# An eval split can leak even when every case ID is unique

You turn a support conversation into two evaluation cases. One asks the original question. The other paraphrases it. If one lands in the tuning set and the other in the holdout, unique case IDs give you a clean report while the underlying incident crosses the split.

I work on products at Lumina. This is a small, fictional check you can run before comparing two versions of a model. The field `incident` identifies the source event that produced the case. It stays the same when a case is paraphrased or annotated again.

Save as `eval_split.py` and run `python3 eval_split.py`.

```python
rows = [
    {"id": "case-01", "incident": "ticket-20", "split": "tuning"},
    {"id": "case-02", "incident": "ticket-20", "split": "holdout"},
    {"id": "case-03", "incident": "ticket-31", "split": "holdout"},
]


def overlapping_incidents(cases):
    tuning = {row["incident"] for row in cases if row["split"] == "tuning"}
    holdout = {row["incident"] for row in cases if row["split"] == "holdout"}
    return tuning & holdout


assert len({row["id"] for row in rows}) == len(rows)
assert overlapping_incidents(rows) == {"ticket-20"}

assignments = {"ticket-20": "tuning", "ticket-31": "holdout"}
grouped = [
    {**row, "split": assignments[row["incident"]]}
    for row in rows
]
assert not overlapping_incidents(grouped)
print("Unique IDs passed; the incident check caught and fixed the overlap.")
```

In practice I would assign the split when the incident first enters the dataset, then persist that assignment with the version. Every new case derived from that incident inherits it. If several tickets are duplicates of one outage, the grouping key may need to be the outage rather than the ticket. The check above catches overlap for the key you supply. It cannot infer missing lineage or discover semantic duplicates by itself.

Keep coverage in view after grouping. A holdout that loses all payment incidents is clean by this check and still misses an important class of failures. Report the mix by split before drawing conclusions from a score.

Our paid [Evaluation Data course at Lumina](https://lumytics.com/courses/offers/data-complete/?utm_source=hashnode&utm_medium=community&utm_campaign=lumina_cursos_2026_11&utm_content=eval_incident_split) includes exercises on keeping evaluation cases traceable as the dataset grows. I built products behind the course. This example runs offline with invented data. Drafted with AI assistance and tested locally.
