I hand-labelled 149 examples before I trusted my own scorer

I built a system that reads long documents and scores them against a structured profile. The generation part took an afternoon. Deciding whether it was any good took considerably longer, and that part is the only reason I trust it now.

This is a short account of building a hand-labelled evaluation set, because it is the piece of LLM work I see written about least and asked about most.

Why eyeballing fails

The first version felt good. I read a dozen outputs, they seemed sensible, and I moved on.

That feeling is worth nothing, for a specific reason: when you read your own system’s output you are reading it charitably. You know what it meant. You fill gaps without noticing. And you sample the cases you happen to think of, which are the cases you designed for.

The failure mode of a scoring system is not that it is obviously wrong. It is that it is quietly wrong in a narrow band you never look at.

Labelling by hand, badly at first

So I scored documents myself. No model in the loop, just me and a number from 0 to 100, recorded before I looked at what the system said.

The first thing I learned is that I was not consistent with myself. Two similar documents, scored on different days, came out eleven points apart. If the ground truth is noisy, no amount of tuning downstream is meaningful.

Fixing that meant writing down what the number actually meant. Not a rubric with weights, just a few sentences per band: what makes something an 80 rather than a 60, and what disqualifies a document entirely. Once the definition existed, my own repeat scores tightened considerably.

That is the real first output of an eval set: a written definition of what good means. The labels are downstream of it.

149

I stopped at 149 examples. Not a round number because it was not a target; it was where the marginal example stopped changing anything.

That is smaller than people expect. A few hundred carefully labelled examples will tell you more than tens of thousands of examples labelled by a model that shares your system’s blind spots. Small and honest beats large and circular.

I kept the set deliberately unbalanced toward the boundary. Obvious accepts and obvious rejects teach you nothing; the examples worth labelling are the ones you had to think about.

Calibrating against it

With ground truth in hand, the scorer became measurable. I used two numbers:

Mean absolute error, because it is in the same units as the thing I care about. MAE 7.4 means that on average my scorer is about seven points away from where I would have put it, on a 100-point scale. I can reason about that. I cannot reason about a loss value.

Rank correlation, because the absolute number often matters less than the ordering. If the top twenty are the right top twenty, a systematic offset is survivable.

Those two disagree in a useful way. A scorer can have decent MAE and poor ordering, which usually means it has learned the average and not the signal.

What the disagreements taught me

The interesting part of the exercise is not the score. It is reading every case where the scorer and I differed by more than fifteen points.

Almost every large disagreement pointed at a missing feature, not a broken model. The scorer had no way to see something I was using unconsciously. Once named, it became a rule, and the rule was cheap, deterministic and fast, which meant fewer model calls rather than more.

That is the pattern I would take to any similar system: large disagreements are feature requests.

Rules first, model second

The system I ended up with rejects the overwhelming majority of inputs deterministically, before any model is involved, on criteria that are not judgement calls at all. Only what survives is worth spending a model call on.

That ordering came directly from the eval set. Without it I would have reached for a bigger model, which is the expensive way to fix a problem that was mostly definitional.

What I would tell someone starting

Write down what good means before you label anything. Label a hundred or so yourself, weighted toward the cases you find genuinely hard. Report error in units you can argue about. Read every big disagreement, because that is where the actual engineering is. And keep the set: the value compounds every time you change the system, because for the first time you can answer “did that help” with something other than a feeling.

Generation is commoditised. Knowing whether your output got better is not.