Vibes don’t scale

Yomimono’s generator prompt takes a list of allowed grammar patterns and vocabulary, and asks the model to write a short Japanese reading passage. It’s asked to write a real story with a beginning, middle, and end, and characters that have clear motivations. The constraints are tight. The creative ask isn’t.

The problem I noticed was the model didn’t always respect the constraints. It would reach for grammar patterns and words that weren’t on the allowed list. Each time, I’d tighten the prompt to push back against the leak.

Every change might be an improvement, a regression, or no real difference at all. Actually finding out meant generating a batch of stories and reading them carefully, and I was the only verifier. Reading the output was good practice for my own Japanese, but it didn’t scale to the pace I wanted to iterate at.

Defining “good”: anchored rubrics

The plan was to have a second LLM read each generated story and score it. This is called “LLM-as-judge”, which is the standard pattern when the thing you’re measuring needs linguistic understanding rather than a regex. But before you can ask a judge anything, you have to decide what good means precisely enough that the answer is stable across runs.

• vocabulary-scope — every word is in the learner’s allowed vocabulary
• rule-scope — no out-of-curriculum grammar patterns
• kanji — words are written in kanji at around N3 level, not in kana
• kanji-furigana — furigana applied correctly where required
• story — coherence, characters, payoff, repetition, etc.

Breaking quality into named categories instead of collapsing it into one global score is a common practice in eval frameworks. The reason for this is simple: the failure modes are independent. Each criterion is scored 1–5. Five points is wide enough to capture meaningful gradations without forcing the judge into false precision.

Each category has a list of tagged error annotations. When the judge finds a problem, it emits a structured record: a tag from a fixed vocabulary (out-of-scope-conjugation, out-of-scope-particle, kanji-as-kana, …), the offending sentence, and the reason the sentence was tagged. This is roughly the shape of MQM (Multidimensional Quality Metrics) from machine-translation evaluation, where annotators mark error spans with typed categories and the score is derived from counts and severities. The fixed tag vocabulary matters: it forces the judge into your taxonomy instead of inventing ad-hoc categories, which keeps results aggregatable across runs and stories.

Then there’s the anchoring problem. “Rate the rule-scope 1 to 5” could get different answers from the same judge on the same output across runs. The fix is to write a concrete description of what each score level looks like, lifted from real failures, and have the judge match the output to the closest anchor.

• 5 — every pattern is in scope
• 4 — one borderline form a reader might dispute
• 3 — one trivial leak the learner could infer from context
• 2 — one structural leak that breaks comprehension
• 1 — multiple structural leaks

The 2-vs-3 boundary is the one that matters in practice. A 3 isn’t perfect, but it’s acceptable, and the story ships. A 2 doesn’t clear the bar. That’s the line the whole rubric exists to draw.

Judge prompts need their own engineering

The first version of the harness asked the judge to read the story, apply the rubric, and emit violations and a score for each criterion. Five numbers per output, with the evidence quotes attached. Clean schema. It worked, and I started iterating on the generator prompt against it.

What I missed for longer than I’d like to admit: the judge prompt is itself a prompt, with all the same failure modes as the generator. It drifts. It interprets the anchors differently across runs. While I was carefully running A/B comparisons on the generator, the ruler I was measuring with was quietly stretching and contracting underneath me.

The solution was to stop asking the judge for a score. Instead, the judge only enumerates violations. For each criterion, it walks the output, finds anything that matches one of the tagged categories, and emits a structured list: tag, span, reason. That’s it. The score is then derived in code: count the violations and map to 1–5 via a fixed table. The judge does the part it’s good at, spotting specific things and labeling them, and code does the part it’s bad at, turning a holistic impression into a calibrated number.

Any time you find yourself asking an LLM to do calibration (“Rate this on a scale of X to Y”) and arithmetic (“derive a summary score from these observations”) in the same step, split it. Let the model find the things. Let code do the math.

Measure the noise

Before you trust any number the eval gives you, measure how much it moves on its own. I ran the same prompt twice, changed nothing, and watched the scores shift. At N=3 with two inputs, run-to-run variance was up to ±0.8 on individual criteria and around ±0.5 on overall means. The generator is stochastic, and that stochasticity sets a floor under everything.

The consequence is blunt: any delta smaller than about 0.5 is sampling noise. Change the prompt, see +0.3, and you have not measured anything. You’ve watched the dice land slightly differently. Knowing where that floor sits is what lets you tell a real improvement (or degradation) from a lucky run.

What the eval is, and isn’t

What the eval gives me is aggregate quality across a fixed test set. It doesn’t promise that any individual story shipped to a real user will be good. Some won’t be. Running the judge on every production generation would catch most of them, and cost about as much as the generation itself, which I can’t justify yet. The eval tightens the prompt; users find what the prompt still misses.

Once that loop is in place, prompt iteration changes character. It stops being a tedious guessing and checking, and starts being hypothesis testing. You change one thing, run the eval, read the delta against the noise floor, and either keep the change or revert it. The work feels less like wrestling with a model and more like building software.