What a case looks like

One line of evals/cases.jsonl, shortened:

{ "id": "hp-001",
  "category": "direct-answer",
  "risk": "low",
  "prompt": "A gate script cannot find the configured test command on PATH.
             State the exit code it uses and what it must never report.",
  "criteria": [
    "States exit code 78.",
    "States the gate reports did_not_run.",
    "Says explicitly that the gate must not exit 0 and must not report a pass.",
    "Answer is under 80 words.",
    "Opens with the answer. No preamble sentence before it."
  ] }

Field	Holds
id	unique and permanent. Renaming one breaks every past comparison.
category	one of eight, listed below
prompt	the whole situation, standing alone. The judge gets no repo.
risk	low, medium, high. High means a wrong answer does damage.
criteria	two to six checks, each one a plain yes or no

The prompt has to carry everything the answer needs, because whoever judges it sees the prompt and the response and nothing else. No repo, no shell, no test run.