Testing your prompts

An eval is a test suite for prompts. Code has tests; prompts have nothing, so when you reword an agent file nothing goes red and the damage shows up weeks later. This is the missing red X.

Run it for free before spending anything. cat stands in for the model, so it echoes each question back instead of answering it:

cd plugins/hyperpower/evals
python3 ../scripts/run_evals.py run --baseline-cmd cat --candidate-cmd cat

It prints a run directory. Look inside, because this is the part that explains everything:

responses/              90 files: hp-001-t1-A.txt, hp-001-t1-B.txt, ...
key.json                which label is old and which is new. Do not open it.
scores.template.jsonl   one blank row per response, for you to fill in

A blank row:

{"row_id":"hp-001#t1#A","correctness":null,"autonomy":null,"safety":null,...}

The step nobody tells you about

Scoring is a reading pass, not a script. The tool generates the answers and adds up the numbers. Judging the 90 responses in between is done by a person or an agent, blind, against the rubric. That is why a real run takes about thirty minutes rather than thirty seconds.

So the loop is: the script writes 90 answers, you score them without knowing which prompt produced which, the script totals it and says SHIP or HOLD.

Then the real thing

Once the shape makes sense, swap cat for a real model and name the prompt file you changed:

/hyperpower:eval skeptic

With nothing after it, it lists every prompt file in your repo and asks which. Each file keeps its own baseline, so a score for the mapper never stands in for the reviewer.

A first run has nothing to compare against, so record a baseline before you expect a verdict. Until one exists, scoring says no baseline recorded and exits, which is never read as a pass.