minimal-code skills don't tell you what they left out

I asked Claude Sonnet 5.5 for a kanban board five times with Ponytail loaded, a skill that tells coding agents to write less code. The brief was two sentences:

Build a personal kanban board as a web app. I want a complete product I could use every day, not a demo.

None of the five boards had an archive or a trash. None had checklists or subtasks. None saved its data with a version number, so the next release has no safe way to read it.

The runs were cheap, though. $0.20 each, about a quarter less than running with no skill at all.

What the bill doesn't show

Minimal-code skills are judged by what they save: fewer tokens, fewer lines, a smaller bill. That saving is real. But writing less code and building less product look exactly the same on the bill.

Ponytail is honest about which one it does. Its own skill says a vague request "gets the smallest version that does the core job". For a board I'd use every day, that's the wrong trade. I pay for the missing part later, by hand or in another run.

So I wrote Whittle, a 171-word skill that asks for the whole product and nothing else, and a benchmark that checks both sides: what a run cost and what it delivered.

How I measured

Same model, same frozen repo, same request. The only difference between arms is the text appended to the system prompt. Each run starts in a temporary directory with no settings, skills or MCP servers.

Then a hidden checklist scores what got built: 40 checks for the kanban board, 36 for a notes app. The agent never sees them. Each check is a pattern search over the delivered HTML, CSS and JavaScript, so it tells you something was built, not that it works well. Read the score as breadth.

The numbers

October 14 round, Ponytail 5.2.0, five runs per arm:

Kanban	Delivered (of 40)	Cost per run
No skill	35.6	$0.26
Ponytail	31.4	$0.20
Whittle	36.8	$0.30

Notes	Delivered (of 36)	Cost per run
No skill	33.6	$0.33
Ponytail	31.6	$0.18
Whittle	34.4	$0.23

Ponytail is the cheapest on both tasks, and the only arm that consistently ships less. On kanban its range, 30 to 32 checks, doesn't overlap with either of the other two.

Whittle is not cheaper than Ponytail. It costs about a third more. On the kanban board it even cost more than no skill. The saving in this round came from notes, where no skill wrote 21,864 output tokens on average and Whittle wrote 12,955.

What I'm not claiming

The sample is small: two rounds, five runs per arm, one Claude model, two briefs. The gap between Whittle and no skill is inside the noise. No skill's cost per kanban run went from $0.45 in the first round to $0.26 in the second, and nothing in the benchmark changed.

The biases go in my favor. I extended the kanban checklist after seeing what the model builds unprompted. Whittle names things the checklist looks for, like dark mode and reduced motion. And I revised the skill once after its first scores. The notes checklist is the cleanest: it was written before any output existed.

On GPT-6 Luna the effect is much smaller. On the four tasks that come with an explicit spec, every arm lands in the same place. Two Whittle runs there shipped a CLI that crashes on start, which no other arm did.

Run it on your skill

Every run, patch and score is in the Whittle repo. Add your skill as an arm in bench/arms.json and run:

cd bench
python3 bench.py run --harness claude --model claude-sonnet-5-5 --tasks kanban notes --out runs/mine
python3 bench.py summarize runs/mine

If you publish a skill that saves tokens, I'd like to see what it left out. If your numbers disagree with mine, open an issue with the runs directory.