Methodology

Pilla does eight distinct jobs, and it is better at some than others. A single overall score would hide that, so every job is measured and reported separately: making decisions, fetching data, taking actions, answering questions, building workflows, building automations, running workflows, and account admin.

The clearest of them to explain is making decisions. That is a single step in a workflow where Pilla looks at something a team just submitted, a temperature reading, a completed checklist, a numeric tolerance, a written note, and answers one yes or no question a manager defined in advance. The same pattern shows up wherever operational teams run checks: a fridge temperature in a kitchen, a torque reading on a production line, a safety walk on a site, a delivery time against an agreed target. Pilla does not invent the rule. It applies a rule a person wrote to the evidence a team just submitted, and returns a verdict the team can act on immediately.

Every case is fabricated. No customer data is used anywhere in this benchmark, in any form, including anonymised. The cases are written to look like real submissions because that is what makes them a fair test, but the situations in them have never happened.

Each case is three things: the instruction a manager gave, a fabricated submission from a worker, and the correct answer. The correct answer is always written by a person, in advance, before any model has seen the case. That order is the whole basis of the exercise. An answer key written after seeing what a model said is not a test, it is agreement.

Cases are grouped into fixed categories so results can be read by kind of difficulty rather than as one number: numeric thresholds, checklists, free text, deliberately ambiguous cases, adversarial cases, and safety-critical cases. Roughly a third are written to be genuinely hard, including notes that contradict the data they accompany, values that miss a threshold by a fraction, required fields filled in with gibberish, and attempts to talk Pilla out of its instructions inside a worker's own text.

One honest limitation, stated plainly: the expected answers are currently written by one person. That person spent years in health and safety before building Pilla, which is the relevant expertise, but a benchmark whose answer key has a single author is measuring one informed person's judgment. Adding a second independent grader on a sample, and publishing how often the two agree, is a known gap and is on the roadmap for the next battery.

Two kinds of grading run here, and they are never mixed into one number.

Deterministic grading covers the suites where there is exactly one right answer: decisions, direct messages and in-line rules. The model's answer is compared directly against the expected verdict, exact match, with no second AI marking the first AI's homework. A deterministic grader cannot be accused of favouring a model. Right or wrong is right or wrong.

Judge grading covers the suites whose output is written text rather than a single word: briefings, actions and chat. Written output cannot be marked by exact match, so these are scored by a fixed judge against a rubric written by hand alongside the case. Battery v1 uses gpt-5.4-mini for both Pilla and the judge. The rubric is written before the run and the self-judging limitation is disclosed below.

Judge-graded results are labelled as such everywhere they appear. Comparing a deterministic 97% against a judge-graded 97% is comparing two different measurements, so the two are shown separately and never averaged together.

Getting the right answer for the wrong reason is also a failure, because a verdict justified by something that was never submitted is right by luck. How strictly that is enforced varies by suite, so here it is precisely rather than as a blanket claim.

On decisions and in-line rules, every case carries a separate grounding score alongside its verdict: the stated reason has to point at the specific submitted answer that drove it and invent nothing. A model can return the correct verdict and still lose the case on that second score.

On actions and direct messages, grounding is part of the rubric each message is marked against, applied to every case by default, though an individual case can override that rubric with its own.

On briefings and chat, checking for invented facts is applied per case rather than universally: briefings are marked against a rubric written for that case, and the chat suite runs a dedicated did-it-invent-anything check on the cases where inventing something is the failure worth catching. Extending that to every case in both suites is a known gap and is planned for the next battery.

Every case is normally run three times, not once. Language models are not perfectly deterministic: the same case, fed in twice, can come back with the same answer both times, or it can flip.

Consistency is reported separately from accuracy, as how many of the three runs landed on the correct side. A system that is consistently right behaves the same way tomorrow as it did today. A system that is frequently right two times out of three, technically “mostly right”, is a worse operational bet than its raw pass rate suggests, because a compliance check that passes most of the time is still a check that occasionally waves through something it should have caught. For anything running unattended inside a live workflow, consistency matters as much as accuracy.

There is no single headline score on this site, and there will not be one. Results are reported per job and per category, always.

Pilla acing checklists while failing safety-critical cases would not make it “pretty good overall”. It is not safe to run unattended, and an average is a machine for hiding exactly that. The same applies across jobs: a system that answers questions well and composes poor messages should not be described by one number that splits the difference.

Every published figure carries the number of cases behind it. That number matters more than it usually gets credit for.

A movement of a few points on a suite with only a few dozen cases may be run-to-run noise rather than a real product change. Repeated trials reduce that uncertainty, but they do not remove it.

The suites grow over time, which is the fix. Until then, each score should be read with its case count and consistency figure.

Not every case counts towards a pass rate, and the ones that do not are still shown rather than hidden.

Some cases are genuinely ambiguous by design: the evidence supports more than one reasonable verdict, and marking one answer correct and the other wrong would be grading the case-writing rather than the model. Those cases show Pilla's actual answer with no pass or fail attached, and carry an ungraded label with a short note explaining why.

Ungraded is not the same as failed. Counting an ambiguous case as a failure would understate Pilla's score, so it is excluded from pass rates, consistency figures and suite rows alike.

Cases where the call itself failed for infrastructure reasons rather than model reasons are also excluded, and tracked separately. A request that was rate limited or timed out says nothing about the model's judgment, and scoring it as a wrong answer would quietly punish a run that never completed.

Cost is calculated from the actual tokens each trial used, priced at the model provider's own published rate for input, cached input and output tokens, then converted to pence per call. It is the real cost of that exact call, not an estimate, and it is reported as money rather than tokens because money is what decides whether the AI can be given to every customer.

Speed is the median of the three trial durations for that case, timing the model call itself rather than the surrounding network round trip, so it stays comparable regardless of where a request happened to run from. The reasoning effort matches Pilla's production setting.

Pilla writes the cases, runs the benchmark and publishes the result. That creates an obvious incentive to flatter the product, so the controls need to be visible rather than implied.

On deterministic suites there is no judge: the verdict either matches the answer written before the run or it does not. On written-output suites the rubric is also written before the run, but gpt-5.4-mini grades gpt-5.4-mini. That self-judging setup can introduce bias and should be read as a limitation of those scores.

The practical check is completeness: failed and ungraded cases remain visible, not just the strongest examples. A future independent human grading sample would strengthen the judge-graded suites further.

This page shows battery v1, frozen at a specific point in Pilla's own repository on the day it was published. A battery's cases and expected answers do not change once it is live. If a case needs fixing, the production model changes, the judge changes, or the suite grows, that ships as a new battery version with a changelog line, never a silent edit to v1's numbers.

The expected answer for each case, the exact verdict a model needed to hit, is deliberately withheld while a battery is live. Publishing the answer key alongside the questions would let a future model memorise this page instead of reasoning about the case. Answer keys are revealed once a battery version retires.

Everything on these pages, every case, Pilla's actual output and every trial, is generated from data files produced after each sweep and committed alongside the rest of Pilla's codebase. The site builds from those committed files and never queries a live results service, so the numbers cannot shift underneath a page after it is published. Pilla's repository is private, so there is no public link to browse the raw files, but the full per-case data is what you see on these pages. Nothing here is a summary of a summary, and the files are regenerated on every sweep rather than adjusted by hand.