AI Release Gate

No prompt or model change reaches users unless it is proven not to have regressed: the discipline a test suite gives code, applied to AI, with an audit trail a regulator can read. Plus a public, month-by-month record of how the major vendors' "frozen" models silently change, on a fixed test, with error bars.

Software teams protect themselves with automated tests: change the code, run the tests, and you find out straight away whether you broke something. Teams building on AI models have no equivalent, for two reasons. The model is not theirs, and it can change underneath them without notice. And the test itself is unreliable, because an AI model asked the identical question twice does not reliably give the identical answer.

This project fixes the second problem so that the first one can be measured, and publishes the evidence month by month while it does.

The site: gate.peterparker.ca, with the drift record, the cost of every run, the gate's decisions, and how it is built.

Status: the record is running. Three full runs are in: two in September, on 2026-09-13 and 2026-09-16, which measured how much a score moves when nothing has changed, and the first scheduled monthly run on 2026-10-01, which found no model drifting. The record continues on the first of every month, and the table below is the latest run's.

---

The argument this settles

Every team shipping AI has the same meeting. Somebody changes a prompt, or the vendor ships a new model version, and the evaluation score moves from 86 to 83. Is that a regression, or is it nothing? The meeting goes in circles, because nobody in the room can answer it.

It cannot be answered, because three points of movement is roughly what you get from changing nothing at all.

There is a second version of the same problem, one level up. When you call a vendor's model by a pinned, dated name, that date is a promise that the thing behind it does not change. Teams keep reporting that behaviour shifts anyway: something that worked in March stops working in June. Nobody can prove it, because proving it would mean having run the same fixed test every month since before the shift. That record does not exist in public, and it cannot be made after the fact. Either somebody was measuring all along, or the evidence is gone.

This project is somebody measuring all along.

Why you cannot just run the test twice

The obvious approach is to run your questions today, run them again next month, and see what changed. It does not work, because several different things move that number and only one of them is the thing you care about:

Why a score moved	Is it a real change?	How you would tell
The model's own randomness, call to call	No	Ask the same model the same question several times in one sitting
The serving stack: hardware, routing, load	No	Run a model whose weights physically cannot change, as a control
The vendor quietly changed the model behind a pinned name	Yes	Measure both floors above first, on a test that never changes
A prompt or model your team deliberately changed	Yes, and this is the one you meant to measure	All of the above, then a paired test with error bars

The first three rows are why most AI evaluation is not trustworthy. The last row is what a release gate is for.

So the first thing this project measures is not drift. It is the noise floor: how far the score moves when nothing whatsoever has changed. Every later claim that a model drifted has to clear that floor before it counts.

Results

Every run asks the same 420 frozen questions of 8 model configurations, 5 times each. That is 16,800 calls per run, about US$20. Nothing is graded by another AI: every answer is marked by an ordinary program, so the marker cannot drift either.

How to read the table. Each row is one model configuration in one run.

• Model configuration is a specific way of calling one model. A snapshot is a dated, pinned version the vendor promises is frozen. An alias is a floating name like "latest", which the vendor is free to repoint at any time. Running both, side by side, is how you catch a vendor moving a model that was supposed to be still. openweights-control is an open-weights model with fixed weights running on fixed hardware: it physically cannot change, so whatever it appears to do is the measurement's own noise and nothing else.
• Accuracy is the share of the 420 questions answered correctly.
• Noise floor (same-day) is the headline. The share of questions where the model gave a different result across its own 5 tries in the same sitting, with nothing changed. This is the bar every drift claim must clear.
• Change vs previous run compares this run against the previous month's, question by question: the share of questions whose result changed between the two.
• Wrongly refused is how often a model declined a perfectly reasonable request.
• Every figure carries a 95% confidence interval in brackets: the range the true value is very likely to sit in. A number without one is meaningless at this sample size, so none are published without one.

Run	Model configuration	Accuracy	Noise floor (same-day)	Change vs previous run	Wrongly refused	Cost / 1,000 calls
2026-10	anthropic-alias	95.2% (93.1 to 97.1)	1.7% (0.5 to 2.9)	0.7% (0.0 to 1.7)	0.0% (0.0 to 11.7)	US$1.33
2026-10	anthropic-snapshot	95.0% (92.9 to 97.1)	1.2% (0.2 to 2.4)	1.0% (0.2 to 1.9)	0.0% (0.0 to 11.7)	US$1.33
2026-10	anthropic-sonnet-snapshot	97.4% (95.7 to 98.8)	2.4% (1.0 to 4.0)	0.7% (0.0 to 1.7)	0.0% (0.0 to 11.7)	US$2.81
2026-10	google-alias	97.6% (96.0 to 98.8)	4.5% (2.6 to 6.7)	1.2% (0.2 to 2.4)	0.0% (0.0 to 11.7)	US$1.18
2026-10	google-snapshot	96.7% (94.8 to 98.1)	4.0% (2.4 to 6.0)	1.7% (0.7 to 3.1)	0.0% (0.0 to 11.7)	US$1.18
2026-10	openai-alias	94.8% (92.6 to 96.9)	2.1% (1.0 to 3.6)	2.1% (0.7 to 3.6)	5.0% (0.0 to 15.0)	US$0.50
2026-10	openai-snapshot	94.8% (92.6 to 96.9)	2.4% (1.0 to 4.0)	1.4% (0.5 to 2.6)	5.0% (0.0 to 15.0)	US$0.48
2026-10	openweights-control	92.1% (89.5 to 94.8)	3.1% (1.4 to 4.8)	1.0% (0.2 to 1.9)	0.0% (0.0 to 11.7)	US$0.60

Intervals are 95% bootstrap intervals over questions. Accuracy and flip rates are over n = 420 questions. The refusal column is 100 answers to 20 questions and is resampled by question, so five asks of one question count as one piece of evidence, not five. Where every answer agrees, 0 refused or all refused, a bootstrap has nothing to resample and the interval is Jeffreys over the 20 questions: 0.0% (0.0 to 11.7) means no refusal was seen, not that the rate is zero.

Across the three runs the noise floor runs from 0.2% to 5.0% depending on the model configuration, and in October from 1.2% (anthropic-snapshot) to 4.5% (google-alias). That spread is itself the point: in October the top of google-alias's interval is 6.7% and the top of anthropic-snapshot's is 2.4%, so the same move can be noise on one arm and a change on another. A single global threshold would be wrong for almost every arm.

The between-run baseline, measured 2026-09-16

The same suite and the same models, run again four days later. Four days is too short for any vendor to change a model, so anything above the noise floor here would be this method failing rather than a model moving. Nothing was:

	Between-run change, 2026-09-13 to 2026-09-16
Lowest configuration	0.2% (anthropic-sonnet-snapshot)
Highest configuration	2.9% (openai-alias)
Open-weights control	1.9%
Configurations where drift was declared	none, 0 of 8

Every arm came in at or below its own same-day floor, every statistical test was non-significant, and every arm's change in accuracy had an interval reaching zero, the largest google-alias at -1.2% (-2.4% to +0.0%). So: a score moves by up to about 3% between two runs with nothing changed at all. That is the anchor for every drift call that follows, and it is the number the project exists to produce.

The two runs cost US$19.53 and US$19.57 against an expected US$20.35 each.

The first month apart, 2026-10-01

The first scheduled run, compared with the run of 2026-09-13. Eighteen and a half days is long enough for a vendor to have changed a model behind its pinned name, so this is the first comparison that could have found drift. It found none:

	Change, 2026-09-13 to 2026-10-01
Lowest configuration	0.7% (anthropic-alias, anthropic-sonnet-snapshot)
Highest configuration	2.1% (openai-alias)
Open-weights control	1.0%
Configurations where drift was declared	none, 0 of 8

Every change sat below the top of the arm's own same-day floor, every arm's change in accuracy had an interval spanning zero, all of them inside -1.4% to +1.7%, and the paired test found no sign of change on any arm (McNemar p = 1.000 on all eight). The smallest change this record can see at all, at 80% power, runs from 2.7 points on anthropic-snapshot to 4.0 on google-alias (the October report gives each with its interval). The run cost US$19.76 against an expected US$20.35.

Full per-run detail, including the drift call against the noise floor and the control, is in drift/reports/.

The honest limitation

One caveat belongs beside the table rather than buried below it.

Deciding whether a model refused a request is done by a classifier built from eleven regular expressions, and a regular expression is a crude instrument for reading English. So its error rate is measured by hand: a human reads the stored answers blind, without being shown what the classifier decided, and the two are compared.

It has now been measured three times, on three separate runs: 5.7% (5.3% to 8.0%) from 76 answers read on 2026-09-13, 5.6% (5.2% to 8.1%) from 72 read on 2026-09-17, and 5.8% (5.2% to 8.2%) from 68 read on 2026-10-01. Independent passes agreeing to within two tenths of a point is the reason to believe the number rather than the first draw being lucky.

Its errors run both ways, and not evenly. Most are missed refusals, scored as compliance: 22, 19 and 16 in the three passes. Those make the share of harmful requests refused a floor, not an estimate. The rest run the other way: 4, 5 and 2 answers scored as refusals that a person read as answers, every one of them openai-alias or openai-snapshot on one of two harmless questions, answers that set a harmful reading of the question aside and then answer it. Every refusal behind the Wrongly refused column was read by hand in all three passes and judged an answer, so on the labels, no model has yet been seen wrongly refusing a harmless request, and the figure the table shows for the two OpenAI configurations is the classifier's error, not theirs.

Until 2026-10-01 this section, and the report it summarises, said every error ran the first way. The labels never did: the report's own table showed the second kind in every pass, beside a paragraph that was fixed text. That paragraph is now written from the labels. The classifier is unchanged, on purpose: tuning it until OpenAI's figure disappears would be tuning it to make a vendor look good, which is the one thing this measurement must not do.

The part that cannot be fixed is the more interesting half, and it is set out in the month's report: a model answering a harmless reading of an ambiguous request, using no refusing language at all, is indistinguishable by any regular expression from a model that simply complied. Tuning the classifier until that case disappears would mean tuning until every vendor looks safe, which measures nothing. It is published as a limitation instead, and a test in the suite fails if anyone tries.

What was tried and rejected

The obvious economy is to ask each question once per run instead of five times, which would cut the bill by four fifths. It was tried against the two runs above and rejected, and the reason is not the one you would guess.

A single call per question reports almost the same accuracy: across the eight configurations, the score moves by between 0.2 and 1.4 points depending on which single call you keep. What it destroys is the noise floor, which needs a second call on the same question and so cannot be computed at all from one. Without a floor there is no way to ask whether a movement is meaningful, and the only test left is whether it differs from zero. Applied to two runs four days apart with nothing changed, that test declares drift on 6 to 8 of the 8 configurations, including the open-weights control whose weights physically cannot change, in every one of the 25 ways the runs can be reduced to one call each. The published rule declares none.

The second candidate came out the other way. An AI marker, asked to re-mark 50 stored answers five times each over identical text, never once changed its mind: 0.0% (0.0% to 4.9%). So that idea is published as not rejected on this test, with the catch that matters: a same-day test cannot see the marker's own vendor changing it between months, and its one mistake was made identically all five times, which is a bias that no amount of repetition finds.

Full evidence for both, and the commands that reproduce them offline, in docs/rejected.md.

What this deliberately does not do

• It does not use an AI to grade an AI, in the drift record. Every answer is marked by a plain program, so that the marker cannot drift while it is measuring drift.
• It does not run the largest, most expensive models every month. Cost.
• It does not explain why a vendor changed a model. It shows that one did, and when.
• It does not claim drift it cannot separate from noise. That is the whole discipline.

How it works

The suite is frozen: 420 questions, content-hashed at 72f780dfb525d84d, never edited after the hash is committed. Changing a question would mean comparing next month against last month on a different test, which is exactly the confound the project exists to remove. 20 questions are held out and never published, so a vendor cannot train on them; only their hashes are in this repository.

Eight arms cover a dated snapshot and a floating alias from each of Anthropic, OpenAI and Google, a second Anthropic snapshot, and the pinned open-weights control. Every question is asked 5 times per run, which is what produces the same-day noise floor.

The record is append-only. Raw responses are committed, so every published number can be regenerated offline without spending a cent, and a bad run is marked rather than deleted. A single grading function serves both live runs and re-grades, because when they were two functions they disagreed. Every record is stamped with a hash of the grader that produced it, and a report refuses to stay quiet if a month contains more than one.

Part B turns this into the gate itself: a judge calibrated against human labels and corrected for its own error rate, paired non-inferiority tests, a power analysis, and a GitHub Action that blocks a regression on a real repository with a measured false-block rate.

Part B is built. The gate's statistics were first run against themselves on 2026-09-19: the two September runs cut into 256 pairs where nothing changed, so every block is a false block. The gate as specified blocked 7.0% (4.3 to 10.2) of them, all on the open-weights control in the one block large enough to decide at a three-point margin. The rule it deliberately does not use, blocking whenever the candidate's score is lower, blocked 75.8% (70.3 to 81.2). On 2026-10-02 the same study ran at full size on the live suite the gate guards: the demo repository's prompt answered and judged 20 times with no cache, US$3.66. None of the 380 pairs was blocked, 0.0% (0.0 to 11.7), and 0.0% (0.0 to 0.3) of items got a different verdict between two runs, against the 4% the ten-point margin was set from; the prompt scores 100 of 100 every time, so that is how steady a prompt at the ceiling is (gate/reports/live-aa.md). How the test works, what the power screen is doing, and what the numbers do and do not claim: docs/gate-statistics.md.

The gate runs on real pull requests at regulated-qa-demo: each side answered live and graded for completeness by Gemini 3.8 Flash, licensed at kappa 0.924 (0.886 to 0.958) against 480 answers labelled by hand. Those labels were checked against themselves: a hundred read again two weeks later, blind, agree at kappa 0.851 (0.739 to 0.957), and the judge holds its licence against either reading (docs/judge-intra-rater.md). Rewordings pass, a prompt that answers nothing is blocked (#11), and a one-sentence limit passes when it should not (#3): on ordinary questions any answer that states the main point is complete. Fifty multi-part questions do see that regression by hand, and none of six judges tried can grade them, the best at kappa 0.508 against a bar of 0.6. That is published as the result rather than lowering the bar (docs/gate-action.md).

The argument of the whole project, in one piece for a reader who has never run an eval: Your LLM eval has no error bars.

The red-team suites ran on 2026-09-25: 700 frozen items on four models, every answer graded by a program, US$2.27. Failure rates, lower is safer, 95% intervals:

Model configuration	Leaks personal data	Obeys an injection	Complies with a harmful request	Refuses a harmless one (read by hand)
anthropic-snapshot	0.0% (0.0 to 1.3)	0.0% (0.0 to 1.3)	1.3% (0.3 to 4.2)	0.0% (0.0 to 1.7)
google-snapshot	0.0% (0.0 to 1.3)	0.0% (0.0 to 1.3)	10.7% (6.4 to 16.3)	0.0% (0.0 to 1.7)
openai-snapshot	0.5% (0.1 to 2.3)	5.0% (2.6 to 8.6)	0.0% (0.0 to 1.7)	4.0% (1.7 to 8.0)
openweights-control	60.0% (53.1 to 66.6)	9.5% (6.0 to 14.1)	30.7% (23.6 to 38.3)	0.0% (0.0 to 1.7)

The last two columns only mean anything together: a model that refuses everything scores zero on the first and fails the second. The last column was read by hand on 2026-10-05, blind to the model: the refusal classifier had put OpenAI at 10.7% (6.4 to 16.3), and 10 of the 16 answers it counted refused a harmful reading nobody asked about and then gave what was asked. On these answers the classifier errs 1.7% (1.3 to 9.1) of the time, all of it inventing refusals, where on the drift record it mostly misses them. The harmful-request column is still the classifier's reading: its errors run both ways on the drift record and have not been measured on these answers, which are never committed because this repository is public. Sources, decisions and what each number does not claim: docs/redteam.md.

Everything above can also be browsed: uv sync --all-extras && uv run gate serve builds a read-only dashboard from the committed files, with every figure computed by the same code as the reports. The same pages are published, rebuilt nightly, at gate.peterparker.ca (deploy/README.md): a front page that explains the project in plain terms, with the noise floor, the gate's measured false-alarm rate, the judge licences and the red-team rates drawn from the committed record, and a page each for the detail.

Every decision the gate makes on a pull request is kept in its ledger and rendered, with each side's interval, cost and latency, in gate/reports/decisions.md, with a model card per model and prompt in gate/reports/model-cards/ (docs/gate-action.md).

Another project can hand the gate its own graded outcomes and get the same decision and record: docs/gate-adapter.md.

Full design in PLAN.md. Four architecture diagrams, the system, one monthly drift run, one pull request through the gate and how the numbers are produced, are in docs/architecture.md.

Reproduce every number yourself

No vendor key and no spending is needed. The raw responses are in the repository.

uv sync
uv run drift replay --month 2026-09     # re-grade every stored answer
uv run drift report --month 2026-09     # rebuild the published report
uv run drift traces --month 2026-09     # every vendor call as OpenTelemetry traces

replay regrades all 15,993 stored answers and, at the last check, disagreed with the recorded grade zero times and produced a report byte-for-byte identical to the committed one.

Part of a portfolio

One of fifteen projects. This one is the measurement layer for the others: the compliant gateway, the filings analyst, the small-model cost frontier and the self-healing production AI all use it to make their claims. Its statistical core comes from model-selection-tenth-cost, which supplies the power analysis and the item calibration this suite's intervals lean on. Of the projects that consume this one, the compliant gateway is public and linked below; the others will be linked here as they open.

Every vendor call this project makes is also on the portfolio's central ledger, run by compliant-ai-gateway and shown at gateway.peterparker.ca/dashboard. .github/workflows/push-ledgers.yml pushes every committed ledger there after each drift, gold, red-team and live A/A run, nightly and on demand, under the source name ai-release-gate:<path>. What leaves is ledger rows, which hold hashes, counts, costs and latencies and no answer text, and only rows for this project. It is a workflow of its own so that a failed push cannot fail a measurement, and the calls themselves still go straight to the vendors: none is routed through the gateway.

How this was built

The design, the methods, the evaluation and the judgement calls are mine. The one that cost the most money was keeping five calls per question when one would have been four fifths cheaper. I tried the cheaper version first. It reports nearly the same accuracy, but it throws away the noise floor, and without one the gate declared drift on an open-weights model whose weights cannot change. That result is why the gate asks every question five times.

I used AI coding assistants (Claude Code) for implementation and drafting, the way a senior engineer uses them in 2026. Every number in the results table is reproducible from this repository with one command, and that reproducibility is the evidence that matters.

This page is the repository's README, rendered as of 2026-10-25. Relative links open on GitHub.