Each figure is re-measured on each release, on real, public code. Where a figure can be verified in the open, the row links there.
Every number on this site, and how it was measured.
| Figure | Value | How it was measured | Verify |
|---|---|---|---|
| Free: apps where the check reaches a verdict | 5 of 10 | Ten real Flask apps, each with one planted sign-in flaw. A fresh agent sets Assure up from the one-liner; the figure is the apps where the check reaches a real reading and names the flaw. | the repo |
| False alarms | 0 | A broken rule reported on code that keeps it, counted on every app in the benchmark and the free gate. | the repo |
| Planted defects caught in open-source apps | 4 of 6 | Open-source PostgreSQL and HTTP apps, each read with a known defect planted; the defects the check reports as broken, with no false alarm on the rules that hold. | |
| Known PostgreSQL CVE exposures caught | 9 of 17 | Seventeen known PostgreSQL CVE exposures: the version in the affected range and the precondition observed, nothing executed; the exposures the check names, against a version-only check that flags all 17 with 17 false alarms. | |
| Assure on Assure: own promises decided | 53 of 61 | Assure's own requirements.md, read by the same check that reads yours; the promises that read holds or disproven rather than not established. | the repo |
| Assure on Assure: engine | 0782f26c | The engine the self-check ran on. | |
| Assure on Assure: date | 2026-10-25 | When the self-check ran. | |
| Assure on Assure: promises | 61 | Every line of Assure's own requirements.md. | |
| Assure on Assure: hold | 51 | Promises the self-check reads as holds. | |
| Assure on Assure: not established | 8 | Promises the self-check cannot decide yet, each with its reason. | |
| Assure on Assure: disproven | 2 | Promises the self-check reads as broken; each has a fix in flight. | |
| Time to the first pull-request comment | 3.6 minutes | Fresh agents, from the one-liner, on ten apps; the median from the one-liner to the first comment on the pull request. | the repo |
| Planted flaws caught by Assure Free | 5 | One route reachable without signing in, planted in each of ten real Flask apps; the apps where Assure names it. | the repo |
| Pull requests merged a day | a median of 66 (peak 130) | Pull requests merged into Assure's own main branch, counted per Sydney day. | |
| Internal development acceleration | 66x | Reviewed changes merged a day into Assure's own main branch: from 1 on 30 Sep to a median of 66 a day over 13–19 Oct (peak 130). | |
| Planted flaws caught by Bandit | 0 | The same ten apps and plants, Bandit at its default rules, on Python 3.12. | the repo |
| Planted flaws caught by CodeQL | 0 | The same ten apps and plants, CodeQL with the default Python query suite. | the repo |
| Bandit findings on the clean code | 275 | Bandit on the ten apps before the plant; every finding itemised, none assessed. | the repo |
| CodeQL findings on the clean code | 11 | CodeQL on the ten apps before the plant; every finding itemised, none assessed. | the repo |
| Translation consistency | 99.3% | A frozen set of 143 requirement lines, each translated and compared with its reference reading; the share that match, with unsafe rows counted separately. | |
| Cost of one check | about USD 0.06 | Assure's own app, 68 files, 2.2 MB, with app_roots, at the metering rate. | /docs/data |
| Checks on the free credit | about 90 | The same app, divided into the free credit. | |
| Free plan input limit | 6 MiB | The published limit, which equals the enforced one. | /docs/data |
| Premium plan input limit | 24 MiB | The published limit, which equals the enforced one. | /docs/data |
| Pro plan input limit | 64 MiB | The published limit, which equals the enforced one. | /docs/data |
| Free credit | USD 5 | From the API quick start. | /docs/quickstart-api |
| Premium, a month | USD 20 | From the API quick start. | /pricing |
| Pro, a month | USD 200 | From the API quick start. | /pricing |
| Premium monthly pool of usage | USD 20 | From the API quick start. | |
| Pro monthly pool of usage | USD 200 | From Your data. | |
| Metering rate | USD 0.10 per million input tokens | From the API quick start. | /docs/quickstart-api |
| Verdict retention | 30 days | From the Action quick start. |
What the release carries
- Input limits by plan: Free 6 MiB, Premium 24 MiB, Pro 64 MiB. Enterprise above that.
- Assure sends only the app's own code, so a check costs cents. A repository over its plan's limit still gets its app checked.
- On Premium the comment carries the formal result and a link to the proof tree.
- Pro runs the Premium build.
- The written report, in plain words, and a line that says what the next plan would check.
- One line to your agent sets Assure up.
- Setup is one command. The agent gets the declaration, a requirements.md tagged by tier and the workflow, and opens the first pull request itself, already checked.
- Requirement lines are translated by default, with zero data retention.
- An adapter bridge lets your agent map an app shape Assure does not read yet onto one it does; Assure re-checks the map on every run.
- A fact only you know, stated in the declaration, reads as holds, assuming it (stated by you).
- Every gap offers a coverage request from the refusal itself.
For the detail
Provenance for every figure: the engine it was measured on, the date, the record and its LAUNCH-FIGURES row. A figure marked release was measured on the engine customers run; one marked projected was measured on main or an integration tree and is re-measured on the release; one with no record is withheld and appears nowhere on the site.
| Figure | Value | Status | Engine | Date (UTC) | Source | Row |
|---|---|---|---|---|---|---|
free_setup_qual | 5 of 10 5 of 10 on the release train (microblog, cookiecutter-demo, jwt-restful, rest-tutorial, flaskr; 0 false Broken), the Director's approved claim for today (trial lane, 20 Oct); 4 of 10 honest on main 7d737d6d; the honest range by launch is 6–8; Thursday's run sets the value | projected | the release train (main 366f011c) | 2026-10-19 | PRODUCTION-WAVE.md; free-wave/gate/pw-main-366f011c/SUMMARY.txt (d543bee06fe4574b) | 1 |
false_alarms | 0 | projected | as rows 1 and 3 | 2026-10-19 | the false-Broken line of the free gate, the false-alarm line of the depth harness, and the Assure table of bench run-002 (9ccdeae750cdec73) | 4 |
ground_truth_caught | 4 of 6 caught 4 of 6 expected disproven; 0 false alarms; 3 should hold; 2 pending (G2-PA-MB-API, G2-PA-AL-AJAX); 3 must-not-hold vacuous; served on production after the v0.1.0 switch (runs from 11:28Z, cases generated 11:46:24Z) | release | production v0.1.0 6539d90d | 2026-10-21 | experiment-arm-001/corpus/GROUND-TRUTH.json (38051d4a4e47a734), run record experiment-arm-001/corpus/GROUND-TRUTH-CASES.json; engine 6539d90da7c699e87787b1621e3eee8f14440d3b, served true | corpus |
cve_bench_caught | 9 of 17 caught 9 of 17; 0 false alarms (version-only baseline: 17 caught, 17 false alarms); 34 should hold; aggregate only; served on production after the v0.1.0 switch (cases generated 11:47:52Z) | release | production v0.1.0 6539d90d | 2026-10-21 | experiment-arm-001/corpus/CVE-GROUND-TRUTH.json (0db127000cc28648), corpus cve-bench-pg; engine 6539d90da7c699e87787b1621e3eee8f14440d3b, served true | corpus |
self_promises | 53 of 61 53 of 61 decided on v0.1.4 (0782f26c), 2026-10-25: 51 hold, 8 not established, 2 disproven | release | the newest self-check release | its meta.json date | content/self-check/0782f26c/meta.json (/self-check/0782f26c), read at build time | 2 |
self_engine | 0782f26c 53 of 61 decided on v0.1.4 (0782f26c), 2026-10-25: 51 hold, 8 not established, 2 disproven | release | the newest self-check release | its meta.json date | content/self-check/0782f26c/meta.json (/self-check/0782f26c), read at build time | 2 |
self_date | 2026-10-25 53 of 61 decided on v0.1.4 (0782f26c), 2026-10-25: 51 hold, 8 not established, 2 disproven | release | the newest self-check release | its meta.json date | content/self-check/0782f26c/meta.json (/self-check/0782f26c), read at build time | 2 |
self_total | 61 53 of 61 decided on v0.1.4 (0782f26c), 2026-10-25: 51 hold, 8 not established, 2 disproven | release | the newest self-check release | its meta.json date | content/self-check/0782f26c/meta.json (/self-check/0782f26c), read at build time | 2 |
self_holds | 51 53 of 61 decided on v0.1.4 (0782f26c), 2026-10-25: 51 hold, 8 not established, 2 disproven | release | the newest self-check release | its meta.json date | content/self-check/0782f26c/meta.json (/self-check/0782f26c), read at build time | 2 |
self_ne | 8 53 of 61 decided on v0.1.4 (0782f26c), 2026-10-25: 51 hold, 8 not established, 2 disproven | release | the newest self-check release | its meta.json date | content/self-check/0782f26c/meta.json (/self-check/0782f26c), read at build time | 2 |
self_disproven | 2 53 of 61 decided on v0.1.4 (0782f26c), 2026-10-25: 51 hold, 8 not established, 2 disproven | release | the newest self-check release | its meta.json date | content/self-check/0782f26c/meta.json (/self-check/0782f26c), read at build time | 2 |
premium_depth_ratio | — 38 vs 11 of 74 promises decided, Premium against Free; 0 false alarms; 0 misses | withheld | candidate e7aaf97e (integration) | 2026-10-18 | experiment-arm-001/depth/runs/candidate10-r1-v5/DEPTH.md (53d8a69e53021d46) | 3 |
premium_setup_qual | 7 withheld 22 Oct 2026 (chief-of-staff 4799, trial 4768): 7 of 10 was an unreleased local build (875ba2b5); the release candidate read 4 of 10. Returns with Friday's measured Premium run, release named. 7 of 10 on candidate10c-r1-v5, projected for the release (the Director, 19 Oct); the release candidate read 4 of 10 on 19 Oct; 8 of 10 comes after launch with #4720(c). Thursday's measurement replaces this | withheld | candidate10c-r1-v5 (integration) | 2026-10-18 | experiment-arm-001/premium-depth-wave/RECORD.md:52 (c488552a91e82c59) | 10 |
time_to_first_comment | 3.6 minutes median, bar 15 minutes; comments posted on 10 of 10 apps | release | production beta a2555bb6 | 2026-10-17 | experiment-arm-001/setup-qual/runs/20261017T233534Z-beta/SUMMARY.md (91e7c645d40a3c50) | 5 |
ten_x_ci | — | withheld | no record found (19 Oct) | 6 | ||
bytes_per_check | — | withheld | owed: production-wave/seats/payload-measure/MEASUREMENT.md | 7 | ||
bench_assure | 5 5 of 10 on the release train, the same reading as row 1 (the Director's approved claim, 20 Oct); 4 of 10 on main 7345ed0a; 1 of 10 on production 7f7b0f24; 0 false alarms on each | projected | the release train (main 366f011c) | 2026-10-20 | experiment-arm-001/production-wave/bench/run-002/RESULT.md (9ccdeae750cdec73) | 9 |
prs_per_day | a median of 66 (peak 130) 13–19 Oct: 92, 41, 39, 42, 66, 130, 90 a day (median 66, peak 130 on 18 Oct); the trial lane's preliminary launch figure is about 100, set on Thursday | projected | GitHub search API, Assure main | 2026-10-19 | LAUNCH-FIGURES row 15; the Director, 20 Oct: PRs merged per day is a key internal metric; the lane's launch-site finalisation, 20 Oct | 15 |
dev_acceleration | 66x 1 reviewed change merged on 30 Sep; 84 on 10 Oct (the founder's showcase chart); 13–19 Oct: 92, 41, 39, 42, 66, 130, 90 a day (row 15, median 66). The Director's framing, 20 Oct: '100x internal development acceleration'; this week's figure is preliminary | projected | GitHub search API, Assure main | 2026-10-20 | the Director, 20 Oct ('change this to 100x internal development acceleration'); LAUNCH-FIGURES row 15; the trial lane's finalisation of 20 Oct had asked for no 100x, overridden by the Director | 15 |
competitor_bandit | 0 | release | Bandit 1.8.6 (proof-7f7b0f24) | 2026-10-19 | trial-records-20261019/public-bench/assure-benchmark/results/proof-7f7b0f24/SUMMARY.md (7af1097d6b96bfe76dd097c5b2312f2ad5b42749c0a2e487f6cf27e67ef84088) | 9 |
competitor_codeql | 0 | release | CodeQL 2.27.1 (proof-7f7b0f24) | 2026-10-19 | trial-records-20261019/public-bench/assure-benchmark/results/proof-7f7b0f24/SUMMARY.md (7af1097d6b96bfe76dd097c5b2312f2ad5b42749c0a2e487f6cf27e67ef84088) | 9 |
bandit_clean_findings | 275 | release | Bandit 1.8.6 (proof-7f7b0f24) | 2026-10-19 | trial-records-20261019/public-bench/assure-benchmark/results/proof-7f7b0f24/SUMMARY.md (7af1097d6b96bfe76dd097c5b2312f2ad5b42749c0a2e487f6cf27e67ef84088) | 9 |
codeql_clean_findings | 11 | release | CodeQL 2.27.1 (proof-7f7b0f24) | 2026-10-19 | trial-records-20261019/public-bench/assure-benchmark/results/proof-7f7b0f24/SUMMARY.md (7af1097d6b96bfe76dd097c5b2312f2ad5b42749c0a2e487f6cf27e67ef84088) | 9 |
translation_consistency | 99.3% | release | production 7f7b0f24 | 2026-10-18 | trial-records-20261018/RELEASE-BRIEF-HOTFIX-4613.md (a5616d0b5d87c9d0) | 12 |
cost_per_check | about USD 0.06 Assure's own app, 68 files, 2.2 MB; about 90 checks on the free credit | projected flag app_roots | main collector | 2026-10-19 | #4904 body; trial-records-20261019/docs-point-at-app-001/ | 13 |
checks_on_free_credit | about 90 | projected flag app_roots | main collector | 2026-10-19 | #4904 body | 13 |
limit_free | 6 MiB | projected flag per_plan_limits | #4905 (open) | 2026-10-19 | the Director's ruling; enforced in #4905 | 14 |
limit_premium | 24 MiB | projected flag per_plan_limits | #4905 (open) | 2026-10-19 | the Director's ruling; enforced in #4905 | 14 |
limit_pro | 64 MiB | projected flag per_plan_limits | #4905 (open) | 2026-10-19 | the Director's ruling; enforced in #4905 | 14 |
price_free_credit | USD 5 | release | QUICKSTART-API.md section 8: each account starts with USD 5.00 of credit | docs | ||
price_premium | USD 20 | release | QUICKSTART-API.md section 14: Premium (usd_20), USD 20 a month | docs | ||
price_pro | USD 200 | release | QUICKSTART-API.md section 14: Pro (usd_200), USD 200 a month | docs | ||
pool_premium | USD 20 | release | QUICKSTART-API.md section 14: one monthly pool of USD 20 of usage | docs | ||
pool_pro | USD 200 | release | DATA.md 'The report': on Premium and Pro, one pool, USD 20 or USD 200 of usage a month | docs | ||
token_price | USD 0.10 per million input tokens | release | QUICKSTART-API.md section 8 | docs | ||
verdict_retention_days | 30 days | release | QUICKSTART-ACTION.md: the server keeps the verdict for 30 days | docs |
Release items and their state
per_plan_limitsexpectedPer-plan input limits: Free 6 MiB, Premium 24 MiB, Pro 64 MiB; Enterprise above that (#4252, #4905)app_rootsexpectedAssure sends only the app's own code (app_roots). A repository over its plan's limit gets its app checked, with an upsell (#4907, the next Action tag)premium_formal_resultexpectedPremium: the formal result in the PR comment, plus the proof tree (#4175, #4562, #4611)pro_runs_premium_buildexpectedPro runs the Premium build (#4175)readable_reportexpectedThe readable report and deterministic upgrade lines (#5006, #4614, #4719)upgrade_lineexpectedThe free comment ends with the deterministic upgrade line (#5006)python_one_linerexpectedThe README and the setup skill lead with the Python one-liner (#4943)agent_first_setupexpectedSetup is a single command: the agent gets the declaration, a requirements.md tagged by tier, and the workflow from the API, then opens the first pull request itself; that PR already shows the check working (Free rows hold; higher-tier rows read 'not established — Premium checks this'). The 'Allow GitHub Actions to create pull requests' step is gone (#4994, #4995, #4996)translation_default_onexpectedTranslation of requirement lines is on by default, with zero data retentionbenchmark_publicshippedSymbolia-Assurance/assure-benchmark is public: ten real Flask apps, the planted flaw for each, every tool version pinned, results per release. Created once Thursday's rerun is in. Flip to pending to hide its linksdemos_publicexpectedThe planted-flaw pull requests on the ten setup-qual repositories are public, each carrying Assure's comment. The audit agent confirms each PR (or swaps a better one) and Thursday's rerun refreshes the comments on the release engine. Flip to pending to hide their linksself_check_publicpendingAssure checks itself, every release: the home card shows the newest release's own comment and /assure-on-assure keeps every release's counts. Off until the first real release drop; until then the fixture renders with a visible sample mark. The Director decides Thursdayadapter_bridgeexpectedThe adapter bridge (assure.adapter-bridge/v1): a customer's agent maps an app shape Assure does not read yet onto a served adapter with a bridge section in .assure/declaration.json; Assure re-verifies the map on every run and its claims read as stated premises (#5285, #5287, #5288)stated_factsexpectedCustomer-stated config facts (config_surfaces) read as stated premises: holds, assuming X (stated by you); never a silent pass; drafted facts arrive unconfirmed (#5289, #5275)request_coverageexpectedA refusal leads to a coverage request on POST /v1/feedback with a pre-filled coverage_gap note; Symbolia grows coverage internally (#5269)video_finalshippedThe ETB launch video, final cut (received 19 Oct): the end card reads symbolia.ai and the lower third reads Billy. Same timeline and audio as V01