# Benchmark

> Ten real Flask apps, one planted sign-in flaw each, every tool version pinned.

**Ten real apps. One planted flaw each.**

[Symbolia-Assurance/assure-benchmark](https://github.com/Symbolia-Assurance/assure-benchmark): the method, the results per release, and run.sh to reproduce it.

One route reachable without signing in, planted in each app. Each tool runs on the clean commit and the planted one. Every finding is itemised.

Planted auth flaws caught: Assure Free 5 / 10. Bandit 0. CodeQL 0. 0 false alarms.

On the same clean code Bandit raised 275 findings and CodeQL 11. Semgrep is not published (rules licence); run it with run.sh. [The measurements](/figures).

The apps: [microblog](https://github.com/miguelgrinberg/microblog) ([PR](https://github.com/Symbolia-Assurance/setup-qual-microblog/pull/5)), [flasky](https://github.com/miguelgrinberg/flasky) ([PR](https://github.com/Symbolia-Assurance/setup-qual-flasky/pull/5)), [flask-base](https://github.com/hack4impact/flask-base) ([PR](https://github.com/Symbolia-Assurance/setup-qual-flask-base/pull/5)), [realworld](https://github.com/gothinkster/flask-realworld-example-app) ([PR](https://github.com/Symbolia-Assurance/setup-qual-realworld/pull/5)), [Flask-User starter](https://github.com/lingthio/Flask-User-starter-app) ([PR](https://github.com/Symbolia-Assurance/setup-qual-flask-user-starter/pull/5)), [cookiecutter-flask demo](https://github.com/jamescurtin/demo-cookiecutter-flask) ([PR](https://github.com/Symbolia-Assurance/setup-qual-cookiecutter-demo/pull/39)), [flask-jwt-auth](https://github.com/oleg-agapov/flask-jwt-auth) ([PR](https://github.com/Symbolia-Assurance/setup-qual-jwt-restful/pull/5)), [REST-tutorial](https://github.com/miguelgrinberg/REST-tutorial) ([PR](https://github.com/Symbolia-Assurance/setup-qual-rest-tutorial/pull/5)), [flaskr](https://github.com/pallets/flask/tree/main/examples/tutorial) ([PR](https://github.com/Symbolia-Assurance/setup-qual-flaskr/pull/7)), [flask-bones](https://github.com/cburmeister/flask-bones) ([PR](https://github.com/Symbolia-Assurance/setup-qual-flask-bones/pull/5)). Each PR is the planted change, with Assure's comment on it.