How do I compare failure rates of a canary cohort and its concurrent control?
The trigger is the configured absolute percentage-point increase, not statistical significance. A decision requires traffic in both groups.
Open Canary Error GapTry the worked example
These are the example inputs from Canary Error Gap. Replace them in the tool to run your own case.
| Feature | Test requests | Test errors | Control requests | Control errors | Trigger pp |
|---|---|---|---|---|---|
| Search | 200 | 8 | 1000 | 10 | 2 |
| Export | 50 | 0 | 120 | 1 | 2 |
Example result
1 canary pause
Who uses this?
Release engineer. The free online tool runs the calculation in your browser. To edit the files and use the tools offline in one business, see the €49 offline edition.
Where the idea came from
Compare failure rates of a canary cohort and its concurrent control. The demonstrated change reaches a pull request and a human approval point before merge. Benchcraft designed this tool’s inputs, decision rule and example for this office task.
Original video