Canary Error Gap
Compare failure rates of a canary cohort and its concurrent control.
Prepare your sheet
Use one row per feature observation. Enter Test requests, Test errors, Control requests, Control errors, Trigger pp. Check that every row belongs to the same review scope.
Decision rule
The trigger is the configured absolute percentage-point increase, not statistical significance. A decision requires traffic in both groups.
Worked result
The fictional example contains 2 input rows. Its result is 1 canary pause.
| Feature | Canary | Control | Gap pp | Action |
|---|---|---|---|---|
| Search | 4% | 1% | 3 | Pause |
| Export | 0% | 0.83% | -0.83 | Observe |
Run it on your own inputs
- Open the tool and choose Start empty or edit the example.
- Enter the inputs, resolve validation messages and review the decision sheet.
- Download the text report or print the summary for the responsible person.
Source and support
Compare failure rates of a canary cohort and its concurrent control. The demonstrated change reaches a pull request and a human approval point before merge. Benchcraft designed this tool’s inputs, decision rule and example for this office task. Reviewed source · Alex Finn.
Request a custom workflow. €149 one-time pilot after written scope agreement. JB Codex adapts one repeated job with a worked example and owner review.