Synthetic example. 400 generated cases illustrate a hidden regression. No real model was evaluated.
The average doesn't tell
the whole story.
An auditable look at two evaluation runs. Find the groups that slipped, inspect the cases behind them, and make the next release with evidence.
Matched by case ID
Baseline fail → candidate pass
Baseline pass → candidate fail
308 both pass · 48 both fail
Follow the evidence.
Each declared slice shows its paired change. Statistical support keeps small groups visible without turning every fluctuation into an alarm.
Try another search or choose a different decision.
Every case has a story.
Start with losses, compare the notes, then look at the wins. Export includes every case matching your filters, across all pages.
| Case / slices | Baseline | Candidate | Outcome |
|---|---|---|---|
| en-001language: en, task: retrieval | PassSynthetic benchmark case | PassSynthetic benchmark case | Both pass |
| en-002language: en, task: reasoning | PassSynthetic benchmark case | PassSynthetic benchmark case | Both pass |
| en-003language: en, task: retrieval | PassSynthetic benchmark case | PassSynthetic benchmark case | Both pass |
| en-004language: en, task: reasoning | PassSynthetic benchmark case | PassSynthetic benchmark case | Both pass |
| en-005language: en, task: retrieval | PassSynthetic benchmark case | PassSynthetic benchmark case | Both pass |
| en-006language: en, task: reasoning | PassSynthetic benchmark case | PassSynthetic benchmark case | Both pass |
| en-007language: en, task: retrieval | PassSynthetic benchmark case | PassSynthetic benchmark case | Both pass |
| en-008language: en, task: reasoning | PassSynthetic benchmark case | PassSynthetic benchmark case | Both pass |
| en-009language: en, task: retrieval | PassSynthetic benchmark case | PassSynthetic benchmark case | Both pass |
| en-010language: en, task: reasoning | PassSynthetic benchmark case | PassSynthetic benchmark case | Both pass |
| en-011language: en, task: retrieval | PassSynthetic benchmark case | PassSynthetic benchmark case | Both pass |
| en-012language: en, task: reasoning | PassSynthetic benchmark case | PassSynthetic benchmark case | Both pass |
| en-013language: en, task: retrieval | PassSynthetic benchmark case | PassSynthetic benchmark case | Both pass |
| en-014language: en, task: reasoning | PassSynthetic benchmark case | PassSynthetic benchmark case | Both pass |
| en-015language: en, task: retrieval | PassSynthetic benchmark case | PassSynthetic benchmark case | Both pass |
| en-016language: en, task: reasoning | PassSynthetic benchmark case | PassSynthetic benchmark case | Both pass |
| en-017language: en, task: retrieval | PassSynthetic benchmark case | PassSynthetic benchmark case | Both pass |
| en-018language: en, task: reasoning | PassSynthetic benchmark case | PassSynthetic benchmark case | Both pass |
| en-019language: en, task: retrieval | PassSynthetic benchmark case | PassSynthetic benchmark case | Both pass |
| en-020language: en, task: reasoning | PassSynthetic benchmark case | PassSynthetic benchmark case | Both pass |
| en-021language: en, task: retrieval | PassSynthetic benchmark case | PassSynthetic benchmark case | Both pass |
| en-022language: en, task: reasoning | PassSynthetic benchmark case | PassSynthetic benchmark case | Both pass |
| en-023language: en, task: retrieval | PassSynthetic benchmark case | PassSynthetic benchmark case | Both pass |
| en-024language: en, task: reasoning | PassSynthetic benchmark case | PassSynthetic benchmark case | Both pass |
| en-025language: en, task: retrieval | PassSynthetic benchmark case | PassSynthetic benchmark case | Both pass |
Showing 25 of 400 paired cases. Enable JavaScript to search, filter, export, and see additional pages.
Built to be questioned.
The thresholds, test, and exact inputs behind this decision. Every number in this report comes from your paired evaluation data.
Decision rule & statistical method
The gate: an evaluation group fails when its pass rate drops by at least the configured tolerance, its support reaches the minimum, and its Holm-adjusted p value is at or below the significance threshold. The overall group and all supported declared slice groups are checked. Unsupported groups never fail the gate.
The test: a one-sided exact paired McNemar test uses discordant cases to test for regression: losses exceeding wins. Holm's step-down correction is applied across the tested overall and declared slice groups. These groups may overlap. Adjusted significance describes evidence of change; it does not explain the cause.
Input fingerprints & declared slices
cc0c25dd0f253eacc0fa4bf0cd8f1e5e6ac973cc1759a2bd382457d54488be45548b42187a8981d6b46bd374ee71753157b660c0bf2a7bd71687acae20fafaceDeclared slice columns: language, task
Report schema: pairglass/v1
SHA256 fingerprints identify the exact input bytes. Optional cost and latency cards report means only when those metrics are supplied.