Controlled benchmark
medicines.org.uk cannot be broken on cue, so the repair loop is measured against a fixture site we control, whose layout can be mutated to order. Its results are kept in a store of their own, deliberately: a benchmark must not be able to disturb the collectors that serve the corpus.
9 mutations over 19 attempts: 8 behaved as expected. 7 of the 8 that should raise a signal did, and 1 that should raise none stayed silent. 1 repaired, 1 verified against the old layout. Open: url_pattern — expected to raise a signal and did not. Every mutation is listed below. A signal chip is filled when that detector fired, outlined when it was expected and stayed silent. Non-regression means the repair still works against the layout it was built for, so it was not overfitted to the break.
| Mutation | Signals | Outcome |
|---|
| class_rename | - row_count expected but silent
- skeleton expected and fired
| detected, not repaired |
| column_reorder | - skeleton expected and fired
| detected, not repaired |
| date_format | - null_rate expected and fired
- schema expected and fired
| detected, not repaired |
| field_split_merge | - skeleton expected and fired
| detected, not repaired |
| link_label | none declared | declared gap · no signal expected |
| pagination | - row_count expected but silent
- schema fired but never declared
- skeleton expected but silent
| detected, not repaired |
| table_to_div | - row_count expected and fired
- skeleton expected and fired
| detected, not repaired |
| url_pattern | - null_rate expected but silent
- schema expected but silent
| not detected |
| wrapper_nesting | - skeleton expected and fired
| repaired · accuracy 1 · non-regression passed |