Behavioural testing, CheckList
The draft's bibliography cites a task-agnostic behavioural testing method for NLP models. The cited work argues that held-out accuracy can overestimate performance and proposes a matrix of capabilities and test types plus a tool to generate tests.
The requirement in plain English
The draft's bibliography cites a task-agnostic behavioural testing method for NLP models. The cited work argues that held-out accuracy can overestimate performance and proposes a matrix of capabilities and test types plus a tool to generate tests.
What the public record says
The bibliography entry names the paper. The public view gives no summary of it and does not say which clause uses it.
Why it matters to you
Behavioural testing is the natural bridge between a statistical claim and a functional one. It gives a structured way to ask what the model should do when an input changes in a way that should not matter, and what it should do when the change should matter. For a robustness pack, that structure is how you justify the perturbations you chose. It does not give you a probability, so it complements a statistical estimate rather than replacing it.
Evidence a reviewer asks for
- The behavioural test matrix: capabilities, test types and the expected behaviour for each.
- The mapping from each test type to a perturbation family in the statistical plan.
- The failures found, with the input that produced them.
- The model and dataset versions used.
Checks to run this month
- Does the test set include changes that should not matter, as well as changes that should?
- Is each perturbation tied to an expected behaviour?
- Are failures recorded, not only pass counts?
Dates
Paper published 2020. FDIS registered 2026-08-21.
Cross-references
- High-dimensional input and why exhaustive testing fails
- Write the sampling and perturbation plan
- A statistical approach to robustness
What this page does not state
The FDIS does not say how it uses this paper, and the paper is not the standard.