Subpopulations
Compare performance across relevant clinical profiles, demographics, equipment or sites.
Robustness and failure points
Stress test a health AI model
A strong average result does not describe the conditions in which a model fails. QSTOM-IT puts the system under targeted stress to identify the populations, data and scenarios where performance degrades.
Testing
Look for failure conditions, not only confirmation of a metric
Degraded data
Test sensitivity to missing data, noise, format changes and quality variation.
Time and drift
Observe stability over time and the effect of changes in population or practice.
Critical errors
Study false negatives, false positives, silent errors and high-consequence scenarios.
Generalisation
Assess what happens when the use context differs from development data.
Generative systems
Examine hallucinations, source fidelity, variability, refusal behaviour and out-of-context use.
Access
Three possible access levels
1
Document review
Documentation, results, protocols and logs, without access to individual-level data.
2
Client environment
Tests run inside infrastructure controlled by the organisation with restricted access.
3
Specialised environment
Where necessary, work with infrastructure or partners appropriate to health data constraints.
Reporting
The output is not a single score
We document failure points, evidence level, reproduction conditions and potential consequences. A critical risk is not averaged away by strong results on other criteria.
- Stress-test plan
- Reproducible results
- Subpopulation analysis
- Failure point map
- Generalisation limits
- Recommendations for additional tests
FAQ
Stress testing FAQ
How long does a stress test take?+
Duration depends on access level, number of scenarios and model complexity. A short scoping phase defines priority tests before work begins.
Can you test LLMs?+
Yes, when the system has a clearly defined use case. The protocol should include sources, error scenarios, response variability and suitable evaluation criteria.
Does a stress test prove that a model is safe?+
No. It documents failure conditions and observed results within a defined scope. It cannot prove the absence of every possible risk.
QSTOM-IT
You know the average performance. Do you know where it breaks?
We can design a stress-test protocol adapted to the model, data and decision at stake.