P
Performance
Metrics, thresholds, calibration, performance by population, comparators and consistency with the stated use case.
- Appropriate metrics
- Operational threshold
- Calibration
- Error analysis
QSTOM-IT framework - in development
PRISM - A methodological framework for assessing health AI
PRISM organises technical and methodological assessment of an AI system across five complementary dimensions. The framework aims to make questions, evidence and tests explicit, versioned and reproducible.
Framework status
PRISM is not a regulatory certification. The first public version of the framework is being prepared and is intended for publication.
PRISM
Five complementary dimensions
R
Robustness
Generalisation, drift, temporal variation, degraded data and the conditions under which performance deteriorates.
- Stress tests
- External validation
- Drift
- Data sensitivity
I
Integrity
Data quality and provenance, cohorts, train/test separation, leakage, traceability and pipeline reproducibility.
- Data lineage
- Cohort separation
- Data leakage
- Reproducibility
S
Safety
Critical errors, false negatives, edge cases, silent errors, human oversight and consequences of an incorrect output.
- Critical cases
- Oversight
- Failure modes
- Use limitations
M
Methodology
Evaluation question, hypotheses, experimental protocol, comparators, validity of conclusions and study limitations.
- Explicit hypothesis
- Protocol
- Comparators
- Scope of claims
Evidence level
Describe the level of evidence instead of averaging everything into one score
PRISM is not designed to produce a score out of 100. A critical data leakage problem should not be offset by strong results on other dimensions.
No conclusion is drawn on this point.
The method or result is described by the assessed team.
QSTOM-IT has run or reviewed a targeted test within the engagement scope.
The result could be reproduced from accessible materials using a documented protocol.
PRISM Assessed
PRISM Assessed - a future assessment mark, not a certification
In time, an assessment performed against a public PRISM version could be referenced as PRISM Assessed, with an identifier, date, scope and limitations. The mark would only attest that an assessment was performed using the stated framework.
Publication
Publication makes the framework open to challenge
PRISM V1 is intended to be published with its definitions, decision rules, evidence levels and a first application to an open medical model. The objective is to enable scientific discussion and revision of the framework.
EU AI Act
PRISM and the EU AI Act
Some PRISM dimensions overlap with themes in the European framework, such as data quality, robustness, documentation and human oversight. This mapping can help structure technical evidence but does not constitute a conformity assessment.
QSTOM-IT
Are you a clinician, researcher, quality or regulatory expert?
QSTOM-IT is looking for external reviewers to challenge the first version of the framework before publication.