Validation protocol (draft)
Pre-specified protocol for the analytical validation of the 30-second single-lead AF decision stage; endpoints, acceptance criteria, datasets, scoring and statistics fixed before the final run.
Status: draft for review, 15 September 2026. Once approved, this page is frozen together with the algorithm (manifest hashes below) and the final run is executed once. Every change afterwards is a documented deviation.
1. Purpose and scope
Analytical (bench) validation of the AF decision stage on 30-second single-lead ECG strips. Output per strip: AF, not AF, or inconclusive. Claim under test: on a strip that the device records and the beat detector processes, the algorithm identifies atrial fibrillation with the sensitivity and specificity below. Out of scope here: continuous monitoring, atrial flutter as a separate class, other arrhythmias, and the clinical (prospective) study, which has its own protocol.
2. Algorithm under test
Frozen configuration, identified by the freeze manifest (file hashes, engine commit, feature list, hyperparameters, decision rule). Windows of 16 consecutive beats stepping 4; plausibility gate (all RR 0.2-3.0 s, window rate <= 200 bpm); gradient-boosting classifier on 10 RR-interval features and 14 beat-morphology and quality features; physiological guard; strip rule: highest window probability >= 0.8 -> AF, <= 0.5 -> not AF, otherwise or no plausible window -> inconclusive. No parameter may be changed after the protocol is approved.
3. Datasets and reference standard
| Set | Role | Reference standard | Never used for training or tuning |
|---|---|---|---|
| Sensor strips, held-out patients (patient-disjoint by hash split) | primary, device data | physician re-annotation of every strip (AF / not AF / uninterpretable), blinded to algorithm output; current labels are the original developer’s and are replaced before the final run | yes |
| PhysioNet/CinC 2017 (8,528 single-lead strips, 9-61 s) | secondary, public, artefact-rich | challenge labels v3 (normal / AF / other / noisy) | yes |
| MIT-BIH AF Database (23 records), cut into consecutive 30-s strips | secondary, public, ectopy-rich | cardiologist rhythm annotations; strip labelled AF if >= 80 % of it is AF, not AF if <= 20 %, otherwise excluded | yes |
| Long-Term AF DB, MIT-BIH Arrhythmia DB, clinical AF set, sensor training patients | training and tuning only | no (excluded from validation) |
Pre-specified handling of the CinC “other rhythm” class: counted as not AF (specificity). The “noisy” class is reported separately: fraction inconclusive, fraction called AF.
4. Endpoints and acceptance criteria
Primary endpoints on the sensor held-out set and on CinC 2017, per strip, over classified strips (inconclusive excluded from Se/Sp, reported separately):
| Endpoint | Acceptance |
|---|---|
| Sensitivity for AF | point >= 95 %, lower two-sided 95 % confidence bound >= 90 % |
| Specificity | point >= 97 %, lower bound >= 95 % |
| Inconclusive rate | <= 15 % on CinC 2017 excluding the noisy class; reported without limit on the artefact-heavy sensor set until the re-annotated set defines interpretable strips |
| PPV / NPV | reported with intervals, no limit (prevalence-dependent) |
Secondary endpoints (MIT-BIH AF DB strips): same statistics; plus EC57 duration sensitivity and positive predictivity by epicmp for information. Beat detection: MIT-BIH Arrhythmia DB bxb Se and +P >= 98 %, unchanged engine.
5. Statistics
Wilson score intervals for all proportions; on the sensor set additionally a patient-level cluster bootstrap (2,000 resamples), reported alongside. Sample size for the pivotal sensor set: with expected Se 97 % and Sp 98 %, at least 130 AF strips and 300 non-AF strips give lower Wilson bounds above 92 % and 95.5 %; for cluster-robust bounds the study must enrol at least 100 subjects with AF and 100 without. Subgroups reported: heart-rate band (< 60, 60-100, 100-130, > 130 bpm), strip quality index band, and, when available, sex and age band.
6. Conduct
- Approve this protocol and the freeze manifest; record both hashes in the run log.
- Replace the sensor labels by the blinded physician re-annotation; resolve disagreements by a second reader before any algorithm output is looked at.
- Run the frozen pipeline once per dataset with the published scripts (
strip_eval.py,strip_ci.py,wfdb_crosscheck.py); keep the raw per-strip output files. - Report every result, including failures; no re-running with changed parameters. Findings that require a change lead to a new algorithm version and a new protocol version.
- Deviations (data errors found during the run, missing files) are listed in the report with their effect.
7. Traceability
Requirement -> endpoint -> script -> output file, one line each, kept in the run log. The current pre-study results (feasibility, not validation) are on the results page; they were obtained with the same scripts but before label re-annotation and before the freeze, and are therefore not the submission numbers.