# AF window-classifier feasibility: results log Pipeline: `feasibility/af_windows.py`. Windows of 32 beats, step 8. Features: 20 RR-interval statistics (normalised RMSSD, CV, pNN50, Shannon entropy, sample entropy, turning-point ratio, Poincare SD1/SD2, irregularity index, normalised MAD), each computed on all beats and on beats with ectopics removed, plus heart rate and ectopic fraction. Model: HistGradientBoosting, depth 4, 300 iterations. Episodes: hysteresis on 0.6 / off 0.4, min 2 windows. Scoring: EC57 episode and duration Se / +P (Python implementation of epicmp). Beat sources: PhysioNet reference annotations for MIT-BIH and LTAFDB, `.qrs` detections for AFDB (no beat types), engine beats for RECAMO. Labels: PhysioNet rhythm annotations; RECAMO whole-segment labels from the developer's annotation (to be replaced by cardiologist re-annotation). ## Experiment 0: train MIT-BIH only (48 records, 965 AF windows), test AFDB (25 records) | Variant | Window Se | Window Sp | Episode Se | Episode +P | Duration Se | Duration +P | |---|---|---|---|---|---|---| | all features | 94.4% | 87.5% | 91.5% | 53.9% | 92.9% | 82.1% | | no ectopy features | 71.6% | 98.5% | 80.9% | 87.9% | 74.1% | 95.8% | (48-beat windows in this run.) ## Experiment 2: train MIT-BIH + RECAMO train split (97 patients), test held-out RECAMO (47 patients) and AFDB Patient split by md5 hash of db:patient, test fraction 0.3. No RECAMO test patient appears in training. | Test set | Windows (AF) | Window Se | Window Sp | Window +P | Episode Se | Episode +P | Duration Se | Duration +P | |---|---|---|---|---|---|---|---|---| | RECAMO held-out | 2572 (416) | 91.1% | 98.6% | 92.4% | 96.4% | 90.5% | 87.4% | 92.4% | | AFDB | 152364 (66177) | 93.4% | 93.2% | 91.4% | 90.8% | 50.3% | 95.1% | 89.0% | RECAMO per 30-s segment (AF present vs detected): 604 segments, TP 161, FP 17, FN 6, TN 420 -> Se 96.4%, Sp 96.1%, PPV 90.4%. Grouped 5-fold CV on the training set: window Se 0.894, Sp 0.991. ### AFDB post-processing sweep (exp2 model) Gap merging reduces the episode count (982 -> 132 at 300 s) without improving episode +P (44-51%), so the false episodes are not fragmentation but false detections in non-AF stretches. Threshold / minimum length: | thr_on | min windows | Episode Se | Episode +P | Duration Se | Duration +P | |---|---|---|---|---|---| | 0.6 | 2 | 90.8% | 44.9% | 96.1% | 87.2% | | 0.6 | 6 | 80.6% | 60.9% | 94.9% | 91.4% | | 0.9 | 2 | 85.0% | 55.8% | 94.7% | 91.6% | | 0.9 | 6 | 75.4% | 68.2% | 93.3% | 93.7% | (merge gap 60 s in all rows). Note: sweeping on the test set is exploratory only; final operating points must be chosen on a tuning split (LTAFDB) before the single AFDB run. ## Experiment 1 (partial LTAFDB: 15 of 84 records, 193k windows): train MIT-BIH + LTAFDB-15, test AFDB | Variant | Window Se | Window Sp | Window +P | Episode Se | Episode +P | Duration Se | Duration +P | |---|---|---|---|---|---|---|---| | all features | 99.5% | 84.1% | 82.8% | 97.3% | 30.1% | 97.8% | 77.0% | | no ectopy features | 93.0% | 97.2% | 96.3% | 92.5% | 47.5% | 97.0% | 92.8% | The "all features" variant over-fires on AFDB because LTAFDB/MIT-BIH carry beat types while AFDB does not (domain shift), so the beat-type-agnostic variant is the honest line for AFDB. ## Experiment 3p: train MIT-BIH + LTAFDB-15 + SCI (218 segments) + RECAMO train (97 patients), no ectopy features | Test set | Window Se | Window Sp | Window +P | Episode Se | Episode +P | Duration Se | Duration +P | |---|---|---|---|---|---|---|---| | AFDB | 93.6% | 97.2% | 96.2% | 92.5% | 48.6% | 97.2% | 92.9% | | RECAMO held-out (47 patients) | 95.4% | 97.4% | 87.6% | 97.6% | 86.7% | 89.0% | 89.6% | ### Where the AFDB false episodes come from (exp1b, 192 false of 366 detected) | Record | Ref ep / hit | Test ep / hit | False ep | Duration +P | |---|---|---|---|---| | 08219 | 39 / 39 | 59 / 4 | 55 | 54% | | 06995 | 4 / 4 | 54 / 4 | 50 | 73% | | 03665 | 7 / 7 | 19 / 1 | 18 | 83% | | 04043 | 82 / 81 | 82 / 64 | 18 | 76% | | 04126 | 7 / 7 | 23 / 6 | 17 | 51% | | 04908 | 4 / 4 | 16 / 4 | 12 | 74% | Six records produce 170 of the 192 false episodes: short irregular non-AF stretches (ectopy, sinus arrhythmia) in records that also contain AF. Without beat types (AFDB .qrs) or P-wave information the RR-only classifier cannot separate them. Planned fix: engine-native features (P-wave amplitude, QRS width, beat quality) from the developer's beat detector; the engine is being run on AFDB for this. ## Experiment 4: engine-native features (P-wave presence/amplitude, QRS width, deformation, quality, PB ventricular fraction, out-of-signal fraction) Same RECAMO patient split as before. Training without LTAFDB unless stated. All rows beat-type-agnostic. | Model | Training sources | RECAMO held-out: window Se / Sp / +P | Episode Se / +P | Duration Se / +P | |---|---|---|---|---| | 4a RR only | MIT-BIH + SCI + RECAMO-train | 96.4 / 94.9 / 78.6 | 97.0 / 78.3 | 89.0 / 80.9 | | 4b RR + morphology | same, engine features | 98.1 / 96.8 / 85.4 | 98.2 / 86.3 | 90.4 / 87.3 | | 4c 4b + LTAFDB-15 (RR-only rows, morphology NaN) | mixed | 98.1 / 97.6 / 88.7 | 98.2 / 87.7 | 90.6 / 89.2 | 4c on AFDB (RR-only rows, engine features not yet available): window 94.1 / 96.3 / 95.1; episode 92.8 / 46.2; duration 97.2 / 91.3. Engine-native AFDB test is queued for when the engine run completes. Top permutation importances (4b, RECAMO): irregularity index 0.18, heart rate 0.14, normalised MAD 0.06, then Shannon entropy, out-of-signal fraction, pNN50, SD1/SD2. Heart-rate dependence is a known weakness to address (rate-stratified evaluation, possibly down-weighting hr). ## Full LTAFDB (84 records, 1.12M windows, stride 4 in training), beat-type-agnostic | Model | Training | Test | Window Se / Sp / +P | Episode Se / +P | Duration Se / +P | |---|---|---|---|---|---| | 1b | MIT-BIH + LTAFDB | AFDB | 96.6 / 97.1 / 96.2 | 92.8 / 43.6 | 97.4 / 92.5 | | 3 | + SCI + RECAMO-train (RR only) | AFDB | 96.5 / 97.1 / 96.3 | 92.8 / 43.5 | 97.3 / 92.5 | | 3 | same | RECAMO held-out | 98.6 / 98.1 / 90.7 | 98.2 / 90.1 | 90.6 / 92.3 | | 4c-full | LTAFDB RR rows + engine-feature rows | RECAMO held-out | 98.1 / 98.1 / 90.9 | 97.0 / 91.5 | 89.3 / 93.1 | | 4c-full | same | AFDB (RR only, morphology NaN) | 96.8 / 96.8 / 95.9 | 92.8 / 43.7 | 97.1 / 92.0 | Window-level AFDB performance is now 96.6 % / 97.1 %. Episode +P on AFDB stays at ~44 %. ### AFDB false-positive diagnostic (model 3) 1,903 FP windows vs 62,712 TP: median heart rate 91 vs 100, irregularity index 0.65 vs 0.68, RR CV 0.205 vs 0.208, model probability 0.85 vs 0.99. On RR intervals alone the false windows look like AF. 213 false episodes, median length 44 s (p25 31 s, p75 90 s, nine longer than 5 min); reference episodes median 98 s with 102 of 293 shorter than 60 s, so a minimum-length rule cannot separate them. Eight records produce 206 of the 213 (08219: 56, 06995: 51, 04126: 33, 03665: 22). Conclusion: P-wave / morphology information from the engine is required; engine-native AFDB test pending. ### Inconclusive output (RECAMO held-out, model 4c-full) | Abstention rule | Inconclusive rate | Se on classifiable | Sp | +P | |---|---|---|---|---| | none | 0 % | 98.1 | 98.1 | 90.9 | | probability in (0.3, 0.7) | 3.3 % | 99.2 | 99.4 | 97.0 | | qi_mean < 90 or out-of-signal > 20 % | 44.4 % | 99.5 | 97.1 | 85.7 | Quality-based rules abstain far too often on wearable data with the engine's current qi scale; the probability-margin rule is cheap and effective. Final rule to be fixed on the tuning split. ### Rate-stratified window performance | Set | HR band | AF windows | Se | non-AF windows | Sp | |---|---|---|---|---|---| | RECAMO held-out | 60-100 | 215 | 98.1 | 348 | 93.4 | | RECAMO held-out | 100-130 | 124 | 98.4 | 121 | 85.1 | | RECAMO held-out | >130 | 77 | 97.4 | 1687 | 100.0 | | AFDB | <60 | 758 | 99.3 | 8679 | 99.9 | | AFDB | 60-100 | 32508 | 95.4 | 66747 | 97.6 | | AFDB | 100-130 | 25919 | 98.1 | 10696 | 92.6 | | AFDB | >130 | 6992 | 95.7 | 65 | 32.3 | Sensitivity is flat across rate bands (95-99 %). Specificity dips for fast non-AF rhythm (100-130 bpm), the expected weak spot of RR-based detection. ### RECAMO held-out: artefact segments excluded (model 4c-full) The 1687 non-AF windows above 130 bpm come from 196 segments flagged saw/noise by the developer, where the engine reports ~119 "beats" per 30 s (spurious). They are trivially classified non-AF and inflate specificity. Honest numbers on the 424 clean held-out segments: | Level | Se | Sp | +P / PPV | Notes | |---|---|---|---|---| | Window, clean segments | 98.1 | 94.7 | 91.0 | 1162 windows, 414 AF | | Window, clean, abstain 0.3
300 s | 109 | 98.2 % | 81 | 100.0 % | Episodes below the 30-s clinical definition of AF are below the resolution of a 32-beat window by design; for episodes >= 30 s sensitivity is 96-100 %. AFDB false episodes: 410, median 33 s, 63 longer than 60 s, two longer than 5 min. Episode +P restricted to detected episodes >= 60 s: 83.9 %. Single run with the fixed operating point: | Test set | Window Se / Sp / +P | Episode Se / +P | Duration Se / +P | Segment Se / Sp / PPV | |---|---|---|---|---| | AFDB (RR only) | 95.5 / 97.3 / 96.5 | 94.5 / 51.9 | 96.5 / 93.4 | n/a | | RECAMO held-out, 424 clean segments | 96.6 / 99.5 / 99.0 | 94.5 / 98.7 | 87.4 / 99.2 | 94.5 / 99.2 / 98.7 (FP 2, FN 9) | ### Head-to-head on the same 424 clean held-out RECAMO segments (segment level) | Model | Se | Sp | PPV | TP / FP / FN / TN | |---|---|---|---|---| | Developer engine output as delivered (rcm_*_rev25 files, engine 0.1a.021/023) | 99.4 | 96.1 | 94.3 | 164 / 10 / 1 / 249 | | Ours, model 6, operating point tuned on LTAFDB, patients never seen | 94.5 | 99.2 | 98.7 | 156 / 2 / 9 / 257 | Caveats: all 47 patients were in the developer's training set, and the RECAMO labels were written by the developer with his engine output at hand, so his sensitivity is in-sample and possibly label-aligned. Our 9 misses include RCM158-5 and RCM160-67 (irregular RR with clear P waves at good quality) that the developer's engine calls AF; cardiologist re-annotation will settle which is right. The two operating points differ in direction: his favours sensitivity, ours precision; the abstention rule can move ours. ## Noise handling and the end-to-end product rule (no expert flags used) Signal-quality study (`sqi.py`): "saw" segments are sawtooth artefacts that the engine turns into spurious beats (median 236 detected beats/min, qi_min 0, 1 % labelled AF); "noise" segments have plausible beat rates but the engine already marks a median 53 % of their duration as out of signal. A trained artefact detector (raw-signal SQIs + engine summaries, logistic regression) reproduces the expert flags only moderately (Se 86 %, Sp 85 % on held-out patients), and the flags themselves have no written criteria. End-to-end evaluation on ALL 755 held-out RECAMO segments (28 % flagged by the expert), expert flags not used. Product rule per 30-s strip: windows need 32 plausible beats (RR 0.2-3.0 s), AF if max window probability >= on, not-AF if <= off, otherwise inconclusive. Model 6. | Rule | Inconclusive | Classified | Se | Sp | PPV | NPV | FP / FN | |---|---|---|---|---|---|---|---| | on 0.5 / off 0.5 (no band) | 20.0 % (151 no valid window) | 604 | 97.0 | 97.9 | 94.7 | 98.8 | 9 / 5 | | on 0.6 / off 0.3 | 23.0 % (151 no window, 23 band) | 581 | 98.8 | 99.3 | 98.1 | 99.5 | 3 / 2 | | on 0.7 / off 0.3 | 23.4 % | 578 | 98.7 | 99.5 | 98.7 | 99.5 | 2 / 2 | | + artefact stage t=0.9 | 38.0 % | 468 | 98.8 | 99.4 | 98.8 | 99.4 | 2 / 2 | The RR plausibility gate alone removes the sawtooth artefacts (they yield no valid window); the probability band removes the uncertain strips; the trained artefact stage adds only refusals. Chosen product rule: on 0.6 / off 0.3, no separate artefact model. The 23 % inconclusive rate reflects this dataset (28 % expert-flagged artefacts); on curated recordings it will be lower (Apple reports ~12 % unclassifiable). ## Training-data audit (`audit_labels.py`, out-of-fold by patient, 111,629 windows) | Source | Windows | Suspect | Finding | |---|---|---|---| | LTAFDB | 99,047 | 1,823 non-AF windows with p > 0.9 | 914 in record 64 lie BEFORE its first rhythm annotation (unannotated ~2 h, irregular RR): a loader bug treated them as non-AF -> fixed (`fix_uncovered.py` removed 11,601 windows of record 64 from the training file; extractor now drops beats before the first rhythm marker). Records 103 and 22: annotated (N, (AB, (SVTA with irregular RR -> genuine hard negatives, kept. | | LTAFDB | | 376 AF windows with p < 0.1, 98 AF on regular RR | records 26 (AF at 63 bpm, irregularity 0.35), 112 and 115 (AF with near-regular RR): kept except the regular-RR rule | | RECAMO set 2 | 1,389 | 694 windows > 200 bpm | unflagged artefact segments with spurious engine beats -> dropped by the new rule | | SCI | 2,041 | 102 AF-labelled regular-RR windows, 55 confident disagreements | whole-segment AF labels covering non-AF stretches -> regular-RR rule drops the former | | MIT-BIH | 8,251 | 103 | scattered; 15 artefact windows -> dropped by the > 200 bpm rule | Cleaning rules adopted (`train --clean`, and the > 200 bpm rule in `test`): windows above 200 beats/min are artefacts; AF-labelled windows with irregularity < 0.1 and CV < 0.05 are label errors. Model-based disagreements are NOT used for cleaning (circular). Model 8 = model 6 protocol on the corrected and cleaned data. ## Experiment 7: second window scale, 16 beats / step 4 (`run_scale16.sh`) Model 7 trained like model 6 on 16-beat windows (134,980 windows, stride 16; grouped-CV window Se 0.949, Sp 0.927); operating point tuned on LTAFDB-tune: on 0.8 / off 0.5 / min 2 windows / no merge. AFDB, RR only: | Episodes on AFDB | Test ep | Episode Se / +P | Duration Se / +P | Se < 10 s | 10-30 s | 30-60 s | > 60 s | |---|---|---|---|---|---|---|---| | model 6 (32-beat) | 853 | 94.5 / 51.9 | 96.5 / 93.4 | 57 % | 92 % | 97 % | 100 % | | model 7 (16-beat) | 1724 | 98.6 / 49.7 | 95.7 / 93.3 | 89 % | 100 % | 97 % | 100 % | | union | 1238 | 98.6 / 36.8 | 96.8 / 91.5 | 89 % | 100 % | 97 % | 100 % | | intersection | 1320 | 94.5 / 63.0 | 95.4 / 95.3 | 57 % | 92 % | 97 % | 100 % | The short scale recovers the short paroxysms without hurting duration metrics; a cascade (16-beat proposes, 32-beat or morphology confirms) is the design to pursue rather than a plain union. Note: model 7 was trained before the record-64 fix; retrain after model 8 results. ## Experiment 8: model 6 protocol on corrected + cleaned data (`run_model8.sh`) Record-64 unannotated stretch removed; `--clean` dropped 754 windows > 200 bpm and 205 AF-labelled regular-RR windows; 109,213 training windows, 243 patients. Grouped-CV window Se 0.973 / Sp 0.955 (model 6: 0.966 / 0.935). Operating point re-tuned on LTAFDB-tune: on 0.6 / off 0.3 / min 1 / no merge (unchanged). | Test | Model 6 | Model 8 | |---|---|---| | AFDB window Se / Sp / +P | 95.5 / 97.3 / 96.5 | 95.6 / 96.7 / 95.8 | | AFDB EC57 episode Se / +P | 94.5 / 51.9 | 93.9 / 46.8 | | AFDB EC57 duration Se / +P | 96.5 / 93.4 | 96.5 / 92.0 | | RECAMO clean windows Se / Sp / +P | 96.6 / 99.5 / 99.0 | 97.1 / 98.8 / 98.8 | | RECAMO clean EC57 episode Se / +P | 94.5 / 98.7 | 96.4 / 98.2 | | RECAMO end-to-end, all 755: inconclusive / Se / Sp / PPV | 23.0 % / 98.8 / 99.3 / 98.1 | 44.5 % / 98.8 / 98.0 / 97.0 | Cleaning raised cross-validated specificity and RECAMO sensitivity; AFDB is within noise. The higher inconclusive rate comes from the new > 200 bpm window rule in the test path (artefact strips now refused rather than classified as non-AF). Breakdown of the 755 held-out segments (model 8): | Category | Clean (not flagged) | Expert-flagged | |---|---|---| | no window at all (< 33 plausible beats) | 120 | 31 | | all windows > 200 bpm (spurious beats) | 42 | 132 | | probability band 0.3-0.7 | 9 | 7 | | classified | 373 | 41 | Finding: 120 clean strips are refused only because a 32-beat window needs 33 beats and a 30-s strip below ~66 bpm does not have them. For 30-s product strips the window must be shorter (16 beats, model 9) or adaptive. The 42 clean strips whose windows all exceed 200 bpm are unflagged artefacts (rubbish the expert missed); refusing them is correct. ## Experiment 9: 16-beat model retrained on corrected + cleaned data; cascade on AFDB (`run_cascade.sh`) Model 9: 131,070 windows, grouped-CV Se 0.955 / Sp 0.941; operating point on LTAFDB-tune: on 0.6 / off 0.5 / min 3. | AFDB, RR only | Test ep | Episode Se / +P | Duration Se / +P | Se < 10 s | 10-30 s | >= 30 s | |---|---|---|---|---|---|---| | model 8 (32-beat) | 964 | 93.9 / 46.8 | 96.5 / 92.0 | 50 % | 89 % | 100 % | | model 9 (16-beat) | 1543 | 96.2 / 45.4 | 96.0 / 92.5 | 61 % | 100 % | 100 % | | cascade: 16 proposes, 32 confirms | 1220 | 93.5 / 55.7 | 95.9 / 94.0 | 46 % | 89 % | 100 % | ### End-to-end product rule on all 755 held-out RECAMO strips, expert flags not used (`run_e2e16.sh`) | Configuration | Inconclusive | Classified | Se | Sp | PPV | NPV | FP / FN | |---|---|---|---|---|---|---|---| | model 8, 32-beat, on 0.6 / off 0.3 | 44.5 % | 419 | 98.8 | 98.0 | 97.0 | 99.2 | 5 / 2 | | model 9, 16-beat, on 0.6 / off 0.3 | 32.2 % | 512 | 98.9 | 94.1 | 89.6 | 99.4 | 20 / 2 | | model 9, 16-beat, on 0.8 / off 0.5 | 31.8 % | 515 | 98.7 | 98.6 | 96.9 | 99.4 | 5 / 2 | | hybrid 32-beat else 16-beat, on 0.7 / off 0.3 | 31.5 % | 517 | 98.2 | 98.0 | 95.9 | 99.1 | 7 / 3 | The 16-beat model removes the slow-rate refusals (clean strips refused: 167 -> 87); the remaining refusals are strips whose engine beats are all above 200 bpm (artefacts, correctly refused) plus the probability band. Chosen product configuration: 16-beat windows, AF if max p >= 0.8, not-AF if <= 0.5, else inconclusive. ## Pending - Engine-native AFDB test (models 5, 6, 4c with P-wave / QRS / quality features): `run_afdb_eng.sh`, waits for the engine run (14041 segments), results in `afdb_eng_results.txt`. - Second window scale, 16 beats / step 4, model 7, tuned on LTAFDB-tune, union with model 6 on AFDB: `run_scale16.sh`, results in `scale16_results.txt`. - Replace RECAMO labels with cardiologist interval annotations when available; re-run the fixed protocol once. - Cross-check the Python EC57 scorer against WFDB `epicmp` / `bxb`. - Version the feasibility scripts in a repository with tags and model hashes before the validation run.