Subject-leakage auditor
An audit script for EEG decoding pipelines that catches the split most people get wrong without noticing: the same subject's data leaking across train and test, quietly inflating accuracy.
The problem
A brain-decoding paper’s headline accuracy is only meaningful if train and test data come from genuinely held-out subjects. Splitting by trial instead of by subject is an easy mistake to make and an easy one to miss: the model partly learns “this is subject 7’s EEG” rather than the thing you actually wanted decoded, and accuracy comes back inflated without any obvious sign that anything went wrong.
What it does
A lightweight audit tool that checks a decoding pipeline’s data split for subject-level leakage before results get reported, targeting the single most common way this class of result is accidentally inflated.
Status
The accompanying paper is currently in review. Code isn’t public yet: this entry will get a source link once it is, rather than one pointing at a private repo.