Use cleaned epochs for sensor-space decoding, time-frequency analysis, and covariance calculation

[hoechenberger]

We accidentally used the non-cleaned epochs for classification. This issue probably snuck in during the recent set of refactorings.

Before merging …

• Changelog has been updated (docs/source/changes.md)

[larsoner]

I don't see any problem with moving this but... how did this fix the failures ?!?

[hoechenberger]

You caught me red-handed there. I believe I'm working around a bug in MNE there. It wouldn't allow me to concatenate epochs where the baseline period was cropped away. So now I first concat, then crop.

[larsoner]

I think that's an okay workaround. A comment along those lines would help

[hoechenberger]

I opened an issue: mne-tools/mne-python#12153

[larsoner]

The ds001810 failure seems real, like somehow an extra condition is sneaking into the classification somehow now. @hoechenberger do you have time to check locally if that's the case?

[hoechenberger]

@hoechenberger do you have time to check locally if that's the case?

Unfortunately not, I've been trying to download that dataset for the past hour but it's just hopeless with our firewall here.

[hoechenberger]

@larsoner Btw what I suspect is happening is that we simply end up with too few clean epochs, hence CV won't work

Maybe we can set the number of splits to 2 or even 1... that should fix the issue

[larsoner]

Ahh makes sense. So use max(5, n_epochs_per_cond_min) or whatever. I'll look and push a commit, including reverting my always: conftest change that hopefully will no longer be necessary.

[hoechenberger]

So use max(5, n_epochs_per_cond_min) or whatever.

I actually do like the fact that we're currently failing hard

We should adjust the test case for that specific dataset instead of auto-adjusting the number of splits

[hoechenberger]

@larsoner I just tried, I cannot reproduce this locally :(

[hoechenberger]

@larsoner It seems the issue occurs during the cathodalpost session:

│18:59:14│ ⏳️ sub-01 ses-cathodalpost Contrasting conditions: 61450 – 61511
FAILED           [100%]

[hoechenberger]

import mne
epo = mne.read_epochs("sub-01_ses-cathodalpost_task-attentionalblink_proc-clean_epo.fif")
epo

gives:

<EpochsFIF |  25 events (all good), -0.199219 – 0.5 s, baseline -0.199219 – 0 s, ~5.1 MB, data loaded, with metadata,
 '61450': 22
 '61511': 3>

[agramfort]

@hoechenberger did you check on a few datasets if it boosts the decoding scores? as decoding learns discriminative features it's usually pretty good at filtering out artifacts.

[hoechenberger]

@agramfort No, I did not systematically check this. Are you saying it may have been a conscious decision to use uncleaned epochs here? Because to me, it looked unintentional (i.e., a bug)

Especially since we also used the uncleaned epochs for covariance calculation

[agramfort]

Yes I think it was intentional although not well motivated and documented. Just my prior experience

[hoechenberger]

Maybe we should add a setting, decoding_use_clean_epochs: bool

[hoechenberger]

The thing is that we always use cleaned epochs to create the Evokeds, and source reconstruction also operates on these clean Evokeds. So we're introducing some sort of an inconsistency if decoding shall operate on un-cleaned epochs.

One could argue that if one doesn't want to decode cleaned epochs, one could just disable SSP/ICA and the PTP rejection step…

[agramfort]

I would just check on a couple of datasets to see what it changes

[hoechenberger]

@agramfort
Now this is anecdotical if you will, but I'm currently working with EEG data and one recording was particularly contaminated with a certain artifact that would spread across many sensors. Neither full-epochs nor time-by-time decoding could find a difference between two experimental conditions, and I was about to exclude this participant from analysis. I then cleaned the data (autoreject local -> ICA -> autoreject local) and now full-epochs decoding scores very well above chance (~0.9 ROC AUC) and several time periods are above chance level (~0.6 ROC AUC) in time-by-time decoding. For several other participants, decoding performance was improved by this approach too.

That is to say, I did find the expected effect even when using the un-cleaned data, but it's now larger and more obvious

[larsoner]

My sense is that things like SSP and ICA probably won't matter but that peak-to-peak rejection could help, depending on the nature of the artifacts. That sounds in line with what you're seeing @hoechenberger . In this case it makes sense for us to use the "clean" epochs for decoding. At some point we could make it an option which to use (but maybe assume YAGNI for now?)

[agramfort]

fine with me ! thanks for the details

…

Message ID: ***@***.***>