Should I use 3D U-Net for 2D segmentation with multiple modalities?

[Schobs]

My input data is 4xHxW (so suffix of 0001, 0002,0003, 0004 for each case), and the label is 1xHxW. Please could you clarify if we should use a 3D U-Net here, right? My assumption is strong yes, but wanted to make sure. Cheers.

[FabianIsensee]

Hey,
no this is a 2D segmentation problem. Your images are 1xHxW in each modality.
Best,
Fabian

[Schobs]

I see. Even though they all share the same segmentation map? So when using 2D setting, nnU-net can recognise that the multiple modalities relate to the same single segmentation, and learn separate 2D kernals for each modality?

[FabianIsensee]

Modalities are treated like RGB images in natural image processing. So yes. The U-Net is built to map 4 x H x W -> C x H x W where C is the number of classes you have. THe final segmentation is generated via argmax on C x H x W -> H x W

[Schobs]

I see. That’s rather obvious in retrospect isn’t it haha. Think I was thrown off by the way the channels are saved. thanks for the clarification!