Error with mask snapshot

[jdeschamps]

Hi,

To get better acquainted with dacapo, I started trying to train on a toy example but I ran into an error.

Data

I created 3D spheres of two different classes, and packaged it into a zarr. The internal organization of the zarr is the following (created here):

/
 ├── test
 │   ├── crop_01
 │   │   ├── labels (2, 256, 256, 256) uint8
 │   │   └── raw (256, 256, 256) uint8
 │   └── crop_06
 │       ├── labels (2, 256, 256, 256) uint8
 │       └── raw (256, 256, 256) uint8
 └── train
     ├── crop_00
     │   ├── labels (2, 256, 256, 256) uint8
     │   └── raw (256, 256, 256) uint8
     ├── crop_02
     │   ├── labels (2, 256, 256, 256) uint8
     │   └── raw (256, 256, 256) uint8
     ├── crop_03
     │   ├── labels (2, 256, 256, 256) uint8
     │   └── raw (256, 256, 256) uint8
     ├── crop_04
     │   ├── labels (2, 256, 256, 256) uint8
     │   └── raw (256, 256, 256) uint8
     ├── crop_05
     │   ├── labels (2, 256, 256, 256) uint8
     │   └── raw (256, 256, 256) uint8
     └── crop_07
         ├── labels (2, 256, 256, 256) uint8
         └── raw (256, 256, 256) uint8

I have not set up any mask, the labels are shaped (2, 256, 256, 256) where the two classes are semantic masks in different channels.

Data is created here: https://github.com/nobias-fht/cellmap_pipelines/blob/main/data/create_toy.py

Training

I built a training pipeline here: https://github.com/nobias-fht/cellmap_pipelines/blob/main/training_toy/train_toy.ipynb

It is a DistanceTaskConfig task, with two channels in the ground-truth.

Error

Upon training, I directly get an error:

File ~/miniconda3/envs/dacapo/lib/python3.10/site-packages/zarr/util.py:561, in check_array_shape(param, array, shape)
    559     raise TypeError(f"parameter {param!r}: expected an array-like object, got {type(array)!r}")
    560 if array.shape != shape:
--> 561     raise ValueError(
    562         f"parameter {param!r}: expected array with shape {shape!r}, got {array.shape!r}"
    563     )

ValueError: parameter 'value': expected array with shape (256, 244, 244), got (2, 2, 256, 244, 244)

You can find the full traceback in the training notebook.

The reason for the error is that a mask is created (shape (2, 2, 256, 244, 244)):

dacapo/dacapo/experiments/trainers/gunpowder_trainer.py

Line 390 in 0b3e2d0

array = open_from_identifier(

and then assigned with a shape mismatch:

dacapo/dacapo/experiments/trainers/gunpowder_trainer.py

Line 404 in 0b3e2d0

array[:] = data

Questions

1. I set up the data as a semantic segmentation rather than an instance one. Is dacapo expecting instances? How are semantic vs instance handled? I think I got an error when I had more than 2 labels in the ground-truth.
2. Do you see any obvious reason for this error?

Thanks a bunch for the help!

[pattonw]

Thanks for the detailed issue. I looked into it and there were a couple small bugs that you were running into. I have fixed them, you can see the details here: #358

As for your questions:

1. dacapo can handle both instance and semantic, it just depends on the task you choose. Affinities will give you instance segmentations, distance transforms and one hot predictions will give you semantic segmentations.
2. Yes, they were caused by essentially typos that affected the saving of snapshots, and unfortunately saving snapshots were not yet covered by tests and failed in some cases. I will extend the tests to cover saving snapshots