Prediction Error in nnunetv2

[Elijah-Beck]

I'm doing a semantic segmentation project on ct-scans, and have prepared my data to do training in nnunetv2. everything worked out fine, thank you. I finished 3D Lowres training and did the prediction on raw data and the evaluation of metrics, so I'm quite sure there are no problem's with the code.

But when I finished the 3D fullres cascade training, I tried to do the prediction on the same raw data using the command nnunetv2_predict -i (path to raw data) -o (path to predictions folder) 3d_cascade_fullres -prev_seg_predictions (path to the predictions of 3D Lowres) but it gives me this error message: nnUNetv2_predict -i /disk02/Lobes-
Segments/nnUNet_raw/Dataset448_BPSencoded/imagesTs/ -o /disk02/Lobes-
Segments/nnUNet_results/Dataset448_BPSencoded/nnUNetTrainer__nnUNetPlans__3d_casc
ade_fullres/imagesTs_cascade_Fullres/ -d 448 -c 3d_cascade_fullres -prev_stage_predictions
/disk02/Lobes-
Segments/nnUNet_results/Dataset448_BPSencoded/nnUNetTrainer__nnUNetPlans__3d_lowre
s/imagesTs_3D_Lowres/
#######################################################################
Please cite the following paper when using nnU-Net: Isensee, F., Jaeger, P. F., Kohl, S. A.,
Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-
based biomedical image segmentation. Nature methods, 18(2), 203-211.
####################################################################### There
are 17 cases in the source folder I am process 0 out of 1 (max process ID is 0, we start counting
with 0!) There are 17 cases that I would like to predict Predicting BPS_73:
perform_everything_on_device: True Prediction on device was unsuccessful, probably due to a
lack of memory. Moving results arrays to CPU
100%|████████████████████████████████████████████████████████
█████████████████████████| 512/512 [08:45<00:00, 1.03s/it] Prediction on device
was unsuccessful, probably due to a lack of memory. Moving results arrays to CPU
100%|████████████████████████████████████████████████████████
█████████████████████████| 512/512 [08:10<00:00, 1.04it/s] Traceback (most
recent call last): File "/home/ali/.conda/envs/env/bin/nnUNetv2_predict", line 8, in <module>
sys.exit(predict_entry_point()) File "/home/ali/.conda/envs/env/lib/python3.9/site-
packages/nnunetv2/inference/predict_from_raw_data.py", line 864, in predict_entry_point
predictor.predict_from_files(args.i, args.o, save_probabilities=args.save_probabilities, File
"/home/ali/.conda/envs/env/lib/python3.9/site-
packages/nnunetv2/inference/predict_from_raw_data.py", line 256, in predict_from_files return
self.predict_from_data_iterator(data_iterator, save_probabilities,
num_processes_segmentation_export) File "/home/ali/.conda/envs/env/lib/python3.9/site-
packages/nnunetv2/inference/predict_from_raw_data.py", line 373, in
predict_from_data_iterator prediction = self.predict_logits_from_preprocessed_data(data).cpu()
File "/home/ali/.conda/envs/env/lib/python3.9/site-
packages/nnunetv2/inference/predict_from_raw_data.py", line 492, in
predict_logits_from_preprocessed_data prediction +=
self.predict_sliding_window_return_logits(data).to('cpu') RuntimeError: Inplace update to
inference tensor outside InferenceMode is not allowed.You can make a clone to get a normal
tensor before doing inplace update.See pytorch/rfcs#17 for more details.
Process SpawnProcess-10: Traceback (most recent call last): File
"/home/ali/.conda/envs/env/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run() File "/home/ali/.conda/envs/env/lib/python3.9/multiprocessing/process.py", line 108, in
run self._target(*self._args, **self._kwargs) File "/home/ali/.conda/envs/env/lib/python3.9/site-
packages/nnunetv2/inference/data_iterators.py", line 58, in
preprocess_fromfiles_save_to_queue raise e File "/home/ali/.conda/envs/env/lib/python3.9/site-
packages/nnunetv2/inference/data_iterators.py", line 50, in

preprocess_fromfiles_save_to_queue target_queue.put(item, timeout=0.01) File "<string>", line
2, in put File "/home/ali/.conda/envs/env/lib/python3.9/multiprocessing/managers.py", line 810,
in _callmethod kind, result = conn.recv() File
"/home/ali/.conda/envs/env/lib/python3.9/multiprocessing/connection.py", line 250, in recv buf =
self._recv_bytes() File "/home/ali/.conda/envs/env/lib/python3.9/multiprocessing/connection.py",
line 414, in _recv_bytes buf = self._recv(4) File
"/home/ali/.conda/envs/env/lib/python3.9/multiprocessing/connection.py", line 383, in _recv raise
EOFError EOFError

Now I'm worried that when I finish the 2D training that I also won't be able to finish predictions and evaluate metrics. Is this error due to my devices lack of memory ? or is it something with the nnunetv2 code when transfering operations to the cpu ? I read the predict_on_raw_data.py file and I saw something called the "OOM error", is this it ? and will this show up again when I finish the 2D or is it strictly about the full resolution thing ?