Cannot inference locally #2047
Unanswered
dengzhipeng123
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi, currently I run the trainning remotely (Linux). After the trainning is finished, I copy the entire nnUNet_results folder to my local PC (Windows). The nnUNet is also fully installed and correctly configured in my PC, which keeps the same file structure as remote. My target is to run the inference locally. However, when I started the inference, the following errors happened:
_#######################################################################
Please cite the following paper when using nnU-Net:
Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
#######################################################################
There are 16 cases in the source folder
I am process 0 out of 1 (max process ID is 0, we start counting with 0!)
There are 16 cases that I would like to predict
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in run_code
File "D:\Work\NS\nnUNet\nnUNet\venv\tf\Scripts\nnUNetv2_predict.exe_main.py", line 7, in
File "D:\Work\NS\nnUNet\nnUNet\nnunetv2\inference\predict_from_raw_data.py", line 850, in predict_entry_point
predictor.predict_from_files(args.i, args.o, save_probabilities=args.save_probabilities,
File "D:\Work\NS\nnUNet\nnUNet\nnunetv2\inference\predict_from_raw_data.py", line 257, in predict_from_files
return self.predict_from_data_iterator(data_iterator, save_probabilities, num_processes_segmentation_export)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Work\NS\nnUNet\nnUNet\nnunetv2\inference\predict_from_raw_data.py", line 350, in predict_from_data_iterator
for preprocessed in data_iterator:
File "D:\Work\NS\nnUNet\nnUNet\nnunetv2\inference\data_iterators.py", line 117, in preprocessing_iterator_fromfiles
[i.pin_memory() for i in item.values() if isinstance(i, torch.Tensor)]
File "D:\Work\NS\nnUNet\nnUNet\nnunetv2\inference\data_iterators.py", line 117, in
[i.pin_memory() for i in item.values() if isinstance(i, torch.Tensor)]
^^^^^^^^^^^^^^
NotImplementedError: Could not run 'aten::_pin_memory' with arguments from the 'CUDA' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'aten::_pin_memory' is only available for these backends: [Meta, NestedTensorCPU, BackendSelect, Python, FuncTorchDynamicLayerBackMode, Functionalize, Named, Conjugate, Negative, ZeroTensor, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradHIP, AutogradXLA, AutogradMPS, AutogradIPU, AutogradXPU, AutogradHPU, AutogradVE, AutogradLazy, AutogradMTIA, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, AutogradMeta, AutogradNestedTensor, Tracer, AutocastCPU, AutocastCUDA, FuncTorchBatched, BatchedNestedTensor, FuncTorchVmapMode, Batched, VmapMode, FuncTorchGradWrapper, PythonTLSSnapshot, FuncTorchDynamicLayerFrontMode, PreDispatch, PythonDispatcher].
Meta: registered at ..\aten\src\ATen\core\MetaFallbackKernel.cpp:23 [backend fallback]
NestedTensorCPU: registered at aten\src\ATen\RegisterNestedTensorCPU.cpp:775 [kernel]
BackendSelect: registered at aten\src\ATen\RegisterBackendSelect.cpp:807 [kernel]
Python: registered at ..\aten\src\ATen\core\PythonFallbackKernel.cpp:154 [backend fallback]
FuncTorchDynamicLayerBackMode: registered at ..\aten\src\ATen\functorch\DynamicLayer.cpp:498 [backend fallback]
Functionalize: registered at ..\aten\src\ATen\FunctionalizeFallbackKernel.cpp:324 [backend fallback]
Named: registered at ..\aten\src\ATen\core\NamedRegistrations.cpp:7 [backend fallback]
Conjugate: registered at ..\aten\src\ATen\ConjugateFallback.cpp:17 [backend fallback]
Negative: registered at ..\aten\src\ATen\native\NegateFallback.cpp:19 [backend fallback]
ZeroTensor: registered at ..\aten\src\ATen\ZeroTensorFallback.cpp:86 [backend fallback]
ADInplaceOrView: fallthrough registered at ..\aten\src\ATen\core\VariableFallbackKernel.cpp:86 [backend fallback]
AutogradOther: registered at ..\torch\csrc\autograd\generated\VariableType_0.cpp:17339 [autograd kernel]
AutogradCPU: registered at ..\torch\csrc\autograd\generated\VariableType_0.cpp:17339 [autograd kernel]
AutogradCUDA: registered at ..\torch\csrc\autograd\generated\VariableType_0.cpp:17339 [autograd kernel]
AutogradHIP: registered at ..\torch\csrc\autograd\generated\VariableType_0.cpp:17339 [autograd kernel]
AutogradXLA: registered at ..\torch\csrc\autograd\generated\VariableType_0.cpp:17339 [autograd kernel]
AutogradMPS: registered at ..\torch\csrc\autograd\generated\VariableType_0.cpp:17339 [autograd kernel]
AutogradIPU: registered at ..\torch\csrc\autograd\generated\VariableType_0.cpp:17339 [autograd kernel]
AutogradXPU: registered at ..\torch\csrc\autograd\generated\VariableType_0.cpp:17339 [autograd kernel]
AutogradHPU: registered at ..\torch\csrc\autograd\generated\VariableType_0.cpp:17339 [autograd kernel]
AutogradVE: registered at ..\torch\csrc\autograd\generated\VariableType_0.cpp:17339 [autograd kernel]
AutogradLazy: registered at ..\torch\csrc\autograd\generated\VariableType_0.cpp:17339 [autograd kernel]
AutogradMTIA: registered at ..\torch\csrc\autograd\generated\VariableType_0.cpp:17339 [autograd kernel]
AutogradPrivateUse1: registered at ..\torch\csrc\autograd\generated\VariableType_0.cpp:17339 [autograd kernel]
AutogradPrivateUse2: registered at ..\torch\csrc\autograd\generated\VariableType_0.cpp:17339 [autograd kernel]
AutogradPrivateUse3: registered at ..\torch\csrc\autograd\generated\VariableType_0.cpp:17339 [autograd kernel]
AutogradMeta: registered at ..\torch\csrc\autograd\generated\VariableType_0.cpp:17339 [autograd kernel]
AutogradNestedTensor: registered at ..\torch\csrc\autograd\generated\VariableType_0.cpp:17339 [autograd kernel]
Tracer: registered at ..\torch\csrc\autograd\generated\TraceType_0.cpp:16968 [kernel]
AutocastCPU: fallthrough registered at ..\aten\src\ATen\autocast_mode.cpp:378 [backend fallback]
AutocastCUDA: fallthrough registered at ..\aten\src\ATen\autocast_mode.cpp:244 [backend fallback]
FuncTorchBatched: registered at ..\aten\src\ATen\functorch\LegacyBatchingRegistrations.cpp:720 [backend fallback]
BatchedNestedTensor: registered at ..\aten\src\ATen\functorch\LegacyBatchingRegistrations.cpp:746 [backend fallback]
FuncTorchVmapMode: fallthrough registered at ..\aten\src\ATen\functorch\VmapModeRegistrations.cpp:28 [backend fallback]
Batched: registered at ..\aten\src\ATen\LegacyBatchingRegistrations.cpp:1075 [backend fallback]
VmapMode: fallthrough registered at ..\aten\src\ATen\VmapModeRegistrations.cpp:33 [backend fallback]
FuncTorchGradWrapper: registered at ..\aten\src\ATen\functorch\TensorWrapper.cpp:203 [backend fallback]
PythonTLSSnapshot: registered at ..\aten\src\ATen\core\PythonFallbackKernel.cpp:162 [backend fallback]
FuncTorchDynamicLayerFrontMode: registered at ..\aten\src\ATen\functorch\DynamicLayer.cpp:494 [backend fallback]
PreDispatch: registered at ..\aten\src\ATen\core\PythonFallbackKernel.cpp:166 [backend fallback]
PythonDispatcher: registered at ..\aten\src\ATen\core\PythonFallbackKernel.cpp:158 [backend fallback]
Process SpawnProcess-6:
Process SpawnProcess-8:
Traceback (most recent call last):
File "D:\Work\NS\nnUNet\nnUNet\nnunetv2\inference\data_iterators.py", line 48, in preprocess_fromfiles_save_to_queue
if abort_event.is_set():
^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Deng\anaconda3\Lib\multiprocessing\managers.py", line 1091, in is_set
return self._callmethod('is_set')
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Deng\anaconda3\Lib\multiprocessing\managers.py", line 837, in _callmethod
raise convert_to_error(kind, result)
Traceback (most recent call last):
multiprocessing.managers.RemoteError:
Traceback (most recent call last):
File "C:\Users\Deng\anaconda3\Lib\multiprocessing\managers.py", line 260, in serve_client
self.id_to_local_proxy_obj[ident]
~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
KeyError: '22779175c90'_
I would like to know: should it be impossible to train remotely and run locally? Or this problem is caused by another reason. Thanks!
Beta Was this translation helpful? Give feedback.
All reactions