Installing the correct version of torch.backends.cudnn

[paul-reiners]

This is more a PyTorch question than a nnUNet question, but I'm hoping someone here will know the answer.

nnUNet training is running slow for me so I tested my version of torch.backends.cudnn:

(nnunet-env) -bash-4.2$ python -c 'import torch;print(torch.backends.cudnn.version())'
7605

It is too old, so I install and run the Docker file as recommended:

(nnunet-env) -bash-4.2$ singularity shell pytorch_21.04-py3.sif 
Singularity> python -c 'import torch;print(torch.backends.cudnn.version())'
7605

It's still using the old version. What now?

[FabianIsensee]

Hi,
what docker image is this? Please try the most recent Nvidia NGC pytorch container https://ngc.nvidia.com/catalog/containers/nvidia:pytorch
Best,
Fabian

[paul-reiners]

It is 21.04. I don't see a more recent release.

[FabianIsensee]

sudo docker run --gpus all -it --rm  --ipc=host nvcr.io/nvidia/pytorch:21.04-py3

NVIDIA Release 21.04 (build 22382700)
PyTorch Version 1.9.0a0+2ecb2c7

Container image Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.

Copyright (c) 2014-2021 Facebook Inc.
Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)
Copyright (c) 2012-2014 Deepmind Technologies (Koray Kavukcuoglu)
Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)
Copyright (c) 2011-2013 NYU (Clement Farabet)
Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston)
Copyright (c) 2006 Idiap Research Institute (Samy Bengio)
Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)
Copyright (c) 2015 Google Inc.
Copyright (c) 2015 Yangqing Jia
Copyright (c) 2013-2016 The Caffe contributors
All rights reserved.

NVIDIA Deep Learning Profiler (dlprof) Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

NOTE: MOFED driver for multi-node communication was not detected.
Multi-node communication performance may be reduced.

root@344370b6c207:/workspace# ipython
Python 3.8.8 (default, Feb 24 2021, 21:46:12)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.22.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import torch

In [2]: torch.__version__
Out[2]: '1.9.0a0+2ecb2c7'

In [3]: torch.backends.cudnn.version()
Out[3]: 8200