pytorch - 找不到有效的 cuDNN 算法来运行卷积
问题描述
我在尝试运行前馈 torch.nn.Conv2d 时收到此消息,得到以下堆栈跟踪:
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-26-04bd4a00565d> in <module>
3
4 # call training function
----> 5 losses = train(D, G, n_epochs=n_epochs)
<ipython-input-24-b539315e0aa0> in train(D, G, n_epochs, print_every)
46 real_images = real_images.cuda()
47
---> 48 D_real = D(real_images)
49 d_real_loss = real_loss(D_real, True) # smoothing label 1 => 0.9
50
~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
548 result = self._slow_forward(*input, **kwargs)
549 else:
--> 550 result = self.forward(*input, **kwargs)
551 for hook in self._forward_hooks.values():
552 hook_result = hook(self, input, result)
<ipython-input-14-bf68e57c25ff> in forward(self, x)
48 """
49
---> 50 x = self.leaky_relu(self.conv1(x))
51 x = self.leaky_relu(self.conv2(x))
52 x = self.leaky_relu(self.conv3(x))
~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
548 result = self._slow_forward(*input, **kwargs)
549 else:
--> 550 result = self.forward(*input, **kwargs)
551 for hook in self._forward_hooks.values():
552 hook_result = hook(self, input, result)
~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/container.py in forward(self, input)
98 def forward(self, input):
99 for module in self:
--> 100 input = module(input)
101 return input
102
~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
548 result = self._slow_forward(*input, **kwargs)
549 else:
--> 550 result = self.forward(*input, **kwargs)
551 for hook in self._forward_hooks.values():
552 hook_result = hook(self, input, result)
~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/conv.py in forward(self, input)
347
348 def forward(self, input):
--> 349 return self._conv_forward(input, self.weight)
350
351 class Conv3d(_ConvNd):
~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight)
344 _pair(0), self.dilation, self.groups)
345 return F.conv2d(input, weight, self.bias, self.stride,
--> 346 self.padding, self.dilation, self.groups)
347
348 def forward(self, input):
RuntimeError: Unable to find a valid cuDNN algorithm to run convolution
运行 nvidia-smi 显示:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01 Driver Version: 440.33.01 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 770 On | 00000000:01:00.0 N/A | N/A |
| 38% 50C P8 N/A / N/A | 624MiB / 4034MiB | N/A Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 Not Supported |
+-----------------------------------------------------------------------------+
我使用的是 Python 3.7、Pytorch 1.5,GPU 是 Nvidia GeForce GTX 770,在 Ubuntu 18.04.2 上运行。我在任何地方都没有找到该错误消息。它会响铃吗?
提前非常感谢。
解决方案
根据this answer for similar issue with tensorflow,它可能是因为达到了VRAM内存限制(从错误消息来看这是相当不直观的)。
对于我的 PyTorch 模型训练案例,减小批量大小有帮助。你可以试试这个,或者减少你的模型大小以消耗更少的 VRAM。
推荐阅读
- python - jinja2.exceptions.UndefinedError: 'dict object' 没有属性 'imageLinks'
- ios - tcp_input [C5.1:3] 标志=[R.] seq=2934102331
- c# - 如何将此 json 字符串转换为 jobject
- wikipedia-api - 维基百科转储文件上的功能
- html - 如何在物化 css 导航栏中将按钮内的图标居中对齐?
- python - python(ctypes)中的回调问题
- python-3.x - 在 Visual Studio 中创建虚拟环境
- node.js - 在 google-cloud/storage 中替代 AWS 的 bucket.putObject() 方法
- python - 张量从 (256, 256, 3) 重塑到 (1,256, 256, 3)
- python - 来自 pytest 的 status_code 不等于真实的