问题描述

我正在尝试使用 GoogleNet（Pytorch）对猫与狗进行分类。每个类包含 4000 张要训练的图像和 1000 张要测试的图像，大小为 300*300。我的电脑有 32GB RAM 和 RTX 2080 Super 显卡。而这个错误发生在它刚开始训练的时候。下面是我的代码 GoogleNet 的一部分：

class Inception(nn.Module) :
    def __init__(self, in_dim, out_dim1, mid_dim3, out_dim3, mid_dim5, out_dim5, pool):
       super(Inception, self).__init__()
        self.lay1 = nn.Sequential(nn.Conv2d(in_dim, out_dim1, kernel_size= 1), nn.BatchNorm2d(out_dim1), nn.ReLU())
        self.lay2 = nn.Sequential(nn.Conv2d(in_dim, mid_dim3, kernel_size = 1), nn.BatchNorm2d(mid_dim3), nn.ReLU(), nn.Conv2d(mid_dim3, out_dim3, kernel_size = 3, padding = 1), nn.BatchNorm2d(out_dim3), nn.ReLU())
        self.lay3 = nn.Sequential(nn.Conv2d(in_dim, mid_dim5, kernel_size = 1), nn.BatchNorm2d(mid_dim5), nn.ReLU(), nn.Conv2d(mid_dim5, out_dim5, kernel_size = 3, padding = 1), nn.BatchNorm2d(out_dim5), nn.ReLU(), nn.Conv2d(out_dim5, out_dim5, kernel_size = 3, padding = 1), nn.BatchNorm2d(out_dim5), nn.ReLU())
        self.lay4 = nn.Sequential(nn.MaxPool2d(3, 1, 1), nn.Conv2d(in_dim, pool, kernel_size = 1), nn.BatchNorm2d(pool), nn.ReLU())

    def forward(self, x):
        y1 = self.lay1(x)
        y2 = self.lay2(x)
        y3 = self.lay3(x)
        y4 = self.lay4(x)

        return torch.cat([y1, y2, y3, y4], 1)


class Google(nn.Module):
    def __init__(self):
        super(Google, self).__init__()
        self.pre_lay = nn.Sequential(nn.Conv2d(1, 48, 3, padding = 1), nn.BatchNorm2d(48), nn.ReLU())
        self.glay1 = Inception(48, 16, 24, 32, 4, 8, 8)
        self.glay2 = Inception(64, 32, 32, 48, 8, 24, 16) # input channel : prev output channel sum(torch.cat)
        self.maxpool = nn.MaxPool2d(3, stride = 2, padding = 1)
        self.glay3 = Inception(120, 48, 24, 52, 4, 12, 16)
        self.glay4 = Inception(128, 40, 28, 56, 6, 16, 16)
        self.glay5 = Inception(128, 32, 32, 64, 6, 16, 16)
        self.glay6 = Inception(128, 28, 36, 72, 8, 16, 16)
        self.glay7 = Inception(132, 64, 40, 80, 8, 32, 32)
        self.glay8 = Inception(208, 64, 40, 80, 8, 32, 32)
        self.glay9 = Inception(208, 96, 48, 96, 12, 32, 32)


        self.avgpool = nn.AvgPool2d(8, stride = 1)
        self.linear = nn.Linear(47872, 2)
        gc.collect()
        torch.cuda.empty_cache()


    def forward(self, x):
        gc.collect()
        torch.cuda.empty_cache()
        # with torch.no_grad() : // 
            out = self.pre_lay(x) # CUDA out of memory Occurs!!
            out = self.glay1(out)
            out = self.glay2(out)
            out = self.maxpool(out)
            out = self.glay3(out)
            out = self.glay4(out)
            out = self.glay5(out)
            out = self.glay6(out)
            out = self.glay7(out)
            out = self.maxpool(out)
            out = self.glay8(out)
            out = self.glay9(out)
            out = self.avgpool(out)
            out = out.view(out.size(0), -1)
            print("Out size : ", out.size())
            out = self.linear(out)

        return out

正如我所写，在 GoogleNet 的第一步发生错误。发生错误的前一行，有人建议添加 torch.no_grad() ：如果存在 cuda 内存错误但内存不是问题，则在模型的 forward 函数中添加。但是随后张量的元素 0 不需要 grad 并且没有 grad_fn错误出来了。我在怀疑大量使用 GPU 内存的每一步都尝试了 empty_cache，但仍然无法正常工作。

如果有人遇到过类似错误或知道原因，您的建议一定非常感谢。

添加完整的错误消息

cuda内存错误

这就是我得到的内存错误。

毕业错误

下面的错误是在我在 forward() 添加 torch.no_grad() 之后

标签： pytorch

pytorch - PyTorch：cuda 内存不足但剩余内存足够（添加错误消息）

问题描述

添加完整的错误消息

解决方案

推荐阅读