时间戳

首页 > 解决方案 > pytorch 中的 Dropconnect 实现

deep-learning - pytorch 中的 Dropconnect 实现

问题描述

我正在尝试为 Conv2D 和 transposeconv2D 层编写 dropconnect 代码。按照https://pytorchnlp.readthedocs.io/en/latest/_modules/torchnlp/nn/weight_drop.html中的教程创建它。

import torch
from torch.nn import Parameter

def _weight_drop(module, weights, dropout):
    for name_w in weights:
        w = getattr(module, name_w)
        del module._parameters[name_w]
        module.register_parameter(name_w + '_raw', Parameter(w))
    original_module_forward = module.forward

    def forward(*args, **kwargs):
        for name_w in weights:
            raw_w = getattr(module, name_w + '_raw')
            w = torch.nn.functional.dropout(raw_w, p=dropout, training=module.training)
            setattr(module, name_w, w)
        return original_module_forward(*args, **kwargs)
    setattr(module, 'forward', forward)

class WeightDropConv2d(torch.nn.Conv2d):
    def __init__(self, *args, weight_dropout=0.0, **kwargs):
        super().__init__(*args, **kwargs)
        weights = ['weight']
        _weight_drop(self, weights, weight_dropout)

class WeightDropConvTranspose2d(torch.nn.ConvTranspose2d):
    def __init__(self, *args, weight_dropout=0.0, **kwargs):
        super().__init__(*args, **kwargs)
        weights = ['weight']
        _weight_drop(self, weights, weight_dropout)

torch.version.cuda：1.1.0 火炬。版本：9.0.176

我在第二个时代收到以下错误：

Traceback (most recent call last):
  File "dropconnect.py", line 110, in <module>
    out = model(image)
  File "/home/sbhand2s/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "dropconnect.py", line 73, in forward
    out = self.c1(x)
  File "/home/sbhand2s/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "dropconnect.py", line 34, in forward
    setattr(module, name_w, w)
  File "/home/sbhand2s/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 558, in __setattr__
    .format(torch.typename(value), name))
TypeError: cannot assign 'torch.cuda.FloatTensor' as parameter 'weight' (torch.nn.Parameter or None expected)

当我从 .eval() 切换到 .train() 时，此错误发生在第二个 epoch。如果我不调用 .eval() 则不会发生此错误

关于为什么会发生此错误或如何以更好的方式实施 dropconnect 的任何建议？

复制问题的代码：

from collections import OrderedDict
import torch
from torch import nn

layers = []
layers.append(("conv_1", WeightDropConv2d(1,3,3,1,1,weight_dropout=0.5)))
layers.append(("conv_2", WeightDropConv2d(3,3,3,1,1,weight_dropout=0.5)))
layers.append(("conv_3", WeightDropConv2d(3,1,3,1,1,weight_dropout=0.5)))

model = nn.Sequential(OrderedDict(layers))

pred = model(torch.randn([1,1,3,3]))

model.eval()
pred = model(torch.randn([1,1,3,3]))

model.train()
pred = model(torch.randn([1,1,3,3]))

标签： deep-learningpytorch

解决方案

我也不能完全让那个方法工作（虽然有一个不同的错误），但这里有一个更简单的方法，它似乎正在工作：

for i in range(num_batches):

    orig_params = []
    for n, p in model.named_parameters():
        orig_params.append(p.clone())
        p.copy_(F.dropout(p.data, p=drop_prob) * (1 - drop_prob))

    output = model(input)
    loss = nn.CrossEntropyLoss()(output, label)
    optimizer.zero_grad()
    loss.backward()

    for orig_p, (n, p) in zip(orig_params, model.named_parameters()):  
        p.copy_(orig_p)

    optimizer.step()

（在 Pytorch 1.4 中测试）

推荐阅读