python - RNN 中的隐藏大小与输入大小
问题描述
前提一:
关于 RNN 层中的神经元 - 我的理解是,在“每个时间步,每个神经元都从前一个时间步 y (t –1) 接收输入向量 x (t) 和输出向量” [1]:
前提2:
我的理解也是,在 Pytorch 的 GRU 层中,input_size和hidden_size的含义如下:
- input_size – 输入 x 中预期特征的数量
- hidden_size – 隐藏状态 h 的特征数量
所以很自然地,hidden_size应该代表 GRU 层中的神经元数量。
我的问题:
给定以下 GRU 层:
# assume that hidden_size = 3
class Encoder(nn.Module):
def __init__(self, src_dictionary_size, hidden_size):
super(Encoder, self).__init__()
self.embedding = nn.Embedding(src_dictionary_size, hidden_size)
self.gru = nn.GRU(input_size = hidden_size, hidden_size = hidden_size)
假设 hidden_size 为 3,我的理解是上面的 GRU 层将有 3 个神经元,每个神经元在每个时间步同时接受一个大小为 3 的输入向量。
我的问题是:为什么hidden_size和input_size的参数必须相等?即,为什么 3 个神经元中的每一个都不能接受大小为 5 的输入向量?
恰当的例子:以下两种情况都会产生尺寸不匹配:
self.gru = nn.GRU(input_size = hidden_size, hidden_size = hidden_size-1)
self.gru = nn.GRU(input_size = hidden_size, hidden_size = hidden_size+1)
[1] 热龙,奥雷利安。使用 Scikit-Learn 和 TensorFlow 进行动手机器学习(第 388 页)。奥莱利媒体。Kindle版。
[3] https://pytorch.org/docs/stable/nn.html#torch.nn.GRU
添加完整代码以实现重现性:
import torch
import torch.nn as nn
class Encoder(nn.Module):
def __init__(self, src_dictionary_size, hidden_size):
super(Encoder, self).__init__()
self.hidden_size = hidden_size
self.embedding = nn.Embedding(src_dictionary_size, hidden_size)
self.gru = nn.GRU(input_size = hidden_size, hidden_size = hidden_size-1)
def forward(self, pad_seqs, seq_lengths, hidden):
"""
Args:
pad_seqs of shape (max_seq_length, batch_size, 1): Padded source sequences.
seq_lengths: List of sequence lengths.
hidden of shape (1, batch_size, hidden_size): Initial states of the GRU.
Returns:
outputs of shape (max_seq_length, batch_size, hidden_size): Padded outputs of GRU at every step.
hidden of shape (1, batch_size, hidden_size): Updated states of the GRU.
"""
embedded_sqs = self.embedding(pad_seqs).squeeze(2)
packed_sqs = pack_padded_sequence(embedded_sqs, seq_lengths)
packed_output, h_n = self.gru(packed_sqs, hidden)
output, input_sizes = pad_packed_sequence(packed_output)
return output, h_n
def init_hidden(self, batch_size=1):
return torch.zeros(1, batch_size, self.hidden_size)
def test_Encoder_shapes():
hidden_size = 5
encoder = Encoder(src_dictionary_size=5, hidden_size=hidden_size)
# maximum word count
max_seq_length = 4
# num sentences
batch_size = 2
hidden = encoder.init_hidden(batch_size=batch_size)
# these are padded sequences (sentences of words). There are 2 sentences (i.e. 2 batches) with a maximum of 4 words.
pad_seqs = torch.tensor([
[1, 2],
[2, 3],
[3, 0],
[4, 0]
]).view(max_seq_length, batch_size, 1)
outputs, new_hidden = encoder.forward(pad_seqs=pad_seqs, seq_lengths=[4, 2], hidden=hidden)
assert outputs.shape == torch.Size([4, batch_size, hidden_size]), f"Bad outputs.shape: {outputs.shape}"
assert new_hidden.shape == torch.Size([1, batch_size, hidden_size]), f"Bad new_hidden.shape: {new_hidden.shape}"
print('Success')
test_Encoder_shapes()
解决方案
我刚刚解决了这个问题,错误是自己造成的。
结论:input_size和hidden_ size 的大小可以不同,这没有固有的问题。问题中的前提是正确陈述的。
上面(完整)代码的问题是 GRU 的初始隐藏状态没有正确的维度。初始隐藏状态必须与后续隐藏状态具有相同的维度。在我的例子中,初始隐藏状态的形状是 (1,2,5) 而不是 (1,2,4)。在前者中,5 表示嵌入向量的维数。4 表示 GRU 中的 hidden_size(神经元数量)。正确的代码如下:
import torch
import torch.nn as nn
class Encoder(nn.Module):
def __init__(self, src_dictionary_size, input_size, hidden_size):
super(Encoder, self).__init__()
self.hidden_size = hidden_size
self.embedding = nn.Embedding(src_dictionary_size, input_size)
self.gru = nn.GRU(input_size = input_size, hidden_size = hidden_size)
def forward(self, pad_seqs, seq_lengths, hidden):
"""
Args:
pad_seqs of shape (max_seq_length, batch_size, 1): Padded source sequences.
seq_lengths: List of sequence lengths.
hidden of shape (1, batch_size, hidden_size): Initial states of the GRU.
Returns:
outputs of shape (max_seq_length, batch_size, hidden_size): Padded outputs of GRU at every step.
hidden of shape (1, batch_size, hidden_size): Updated states of the GRU.
"""
embedded_sqs = self.embedding(pad_seqs).squeeze(2)
packed_sqs = pack_padded_sequence(embedded_sqs, seq_lengths)
packed_output, h_n = self.gru(packed_sqs, hidden)
output, input_sizes = pad_packed_sequence(packed_output)
return output, h_n
def init_hidden(self, batch_size=1):
return torch.zeros(1, batch_size, self.hidden_size)
def test_Encoder_shapes():
hidden_size = 4
embedding_size = 5
encoder = Encoder(src_dictionary_size=5, input_size = embedding_size, hidden_size = hidden_size)
print(encoder)
max_seq_length = 4
batch_size = 2
hidden = encoder.init_hidden(batch_size=batch_size)
pad_seqs = torch.tensor([
[1, 2],
[2, 3],
[3, 0],
[4, 0]
]).view(max_seq_length, batch_size, 1)
outputs, new_hidden = encoder.forward(pad_seqs=pad_seqs, seq_lengths=[4, 2], hidden=hidden)
assert outputs.shape == torch.Size([4, batch_size, hidden_size]), f"Bad outputs.shape: {outputs.shape}"
assert new_hidden.shape == torch.Size([1, batch_size, hidden_size]), f"Bad new_hidden.shape: {new_hidden.shape}"
print('Success')
test_Encoder_shapes()
推荐阅读
- javascript - 如何让 flexbox 子项填满垂直空间?
- c++ - 基于范围的 for 循环与 auto 说明符结合 static_cast
- python - 类型检查对象参数的正确方法
- mongodb - MongoDB - 更新数组的子文档以包含计算值
- javascript - 全局导入 Nuxt.js 模块
- xcode - lldb 抱怨名为“this”的变量
- c# - C# 任务卡住并且不会做任何事情
- maven - 在 Dockerfile 中包含一个 Maven 工件
- xaml - 如何绑定到 Xamarin 表单中的可重用内容视图?
- javascript - 是否有可能在选择选项中同时使用 JavaScript 和 PHP 的解决方案?