python - 无法在多模式 seq2seq 中计算输出 KerasTensor

Question

我正在尝试使用帖子的图像、标题和源 subreddit从 subreddit 帖子生成评论。

如果您不知道 subreddit 是什么，只需将其视为帖子的一个类别，例如猫、狗、汽车

我将 CNN 用于图像，一个简单的神经网络用于 subreddits，LSTM 用于标题，最后将 LSTM 用于评论生成。

我让它工作了，但推理部分很难编码，所以我重构了它，所以最终模型的每个组件都是一个小模型。但是，当我尝试将所有内容放在一起时，出现如下所示的错误。

我该如何解决它，问题的原因是什么？

错误

AssertionError                            Traceback (most recent call last)
/tmp/ipykernel_42/2610877267.py in <module>
      5 inputs = encoder.inputs
      6 
----> 7 result = decoder(decoder_target, encoder(inputs))

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py in __call__(self, *args, **kwargs)
    950     if _in_functional_construction_mode(self, inputs, args, kwargs, input_list):
    951       return self._functional_construction_call(inputs, args, kwargs,
--> 952                                                 input_list)
    953 
    954     # Maintains info about the `Layer.call` stack.

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py in _functional_construction_call(self, inputs, args, kwargs, input_list)
   1089         # Check input assumptions set after layer building, e.g. input shape.
   1090         outputs = self._keras_tensor_symbolic_call(
-> 1091             inputs, input_masks, args, kwargs)
   1092 
   1093         if outputs is None:

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py in _keras_tensor_symbolic_call(self, inputs, input_masks, args, kwargs)
    820       return nest.map_structure(keras_tensor.KerasTensor, output_signature)
    821     else:
--> 822       return self._infer_output_signature(inputs, args, kwargs, input_masks)
    823 
    824   def _infer_output_signature(self, inputs, args, kwargs, input_masks):

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py in _infer_output_signature(self, inputs, args, kwargs, input_masks)
    861           # TODO(kaftan): do we maybe_build here, or have we already done it?
    862           self._maybe_build(inputs)
--> 863           outputs = call_fn(inputs, *args, **kwargs)
    864 
    865         self._handle_activity_regularization(inputs, outputs)

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py in call(self, inputs, training, mask)
    423     """
    424     return self._run_internal_graph(
--> 425         inputs, training=training, mask=mask)
    426 
    427   def compute_output_shape(self, input_shape):

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py in _run_internal_graph(self, inputs, training, mask)
    567     for x in self.outputs:
    568       x_id = str(id(x))
--> 569       assert x_id in tensor_dict, 'Could not compute output ' + str(x)
    570       output_tensors.append(tensor_dict[x_id].pop())
    571 

AssertionError: Could not compute output KerasTensor(type_spec=TensorSpec(shape=(None, 80, 5000), dtype=tf.float32, name=None), name='dense_25/truediv:0', description="created by layer 'dense_25'")

标题编码器

获取嵌入它的帖子的标记化标题并返回 LSTM 的状态。

def build_title_embeddings():
    encoder_inputs = Input(shape=(max_title_len), name='Post title')
    encoder_emb =  Embedding(vocab_size, title_embedding_size)(encoder_inputs)
    encoder = LSTM(title_hidden_size, return_state=True)
    _, state_h, state_c = encoder(encoder_emb)
    
    return Model(inputs=encoder_inputs, outputs=[state_h, state_c], name="Title encoder")

美国有线电视新闻网

Cnn 在 imagenet 上预训练keras.applications.EfficientNetB3

def build_cnn():
    cnn = EfficientNetB3(include_top=False, input_shape=(300,300,3), pooling="avg")
    cnn.trainable = False
    return Model(inputs=cnn.inputs, outputs=cnn.outputs, name="cnn")

子版块嵌入

将一个热编码向量作为输入。它的目的是减少输入的维度

def build_subreddit_embeddings():
    input_subreddit = Input(shape=(num_subreddits), name='One hot encoded subreddit')
    sub_emb = Dense(subreddit_embedding_size, activation='relu')(input_subreddit)
    
    return Model(inputs=input_subreddit, outputs=sub_emb, name="Subreddit embedding")

最终编码器

获取上述所有内容并为解码器生成初始状态

def build_final_encoder():
    cnn = build_cnn()
    subreddit_embeddings = build_subreddit_embeddings()
    title_embeddings = build_title_embeddings()
    
    merged = Concatenate()([cnn.output, subreddit_embeddings.output, *title_embeddings.output])

    intermediate_layer = Dense(intermediate_layer_size, activation='relu')(merged)

    state1 = Dense(decoder_hidden_size, activation='relu')(intermediate_layer)
    state2 = Dense(decoder_hidden_size, activation='relu')(intermediate_layer)

    return Model(inputs=[cnn.inputs, subreddit_embeddings.inputs, title_embeddings.inputs], outputs=[state1,state2], name="post_encoder")

解码器

作为最终编码器的输入输出并生成评论标记。

解码器的输入是自己预测的先前标记

def build_decoder():
    decoder_inputs = Input(shape=(max_decoder_len,), name="Decoder inputs")

    state_inputs1 = Input(shape=(decoder_hidden_size,), name="Decoder state input 1")
    state_inputs2 = Input(shape=(decoder_hidden_size,), name="Decoder state input 2")

    decoder_emb =  Embedding(vocab_size, decoder_embedding_size)(decoder_inputs)

    decoder_lstm = LSTM(decoder_hidden_size, return_sequences=True, return_state=True, name="decoder_lstm")
    
    decoder_outputs, _, _ = decoder_lstm(decoder_emb, initial_state=[state_inputs1, state_inputs2])
    
    decoder_outputs = Dense(vocab_size, activation="softmax")(decoder_outputs)
    
    return Model(inputs=[decoder_inputs, [state_inputs1,state_inputs2]], outputs=decoder_outputs, name="final_decoder")

尝试将所有东西放在一起

解码器目标是正确的令牌 id 在训练期间移动了一个，在推理期间是先前的预测

例如在训练期间

解码器目标 -['<start_of_sentence_token>','I','like','pizza']

正确答案 -['I','like','pizza','<end_of_sentence_token>']

decoder_target = Input(shape=(max_decoder_len,), name="Decoder target")

encoder = build_final_encoder()
decoder = build_decoder()
inputs = encoder.inputs

result = decoder(decoder_target, encoder(inputs)) # Errors happen here

标签： pythontensorflowkerastf.kerasseq2seq