时间戳

首页 > 解决方案 > How to concatenate two models in keras?

python - How to concatenate two models in keras?

问题描述

I wanted to use this model but we cannot use merge anymore.

image_model = Sequential([
    Dense(embedding_size, input_shape=(2048,), activation='relu'),
    RepeatVector(max_len)
])
caption_model = Sequential([
    Embedding(vocab_size, embedding_size, input_length=max_len),
    LSTM(256, return_sequences=True),
    TimeDistributed(Dense(300))
])
final_model = Sequential([
    Merge([image_model, caption_model], mode='concat', concat_axis=1),
    Bidirectional(LSTM(256, return_sequences=False)),
    Dense(vocab_size),
    Activation('softmax')
])

i rewrote this in following way,excluding final_model:

image_in = Input(shape=(2048,))
caption_in = Input(shape=(max_len, vocab_size))
merged = concatenate([image_model(image_in), caption_model(caption_in)],axis=0)
latent = Bidirectional(LSTM(256, return_sequences=False))(merged)
out = Dense(vocab_size, activation='softmax')(latent)
final_model = Model([image_in, caption_in], out)

final_model.compile(loss='categorical_crossentropy', optimizer=RMSprop(), metrics=['accuracy'])
final_model.summary()

This also gave me:

ValueError: "input_length" is 40, but received input has shape (None, 40, 8256).

Can anyone help to fix it? source:https://github.com/yashk2810/Image-Captioning/blob/master/Image%20Captioning%20InceptionV3.ipynb

标签： pythontensorflowmachine-learningkerastf.keras

解决方案

you should define the caption_in as 2D: Input(shape=(max_len,)). in your case, the concatenation must be operated on the last axis: axis=-1. the rest seems ok

embedding_size=300
max_len=40
vocab_size=8256

image_model = Sequential([
    Dense(embedding_size, input_shape=(2048,), activation='relu'),
    RepeatVector(max_len)
])
caption_model = Sequential([
    Embedding(vocab_size, embedding_size, input_length=max_len),
    LSTM(256, return_sequences=True),
    TimeDistributed(Dense(300))
])

image_in = Input(shape=(2048,))
caption_in = Input(shape=(max_len,))
merged = concatenate([image_model(image_in), caption_model(caption_in)],axis=-1)
latent = Bidirectional(LSTM(256, return_sequences=False))(merged)
out = Dense(vocab_size, activation='softmax')(latent)
final_model = Model([image_in, caption_in], out)

final_model.compile(loss='categorical_crossentropy', optimizer=RMSprop(), metrics=['accuracy'])
final_model.summary()

推荐阅读