python - Custom ImageDataGenerator (Keras) for large Dataset and multi input

Question

I have been trying to implement a custom ImageDataGenerator for a two input and one output image classification model from an hdf5 file (large dataset of 88k paired images). The network works as follow:

1. Each pair of images is fed into a VGG16 model, for feature extraction purposes, (both network have shared parameters and the layers are frozen for training purposes)
2. The output of each VGG16 net is concatenated and fed into a 3-FC layers.
3. The final output is a probability to decide if these two images are compatible (top and bottom cloth types for outfit matching).

This is the code of my custom generator which reads an hdf5 with two dataset on it, the paired images (88000,2,224,224,3) and the labels (88000,), 1-match 0-notmatch.

class HDF5DataGenerator:
    def __init__(self, dbPath, batchSize, preprocessors=None,aug=None,binarize=True,classes=2):
        self.batchSize = batchSize
        self.preprocessors = preprocessors
        self.aug = aug
        self.binarize = binarize
        self.classes = classes
        
        self.db = h5py.File(dbPath, 'r')
        self.numImages = self.db['images'].shape[0]
     
    def generator(self, passes=np.inf):
        epochs=0
        while epochs < passes:
            idx = np.array(range(0,numImages),dtype='int')
            np.random.shuffle(idx)
            for i in np.arange(0, self.numImages, self.batchSize):
                idxBatch = np.array(idx[i:i+batchSize])
                idxBatch.sort()
                
                imagesA = self.db['images'][idxBatch,0]
                imagesB = self.db['images'][idxBatch,1]
                labels = self.db['labels'][idxBatch]
                
                if self.binarize:
                    labels = to_categorical(labels, self.classes)
                    
                if self.preprocessors is not None:
                    procImagesA = []
                    for image in imagesA:
                        for p in self.preprocessors:
                            image = p.preprocess(image)
                        procImagesA.append(image)
                    imagesA = np.array(procImagesA)
                    
                    procImagesB = []
                    for image in imagesB:
                        for p in self.preprocessors:
                            image = p.preprocess(image)
                        procImagesB.append(image)
                    imagesB = np.array(procImagesB)
                
                if self.aug is not None:
                    (imagesA,labels) = next(self.aug.flow(imagesA, labels, batch_size=self.batchSize))
                    (imagesB,labels) = next(self.aug.flow(imagesB, labels, batch_size=self.batchSize))
                
                yield [imagesA,imagesB],labels

            epochs +=1
    
    def close(self):
        self.db.close()                           

When passing the generator to the fit_generation function as the following:

trainGen = HDF5DataGenerator('train.hdf5',
                             BATCH_SIZE,
                             preprocessors=[mp,iap],
                             aug=aug,
                             classes=2)

history =  model.fit(trainGen.generator(),
                steps_per_epoch = trainGen.numImages // BATCH_SIZE,
                #validation_data= testGen.generator(),
                #validation_steps = testGen.numImages // BATCH_SIZE,
                epochs=EPOCHS, 
                max_queue_size=10)

I get the following error which frankly I don't understand. I already checked the dimensionality of all the images written in the file since the incompatibility erro shows (None,224,224,1), which make me think that there was something wrong with the data, but that was not the issue.

ValueError: in user code:

    /Users/nicolas/.virtualenvs/cv/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:805 train_function  *
        return step_function(self, iterator)
    /Users/nicolas/.virtualenvs/cv/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:795 step_function  **
        outputs = model.distribute_strategy.run(run_step, args=(data,))
    /Users/nicolas/.virtualenvs/cv/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py:1259 run
        return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
    /Users/nicolas/.virtualenvs/cv/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py:2730 call_for_each_replica
        return self._call_for_each_replica(fn, args, kwargs)
    /Users/nicolas/.virtualenvs/cv/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py:3417 _call_for_each_replica
        return fn(*args, **kwargs)
    /Users/nicolas/.virtualenvs/cv/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:788 run_step  **
        outputs = model.train_step(data)
    /Users/nicolas/.virtualenvs/cv/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:755 train_step
        loss = self.compiled_loss(
    /Users/nicolas/.virtualenvs/cv/lib/python3.8/site-packages/tensorflow/python/keras/engine/compile_utils.py:203 __call__
        loss_value = loss_obj(y_t, y_p, sample_weight=sw)
    /Users/nicolas/.virtualenvs/cv/lib/python3.8/site-packages/tensorflow/python/keras/losses.py:152 __call__
        losses = call_fn(y_true, y_pred)
    /Users/nicolas/.virtualenvs/cv/lib/python3.8/site-packages/tensorflow/python/keras/losses.py:256 call  **
        return ag_fn(y_true, y_pred, **self._fn_kwargs)
    /Users/nicolas/.virtualenvs/cv/lib/python3.8/site-packages/tensorflow/python/util/dispatch.py:201 wrapper
        return target(*args, **kwargs)
    /Users/nicolas/.virtualenvs/cv/lib/python3.8/site-packages/tensorflow/python/keras/losses.py:1537 categorical_crossentropy
        return K.categorical_crossentropy(y_true, y_pred, from_logits=from_logits)
    /Users/nicolas/.virtualenvs/cv/lib/python3.8/site-packages/tensorflow/python/util/dispatch.py:201 wrapper
        return target(*args, **kwargs)
    /Users/nicolas/.virtualenvs/cv/lib/python3.8/site-packages/tensorflow/python/keras/backend.py:4833 categorical_crossentropy
        target.shape.assert_is_compatible_with(output.shape)
    /Users/nicolas/.virtualenvs/cv/lib/python3.8/site-packages/tensorflow/python/framework/tensor_shape.py:1134 assert_is_compatible_with
        raise ValueError("Shapes %s and %s are incompatible" % (self, other))

    ValueError: Shapes (None, None) and (None, 224, 224, 1) are incompatible

Hope you guys can help me to point me in the right direction to solve the issue.

Thank you for taking some time to read the post!

---

EDIT-1:

The following is the code of the model to be trained. I fixed the dimension problem.

from keras.layers import concatenate

img_shape = (224,224,3)

img_top = Input(shape=img_shape)
img_bottom = Input(shape=img_shape)
featureExtractor = vgg(img_shape)

feats_top = featureExtractor(img_top)
feats_bottom = featureExtractor(img_bottom)

combined = concatenate([feats_top,feats_bottom]) 

x = Dense(4096, activation='relu')(combined)
x = Dense(4096, activation='relu')(x)
x = Dense(4096, activation='relu')(x)
x = Dense(2, activation='softmax')(x)

model = Model(inputs=[img_top,img_bottom], outputs=x)

标签： pythontensorflowkerasdata-generation