How to join 2 keras models into 1 with reshaping inbetween to train those models together as 1 model-CodePudding

I have 2 keras models. The 1-st is for encoding 256x192 images into 128-dimensional vectors. It's input shape is (None, 192, 256, 1) and output is (None, 128) (None is for batch size, it is always the first element of shape in all keras models). And the 2-nd model should compare 2 vectors from the 1-st model. So it's input shape is (None, 256) and output shape is (None, 1).

I want to train those models with regular fit method. So I need to somehow concatenate them into one model to train it. The input shape of that model should probably be (None, 2, 192, 256, 1) (2 is for 2 images that should be compared), output shape - (None, 1). And also I want to keep first 2 models to use them separately later.

Here's what I've tried:

class _layer_1(keras.layers.Layer):
    def call(self, x):
        return tf.reshape(x, (x.shape[0] * x.shape[1],)   x.shape[2:])

class _layer_2(keras.layers.Layer):
    def call(self, x):
        return tf.reshape(x, (x.shape[0] // 2, x.shape[1] * 2)   x.shape[2:])

model = keras.Sequential([
    keras.Input(shape=(2, 192, 256, 1)),
    _layer_1(),
    encoder_model,
    _layer_2(),
    comparator_model
])

model.compile(optimizer=keras.optimizers.Adam(learning_rate=3e-5), loss='mean_squared_error')

This should have manually reshaped tensors as I need. But it doesn't work. Here's the issue that appeared when I tried to use the fit method:

Traceback (most recent call last):
  File "comparator_train.py", line 61, in <module>
    get_models()[2].fit(np.random.sample((30, 2, 192, 256, 1),), np.random.sample((30, 1),))
  File "comparator_train.py", line 54, in get_models
    comparator_model
  File "/opt/miniconda3/lib/python3.7/site-packages/tensorflow/python/training/tracking/base.py", line 587, in _method_wrapper
    result = method(self, *args, **kwargs)
  File "/opt/miniconda3/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/tmp/__autograph_generated_filecoqup163.py", line 12, in tf__call
    retval_ = ag__.converted_call(ag__.ld(tf).reshape, (ag__.ld(x), (((ag__.ld(x).shape[0] * ag__.ld(x).shape[1]),)   ag__.ld(x).shape[2:])), None, fscope)
TypeError: Exception encountered when calling layer "private_layer_1" (type _layer_1).

in user code:

    File "comparator_train.py", line 43, in call  *
        return tf.reshape(x, (x.shape[0] * x.shape[1],)   x.shape[2:])

    TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'


Call arguments received by layer "private_layer_1" (type _layer_1):
  • x=tf.Tensor(shape=(None, 2, 192, 256, 1), dtype=float32)

So how do I properly make that concatenated model?

Thanks for any help!

Update: Here are the models:

encoder_model = keras.Sequential([
    keras.layers.Input(shape=(192, 256, 1)),
    keras.layers.SeparableConv2D(filters=128, kernel_size=12, strides=2, padding='same', activation=keras.layers.LeakyReLU(alpha=0.1)),
    keras.layers.BatchNormalization(),
    keras.layers.SeparableConv2D(filters=256, kernel_size=12, strides=3, padding='same', activation=keras.layers.LeakyReLU(alpha=0.1)),
    keras.layers.BatchNormalization(),
    keras.layers.SeparableConv2D(filters=256, kernel_size=(12, 12), strides=2, padding='same', activation=keras.layers.LeakyReLU(alpha=0.1)),
    keras.layers.BatchNormalization(),
    keras.layers.Dropout(0.8),
    keras.layers.SeparableConv2D(filters=128, kernel_size=(8, 11), strides=1, padding='valid', activation=keras.layers.LeakyReLU(alpha=0.1)),
    keras.layers.Dropout(0.14),
    keras.layers.MaxPooling2D(pool_size=(3, 3)),
    keras.layers.Flatten(),
    keras.layers.Dense(units=128, activation='sigmoid')
])

comparator_model = keras.Sequential([
    keras.layers.Input(shape=(256,)),
    keras.layers.Dense(units=256, activation=keras.layers.LeakyReLU(alpha=0.1)),
    keras.layers.Dense(units=256, activation=keras.layers.LeakyReLU(alpha=0.1)),
    keras.layers.Dense(units=1, activation='sigmoid')
])

CodePudding user response：

I finally found the solution! It is to separate the batch of pairs of images into 2 batches of images (each of them having the original batch_size) so that i don't actually need to double the batch size. Then it is possible to apply encoder_model on each of those batches separately and join the outputs for the comparator_model. Here is the code that works:

inputs = keras.Input(shape=(2, 192, 256, 1)) # original batch of shape (None, 2, image_shape)
images1, images2 = tf.transpose(inputs, (1, 0, 2, 3, 4)) # (None, 2, image_shape) -> (2, None, image_shape) -> batch1=(None, image_shape), batch2=(None, image_shape)
v1, v2 = encoder_model(images1), encoder_model(images2) # applying 1-st model to get output1=(None, vector_size) and output2=(None, vector_size)
v = keras.layers.Reshape((256,),)(tf.transpose([v1, v2], (1, 0, 2))) # output1, output2 -> (2, None, vector_size) -> (None, 2, vector_size) -> (None, 2*vector_size)
outputs = comparator_model(v) # applying 2-nd model
model = keras.Model(inputs=inputs, outputs=outputs) # combined model

model.compile(optimizer=keras.optimizers.Adam(learning_rate=3e-5), loss='mean_squared_error')

(Then I will train my new model with model.fit so that encoder and comparator models will be trained within that model)

CodePudding user response：

This is essentially your accepted answer as a Functional API model. As you can see, what you have considered "separate models" are not separate models, but just layers inside a variable. If you want separate models, you'll need to independently train them. You have one model and it's not even an ensemble model.

inputs = keras.Input(shape=(2, 192, 256, 1))
images1, images2 = tf.transpose(inputs, (1, 0, 2, 3, 4))
v1 = Input(shape=(192, 256, 1))(images1)
v1 = SeparableConv2D(filters=128, kernel_size=12, strides=2, padding='same', activation=keras.layers.LeakyReLU(alpha=0.1))(v1)
v1 = BatchNormalization()(v1)
v1 = ...
v1 = Flatten()(v1),
v1 = Dense(units=128, activation='sigmoid')(v1)

v2 = Input(shape=(192, 256, 1))(images2)
v2 = ...
v2 = Dense(units=128, activation='sigmoid')(v2)

v = Reshape((256,),)(tf.transpose([v1, v2], (1, 0, 2)))
# your original question said concatenate so this is how that wouldve been
# v = Concatenate()([v1, v2])

out = Dense(units=256, activation=keras.layers.LeakyReLU(alpha=0.1))(v)
out = Dense(units=256, activation=keras.layers.LeakyReLU(alpha=0.1))(out)
out = Dense(units=1, activation='sigmoid')(out)