Joining the output of 2 Keras layers-CodePudding

I am trying to implement a joint model using Keras, and this is the architecture of the model.

However, I have difficulty in the concatenation of inputs from the subnetwork and the main network. The following are my codes:

import tensorflow as tf
from tensorflow.keras import layers
from tensorflow.keras.datasets import mnist
from tensorflow.keras.models import Model, Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Dense, Flatten, Reshape, Concatenate


def Autoencoder():
  input = layers.Input(shape=(256, 256, 5))
  layers.Flatten()

  x = layers.Conv2D(32, (3, 3), activation="relu", padding="same")(input)
  x = layers.Conv2D(32, (3, 3), activation="relu", padding="same")(x)
  x = layers.MaxPooling2D((2, 2), padding="same")(x)

  x = layers.Conv2D(64, (3, 3), activation="relu", padding="same")(x)
  x = layers.Conv2D(64, (3, 3), activation="relu", padding="same")(x)
  x = layers.MaxPooling2D((2, 2), padding="same")(x)

  x = layers.Conv2D(128, (3, 3), activation="relu", padding="same")(x)
  x = layers.Conv2D(128, (3, 3), activation="relu", padding="same")(x)
  x = layers.MaxPooling2D((2, 2), padding="same", name='last_layer')(x)
  autoencoder = Model(input, x)
  return autoencoder.get_layer('last_layer')



def Subnetwork():
  input = layers.Input(shape=(12,1))

  x = layers.Flatten()(input)
  x = layers.Dense(4096, activation="relu")(x)
  x = layers.Reshape((32, 32, 4), name='last_layer')(x)
  subnetwork = Model(input, x)
  return subnetwork.get_layer('last_layer')

def Joint():
  layer_autoencoder = Autoencoder()
  layer_subnetwork = Subnetwork()
  merged= Concatenate([layer_autoencoder, layer_subnetwork])
  model = Model(inputs=[layer_autoencoder, layer_subnetwork], outputs=merged)
  return model

Model = Joint()
Model.summary()

The error message looks like this:

ValueError: Found unexpected instance while processing input tensors for keras functional model. Expecting KerasTensor which is from tf.keras.Input() or output from keras layer call(). Got: <keras.layers.pooling.MaxPooling2D object at 0x7fbfd8634990>

Do anyone know what causes the error and what is the correct solution?

CodePudding user response：

You should redefine the input layer also in the Joint model

def Joint():

  input_autoencoder = layers.Input(shape=(256, 256, 5))  ### define input layer
  layer_autoencoder = Autoencoder()(input_autoencoder)  ### pass input to Autoencoder
  input_subnetwork = layers.Input(shape=(12, 1))  ### define input layer
  layer_subnetwork = Subnetwork()(input_subnetwork)  ### pass input to Subnetwork
  
  merged= Concatenate()([layer_autoencoder, layer_subnetwork])  ### it's Concatenate()([...]) not Concatenate([...])
  model = Model([input_autoencoder, input_subnetwork], merged)  ### use correct inputs
  return model

Pay attention also to Autoencoder and Subnetwork. They must return TF model instances. So they become:

def Autoencoder():
  ...
  autoencoder = Model(input, x)
  return autoencoder

def Subnetwork():
  ...
  subnetwork = Model(input, x)
  return subnetwork