Meta-learning to find optimal model from pre-trained models in Tensorflow-CodePudding

I have many pre-trained models with a different number of layers (Models are not Sequential). Training data had a shape (1, 1, 103) for these models and output was a class label between 0 and 9.

I loaded these saved models, set all layers as non-trainable. I used theses models into new architecture as follows:

inp = keras.layers.Input(shape=(1,1,103), name = "new_input")

out_1 = model_1(inp) # model_1 is the name of variable where I loaded trained model
out_2 = model_2(inp)
out_3 = model_3(inp)
out_4 = model_4(inp)

x = keras.layers.concatenate([out_1, out_2, out_3, out_4])
out = keras.layers.dense(1)(x)

model = keras.models.Model(inputs=inp, outputs=out, name = "meta_model")

When I compile this model with optimizer = "sgd" and loss = "mse".

I didn't get any error until this point, but when I run model.fit(), I get this error TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'

I'm not sure where I'm going wrong.

The previous models were trained with "adam" optimizer and "sparse_categorical_crossentropy" loss and the dataset had 10 classes.

The objective of this model was to train this model with the same data and try to find out which model amongst the previously trained model was optimal.

Any other solution/suggestion to find an optimal number of layers using meta-learning would also be appreciated. I can manually find the optimal number of layers by trial and error but I want meta-model to find this based on the dataset.

eg: by training on dataset1 I found that there was no significant increase in accuracy after 7 layers whereas for dataset2 it reached its peak at 4 layers and adding more layers was useless.

CodePudding user response：

For Hyperparameters tunning I can recommend Ray Tune. I use it and I like this framework very much.

https://docs.ray.io/en/latest/tune/examples/tune_mnist_keras.html

CodePudding user response：

Regarding your error, try something like this:

import tensorflow as tf

def create_model():
  inputs = tf.keras.Input(shape=(1, 1, 103))
  x = tf.keras.layers.Flatten()(inputs)
  x = tf.keras.layers.Dense(5)(x)
  return tf.keras.Model(inputs, x)

model_1 = create_model()
model_2 = create_model()
model_3 = create_model()
model_4 = create_model()

inp = tf.keras.layers.Input(shape=(1,1,103), name = "new_input")

out_1 = model_1(inp)
out_2 = model_2(inp)
out_3 = model_3(inp)
out_4 = model_4(inp)

x = tf.keras.layers.Concatenate()([out_1, out_2, out_3, out_4])
out = tf.keras.layers.Dense(1)(x)

model = tf.keras.Model(inputs=inp, outputs=out, name = "meta_model")

model.compile(optimizer='adam', loss='mse')
model.summary()
x = tf.random.normal((10, 1,1,103))
y = tf.random.normal((10, 1 ))
model.fit(x, y, batch_size=2, epochs=2)

Model: "meta_model"
__________________________________________________________________________________________________
 Layer (type)                   Output Shape         Param #     Connected to                     
==================================================================================================
 new_input (InputLayer)         [(None, 1, 1, 103)]  0           []                               
                                                                                                  
 model_5 (Functional)           (None, 5)            520         ['new_input[0][0]']              
                                                                                                  
 model_6 (Functional)           (None, 5)            520         ['new_input[0][0]']              
                                                                                                  
 model_7 (Functional)           (None, 5)            520         ['new_input[0][0]']              
                                                                                                  
 model_8 (Functional)           (None, 5)            520         ['new_input[0][0]']              
                                                                                                  
 concatenate_1 (Concatenate)    (None, 20)           0           ['model_5[0][0]',                
                                                                  'model_6[0][0]',                
                                                                  'model_7[0][0]',                
                                                                  'model_8[0][0]']                
                                                                                                  
 dense_10 (Dense)               (None, 1)            21          ['concatenate_1[0][0]']          
                                                                                                  
==================================================================================================
Total params: 2,101
Trainable params: 2,101
Non-trainable params: 0
__________________________________________________________________________________________________
Epoch 1/2
5/5 [==============================] - 1s 3ms/step - loss: 3.1462
Epoch 2/2
5/5 [==============================] - 0s 4ms/step - loss: 1.8966
<keras.callbacks.History at 0x7fef00d17b10>