How to disable dropout AGAIN after setting training to true in Keras?-CodePudding

I have the following code:

def create_keras_model(num_classes):
    """
    This function compiles and returns a Keras model.
    Should be passed to KerasClassifier in the Keras scikit-learn API.
    """
    input_shape = (28, 28, 1)
    
    
    x_in = keras.Input(shape=input_shape)
    x = layers.Conv2D(32, kernel_size=(3, 3), activation="relu")(x_in)
    x = layers.Dropout(0.25)(x,training=True)
    x = layers.MaxPool2D(pool_size=(2, 2))(x)
    x = layers.Conv2D(64, kernel_size=(3, 3), activation="relu")(x)
    x = layers.Dropout(0.25)(x,training=True)    
    x = layers.MaxPool2D(pool_size=(2, 2))(x)
    x = layers.Flatten()(x)
    x = layers.Dropout(0.5)(x,training=True)
    x = layers.Dense(num_classes)(x)

    model = Model(inputs=x_in, outputs=x)    

    model.compile(loss='categorical_crossentropy', 
                  optimizer='adam',
                  metrics=['accuracy'])

    return model

I need training=True for my purposes. However, after that purpose I need training=False in the Dropout-Layers. Is there a way to achieve that easily?

One way would be to load the model weights and load them into a second model that does not have any Dropout-Layer in the first place, but this seems over complicated.

Setting "trainable=False" like:

model.layers[-2].training = False
model.layers[-5].training = False
model.layers[-8].training = False

does not work. Calling predict several times on the same input data still yields different results.

CodePudding user response：

IIUC, you could try omitting the Dropout layers during inference by creating a new model:

import tensorflow as tf

def create_keras_model(num_classes):
    """
    This function compiles and returns a Keras model.
    Should be passed to KerasClassifier in the Keras scikit-learn API.
    """
    input_shape = (28, 28, 1)
    
    
    x_in = tf.keras.Input(shape=input_shape)
    x = tf.keras.layers.Conv2D(32, kernel_size=(3, 3), activation="relu")(x_in)
    x = tf.keras.layers.Dropout(0.25)(x,training=True)
    x = tf.keras.layers.MaxPool2D(pool_size=(2, 2))(x)
    x = tf.keras.layers.Conv2D(64, kernel_size=(3, 3), activation="relu")(x)
    x = tf.keras.layers.Dropout(0.25)(x,training=True)    
    x = tf.keras.layers.MaxPool2D(pool_size=(2, 2))(x)
    x = tf.keras.layers.Flatten()(x)
    x = tf.keras.layers.Dropout(0.5)(x,training=True)
    x = tf.keras.layers.Dense(num_classes)(x)

    model = tf.keras.Model(inputs=x_in, outputs=x)    

    model.compile(loss='binary_crossentropy', 
                  optimizer='adam',
                  metrics=['accuracy'])
    return model 

model = create_keras_model(1)

new_model = tf.keras.Sequential()

for idx, l in enumerate(model.layers):
  if not l.name.startswith('dropout'):
    if idx==0:
      new_model.add(tf.keras.layers.InputLayer(input_shape=(28, 28, 1)))
    new_model.add(l)

What you can also try doing is setting the dropout rate to zero during inference:

for layer in model.layers:
    if isinstance(layer, tf.keras.layers.Dropout) and hasattr(layer, 'rate'):
        layer.rate = 0.0

Also check the docs regarding the Dropout layer:

Note that the Dropout layer only applies when training is set to True such that no values are dropped during inference. When using model.fit, training will be appropriately set to True automatically, and in other contexts, you can set the kwarg explicitly to True when calling the layer.

(This is in contrast to setting trainable=False for a Dropout layer. trainable does not affect the layer's behavior, as Dropout does not have any variables/weights that can be frozen during training.)

As mentioned in the comments, you can create two models before the loop, one with and one without dropout and use this: model2.set_weights(model.get_weights()).