I've been joining this hackathon and playing with keras callbacks and neural network, may I know if there is a way to monitor not only loss or val_loss but BOTH of them to avoid overfitting either the test or train set? e.g: can i put a function for the monitor field instead of just one field name?

If I want to monitor val_loss to pick the lowest but I also want a second criteria to pick the minimum difference between val_loss and loss.

CodePudding user response：

You can choose between two approaches:

1. Create a custom metric to record the metric you want, by subclassing tf.keras.metrics.Metric. See https://www.tensorflow.org/api_docs/python/tf/keras/metrics/Metric for an example.
   You can then use your metric in standard callbacks e.g. EarlyStopping()

2. Create a custom callback to do the calculation (and take the action) you want, by subclassing tf.keras.callbacks.CallBack. See https://www.tensorflow.org/guide/keras/custom_callback for how to do this.

CodePudding user response：

I have an answer to a problem that is pretty similar to this, here.

Basically, it is not possible to monitor multiple metrics with keras callbacks. However you could define a custom callback (see the documentation for more info) that can access the logs at each epoch and do some operations.

Let's say if you want to monitor loss and val_loss you can do something like this:

class CombineCallback(tf.keras.callbacks.Callback):

    def __init__(self, **kargs):
        super(CombineCallback, self).__init__(**kargs)

    def on_epoch_end(self, epoch, logs={}):
        logs['combine_metric'] = logs['val_loss']   logs['loss']

Side note: the most important thing in my opinion is to monitor the validation loss. Train loss of course will keep dropping, so it is not really that meaningful to observe. If you really want to monitor them both I suggest you adding a multiplicative factor and give more weight to validation loss. In this case:

class CombineCallback(tf.keras.callbacks.Callback):

    def __init__(self, **kargs):
        super(CombineCallback, self).__init__(**kargs)

    def on_epoch_end(self, epoch, logs={}):
        factor = 0.8
        logs['combine_metric'] = factor * logs['val_loss']   (1-factor) * logs['loss']

Then you can use it like this:

model.fit(
    ...
    callbacks=[CombineCallback()],
)

Also you can draw more inspiration here.