Helle,

I am trying to train Bertclassifier on a classification task by finetuning it but I am having troubles understanding what is display during the training.

I put a small sample of what I get

{'loss': 1.1328, 'learning_rate': 4.994266055045872e-05, 'epoch': 0.0}

{'loss': 1.0283, 'learning_rate': 4.942660550458716e-05, 'epoch': 0.02}

{'eval_loss': 0.994676947593689, 'eval_accuracy': 0.507755277897458, 'eval_f1': array([0.00770713, 0.6359277 , 0.44546742]), 'eval_f1_mi': 0.507755277897458, 'eval_f1_ma': 0.36303408438190915, 'eval_runtime': 10.8296, 'eval_samples_per_second': 428.642, 'eval_steps_per_second': 13.482, 'epoch': 0.02}

{'loss': 1.0075, 'learning_rate': 4.8853211009174314e-05, 'epoch': 0.05}
{'eval_loss': 1.0286471843719482, 'eval_accuracy': 0.46122361051271005, 'eval_f1': array([0.25      , 0.48133484, 0.51830986]), 'eval_f1_mi': 0.46122361051271005, 'eval_f1_ma': 0.41654823359462956, 'eval_runtime': 10.8256, 'eval_samples_per_second': 428.796, 'eval_steps_per_second': 13.486, 'epoch': 0.05}

{'loss': 0.9855, 'learning_rate': 4.827981651376147e-05, 'epoch': 0.07}
{'eval_loss': 0.9796209335327148, 'eval_accuracy': 0.5320982335200345, 'eval_f1': array([0.14783347, 0.6772202 , 0.2726257 ]), 'eval_f1_mi': 0.5320982335200345, 'eval_f1_ma': 0.36589312424069026, 'eval_runtime': 10.8505, 'eval_samples_per_second': 427.813, 'eval_steps_per_second': 13.456, 'epoch': 0.07}

{'loss': 1.0022, 'learning_rate': 4.7706422018348626e-05, 'epoch': 0.09}
{'eval_loss': 0.968146026134491, 'eval_accuracy': 0.5364067212408444, 'eval_f1': array([0.38389789, 0.60565553, 0.5487042 ]), 'eval_f1_mi': 0.5364067212408444, 'eval_f1_ma': 0.5127525387411823, 'eval_runtime': 10.9701, 'eval_samples_per_second': 423.15, 'eval_steps_per_second': 13.309, 'epoch': 0.09}

{'loss': 0.9891, 'learning_rate': 4.713302752293578e-05, 'epoch': 0.11}
{'eval_loss': 0.9413465261459351, 'eval_accuracy': 0.556872037914692, 'eval_f1': array([0.37663886, 0.68815745, 0.28154206]), 'eval_f1_mi': 0.556872037914692, 'eval_f1_ma': 0.4487794533693059, 'eval_runtime': 10.9316, 'eval_samples_per_second': 424.642, 'eval_steps_per_second': 13.356, 'epoch': 0.11}

{'loss': 0.9346, 'learning_rate': 4.655963302752294e-05, 'epoch': 0.14}
{'eval_loss': 0.9142090082168579, 'eval_accuracy': 0.5769065058164584, 'eval_f1': array([0.19836066, 0.68580399, 0.570319  ]), 'eval_f1_mi': 0.5769065058164584, 'eval_f1_ma': 0.4848278830170361, 'eval_runtime': 10.9471, 'eval_samples_per_second': 424.04, 'eval_steps_per_second': 13.337, 'epoch': 0.14}

{'loss': 0.9394, 'learning_rate': 4.5986238532110096e-05, 'epoch': 0.16}
{'eval_loss': 0.8802705407142639, 'eval_accuracy': 0.5857389056441189, 'eval_f1': array([0.30735931, 0.71269565, 0.4255121 ]), 'eval_f1_mi': 0.5857389056441189, 'eval_f1_ma': 0.4818556879387581, 'eval_runtime': 10.9824, 'eval_samples_per_second': 422.677, 'eval_steps_per_second': 13.294, 'epoch': 0.16}

{'loss': 0.8993, 'learning_rate': 4.541284403669725e-05, 'epoch': 0.18}
{'eval_loss': 0.8535333871841431, 'eval_accuracy': 0.5980180956484275, 'eval_f1': array([0.37174211, 0.7155305 , 0.41662443]), 'eval_f1_mi': 0.5980180956484275, 'eval_f1_ma': 0.5012990131553724, 'eval_runtime': 10.8245, 'eval_samples_per_second': 428.842, 'eval_steps_per_second': 13.488, 'epoch': 0.18}

{'loss': 0.9482, 'learning_rate': 4.483944954128441e-05, 'epoch': 0.21}
{'eval_loss': 0.9535377621650696, 'eval_accuracy': 0.541792330891857, 'eval_f1': array([0.31955151, 0.59248471, 0.57414105]), 'eval_f1_mi': 0.541792330891857, 'eval_f1_ma': 0.4953924209116825, 'eval_runtime': 10.9767, 'eval_samples_per_second': 422.896, 'eval_steps_per_second': 13.301, 'epoch': 0.21}

{'loss': 0.8488, 'learning_rate': 4.426605504587156e-05, 'epoch': 0.23}
{'eval_loss': 0.8357231020927429, 'eval_accuracy': 0.6214993537268418, 'eval_f1': array([0.35536603, 0.73122392, 0.50070588]), 'eval_f1_mi': 0.6214993537268418, 'eval_f1_ma': 0.5290986104916023, 'eval_runtime': 10.9206, 'eval_samples_per_second': 425.069, 'eval_steps_per_second': 13.369, 'epoch': 0.23}

{'loss': 0.8893, 'learning_rate': 4.369266055045872e-05, 'epoch': 0.25}
{'eval_loss': 0.7578970789909363, 'eval_accuracy': 0.6712623869021973, 'eval_f1': array([0.41198502, 0.77171541, 0.65677419]), 'eval_f1_mi': 0.6712623869021973, 'eval_f1_ma': 0.6134915401312347, 'eval_runtime': 10.9765, 'eval_samples_per_second': 422.902, 'eval_steps_per_second': 13.301, 'epoch': 0.25}

{'loss': 0.9003, 'learning_rate': 4.311926605504588e-05, 'epoch': 0.28}
{'eval_loss': 0.791412353515625, 'eval_accuracy': 0.6535975872468763, 'eval_f1': array([0.45641646, 0.76072942, 0.53744893]), 'eval_f1_mi': 0.6535975872468763, 'eval_f1_ma': 0.5848649380875267, 'eval_runtime': 10.9302, 'eval_samples_per_second': 424.696, 'eval_steps_per_second': 13.358, 'epoch': 0.28}

{'loss': 0.8345, 'learning_rate': 4.2545871559633024e-05, 'epoch': 0.3}
{'eval_loss': 0.7060380578041077, 'eval_accuracy': 0.6999138302455838, 'eval_f1': array([0.50152905, 0.79205975, 0.64349863]), 'eval_f1_mi': 0.6999138302455838, 'eval_f1_ma': 0.6456958112539298, 'eval_runtime': 10.9475, 'eval_samples_per_second': 424.023, 'eval_steps_per_second': 13.336, 'epoch': 0.3}

{'loss': 0.8149, 'learning_rate': 4.1972477064220184e-05, 'epoch': 0.32}
{'eval_loss': 0.6717478036880493, 'eval_accuracy': 0.7259801809564843, 'eval_f1': array([0.50805932, 0.81245738, 0.71325735]), 'eval_f1_mi': 0.7259801809564843, 'eval_f1_ma': 0.6779246805922554, 'eval_runtime': 10.7574, 'eval_samples_per_second': 431.519, 'eval_steps_per_second': 13.572, 'epoch': 0.32}

{'loss': 0.8343, 'learning_rate': 4.139908256880734e-05, 'epoch': 0.34}
{'eval_loss': 0.6306226253509521, 'eval_accuracy': 0.7455838000861698, 'eval_f1': array([0.58873995, 0.82795018, 0.70917226]), 'eval_f1_mi': 0.7455838000861698, 'eval_f1_ma': 0.7086207951089967, 'eval_runtime': 10.9006, 'eval_samples_per_second': 425.849, 'eval_steps_per_second': 13.394, 'epoch': 0.34}

{'loss': 0.7711, 'learning_rate': 4.0825688073394495e-05, 'epoch': 0.37}
{'eval_loss': 0.6052485108375549, 'eval_accuracy': 0.7619560534252477, 'eval_f1': array([0.62346588, 0.84259464, 0.73186813]), 'eval_f1_mi': 0.7619560534252476, 'eval_f1_ma': 0.7326428851759276, 'eval_runtime': 10.8422, 'eval_samples_per_second': 428.143, 'eval_steps_per_second': 13.466, 'epoch': 0.37}

1. Why does the loss start at 1.1328 ?
2. Why the learning rate is changing at each epoch and it is not fixed ? I fixed it at 5e-5 at the beguinning ?
3. How to intrepret the results ? For me the model seems to learn better since the loss decreases at each epoch ? But how to explain it with the change in the learning the rate ?

training_args = TrainingArguments(
    output_dir='/gpfswork/rech/kpf/umg16uw/results_hf',          
    logging_dir='/gpfswork/rech/kpf/umg16uw/logs',
    do_train=True,
    do_eval=True,
    evaluation_strategy="steps",
    logging_first_step=True,
    logging_steps=10,
    num_train_epochs=2.0,              
    per_device_train_batch_size=8,
    per_device_eval_batch_size=8,   
    learning_rate=5e-5,
    weight_decay=0.01
)

CodePudding user response：

1. The loss starts at 1.3, which is arbitrary, because the first epoch is a randomisation of the weights, and so you would be extremely lucky to be accurate early on.
2. The learning rate you supply to TrainingArguments is just the initial learning rate, the training method adapts this automatically. The learning rate changing indicates that the initial rate may be too high or too low, and the method is adapting to prevent over-fitting or under-fitting the data based on the returned loss and accuracy of each epoch.
3. The accuracy and loss are good measures to track across the epochs, less loss is better, more accuracy is better, if you also had an accuracy measure you could compare accuracy to eval_accuracy and if the eval_accuracy becomes higher than the accuracy then you are starting to overfit the data.