Why a hyper-parameter like regularization parameter (a real number) cannot be trained over training data along with model parameters? What will go wrong?

CodePudding user response：

This is generally done to prevent overfitting. Model parameters are trained using the training set. Hyper-parameter tuning is done using a validation set that is (ideally) completely independent of the training data. The final performance should be evaluated on a test set. Typical splits are 80/10/10 or 60/20/20.

If you tune your hypermeters on the training set, you will very likely vastly overfit and suffer a performance hit on the test set.

Try it out! See the difference in performance on your test set when you do hyper-parameter tuning on the training set, vs on a separate validation set.