Inconsistent loss/metrics with jax backend

[dkgaraujo]

Training an LSTM-based model with mean_squared_error loss, I got the following training results, for which the math doesn't add up: the values of the loss (MSE) and metric (RMSE) are inconsistent.

Would anyone have an insight as to what could be happening here? Thank you in advance.

[dkgaraujo]

Apologies, the data is proprietary so I cannot post a reproducible example. However, the network is relatively simple and comprises only off-the-shelf layers.

[dkgaraujo]

Update: I noted a similar thing also on tensorflow backend. Shouldn't the metric and the loss be calculated on the same data, the same way?

[dkgaraujo]

For context: found this post here suggesting that kernel regularization is also used when calculating the "official" loss metric but not the validation metric. However, I couldn't find a confirmation in the documentation.

The model I am training is indeed using a L2 regularisation.

[jeffcarp]

@dkgaraujo thanks for reporting, can you see what the results are with L2 normalization off?

[jeffcarp]

One guess: it could be because the metrics are computed at a different frequency than losses.

[fchollet]

What's your code for RMSE? The most likely explanation is non-commutativity of the sqrt operation.

[github-actions]

This issue is stale because it has been open for 14 days with no activity. It will be closed if no further activity occurs. Thank you.

[dkgaraujo]

Actually I was able to track the issue. Apparently the loss (MSE) and the metric (RMSE) diverge when there is a regularization penalty added to a layer (https://keras.io/api/layers/regularizers/), because this gets added up to the loss but is not reflected in the metric. So from my perspective as an user, not a bug, but could be better documented perhaps.

@fchollet, I did not use custom RMSE code.

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No