Exercise 5.2 - Surprised by expected output

[spayot]

Hi Sebastian,
Thanks for putting together this excellent content! it's been a bliss to go through the first 5 units. :)

Question with regards to exercise 5.2:
Although i am able to reproduce the "expected" output captured in the screenshot [example-output-1.png](https://github.com/Lightning-AI/dl-fundamentals/blob/main/unit05-lightning/exercises/2_custom-callback/example-output-1.png) (77.20%, 91.43%, ..., 97.36%), i am a bit confused by those values. unless there is massive overfitting, i would have expected instead to see differences in the single digits or low double digits.
it seems like the differences shown in that screenshot are computed at the end of the training epoch but before the validation epoch has actually run (hence the high difference). if we use the on_validation_epoch_end handle instead, then we get train-val accuracy difference values that look much more in line with what i would have expected:

is this alternative approach reasonable? or did i miss the point in terms of the value we are actually trying to calculate with that callback? :)

Follow up question:
with that alternative approach, the (admittedly small) challenge becomes that our 3 calls to trainer.validate at the bottom of trainer.py also "trigger" the custom callback. would love to know if there is an appropriate way to get a callback to be called during the trainer.fit training loop, but not outside of said training loop.
(i tried to overwrite by adding a line with trainer.callbacks = [] after fitting but it did not seem to produce the desired effect...)

Thanks in advance!
Looking forward to continuing the course!

Sylvain

[rasbt]

Glad to hear you are liking the Units so far! And wow, you are absolutely right. Not sure what I did there. Just fixed it!

Your results look good btw!

with that alternative approach, the (admittedly small) challenge becomes that our 3 calls to trainer.validate at the bottom of trainer.py also "trigger" the custom callback. would love to know if there is an appropriate way to get a callback to be called during the trainer.fit training loop, but not outside of said training loop.

That's a good point. Off the top of my head, I don't think there is an easy way. But one way to avoid the trigger here is to perhaps swap the lines

    train_acc = trainer.validate(dataloaders=dm.train_dataloader())[0]["val_acc"]
    val_acc = trainer.validate(datamodule=dm)[0]["val_acc"]
    test_acc = trainer.test(datamodule=dm)[0]["test_acc"]

with

    train_acc = trainer.test(dataloaders=dm.train_dataloader())[0]["test_acc"]
    val_acc = trainer.test(dataloaders=dm.val_dataloader())[0]["test_acc"]
    test_acc = trainer.test(datamodule=dm)[0]["test_acc"]

(I just did that in the template to avoid confusion if others try this exercise)