Issues replicating creation of the model shipped with the code

[marcverhagen]

Because

Given an old-style configuration dump after training (20231102-165708.kfold_results.txt) which was the one created when the current shipped model in modeling/models/20231214-191952.convnext_tiny.kfold_000.pt was created, I created a new config file (trainer-replicated.txt, attached as a txt file since Yaml was not supported).

However, when I run the current trainer I get different results:

• Mean f-1 is 0.75 instead of 0.81
• Only three folds have enough data: folds 001, 003 and 007. Something like this may have been the case previously as well, but the model in modeling/models is for fold 000.
• The size of the pt files is 1.1MB instead of 639KB.

One thing I noticed is that the timestamp on 20231102-165708.kfold_results.txt is different from the one on 20231214-191952.convnext_tiny.kfold_000.pt. The timestamp on the config file 20231214-193543.convnext_tiny.kfold_config.yml is yet another one.

Another thing that may make exact replication hard is that when I run the trainer two times in a row (with the same code base, config and features) I get different results. The differences are small though with mean f-1's of 0.7526 and 0.7421

Done when

No response

Additional context

No response

[keighrim]

(20231102-165708.kfold_results.txt) which was the one created when the current shipped model in modeling/models/20231214-191952.convnext_tiny.kfold_000.pt as created,

It doesn't make sense that 20231102 txt file is created during training of 20231214 pt file. Isn't the txt file the one that I passed earlier today during our zoom call? That txt file (and 20231102 pt files I passed along with) were created on 11/2 when we were running the final rounds of experiments with sinusoidal positional encodings and pre-/postbin combinations.

20231214-191952.convnext_tiny.kfold_000.pt. The timestamp on the config file 20231214-193543.convnext_tiny.kfold_config.yml is yet another one.

This might be an indication that the config and model actually do not match...? I don't have any more details on this part of problem as I didn't participate 12/14 model training.

Also, 11/02 training settings include postbin, so I don't think the model (pt files) will work with current classifier implementation (#46 ).

[marcverhagen]

Also, 11/02 training settings include postbin, so I don't think the model (pt files) will work with current classifier implementation (#46 ).

But the classifier model itself is not impacted by the post binning, so the frame classification part should work. The post binning isn't really a classifier config setting but an evaluation or classifier usage setting. Does that sound right?

[keighrim]

Model files in v2.1 are matched to the configurations included in the same version (was different in v2.0, IIRC). Closing this as resolved.

[keighrim]

I found a serous bug that effectively disables positional encoding that was introduced by me in 3683c53 (12/8/2023, merged in v2.0), and given the timestamp on the commit, the performance regression reported in this issue is probably because of it. Reopening the issue to send a PR with a fix.

[keighrim]

#92 fixes the bug mentioned above.
I couldn't do cohesive ablation evaluation due to recent changes in the training code that nuked postbin feature from the training code (#58), but looking at the average F-1 scores of the raw labels we are interested in (mixed F-1 scores of different positional encoding scheme and positional parameters), it looks like the fix indeed significantly boosts the performance by re-enabling positional encoding.

	fix-convnext_lg	fix-convnext_tiny	nonfix-convnext_lg	nonfix-convnext_tiny
S	0.735595	0.749463	0.640382	0.663854
I	0.785745	0.793643	0.795447	0.792551
N	0.0186817	0.00594677	0.0134049	0.00434189
Y	0.169305	0.197555	0.180216	0.228871
C	0.566859	0.555965	0.488594	0.481486

[keighrim]

Full results from different positional encodings experiments.

	S	I	N	Y	C
convnext_lg-fractional	0.588877	0.763584	0.00803828	0.0445558	0.483063
convnext_lg-none	0.617154	0.731537	0.00779727	0.125528	0.468762
convnext_lg-sinusoidal-add-128d-u1000	0.877502	0.682999	0	0.0560315	0.597987
convnext_lg-sinusoidal-add-128d-u60000	0.869328	0.735252	0	0.0791942	0.621342
convnext_lg-sinusoidal-add-256d-u1000	0.860286	0.688204	0	0.0294172	0.622367
convnext_lg-sinusoidal-add-256d-u60000	0.825281	0.703316	0.00367197	0.0393246	0.631081
convnext_lg-sinusoidal-add-512d-u1000	0.840954	0.677562	0.00909021	0.0446321	0.627861
convnext_lg-sinusoidal-add-512d-u60000	0.891818	0.738426	0.0131579	0.0676896	0.611835
convnext_lg-sinusoidal-concat-128d-u1000	0.82534	0.762044	0.0318489	0.0888473	0.620642
convnext_lg-sinusoidal-concat-128d-u60000	0.798363	0.777416	0.00877193	0.0832917	0.575706
convnext_lg-sinusoidal-concat-256d-u1000	0.851984	0.763736	0.0467836	0.0915322	0.600583
convnext_lg-sinusoidal-concat-256d-u60000	0.864951	0.775126	0.00362976	0.11147	0.593215
convnext_lg-sinusoidal-concat-512d-u1000	0.85326	0.741461	0.0143541	0.0426628	0.598556
convnext_lg-sinusoidal-concat-512d-u60000	0.897169	0.754004	0	0.0669994	0.629205
convnext_tiny-fractional	0.707199	0.742368	0	0.0977915	0.463421
convnext_tiny-none	0.613333	0.773009	0	0.0906449	0.431349
convnext_tiny-sinusoidal-add-128d-u1000	0.847737	0.703982	0	0.123759	0.547839
convnext_tiny-sinusoidal-add-128d-u60000	0.851759	0.743418	0.00657895	0.0682472	0.631584
convnext_tiny-sinusoidal-add-256d-u1000	0.820522	0.682649	0	0.0852744	0.580606
convnext_tiny-sinusoidal-add-256d-u60000	0.836312	0.737819	0.00956938	0.0556193	0.646125
convnext_tiny-sinusoidal-add-512d-u1000	0.779704	0.7114	0	0.0863149	0.597859
convnext_tiny-sinusoidal-add-512d-u60000	0.840244	0.755377	0.00956938	0.133078	0.619044
convnext_tiny-sinusoidal-concat-128d-u1000	0.846883	0.762319	0	0.103446	0.662111
convnext_tiny-sinusoidal-concat-128d-u60000	0.769048	0.761481	0	0.0686968	0.579399
convnext_tiny-sinusoidal-concat-256d-u1000	0.841994	0.749686	0	0.120696	0.552216
convnext_tiny-sinusoidal-concat-256d-u60000	0.761151	0.752826	0.00619195	0.075509	0.548297
convnext_tiny-sinusoidal-concat-512d-u1000	0.868484	0.73896	0.00701754	0.0906838	0.600208
convnext_tiny-sinusoidal-concat-512d-u60000	0.816375	0.75487	0	0.154386	0.577397