BestRQ implementation

[Judyxujj]

add the two components, i.e. mask and quantiser, for BestRQ.

[michelwi]

Not sure if there are problems in the code or if I misunderstood what it is doing 🤷

[albertz]

Does it make sense to include this in i6_models? What exactly is our policy w.r.t. what we want to have here? I thought i6_models is intended for models (and maybe other functions) which are supposed to be used by a larger amount of people among us. But it is not intended for research code, where you want to try something out. Here, as I understand, this is mostly adapted from Fairseq, so maybe it's already well tested, but I don't know. Did you test this?

[Judyxujj]

Does it make sense to include this in i6_models? What exactly is our policy w.r.t. what we want to have here? I thought i6_models is intended for models (and maybe other functions) which are supposed to be used by a larger amount of people among us. But it is not intended for research code, where you want to try something out. Here, as I understand, this is mostly adapted from Fairseq, so maybe it's already well tested, but I don't know. Did you test this?

I think BestRQ is a very general pre-training architecture that we might use later also in the group, so I add it here. I ran them with a small dataset and seems to work.

[curufinwe]

Does it make sense to include this in i6_models? What exactly is our policy w.r.t. what we want to have here? I thought i6_models is intended for models (and maybe other functions) which are supposed to be used by a larger amount of people among us. But it is not intended for research code, where you want to try something out. Here, as I understand, this is mostly adapted from Fairseq, so maybe it's already well tested, but I don't know. Did you test this?

I have no problem adding also some code that is a bit more "research-y" to the repo. But of course once it's in we would require that new variants get a separate class name to not break the old behavior (bugs are the exception of course). If we see that some model / part of a model is changing rapidly we might want to delay things until it settles down a bit, but in this instance I would expect the code from fairseq to broadly work.