[WIP] Add mammut

[gpucce]

Initial attempts at adding MaMMuT https://arxiv.org/abs/2303.16839

@mehdidc @JeniaJitsev @rom1504 I don´t know marianna github name can you tag her too?

This is the PR, will add all of you as collaborators.

[JeniaJitsev]

Initial attempts at adding MaMMuT https://arxiv.org/abs/2303.16839

@mehdidc @JeniaJitsev @rom1504 I don´t know marianna github name can you tag her too?

This is the PR, will add all of you as collaborators.

@marianna13 is the tag. We have a version in https://github.com/SLAMPAI/open_clip_all_at_once , can add you there to work on fusing everything we got on Mammut.

[gpucce]

@JeniaJitsev sure!

[mehdidc]

@gpucce actually a part from resolving conflicts, the only thing I see that is needed is to support logit_bias for siglip training (e.g., like 6f05e1d#diff-2c6f604038a68c220ab530723839b6716c1ca011291158de70c3f8e17c0a74deR86) and adding missing configs (https://github.com/SLAMPAI/open_clip_all_at_once/commit/0ec7475afb20280eaae6fdfdf389e92edded0cfa).
I would say the rest of the modifications are out of the scope of this PR.

[gpucce]

@mehdidc I am not in https://github.com/SLAMPAI/open_clip_all_at_once/

Can you add me there?

[gpucce]

I will have a look at fixing conflicts and making this mergeable in the next couple of days

[mehdidc]

@gpucce Cool ! sure, added you now.