Interested in providing open-source baseline model weights for Meltingpot

[ezhang7423]

Hello Meltingpot Team,

I am reaching out to express my interest in contributing to the Meltingpot project by providing open-source baseline model weights. I believe that having accessible baseline models can significantly benefit the community by enabling more efficient research and development efforts. Below, I've outlined my proposal and how I plan to ensure the quality and utility of these model weights.

Background

I have been working on Social Environment Design that aligns closely with the social goals and challenges addressed by Meltingpot. Throughout my project, I have developed and trained models on a subset of scenarios that I believe could serve as valuable baselines for other researchers and developers working in this space. If it would be useful, I am planning to scale this to train for all scenarios and substrates outlined in this repository.

Proposal

I propose to contribute the following corresponding model weights and algorithms to the Meltingpot repository:

1. MADDPG
2. MASAC
3. MAPPO
4. IDDPG

Quality Assurance

To ensure the quality and reproducibility of the contributed model weights, I will also provide:

• Detailed training procedures and hyperparameters.
• Source code used for training the models, with documentation.
• Evaluation scripts and instructions to reproduce the reported performance metrics following https://github.com/instadeepai/marl-eval.

Conclusion

I am excited about the opportunity to contribute to the Meltingpot project and believe that these baseline models can be a valuable resource for the community. I am open to feedback and suggestions from the Meltingpot team on how to best integrate these models into the project.

Please let me know if there are specific requirements or guidelines I should follow to contribute these model weights, and if this would be a useful contribution. I look forward to your response and hope we can make these resources available to the wider community soon.

Thank you for considering my proposal.

Best regards,
Eddie

[jzleibo]

Sounds good to me! From my perspective, it would be great to have standard models we could point people too. Rakshit Trivedi is the person to coordinate with on this, since he handled the baselines for the challenge. Also John and Edgar likely have thoughts on how to go about it too.

…

On Sun, Mar 31, 2024, 9:10 PM Eddie Zhang ***@***.***> wrote: Hello Meltingpot Team, I am reaching out to express my interest in contributing to the Meltingpot project by providing open-source baseline model weights. I believe that having accessible baseline models can significantly benefit the community by enabling more efficient research and development efforts. Below, I've outlined my proposal and how I plan to ensure the quality and utility of these model weights. Background I have been working on Social Environment Design <https://arxiv.org/abs/2402.14090> that aligns closely with the social goals and challenges addressed by Meltingpot. Throughout my project, I have developed and trained models on a subset of scenarios that I believe could serve as valuable baselines for other researchers and developers working in this space. If it would be useful, I am planning to scale this to train for all scenarios and substrates outlined in this repository. Proposal I propose to contribute the following corresponding model weights and algorithms to the Meltingpot repository: 1. *MADDPG* 2. *MASAC* 3. *MAPPO* 4. *IDDPG* Quality Assurance To ensure the quality and reproducibility of the contributed model weights, I will also provide: - Detailed training procedures and hyperparameters. - Source code used for training the models, with documentation. - Evaluation scripts and instructions to reproduce the reported performance metrics following https://github.com/instadeepai/marl-eval. Conclusion I am excited about the opportunity to contribute to the Meltingpot project and believe that these baseline models can be a valuable resource for the community. I am open to feedback and suggestions from the Meltingpot team on how to best integrate these models into the project. Please let me know if there are specific requirements or guidelines I should follow to contribute these model weights, and if this would be a useful contribution. I look forward to your response and hope we can make these resources available to the wider community soon. Thank you for considering my proposal. Best regards, Eddie — Reply to this email directly, view it on GitHub <#223>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHFJWSAMCWXGP3NHW54UALY274JTAVCNFSM6AAAAABFQLUY4GVHI2DSMVQWIX3LMV43ASLTON2WKOZSGIYTMOJTGY4DENI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[jagapiou]

I agree that shared standard models would be helpful for people working on Melting Pot.

However, I don't think we should add an agent training pipeline to the
dm-meltingpot library as it doesn't currently contain any agent-side
stuff.

We could maybe add these baselines as examples on GitHub but that would be
quite heavyweight, hard to test internally, and a maintenance burden.
(Our existing/examples are already broken
#232).

I think it would be better if this were a separate project (e.g.
meltingpot-baselines) hosted and maintained by Eddie that uses
meltingpot (we can link to it from our README.md or elsewhere).

[rstrivedi]

Hi John -- I agree that a separate repo would be useful and to this end, we're already maintaining baselines at the repository we used for the contest and I'll keep refreshing is as new models are added. Here is the link to the repo. Happy to rename it as you have suggested.

As Joel suggested, it would be great for Eddie to coordinate with me on this and I can help him integrate his models to that repo.

Hi Eddie -- Nice to e-meet you. Please reach out to me to discuss the plan.

[ezhang7423]

Thank you all for the response to this matter. I've reached out individually to you Rakshit - will update the team later on after we meet.

[duenez]

Closing. Thanks!

[jzleibo]

Actually, I believe we decided to update the main repo to match the contest baseline. It's very confusing to have two repos.

…

On Tue, Apr 2, 2024, 1:14 PM Rakshit Trivedi ***@***.***> wrote: Hi John -- I agree that a separate repo would be useful and to this end, we're already maintaining baselines at the repository we used for the contest and I'll keep refreshing is as new models are added. Here is the link <https://github.com/rstrivedi/Melting-Pot-Contest-2023> to the repo. Happy to rename it as you have suggested. As Joel suggested, it would be great for Eddie to coordinate with me on this and I can help him integrate his models to that repo. Hi Eddie -- Nice to e-meet you. Please reach out to me to discuss the plan. — Reply to this email directly, view it on GitHub <#223 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHFJWSDYVPZRSAO6GJ63FDY3LRPRAVCNFSM6AAAAABFQLUY4GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZSGYYDSNZXGI> . You are receiving this because you commented.Message ID: ***@***.***>

[rstrivedi]

Yes, it is still the case regarding main repo to match contest repo -- messaged via email for more details. Thanks!