The N Implementation Details of RLHF with PPO

[vwxyzjn]

Adding the blog post here 🤗. CC @liutianlin0121

[liutianlin0121]

Adding the blog post here 🤗. CC @liutianlin0121

Looks awesome! 🤗

[liutianlin0121]

Just a small recommendation (do feel free to disregard!): perhaps adding a Bibtex citation section at the end of the blog like in https://huggingface.co/blog/rlhf and in https://lilianweng.github.io/posts/2018-04-08-policy-gradient/?

For instance:

@article{Huang2023implementation,
  author = {Huang, Costa and Liu, Tianlin and von Werra, Leandro},
  title = {The N Implementation Details of RLHF with PPO},
  journal = {Hugging Face Blog},
  year = {2023},
  note = {https://huggingface.co/blog/the_n_implementation_details_of_rlhf_with_ppo},
}

[vwxyzjn]

@liutianlin0121 thanks for the suggestion. Done! Btw would you mind creating an HF account and giving me the username? It's gonna be highlighted like this:

[liutianlin0121]

@liutianlin0121 thanks for the suggestion. Done! Btw would you mind creating an HF account and giving me the username? It's gonna be highlighted like this:

Thanks! My HF account is tianlinliu0121: https://huggingface.co/tianlinliu0121

[pcuenca]

This is a great, deep technical post, I just took a quick initial look. As a personal opinion, I think it reads too much like a paper and therefore demands a lot of effort from the reader (for example, they have to go through several intros and lists of resources before getting to the "meat"). I'm sure there's a lot of interest in RLHF and PPO from non-specialists, so I would have loved if it was more engaging for that audience, contextualizing stuff and explaining what this means and why it's important in addition to how it works.

I understand a rewrite is a lot of work and impractical at this point, but there may be some small quick wins in the way we present things or introduce concepts.

Of course, feel free to completely disregard this opinion if you don't agree!

[pcuenca]

Maybe a tad too paper-y for me :) The blog style is more informal than academia, but happy to keep it if you think it's necessary.

[vwxyzjn]

removed

[vwxyzjn]

Thanks so much @pcuenca for the review. I have addressed most of your comments and updated the _blog.yml.

[lvwerra]

LGTM :)