[CGPO] CGPO Trainer (single task single objective)

[gaetanlop]

What does this PR do?

This PR introduces the CGPOTrainer to the trl library, as introduced in the paper CGPO paper.
The current implementation follows the approach of section 4.1 of the paper (CGPO in Single Task with Single Objective) and adds the three possible policy optimization methods described: codpo, crraft and crpg.

This PR was initially discussed in #2156.
This PR depends on #2159 which introduces the MOJs.

I plan to address "Multi-Taks with Multi-Objectives" part (section 4.2) in a separate PR once this one has been successfully merged and tested.

Note: This PR relies on the DataCollatorForChatML which has a small bug. Refer to #2169.

Before submitting

• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a GitHub issue? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines.
• Did you write any new necessary tests?

To do

• Log metrics
• Mini batch in crpg policy optimization
• Approval of 🤝 Mixture of judges #2159
• Add documentation
• Add a more comprehensive test suites
• Handle case in CODPO where no completions satisfy all constraints.

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[gaetanlop]

Some judges in CGPOTrainer need gold answers or metadata to make decisions. This gold answer can be used to compare with the policy's output or store metadata for rule-based judges (examples of this metadata are in Table 4).

I’m using the DataCollatorForChatML in the CGPOTrainer right now. Should we create a new dataset format with a prompt, completion, and gold answer in a separate PR, or should we modify DataCollatorForChatML to return the non-tokenized gold answer along with the current parameters?

What do you prefer ?@qgallouedec @kashif