Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FT] Add Gemba MQM Translation Metric #397

Closed
JoelNiklaus opened this issue Nov 19, 2024 · 3 comments
Closed

[FT] Add Gemba MQM Translation Metric #397

JoelNiklaus opened this issue Nov 19, 2024 · 3 comments
Labels
feature request New feature/request

Comments

@JoelNiklaus
Copy link
Contributor

Issue encountered

The metrics only include rather outdated translation metrics.

Solution/Feature

Gemba MQM seems to be a current metric. Adding it would make translation evaluation better.

@JoelNiklaus JoelNiklaus added the feature request New feature/request label Nov 19, 2024
@NathanHB
Copy link
Member

Looks nice ! This is using external APIs and does not seem to have a PyPI package so we would need to implement it in Lighteval. This not not high priority but if you need it feel free to open a PR and we can help you set it up :)

@JoelNiklaus
Copy link
Contributor Author

Great, thanks!
Yes, I see two avenues:

  1. Fork their repo, publish a pip package and integrate it like that.
  2. Just copy their prompts and post processing functions.

IMO option 1 is cleaner and also allows other people to use the metric more easily.

@chuandudx Would you be interested in taking this?

@JoelNiklaus
Copy link
Contributor Author

This is done: I forked the repo (https://github.com/joelniklaus/gemba) and deployed it to pypi (https://pypi.org/project/gemba/). I also included it in PR #389. We can also think about pulling some metrics from this PR to the metrics_sample so everyone can use them afterwards.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature/request
Projects
None yet
Development

No branches or pull requests

2 participants