Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/support weights passed from dataframe #152

Merged
merged 3 commits into from
Dec 18, 2024

Conversation

L-M-Sherlock
Copy link
Member

No description provided.

@L-M-Sherlock L-M-Sherlock added the enhancement New feature or request label Dec 17, 2024
@Expertium
Copy link
Contributor

I'll copy what I said in Discord

I think we shouldn't use weights, since that would make it easy to game the metrics just by choosing some oddly specific weighting function.
And the value of logloss would become essentially arbitrary since it depends on the choice of the weighting function
That being said, I do see a good usecase for this - finally making a proper "Ignore reviews before"
By setting weights for all reviews before a selected date to 0
In that case, the weights would have to be passed to RMSE as well, not just to logloss

@L-M-Sherlock
Copy link
Member Author

I think we shouldn't use weights, since that would make it easy to game the metrics just by choosing some oddly specific weighting function.

The PR doesn't game the metrics. The weights are used for the optimization. In the evaluation, all reviews' weights are set to one.

@Expertium
Copy link
Contributor

Good to know. I'll be eagerly waiting for you to update the benchmark code in other.py

@L-M-Sherlock L-M-Sherlock marked this pull request as ready for review December 18, 2024 02:13
@L-M-Sherlock L-M-Sherlock added this pull request to the merge queue Dec 18, 2024
Merged via the queue into main with commit 572ead0 Dec 18, 2024
4 checks passed
@L-M-Sherlock L-M-Sherlock deleted the Feat/support-weights-passed-from-dataframe branch December 18, 2024 02:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants