Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Erratic eval @CCC #4735

Open
mstembera opened this issue Aug 10, 2023 · 5 comments
Open

Erratic eval @CCC #4735

mstembera opened this issue Aug 10, 2023 · 5 comments
Labels
discussion needs-analysis Needs further analysis

Comments

@mstembera
Copy link
Contributor

Describe the issue

In game https://www.chess.com/computer-chess-championship#event=ccc21-blitz-semifinals&game=337 starting on move 88 the eval suddenly jumps and stays erratic till move 106. Given only 7 pieces and the pre and post evals this seems erroneous.

Expected behavior

Eval isn't erratic.

Steps to reproduce

See https://www.chess.com/computer-chess-championship#event=ccc21-blitz-semifinals&game=337 starting on move 88.

Anything else?

No response

Operating system

All

Stockfish version

dev-20230729-65ece7d9

@Disservin Disservin added needs-analysis Needs further analysis discussion labels Aug 10, 2023
@Clausable
Copy link
Contributor

Clausable commented Oct 14, 2023

@reggie1337
Copy link

I don't think the eval is behaving unexpectedly since white stays generally positive with two minor pieces in play vs blacks one, the eval bar seems relatively consistent based off moves made.

@cmwetherell
Copy link

The evaluation function/network is not perfect. Maybe positions like this could be compiled with the "known" answer, i.e., it is a draw and the data added to the training data for the nets. I doubt this is a "bug" that can/should be targeted aside from improving training data.

If you view the position in Lichess for SF 16, it gives a more reasonable eval: https://lichess.org/analysis/3bk3/8/6p1/3PK3/1B6/7P/8/8_b_-_-_0_1

@cj5716
Copy link
Contributor

cj5716 commented Dec 18, 2023

positions like this could be compiled with the "known" answer, i.e., it is a draw and the data added to the training data for the nets

I highly doubt that will work given that the nets are trained on over 100GB of data (in compressed form)

@peregrineshahin
Copy link
Contributor

it's been a long long time since this happened.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
discussion needs-analysis Needs further analysis
Projects
None yet
Development

No branches or pull requests

8 participants
@mstembera @cmwetherell @peregrineshahin @Disservin @reggie1337 @Clausable @cj5716 and others