add bananalyzer eval lib #75
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
navidkpr
force-pushed
the
npour/more-evals
branch
from
19a90ee to
a6930a0
Compare
navidkpr
changed the title
| For running the bananalyzer eval, you'll need to run the init script first to download the static assets: | ||
|
|
||
| `./evals/bananalyzer-ts/init.sh` | ||
|
|
There was a problem hiding this comment.
@navidkpr can you pls add instructions how to run the bananalyzer evals?
pkiv
approved these changes
Sep 30, 2024
filip-michalsky
approved these changes
Sep 30, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
why
Mainly because we needed a suite of reliable evals to improve Stagehand on.
We can now add any of the 319 test cases from bananalyzer with a single line. I will be adding the useful test cases in a separate PR.
what changed
test plan
This is tested by default every time we run the eval system.
For the future
Our current eval system uses active urls. This can cause a problem because the urls we have might change over time. For example, a single "." change will cause the eval to fail on an extract job. To get around that we can take snapshots of the websites we use in each eval using playwright and export that into a mhtml file.
This PR adds the functionality to test Stagehand on those snapshots. Making the site change issue null.
Also, other eval systems for web agents also use MHTML format (for example Mind2Web). So next time it will be much easier to integrate with an external eval source.