Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Plan much too long for repeating test fixes #178

Open
viraptor opened this issue Aug 2, 2024 · 1 comment
Open

Plan much too long for repeating test fixes #178

viraptor opened this issue Aug 2, 2024 · 1 comment

Comments

@viraptor
Copy link

viraptor commented Aug 2, 2024

I've tried to get Plandex to fix a few tests where the expected values needed updating. I was explicit about the task with:

The values in the tests need to be updated to match the new results. Here are the failures:
(8 cases of test failures including "expected ... got ..." from an rspec run)

Only one Ruby file in context, with ~200 lines. (Company internal code, so can't provide it, but it's trivial to produce a similar one)

Unfortunately that seems to have been split into way too many tiny tasks. 50 gpt4o requests later, Plandex was still working on the second failing test.

I'm hoping there's some prompt adjustment that could help with tiny repeated tasks like that.

@danenania
Copy link
Contributor

Thanks for reporting this @viraptor—we're working on evals to improve results for these kinds of tasks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants