Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the ablity to continue generation. #7

Open
cuyler72 opened this issue May 30, 2024 · 1 comment
Open

Add the ablity to continue generation. #7

cuyler72 opened this issue May 30, 2024 · 1 comment
Labels
enhancement New feature or request

Comments

@cuyler72
Copy link

Currently if you are not satisfied with the provided options the only way to continue generation is to increase the beam limit and regenerate, It would be nice to be able to continue generation for another cycle.

@the-crypt-keeper
Copy link
Owner

@cuyler72 This is actually a really interesting thought. The original implementation of the search was depth-first (once a beam is identified we rode its splits all the way until the end, then backend up) but I've since flipped it to breadth-first (for each beam we step one token at a time and keep lots of beams running at once). I originally did this to support parallelism but it turned out to have some other nice properties like being able to support max-suggestions coherently. Now that you mention it, BFS also enables a simple "Step Next Tokens" button that can either supplement or replace the depth and breadth controls. I will look into this one!

@the-crypt-keeper the-crypt-keeper added the enhancement New feature or request label May 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants