New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

input prompt optimization problem #56

Open

money0534 opened this issue Mar 22, 2024 · 0 comments

money0534 commented Mar 22, 2024

Hi,
According to this blog:

The optimal bucket sizes should be determined by prompt length distribution in a target application. Here, we adopt bucket lengths: 128, 256, 384, 512. Any input prompt with up to 2,047 tokens requires up to 4 graph executions. For example, a 1,500 input prompt with generation length of 256 requires 260 graph executions - 4 to process the input, and 256 to generate the output.

How is a 1,500 input prompt divided into 4 excutions? What are the section lengths? Could we have some more details, please?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment