We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
If we try to perform inference in float32, we get the error:
AssertionError: Key and Value Dtypes should match
This error comes from this line.
The origin of the error is that the cache dtype is set to jnp.int8 if quantize_kvcache else jnp.bfloat16 but never to jnp.float32.
dtype
jnp.int8 if quantize_kvcache else jnp.bfloat16
jnp.float32
The text was updated successfully, but these errors were encountered:
What are you setting that triggets this? (Activations to float32?)
Sorry, something went wrong.
Yes it's the dtype:
maxtext/MaxText/configs/base.yml
Line 61 in f52e6f7
bvandermoon
No branches or pull requests
If we try to perform inference in float32, we get the error:
This error comes from this line.
The origin of the error is that the cache
dtype
is set tojnp.int8 if quantize_kvcache else jnp.bfloat16
but never tojnp.float32
.The text was updated successfully, but these errors were encountered: