Code

Store tokens in u32 instead of usize#251

Code

Merged

hauntsaninja merged 1 commit intomainopenai/tiktoken:mainfrom paplorinc-1openai/tiktoken:paplorinc-1Copy head branch name to clipboardFeb 9, 2024

+38-35

Commits

Commits on Feb 9, 2024

• Store tokens in u32 instead of usize

  Show description for 1b81474
  

  Lőrinc

  authored and

  hauntsaninja

  committed
  1b81474

  Copy full SHA for 1b81474