Skip to content

Rust v0.10.0

Compare
Choose a tag to compare
@n1t0 n1t0 released this 08 Apr 18:23

Changed

  • [#222]: All Tokenizer's subparts must now be Send + Sync

Added

  • [#208]: Ability to retrieve the vocabulary from the Tokenizer & Model

Fixed

  • [#205]: Trim the decoded string in BPEDecoder
  • [b770f36]: Fix a bug with added tokens generated IDs