Code

Tokenizer BPE fixes#7530

Code

Merged

jaime-m-p merged 29 commits intoggerganov:masterggerganov/llama.cpp:masterfrom jaime-m-p:tokenizer-bpe-fixesjaime-m-p/llama.cpp:tokenizer-bpe-fixesCopy head branch name to clipboardJun 18, 2024

+1,285-1,055

Commits

Commits on May 24, 2024

• Update random test: add_bos_token

  

  jaime-m-p

  committed
  e013b23

• Add BPE models for testing

  

  jaime-m-p

  committed
  55e387b

Commits on May 25, 2024

• bugfix: custom regex split fails with codepoint 0

  

  jaime-m-p

  committed
  fe3c531

• Refactor llm_tokenizer_bpe: move code to constructor

  

  jaime-m-p

  committed
  6f4c300

• Update random test: add_eos_token

  

  jaime-m-p

  committed
  614d0bb

• Add BPE models for testing

  

  jaime-m-p

  committed
  6168399

• Move 'add_special_bos/eos' logic to llm_tokenizer_bpe

  

  jaime-m-p

  committed
  0794b77

• Fix falcon punctuation regex

  

  jaime-m-p

  committed
  51e933a

• Better name functions to append token/bos/eos

  

  jaime-m-p

  committed
  1d2f3ad

• Move tokenizer flags to vocab structure.

  Show description for c83ea1a
  

  jaime-m-p

  committed
  c83ea1a

• Allow lstrip for 'added_tokens'

  Show description for 615f425
  

  jaime-m-p

  committed
  615f425

• Default values for special_add_bos/eos

  

  jaime-m-p

  committed
  f84b04f

• Fix default value for WPM special_add_eos

  

  jaime-m-p

  committed
  7a5578f

Commits on May 26, 2024

• Better variable names

  

  jaime-m-p

  committed
  173ab69

Commits on May 29, 2024

• Build vocab.special_tokens_cache using vocab token types

  

  jaime-m-p

  committed
  fef9915

Commits on Jun 11, 2024

• Merge commit '148995e5' into tokenizer-bpe-fixes

  

  jaime-m-p

  committed
  d67de1a

Commits on Jun 12, 2024

• Generalize 'jina-v2' per token attributes

  

  jaime-m-p

  committed
  c863752

• Fix merge: 'smaug'

  

  jaime-m-p

  committed
  75840fe

Commits on Jun 13, 2024

• update brute force random test

  

  jaime-m-p

  committed
  f58de31

• Fix 'jina-v2' per token attributes

  

  jaime-m-p

  committed
  974d40b

• Fix unicode whitespaces (deepseek-coder)

  

  jaime-m-p

  committed
  07530a8

Commits on Jun 14, 2024

• Fix unicode whitespaces (deepseek-llm)

  

  jaime-m-p

  committed
  4ff15d4

• Skip missing byte tokens (falcon)

  

  jaime-m-p

  committed
  0575023

• Update brute force random test

  

  jaime-m-p

  committed
  8cda5af

• Better unicode data generation

  

  jaime-m-p

  committed
  4af5478

Commits on Jun 15, 2024

• Merge branch 'master' into tokenizer-bpe-fixes

  

  jaime-m-p

  committed
  e28d0e4

• Fix merge: renamed and deleted files

  

  jaime-m-p

  committed
  903e47f

Commits on Jun 17, 2024

• Replace char32_t with uint32_t

  

  jaime-m-p

  committed
  b7ee827

• Merge branch 'master' into tokenizer-bpe-fixes

  jaime-m-p

  authored
  b8929d5