Optimize byte pair merge for really big tokens (40x faster for a 2500 token word)#239
Open
l0rinc wants to merge 2 commits intoopenai:mainfrom l0rinc:paplorinc/add-linearithmic-byte-pair-merge
+87-1
Commits
Commits on Feb 11, 2024
- committed
Lőrinc - committedLőrinc