docs: clarify which tokenizers are affected

benbrandt · Jan 16, 2025 · 4b2691e · 4b2691e
1 parent 35bec4c
commit 4b2691e
Showing 1 changed file with 1 addition and 1 deletion.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -4,7 +4,7 @@
 
 ### Breaking Changes
 
-- Special tokens are now also encoded by both Huggingface and Tiktoken tokenizers. This is closer to the default behavior on the Python side, and should make sure if a model adds tokens at the beginning or end of a sequence, these are accounted for as well.
+- Special tokens are now also encoded by both Huggingface and Tiktoken tokenizers. This is closer to the default behavior on the Python side, and should make sure if a model adds tokens at the beginning or end of a sequence, these are accounted for as well. This is especially important for embedding models that can add a special token to the beginning of the sequence.
 
 ### What's New