Search results

added cached tokenizer

Core: TokenizationInternals of the library; Tokenization.Internals of the library; Tokenization.

Status: Draft (not ready).

huggingface/transformersnumber 35218

#35218 In huggingface/transformers;

· Deep-unlearning opened on Dec 11, 2024

Tokenizer discard data that exceed max_length

Core: TokenizationInternals of the library; Tokenization.Internals of the library; Tokenization.

Feature requestRequest for a new featureRequest for a new feature

Status: Open.

#31627 In huggingface/transformers;

· fengyunflya opened on Jun 26, 2024

[LLaMA3] 'add_bos_token=True, add_eos_token=True' seems not taking effect

Core: TokenizationInternals of the library; Tokenization.Internals of the library; Tokenization.

Feature requestRequest for a new featureRequest for a new feature

Status: Open.

#30947 In huggingface/transformers;

· kiva12138 opened on May 22, 2024

(Willing to PR) Make <code>tokenizer.padding_side</code> an argument instead of only being a field

Core: TokenizationInternals of the library; Tokenization.Internals of the library; Tokenization.

Feature requestRequest for a new featureRequest for a new feature

Status: Open.

#30447 In huggingface/transformers;

· fzyzcjy opened on Apr 24, 2024

<code>test_encode_decode_fast_slow_all_tokens</code> is failing

Core: TokenizationInternals of the library; Tokenization.Internals of the library; Tokenization.

WIPLabel your PR/Issue with WIP for some long outstanding Issues/PRs that are work in progressLabel your PR/Issue with WIP for some long outstanding Issues/PRs that are work in progress

Status: Open.

#30045 In huggingface/transformers;

· ydshieh opened on Apr 4, 2024