Should tokenizer_legacy be default as false?

[tongyx361]

⚠️ Please check that this feature request hasn't been suggested before.

• I searched previous Ideas in Discussions didn't find any similar feature requests.
• I searched previous Issues didn't find any similar feature requests.

🔖 Feature description

@NanoCode012 #401 (comment)
This was from a year ago. If I recall correctly, it was that the new behavior was not correct/created some issues esp for llama. It's very likely they fixed it and this should be reverted. If you would like, please create a new Issue on this, so we can track this better.

✔️ Solution

Default tokenizer_legacy as false?

❓ Alternatives

No response

📝 Additional Context

No response

Acknowledgements

• My issue title is concise, descriptive, and in title casing.
• I have searched the existing issues to make sure this feature has not been requested yet.
• I have provided enough information for the maintainers to understand and evaluate this request.