Fixing tokenizer not stripping "Ideographic Full Stop" (chinese full …#783

Merged

adbar merged 8 commits intoadbar:masterfrom reinoldus:fix/fingerprint-tokenizer-bug

Feb 8, 2025

Commits on Feb 5, 2025

Commits on Feb 6, 2025

Commits on Feb 7, 2025

Commits on Feb 8, 2025

Removing sys dependcy in deduplication and making translate table static
reinoldus
committed