Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unicode normalization #3

Open
gregorycrane opened this issue Oct 13, 2024 · 0 comments
Open

unicode normalization #3

gregorycrane opened this issue Oct 13, 2024 · 0 comments

Comments

@gregorycrane
Copy link

I wonder if we should not normalize unicode as part of our Atlas data prep. I was looking on line about how to do it and found this code from some guy named Tauber ....
@jtauber @lcerrato @AlisonBabeu

from unicodedata import normalize
curword = normalize("NFC",m[1])

My thinking:

  1. Anything in our repos should probably be normalized (e.g., the Greek from the Greco-Arabic corpus).
  2. Anything we import into Atlas, we should normalize. That would imply some code in the Atlas data prep pipeline (I think)

Thoughts?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant