Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clean up vocab creation #300

Open
dakinggg opened this issue Jan 27, 2021 · 0 comments
Open

Clean up vocab creation #300

dakinggg opened this issue Jan 27, 2021 · 0 comments

Comments

@dakinggg
Copy link
Collaborator

This script is getting quite a few steps removed from the original corpus now. It might be better to convert this to a script which reads a large corpus and creates the vocabularies directly, rather than us having created this intermediate file with the word/doc counts in it, and then having this one generate a vocabulary file which is not substantially different apart from how it is filtered.

Originally posted by @DeNeutoy in #295 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant