New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: upgrade from xx_sent_ud_sm to SaT #74

Merged

lsorber merged 1 commit into main from ls-sat

Jan 5, 2025

+109 −720

Member

lsorber commented Jan 2, 2025

Changes:

Replace spaCy's xx_sent_ud_sm with wtpsplit's Segment any Text (SaT) model for sentence splitting.
Expose a way to provide known sentence boundary probabilities to the sentence splitter, with the default being a method that defines Markdown headings in the document as contiguous sentences.
Add wtpsplit-lite as a dependency.
Remove spaCy as a dependency and simplify installation instructions.

lsorber self-assigned this


          feat: upgrade from xx_sent_ud_sm to SaT

bbf4f65

lsorber force-pushed the ls-sat branch from cc5b9b7 to bbf4f65 Compare

January 2, 2025 16:59

lsorber merged commit 41525c8 into main

2 checks passed

lsorber deleted the ls-sat branch

January 5, 2025 15:18

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet