-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Issue #8, convert hunspell output to spellchecker JSON format #9
Open
teners
wants to merge
22
commits into
master
Choose a base branch
from
tkt_8_convert_hunspell_to_json
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 2 commits
Commits
Show all changes
22 commits
Select commit
Hold shift + click to select a range
7833457
Initial commit
a197b6e
Removed output file option, added corrections with spaces support,
ccada06
Fixed duplicates issue, comma-after-last-element issue,
d225ac5
Removed dummy sed solution for the last comma issue,
d1a31dd
Deleted shell script and readme, added gitignore for .idea stuff
768d0e2
Iniital commit for Java implementation
2429ed8
Merged something for some reason
f3f1597
Removed Java spellchecker implementation
1a2ee88
Initial commit for python grpc version
039864a
Deleted .pyc file and added that extension to gitignore
54a60d2
Added author's name, removed unneeded test.py
81adc92
Replaced json-string with typed interface in .proto file
77b1f9f
Added path to shared library as constructor argument in Parser
893fca6
Replaced json with protobuf interface, fixed small flaws
8c0e6fd
Fixed Suggestions interface in proto file
9e4af1a
Replaced all string literals with command line arguments, minor fixes
01d8e7a
Fixed docstring
9e79d4d
Reformatted parser.cpp
eb1d37e
Removed unused ujson import
d8aaff2
Added requirements.txt
9ae2d84
Moved new-hunspell-instance-initialization on request from check_text
a7e2db5
Removed "language" command line argument
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -13,10 +13,10 @@ class Spellchecker: | |
Class contains multiple hunspell instances (one per language) and provides | ||
text spellchecking function using all chosen dictionaries. | ||
""" | ||
def __init__(self, path: str = '', language: str ='', log: object = None): | ||
self._hunspell_instances = {} | ||
self._parser = Parser("../libparser/build/libparser.so") | ||
self._log = log | ||
def __init__(self, libparser: str, path: str = '', language: str ='', log: object = None): | ||
self.hunspell_instances = {} | ||
self.parser = Parser(libparser) | ||
self.log = log | ||
|
||
if path is not '': | ||
self.add_dictionary(path, language) | ||
|
@@ -28,21 +28,21 @@ def add_dictionary(self, path: str, language: str) -> None: | |
:param path: Path to .dic and .aff files. | ||
:param language: Language that corresponds .dic and .aff files. | ||
""" | ||
if language in self._hunspell_instances: | ||
if language in self.hunspell_instances: | ||
return | ||
|
||
dictionary = "{}/{}.dic".format(path, language) | ||
if not os.path.isfile(dictionary): | ||
if self._log: | ||
self._log.write("File not found: {}".format(dictionary)) | ||
if self.log: | ||
self.log.write("File not found: {}".format(dictionary)) | ||
return | ||
affix = "{}/{}.aff".format(path, language) | ||
if not os.path.isfile(affix): | ||
if self._log: | ||
self._log.write("File not found: {}".format(affix)) | ||
if self.log: | ||
self.log.write("File not found: {}".format(affix)) | ||
return | ||
|
||
self._hunspell_instances[language] = hunspell.HunSpell(dictionary, affix) | ||
self.hunspell_instances[language] = hunspell.HunSpell(dictionary, affix) | ||
|
||
def check_text(self, text: str, languages) -> str: | ||
""" | ||
|
@@ -53,13 +53,12 @@ def check_text(self, text: str, languages) -> str: | |
:return: JSON with suggestions for misspelled words. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. this comment is no more actual, as well as "return type" |
||
""" | ||
suggestions_map = Suggestions() | ||
hunspells = (self._hunspell_instances[lang] for lang in languages if lang in self._hunspell_instances) | ||
tokens = self._parser.tokenize(text) | ||
hunspells = (self.hunspell_instances[lang] for lang in languages if lang in self.hunspell_instances) | ||
tokens = self.parser.tokenize(text) | ||
|
||
for h in hunspells: | ||
for token in tokens: | ||
if not h.spell(token): | ||
suggestions_map.suggestions[token].values.extend(h.suggest(token)) | ||
|
||
# return ujson.dumps(suggestions) | ||
return suggestions_map |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
language is defined in the request, no?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Request from client? Nope, in the request we ask spellchecker to use those languages and that demand is satisfyed only if spellchecker already has hunspell instance initialized with a dictionaries of that languages.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
which means that we're restricted to one language, since this argument is single-valued.
Why don't you initialize hunspell instance with the languagw requested bythe client on demand?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Alrighty then, I'll work that out.