[Improvement] Fuzzy matching #137
Conversation
![orca-security-eu[bot]](https://avatars.githubusercontent.com/in/276244?s=60&v=4){kind=small}
There was a problem hiding this comment.
Orca Security Scan Summary
| Status | Check | Issues by priority | |
|---|---|---|---|
 Passed |
Infrastructure as Code |  0  0  0  0 |
View in Orca |
| Passed |
SAST | 0 0 0 0 |
View in Orca |
| Passed |
Secrets | 0 0 0 0 |
View in Orca |
| Passed |
Vulnerabilities | 0 0 0 0 |
View in Orca |
|
Great to see you again! Thanks for the contribution. |
| @@ -316,6 +316,7 @@ | |||
| # highlight-start | |||
| index_filterable=True, | |||
| index_searchable=True, | |||
| tokenization="word", | |||
There was a problem hiding this comment.
I think this should be:
from weaviate.classes.config import Tokenization
Property(
# ...
tokenization=Tokenization.WORD
)(unless it accepts raw strings also? - I guess maybe it does since it's just an enum)
The function signature is this: tokenization: Optional[Tokenization] = Field(default=None) - so even if the string works I wonder if the enum is preferable.
| @@ -324,6 +325,7 @@ | |||
| # highlight-start | |||
| index_filterable=True, | |||
| index_searchable=True, | |||
| tokenization="field", | |||
There was a problem hiding this comment.
I think this should be:
from weaviate.classes.config import Tokenization
Property(
# ...
tokenization=Tokenization.FIELD
)
|
|
||
| - Creates larger inverted indexes due to more tokens | ||
| - May impact query performance for large datasets | ||
|
|
There was a problem hiding this comment.
Filtering behavior will change significantly, as text filtering will be done based on trigram-tokenized text, instead of whole words.
|
|
||
| - Use trigram tokenization selectively on fields that need fuzzy matching. | ||
| - Keep exact-match fields with `word` or `field` tokenization for precision. | ||
|
|
There was a problem hiding this comment.
Note trigram tokenization will impact filtering behavior, as token comparisons will be based on trigrams, rather than words.
What's being changed:
Add docs about using trigram tokenization for fuzzy string searches.
Type of change:
How has this been tested?
yarn start