[Feature Request]: More complex metadata filtering

[KylePancamo]

Describe the problem

When filtering on metadata, there are only 6 operators for performing filtering. When dealing with string filtering, these are not enough for more desired complexities.

Example:
Let's say you want to store emails in chroma, and have the ability to do question answering on those emails, but also filter them by certain criteria. For example, say subject, to, cc, etc.

If you'd like to filter on the subject for example, you would have to provide an exact match with the $eq operator. If the subject is long and verbose, it's highly likely the user won't recall the subject to a precise match, then filtering would be pointless on the field.

Describe the proposed solution

Add in a $contains operator or some similar operator

{"subject": {"$contains": search_term}}

Example:

Document - Text containing this feature request
metadata:
- subject: "[Feature Request]: More complex metadata filtering"
- to: "[email protected]"

Possible filter terms on subject
-- "[Feature Request]:"
-- "Complex filtering"
-- "Metadata filtering"
-- "More filtering"

The search would return this or any documents containing the those keyword matches. This is just one example, but there are without a doubt more use cases where this could be beneficial. Having the ability to do keyword or similarity search on metadata values can lead to greater accuracy when querying a collection.

Alternatives considered

No response

Importance

would make my life easier

Additional Information

No response

[tazarov]

@KylePancamo thank you for this.

I think we can easily use SQLite's LIKE to achieve this:
e.g.

SELECT * FROM table_name WHERE column_name LIKE '%substring%';

[jeffchuber]

@tazarov feels like this should be a CIP - as it has implications for single-node and the distributed architecture and we will want to think somewhat carefully about it.

[l4b4r4b4b4]

That would be a good feature together with generally allowing complex metadata types.

[KylePancamo]

@tazarov @jeffchuber Haven't been keeping up here. Has this been further considered?

[tazarov]

@KylePancamo @jeffchuber I think this is addressed in #1196

@jeffchuber, regarding distributed, I'll have to look into the PR whether that would be a problem.