Making use of sqlite-vec's metadata filtering

[forrestbao]

We previously manually do metadata filtering. Now sqlite-vec just added this ability. So we can simplify our code with that asg017/sqlite-vec#26

[forrestbao]

Two changes need to be made here:

First, combine the tables chunks with embeddings by adding a column embedding in chunks to store the vector embedding of texts in the column text.

Then perform metadata (here the metadata is sample_id and text_type) filtering in SQL like this:

opposite_text_type = {'source': 'summary', 'summary': 'source'}

sql_cmd = 
f"
SELECT
        chunk_id,
        distance
    FROM embeddings
    WHERE embedding MATCH {embedding_of_selected_text_span: str}
    AND sample_id = {sample_id_of_selected_text_span: int}
    AND text_type = opposite_text_type[text_type_of_selected_text_span: str]
    ORDER BY distance
"

For more details, see this blog post of sqlite-vec. To understand embeddings and vector search in general, try this Youtube video.

Below is our current manual row filtering

https://github.com/TexteaInc/mercury/blob/sqlite/README.md?plain=1#L232-L292