(EAI-375): Ingest snooty docs facets and meta

[mongodben]

Jira: https://jira.mongodb.org/browse/EAI-375

Changes

• Ingest Snooty docs meta and facets fields as Page.metadata
• Add chunking concurrency

Notes

• This PR only ingests the facets that the docs put inside of pages, not in the facets.toml files. There's a DOP ticket to capture the facets.toml https://jira.mongodb.org/browse/DOP-5182

Experiment Results

Experiment compares using the ingestion pipeline with the new snooty metadata to the previous baseline. The results can be found here: mongodb-chatbot-retrieval/experiments/mongodb-chatbot-retrieval-snooty-metadata

The results actually show a very slight decrease in search quality as a result of these changes:

• BinaryNDCG@5 goes from 33.20% -> 33.05% (-0.15%)
• F1@5 goes from 23.33% -> 22.88% (-0.45%)

Next Steps

Based on the results, I think we should take one of the following paths:

1. YOLO merge changes. The difference is pretty marginal in the evals. Probably won't have a meaningful impact on system. Could potentially have positive impact later on with things like changing embedding model, etc.
2. Make additional 'Retrieval system upgrade' changes to chatbot (new embedding model, hybrid search, etc.). See if the additional metadata has positive impact then.
3. Experiment with only embedding certain fields of the metadata, changing the name etc.
4. Don't include the metadata in chunks, just ingest into pages collection.

• Note that this will require changing our chunking system to accommodate this.

My inclination is to proceed with path 2. Leave this to the side, and if we can't the metadata inclusion to improve system performance, then do some light experimentation on path 3, and only do 4 if we can't get it to work.

But then again, maybe I'm overcomplicating and we should just YOLO path 1, shouldn't having meaningful negative impact...

[nlarew]

This works for now. Longer term if we plan to maintain this Snooty AST glue code then it might be nice to move to Zod parsing for these instead of type casts.