How can local.metadata.yaml file's embedder_config=>provider be truefoundry for openai's "text-embedding-ada-002" model?

[avibathula]

THANK YOU FOR creating this repo.

Really interesting project - I was attending a webinar in Oreilly's on LLMs and one of the participants suggested me to look into Cognito - it looks very promising.

---

Question 1

Is there any documentation for what all is allowed in local.metadata.yaml file?

Question 2

Looking at the sample provided - how can embedder_config.provider be truefoundry for model being openai's model openai-main/text-embedding-ada-002

What does the usage of "openai-main" mean?

collection_name: finance
data_source:
    type: localdir
    uri: ./sample-data/finance
parser_config:
    chunk_size: 1000
    chunk_overlap: 20
    parser_map:
        ".pdf": MultiModalParser
embedder_config:
    provider: truefoundry
    config:
        model: "openai-main/text-embedding-ada-002"

[S1LV3RJ1NX]

Answer to Q1: The contents are equal to given in local.metadata.sample. It contains:

• collection name
• data source -> Includes type (from registry of data loaders), uri
• parser config
• embedder_config

Answer to Q2: openai-main is the the name given to group of models by us hosted on truefoundy and served via LLM Gateway

[avibathula]

I ran a search for local.metadata.sample and there was no such file in the repo. Did you mean to suggest seeing local.metadata.yaml as a sample?

Also, where can I find more info about what all groups (like "openai-main") does truefoundry provide?

Searching in the code itself, the only place I saw some usages at cognita/backend/modules/query_controllers/example /payload.py and I only see openai-main/gpt-3-5-turbo - is that the only one in this group?

Are you folks planning to adding any docs site for Cognito, like what you have for Truefoundry platform? It will make onboarding a lot more easier.

PS: Just a small sincere request. It would be really helpful if someone can respond to questions faster. I thought this project has also gone the route of https://github.com/pinecone-io/canopy/ which got abandoned after initial release. So, I sort of moved on to try SciPhi's R2R and see if that is better suited for my needs, one of them being activity in the community. I understand it is a new community, and that's why it needs to be jumpstarted for the flywheel to get moving. If you prefer to have a Discord server, that works too. Turnaround time of 1+ weeks is too slow and just doesn't work. Once the community starts humming, it will become self-sustaining - but until then, it needs some TLC. Just sharing my 2 cents and I see this repo to have a lot of potential. Thank you.

[agutgutia1991]

Hi Avi, Thanks for the vote of confidence on the repo. We appreciate your feedback and agree that 1+ weeks to reply is a huge delay. We will take it in priority to create a place where questions can be answered faster by our internal team till we have the community support. Give us 1-2 days to have a framework for faster resolution. We will also get back on the other questions + questions related to docs you have asked by morning IST tomorrow. PS: We are also looking for support from folks who can start contributing to the repo. Thanks, Anuraag

…

On Thu, 30 May, 2024, 02:23 Avi Bathula, ***@***.***> wrote: I ran a search for local.metadata.sample and there was no such file in the repo. Did you mean to suggest seeing local.metadata.yaml as a sample? Also, where can I find more info about what all groups (like "openai-main") does truefoundry provide? Searching in the code itself, the only place I saw some usages at cognita/backend/modules/query_controllers/example /payload.py and I only see openai-main/gpt-3-5-turbo - is that the only one in this group? Are you folks planning to adding any docs site for Cognito, like what you have for Truefoundry platform? It will make onboarding a lot more easier. PS: Just a small sincere request. It would be really helpful if someone can respond to questions faster - I sort of moved on to try SciPhi's R2R and see if that is better suited for my needs, one of them being activity in the community. I understand it is a new community, and that's why it needs to be jumpstarted for the flywheel to get moving. If you prefer to have a Discord server, that works too. Turnaround time of 1+ weeks is too slow and just doesn't work. Once the community starts humming, it will become self-sustaining - but until then, it needs some TLC. Just sharing my 2 cents and I see this repo to have a lot of potential. Thank you. — Reply to this email directly, view it on GitHub <#161 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADKJQF6CDJ7H25BXAD5M6NTZEY527AVCNFSM6AAAAABH6PTBGCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TKOJYGQZDE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>