Metadata array not working through Llamaindex pinecone

[selvagmm]

We are accessing pinecone using Llamaindex library. One of our use cases is that we need to add an array of strings as metadata. However, we could not do because Llamaindex throwing below error when we add an array as metadata.

Error: Value for metadata tag_id must be one of (str, int, float, None)

Llamaindex version is 0.8.65.

Please let us know if there is a support for metadata array in Llamaindex- pinecone connector in the latest or future versions. Or Let us know of there is any work around.

Thanks,

Selva.

[logan-markewich]

Does pinecone support lists as metadata? If so, we can fix that. If not though, I would advise to use another vector db

[selvagmm]

Hi Logan, Yes, pinecone supports lists as metadata. Please help us fix this issue. https://docs.pinecone.io/docs/metadata-filtering Thanks and Regards, Selvaganapathy C Sr Technical Architect [cid:efd0a270-367f-458a-a386-15416ad436d7] Mintmesh Inc| ISO & SOC 2 Certified[cid:cabd647c-dfd8-40f7-a624-9a9833878b66] Corporate Office: 400 Renaissance Center, Suite 2600, Detroit, MI 48243 M:+91 9884336369 | E: ***@***.******@***.***> | www.mintmesh.ai<http://www.mintmesh.ai/>

…

________________________________ From: Logan ***@***.***> Sent: 07 January 2024 22:39 To: run-llama/llama_index ***@***.***> Cc: Selvaganapathy C ***@***.***>; Author ***@***.***> Subject: Re: [run-llama/llama_index] Metadata array not working through Llamaindex pinecone (Discussion #9048) Does pinecone support lists as metadata? If so, we can fix that. If not though, I would advise to use another vector db — Reply to this email directly, view it on GitHub<#9048 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AWG4FY6VE2RJVFW5EDHX3NDYNLJEBAVCNFSM6AAAAAA7UE3MW2VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DAMZYGY3TA>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[aukinfo]

Hi, you can fix this with a custom metadata extractor - i had same problem:

Here are the code parts that differ from the shipped class:

async def _aextract_entities_from_node(self, node: BaseNode) -> Dict[str, str]:
        """Extract entities from a node and return it's metadata dict."""
        if self.is_text_node_only and not isinstance(node, TextNode):
            return {}

        context_str = node.get_content(metadata_mode=self.metadata_mode)
        entities = await self.llm.apredict(
            PromptTemplate(template=self.prompt_template),
            num_entities=self.num_entities,
            context_str=context_str,
        )

        # Remove the backticks and ```json part using regex
        entities = utils.trim_json_string(entities.strip())
        entities = json.loads(entities)

        return {"excerpt_entities": entities}

And the template:

DEFAULT_ENTITY_EXTRACT_TEMPLATE = """\
Context: {context_str}.\n Provide an array of up to {num_entities} entities from the \
provided context. Be sure to include all port names, project names, people, companies, towns, vessels etc. Proper nouns have priority. Format as an array of strings. Entities: """

And the json cleaner (needed if openai is your llm)

def trim_json_string(input_str):
    """
    Trims surrounding noise from a JSON string.

    Args:
    input_str (str): The input string containing JSON data surrounded by noise.

    Returns:
    str: A cleaned JSON string.
    """
    # Define a regular expression pattern to extract JSON object or array
    pattern = r'(\{.*\}|\[.*\])'
    
    # Search for the pattern in the input string
    match = re.search(pattern, input_str, re.DOTALL)
    
    if match:
        # Extract the JSON part
        json_str = match.group(0)
        
        try:
            # Validate the JSON format
            json.loads(json_str)
            return json_str
        except json.JSONDecodeError:
            return "Invalid JSON found within the input string."
    else:
        return "No valid JSON object or array found in the input string."