Feat: UnstructuredFileConverter meta field

[lambda-science]

Fix: #241

This pull request add a new optional meta field to add custom metadata to Unstructured documents.
From:

class UnstructuredFileConverter:

    @component.output_types(documents=List[Document])
    def run(self, paths: Union[List[str], List[os.PathLike]]):

To:

class UnstructuredFileConverter:

    @component.output_types(documents=List[Document])
    def run(
        self,
        paths: Union[List[str], List[os.PathLike]],
        meta: Optional[Union[Dict[str, Any], List[Dict[str, Any]]]] = None,
    ):

The implementation is inspired from the handling of metadata in the PyPDFToDocument component from Haystack main repo.

Ping @anakin87

[CLAassistant]

All committers have signed the CLA.

[lambda-science]

Not sure I understand why the test are failing on 3.9

[lambda-science]

And maybe I should add more test case because all test case only have one file with one metadata dict. Maybe a test case with two files and a meta list of dict

EDIT: Done.

[anakin87]

@lambda-science tests are failing because of mypy. Something in the code should be fixed.

Thanks for your PR. I will take a look later or tomorrow...

[lambda-science]

@lambda-science tests are failing because of mypy. Something in the code should be fixed.

Thanks for your PR. I will take a look later or tomorrow...

No worries, I realized I could use my own fix before it is merged and released with a simple requirement.txt such as:

unstructured-fileconverter-haystack @ git+https://github.com/deepset-ai/haystack-core-integrations@3bc51a788356545dc692183591cf91bdf5391713#subdirectory=integrations/unstructured

That's very cool ! So it's not urgent at all 🌞

[lambda-science]

I have the feeling that something is off with metadata, because the auto-generated page_number is always the last page with one doc per elem
SOLVED: ..... mutable data in python, always have been...... replacing metadata = meta by metadata = meta.copy() solved it 😭

[anakin87]

Hey @lambda-science, thanks for the good work!

• I left a comment about metadata handling
• I would add a test where the path is a directory

[anakin87]

Hey... Sorry for the long wait.

I found some small opportunities for improvement, then we can merge this good PR!

[anakin87]

Looks good...

Thanks @lambda-science!

I'm going to merge this PR and release a new version...

[anakin87]

New version: https://pypi.org/project/unstructured-fileconverter-haystack/0.3.0/