Move over unstructured parser

[flash1293]

What

Closes #31614

This PR moves the unstructured parser currently in the S3 connector over to the CDK.
This will enable unstructured file extraction for all connectors based on the file based CDK.

⚠️ Based on the dependencies, the hi_res parsing strategy can't be applied - it will default to fast and fall back to ocr_only if necessary. ⚠️

Open questions:

• Should there be one parser per file type? The downside of that is that the user needs to configure more stuff
• Should it be possible to opt out of unstructured files? Maybe there are some file sources that don't want to support this

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 26, 2023 10:38am

[github-actions]

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

• PR name follows PR naming conventions
• Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
• Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
• You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
• Secrets in the connector's spec are annotated with airbyte_secret
• All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
• Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
• Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
• If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

1. Check for hidden checklists in your PR description

2. Toggle the github label checklist-action-run on/off to re-run the checklist CI.

[aaronsteers]

Regarding these inline imports, are these not on the top of the file because they are extras in the CDK, and because they might not be available at runtime if the extra is not included?

I think the general best practice for these scenarios would be something like this at the top of the file...

try:
    import unstructured
    from unstructured.file_utils.filetype import FileType
    from unstructured.partition.auto import partition
    from unstructured.partition.md import optional_decode
except ImportError as ex:
    logger.warning(f"Unstructured libraries could not be imported. {ex}")
    FileType = None
    partition = None
    optional_decode = None

Then within the individual functions, we can either assume that the code path will not be traversed without the dependencies, or in certain points we can just check the needed import at top of the function.

    def _read_file(self, file_handle: IOBase, file_name: str) -> str:
        if unstructured is none:
            raise ImportError("Unstructured library is not available. Please install any missing libraries (...) and try again.")

[flash1293]

They are inline because they are quite slow to import - if they are on the top of the file it would slow down the connector even if the parser is not used because python has to load so much code. I can add a comment to explain this (or maybe there is a more elegant way to handle this I’m not aware of)

[aaronsteers]

In that case, what do you think about putting into a helper function called by the class constructor?

Pseudocode, may need adjustment:

...
unstructured_lib = None

def _import_unstructured():
    """Dynamically imported as needed, due to slow import speed."""
    global unstructured_lib

    import unstructured
    unstructured_lib = unstructured

class UnstructuredParser(FileTypeParser):
    def __init__():
        _import_unstructured()

[flash1293]

Thanks for the review @aaronsteers , I changed the way the dynamic imports are handled, could you take another look?

[clnoll]

LGTM @flash1293!

[aaronsteers]

Looks good!
Thanks for those changes. 👍