Enable support for custom filesystem

[thiagosalvatore]

Custom FileSystems that are now available on llama-index were not available on llama-parse. We can now use it to download files using any reader like S3Reader.

Fixes #116

[thiagosalvatore]

@logan-markewich can you take a look at this when you have some time, please?

[logan-markewich]

I don't think this works, at least locally, it doesn't work for me

Since we already have the path, can't we use that?

[thiagosalvatore]

I am using it in production right now and it is working properly. What is the error that you are getting locally?

The issue with the path is that I've seen situations in production where people upload a file with .txt but the file is actually a .csv and it doesn't work.

[logan-markewich]

:sigh: this still doesn't work guys lol

>>> from llama_parse import LlamaParse
>>> documents = LlamaParse(api_key="llx-...").load_data("2023.acl-srw.0.pdf")
Error while parsing the file '2023.acl-srw.0.pdf': '_io.BufferedReader' object has no attribute 'full_name'
>>> 

[sheresaidon]

@logan-markewich whenever you have a chance could you please review, currently llama parse is giving me the fs error when trying to use it with GCS Reader

parser = LlamaParse(result_type="markdown", api_key=LLAMA_PARSE_API_KEY,verbose=True, show_progress=True)

reader = GCSReader(
    bucket=bucket_name,
    key=file_path,  # Use the extracted file path
    file_extractor={".pdf": parser, ".docx": parser, ".txt": parser},
    service_account_key=service_account_key_dict
)

LlamaParse.load_data() got an unexpected keyword argument 'fs'. Skipping...

[thiagosalvatore]

@logan-markewich any updates here? I've been using this fix in prod for a while without any issues

[gjyoungjr]

any timeline on when this PR will get merged? my team is also blocked by not having this featue.

does anyone know a current work around to load files that are not local in your directory?

[logan-markewich]

this still doesn't work 😅

>>> from llama_parse import LlamaParse
>>> documents = LlamaParse(api_key="llx-...").load_data("2023.acl-srw.0.pdf")
Error while parsing the file '2023.acl-srw.0.pdf': '_io.BufferedReader' object has no attribute 'full_name'
>>> 

[logan-markewich]

This is still broken actually, see above

[logan-markewich]

Ok, I think this works, but haven't actually tested it with an external file system

[rohanprashant12]

Thanks @logan-markewich and @thiagosalvatore!