Integrate Arrow Data fusion to support wide variety of connectors

[mediuminvader]

Current list of connectors doesn't allow to expose a datalake/S3 as APIs.
However, with the integration of Data Fusion as a source connector, it will be possible to retrieve data from an object store and ingest in the Dozer pipeline.

[xudong963]

Do you mean Arrow Datafusion?

[v3g42]

This is great. We just recently caught up with @andygrove from arrow and discussed a good way to integrate batch workflows or bring the support of data fusion sources in the dozer eco system.

This means we will be able to merge S3 data with postgres on the fly and serve it as APIs efficiently backed by a cache.

It could be a good use case.

[v3g42]

@karolisg @mediuminvader We should probably include a some more documentation on how to extend or implement Dozer connectors.