Ibis backend blowout: a proposal

[lostmygithubaccount]

Issue Description

Background:
At present, Ibis supports a range of backends for data analysis and manipulation. However, there is an opportunity to improve usability and compatibility by expanding support for different "flavors" or "families" within certain backends. This proposal aims to address this gap by adding support for additional backend variants.

Motivation

Improved User Experience:
By enhancing backend support within Ibis, I aim to provide a better user experience and make it easier for users to find and confirm the supported backends in the documentation and search. This includes improving the documentation, ensuring better searchability, and providing more comprehensive information on specific backend variants. This will allow users to make informed decisions when selecting a backend that best suits their needs.

Better Defaults and Specifics:
As different backends evolve, they may introduce variations and specific features that can differ from the standard implementation. By expanding backend support in Ibis, I can stay up-to-date with these changes and provide better defaults and specifics for each variant. For example, Presto may have certain functionality that differs from Trino, and by supporting both, we can ensure that users have access to the most relevant and accurate information for their chosen backend.

Proposal:

I propose extending Ibis's backend support to incorporate various flavors or families within specific backends. By doing so, we can enhance compatibility, address the unique requirements of diverse user bases, and provide a better overall user experience. To begin, I suggest expanding support for the following backends:

1. PySpark:

• Databricks
• Azure Synapse
• GCP Dataproc (?)
• AWS EMR/Glue/something (?)

2. Trino:

• Presto
• Starburst

3. PostgreSQL:

• AWS Redshift
• AWS Aurora
• CockroachDB (?)
• ...

I invite the community members, developers, and users to provide input and suggestions regarding these proposed additions. Your expertise and insights will be invaluable in shaping the future direction of Ibis's backend support. We can prioritize based on input.

Open Questions:

It is important to note that this proposal will introduce some engineering overhead. However, it seems like a good case for inheritance to handle the variations among backend variants. Further discussions and considerations are needed to determine the best approach to handle the engineering overhead efficiently.

---

I'm proposing this to the maintainers and the community to gather feedback and gauge interest in this enhancement.

Thank you for your collaboration!

[cpcloud]

Thanks for the issue!

I'm not sure I follow whether there's any actionable feedback here.

We won't really know exactly what any kind of reusable architecture looks like here until we see what the variations are in the Trino-like, PySpark-like, etc backends are.

Referring to a specific pattern such as inheritance feels a bit premature, given that we don't know whether that's a pattern that will address the problem of backend variation. It might, but it might not, we won't know until we see where things differ.

I don't see any issue with creating a new backend for the new variations, and perhaps inheriting from the backend that defines common functionality.

I definitely don't think we need a new abstraction or user-facing concept like a backend family to address the variations.

Again though we'll have to see where things differ.