ideas: Split opendal rust core into multiple crates

[Xuanwo]

This idea is part of The Way to OpenDAL Rust Core v1.0

Background

OpenDAL Rust expose tow levels of APIs:

• Public API: all APIs that exposed under opendal::* like Operator, Reader, Writer.
• Raw API: all APIs that exposed under opendal::raw::* like Access, oio::Read, oio::Write

OpenDAL's public APIs rarely change, yet the raw APIs are frequently refactored. Even when there are no changes within the public API, users still need to upgrade simultaneously.

At the same time, the OpenDAL Rust core is quite extensive and includes:

• all services: services::s3, services::gcs, etc.
• all layers: layers::RetryLayer, layers::MetricsLayer, etc.

We used to increment OpenDAL versions solely due to an API change in services::s3.

I'm considering whether it would be beneficial to split the OpenDAL Rust core into multiple crates, allowing users to independently upgrade services like S3 as needed. More importantly, this approach would provide users with a stable API for Operator, ensuring compatibility across all versions.

Idea

Plan A: Split all and keep the shell

Divide the opendal rust core into:

• opendal
• opendal_core
• opendal_service_xxx (like opendal_service_s3)
• opendal_layer_xxx (like opendal_layer_retry)

The crate opendal serves as a shell that includes all widely used services and layers. Throughout its evolution, we may remove some services.

Plan B: Split Core Only

Only divide opendal core out:

• opendal
• opendal_core

opendal will depend on and re-expore all API from opendal_core instead. This will make it super easier for users to implement their own services.

Plan C: Split all

Remove all services and layers from opendal:

• opendal
• opendal_service_xxx (like opendal_service_s3)
• opendal_layer_xxx (like opendal_layer_retry)

opendal itself will not provide any services. Following this change, users will need to add new crates instead of incorporating new features.

---

What are your thoughts? I welcome any comments or feedback.

[simonsan]

Plan A sounds like a good approach to me. Having an umbrella crate that is the entrypoint to the opendal ecosystem, and then being able to also compose your own with pulling in directly opendal_core and the layers and services. Having the essential traits in opendal_core will still make it easy to implement your own services and layers on top of it. 👍🏽

[yuchanns]

Plan A and C are beneficial to bindings.

[morristai]

Plan A + 1

[PsiACE]

I think plan A or C would be great.

[tisonkun]

Where do the configs stay after each plan? Currently, configs are always available while their corresponding services is hidden behind the feature flag.

[Xuanwo]

That's an good question.

I currently feel that it's impossible for us to maintain all configurations within the same crate. I aim to enable the implementation of breaking changes to a specific service configuration without affecting opendal-core.

Therefore, my current plan is to move the configuration to its own separate crate.

[tisonkun]

Then if the service implementation gated by a feature flag? The original purpose to lift configs is that downstream projects can passing and storing configure the operator without polling dependencies to the real service implementation.

[Xuanwo]

Yes, it's exactly the place I want to improve.

Our current design prohibits the implementation of breaking changes; even a minor correction, such as fixing a typo, could completely disrupt a user's stored configuration.

I plan to introduce a version field for each configuration, the value of which will always indicate the current version of the configuration. We will document the evolution of each configuration, allowing server maintainers to determine how many versions they wish to remain compatible with, such as through renaming or printing debug messages.

The aim is to ensure that users can safely pass and store a HashMap<String, String> or JSON bytes without disruption and without directly depending on the configuration structure.

This means that users can directly save a hashmap or JSON bytes in their own storage and rely on the service implementation only when necessary.

[simonsan]

The aim is to ensure that users can safely pass and store a HashMap<String, String> or JSON bytes without disruption and without directly depending on the configuration structure.

Yeah, I think that would be nice! 👍🏽

[ho-229]

Hi, I want to know if there is any user feedback related to the background or how users can use OpenDAL.

[Xuanwo]

Hello, most of the background information can be found at #5189.

We are planning to advance OpenDAL to version 1.0 (specifically the OpenDAL Rust core 1.0). It's clear that we cannot move to version 1.0 in its current state. We are aware that there are breaking changes ahead, and we need to clearly define the boundaries: which parts will be stable and which will not.

By splitting the rust core of OpenDAL into opendal_core and opendal_services_s3, we establish a stable opendal_core first while continuing to develop the services.