Open Data Access Layer: Access data freely, painlessly, and efficiently
- Documentation: stable | main
- Release notes
- azblob: Azure Storage Blob services.
- azdfs: Azure Data Lake Storage Gen2 services. (As known as abfs)
- fs: POSIX alike file system.
- ftp: FTP and FTPS support.
- gcs: Google Cloud Storage Service.
- ghac: Github Action Cache Service.
- hdfs: Hadoop Distributed File System(HDFS).
- http: HTTP read-only services.
- ipfs: InterPlanetary File System HTTP Gateway support.
- ipmfs: InterPlanetary File System MFS API support.
- memcached: Memcached service support.
- memory: In memory backend.
- moka: moka backend support.
- obs: Huawei Cloud Object Storage Service (OBS).
- oss: Aliyun Object Storage Service (OSS).
- redis: Redis services support.
- rocksdb: RocksDB services support.
- s3: AWS S3 alike services.
- sled: sled services support.
- webdav: WebDAV Service Support.
- webhdfs: WebHDFS Service Support.
Access data freely
- Access different storage services in the same way
- Behavior tests for all services
- Cross language/project bindings (working on)
Access data painlessly
- 100% documents covered
- Powerful
Layers
- Automatic retry support
- Full observability support: logging, tracing, metrics.
- Native decompress support
- Native service-side encryption support
Access data efficiently
- Zero cost: mapping to underlying API calls directly
- Best effort: auto pick the best
read
/seek
/next
implementations based on services - Auto metadata reuse: avoid extra
metadata
calls
use opendal::Result;
use opendal::layers::LoggingLayer;
use opendal::services;
use opendal::Operator;
#[tokio::main]
async fn main() -> Result<()> {
// Pick a builder and configure it.
let mut builder = services::S3::default();
builder.bucket("test");
// Init an operator
let op = Operator::create(builder)?
// Init with logging layer enabled.
.layer(LoggingLayer::default())
.finish();
// Create object handler.
let o = op.object("test_file");
// Write data
o.write("Hello, World!").await?;
// Read data
let bs = o.read().await?;
// Fetch metadata
let meta = o.metadata().await?;
let mode = meta.mode();
let length = meta.content_length();
// Delete
o.delete().await?;
Ok(())
}
More examples could be found at Documentation.
- Databend: A modern Elasticity and Performance cloud data warehouse.
- GreptimeDB: An open-source, cloud-native, distributed time-series database.
- deepeth/mars: The powerful analysis platform to explore and visualize data from blockchain.
- mozilla/sccache: sccache is ccache with cloud storage
- risingwave: A Distributed SQL Database for Stream Processing
Check out the CONTRIBUTING.md guide for more details on getting started with contributing to this project.
Submit issues for bug report or asking questions in discussion.
Licensed under Apache License, Version 2.0.