README.md

OpenDAL  

OpenDAL is a data access layer that allows users to easily and efficiently retrieve data from various storage services in a unified way.

• Documentation: stable | main
• Release notes

Services

Standard Storage Protocols (like ftp, webdav)

• ftp: FTP and FTPS
• http: HTTP read-only services
• sftp: SFTP services being worked on
• webdav: WebDAV Service

Object Storage Services (like s3, gcs, azblob)

• azblob: Azure Storage Blob services
• cos: Tencent Cloud Object Storage services
• gcs: Google Cloud Storage Service
• obs: Huawei Cloud Object Storage Service (OBS)
• oss: Aliyun Object Storage Service (OSS)
• s3: AWS S3 alike services
• supabase: Supabase Storage Service being worked on
• wasabi: Wasabi Cloud Storage

File Storage Services (like fs, azdfs, hdfs)

• fs: POSIX alike file system
• azdfs: Azure Data Lake Storage Gen2 services (As known as abfs)
• hdfs: Hadoop Distributed File System(HDFS)
• ipfs: InterPlanetary File System HTTP Gateway
• ipmfs: InterPlanetary File System MFS API being worked on
• webhdfs: WebHDFS Service

Consumer Cloud Storage Service (like gdrive, onedrive)

• gdrive: Google Drive being worked on
• onedrive: OneDrive being worked on

Key-Value Storage Service (like rocksdb, sled)

• cacache: cacache backend
• dashmap: dashmap backend
• memory: In memory backend
• persy: persy backend
• redis: Redis services
• rocksdb: RocksDB services
• sled: sled backend
• redb: redb backend
• tikv: tikv backend

Cache Storage Service (like memcached, moka)

• ghac: Github Action Cache Service
• memcached: Memcached service
• mini_moka: Mini Moka backend
• moka: Moka backend
• vercel_artifacts: Vercel Remote Caching Service being worked on

Welcome to add any services that are not currently supported here.

Features

Access data freely

• Access different storage services in the same way
• Behavior tests for all services

Access data painlessly

• 100% documents covered
• Powerful Layers
• Automatic retry support
• Full observability: logging, tracing, metrics.
• Native chaos testing

Access data efficiently

• Zero cost: Maps to API calls directly
• Best effort: Automatically selects best read/seek/next based on services
• Avoid extra calls: Reuses metadata when possible

Quickstart

use opendal::Result;
use opendal::layers::LoggingLayer;
use opendal::services;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    // Pick a builder and configure it.
    let mut builder = services::S3::default();
    builder.bucket("test");

    // Init an operator
    let op = Operator::new(builder)?
        // Init with logging layer enabled.
        .layer(LoggingLayer::default())
        .finish();

    // Write data
    op.write("hello.txt", "Hello, World!").await?;

    // Read data
    let bs = op.read("hello.txt").await?;

    // Fetch metadata
    let meta = op.stat("hello.txt").await?;
    let mode = meta.mode();
    let length = meta.content_length();

    // Delete
    op.delete("hello.txt").await?;

    Ok(())
}

Examples

The examples are available at here.

Contributing

Check out the CONTRIBUTING guide for more details on getting started with contributing to this project.

License

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0