Oracle Cloud Infrastructure Object Storage fsspec Implementation

​ The Oracle Cloud Infrastructure Object Storage service is an internet-scale, high-performance storage platform that offers reliable and cost-efficient data durability. With Object Storage, you can safely and securely store or retrieve data directly from the internet or from within the cloud platform. ​ ocifs is part of the fsspec intake/filesystem_spec ecosystem ​

a template or specification for a file-system interface, that specific implementations should follow, so that applications making use of them can rely on a common interface and not have to worry about the specific internal implementation decisions with any given backend. ​ ocifs joins the list of file systems supported with this package. ​ The intake/filesystem_spec project is used by Pandas, Dask and other data libraries in python, this package adds Oracle OCI Object Storage capabilties to these libraries. ​

Example Usage

from ocifs import OCIFileSystem

fs = OCIFilesystem("~/.oci/config")
fs.ls("oci://<my_bucket>@<my_namespace>/<my_prefix>")
# [<my_bucket>@<my_namespace>/<my_prefix>/obj1, <my_bucket>@<my_namespace>/<my_prefix>/obj2]

fs.cat("oci://<my_bucket>@<my_namespace>/<my_prefix>/obj1")
# b"Hello World"

with fs.open("oci://<my_bucket>@<my_namespace>/<my_prefix>/obj3", 'w') as f:
    f.write("Adding a third object.")

fs.copy("oci://<my_bucket>@<my_namespace>/<my_prefix>/obj3", "oci://<my_bucket>@<my_namespace>/<my_prefix>/obj1")

with fs.open("oci://<my_bucket>@<my_namespace>/<my_prefix>/obj1") as f:
    print(f.read())
# b"Adding a third object."

Or Use With Pandas

​

import pandas as pd
import ocifs
​
df = pd.read_csv(
    "oci://my_bucket@my_namespace/my_object.csv",
    storage_options={"config": "~/.oci/config"},
)

​

Getting Started

python3 -m pip install ocifs

Software Prerequisites

Python >= 3.6

Environment Variables for Authentication:

export OCIFS_IAM_TYPE=api_key
export OCIFS_CONFIG_LOCATION=~/.oci/config
export OCIFS_CONFIG_PROFILE=DEFAULT

Note, if you are operating on OCI with an alternative valid signer, such as resource principal, instead set the following:

export OCIFS_IAM_TYPE=resource_principal

Documentation

• 
• ocifs Documentation
• ocifs GitHub

Support

The built-in filesystems in fsspec are maintained by the intake project team, where as ocifs is an external implementation (similar to s3fs, gcsfs, adl/abfs, and so on), which is maintained by Oracle.

Contributing

This project welcomes contributions from the community. Before submitting a pull request, please review our contribution guide

Security

Please consult the security guide for our responsible security vulnerability disclosure process

License

Copyright (c) 2021, 2023 Oracle and/or its affiliates.

Released under the Universal Permissive License v1.0 as shown at https://oss.oracle.com/licenses/upl/.