openfs

An S3 feature store client for data pipelines.

Openfs provides a simple api to boost the quality of your training data while keeping your data pipelines clean and manageable.

Installation

pip install openfs

Quick Start

Creating a Store

import openfs as fs
from openfs.stores import FeatureStore
from openfs.boosters import Booster
import os

# <- import files to upload here

# Connect to store bucket
fs.client.connect(
    region_name=os.environ['FSTORE_REGION'],
    endpoint_url=os.environ['FSTORE_ENDPOINT_URL'],
    access_key_id=os.environ['FSTORE_ACCESS_KEY'],
    secret_access_key=os.environ['FSTORE_SECRET_KEY']
)

# Create store
store = FeatureStore("store_name", "description of store", "some_primary_key")

# Upload store
response = store.upload(files, filenames)

Creating a Booster Dataset

# Create booster
booster = Booster(store_id=response['store_id'])

# Add features
booster.add_single("feature_1", alias="alias_for_feature")
booster.add_group(["feature_2", "feature_3"], alias="grouped_feature", how='sum')

# pull features from store (for testing)
df = booster.create_df()

# upload booster
booster.upload(name="booster_name", description="booster description")

Viewing Stores

fb.client.list_stores()

Contributors

We'd love to welcome contributors to openfs to help make training data richer and more open for everyone. We're working on our contributor docs at the moment, but if you're interested in contributing, please send us a message at [email protected].

Thanks for trying openfs!

Made with ❤️ in Australia

---

© copyright xplainable pty ltd