Dagster × OpenAI Fine-Tune Demo

Fine-tune a custom model to detect specific features from Goodreads data.

In this example project we show how you can write a pipeline that ingests data from Goodreads with DuckDB and then generates features for modeling. You can then use this data to fine-tune a model in OpenAI to identify that feature while also validating the model against the base model is was built against.

Example Asset Lineage

Getting started

Install the project dependencies:

pip install -e ".[dev]"

Run Dagster:

dagster dev

Open http://localhost:3000 in your browser.

References

Dagster

• Dagster Docs
• Dagster Docs: DuckDB
• Dagster Docs: OpenAI Integration

DuckDB

• DuckDB Docs

OpenAI

• OpenAI Fine-Tuning
• OpenAI Cookbook: How to fine-tune chat models
• OpenAI Cookbook: Data preparation and analysis for chat model fine-tuning