Cookiecutter - MLOps Package

Jumpstart your MLOps projects with this comprehensive Cookiecutter template.

The template provides a robust foundation for building, testing, packaging, and deploying Python packages and Docker Images tailored for MLOps tasks.

Related resources:

• MLOps Coding Course (Learning): Learn how to create, develop, and maintain a state-of-the-art MLOps code base.
• MLOps Python Package (Example): Kickstart your MLOps initiative with a flexible, robust, and productive Python package.
• LLMOps Coding Package (Example): Example with best practices and tools to support your LLMOps projects.

Philosophy

This Cookiecutter is designed to be a common ground for diverse MLOps environments. Whether you're working with Kubernetes, Vertex AI, Databricks, Azure ML, or AWS SageMaker, the core principles of using Python packages and Docker images remain consistent.

This template equips you with the essentials for creating, testing, and packaging your AI/ML code, providing a solid base for integration into your chosen MLOps platform. To fully leverage its capabilities within a specific environment, you might need to combine it with external tools like Airflow for orchestration or platform-specific SDKs for deployment.

You have the freedom to structure your src/ and tests/ directories according to your preferences. Alternatively, you can draw inspiration from the structure used in the MLOps Python Package project for a ready-made implementation.

Key Features

• Streamlined Project Structure: A well-defined directory layout for source code, tests, documentation, tasks, and Docker configurations.
• Uv Integration: Effortless dependency management and packaging with uv.
• Automated Testing and Checks: Pre-configured workflows using Pytest, Ruff, Mypy, Bandit, and Coverage to ensure code quality, style, security, and type safety.
• Pre-commit Hooks: Automatic code formatting and linting with Ruff and other pre-commit hooks to maintain consistency.
• Dockerized Deployment: Dockerfile and docker-compose.yml for building and running the package within a containerized environment (Docker).
• Invoke Task Automation: PyInvoke tasks to simplify development workflows such as cleaning, installing, formatting, checking, building, documenting, and running MLflow projects.
• Comprehensive Documentation: pdoc generates API documentation, and Markdown files provide clear usage instructions.
• GitHub Workflow Integration: Continuous integration and deployment workflows are set up using GitHub Actions, automating testing, checks, and publishing.

Quick Start

1. Generate your project:

pip install cookiecutter
cookiecutter gh:fmind/cookiecutter-mlops-package

You'll be prompted for the following variables:

• user: Your GitHub username.
• name: The name of your project.
• repository: The name of your GitHub repository.
• package: The name of your Python package.
• license: The license for your project.
• version: The initial version of your project.
• description: A brief description of your project.
• python_version: The Python version to use (e.g., 3.12).
• mlflow_version: The MLflow version to use (e.g., 2.19.0).

2. Initialize a git repository:

cd {{ cookiecutter.repository }}
git init

3. Enable GitHub Pages Workflow:

• Navigate to your repository settings on GitHub: "Settings" -> "Actions" -> "General."
• Under "Workflow permissions," ensure "Read and write permissions" is selected.
  • This allows the workflow to automatically publish your documentation.

4. Explore the generated project:

• src/{{cookiecutter.package}}: Your Python package source code.
• tests/: Unit tests for your package.
• tasks/: PyInvoke tasks for automation.
• Dockerfile: Configuration for building your Docker image.
• docker-compose.yml: Orchestration file for running MLflow and your project.

5. Start developing!

Use the provided Invoke tasks to manage your development workflow:

• invoke installs: Install dependencies and pre-commit hooks.
• invoke formats: Format your code.
• invoke checks: Run code quality, type, security, and test checks.
• invoke docs: Generate API documentation.
• invoke packages: Build your Python package.
• invoke projects: Run MLflow projects.
• invoke containers: Build and run your Docker image.

Example Usage

Running an MLflow Project

After installing dependencies and setting up MLflow:

invoke projects

This will execute the job with the configuration file in your confs folder.

Building and Running Your Docker Image

invoke containers

This builds a Docker image based on your Dockerfile and runs it. The CMD in the Dockerfile executes your package with the --help flag.

Contributions

We welcome contributions to enhance this Cookiecutter template for generating MLOps projects.

Feel free to open issues or pull requests for any improvements, bug fixes, or feature requests.

License

This project is licensed under the MIT License. See the LICENSE.txt file for details.