Skip to content

Rewrite fine-tuning guide #408

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Rewrite fine-tuning guide #408

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d0894f1
Rewrite fine-tuning guide
muhsinking Oct 9, 2025
230ae57
Update
muhsinking Oct 9, 2025
6ce89a8
Merge branch 'main' into fine-tuning-rewrite
muhsinking Oct 14, 2025
2d543e4
Merge branch 'main' into fine-tuning-rewrite
muhsinking Oct 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
177 changes: 88 additions & 89 deletions fine-tune.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,95 +1,94 @@
---
title: "Fine-tune a model"
description: "Learn how to fine-tune a large language model on Runpod using Axolotl."
---

This guide explains how to fine-tune a large language model using Runpod and Axolotl. You'll learn how to select a base model, configure your training environment, and start the fine-tuning process.
Fine-tuning is the process of taking a pre-trained large language model (LLM) and further training it on a smaller, specific dataset. This process adapts the model to a particular task or domain, improving its performance and accuracy for your use case.

## Prerequisites
This guide explains how to use Runpod's fine-tuning feature, powered by [Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl), to customize an LLM. You'll learn how to select a base model, choose a dataset, configure your training environment, and deploy your fine-tuned model.

Before you begin fine-tuning, ensure you have:
For more information about fine-tuning with Axolotl, see the [Axolotl Documentation](https://github.com/OpenAccess-AI-Collective/axolotl).

* A Runpod account with access to the Fine Tuning feature
* (Optional) A Hugging Face access token for gated models
## Requirements

## Select a base model
Before you begin, ensure you have:

To start fine-tuning, you'll need to choose a base model from Hugging Face:
* A Runpod account.
* (Optional) A [Hugging Face](https://huggingface.co/) account and an access token if you plan to use gated models or upload your fine-tuned model.

1. Navigate to the **Fine Tuning** section in the sidebar

2. Enter the Hugging Face model ID in the **Base Model** field
## Select a base model and dataset

* Example: `NousResearch/Meta-Llama-3-8B`
The base model is the starting point for your fine-tuning process, while the dataset provides the specific knowledge needed to adapt the base model to your task.

3. For gated models (requiring special access):
You can choose from thousands of models and datasets on [Hugging Face](https://huggingface.co/models).

1. Generate a Hugging Face token with appropriate permissions
2. Add your token in the designated field
<Tip>
If you're experimenting with fine-tuning for the first time, we recommend trying the [NousResearch/Meta-Llama-3-8B](https://huggingface.co/NousResearch/Meta-Llama-3-8B) model and the [tatsu-lab/alpaca](https://huggingface.co/datasets/tatsu-lab/alpaca) dataset.
</Tip>

## Select a dataset
## Deploy a fine-tuning Pod

You can choose a dataset from Hugging Face for fine-tuning:
<Steps>
<Step title="Go to the Fine-Tuning page">
Navigate to the [Fine-Tuning](https://console.runpod.io/fine-tuning) section in the Runpod console.
</Step>

1. Browse available datasets on [Hugging Face](https://huggingface.co/datasets?task_categories=task_categories:text-generation\&sort=trending)
2. Enter your chosen dataset identifier in the **Dataset** field
* Example: `tatsu-lab/alpaca`
<Step title="Specify the base model and dataset">
In the **Base Model** field, enter the Hugging Face model ID (e.g `NousResearch/Meta-Llama-3-8B`). In the **Dataset** field, enter the Hugging Face dataset ID (e.g `tatsu-lab/alpaca`).
</Step>

## Deploy the fine-tuning pod
<Step title="Provide a Hugging Face token (if needed)">
If you're using a gated model that requires special access, generate a Hugging Face token with the necessary permissions and add it to the **Hugging Face Access Token** field.
</Step>

Follow these steps to set up your training environment:
<Step title="Continue to the next step">
Click **Deploy the Fine-Tuning Pod** to start configuring your fine-tuning Pod.
</Step>

1. Click **Deploy the Fine Tuning Pod**
<Step title="Choose a GPU for the Pod">
Select a GPU instance based on your model's requirements. Larger models and datasets require GPUs with more memory.
</Step>

2. Select a GPU instance based on your model's requirements:
<Step title="Deploy the Pod">
Finishing configuring the Pod, then click **Deploy on-demand**. This should open the detail pane for your Pod automatically.
</Step>

* Smaller models: Choose GPUs with less memory
* Larger models/datasets: Choose GPUs with higher memory capacity
<Step title="Monitor the Pod's deployment">
Click **Logs** to monitor the system logs for deployment progress. Wait for the success message: `"You've successfully configured your training environment!"` Depending on the size of your model and dataset, this may take some time.
</Step>

3. Monitor the system logs for deployment progress
<Step title="Connect to your training environment">
Once your training environment is ready, you can connect to it to configure and start the fine-tuning process.

4. Wait for the success message: `"You've successfully configured your training environment!"`
Click **Connect** and choose your preferred connection method:
* **Jupyter Notebook**: A browser-based notebook interface.
* **Web Terminal**: A browser-based terminal.
* **SSH**: A secure connection from your local machine.

## Connect to your training environment

After your pod is deployed and active, you can connect using any of these methods:

1. Go to your Fine Tuning pod dashboard

2. Click **Connect** and choose your preferred connection method:

* **Jupyter Notebook**: Browser-based notebook interface
* **Web Terminal**: Browser-based terminal
* **SSH**: Local machine terminal connection

<Info>

To use SSH, add your public SSH key in your account settings. The system automatically adds your key to the pod's `authorized_keys` file.

</Info>
<Tip>
To use SSH, add your public SSH key in your account settings. The system automatically adds your key to the Pod's `authorized_keys` file. For more information, see [Connect to a Pod with SSH](/pods/configuration/use-ssh).
</Tip>
</Step>
</Steps>

## Configure your environment

Your training environment includes this directory structure in `/workspace/fine-tuning/`:

* `examples/`: Sample configurations and scripts
* `outputs/`: Training results and model outputs
* `config.yaml`: Training parameters for your model

The system generates an initial `config.yaml` based on your selected base model and dataset.
Your training environment is located in the `/workspace/fine-tuning/` directory and has the following structure:

## Review and modify the configuration
* `examples/`: Sample configurations and scripts.
* `outputs/`: Where your training results and model outputs will be saved.
* `config.yaml`: The main configuration file for your training parameters.

The `config.yaml` file controls your fine-tuning parameters. Here's how to customize it:
The system generates an initial `config.yaml` based on your selected base model and dataset. This is where you define all the hyperparameters for your fine-tuning job. You may need to experiment with these settings to achieve the best results:

1. Open the configuration file:
Open the configuration file (`config.yaml`) in JupyterLab or using your preferred text editor to review and adjust the fine-tuning parameters:

```sh
nano config.yaml
```

2. Review and adjust the parameters based on your specific use case
```sh
nano config.yaml
```

Here's an example configuration with common parameters:
Below is an example with common settings. You can replace the `NousResearch/Meta-Llama-3-8B` and `tatsu-lab/alpaca` with whichever base model and dataset you selected:

```yaml
base_model: NousResearch/Meta-Llama-3.1-8B
Expand All @@ -101,7 +100,7 @@ strict: false

# Dataset configuration
datasets:
- path: tatsu-lab/alpaca
- path: tatsu-lab/alpaca
type: alpaca
dataset_prepared_path: last_run_prepared
val_set_size: 0.05
Expand Down Expand Up @@ -136,7 +135,7 @@ tf32: false

gradient_checkpointing: true
gradient_checkpointing_kwargs:
use_reentrant: false
use_reentrant: false
early_stopping_patience:
resume_from_checkpoint:
logging_steps: 1
Expand All @@ -153,51 +152,51 @@ weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
pad_token: <|end_of_text|>
pad_token: <|end_of_text|>
```

<Info>

The `config.yaml` file contains all hyperparameters needed for fine-tuning. You may need to iterate on these settings to achieve optimal results.

</Info>

For more configuration examples, visit the [Axolotl examples repository](https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples).
For more configuration examples, refer to the [Axolotl examples repository](https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples).

## Start the fine-tuning process

Once your configuration is ready, follow these steps:
Once you're satisfied with your configuration, you can start the training.

1. Start the training process:
Run the following command in your terminal:

```sh
axolotl train config.yaml
```
```sh
axolotl train config.yaml
```

2. Monitor the training progress in your terminal
Monitor the training progress in your terminal. The output will show the training loss, validation loss, and other metrics.

## Push your model to Hugging Face

After completing the fine-tuning process, you can share your model:
After the fine-tuning process is complete, you can upload your model to the Hugging Face Hub to share it with the community or use it in your applications.

1. Log in to Hugging Face:
<Steps>
<Step title="Log in to Hugging Face">

```sh
huggingface-cli login
```
Run this command to log in to your Hugging Face account:

2. Create a new repository on Hugging Face if needed
```sh
huggingface-cli login
```
</Step>
<Step title="Upload your model files">
To upload your model files to the Hugging Face Hub, run this command:

3. Upload your model:
```sh
huggingface-cli upload YOUR_USERNAME/MODEL_NAME ./outputs/out
```
Replace `YOUR_USERNAME` with your Hugging Face username and `MODEL_NAME` with your desired model name.

```sh
huggingface-cli upload <your-username>/<model-name> ./output
```
</Step>
</Steps>

Replace `<your-username>` with your Hugging Face username and `<model-name>` with your desired model name.
## Next steps

## Additional resources
Now that you have a fine-tuned model, you can deploy it on Runpod for inference:

For more information about fine-tuning with Axolotl, see:
* **[Serverless](/serverless/overview)**: Deploy your model as a Serverless endpoint for a pay-as-you-go, scalable solution.
* **[Pods](/pods/overview)**: Run your model on a dedicated Pod for more control and persistent workloads.

* [Axolotl Documentation](https://github.com/OpenAccess-AI-Collective/axolotl)