Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Guide For Multi-Node Distributed Finetuning #1477

Closed
wants to merge 6 commits into from

Conversation

shahdivax
Copy link

Title: Distributed Finetuning for Multi-Node Setup Guide

Description:
This PR introduces a comprehensive guide for setting up a distributed finetuning environment using Axolotl and Accelerate. The guide covers the following steps:

  1. Configuring SSH for passwordless access across multiple nodes
  2. Generating and exchanging public keys for secure communication
  3. Configuring Axolotl with shared settings and host files
  4. Configuring Accelerate for multi-node training with Deepspeed
  5. Running distributed finetuning using Accelerate

Title: Distributed Finetuning Setup Guide

Description:
This PR introduces a comprehensive guide for setting up a distributed finetuning environment using Axolotl and Accelerate. The guide covers the following steps:

1. Configuring SSH for passwordless access across multiple nodes
2. Generating and exchanging public keys for secure communication
3. Configuring Axolotl with shared settings and host files
4. Configuring Accelerate for multi-node training with Deepspeed
5. Running distributed finetuning using Accelerate
Create Distributed_Finetuning_Multi-Node.md
@NanoCode012
Copy link
Collaborator

NanoCode012 commented Apr 18, 2024

Thanks for sharing. I think we converted to a qmd. Also, we have an existing multi-node doc. Maybe you could edit that one instead?

The previous one uses fsdp whereas yours use deepspeed, so you can create a heading for fsdp which is the current doc, then one for deepspeed (or you can combine them to make it clearer).

@shahdivax
Copy link
Author

Thanks for sharing. I think we converted to a qmd. Also, we have an existing multi-node doc. Maybe you could edit that one instead?

The previous one uses fsdp whereas yours use deepspeed, so you can create a heading for fsdp which is the current doc, then one for deepspeed (or you can combine them to make it clearer).

Sure, will edit the existing one. As you mentioned.

@shahdivax
Copy link
Author

Thanks for sharing. I think we converted to a qmd. Also, we have an existing multi-node doc. Maybe you could edit that one instead?

The previous one uses fsdp whereas yours use deepspeed, so you can create a heading for fsdp which is the current doc, then one for deepspeed (or you can combine them to make it clearer).

Hello, I have created another pull request for the same #1688
please review it thanks

@shahdivax shahdivax closed this Jun 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants