Skip to content

Commit

Permalink
improve gradient accumulation (rasbt#300)
Browse files Browse the repository at this point in the history
  • Loading branch information
rasbt authored Aug 5, 2024
1 parent 36fbc7a commit 70e5714
Showing 1 changed file with 2 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -259,7 +259,8 @@ def train_classifier_simple(model, train_loader, val_loader, optimizer, device,
loss.backward() # Calculate loss gradients

# Use gradient accumulation if accumulation_steps > 1
if batch_idx % accumulation_steps == 0:
is_update_step = ((batch_idx + 1) % accumulation_steps == 0) or ((batch_idx + 1) == len(train_loader))
if is_update_step:
optimizer.step() # Update model weights using loss gradients
optimizer.zero_grad() # Reset loss gradients from previous batch iteration

Expand Down

0 comments on commit 70e5714

Please sign in to comment.