v3.3.4.md

Change Log

Feature

• Support OrthoGrad feature for create_optimizer(). (#324)
• Enhanced flexibility for the optimizer parameter in Lookahead, TRAC, and OrthoGrad optimizers. (#324)
  • Now supports both torch.optim.Optimizer instances and classes
  • You can now use Lookahead optimizer in two ways.
    • Lookahead(AdamW(model.parameters(), lr=1e-3), k=5, alpha=0.5)
    • Lookahead(AdamW, k=5, alpha=0.5, params=model.parameters())
• Implement SPAM optimizer. (#324)
  • Spike-Aware Adam with Momentum Reset for Stable LLM Training
• Implement TAM, and AdaTAM optimizers. (#325)
  • Torque-Aware Momentum