Add New Optimizer

[SoheilFM]

Feature request

I want to add a new optimizer that I worked on it
Is there any doc or anything how to add this new optimizer to all files of transformers library?

Motivation

This optimizer would be great to implement on transformers and reduce time to train a new model

Your contribution

I can make a clone of this repo and then changes file and submit PR. But I can't find a document how to implement my algorithm to transformers Files because I found so much classes and there are a lots of file that need changes.

[deven367]

Hi @SoheilFM, you can follow the link here → https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md

[SoheilFM]

Hi @SoheilFM, you can follow the link here → https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md

I read that, but I couldn't find any information about how optimizers are implemented.

[qubvel]

Hi @SoheilFM, thanks for your initiative to add a new optimizer!

It would be cool to have some descriptions, benchmarks, and links to your optimizer, to understand better the motivation regarding adding it. Is there any paper published with benchmarks? Is there any working implementation with pip package?

Later on, if the feature is approved, you could probably take a look at recent PRs regarding adding optimizers (in case your optimizer has a pip package already):

• FEAT / Optim: Add QGaLore optimizer #31936
• FEAT / Optim: Add GaLore optimizer #29588

[SoheilFM]

Hi @SoheilFM, thanks for your initiative to add a new optimizer!

It would be cool to have some descriptions, benchmarks, and links to your optimizer, to understand better the motivation regarding adding it. Is there any paper published with benchmarks? Is there any working implementation with pip package?

Later on, if the feature is approved, you could probably take a look at recent PRs regarding adding optimizers (in case your optimizer has a pip package already):

• FEAT / Optim: Add QGaLore optimizer #31936
• FEAT / Optim: Add GaLore optimizer #29588

Hello Dear @qubvel,

Thank you for your reply.

We are currently working on the optimizer mentioned in our research paper, but unfortunately, I cannot share it publicly at this time. The general concept is similar to the optimizer they added to transformers, which they named LION.

This algorithm is written in pure Python and has not been implemented in pip or any package management system. It has been tested on unimodal benchmarks such as Sphere, Rastrigin, Rosenbrock, and Ackley, showing promising results.

Our next step is to implement this algorithm in the transformers library to train some models and evaluate benchmarks to determine its effectiveness as an optimizer.

I have read the two PRs you mentioned, but they require numerous file changes and include functions and parameters that I do not fully understand. Different developers have different approaches, possibly based on their use cases.

Could you please let me know if there are any specific issues or documentation that would help us implement a new optimizer in the transformers library, or should we rely on our knowledge and navigate through the various modules and code?

Best regards,

[SoheilFM]

Any News?

[qubvel]

Hi @SoheilFM,

If you just need to conduct experiments with your custom optimizer, you don't need to integrate it into transofrmers library, Trainer has option to provide an external optimizer and lr scheduler for training, see here:

• https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Trainer.optimizers

optimizer = MyOptimizer(model.parameters(), ...)

trainer = Trainer(
   ...
   optimizers=(optimizer, scheduler),
   ...
)
trainer.train()
...

Let me know if that's something you are looking for.