Add adabelief to the readme. (#210)

jettify · Oct 20, 2020 · efeea8f · efeea8f
1 parent 9c72aa0
commit efeea8f
Show file tree

Hide file tree

Showing 4 changed files with 36 additions and 0 deletions.
diff --git a/README.rst b/README.rst
@@ -60,6 +60,9 @@ Supported Optimizers
 | `AccSGD`_   | https://arxiv.org/abs/1803.05591                                              |
 +-------------+-------------------------------------------------------------------------------+
 |             |                                                                               |
+| `AdaBelief`_| https://arxiv.org/abs/2010.07468                                              |
++-------------+-------------------------------------------------------------------------------+
+|             |                                                                               |
 | `AdaBound`_ | https://arxiv.org/abs/1902.09843                                              |
 +-------------+-------------------------------------------------------------------------------+
 |             |                                                                               |
@@ -261,6 +264,38 @@ AccSGD
 
 **Reference Code**: https://github.com/rahulkidambi/AccSGD
 
+
+AdaBelief
+---------
+
++-------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
+| .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rastrigin_AdaBelief.png  |  .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rosenbrock_AdaBelief.png |
++-------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
+
+.. code:: python
+
+    import torch_optimizer as optim
+
+    # model = ...
+    optimizer = optim.AdaBelief(
+        m.parameters(),
+        lr= 1e-3,
+        betas=(0.9, 0.999),
+        eps=1e-3,
+        weight_decay=0,
+        amsgrad=False,
+        weight_decouple=False,
+        fixed_decay=False,
+        rectify=False,
+    )
+    optimizer.step()
+
+
+**Paper**: *AdaBelief Optimizer, adapting stepsizes by the belief in observed gradients* (2020) [https://arxiv.org/abs/2010.07468]
+
+**Reference Code**: https://github.com/juntang-zhuang/Adabelief-Optimizer
+
+
 AdaBound
 --------
 

diff --git a/docs/rastrigin_AdaBelief.png b/docs/rastrigin_AdaBelief.png
diff --git a/docs/rosenbrock_AdaBelief.png b/docs/rosenbrock_AdaBelief.png
diff --git a/examples/viz_optimizers.py b/examples/viz_optimizers.py
@@ -191,6 +191,7 @@ def LookaheadYogi(*a, **kw):
         (optim.A2GradUni, -8, 0.1),
         (optim.A2GradInc, -8, 0.1),
         (optim.A2GradExp, -8, 0.1),
+        (optim.AdaBelief, -8, 0.1),
     ]
     execute_experiments(
         optimizers,