[Update] Optimizer visualization improved (#345)

* update: visualize_optimizers * Update visualize_optimizers.py * Update visualize_optimizers.py * Code refactor * Code format --------- Co-authored-by: Aydin <[email protected]>
kozistr · Feb 14, 2025 · 5f4e62f · 5f4e62f
1 parent 0386506
commit 5f4e62f
Show file tree

Hide file tree

Showing 183 changed files with 357 additions and 74 deletions.
diff --git a/docs/visualizations/rastrigin_ADOPT.png b/docs/visualizations/rastrigin_ADOPT.png
diff --git a/docs/visualizations/rastrigin_APOLLO.png b/docs/visualizations/rastrigin_APOLLO.png
diff --git a/docs/visualizations/rastrigin_ASGD.png b/docs/visualizations/rastrigin_ASGD.png
diff --git a/docs/visualizations/rastrigin_AccSGD.png b/docs/visualizations/rastrigin_AccSGD.png
diff --git a/docs/visualizations/rastrigin_AdEMAMix.png b/docs/visualizations/rastrigin_AdEMAMix.png
diff --git a/docs/visualizations/rastrigin_AdaBelief.png b/docs/visualizations/rastrigin_AdaBelief.png
diff --git a/docs/visualizations/rastrigin_AdaBound.png b/docs/visualizations/rastrigin_AdaBound.png
diff --git a/docs/visualizations/rastrigin_AdaDelta.png b/docs/visualizations/rastrigin_AdaDelta.png
diff --git a/docs/visualizations/rastrigin_AdaFactor.png b/docs/visualizations/rastrigin_AdaFactor.png
diff --git a/docs/visualizations/rastrigin_AdaHessian.png b/docs/visualizations/rastrigin_AdaHessian.png
diff --git a/docs/visualizations/rastrigin_AdaMax.png b/docs/visualizations/rastrigin_AdaMax.png
diff --git a/docs/visualizations/rastrigin_AdaMod.png b/docs/visualizations/rastrigin_AdaMod.png
diff --git a/docs/visualizations/rastrigin_AdaNorm.png b/docs/visualizations/rastrigin_AdaNorm.png
diff --git a/docs/visualizations/rastrigin_AdaPNM.png b/docs/visualizations/rastrigin_AdaPNM.png
diff --git a/docs/visualizations/rastrigin_AdaShift.png b/docs/visualizations/rastrigin_AdaShift.png
diff --git a/docs/visualizations/rastrigin_AdaSmooth.png b/docs/visualizations/rastrigin_AdaSmooth.png
diff --git a/docs/visualizations/rastrigin_AdaTAM.png b/docs/visualizations/rastrigin_AdaTAM.png
diff --git a/docs/visualizations/rastrigin_Adai.png b/docs/visualizations/rastrigin_Adai.png
diff --git a/docs/visualizations/rastrigin_Adalite.png b/docs/visualizations/rastrigin_Adalite.png
diff --git a/docs/visualizations/rastrigin_Adam.png b/docs/visualizations/rastrigin_Adam.png
diff --git a/docs/visualizations/rastrigin_AdamG.png b/docs/visualizations/rastrigin_AdamG.png
diff --git a/docs/visualizations/rastrigin_AdamMini.png b/docs/visualizations/rastrigin_AdamMini.png
diff --git a/docs/visualizations/rastrigin_AdamP.png b/docs/visualizations/rastrigin_AdamP.png
diff --git a/docs/visualizations/rastrigin_AdamS.png b/docs/visualizations/rastrigin_AdamS.png
diff --git a/docs/visualizations/rastrigin_AdamW.png b/docs/visualizations/rastrigin_AdamW.png
diff --git a/docs/visualizations/rastrigin_Adan.png b/docs/visualizations/rastrigin_Adan.png
diff --git a/docs/visualizations/rastrigin_AggMo.png b/docs/visualizations/rastrigin_AggMo.png
diff --git a/docs/visualizations/rastrigin_Aida.png b/docs/visualizations/rastrigin_Aida.png
diff --git a/docs/visualizations/rastrigin_AliG.png b/docs/visualizations/rastrigin_AliG.png
diff --git a/docs/visualizations/rastrigin_Amos.png b/docs/visualizations/rastrigin_Amos.png
diff --git a/docs/visualizations/rastrigin_ApolloDQN.png b/docs/visualizations/rastrigin_ApolloDQN.png
diff --git a/docs/visualizations/rastrigin_AvaGrad.png b/docs/visualizations/rastrigin_AvaGrad.png
diff --git a/docs/visualizations/rastrigin_BSAM.png b/docs/visualizations/rastrigin_BSAM.png
diff --git a/docs/visualizations/rastrigin_CAME.png b/docs/visualizations/rastrigin_CAME.png
diff --git a/docs/visualizations/rastrigin_DAdaptAdaGrad.png b/docs/visualizations/rastrigin_DAdaptAdaGrad.png
diff --git a/docs/visualizations/rastrigin_DAdaptAdam.png b/docs/visualizations/rastrigin_DAdaptAdam.png
diff --git a/docs/visualizations/rastrigin_DAdaptAdan.png b/docs/visualizations/rastrigin_DAdaptAdan.png
diff --git a/docs/visualizations/rastrigin_DAdaptLion.png b/docs/visualizations/rastrigin_DAdaptLion.png
diff --git a/docs/visualizations/rastrigin_DAdaptSGD.png b/docs/visualizations/rastrigin_DAdaptSGD.png
diff --git a/docs/visualizations/rastrigin_DiffGrad.png b/docs/visualizations/rastrigin_DiffGrad.png
diff --git a/docs/visualizations/rastrigin_EXAdam.png b/docs/visualizations/rastrigin_EXAdam.png
diff --git a/docs/visualizations/rastrigin_FAdam.png b/docs/visualizations/rastrigin_FAdam.png
diff --git a/docs/visualizations/rastrigin_FOCUS.png b/docs/visualizations/rastrigin_FOCUS.png
diff --git a/docs/visualizations/rastrigin_FTRL.png b/docs/visualizations/rastrigin_FTRL.png
diff --git a/docs/visualizations/rastrigin_Fromage.png b/docs/visualizations/rastrigin_Fromage.png
diff --git a/docs/visualizations/rastrigin_GaLore.png b/docs/visualizations/rastrigin_GaLore.png
diff --git a/docs/visualizations/rastrigin_Grams.png b/docs/visualizations/rastrigin_Grams.png
diff --git a/docs/visualizations/rastrigin_Gravity.png b/docs/visualizations/rastrigin_Gravity.png
diff --git a/docs/visualizations/rastrigin_GrokFastAdamW.png b/docs/visualizations/rastrigin_GrokFastAdamW.png
diff --git a/docs/visualizations/rastrigin_Kate.png b/docs/visualizations/rastrigin_Kate.png
diff --git a/docs/visualizations/rastrigin_Kron.png b/docs/visualizations/rastrigin_Kron.png
diff --git a/docs/visualizations/rastrigin_LARS.png b/docs/visualizations/rastrigin_LARS.png
diff --git a/docs/visualizations/rastrigin_LaProp.png b/docs/visualizations/rastrigin_LaProp.png
diff --git a/docs/visualizations/rastrigin_Lamb.png b/docs/visualizations/rastrigin_Lamb.png
diff --git a/docs/visualizations/rastrigin_Lion.png b/docs/visualizations/rastrigin_Lion.png
diff --git a/docs/visualizations/rastrigin_MADGRAD.png b/docs/visualizations/rastrigin_MADGRAD.png
diff --git a/docs/visualizations/rastrigin_MARS.png b/docs/visualizations/rastrigin_MARS.png
diff --git a/docs/visualizations/rastrigin_MSVAG.png b/docs/visualizations/rastrigin_MSVAG.png
diff --git a/docs/visualizations/rastrigin_Muon.png b/docs/visualizations/rastrigin_Muon.png
diff --git a/docs/visualizations/rastrigin_Nero.png b/docs/visualizations/rastrigin_Nero.png
diff --git a/docs/visualizations/rastrigin_NovoGrad.png b/docs/visualizations/rastrigin_NovoGrad.png
diff --git a/docs/visualizations/rastrigin_PAdam.png b/docs/visualizations/rastrigin_PAdam.png
diff --git a/docs/visualizations/rastrigin_PID.png b/docs/visualizations/rastrigin_PID.png
diff --git a/docs/visualizations/rastrigin_PNM.png b/docs/visualizations/rastrigin_PNM.png
diff --git a/docs/visualizations/rastrigin_Prodigy.png b/docs/visualizations/rastrigin_Prodigy.png
diff --git a/docs/visualizations/rastrigin_QHAdam.png b/docs/visualizations/rastrigin_QHAdam.png
diff --git a/docs/visualizations/rastrigin_QHM.png b/docs/visualizations/rastrigin_QHM.png
diff --git a/docs/visualizations/rastrigin_RAdam.png b/docs/visualizations/rastrigin_RAdam.png
diff --git a/docs/visualizations/rastrigin_Ranger.png b/docs/visualizations/rastrigin_Ranger.png
diff --git a/docs/visualizations/rastrigin_Ranger21.png b/docs/visualizations/rastrigin_Ranger21.png
diff --git a/docs/visualizations/rastrigin_Ranger25.png b/docs/visualizations/rastrigin_Ranger25.png
diff --git a/docs/visualizations/rastrigin_SGD.png b/docs/visualizations/rastrigin_SGD.png
diff --git a/docs/visualizations/rastrigin_SGDP.png b/docs/visualizations/rastrigin_SGDP.png
diff --git a/docs/visualizations/rastrigin_SGDSaI.png b/docs/visualizations/rastrigin_SGDSaI.png
diff --git a/docs/visualizations/rastrigin_SGDW.png b/docs/visualizations/rastrigin_SGDW.png
diff --git a/docs/visualizations/rastrigin_SM3.png b/docs/visualizations/rastrigin_SM3.png
diff --git a/docs/visualizations/rastrigin_SOAP.png b/docs/visualizations/rastrigin_SOAP.png
diff --git a/docs/visualizations/rastrigin_SPAM.png b/docs/visualizations/rastrigin_SPAM.png
diff --git a/docs/visualizations/rastrigin_SRMM.png b/docs/visualizations/rastrigin_SRMM.png
diff --git a/docs/visualizations/rastrigin_SWATS.png b/docs/visualizations/rastrigin_SWATS.png
diff --git a/docs/visualizations/rastrigin_ScalableShampoo.png b/docs/visualizations/rastrigin_ScalableShampoo.png
diff --git a/docs/visualizations/rastrigin_ScheduleFreeAdamW.png b/docs/visualizations/rastrigin_ScheduleFreeAdamW.png
diff --git a/docs/visualizations/rastrigin_ScheduleFreeRAdam.png b/docs/visualizations/rastrigin_ScheduleFreeRAdam.png
diff --git a/docs/visualizations/rastrigin_ScheduleFreeSGD.png b/docs/visualizations/rastrigin_ScheduleFreeSGD.png
diff --git a/docs/visualizations/rastrigin_Shampoo.png b/docs/visualizations/rastrigin_Shampoo.png
diff --git a/docs/visualizations/rastrigin_SignSGD.png b/docs/visualizations/rastrigin_SignSGD.png
diff --git a/docs/visualizations/rastrigin_SophiaH.png b/docs/visualizations/rastrigin_SophiaH.png
diff --git a/docs/visualizations/rastrigin_StableAdamW.png b/docs/visualizations/rastrigin_StableAdamW.png
diff --git a/docs/visualizations/rastrigin_TAM.png b/docs/visualizations/rastrigin_TAM.png
diff --git a/docs/visualizations/rastrigin_Tiger.png b/docs/visualizations/rastrigin_Tiger.png
diff --git a/docs/visualizations/rastrigin_Yogi.png b/docs/visualizations/rastrigin_Yogi.png
diff --git a/docs/visualizations/rosenbrock_ADOPT.png b/docs/visualizations/rosenbrock_ADOPT.png
diff --git a/docs/visualizations/rosenbrock_APOLLO.png b/docs/visualizations/rosenbrock_APOLLO.png
diff --git a/docs/visualizations/rosenbrock_ASGD.png b/docs/visualizations/rosenbrock_ASGD.png
diff --git a/docs/visualizations/rosenbrock_AccSGD.png b/docs/visualizations/rosenbrock_AccSGD.png
diff --git a/docs/visualizations/rosenbrock_AdEMAMix.png b/docs/visualizations/rosenbrock_AdEMAMix.png
diff --git a/docs/visualizations/rosenbrock_AdaBelief.png b/docs/visualizations/rosenbrock_AdaBelief.png
diff --git a/docs/visualizations/rosenbrock_AdaBound.png b/docs/visualizations/rosenbrock_AdaBound.png
diff --git a/docs/visualizations/rosenbrock_AdaDelta.png b/docs/visualizations/rosenbrock_AdaDelta.png
diff --git a/docs/visualizations/rosenbrock_AdaFactor.png b/docs/visualizations/rosenbrock_AdaFactor.png
diff --git a/docs/visualizations/rosenbrock_AdaHessian.png b/docs/visualizations/rosenbrock_AdaHessian.png
diff --git a/docs/visualizations/rosenbrock_AdaMax.png b/docs/visualizations/rosenbrock_AdaMax.png
diff --git a/docs/visualizations/rosenbrock_AdaMod.png b/docs/visualizations/rosenbrock_AdaMod.png
diff --git a/docs/visualizations/rosenbrock_AdaNorm.png b/docs/visualizations/rosenbrock_AdaNorm.png
diff --git a/docs/visualizations/rosenbrock_AdaPNM.png b/docs/visualizations/rosenbrock_AdaPNM.png
diff --git a/docs/visualizations/rosenbrock_AdaShift.png b/docs/visualizations/rosenbrock_AdaShift.png
diff --git a/docs/visualizations/rosenbrock_AdaSmooth.png b/docs/visualizations/rosenbrock_AdaSmooth.png
diff --git a/docs/visualizations/rosenbrock_AdaTAM.png b/docs/visualizations/rosenbrock_AdaTAM.png
diff --git a/docs/visualizations/rosenbrock_Adai.png b/docs/visualizations/rosenbrock_Adai.png
diff --git a/docs/visualizations/rosenbrock_Adalite.png b/docs/visualizations/rosenbrock_Adalite.png
diff --git a/docs/visualizations/rosenbrock_Adam.png b/docs/visualizations/rosenbrock_Adam.png
diff --git a/docs/visualizations/rosenbrock_AdamG.png b/docs/visualizations/rosenbrock_AdamG.png
diff --git a/docs/visualizations/rosenbrock_AdamMini.png b/docs/visualizations/rosenbrock_AdamMini.png
diff --git a/docs/visualizations/rosenbrock_AdamP.png b/docs/visualizations/rosenbrock_AdamP.png
diff --git a/docs/visualizations/rosenbrock_AdamS.png b/docs/visualizations/rosenbrock_AdamS.png
diff --git a/docs/visualizations/rosenbrock_AdamW.png b/docs/visualizations/rosenbrock_AdamW.png
diff --git a/docs/visualizations/rosenbrock_Adan.png b/docs/visualizations/rosenbrock_Adan.png
diff --git a/docs/visualizations/rosenbrock_AggMo.png b/docs/visualizations/rosenbrock_AggMo.png
diff --git a/docs/visualizations/rosenbrock_Aida.png b/docs/visualizations/rosenbrock_Aida.png
diff --git a/docs/visualizations/rosenbrock_AliG.png b/docs/visualizations/rosenbrock_AliG.png
diff --git a/docs/visualizations/rosenbrock_Amos.png b/docs/visualizations/rosenbrock_Amos.png
diff --git a/docs/visualizations/rosenbrock_ApolloDQN.png b/docs/visualizations/rosenbrock_ApolloDQN.png
diff --git a/docs/visualizations/rosenbrock_AvaGrad.png b/docs/visualizations/rosenbrock_AvaGrad.png
diff --git a/docs/visualizations/rosenbrock_BSAM.png b/docs/visualizations/rosenbrock_BSAM.png
diff --git a/docs/visualizations/rosenbrock_CAME.png b/docs/visualizations/rosenbrock_CAME.png
diff --git a/docs/visualizations/rosenbrock_DAdaptAdaGrad.png b/docs/visualizations/rosenbrock_DAdaptAdaGrad.png
diff --git a/docs/visualizations/rosenbrock_DAdaptAdam.png b/docs/visualizations/rosenbrock_DAdaptAdam.png
diff --git a/docs/visualizations/rosenbrock_DAdaptAdan.png b/docs/visualizations/rosenbrock_DAdaptAdan.png
diff --git a/docs/visualizations/rosenbrock_DAdaptLion.png b/docs/visualizations/rosenbrock_DAdaptLion.png
diff --git a/docs/visualizations/rosenbrock_DAdaptSGD.png b/docs/visualizations/rosenbrock_DAdaptSGD.png
diff --git a/docs/visualizations/rosenbrock_DiffGrad.png b/docs/visualizations/rosenbrock_DiffGrad.png
diff --git a/docs/visualizations/rosenbrock_EXAdam.png b/docs/visualizations/rosenbrock_EXAdam.png
diff --git a/docs/visualizations/rosenbrock_FAdam.png b/docs/visualizations/rosenbrock_FAdam.png
diff --git a/docs/visualizations/rosenbrock_FOCUS.png b/docs/visualizations/rosenbrock_FOCUS.png
diff --git a/docs/visualizations/rosenbrock_FTRL.png b/docs/visualizations/rosenbrock_FTRL.png
diff --git a/docs/visualizations/rosenbrock_Fromage.png b/docs/visualizations/rosenbrock_Fromage.png
diff --git a/docs/visualizations/rosenbrock_GaLore.png b/docs/visualizations/rosenbrock_GaLore.png
diff --git a/docs/visualizations/rosenbrock_Grams.png b/docs/visualizations/rosenbrock_Grams.png
diff --git a/docs/visualizations/rosenbrock_Gravity.png b/docs/visualizations/rosenbrock_Gravity.png
diff --git a/docs/visualizations/rosenbrock_GrokFastAdamW.png b/docs/visualizations/rosenbrock_GrokFastAdamW.png
diff --git a/docs/visualizations/rosenbrock_Kate.png b/docs/visualizations/rosenbrock_Kate.png
diff --git a/docs/visualizations/rosenbrock_Kron.png b/docs/visualizations/rosenbrock_Kron.png
diff --git a/docs/visualizations/rosenbrock_LARS.png b/docs/visualizations/rosenbrock_LARS.png
diff --git a/docs/visualizations/rosenbrock_LaProp.png b/docs/visualizations/rosenbrock_LaProp.png
diff --git a/docs/visualizations/rosenbrock_Lamb.png b/docs/visualizations/rosenbrock_Lamb.png
diff --git a/docs/visualizations/rosenbrock_Lion.png b/docs/visualizations/rosenbrock_Lion.png
diff --git a/docs/visualizations/rosenbrock_MADGRAD.png b/docs/visualizations/rosenbrock_MADGRAD.png
diff --git a/docs/visualizations/rosenbrock_MARS.png b/docs/visualizations/rosenbrock_MARS.png
diff --git a/docs/visualizations/rosenbrock_MSVAG.png b/docs/visualizations/rosenbrock_MSVAG.png
diff --git a/docs/visualizations/rosenbrock_Muon.png b/docs/visualizations/rosenbrock_Muon.png
diff --git a/docs/visualizations/rosenbrock_Nero.png b/docs/visualizations/rosenbrock_Nero.png
diff --git a/docs/visualizations/rosenbrock_NovoGrad.png b/docs/visualizations/rosenbrock_NovoGrad.png
diff --git a/docs/visualizations/rosenbrock_PAdam.png b/docs/visualizations/rosenbrock_PAdam.png
diff --git a/docs/visualizations/rosenbrock_PID.png b/docs/visualizations/rosenbrock_PID.png
diff --git a/docs/visualizations/rosenbrock_PNM.png b/docs/visualizations/rosenbrock_PNM.png
diff --git a/docs/visualizations/rosenbrock_Prodigy.png b/docs/visualizations/rosenbrock_Prodigy.png
diff --git a/docs/visualizations/rosenbrock_QHAdam.png b/docs/visualizations/rosenbrock_QHAdam.png
diff --git a/docs/visualizations/rosenbrock_QHM.png b/docs/visualizations/rosenbrock_QHM.png
diff --git a/docs/visualizations/rosenbrock_RAdam.png b/docs/visualizations/rosenbrock_RAdam.png
diff --git a/docs/visualizations/rosenbrock_Ranger.png b/docs/visualizations/rosenbrock_Ranger.png
diff --git a/docs/visualizations/rosenbrock_Ranger21.png b/docs/visualizations/rosenbrock_Ranger21.png
diff --git a/docs/visualizations/rosenbrock_Ranger25.png b/docs/visualizations/rosenbrock_Ranger25.png
diff --git a/docs/visualizations/rosenbrock_SGD.png b/docs/visualizations/rosenbrock_SGD.png
diff --git a/docs/visualizations/rosenbrock_SGDP.png b/docs/visualizations/rosenbrock_SGDP.png
diff --git a/docs/visualizations/rosenbrock_SGDSaI.png b/docs/visualizations/rosenbrock_SGDSaI.png
diff --git a/docs/visualizations/rosenbrock_SGDW.png b/docs/visualizations/rosenbrock_SGDW.png
diff --git a/docs/visualizations/rosenbrock_SM3.png b/docs/visualizations/rosenbrock_SM3.png
diff --git a/docs/visualizations/rosenbrock_SOAP.png b/docs/visualizations/rosenbrock_SOAP.png
diff --git a/docs/visualizations/rosenbrock_SPAM.png b/docs/visualizations/rosenbrock_SPAM.png
diff --git a/docs/visualizations/rosenbrock_SRMM.png b/docs/visualizations/rosenbrock_SRMM.png
diff --git a/docs/visualizations/rosenbrock_SWATS.png b/docs/visualizations/rosenbrock_SWATS.png
diff --git a/docs/visualizations/rosenbrock_ScalableShampoo.png b/docs/visualizations/rosenbrock_ScalableShampoo.png
diff --git a/docs/visualizations/rosenbrock_ScheduleFreeAdamW.png b/docs/visualizations/rosenbrock_ScheduleFreeAdamW.png
diff --git a/docs/visualizations/rosenbrock_ScheduleFreeRAdam.png b/docs/visualizations/rosenbrock_ScheduleFreeRAdam.png
diff --git a/docs/visualizations/rosenbrock_ScheduleFreeSGD.png b/docs/visualizations/rosenbrock_ScheduleFreeSGD.png
diff --git a/docs/visualizations/rosenbrock_Shampoo.png b/docs/visualizations/rosenbrock_Shampoo.png
diff --git a/docs/visualizations/rosenbrock_SignSGD.png b/docs/visualizations/rosenbrock_SignSGD.png
diff --git a/docs/visualizations/rosenbrock_SophiaH.png b/docs/visualizations/rosenbrock_SophiaH.png
diff --git a/docs/visualizations/rosenbrock_StableAdamW.png b/docs/visualizations/rosenbrock_StableAdamW.png
diff --git a/docs/visualizations/rosenbrock_TAM.png b/docs/visualizations/rosenbrock_TAM.png
diff --git a/docs/visualizations/rosenbrock_Tiger.png b/docs/visualizations/rosenbrock_Tiger.png
diff --git a/docs/visualizations/rosenbrock_Yogi.png b/docs/visualizations/rosenbrock_Yogi.png
diff --git a/examples/visualize_optimizers.py b/examples/visualize_optimizers.py