Unstable critic training

[atagle123]

Hi, great work! I'm trying to reproduce results on Antmaze and MuJoCo, but the critic training becomes unstable (loss explodes) in certain domains. Could you provide the batch size hyperparameter for each experiment to help avoid training instabilities,? Do you use cossine annealing in the critic training? I really appreciate it. Thank you!