Hyperparameters for reproducing results in paper

[edwardjjj]

Hi, Thank you very much for sharing your dataset. We tried to reproduce the BC training on lamp_low and only achieved ~3% success rate. I used the following arguments:

python -m src.train.bc +experiment=state/diff_unet \
    task=lamp \
    rollout=rollout rollout.every=100 rollout.max_steps=1000 rollout.num_envs=512 \
    rollout.randomness=low \
    pred_horizon=32 action_horizon=8 obs_horizon=1 control.controller=diffik \
    demo_source=teleop randomness='[low,low_perturb]' \
    training.batch_size=128 training.actor_lr=1e-4 training.num_epochs=1000 \
    training.steps_per_epoch=1000 \
    wandb.project=ol-state-low \
    dryrun=false

what parameters should we adjust to achieve 7% success rate as listed in paper, or it's just a margin of error that we are experiencing?

[ankile]

Hi, thanks for reaching out!

Unfortunately, we and many others find that imitation learning training is not very predictable or monotonous. Most of the time, one has to save and evaluate a series of checkpoints throughout training to find the best one—this varies from run to run.

That said, I don't see any obvious differences between your HPs there and the ones I used, except for the batch size of 128, where I typically used 256 (or higher) for state-based training.

So, I'd suggest you test slightly larger batch sizes and a couple more seeds and evaluate more checkpoints throughout training. I'm also in the process of releasing the weights for the models I trained.

I hope this helps!

Best, Lars