LIDAR Example code

[Rajivrocks-Ltd]

Hello, I was wondering how we could utilize the LIDAR RTGym environment in code. The example code in the competition folder only shows how we can interact and use the raw images. But we assume LIDAR is much quicker in training? So we'd like to use the LIDAR as well and compare performance. Is there some example code to show how to use this LIDAR environment in code for agent training?

Also, a ssecond part. I am saving my model by setting the parameter in the config "SAVE_MODEL_EVERY" to number I want. Now I have a .tcpt file and .tmod but I don't see 2 things

1. How do I restart training with these files?
2. How do I perform inference using the weights? I use the command py -m tmrl --test. Here I set the run name to the name of the weights file.

But doing this gives the following error:
Traceback (most recent call last):
File "", line 198, in run_module_as_main
File "", line 88, in run_code
File "D:\Project\MGAIA\MGAIA_A3\venv\Lib\site-packages\tmrl_main.py", line 88, in
main(arguments)
File "D:\Project\MGAIA\MGAIA_A3\venv\Lib\site-packages\tmrl_main.py", line 26, in main
rw = RolloutWorker(env_cls=partial(GenericGymEnv, id=cfg.RTGYM_VERSION, gym_kwargs={"config": config}),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Project\MGAIA\MGAIA_A3\venv\Lib\site-packages\tmrl\networking.py", line 504, in init
self.actor = self.actor.load(self.model_path, device=self.device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Project\MGAIA\MGAIA_A3\venv\Lib\site-packages\tmrl\actor.py", line 137, in load
self.load_state_dict(torch.load(path, map_location=self.device))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Project\MGAIA\MGAIA_A3\venv\Lib\site-packages\torch\serialization.py", line 1040, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Project\MGAIA\MGAIA_A3\venv\Lib\site-packages\torch\serialization.py", line 1262, in _legacy_load
magic_number = pickle_module.load(f, **pickle_load_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_pickle.UnpicklingError: invalid load key, '{'.

I saw there was a pickled reward function from the example, do I need that as well? if so, how do I save that to use it with my weights to perform inference with later.

Thanks in advance for any advice!

[yannbouteiller]

Hi,

If you set the "ENV" entry in config.json to a LIDAR environment as described in the readme, the default pipeline that is launched with tmrl --worker and tmrl --trainer will use it. It will also be the environment returned by the get_environment() method. If you want to use this environment in your own tmrl training pipeline, you can retrieve the corresponding rtgym configuration dictionary with from tmrl.config.config_objects import CONFIG_DICT (you'll need that because the RolloutWorker class expects the environment class as described in the tutorials, not an environment instance I believe)

If you want to code your own LIDAR environment, the code for the default LIDAR environment is here.

If you want to test one of the models saved in the history, you need to simply rename this model (a .tmod file) by removing the _number part so that the name matches the name of the normal model (the one that is always saved regardless of whether you are using the history-saving option). Then tmrl --test should do what you want.

[yannbouteiller]

Now that I'm thinking about it, it would make sense to give the option to pass an environment instance rather than the environment class to the RolloutWorker class. It should be easy to implement: we just need to add a __call__ method that returns self to the GenericGymEnv class I believe (unless Gymnasium environments have a __call__ method already).

[Rajivrocks-Ltd]

Hi, thanks for the reply! Could you elaborate more on the second part, using your own trained model? What I do now is use the .tmod file in the TmrlData/weights folder that is saved (as in my own model, not the pretrained model that is provided with TMRL). I rename the "RUN_NAME" to my own trained model name (for example, it's called "test_123.tmod", so the "RUN_NAME" will be "test_123" in the config.json file) and run the command you suggested "py -m tmrl --test" but this will give the above stacktrace back.

Also, how can I continue the training process with the .tcpt file in the TmrlData/Checkpoint folder?

Thanks again for your reply

[yannbouteiller]

Just leave your run name as it is and instead rename the weights files.

For the checkpoint, training resumes automatically from there unless you change the run name.

(In config.json there is also an option to reset training and only keep the replay buffer from the checkpoint, it is disabled by default I believe)

[Rajivrocks-Ltd]

Okay, sorry I am still confused at the moment. So if my run name is 'test_123' my weight file (.tmod) file should also be 'test_123'? Because this is already the case.

Also, I think the issue is that when I look at the content of the .tmod file of mine, it is in json format. But when I open the SAC_4_imgs_pretrained.tmod it looks like hex values. But my own .tmod file, which is being saved in this /weights folder is, like I said, in .json format. Is it not being saved in the correct format? I am using the example code in the 'tmrl/tuto/competition' directory

EDIT: Okay so I uncommented the standard torch.save method that was commented before. This might have been the problem before. I'll update this post when I test it

[yannbouteiller]

Oh I see, sorry I didn't get that!

The competition script saves stuff in .json format only because we don't want people to submit pickle files to the competition for security reasons.

If you are training something locally, you can use the default serializer instead, it should solve your issue.

Your issue comes from the fact that tmrl --test uses the default (pickle) deserializer. If you want to test your trained policy serialized in .json, you can use the competition evaluation script

[Rajivrocks-Ltd]

Ooooh okay, thanks a bunch Yann! I figured it out and it works now!

[yannbouteiller]

Also a note for the future, we might like to change this json format for safetensor in the competition script, as this has kind of become the standard on huggingface.