[Question]How to Maximize Step Execution Speed in Reinforcement Learning?

[ItsBean]

I found that sim_freq can improve the speed of env.step. When I set it to 1, the game runs faster and normally, but when I set it to 0.01, my agent can no longer respond correctly. Typically, the agent can only interact for 2 steps before the game truncates when both policy_freq and sim_freq are set to 0.01. So, I have two questions:

• How can I maximize the interaction speed?
• What should the values for sim_freq and policy_freq be?

Here is my test code:

# here is the code for init.
    simulation_freq = 0.5
    action_config = {
        "action": {"type": "ContinuousAction"},
        "simulation_frequency": simulation_freq,  # Example frequency in Hz
        "duration": int(100),  # Example duration in seconds
        "policy_frequency": 0.5,  # Number of steps between policy updates
    }
    print('action_config', action_config)

    env = gym.make('highway-v0', config=action_config)

    # here is the sample code:
    for t in range(int(args.max_timesteps)):

        episode_timesteps += 1

        # Select action randomly or according to policy
        if t < args.start_timesteps:
            action = env.action_space.sample()
        else:
            action = (
                    policy.select_action(np.array(state))
                    + np.random.normal(0, max_action * args.expl_noise, size=action_dim)
            ).clip(-max_action, max_action)

        # Perform action
        next_state, reward, terminated, truncated, _ = env.step(action)
        next_state = next_state.flatten()
        done = terminated or truncated
        # truncated will be assigned with zero
        done_bool = float(done) if not truncated else 0