The data format

[kargarisaac]

What is the sequence of 6 input matrices in the field of 'data' in the dataset? I mean the inputs and the shape of them. The size of the input in every batch is something like [64, 6, 144, 192] and when I plot it, there 6 different inputs.
Did you use a grayscale image for roadmap? I cannot differentiate between the second and third frames.
It would be great if you can explain a little about 'future_poses_regr_offset' and 'future_penalty_maps' in the dataset.

Thank you

[Iftimie]

Hi,
I tried to replicate as in figure 1 of the paper. At the following line I assigned the channels for the input tensor.
First 3 channels are the BGR channels (from opencv drawing) of only lanes and traffic lights. Lanes are usually drawn gray, but the traffic lights are coloured red, yellow, green, depending on their state.
The fourth channel is the image with the vehicle. That is always a simple rectangle because all rendering is done with respect to position and orientation of the vehicle.
The fifth channel represents the GPS-like route(that fat line)
The last channel represents the previous locations of the vehicle ( with a skip of about 5 frames or more, otherwise it would look like a continous line instead of those tiny gray dots)

ChauffeurNet/network/models/Dataset.py

Line 203 in deddf29

image_concatenated = np.empty((6, Config.r_res[0], Config.r_res[1]), np.uint8)

About 'future_poses_regr_offset' and 'future_penalty_maps':
The waypoints are predicted as a spatial argmax after softmax. If we predict 5 waypoints we would need 5 pairs of x y coordinates of ground truth in order to apply a spatial cross entropy loss. However I believed that if the network predicted x+1, y, it should not be penalized too much, thus I needed a penalty map where around the ground truth x y coordinate, would be a gaussian zone of reduced penalty.
I got inspired from CornerNet paper equation (1) and equation (2) https://arxiv.org/pdf/1808.01244.pdf

Since the network is doing the spatial softmax at a reduced resolution, say 3x smaller, then the prediction would be coarse. To improve the prediction. For each waypoint, the network learns to predict an offset for both x and y from the argmax location. Thus each prediction of a waypoint consists of 3 feature maps. For one of them is applied a spatial softmax where we get x y. Another is the regression offset for the selected x and another one is the regression offset for the selected y. At train time the loss is computed only for the ground truth position.

This is optional but since you were looking over the code, it might be of interest.
At the following line I tried to replicate the trajectory perturbation (figure 5 in the paper)
To do this, I decided to select the range of vehicle coordinates points from N previous frames and N future frames and altered them with to look like a bell. This helped the network recover from bad positions.

ChauffeurNet/simulator/util/Path.py

Line 99 in deddf29

def apply_dropout(self, path_idx, vehicle):

The function is called in

ChauffeurNet/network/models/Dataset.py

Line 132 in deddf29

self.path.apply_dropout(idx, self.vehicle)

Hope my explanation helped a bit

[kargarisaac]

Thank you for your answer. I have one more question. In the pkl dataset, I see one speed and one steering for one trajectory. I mean there are several points in "future_penalty_maps" and only one speed and steering.

For example, for the above data sample, I think the speed and steering data is not correct. Am I right?

Thank you again for your help

[Iftimie]

Hi!

Yes you are right.. There are several points predicted using a RNN, however for the speed and steering predicition I did not attached them to the RNN.
It's true that in the paper they predicted 3 outputs using the RNN (waypoints, regression offsets and speed for the all k iterations, prediction horizon). In my case I wasn't so sure how to connect the speed prediction to the RNN, thus I kept it separate in a simple MLP.

ChauffeurNet/network/models/SimpleConv.py

Line 62 in deddf29

class SteeringPredictor(nn.Module):

ChauffeurNet/network/models/SimpleConv.py

Line 183 in deddf29

nn_outputs["steering"] = self.steering_predictor(features)

And even if I was training the network with speed prediction, I did not incorporated into the vehicle control at test time, because I had no idea how. I do not know much about control theory. I calculated the speed solely based on the waypoints using some heuristics.
Here, the waypoints are predicted, converted from 2D to 3D coordinates (ray casting from camera onto the ground plane), then passed to the vehicle.

ChauffeurNet/simulator/control/car_controller/NeuralController.py

Line 48 in deddf29

self.vehicle.simulate_given_waypoints(waypoints_3D)

I calculated the speed in the following way: I calculated the average distance between every waypoint. I observed that near an intersection they would be closer togheter, while on a simple straight road they would be far stretched.
Then, I interpolated the average distance from 0 and Config.num_skip_poses * Config.max_speed (which says how far can be two waypoints in 3D) and [0, Config.max_speed].

ChauffeurNet/simulator/util/Vehicle.py

Line 299 in deddf29

interp_obj = interp1d([0, Config.num_skip_poses * Config.max_speed], [0, Config.max_speed], fill_value="extrapolate")

For the steering angle I selected two predicted waypoints, I took their coordinates on the ground plane (x, and z) and calculated the required angle that would bring the car that coordinate.

ChauffeurNet/simulator/util/Vehicle.py

Line 310 in deddf29

x,z =waypoints[0][Config.test_waypoint_idx_steer],waypoints[2][Config.test_waypoint_idx_steer]

If you have any more questions, feel free to ask them. I am happy that someone took that much interest into this project.

[kargarisaac]

Thank you for your answer. I have to read the code for simulator too. For now, I focused on the dataset you provided.
Do you think I can use this simulator to train a reinforcement learning policy? for example to follow the planned route.

[Iftimie]

I am not very familiar with reinforcement learning. It could be doable. The simulator should be simple enough in order to have access to the entire state of the world.
For example I crated 3 controllers: BagEventController is just for replaying what you recorded (what you have in the .pkl file), Live controller is for the user to play with the keys and mouse and NeuralController is used when you test the model.
I guess that another class could be added for reinforcement learning.
https://github.com/Iftimie/ChauffeurNet/tree/master/simulator/control/car_controller