Week 23 - Semantic Segmentation
Once we have established a solid base for the deep learning project we move on to other tasks. With that being said, we are going to make use of a segmentation sensor already implemented in the Carla Simulator. With this sensor we can segmentate an image like the one we showed the previous week. Starting the training process we needed to record a new dataset but with segmentated images, so we proceeded to extract a dataset similar to the one we had on the previous weeks were we will make the car run clockwise and anticlockwise in three different circuits, and then we gather more data, but this time focused on the turns, for the straight instances are much more larger and we need a balanced and rich dataset.
While we were recolecting data we encounter a little problem exceptionally present in the Town05.
As we can see from the previous video, we can observe that there is a black flickering on the images from using a hardcoded autopilot (from Carla). The reason to highlight this last part, is that this flickering can be seen when we use a neural network, or when we overload the code with tasks that slow the segmentation process. But this doens’t happens in other towns as frequently as it happens in the town05. Until we sort this problem out, we decided to change the training maps, and instead of using the towns: 02, 03 and 05, we are going to use the 02, 03 and 07, where we can also try different and more complex turns.
Now, to balance our dataset we do the same thing we have been doing this past weeks, in order to have a model that trains equally the left and right turns without overfitting.
Once the data is set, we train the model, trying to check how far can we train it and if it is even worth it. From the looks of it, we can see some improvements along the 240 epochs that we make it train for.
Now, for the results we can easily see that the car has grown to be more robust, being able to keep itself in the lane in more situations given the fact that by having segmentated images, we can simplify the features learned and therefore making it more robust to different types of roads.
In the first video we take a look at how the car performs on a training town such as 07, where the curves are quite unique.
The next video shows us a run on town 01, similar to the town 02 with some differences such as the presence of puddles and different lighting, where as we can see, the segmentation is pretty much the same as the default weather and lighting settings.
In the last video, we can see more oscilation with the car, trying to maintain itself inside its lane and picking up a lot of speed. Given that the speed is something we are not focusing on right now, we try to understand why it is oscillating, and if we go back for just a moment, when we saw the histogram of the dataset we can appreciate a curious case. The left hard steering is much more represented that the right hard steering values. We could be wrong, but given that the car tendencies grow bigger when it approaches the left lane, we could assume that this could be a possible explanation. In maps like town 04, when we put the car on bigger roads like one with 4 lanes, the lines aren’t too close to each other, giving a space for the car to veer to the left being able to change lanes when this is not the required behaviour.
To further explore into our project, we are going to try to focus on two things:
- Balance better the dataset for an improved behaviour
- Introduce more agents other than our car. We will try to teach the car to brake when it has a car running in front of it.