Juan Simó Álvarez

Robotics Software Engineer

5 minute read

Progress This Week

Objective

The objective was to create subsets for every dataset, one for the data of curves and another one for the data of lines. And with those new sets train 2 expert pilots that would be coordinated the same way the images where divided.

Classifier

Creating the subdatasets should be done by hand, however, this would take an eternity. Another way would be just taking into account the angular speed, however, that would be using the expert pilot, not a real classifier.

I tried various things with different degrees of success. One of the best was dividing the image into nine quadrants and depending on the distribution of the red line on each of them classifying the image as a line or curve. However, there were too many cases and exceptions and I ended up implementing a simpler program. This one picked the center of the red pixels of the bottom line (if there is no bottom line we assume it’s a curve) and create an imaginary column from there. Afterwards, looking only at the top rows for agility reasons, it would look for any red pixels outside a specified margin apart from the column. ..red....|margin|margin|..... (not sure if that helps, I’ll create a diagram).

With this and adjusting the parameters of how many and which top rows it looked and how wide was the margin I got different subsets, I kept adjusting because it always learned to take straight lines, which I don’t like because I want it to only be able to take curves and to oscillate in straight lines.

Line expert

The line expert as expected just knows how to step on the gas, but if left alone it would only crash in the first curve it found as can be seen here: TODO

Curve expert

This expert should be able to complete the whole circuit itself, but slower than paired with the line expert. In general, it should be similar to our neural pilot but worse on straight lines. However, even if it did not accelerate much on straight lines, it went straight. This expert was supposed to only know how to turn, and thus to oscillate when presented with a straight line, however, even after a few different subnets (each with less data as the parameters got more restrictive) I got three good curve pilots that went straight in straight lines. I intend to try and take it to the limit.

100 margin 0-50 rows

This was the first approximation, that took a few straight lines into the subset but was the one how left fewer curve lines go into the straight subset (being this the only straight subset I have used). It left a 100 pixels unchecked at each side of the column picked and checked the first 50 rows of the image.

The result can be seen here: https://youtu.be/YLj_dbvlQWQ

150 margin 10-30 rows

Augmenting the margin I tried excluding straight lines that were left to a side by the Expert pilot and thus were diagonal, getting out of the previous margin and being taken into account as curves. Reducing the number of rows tried to, in case of diagonal lines avoid the furthest points, and by reducing the down limit I tried to avoid the widest parts of the line.

However, the result was similar as can be seen here: https://youtu.be/Pi8jpU8O3Fk

200 margin 10-30 rows

Again the result was able to go straight in most straight lines as appreciated in the video: https://youtu.be/IwRICPQ-V0c

Double Neural Pilot

With the classifier already done it was not too difficult to create a pilot that got the image, classified it into straight line or curve (exactly the same way as done to create the subsets) and depending on the result ask for the velocities to one expert or another. This was a first attempt success with all three curves experts, however, being as similar as they are, I will just record using the first one.

Recap

During the past weeks, I transitioned from a working model as I had to a better way of doing things. However, this forced me to implement a lot of new things such as:

  • Adjusting Expert Pilot
  • Dataset analysis.
  • Training and testing graphs
  • Off-line testing
  • Separating rosbags and datasets
  • Affine
  • Subset classifier

After all of this work we finally have some results (with everything going on the RTF is sometimes less than 1, so it will afect the time it takes and sometimes the precission of the pilot):

Expert Pilot 2.0

Our Expert Pilot that drives a bit slower but without leaving the line (almost never in the many curves circuit), from which every dataset is created, this pilot has been tested in four circuits with being worse in the more complex ones:

Simple: https://youtu.be/um7ybjK6Sj0 Many curves: https://youtu.be/0IGZdLkvS8k Montreal: https://youtu.be/qxh8F2iCVZc Montmelo: https://youtu.be/Zd4tx1RvVHw

Montmelo was quite bad, but at the same time gave me a dataset of the pilot actually overcoming getting far from the line, which provided very valuable data to the neural net on how to recover.

Neural Pilot

The fact the best pilot the one with the affine was a huge relief given the amount of time spent on making the affine work. The results can be compared to the Expert Pilot:

Simple: https://youtu.be/oLDNV-7qnoc Many curves: https://youtu.be/_zfzEvDSaOQ Montreal: https://youtu.be/UPoMY70P7xY Montmelo: https://youtu.be/1_gollvmBD0

In montreal The quality might be a problem for the net, but in montmelo crashes when suddenly when following nicley a curve, this shows there is probably more data needed.

Double Neural Pilot

This pilot brought together and managed by my classifier was definitely the best one in the test circuits and can be seen performing every other one in the next videos:

Simple: https://youtu.be/cQOunPa7UKI

Ends up failing in similar ways in Montreal and Montmelo as the Neural Pilot. When not recording does better, but fails either way.

End

The next steps are gonna be hard, probably I will have to change some of the things I already have and the intention is to create a neural net that is divided, it first analyzes the image and then passes the results (not an output but what you would see in the middle of the net) to one of two experts, which won’t need to re-analyze the image and will work only with the parameters received, after the first part analyzed and extracted the information from the image, to give the commands as an output. Achieving for real an end to end pilot.

However, I am quite happy to where I am and I am finally starting to see something I can actually present as my thesis, which I hope to present this course to graduate within 4 years of starting the degree.