Distillation Confirmed, and New Data for the Corners Where Models Fail
November 10, 2026
๐ Part 1 โ Overview
๐ What Changed This Week
Distillation is confirmed and adopted. Teaching each model to imitate the average answer of a group of five reduced the difference between seeds on three datasets, and those models also drive better: 14 of 15 reach the end of the route, against 12 of 15 without it.
Two explanations were ruled out. The failures are not caused by the smoothing of the steering wheel, and the shape of the training data (its steering histogram) does not explain why one dataset drives better than another.
The weak spot was found. The three hard places on the test route are left-hand corners without a junction, a situation that almost never appears in the training data. New data was captured for exactly that situation, and the models that use it are training now.
Short answer: the seed problem now has a working solution, and the remaining failures point to missing data rather than to the training method.
๐ฑ Part 2 โ Distillation, Confirmed on Three Datasets
Last week, distillation was the only idea that made models trained with different seeds more alike, but it had been tried on a single dataset and nobody had checked whether those models drive well. This week it was repeated on three datasets, with five seeds each, and every model was put to drive the test route in Town02. The route has two difficult points (A at about 60 m and C at about 240 m) and ends at a gas station (B, about 430 m). Reaching B means completing the route.
How distillation works: five normally trained models form the teacher; each student is trained on the same images to reproduce the teacher’s average steering, and only the student is used for driving.
Group (3 datasets ร 5 seeds)
Reach the end (B)
Difference between seeds
Week 48 models
11/15
reference
Normal training (controls)
12/15
reference for the teacher
Distillation, full imitation (ฮฑ = 1)
14/15
11 % to 23 % lower
Distillation, half imitation (ฮฑ = 0.5)
0/5
โ
The "difference between seeds" is the fine measure introduced last week: all models look at the same 2,122 images of the route, and we measure how much their steering decisions disagree. Full imitation works; mixing the original label and the teacher's answer half and half breaks the model completely. The code was checked and is correct, so this is a real effect of the mixture.
Where the idea comes from
Using distillation against differences between runs is not new. Jiang et al. [1] use it to reduce churn: the differences in prediction between models that should be equivalent, exactly our seed problem. Bhojanapalli et al. [2] show that two networks trained with the same recipe disagree on many predictions, and reduce that disagreement with minimum-entropy regularizers and co-distillation, a variant where models learn from each other while they train. Our version applies the same idea to steering regression with a teacher of five models, and checks the effect on driving, not only on the predictions.
Is it just a noisy steering wheel?
The driver softens the model's steering with a smoothing filter. One possibility was that the filter made the car react too late in the corners. The same models were driven again without the filter, and the result was identical: 12 of 15. The failures are not noise. The models make the same wrong decision in the same places.
๐ Part 3 โ Does the Data Explain the Driving?
Two questions were asked about the training data. First, whether the dataset whose steering histogram is most "typical" gives the best drivers. Second, whether changing the proportions of each type of situation (straight roads, turns, recoveries) gives better drivers.
1. The histograms of our datasets only differ by chance
The distance between the steering histograms of the different datasets (Wasserstein distance, 0.0006 to 0.0016) is about the same size as the difference expected from random sampling alone (median 0.0009, 95th percentile 0.0014). Differences that small cannot explain why one dataset drives better than another. The large dataset (309,000 images) looked the most "typical" only because more samples means less random variation.
For the second question, three new mixtures were built with 62,000 images each, three versions each and five seeds per version, and compared with the current mixture (Comp0):
Mixture
Change from Comp0
Reach the end
Fail at A
Fail at C
Comp0
current mixture
12/15
2
1
Comp2
symmetric: as many left as right turns, more straight road
11/15
โ
โ
Comp3
20 % lane recoveries
10/15
5
0
Comp4
less straight road, more right turns
7/15
4
4
2. No new mixture beats the current one
More lane recoveries fixes point C but makes point A worse; less straight road makes both worse. Rebalancing what already exists only moves the problem from one corner to another. This suggested that something was missing from the data, not badly proportioned.
๐งญ Part 4 โ The Missing Situation: Left-Hand Corners
Looking closely at the route, A, C and the gas station have the same shape: a 90ยฐ left-hand corner with no junction, an "L". In our training maps the driving rule at junctions is "turn right if you can; otherwise go straight; turn left only if there is no other option". As a result, the models had seen very few left L-corners and had never seen how to recover from a bad entry into one.
Instead of copying the gas station area into other maps, new examples were recorded in the training maps (Town01 and Town04), leaving the test map untouched:
Find the corners. A tool now lists every left L-corner of a map, respecting the driving rule. Town01 has four useful corners and Town04 two.
Record recoveries. The capture tool places the car directly at a corner. A short push of the steering wheel (reduced from 0.7 to 0.15) or random noise of 0.15 knocks the car off its line, and the human driver brings it back. Only the recovery is kept.
Get twice the data from the same work. Each corner was also driven on the left side of the road, British style, and those images were mirrored: a right-hand British corner becomes a normal left-hand corner.
Prepare the images exactly like the rest of the data (remove frames under 10 km/h, crop, resize).
The result is 445 recordings and 33,431 new images of left-corner recoveries, a type of data the models did not have until now.
โณ Part 5 โ In Progress: Comp5 with Distillation
The new mixture, Comp5, adds 10 % of the new left-corner recoveries. It follows the same design as before (62,000 images, three versions, five seeds), so it can be compared directly with Comp0:
Straight road
Gentle weaving
Right turns
Left turns
Left-corner recoveries (new)
35 %
25 %
20 %
10 %
10 %
โ ๏ธ Where We Are
The 15 normal models (the teachers) and the 15 distilled models are being trained in a single queue that ends today. Then they will drive the test route. The question is whether the new data removes the failures at A and C. If it does, distilled Comp5 becomes the new reference. A larger version with all available images (about 309,000) was postponed, because it needs almost a full day of training.
๐๏ธ Part 6 โ Project Housekeeping and Behavior Metrics
Order. The scripts are now grouped by role (common model, training, driving, analysis, archive), with a single definition of the network shared by all of them.
Reproducibility. The project is in a private GitHub repository with restore points; the simulator runs from our own Docker image, backed up; both Python environments are frozen in requirement files; and a command guide documents how to drive, build datasets, train and evaluate.
Behavior Metrics. The maintainers reviewed both proposals. For #742 (already-running simulator) they ask to send one small timing fix as a separate proposal to their main ROS branch. For #747 (new metrics) they ask to compute the lane percentages over simulated time instead of over the number of samples. The replies are drafted and the decision on how to proceed is being taken this week.
๐ WEEK 50 SUMMARY โ NOVEMBER 10, 2026
๐ฏ Objective: Confirm distillation as the solution to seed dependence, understand why models fail at the hard points of the test route, and give them the data they lack, while organizing and documenting the project.
๐งช Methodologies: Distillation (ฮฑ = 1 and ฮฑ = 0.5) on three datasets with five seeds each, evaluated with the fine seed-difference measure and by driving the Town02 route. Driving without steering smoothing. Wasserstein distance between steering histograms compared with the sampling-noise floor. Three new data mixtures (Comp2โComp4), 15 models each. A corner finder and an improved capture tool to record left-corner recoveries in Town01 and Town04, with British-side driving and mirroring.
๐ Key Findings: Distillation with full imitation reduces the difference between seeds by 11โ23 % and improves driving from 12/15 to 14/15; half imitation fails (0/5). The failures are systematic decisions, not steering noise. Histogram differences between our datasets are at sampling-noise level and cannot explain driving. No new mixture beats Comp0: lane recoveries fix C but hurt A. A, C and the gas station are left L-corners without a junction, a situation almost absent from training.
โ Achievements: Distillation adopted. 445 recordings and 33,431 new images of left-corner recoveries. Comp5 built; its 15 normal and 15 distilled models in training. Project reorganized, on GitHub, in a Docker image and documented. Review answers for Behavior Metrics drafted.
โ Limitations: Comp5 driving results are not available yet. The large Comp5 version was postponed. The capture tool leaves the car still for 3โ14 s after it appears. The histogram analysis has only been done within one mixture.
๐ Next Steps: Drive the distilled Comp5 models and compare them with distilled Comp0. Train and distill the large Comp5 version. Repeat the histogram analysis across mixtures. Answer the reviews of #742 and #747. Fix the capture tool's start-up delay.
๐ References
H. Jiang, H. Narasimhan, D. Bahri, A. Cotter, A. Rostamizadeh. Churn Reduction via Distillation. ICLR 2022. arXiv:2106.02654 ยท ICLR
S. Bhojanapalli, K. Wilber, A. Veit, A. S. Rawat, S. Kim, A. Menon, S. Kumar. On the Reproducibility of Neural Network Predictions. 2021. arXiv:2102.03349