Week 50 ยท November 10, 2026

Distillation Confirmed, and New Data for the Corners Where Models Fail

November 10, 2026

๐Ÿ“Š Part 1 โ€” Overview

๐Ÿ“Œ What Changed This Week

  • Distillation is confirmed and adopted. Teaching each model to imitate the average answer of a group of five reduced the difference between seeds on three datasets, and those models also drive better: 14 of 15 reach the end of the route, against 12 of 15 without it.
  • Two explanations were ruled out. The failures are not caused by the smoothing of the steering wheel, and the shape of the training data (its steering histogram) does not explain why one dataset drives better than another.
  • The weak spot was found. The three hard places on the test route are left-hand corners without a junction, a situation that almost never appears in the training data. New data was captured for exactly that situation, and the models that use it are training now.

Short answer: the seed problem now has a working solution, and the remaining failures point to missing data rather than to the training method.

๐ŸŒฑ Part 2 โ€” Distillation, Confirmed on Three Datasets

Last week, distillation was the only idea that made models trained with different seeds more alike, but it had been tried on a single dataset and nobody had checked whether those models drive well. This week it was repeated on three datasets, with five seeds each, and every model was put to drive the test route in Town02. The route has two difficult points (A at about 60 m and C at about 240 m) and ends at a gas station (B, about 430 m). Reaching B means completing the route.

The student learns the average answer of 5 models and is still one model Dataset 62,000 images 1 ยท Normal training Control s42 Control s1 Control s2 Control s3 Control s4 2 ยท Teacher Teacher average of the 5 answers per image target 3 ยท Student (ร—5 seeds) Student (seed s) imitates the teacher ฮฑ = 1 One single model as fast as before when driving same training images ฮฑ = 1: imitates the teacher only (14/15 reach B) ยท ฮฑ = 0.5: half label, half teacher (0/5)
How distillation works: five normally trained models form the teacher; each student is trained on the same images to reproduce the teacher’s average steering, and only the student is used for driving.
Group (3 datasets ร— 5 seeds)Reach the end (B)Difference between seeds
Week 48 models11/15reference
Normal training (controls)12/15reference for the teacher
Distillation, full imitation (ฮฑ = 1)14/1511 % to 23 % lower
Distillation, half imitation (ฮฑ = 0.5)0/5โ€”

The "difference between seeds" is the fine measure introduced last week: all models look at the same 2,122 images of the route, and we measure how much their steering decisions disagree. Full imitation works; mixing the original label and the teacher's answer half and half breaks the model completely. The code was checked and is correct, so this is a real effect of the mixture.

Where the idea comes from

Using distillation against differences between runs is not new. Jiang et al. [1] use it to reduce churn: the differences in prediction between models that should be equivalent, exactly our seed problem. Bhojanapalli et al. [2] show that two networks trained with the same recipe disagree on many predictions, and reduce that disagreement with minimum-entropy regularizers and co-distillation, a variant where models learn from each other while they train. Our version applies the same idea to steering regression with a teacher of five models, and checks the effect on driving, not only on the predictions.

Is it just a noisy steering wheel?

The driver softens the model's steering with a smoothing filter. One possibility was that the filter made the car react too late in the corners. The same models were driven again without the filter, and the result was identical: 12 of 15. The failures are not noise. The models make the same wrong decision in the same places.

๐Ÿ” Part 3 โ€” Does the Data Explain the Driving?

Two questions were asked about the training data. First, whether the dataset whose steering histogram is most "typical" gives the best drivers. Second, whether changing the proportions of each type of situation (straight roads, turns, recoveries) gives better drivers.

1. The histograms of our datasets only differ by chance

The distance between the steering histograms of the different datasets (Wasserstein distance, 0.0006 to 0.0016) is about the same size as the difference expected from random sampling alone (median 0.0009, 95th percentile 0.0014). Differences that small cannot explain why one dataset drives better than another. The large dataset (309,000 images) looked the most "typical" only because more samples means less random variation.

For the second question, three new mixtures were built with 62,000 images each, three versions each and five seeds per version, and compared with the current mixture (Comp0):

MixtureChange from Comp0Reach the endFail at AFail at C
Comp0current mixture12/1521
Comp2symmetric: as many left as right turns, more straight road11/15โ€”โ€”
Comp320 % lane recoveries10/1550
Comp4less straight road, more right turns7/1544

2. No new mixture beats the current one

More lane recoveries fixes point C but makes point A worse; less straight road makes both worse. Rebalancing what already exists only moves the problem from one corner to another. This suggested that something was missing from the data, not badly proportioned.

๐Ÿงญ Part 4 โ€” The Missing Situation: Left-Hand Corners

Looking closely at the route, A, C and the gas station have the same shape: a 90ยฐ left-hand corner with no junction, an "L". In our training maps the driving rule at junctions is "turn right if you can; otherwise go straight; turn left only if there is no other option". As a result, the models had seen very few left L-corners and had never seen how to recover from a bad entry into one.

Instead of copying the gas station area into other maps, new examples were recorded in the training maps (Town01 and Town04), leaving the test map untouched:

  1. Find the corners. A tool now lists every left L-corner of a map, respecting the driving rule. Town01 has four useful corners and Town04 two.
  2. Record recoveries. The capture tool places the car directly at a corner. A short push of the steering wheel (reduced from 0.7 to 0.15) or random noise of 0.15 knocks the car off its line, and the human driver brings it back. Only the recovery is kept.
  3. Get twice the data from the same work. Each corner was also driven on the left side of the road, British style, and those images were mirrored: a right-hand British corner becomes a normal left-hand corner.
  4. Prepare the images exactly like the rest of the data (remove frames under 10 km/h, crop, resize).

The result is 445 recordings and 33,431 new images of left-corner recoveries, a type of data the models did not have until now.

โณ Part 5 โ€” In Progress: Comp5 with Distillation

The new mixture, Comp5, adds 10 % of the new left-corner recoveries. It follows the same design as before (62,000 images, three versions, five seeds), so it can be compared directly with Comp0:

Straight roadGentle weavingRight turnsLeft turnsLeft-corner recoveries (new)
35 %25 %20 %10 %10 %

โš ๏ธ Where We Are

The 15 normal models (the teachers) and the 15 distilled models are being trained in a single queue that ends today. Then they will drive the test route. The question is whether the new data removes the failures at A and C. If it does, distilled Comp5 becomes the new reference. A larger version with all available images (about 309,000) was postponed, because it needs almost a full day of training.

๐Ÿ—‚๏ธ Part 6 โ€” Project Housekeeping and Behavior Metrics

  • Order. The scripts are now grouped by role (common model, training, driving, analysis, archive), with a single definition of the network shared by all of them.
  • Reproducibility. The project is in a private GitHub repository with restore points; the simulator runs from our own Docker image, backed up; both Python environments are frozen in requirement files; and a command guide documents how to drive, build datasets, train and evaluate.
  • Behavior Metrics. The maintainers reviewed both proposals. For #742 (already-running simulator) they ask to send one small timing fix as a separate proposal to their main ROS branch. For #747 (new metrics) they ask to compute the lane percentages over simulated time instead of over the number of samples. The replies are drafted and the decision on how to proceed is being taken this week.

๐Ÿ“Œ WEEK 50 SUMMARY โ€“ NOVEMBER 10, 2026

๐ŸŽฏ Objective: Confirm distillation as the solution to seed dependence, understand why models fail at the hard points of the test route, and give them the data they lack, while organizing and documenting the project.

๐Ÿงช Methodologies: Distillation (ฮฑ = 1 and ฮฑ = 0.5) on three datasets with five seeds each, evaluated with the fine seed-difference measure and by driving the Town02 route. Driving without steering smoothing. Wasserstein distance between steering histograms compared with the sampling-noise floor. Three new data mixtures (Comp2โ€“Comp4), 15 models each. A corner finder and an improved capture tool to record left-corner recoveries in Town01 and Town04, with British-side driving and mirroring.

๐Ÿ“ˆ Key Findings: Distillation with full imitation reduces the difference between seeds by 11โ€“23 % and improves driving from 12/15 to 14/15; half imitation fails (0/5). The failures are systematic decisions, not steering noise. Histogram differences between our datasets are at sampling-noise level and cannot explain driving. No new mixture beats Comp0: lane recoveries fix C but hurt A. A, C and the gas station are left L-corners without a junction, a situation almost absent from training.

โœ… Achievements: Distillation adopted. 445 recordings and 33,431 new images of left-corner recoveries. Comp5 built; its 15 normal and 15 distilled models in training. Project reorganized, on GitHub, in a Docker image and documented. Review answers for Behavior Metrics drafted.

โŒ Limitations: Comp5 driving results are not available yet. The large Comp5 version was postponed. The capture tool leaves the car still for 3โ€“14 s after it appears. The histogram analysis has only been done within one mixture.

๐Ÿ”œ Next Steps: Drive the distilled Comp5 models and compare them with distilled Comp0. Train and distill the large Comp5 version. Repeat the histogram analysis across mixtures. Answer the reviews of #742 and #747. Fix the capture tool's start-up delay.

๐Ÿ“š References

  1. H. Jiang, H. Narasimhan, D. Bahri, A. Cotter, A. Rostamizadeh. Churn Reduction via Distillation. ICLR 2022. arXiv:2106.02654 ยท ICLR
  2. S. Bhojanapalli, K. Wilber, A. Veit, A. S. Rawat, S. Kim, A. Menon, S. Kumar. On the Reproducibility of Neural Network Predictions. 2021. arXiv:2102.03349