Week 49 Β· November 3, 2026

Seed Independence in Ideas and Four Upstream Contributions to Behavior Metrics

November 3, 2026

πŸ“Š Part 1 β€” Overview

πŸ“Œ Two Lines of Work Closed This Week

  • Seed independence, in ideas. The goal was a training procedure that almost always yields the same driving model, while remaining a single model as fast as today. Six ideas were tested; only distillation (learning to imitate a group of five models) reduced the difference between seeds, by roughly a quarter.
  • Upstream contribution to Behavior Metrics. Four improvements were proposed to the open-source project. Two are already submitted for review with documentation; the other two are raised as issues awaiting the maintainers' opinion.

Short answer: training is chaotic and no single "culprit" piece can be fixed; seed independence splits into "thinking alike" (very hard) and "driving alike" (depends on margin in the hard stretches). Four contributions cover external simulator, weather, exact stepping and new metrics.

🌱 Part 2 β€” Seed Independence: The Question and the Path

Training a neural network involves a lot of randomness: how its connections start, in what order it sees the examples, which parts are randomly switched off while it learns. All of that randomness is controlled by a single number, the seed. With the same data and the same recipe, changing only that number produces models that drive differently: one passes a difficult curve and another goes off at the same place. It is as if the result depended on tossing a coin.

πŸ“Œ Why It Matters

  • No confident comparison. If one dataset yields a better model than another, we do not know whether it is because of the data or because that one got lucky.
  • No guaranteed result. When delivering a model, we do not know whether it is a typical one or a fortunate one.

The week's goal: find a way of training that always, or almost always, gives the same model β€” with one condition: at the end it must be a single model, as fast as today.

Each idea discarded an explanation and led to the next:

#IdeaWhat was doneOutcome
1Average several models into one (Model Soup)Train several models starting from the same point and blend their connections into a single modelThe blend worked, but the starting models came out worse, so the final result did not improve
2Let each model finish learningTraining stopped early, while the model was still improving; this was correctedFixing it did no harm, but did not make the models more similar to each other either
3Measure better"Reached the end of the route or not" is a very coarse measure; a finer measure was created: show all models the same images from the route and see how much their steering decisions differEnabled comparing ideas clearly in minutes and without driving
4Search for the causeSeparate the randomness into its five pieces to see which weighed mostSurprise: any one of them, on its own, is enough to produce the whole difference. Even the graphics card's plain rounding is enough
5Give more dataTrain a model with five times more examplesIt differs between seeds by the same amount as the others
6Teach it to imitate a group (distillation)Instead of learning from the examples directly, the model learns to imitate the average response of a group of five modelsThe only idea that reduced the difference between seeds, by approximately a quarter

πŸ” Part 3 β€” What Was Discovered

1. Learning is chaotic

As in the butterfly effect, a minimal difference at the beginning ends in a different model. It does not matter where the difference comes from: the order of the examples, the starting point or a rounding of the graphics card. That is why there is no single "culprit piece" that can be fixed.

2. There are two ways of being seed-independent

  • Thinking alike: all models make the same decisions on the same image. This turns out to be very difficult; almost nothing improves it.
  • Driving alike: all models reach the end of the route, even if internally they decide something different. This seems to depend on how much margin the model has in the difficult stretches. With more data, all models arrived.

3. Models fail where they are at the limit

In two specific stretches of the route, the models go right at the edge: a few more hours of training are enough for the same model to go from failing to arriving, or the other way round. The seed does not make models bad; it only decides which side of the edge they fall on.

4. Averaging predictions works better than averaging connections

Blending the connections of several models did not help. Teaching a model to imitate what a group says did reduce the difference between seeds, and it is still a single model when used.

5. A fine measure is worth more than many coarse tests

With only "arrived or not arrived" and few models, the results seemed to change by chance. The new measure made it possible to compare ideas clearly in minutes and without driving.

πŸ”œ Part 4 β€” What Follows (Seed Independence)

  • Confirm the imitate-a-group idea. Test it on another dataset and verify that those models also drive well, not only that they resemble each other.
  • Give the model margin in the difficult stretches. Understand what those two stretches are like and create similar situations in the training maps, without touching the test map.
  • Check the effect of more data with more seeds, to know whether all of them arriving was a real effect or luck.
  • Resume the other tasks of the week: the gas station in a training map, the documentation for Behavior Metrics, and the relationship between data and driving.

πŸ”€ Part 5 β€” Contribution to Behavior Metrics: Four Improvements

Behavior Metrics is an open-source tool from the JdeRobot group for evaluating automatic "drivers" in a simulator. It puts a model to drive and measures how it does: distance travelled, crashes, lane departures and smoothness. While using it with our models we found three limitations and missed two metrics. We fixed them in our copy in order to work, and then decided to propose them to the project, so that anyone using the tool under similar conditions benefits.

ImprovementIn a Few WordsWhat It ContributesStatus
A. Use an already-running simulator More ways to set up experiments Until now the tool closed the simulator if it ran separately, for example in a container or on another server, and required indicating where it was installed. With the improvement, it connects to the simulator that is already running without closing it. Enables using the tool with the simulator in Docker, on a remote server or in the cloud. Submitted, with documentation
B. Choose the weather Comparable experiments Until now each test started with the weather the map brought (cloudy, rainy, low sun…), with no way to choose it. With the improvement, the weather is indicated in the experiment configuration. Enables evaluating under the same conditions in which the model was trained and comparing experiments fairly. Raised; pending indication of where to integrate it
C. Exact step-by-step simulation Reliable and repeatable results Until now the simulation advanced irregularly and the model could decide with delayed images, so the result depended on the computer's speed. The improvement makes the world advance a fixed step for each decision, with the image from that same instant. In our test, a model went from crashing almost immediately to covering 428 m. Raised; pending agreement on the design
D. New driving metrics More information about how the model drives The tool did not measure whether the car stays in its lane nor whether the steering wheel oscillates from one side to the other. The improvement adds two metrics: the percentage of time in the correct lane and how many times per second the steering wheel changes side. Enables distinguishing a stable and correct model from one biased to one side, from a nervous one or from an unstable one. In addition, the documentation fills the project website's metrics page, which was empty. Submitted, with documentation

All four only affect the non-ROS mode of use, which is the one we use. While measuring we also found two counting bugs (collisions and lane departures counted more than they should), but we did not propose them because another person in the project is already fixing them.

How It Was Done

  1. Review what already exists. We checked that nobody was working on the same thing and detected two nearby works with which coordination is needed.
  2. Open one issue per improvement. An issue is a public record that describes the problem, how to reproduce it and a possible solution. We opened four.
  3. Prepare the change in our own copy. We made a copy of the project in our account, applied improvements A and D and tested them with the simulator.
  4. Propose them. For each one we sent two change requests (pull requests): one with the code and one with the documentation for the project website. Both were linked to the issue.

Along the way there were minor GitHub access obstacles (the account signs in with Google and GitHub does not accept passwords from the terminal). They were solved by signing in with the official GitHub tool through the browser.

⚠️ Where We Are and What Follows

Everything that depended on us is done. Now it is the project maintainers' turn to review and respond.

ReferenceNext Step
A. Already-running simulatorIssue #741; proposals #742 (code) and #743 (documentation). Address the review comments until accepted.
B. WeatherIssue #744. When they respond, send the change, which is already prepared.
C. Exact step-by-stepIssue #745. Agree on the design, program it and send it.
D. New metricsIssue #746; proposals #747 (code) and #748 (documentation). Address the review comments until accepted.

Meanwhile, our installation keeps working with the improvements applied locally, so the evaluation of models does not stop. If there is no response in two or three weeks, a friendly reminder will be sent.

πŸ“Œ WEEK 49 SUMMARY – NOVEMBER 3, 2026

🎯 Objective: Find a training procedure that almost always yields the same driving model while remaining a single model as fast as today, and propose to Behavior Metrics the improvements that arose while evaluating driving models in CARLA.

πŸ§ͺ Methodologies: Six seed-independence ideas tested in sequence: model soup (connection averaging), full training without early stopping, a fine steering-decision measure on shared images, separation of randomness into five components, five times more data, and distillation (imitating the average response of a group of five models). For upstream: one issue per improvement, a personal fork, local testing with the simulator, and paired code and documentation pull requests linked to each issue.

πŸ“ˆ Key Findings: Learning is chaotic β€” a minimal initial difference (example order, starting point, even graphics-card rounding) ends in a different model, so there is no single fixable culprit. Seed independence splits into "thinking alike" (very hard; almost nothing improves it) and "driving alike" (depends on margin in the two edge stretches of the route; with more data all models arrived). Models fail where they are at the limit: a few more hours of training can flip the same model from failing to arriving. Averaging predictions (distillation) works better than averaging connections and reduced the seed difference by about a quarter, still yielding a single model. A fine measure is worth more than many coarse tests. For Behavior Metrics: four improvements β€” already-running simulator (A), weather selection (B), exact step-by-step simulation (C, which took a model from crashing almost immediately to 428 m) and new lane/steering metrics (D) β€” all affecting only the non-ROS mode; two counting bugs found but not proposed, as another contributor is already fixing them.

βœ… Achievements: Distillation is the only idea that reduced seed dependence, by approximately a quarter, while keeping a single fast model. A fine offline measure now allows comparing ideas clearly in minutes without driving. Improvements A and D are submitted with code and documentation (PRs #742/#743 and #747/#748); issues #744 and #745 are raised and their changes are prepared. Local installation keeps working with the patches, so model evaluation continues.

❌ Limitations: No single source of randomness can be blamed, so there is no simple fix. Distillation has not yet been confirmed on another dataset, nor verified that the resulting models drive well. The "more data" result is a single case with a small number of seeds. The two difficult stretches are still not understood. No ROS on Ubuntu 26.04, so upstream changes cannot be tested in ROS mode. Review and integration depend on maintainers.

πŸ”œ Next Steps: Confirm the imitate-a-group idea on another dataset and verify driving quality, not just resemblance. Give the model margin in the two difficult stretches by creating similar situations in the training maps, without touching the test map. Check the effect of more data with more seeds. Resume the gas station in a training map, the Behavior Metrics documentation and the data–driving relationship. Address review comments on PRs #742/#743 and #747/#748, send the prepared weather change after issue #744, and agree on the design for #745. Send a friendly reminder if there is no response in two or three weeks.