Week 49 Β· November 3, 2026
Seed Independence in Ideas and Four Upstream Contributions to Behavior Metrics
November 3, 2026
π Two Lines of Work Closed This Week
- Seed independence, in ideas. The goal was a training procedure that almost always yields the same driving model, while remaining a single model as fast as today. Six ideas were tested; only distillation (learning to imitate a group of five models) reduced the difference between seeds, by roughly a quarter.
- Upstream contribution to Behavior Metrics. Four improvements were proposed to the open-source project. Two are already submitted for review with documentation; the other two are raised as issues awaiting the maintainers' opinion.
Short answer: training is chaotic and no single "culprit" piece can be fixed; seed independence splits into "thinking alike" (very hard) and "driving alike" (depends on margin in the hard stretches). Four contributions cover external simulator, weather, exact stepping and new metrics.
Training a neural network involves a lot of randomness: how its connections start, in what order it sees the examples, which parts are randomly switched off while it learns. All of that randomness is controlled by a single number, the seed. With the same data and the same recipe, changing only that number produces models that drive differently: one passes a difficult curve and another goes off at the same place. It is as if the result depended on tossing a coin.
π Why It Matters
- No confident comparison. If one dataset yields a better model than another, we do not know whether it is because of the data or because that one got lucky.
- No guaranteed result. When delivering a model, we do not know whether it is a typical one or a fortunate one.
The week's goal: find a way of training that always, or almost always, gives the same model β with one condition: at the end it must be a single model, as fast as today.
Each idea discarded an explanation and led to the next:
1. Learning is chaotic
As in the butterfly effect, a minimal difference at the beginning ends in a different model. It does not matter where the difference comes from: the order of the examples, the starting point or a rounding of the graphics card. That is why there is no single "culprit piece" that can be fixed.
2. There are two ways of being seed-independent
- Thinking alike: all models make the same decisions on the same image. This turns out to be very difficult; almost nothing improves it.
- Driving alike: all models reach the end of the route, even if internally they decide something different. This seems to depend on how much margin the model has in the difficult stretches. With more data, all models arrived.
3. Models fail where they are at the limit
In two specific stretches of the route, the models go right at the edge: a few more hours of training are enough for the same model to go from failing to arriving, or the other way round. The seed does not make models bad; it only decides which side of the edge they fall on.
4. Averaging predictions works better than averaging connections
Blending the connections of several models did not help. Teaching a model to imitate what a group says did reduce the difference between seeds, and it is still a single model when used.
5. A fine measure is worth more than many coarse tests
With only "arrived or not arrived" and few models, the results seemed to change by chance. The new measure made it possible to compare ideas clearly in minutes and without driving.
- Confirm the imitate-a-group idea. Test it on another dataset and verify that those models also drive well, not only that they resemble each other.
- Give the model margin in the difficult stretches. Understand what those two stretches are like and create similar situations in the training maps, without touching the test map.
- Check the effect of more data with more seeds, to know whether all of them arriving was a real effect or luck.
- Resume the other tasks of the week: the gas station in a training map, the documentation for Behavior Metrics, and the relationship between data and driving.
Behavior Metrics is an open-source tool from the JdeRobot group for evaluating automatic "drivers" in a simulator. It puts a model to drive and measures how it does: distance travelled, crashes, lane departures and smoothness. While using it with our models we found three limitations and missed two metrics. We fixed them in our copy in order to work, and then decided to propose them to the project, so that anyone using the tool under similar conditions benefits.
All four only affect the non-ROS mode of use, which is the one we use. While measuring we also found two counting bugs (collisions and lane departures counted more than they should), but we did not propose them because another person in the project is already fixing them.
How It Was Done
- Review what already exists. We checked that nobody was working on the same thing and detected two nearby works with which coordination is needed.
- Open one issue per improvement. An issue is a public record that describes the problem, how to reproduce it and a possible solution. We opened four.
- Prepare the change in our own copy. We made a copy of the project in our account, applied improvements A and D and tested them with the simulator.
- Propose them. For each one we sent two change requests (pull requests): one with the code and one with the documentation for the project website. Both were linked to the issue.
Along the way there were minor GitHub access obstacles (the account signs in with Google and GitHub does not accept passwords from the terminal). They were solved by signing in with the official GitHub tool through the browser.
β οΈ Where We Are and What Follows
Everything that depended on us is done. Now it is the project maintainers' turn to review and respond.
Meanwhile, our installation keeps working with the improvements applied locally, so the evaluation of models does not stop. If there is no response in two or three weeks, a friendly reminder will be sent.
π WEEK 49 SUMMARY β NOVEMBER 3, 2026
π― Objective: Find a training procedure that almost always yields the same driving model while remaining a single model as fast as today, and propose to Behavior Metrics the improvements that arose while evaluating driving models in CARLA.
π§ͺ Methodologies: Six seed-independence ideas tested in sequence: model soup (connection averaging), full training without early stopping, a fine steering-decision measure on shared images, separation of randomness into five components, five times more data, and distillation (imitating the average response of a group of five models). For upstream: one issue per improvement, a personal fork, local testing with the simulator, and paired code and documentation pull requests linked to each issue.
π Key Findings: Learning is chaotic β a minimal initial difference (example order, starting point, even graphics-card rounding) ends in a different model, so there is no single fixable culprit. Seed independence splits into "thinking alike" (very hard; almost nothing improves it) and "driving alike" (depends on margin in the two edge stretches of the route; with more data all models arrived). Models fail where they are at the limit: a few more hours of training can flip the same model from failing to arriving. Averaging predictions (distillation) works better than averaging connections and reduced the seed difference by about a quarter, still yielding a single model. A fine measure is worth more than many coarse tests. For Behavior Metrics: four improvements β already-running simulator (A), weather selection (B), exact step-by-step simulation (C, which took a model from crashing almost immediately to 428 m) and new lane/steering metrics (D) β all affecting only the non-ROS mode; two counting bugs found but not proposed, as another contributor is already fixing them.
β
Achievements: Distillation is the only idea that reduced seed dependence, by approximately a quarter, while keeping a single fast model. A fine offline measure now allows comparing ideas clearly in minutes without driving. Improvements A and D are submitted with code and documentation (PRs #742/#743 and #747/#748); issues #744 and #745 are raised and their changes are prepared. Local installation keeps working with the patches, so model evaluation continues.
β Limitations: No single source of randomness can be blamed, so there is no simple fix. Distillation has not yet been confirmed on another dataset, nor verified that the resulting models drive well. The "more data" result is a single case with a small number of seeds. The two difficult stretches are still not understood. No ROS on Ubuntu 26.04, so upstream changes cannot be tested in ROS mode. Review and integration depend on maintainers.
π Next Steps: Confirm the imitate-a-group idea on another dataset and verify driving quality, not just resemblance. Give the model margin in the two difficult stretches by creating similar situations in the training maps, without touching the test map. Check the effect of more data with more seeds. Resume the gas station in a training map, the Behavior Metrics documentation and the dataβdriving relationship. Address review comments on PRs #742/#743 and #747/#748, send the prepared weather change after issue #744, and agree on the design for #745. Send a friendly reminder if there is no response in two or three weeks.