Week 46 Β· October 13, 2026
Seed Variance: Deterministic Training, EMA, and Ensembles β First Online Evaluation of 200 Models
October 13, 2026
π The Problem in One Sentence
Models trained in PyTorch on the same dataset with the same seed produce different results, so model reproducibility could not be achieved β and similar offline metrics do not guarantee similar driving behavior.
π How the Problem Was Evidenced
The Week 45 results already showed that training in PyTorch with the same dataset produces different models, so reproducibility was not reached. To prove it, the same dataset was trained three times, obtaining the following results:
| Run |
best_val_loss |
final train loss |
final val loss |
| First training | 0.014067 | 0.006712 | 0.014339 |
| Second training | 0.013050 | 0.006764 | 0.013776 |
| Third training | 0.013085 | 0.006633 | 0.013404 |
β οΈ The Key Observation
Although the metrics are similar, the driving results are quite different. Two models with nearly identical validation loss can behave completely differently on the road. This is the practical consequence of seed variance.
π Literature Reviewed
The seed-sensitivity topic has been reviewed in depth. The following works cover exactly this line of research:
"torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision"
Works with ImageNet and CIFAR, makes the problem very clear, and also addresses the practice of taking the best "run" of a trained model. arXiv:2109.08203
"Sources of Irreproducibility in Machine Learning: A Review"
Analyzes the sources of irreproducibility and identifies random initialization as one of them. arXiv:2204.07610
"Measuring training variability from stochastic optimization using robust nonparametric testing"
Focuses on how to measure the impact of randomness. Both works point to the effects of seeds at the time of initializing training. arXiv:2406.08307
π Status of the Problem
This is still an open problem within the research community. There is no single accepted solution; the practical approach is to reduce the stochastic effect and to measure with several seeds rather than trusting one run.
To reduce the stochastic effect of seed selection, two complementary approaches were found.
β
2.1 β In Training: Exponential Moving Average (EMA)
EMA averages, for each training batch, the weight of every connection. This is done for each seed being trained, and the result is to reduce the effect of the seed during training, producing models with similar behavior; that is, it reduces the "long tail" effect (models very far from the mean) and the "black swan" effect (models with special behaviors).
The process is defined by the equation (Lakshmanan 2026, A Short Survey of Averaging Techniques in Stochastic Gradient Methods):
x_k = Ξ² Β· x_(kβ1) + (1 β Ξ²) Β· ΞΈ_k
Other alternatives that could be used are PolyakβRuppert, Stochastic Weight Averaging (SWA), and Exponential Moving Average (EMA).
β
2.2 β At Driving Time: Ensembles
The idea is to train between 5 and 7 models on the same dataset but with a different seed (a larger number does not seem to produce changes), and then, while driving, average the inference of these 5 models. This is called an ensemble, as indicated in "Machine Learning Prediction of Airfoil Aerodynamic Performance Using Neural Network Ensembles" (MDPI 2076-3417/15/14/7720).
3.1 β Training Determinism
β οΈ What Is Fixed Today and What Is Not
The training script was modified to fix the PyTorch seed, but the GPU computes convolutions with algorithms that do not guarantee bit-for-bit identical results. That minimal difference grows with every training step until it produces different models.
In main(), the line torch.manual_seed(args.seed) was replaced by:
random.seed(args.seed)
np.random.seed(args.seed)
torch.manual_seed(args.seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
The price paid for fixing the seed is that the training algorithm becomes somewhat slower (no deeper work was done in this direction during the week).
β
Verification: Determinism in Training Was Achieved
Several models were trained consecutively, verifying training parameters such as MAE and obtaining the same results. The final driving test also returned the same results, which is why determinism in training is now guaranteed.
3.2 β Search for Seed Independence
β
EMA Implementation
EMA was implemented with an exponential decay value Ξ² = 0.999.
β
Ensembles
Since Week 40, a dataset bank was consolidated, made up of 8 different compositions (different percentages per maneuver) and generated with 5 seeds for each composition (42, 123, 456, 789, 999). For this reason, 5 different models were trained for each composition (1 per seed).
In total, with the previous process, 8 compositions Γ 5 dataset seeds Γ 5 training seeds = 200 models were trained.
Offline metrics (training)
As shown in Part 1, the three repeated runs gave very similar offline metrics (best_val_loss β 0.0130β0.0141, final val loss β 0.0134β0.0143). The offline metrics alone do not separate good from bad models; that is why the online evaluation is the decisive one.
Online metrics (driving)
The file ranking_ensembles_run81.csv summarizes the results. In particular, the columns dataset, metros, segundos, fuera_pct, and cte_media.
How to read the online metrics
- fuera_pct β percentage of frames outside the lane.
- cte_media (cross-track error) β the lateral distance in meters between the car and the center of the lane, averaged over all frames. The driver looks for the closest point of the lane center and measures how far the car separates from it. 0 would be driving always perfectly centered. With 0.60 m, the car drives on average 60 cm from the center; with 0.97 m, almost one meter.
- cte_p95 β the 95th percentile of that same distance. 95% of the time the car was closer to the center than this value, so it measures the worst moments, normally the curves. A 1.49 m indicates that in curves it opens up to almost 1.5 m.
π Interpretation of the Ranking
The ranking is ordered by cte_media, from best (lowest) to worst. The best ensembles keep the car centered at roughly 0.54β0.69 m on average and complete the route (~430 m in ~105 s), with fuera_pct in the 1.5β6% range. The worst cases fall into two clearly different failure modes:
- High lateral error without crashing β e.g.,
Comp7-seed999 (cte_media 1.22 m, fuera_pct 7.63%) and Comp6-seed999 (cte_media 1.05 m, fuera_pct 10.49%). The car completes the route but drifts far from the center.
- Early termination / getting stuck β the last four rows (
Comp2-seed789, Comp6-seed123, Comp5-seed123, Comp7-seed123, Comp7-seed42) travel only 62β103 m in 15β25 s with fuera_pct between 13% and 33% and very high cte_p95 (6.8β9.2 m). These are the seed-variance "black swans": models that fail early on the same route where other seeds succeed.
Key takeaway: the dataset composition and the training seed both matter, but the seed effect is large enough to turn a ~430 m run into a ~62 m run on the same composition (compare Comp7-seed456 at 430.5 m / cte_media 0.765 vs. Comp7-seed42 at 63.6 m / cte_media 2.017, and Comp7-seed123 at 63.4 m). This is exactly the run-to-run variance the week set out to reduce, and it justifies both EMA and ensembles.
π§ͺ What the Online Results Confirm
All 40 evaluated configurations correspond to the same dataset bank (Compositions 0β7, seeds 42/123/456/789/999). Fourteen of them (the last rows) fail early or drift far from the lane center. Because the number of models is small, these results are descriptive: they show that seed variance is large enough to change driving quality, but they do not yet quantify a confidence interval. That is exactly why EMA and ensembles are the next step: to shrink the long tail and the black swans before comparing configurations.
π WEEK 46 SUMMARY β OCTOBER 13, 2026
π― Objective: Reproduce and document run-to-run (seed) variance in training, implement determinism, and reduce seed sensitivity through EMA and ensembles, then evaluate the resulting models online.
π§ͺ Methodologies: Three repeated trainings with the same seed to expose variance; literature review of seed sensitivity (arXiv:2109.08203, arXiv:2204.07610, arXiv:2406.08307); deterministic training via random.seed, np.random.seed, torch.manual_seed, cudnn.deterministic=True, cudnn.benchmark=False; EMA with Ξ² = 0.999; ensemble evaluation over 8 dataset compositions Γ 5 dataset seeds Γ 5 training seeds = 200 models.
π Key Findings: Similar offline metrics (best_val_loss 0.0130β0.0141) produced very different driving behavior. Determinism in training was verified by re-training and re-testing consecutively. In the online ranking, the best ensembles complete the route (~430 m in ~105 s) with cte_media 0.54β0.69 m and fuera_pct 1.5β6%, while the worst seeds travel only 62β103 m with fuera_pct 13β33% and cte_p95 up to 9.2 m on the same compositions β a direct demonstration of seed variance.
β
Achievements: Deterministic training guaranteed and verified. EMA implemented (Ξ² = 0.999). A 200-model bank (8 compositions Γ 5 dataset seeds Γ 5 training seeds) trained. First online ranking of ensembles generated (ranking_ensembles_run81.csv) with interpretable best and worst cases.
β Limitations: Offline metrics do not predict driving quality. The online ranking is descriptive and still without confidence intervals. Determinism only holds within the same GPU/PyTorch/CUDA/cuDNN environment and does not recover previous runs. The frame_0 image display and the PilotNet brain connection inherited from Week 45 remain pending for the Behavior Metrics evaluation.
π Next Steps: Compare EMA models against non-EMA models on the same route to quantify the reduction of the long tail and black swans. Formalize the ensemble inference (5β7 models averaged) and re-run the ranking to check whether the worst seeds move toward the mean. Add a per-composition breakdown of cte_media and fuera_pct to separate dataset-composition effects from training-seed effects. Complete the Behavior Metrics frame_0 display and connect the PilotNet brain so that the best model (dataset_w40-62000_Comp0-seed42 in this ranking) can be cross-checked against the in-house LaneMonitor metrics.