Week 46 Β· October 13, 2026

Seed Variance: Deterministic Training, EMA, and Ensembles β€” First Online Evaluation of 200 Models

October 13, 2026

🎲 Part 1 β€” Run-to-Run Variance (Seed Variance): The Problem Behind Irreproducible Models

πŸ“Œ The Problem in One Sentence

Models trained in PyTorch on the same dataset with the same seed produce different results, so model reproducibility could not be achieved β€” and similar offline metrics do not guarantee similar driving behavior.

πŸ” How the Problem Was Evidenced

The Week 45 results already showed that training in PyTorch with the same dataset produces different models, so reproducibility was not reached. To prove it, the same dataset was trained three times, obtaining the following results:

Run best_val_loss final train loss final val loss
First training0.0140670.0067120.014339
Second training0.0130500.0067640.013776
Third training0.0130850.0066330.013404

⚠️ The Key Observation

Although the metrics are similar, the driving results are quite different. Two models with nearly identical validation loss can behave completely differently on the road. This is the practical consequence of seed variance.

πŸ“š Literature Reviewed

The seed-sensitivity topic has been reviewed in depth. The following works cover exactly this line of research:

"torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision"

Works with ImageNet and CIFAR, makes the problem very clear, and also addresses the practice of taking the best "run" of a trained model. arXiv:2109.08203

"Sources of Irreproducibility in Machine Learning: A Review"

Analyzes the sources of irreproducibility and identifies random initialization as one of them. arXiv:2204.07610

"Measuring training variability from stochastic optimization using robust nonparametric testing"

Focuses on how to measure the impact of randomness. Both works point to the effects of seeds at the time of initializing training. arXiv:2406.08307

πŸ“Œ Status of the Problem

This is still an open problem within the research community. There is no single accepted solution; the practical approach is to reduce the stochastic effect and to measure with several seeds rather than trusting one run.

🧩 Part 2 β€” Proposed Solution: EMA and Ensembles

To reduce the stochastic effect of seed selection, two complementary approaches were found.

βœ… 2.1 β€” In Training: Exponential Moving Average (EMA)

EMA averages, for each training batch, the weight of every connection. This is done for each seed being trained, and the result is to reduce the effect of the seed during training, producing models with similar behavior; that is, it reduces the "long tail" effect (models very far from the mean) and the "black swan" effect (models with special behaviors).

The process is defined by the equation (Lakshmanan 2026, A Short Survey of Averaging Techniques in Stochastic Gradient Methods):

x_k = Ξ² Β· x_(kβˆ’1) + (1 βˆ’ Ξ²) Β· ΞΈ_k

Other alternatives that could be used are Polyak–Ruppert, Stochastic Weight Averaging (SWA), and Exponential Moving Average (EMA).

βœ… 2.2 β€” At Driving Time: Ensembles

The idea is to train between 5 and 7 models on the same dataset but with a different seed (a larger number does not seem to produce changes), and then, while driving, average the inference of these 5 models. This is called an ensemble, as indicated in "Machine Learning Prediction of Airfoil Aerodynamic Performance Using Neural Network Ensembles" (MDPI 2076-3417/15/14/7720).

πŸ› οΈ Part 3 β€” Implementation

3.1 β€” Training Determinism

⚠️ What Is Fixed Today and What Is Not

Source of randomness Status Why
Train / val split βœ… Fixed Uses np.random.RandomState(seed).
Initial weights, dropout, batch order βœ… Fixed Depend on torch.manual_seed(args.seed).
Augmentation (flip, brightness, translation) ⚠️ Fixed but fragile Uses Python's random, which the script does not seed. It is saved because PyTorch reseeds random in each DataLoader worker. With num_workers=0 it would stop being reproducible.
GPU convolutions (cuDNN) ❌ Not fixed Some algorithms, especially in gradient computation, sum in an order that varies between runs.

The training script was modified to fix the PyTorch seed, but the GPU computes convolutions with algorithms that do not guarantee bit-for-bit identical results. That minimal difference grows with every training step until it produces different models.

In main(), the line torch.manual_seed(args.seed) was replaced by:

random.seed(args.seed)
np.random.seed(args.seed)
torch.manual_seed(args.seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

The price paid for fixing the seed is that the training algorithm becomes somewhat slower (no deeper work was done in this direction during the week).

βœ… Verification: Determinism in Training Was Achieved

Several models were trained consecutively, verifying training parameters such as MAE and obtaining the same results. The final driving test also returned the same results, which is why determinism in training is now guaranteed.

3.2 β€” Search for Seed Independence

βœ… EMA Implementation

EMA was implemented with an exponential decay value Ξ² = 0.999.

βœ… Ensembles

Since Week 40, a dataset bank was consolidated, made up of 8 different compositions (different percentages per maneuver) and generated with 5 seeds for each composition (42, 123, 456, 789, 999). For this reason, 5 different models were trained for each composition (1 per seed).

In total, with the previous process, 8 compositions Γ— 5 dataset seeds Γ— 5 training seeds = 200 models were trained.

πŸ“Š Part 4 β€” Results

Offline metrics (training)

As shown in Part 1, the three repeated runs gave very similar offline metrics (best_val_loss β‰ˆ 0.0130–0.0141, final val loss β‰ˆ 0.0134–0.0143). The offline metrics alone do not separate good from bad models; that is why the online evaluation is the decisive one.

Online metrics (driving)

The file ranking_ensembles_run81.csv summarizes the results. In particular, the columns dataset, metros, segundos, fuera_pct, and cte_media.

How to read the online metrics
  • fuera_pct β€” percentage of frames outside the lane.
  • cte_media (cross-track error) β€” the lateral distance in meters between the car and the center of the lane, averaged over all frames. The driver looks for the closest point of the lane center and measures how far the car separates from it. 0 would be driving always perfectly centered. With 0.60 m, the car drives on average 60 cm from the center; with 0.97 m, almost one meter.
  • cte_p95 β€” the 95th percentile of that same distance. 95% of the time the car was closer to the center than this value, so it measures the worst moments, normally the curves. A 1.49 m indicates that in curves it opens up to almost 1.5 m.

πŸ“Œ Interpretation of the Ranking

The ranking is ordered by cte_media, from best (lowest) to worst. The best ensembles keep the car centered at roughly 0.54–0.69 m on average and complete the route (~430 m in ~105 s), with fuera_pct in the 1.5–6% range. The worst cases fall into two clearly different failure modes:

  • High lateral error without crashing β€” e.g., Comp7-seed999 (cte_media 1.22 m, fuera_pct 7.63%) and Comp6-seed999 (cte_media 1.05 m, fuera_pct 10.49%). The car completes the route but drifts far from the center.
  • Early termination / getting stuck β€” the last four rows (Comp2-seed789, Comp6-seed123, Comp5-seed123, Comp7-seed123, Comp7-seed42) travel only 62–103 m in 15–25 s with fuera_pct between 13% and 33% and very high cte_p95 (6.8–9.2 m). These are the seed-variance "black swans": models that fail early on the same route where other seeds succeed.

Key takeaway: the dataset composition and the training seed both matter, but the seed effect is large enough to turn a ~430 m run into a ~62 m run on the same composition (compare Comp7-seed456 at 430.5 m / cte_media 0.765 vs. Comp7-seed42 at 63.6 m / cte_media 2.017, and Comp7-seed123 at 63.4 m). This is exactly the run-to-run variance the week set out to reduce, and it justifies both EMA and ensembles.

# dataset metros segundos fuera_pct cte_media cte_p95 final_x final_y
1dataset_w40-62000_Comp0-seed42429.9104.96.240.53531.4903191.2108.3
2dataset_w40-62000_Comp0-seed999433.1105.61.890.58131.3576191.5108.9
3dataset_w40-62000_Comp0-seed789429.9104.83.580.60311.496192.2109.6
4dataset_w40-62000_Comp0-seed456430.51054.380.60951.5999191.2108.5
5dataset_w40-62000_Comp5-seed42430.9104.76.30.67311.5114189.3106.8
6dataset_w40-62000_Comp0-seed123431.1105.22.280.68581.5191192.2109.7
7dataset_w40-62000_Comp3-seed42431.7104.96.340.6941.4079188.8106.5
8dataset_w40-62000_Comp2-seed42430.1104.67.270.70411.4767189.3106.8
9dataset_w40-62000_Comp4-seed999432.9105.46.170.71411.7195189.1106.9
10dataset_w40-62000_Comp4-seed123432.2105.11.470.73211.5109191.8109.2
11dataset_w40-62000_Comp1-seed999430.8104.83.390.73671.4751191.9109.4
12dataset_w40-62000_Comp1-seed789430.9104.91.910.74951.4434192109.3
13dataset_w40-62000_Comp7-seed456430.5104.52.340.7651.3611191.6108.9
14dataset_w40-62000_Comp7-seed789429.5104.36.230.77271.5688190.9108
15dataset_w40-62000_Comp1-seed42429.6104.57.130.77571.4908190.4107.6
16dataset_w40-62000_Comp3-seed789429.1104.22.690.78271.5765192.2109.6
17dataset_w40-62000_Comp2-seed999428.11047.070.79161.557191.5108.7
18dataset_w40-62000_Comp1-seed456428.6104.16.580.81151.4442191.6108.8
19dataset_w40-62000_Comp4-seed456428.8104.23.170.81571.5732192.5110
20dataset_w40-62000_Comp2-seed456431.3104.82.240.81841.4348192.1109.5
21dataset_w40-62000_Comp5-seed456430.3104.52.870.82111.5002190.9108.2
22dataset_w40-62000_Comp3-seed999428.8104.16.720.83761.4873191.4108.6
23dataset_w40-62000_Comp4-seed42430.2104.46.750.83851.5101189.3106.8
24dataset_w40-62000_Comp3-seed456428.8104.23.790.84131.7091192.4109.9
25dataset_w40-62000_Comp4-seed789429.3104.34.890.85691.7443191.6108.8
26dataset_w40-62000_Comp3-seed123430.5104.62.630.86221.634192.2109.6
27dataset_w40-62000_Comp2-seed1234321051.570.88351.495192.1109.4
28dataset_w40-62000_Comp5-seed789427.61047.550.89951.5525191.9109.3
29dataset_w40-62000_Comp1-seed123430.8104.83.670.90061.5616191.7108.9
30dataset_w40-62000_Comp5-seed999432.1104.92.340.91141.4611190.2107.5
31dataset_w40-62000_Comp6-seed456432.1104.96.860.92371.7282187.4105.9
32dataset_w40-62000_Comp6-seed42431104.62.820.941.4806190.4107.7
33dataset_w40-62000_Comp6-seed789428.2103.96.540.95921.6033191.6109
34dataset_w40-62000_Comp6-seed999431.5104.810.491.05151.8844191.2108.5
35dataset_w40-62000_Comp7-seed999431.4104.97.631.21631.9574192.3109.7
36dataset_w40-62000_Comp2-seed789103.425.512.940.74311.675157.9306.3
37dataset_w40-62000_Comp6-seed12362.215.530.651.73946.894712.5315
38dataset_w40-62000_Comp5-seed1236215.531.391.74646.792212.5314.2
39dataset_w40-62000_Comp7-seed12363.415.831.961.91938.822812.4316.3
40dataset_w40-62000_Comp7-seed4263.615.833.122.01719.174512.5316.4

πŸ§ͺ What the Online Results Confirm

All 40 evaluated configurations correspond to the same dataset bank (Compositions 0–7, seeds 42/123/456/789/999). Fourteen of them (the last rows) fail early or drift far from the lane center. Because the number of models is small, these results are descriptive: they show that seed variance is large enough to change driving quality, but they do not yet quantify a confidence interval. That is exactly why EMA and ensembles are the next step: to shrink the long tail and the black swans before comparing configurations.

πŸ“Œ WEEK 46 SUMMARY – OCTOBER 13, 2026

🎯 Objective: Reproduce and document run-to-run (seed) variance in training, implement determinism, and reduce seed sensitivity through EMA and ensembles, then evaluate the resulting models online.

πŸ§ͺ Methodologies: Three repeated trainings with the same seed to expose variance; literature review of seed sensitivity (arXiv:2109.08203, arXiv:2204.07610, arXiv:2406.08307); deterministic training via random.seed, np.random.seed, torch.manual_seed, cudnn.deterministic=True, cudnn.benchmark=False; EMA with Ξ² = 0.999; ensemble evaluation over 8 dataset compositions Γ— 5 dataset seeds Γ— 5 training seeds = 200 models.

πŸ“ˆ Key Findings: Similar offline metrics (best_val_loss 0.0130–0.0141) produced very different driving behavior. Determinism in training was verified by re-training and re-testing consecutively. In the online ranking, the best ensembles complete the route (~430 m in ~105 s) with cte_media 0.54–0.69 m and fuera_pct 1.5–6%, while the worst seeds travel only 62–103 m with fuera_pct 13–33% and cte_p95 up to 9.2 m on the same compositions β€” a direct demonstration of seed variance.

βœ… Achievements: Deterministic training guaranteed and verified. EMA implemented (Ξ² = 0.999). A 200-model bank (8 compositions Γ— 5 dataset seeds Γ— 5 training seeds) trained. First online ranking of ensembles generated (ranking_ensembles_run81.csv) with interpretable best and worst cases.

❌ Limitations: Offline metrics do not predict driving quality. The online ranking is descriptive and still without confidence intervals. Determinism only holds within the same GPU/PyTorch/CUDA/cuDNN environment and does not recover previous runs. The frame_0 image display and the PilotNet brain connection inherited from Week 45 remain pending for the Behavior Metrics evaluation.

πŸ”œ Next Steps: Compare EMA models against non-EMA models on the same route to quantify the reduction of the long tail and black swans. Formalize the ensemble inference (5–7 models averaged) and re-run the ranking to check whether the worst seeds move toward the mean. Add a per-composition breakdown of cte_media and fuera_pct to separate dataset-composition effects from training-seed effects. Complete the Behavior Metrics frame_0 display and connect the PilotNet brain so that the best model (dataset_w40-62000_Comp0-seed42 in this ranking) can be cross-checked against the in-house LaneMonitor metrics.