Robotics Lab URJC

Programming Robot Intelligence

Week 38 · July 24, 2026

Randomness Seed Impact on Dataset Construction & Model Performance

July 24, 2026

Overview

Based on the results from the previous week, it was observed that while the generated datasets maintained proportionality of samples across Towns and manoeuvres, they did not produce true randomness due to the use of a single fixed seed (42) in all scripts.

This week's work plan focused on generating additional datasets by varying the randomness seed during dataset construction to identify the impact of this seed on the driving performance of the trained PilotNet model.

Experimental Setup

The same seven dataset compositions from the previous week were compared:

CompositionBase (Week 36)Town 03Town 07Town 12
1✅✅✅✅
2✅✅✅❌
3✅✅❌❌
4✅✅❌✅
5✅❌✅✅
6✅❌❌✅
7✅❌✅❌

In the previous week, all these compositions were built using a fixed randomness seed of 42. For this week, the same dataset compositions were constructed but with 3 different seeds:

123
Seed 1
456
Seed 2
999
Seed 3

For each dataset constructed with these seeds, the procedure from Weeks 36 and 37 was applied to obtain trained PilotNet models.

📊 Results

Overall, improvements were obtained compared to the previous week's fixed‑seed results. The two compositions that delivered the best performance were:

Composition Seed Key Improvement Performance
Composition 7
(Base + Town 07)		 999 Reduced confusion on light‑coloured walls and open spaces BEST
Composition 3
(Base + Town 03)		 999 Reduced confusion on forests and open spaces BEST

🎯 Key Findings

  • Composition 7 with seed 999 – effectively reduced confusion in light‑coloured walls and open spaces, resulting in improved autonomous driving with fewer lane departures and longer autonomous driving segments between errors.
  • Composition 3 with seed 999 – delivered similar improvements for forests and open spaces, also achieving more stable autonomous driving with reduced errors.

⚠️ Important Observation

Both compositions using seed 999 showed clear improvements over the Week 36 baseline model. This confirms that the randomness seed plays a significant role in the quality and generalisation capability of the resulting model, even when the dataset composition and proportions remain identical.

🎥 Validation Videos

Composition 7 – Seed 999
youtu.be/IQJnew2bBQY

Improved performance on light‑coloured walls and open spaces. Fewer lane departures, longer autonomous segments.

Composition 3 – Seed 999
youtu.be/1Znogmp2KU8

Improved performance on forests and open spaces. Stable driving with reduced confusion.

Both videos demonstrate autonomous driving with fewer errors and longer continuous driving segments compared to the previous week's fixed‑seed results.

📈 Comparison with Week 37

Week 37 (fixed seed 42): Composition 7 showed the best results, but still exhibited confusion on light‑coloured walls, plazas, and open spaces.

Week 38 (seed 999): Composition 7 with seed 999 reduced confusion on light‑coloured walls and open spaces, achieving better overall performance.

New finding: Composition 3 with seed 999 emerged as a strong performer, particularly for forests and open spaces – a composition that was not competitive in the previous week.

💡 Key Insight

The randomness seed significantly influences which dataset composition yields the best model. A composition that performs poorly with one seed can become the best performer with a different seed. This highlights the importance of multiple random realisations when evaluating dataset construction strategies.

📌 Conclusions

  1. Seed matters: The randomness seed has a significant impact on model performance, even when dataset composition and proportions remain identical.
  2. Best results with seed 999: Both Composition 7 and Composition 3 with seed 999 outperformed the previous week's fixed‑seed results.
  3. Composition 7 (seed 999): Reduced confusion on light‑coloured walls and open spaces, with longer autonomous driving segments.
  4. Composition 3 (seed 999): Emerged as a strong alternative, particularly effective for forests and open spaces.
  5. Practical implication: When constructing datasets for autonomous driving, multiple random seeds should be tested to identify the most robust configuration, rather than relying on a single fixed seed.

🔜 Next Steps (Week 39)

  • Further seed exploration: Test additional seeds to confirm the robustness of the findings and identify optimal seed values.
  • Combine best compositions: Investigate whether combining Composition 3 and Composition 7 (both with seed 999) yields even better performance.
  • Complete robustness tests: Finalise the weather variation tests (morning/evening conditions) that were partially completed in Week 37.
  • Evaluate on additional towns: Test the best models on unseen towns (e.g., Town 06, Town 10) to assess generalisation.

📌 WEEK 38 SUMMARY – JULY 24, 2026

🎯 Objective: Vary randomness seed (123, 456, 999) to evaluate impact on dataset quality and model performance.

🧪 7 compositions tested × 3 seeds = 21 datasets/models evaluated.

🏆 Best results: Composition 7 (Base + Town 07) and Composition 3 (Base + Town 03) both with seed 999.

✅ Composition 7 (seed 999): Reduced confusion on light‑coloured walls and open spaces.

✅ Composition 3 (seed 999): Reduced confusion on forests and open spaces.

📈 Key insight: Seed choice significantly influences which composition performs best – multiple seeds are essential for robust evaluation.

🎥 Videos: Comp 7 (seed 999) · Comp 3 (seed 999)

References

  • [1] Bojarski, M., et al. (2016): End to end learning for self‑driving cars (PilotNet). arXiv:1604.07316
  • [2] Mateus, A. (2026): Week 36 report – Validation in Town02: Stability Gains & Remaining Challenges.
  • [3] Mateus, A. (2026): Week 37 report – Dataset Composition Experiments & Robustness Testing.
  • [4] CARLA Simulator (2026): Documentation – Random seed effects on dataset sampling.

— Armando Mateus, Robotics Lab URJC