Robustness Evaluation: Temporal Condition Variations (Morning & Evening)
August 7, 2026
Overview
Building upon the findings from the previous weeks (Week 36: driving confusion in forests and plazas; Weeks 37-38: improvements in confusion but with lingering issues), this week marks the commencement of a systematic evaluation of model robustness. The primary objective is to assess the performance of the trained PilotNet models under varying operational conditions.
Two principal scenarios are considered for robustness assessment:
- 1. Temporal Condition Changes: The models were trained on datasets collected at midday (12:00). This week, we evaluate performance at 9:00 (morning) and 16:00 (evening) to test sensitivity to illumination changes.
- 2. Start Position & Orientation: While the models were primarily trained with appropriate orientation and lane-centred starts (though incorporating drunk-dagger for departures), a dedicated evaluation of robustness to non-optimal starting poses is planned for future work.
This report focuses exclusively on the temporal condition variations.
Methodology
To systematically evaluate robustness to illumination changes, three distinct times of day were selected for evaluation: 12:00 (noon), 9:00 (morning), and 16:00 (evening).
The benchmark models are the best-performing architectures from Weeks 36 and 37-38. New datasets for morning (9:00) and evening (16:00) conditions were generated following the same methodology used for the Week 36 dataset construction, ensuring consistency in data collection protocols.
For reference, the training configurations of the benchmark models are summarised below:
- Week 36 Model: Trained on midday data (62,000 examples from Towns 01 and 04). Composition: 45% straight, 15% left turns, 15% right turns, 25% soft drunk-dagger.
- Weeks 37-38 Model: Trained on midday data (62,000 examples from Towns 01, 04, and 07). Composition: 45% straight, 15% left turns, 15% right turns, 25% soft drunk-dagger.
Results: Week 36 Benchmark Model
The following tables summarise the performance of the Week 36 model across the three temporal conditions.
12:00 (Noon)
- Frames in Lane: 2201 (89.3%)
- Frames Out of Lane: 259 (10.7%)
- Lane Departures: 2 events
- Lane Invasions: 2 events
- Collisions: 1 (Execution terminated)
1. Duration: 10.67s, Frames: 177, MaxSteering: 0.345
2. Duration: 4.68s, Frames: 82, MaxSteering: 0.174
9:00 (Morning)
- Frames in Lane: 489 (61.1%)
- Frames Out of Lane: 306 (38.9%)
- Lane Departures: 0 events
- Lane Invasions: 1 event
- Collisions: 2 (Execution terminated)
16:00 (Evening)
- Frames in Lane: 104 (58.9%)
- Frames Out of Lane: 71 (41.1%)
- Lane Departures: 0 events
- Lane Invasions: 1 event
- Collisions: 2 (Execution terminated)
⚠️ Observation
The Week 36 model shows a significant degradation in performance under morning and evening conditions, with a marked increase in frames out of lane and a higher incidence of collisions. The model, trained exclusively on midday data, does not generalise well to different illumination levels.
Results: Weeks 37-38 Benchmark Model
The following tables summarise the performance of the Weeks 37-38 model.
12:00 (Noon)
- Frames in Lane: 3303 (64.8%)
- Frames Out of Lane: 1798 (35.2%)
- Lane Departures: 3 events
- Lane Invasions: 3 events
- Collisions: 0
1. Duration: 8.63s, Frames: 139, MaxSteering: 0.280
2. Duration: 95.29s, Frames: 1628, MaxSteering: 0.195
3. Duration: 1.75s, Frames: 31, MaxSteering: 0.317
9:00 (Morning)
- Frames in Lane: 782 (46.1%)
- Frames Out of Lane: 905 (53.9%)
- Lane Departures: 2 events
- Lane Invasions: 3 events
- Collisions: 1 (Execution terminated)
1. Duration: 28.14s, Frames: 469, MaxSteering: 0.244
2. Duration: 2.50s, Frames: 43, MaxSteering: 0.222
16:00 (Evening)
- Frames in Lane: 680 (73.8%)
- Frames Out of Lane: 241 (26.2%)
- Lane Departures: 1 event
- Lane Invasions: 2 events
- Collisions: 1 (Execution terminated)
1. Duration: 12.17s, Frames: 208, MaxSteering: 0.213
📊 Analysis
The Weeks 37-38 model, despite including a third town (Town 07) in its training set, still exhibits a notable performance drop in morning conditions (53.9% frames out of lane). Evening performance (26.2% out of lane) is comparatively better but still inferior to the midday results.
Supplementary Results: Week 13 Models
For completeness, the performance of three Week 13 models (optimised for noon, morning, and evening conditions, respectively) was also evaluated. These models were trained on a smaller, earlier dataset.
Week 13 – Best Noon Model
- Noon: Video – In Lane: 21.7%
- Morning: Video – In Lane: 33.3% (Lane Dep: 2)
- Evening: Video – In Lane: 60.9% (Lane Dep: 2)
Week 13 – Best Morning Model
- Noon: Video – In Lane: 40.2% (Collisions: 2)
- Morning: Video – In Lane: 25.6% (Lane Dep: 1, Coll: 2)
- Evening: Video – In Lane: 66.7% (Coll: 2)
Week 13 – Best Evening Model
- Noon: Video – In Lane: 95.8% (Coll: 2)
- Morning: Video – In Lane: 39.9% (Lane Dep: 1, Coll: 2)
- Evening: Video – In Lane: N/A (Coll: 2, Curves: 64.8% open)
Discussion
Performance Degradation: Both the Week 36 and Weeks 37-38 models exhibit a clear degradation in performance when evaluated under morning and evening conditions compared to midday. This is particularly pronounced in the morning (9:00), where lighting conditions (shadows, lower sun angle) differ most significantly from the training distribution.
No Universal Model: The results suggest that a single model trained exclusively on midday data is not robust to temporal illumination changes. While the Weeks 37-38 model (trained with an additional town) showed improved midday performance and slightly better evening generalisation, it still struggled in the morning.
Condition-Specific Specialisation: The Week 13 models, trained for specific times of day, demonstrate that a model can be specialised for a particular condition. However, this comes at the cost of performance in other conditions, confirming the absence of a universal solution with the current approach.
💡 Key Insight
To achieve robust performance across different times of day, training data must include representative samples from each target condition. Simply increasing the number of towns in the training set (while keeping the time fixed) does not confer robustness to illumination changes.
📌 Conclusions
- Robustness Gap: Models trained exclusively on midday data exhibit a significant performance drop (≥20% reduction in in-lane frames) when evaluated under morning and evening conditions.
- Conditional Training Required: The best performance for a specific time of day is achieved by training on data from that same time. A model trained on mixed temporal data is necessary for universal robustness.
- Degradation Patterns: Morning conditions (9:00) consistently pose the greatest challenge, likely due to more dramatic illumination changes (shadows, low sun angle) compared to the midday baseline.
- No Single Best Model: There is no universal model that performs optimally across all conditions with the current dataset composition. The choice of model must be tailored to the expected operational environment.
🔜 Future Work
- Mixed Temporal Training: In the coming weeks, we will construct datasets that include samples from morning, midday, and evening conditions to train a model with improved temporal robustness.
- Start Position & Orientation Robustness: We will conduct a systematic evaluation of model sensitivity to non-optimal starting positions and orientations, extending beyond the current drunk-dagger approach.
- Weather Variation: Future work will also investigate robustness to weather conditions (rain, fog) as a complementary dimension of environmental variation.
📌 WEEK 39 SUMMARY – AUGUST 7, 2026
🎯 Objective: Evaluate robustness of PilotNet models to temporal illumination changes (morning, midday, evening).
🧪 Models Evaluated: Week 36 (Towns 01,04) and Weeks 37-38 (Towns 01,04,07) benchmarks.
📉 Key Finding: Both models show significant performance degradation under morning and evening conditions, with morning (9:00) being the most challenging.
🔄 Complementary Findings: Week 13 condition-specific models confirm that training on target conditions yields better performance, but no single model generalises universally.
📊 Performance Drop (Week 36): Noon (89.3% in-lane) → Morning (61.1%) → Evening (58.9%).
📊 Performance Drop (Weeks 37-38): Noon (64.8% in-lane) → Morning (46.1%) → Evening (73.8%).
🔜 Next Steps: Construct mixed-temporal datasets and evaluate robustness to starting position/orientation.
References
- [1] Bojarski, M., et al. (2016): End to end learning for self‑driving cars (PilotNet). arXiv:1604.07316
- [2] Mateus, A. (2026): Week 36 report – Validation in Town02: Stability Gains & Remaining Challenges.
- [3] Mateus, A. (2026): Week 37-38 report – Dataset Composition Experiments & Robustness Testing.
- [4] CARLA Simulator (2026): Documentation – Environmental condition simulation (time of day, weather).
— Armando Mateus, Robotics Lab URJC