Week 14 - Dataset Refinement and Comparative Evaluation of MobileNet and PilotNet
December 30, 2025
Enhancing dataset with challenging scenarios and comprehensive offline/online validation of both architectures
This week's work focused on refining the dataset to address autonomous driving issues identified during online testing. While offline tests for both PilotNet (TensorFlow) and MobileNet (PyTorch) models were satisfactory, online evaluations revealed errors in lane-return behavior and steering stability. To address these issues, the dataset was augmented with specific driving scenarios that exemplify cases where the autonomous controller performed poorly.
Figure 1: Balanced dataset composition. The "raw steering" distribution indicates that only 5 of the 7 originally defined categories represent the majority of actions. This differs from vehicle deviation, which requires all 7 categories.
1. Dataset Enhancement and Balancing:
The resulting balanced dataset (Figure 1) shows enrichment in the number of samples per category compared to Week 13. The steering data composition reveals that only 5 of the 7 original categories contain significant samples, while vehicle deviation data utilizes all 7 categories effectively. The balanced dataset contains 28,784 samples distributed across steering categories to address previous performance gaps.
2. Offline Testing Methodology:
Two evaluation scenarios were designed for offline testing:
SCENARIO 1: STRAIGHT - RIGHT - STRAIGHT - RIGHT
Evaluates the model's ability to follow expert driving commands in a sequence of straight driving followed by right turns.
SCENARIO 2: SOFT LEFT - STRAIGHT LEFT
Tests correction capability from a right-biased vehicle position, followed by straight driving and a left turn.
3. Offline Test Results:
Scenario 1: STRAIGHT - RIGHT - STRAIGHT - RIGHT
Figure 2: Offline results for MobileNet model in Scenario 1.
Figure 3: Offline results for PilotNet model in Scenario 1.
Scenario 2: SOFT LEFT - STRAIGHT - LEFT
Figure 4: Offline results for MobileNet model in Scenario 2.
Figure 5: Offline results for PilotNet model in Scenario 2.
4. Online Testing Results:
Both trained models (MobileNet and PilotNet) were tested in CARLA Simulator to validate their online performance:
MOBILENET (PYTORCH):
• Stable driving with appropriate lane correction
• Correct execution of turns
• Area for improvement: Response speed occasionally causes road departure and collisions with obstacles
PILOTNET (TENSORFLOW):
• Faster maneuver execution but less stable driving
• Correct turns but occasional premature turn initiation
• Issue: Early turns cause lane departure from the inner side
5. Preliminary Comparative Analysis:
PRELIMINARY COMPARATIVE TABLE
| Characteristic | PilotNet (TensorFlow) | MobileNet (PyTorch) |
|---|---|---|
| Framework | TensorFlow 2.13.0 | PyTorch |
| Architecture | Specific CNN (NVIDIA) | Pre-trained CNN (MobileNet) |
| Parameters | 804,203 | ~3-5 million (estimated) |
| Model Size | 3.07 MB | ~12-20 MB (estimated) |
| Training Approach | From scratch | Transfer Learning |
| Test MAE | 0.0462 | 0.0313337229 |
| Test Loss | 0.0051 | 0.006225 |
| Inference Speed | Fast (lightweight CNN) | Moderate |
| Resources | CPU only | CUDA/GPU |
| Dataset Used | 28,784 samples (balanced) | 28,784 samples (balanced) |
Conclusion:
This week's work successfully enhanced the dataset with challenging driving scenarios, resulting in improved model performance. The comparative analysis reveals complementary strengths: MobileNet offers stable driving with appropriate corrections but requires optimization for response speed, while PilotNet provides faster inference but exhibits stability issues and premature turns. The next phase will focus on model-specific optimizations—improving MobileNet's response time and refining PilotNet's turning behavior—while continuing to expand the dataset with edge cases to enhance overall autonomous driving robustness.