PilotNet PyTorch vs TensorFlow: Root-Cause Diagnosis and Lateral Recovery Fix
September 15, 2026
Overview
This week's work focused on a deep diagnostic of the PyTorch PilotNet implementation, which consistently underperformed the existing TensorFlow reference model when driving in Town02 (an unseen map). What initially looked like a curve-steering problem turned out to be a chain of hidden bugs: a broken module import, a silently wrong data augmentation, and misleading offline metrics. This report documents the root causes found and the fix applied.
๐ Context
- Goal: Imitation learning with PilotNet, trained on Town01+Town04, tested on Town02. Match a pre-existing TF model (
train00_datasets_for_models.py) that generalizes to Town02. - Environments:
venv_py37(CARLA + PyTorch cu117, CPU inference, also TF 2.11.0) for driving;venv_py310(torch 2.11.0+cu128, RTX 5070) for training. - CARLA: Docker, port 2001.
- Documented "solved" recipe: V3 architecture (BatchNorm+Dropout+tanh, 120ร160, normalization
[-1,1]),weighted_steering_loss(turn_weight=2.0), augmentation (flip 50%, brightness 40%, lateral translation 30%), camerax=1.5, y=0, z=2.4.
ยง1 Timeline: How We Got Here
The project objective: a PyTorch PilotNet, trained on Town01+Town04, that generalizes to Town02 as well as the reference TensorFlow model. This is how we arrived at the current state.
- The import pointed to the wrong file:
common/pilotnet_model.pyhad the old architecture (no BatchNorm). The corrected version existed, but in another folder โ it was never copied to the location from which it is actually imported. - Three fine-tuning experiments, three dead ends: Increasing
turn_weight, matching BatchNorm momentum/epsilon to Keras, matching weight initialization โ none significantly moved the bias. - Offline comparison did not predict real driving:
validate_curvas.pyshowed TF with worse frame-to-frame error than PyTorch. But when actually driving in CARLA, TF went much further. We had to compare driving, not just isolated predictions. - Driving drivers had their own bugs: Collisions counted 61 times for a single crash,
cross_track_errorhidden when it was most needed, third-person camera disabled by default. - The real bug: PIL inverts the sign of the recovery correction. The "recovery" augmentation (lateral translation + steering correction) copied the TF formula, but implemented with a library that moves the image in the opposite direction. The model learned to reinforce drift, not correct it.
- Current status: Bias in tight curves dropped ~19% after the fix. The car no longer crashes at the same point โ but it still struggles with wide curves. Research ongoing.
๐ Quick Stats
56 m traveled by PyTorch before first crash (original model) ยท 431 m traveled by TF before its own crash ยท 4 implementation bugs found and fixed ยท 1 underlying problem still unsolved
Root Cause 1: Broken Import โ The Wrong Architecture Was Training All Along
โ ๏ธ The Bug
common/pilotnet_model.py โ the file actually imported during both training and driving โ contained the old V1 architecture (no BatchNorm/Dropout/tanh, 66ร200 input, normalization [0,1]). The correct V3 file (BatchNorm, 120ร160, [-1,1]) existed in the project root but was never copied into common/.
Impact: Every previous PyTorch training run used the outdated architecture without anyone noticing. The documented "V3 recipe" was never actually executed.
โ Fix Applied
cp pilotnet_model.py common/pilotnet_model.py
Confirmed with check_model_load.py: old checkpoints, trained unintentionally with V1, stopped loading against the new architecture (as expected).
Additionally, 3 more duplicated files with the same pattern were found and cleaned: entrenar_definitivo.py, check_model_load.py, and driverAgent_PyTorch_fixed.py inside common/ (all obsolete).
Design Decision: Constant Speed in Curves
๐ Decision
--curve-speed-factor (braking in curves) was discarded. It is not faithful to the reference TF driver, which maintains a fixed target speed (15 km/h) regardless of steering.
ยง3 Three Experiments That Did Not Solve the Problem
With the architecture already corrected, three adjustments based on real differences between PyTorch and TF were tested. None resolved the underestimation bias in tight curves โ measured with validate_curvas.py, comparing prediction error segmented by steering magnitude.
| Change Tested | Tight-Curve Bias (before โ after) | Improvement |
|---|---|---|
turn_weight 2.0 โ 4.0 |
-0.104 โ -0.097 | ~7% |
BatchNorm momentum/epsilon = Keras |
-0.104 โ -0.101 | ~3% |
Weight initialization (he_normal/glorot) |
No significant change | โ |
Offline Validation Revealed Systematic Bias
๐ Finding
validate_curvas.py (custom script measuring error per steering-magnitude bin) showed the PyTorch model underestimates steering in tight curves โ a consistent bias of roughly -0.10 to -0.12 across all runs (n=400 samples).
Key Turning Point: Real Comparison Against TF with Correct Data
๐ What Happened
After fixing a dataset mismatch (the reference .h5 was from w41_comp4_seed123, not w40_comp7), validate_curvas_tf.py (same methodology, but for TF) was run on the correct dataset.
- TF performed worse than PyTorch in all 4 bins, including "straight" โ contradicting the premise that "TF is more accurate frame-to-frame".
- But in real CARLA driving (not just offline validation), TF drove 431 m without crashing; PyTorch crashed at 56 m.
โ ๏ธ Conclusion
Frame-to-frame offline error does not predict closed-loop behavior well. Real driving had to be compared, not just validate_curvas.py.
Real Crash-Position Comparison (with loc_x/loc_y/yaw already logged)
Running real driving with recorded position (loc_x, loc_y, yaw) made it possible to pinpoint exactly where and how each model fails โ with real data from the runs, not just estimates.
| Model | Crash Distance | Crash Location | Cause |
|---|---|---|---|
| TF | 431 m | Guardrail (xโ193, yโ110) |
Insufficient sustained steering in a curve โ even the reference model fails there. |
| PyTorch | 56 m | Pole (xโ10, yโ308) |
Progressive, sustained lateral drift (0.22 m โ 1.34 m in ~17 s) with no active correction, until hitting an obstacle within the tolerated threshold (2.8 m). |
โ ๏ธ Diagnosis
It is not "PyTorch turns worse in tight curves" in general โ it is that it does not actively correct a lateral drift once it starts, a classic imitation-learning symptom (the model barely sees "recovery" examples in normal driving data).
loc_x/loc_y from CARLA), in the zone where all three models end up diverging. TF stays on the lane axis the whole stretch. The pre-fix model deviates only ~1.3 m from center โ almost imperceptible โ but that's enough to hit a pole. The post-fix model corrects the subtle drift, but opens up much more in the following curve: it improved a symptom, not the underlying cause.
๐ Reading the Graph
Before the fix: PyTorch stayed almost glued to TF's path โ the problem was a minimal, progressive lateral deviation, with no correction, just enough to graze a specific obstacle.
After the fix: That behavior changed: the car no longer drifts subtly, but in the next wide curve it underestimates the required turning radius and opens up several meters โ a different symptom, with the same underlying root (curves where the required steering is more subtle than an obvious tight turn).
ยง6 Bugs Found and Fixed in the Driving Drivers
These appeared while instrumenting both drivers to diagnose the above. None change the model's behavior, but without fixing them the diagnostic data was misleading.
- Collisions counted up to 61 times for a single crash: CARLA's collision sensor can re-fire on every tick while the car remains in contact during the post-crash braking window. Without a guard, each re-fire added a collision โ one report said "61 collisions" when exactly one crash had occurred. Fixed with an
is_first_collisionguard in both drivers. cross_track_errorhidden when it mattered most: The CSV forced that column to0.0wheneveris_inside_lane=Trueโ exactly when it was most useful to see how far from center the car was before crossing the alert threshold. It also wasn't updated in the "Against traffic" branch, leaving a previous frame's value. Fixed in both drivers.- Third-person camera disabled by default (TF driver): Gated by a
run_number >= 2condition that never triggered unless--runwas passed explicitly. Reports were also saved to~/Videos/model_reports/without any console notice. Fixed. - Seed extraction regex failed on folder names without a trailing separator (
w41_comp4_s123โ seed lost). Fixed.
All four were corrected with guards and explicit logging in both drivers (driverAgent_PyTorch_fixed.py and driverAgent_H5_35-crop_simple_BCK.py).
ยง5 Root Cause 2 (Most Likely): Inverted Sign in Recovery Augmentation
โ ๏ธ The Bug
The "recovery" augmentation shifts the image laterally and corrects the steering accordingly โ the standard technique to teach an imitation-learning model to return to the lane center. TF implements it with cv2.warpAffine; PyTorch, with PIL.Image.transform. Same formula, same translation sign โ but the two libraries interpret that matrix in opposite directions.
Empirically verified with a synthetic image (central white stripe) before touching the code: with the same matrix (1,0,tx,0,1,0) and tx=+5, cv2 shifts the content to the right; PIL shifts it to the left. Opposite directions, same code.
Result: in the ~30% of training frames where this augmentation was applied, the image moved to one side but the steering correction was still calibrated for the opposite side. The model did not learn "I'm off-center, correct toward the center" โ it learned, in those frames, exactly the opposite.
โ Fix Applied
Negate translation in the PIL matrix: (1, 0, -translation, 0, 1, 0), keeping the steering-correction formula identical to TF.
๐ Measured Impact
Retraining with the fix (same dataset, same turn_weight=2.0): tight-curve bias dropped from -0.119 to -0.096 (~19% improvement) โ the largest improvement of all changes tested so far, and the first time in the entire investigation that a change is clearly visible in validate_curvas.py.
Current Status
๐ Retraining in Progress
Retraining is underway on dataset_w41-74400_Comp4-seed123 with the sign fix applied, turn_weight=2.0, and the V3 architecture with BatchNorm/init already matched to TF.
| Item | Status |
|---|---|
V3 architecture (BatchNorm/Dropout/tanh) actually running in common/ |
โ Done |
| BatchNorm momentum/epsilon and initialization matched to TF | โ Done |
| Sign bug in recovery augmentation fixed | โ Done |
| Both drivers instrumented with real position and no counting bugs | โ Done |
Confirm whether new model avoids original crash point (x=10, y=308) |
โ Pending |
| Diagnose wide curve where post-fix model still opens ~10 m | โ Pending |
| Capture real camera frame at divergence point for direct visual inspection | โ Pending |
Try higher --steering-smoothing as additional lever, without retraining |
โ Pending |
Next Steps When Training Finishes
๐ Validation Checklist
check_model_load.pyโ confirm correct load (58 keys).config.jsonโ confirmturn_weight: 2.0, reviewbest_val_loss/epochs.validate_curvas.pyโ check whether the tight-curve bias decreased.- The test that really matters: run
driverAgent_PyTorch_fixed.pyat the same spawn (--duration 300 --curve-speed-factor 0) and see whether the car corrects the drift beforexโ10, yโ308instead of crashing into the pole at 56 m.
Workaround: Parallel TensorFlow Training
๐ Alternative Path While PyTorch Diagnosis Continues
Given that training with PilotNet (PyTorch) still does not produce models with the same performance, models are being trained with TensorFlow as a workaround (each one takes approximately 3 hours). In total, the training of 25 models has been planned, and they have not yet finished training.
๐ Dataset Composition
Following the training philosophy, progressive integrations have been developed with 4,600 / 6,600 / 10,000 / 12,000 / 20,000 lane-recovery examples.
Objective: Find the number of recovery samples with the best performance, aiming to provide recovery capability to the best-performing model obtained so far (Week 41, comp4 seed 123).
๐ Rationale
The PyTorch sign bug (PIL vs cv2) was the root cause of the failed recovery behavior. While the retrained PyTorch model is being validated, the TF pipeline is being leveraged to systematically explore the effect of recovery-sample volume on closed-loop performance โ isolating the dataset-size variable from the framework-implementation variable.
| Recovery Examples | Status |
|---|---|
| 4,600 | ๐ Training |
| 6,600 | ๐ Training |
| 10,000 | ๐ Training |
| 12,000 | ๐ Training |
| 20,000 | ๐ Training |
๐ WEEK 42 SUMMARY โ SEPTEMBER 15, 2026
๐ฏ Objective: Diagnose why the PyTorch PilotNet underperforms the TF reference model on the unseen Town02 map, and fix the root causes.
๐งช Methodologies: (1) Module-import audit and architecture verification; (2) offline curve-error validation against both models; (3) real CARLA closed-loop driving comparison with crash-position logging.
๐ Key Findings: The wrong (V1) architecture was silently being trained; offline frame-to-frame error does not predict closed-loop driving; the PyTorch model fails due to uncorrected lateral drift, not curve steering per se; PIL and cv2 interpret affine translation matrices in opposite directions, inverting the recovery augmentation signal in ~30% of training frames.
โ Achievements: Root cause identified โ an inverted sign in PIL-based lateral-translation augmentation was teaching the model to reinforce drift instead of correcting it. Fix applied and retraining started. Tight-curve bias reduced ~19% (from -0.119 to -0.096). Four additional driver bugs fixed.
โ Limitations: Real driving performance still being validated; the wide-curve scenario remains challenging for the post-fix model; plaza and complex-intersection scenarios remain difficult for both models.
๐ Next Steps: Complete retraining, run the full validation checklist, confirm drift correction in closed-loop CARLA driving, and diagnose the remaining wide-curve weakness.