PilotNet in Behavior Metrics: Brain v2 vs. Own Driver — Seed Independence with EMA
October 27, 2026
📊 Part 1 — Overview
📌 Two Questions Closed This Week
- PilotNet in Behavior Metrics. Models that drove well in the training pipeline crashed quickly inside Behavior Metrics. A new brain (v2) replicates the own driver's semantics and was validated in Town02.
- Seed independence with EMA. Whether an EMA of the weights can make single models independent of the training seed, as the 5-seed ensemble had achieved.
Short answer: brain v2 works (428 m before understeering into a guardrail), but EMA 0.9999 does not deliver seed independence (14 of 60 single models still fail).
🛠️ Part 2 — Diagnosis and Brain v2
The initial hypothesis (FPS and sync/async mode) was only partly right. Three differences carried the weight:
| Aspect | Own driver | Previous brain in BM | Weight |
|---|---|---|---|
| Simulation mode | Synchronous, fixed_delta_seconds = 0.05, one tick() per decision | Asynchronous; brain decides every 50 ms real time | High |
| Image seen by model | From the current tick, via a frame-indexed queue | Last image delivered by callback, variable delay | High |
| Speed control | PID in simulated time (Kp 0.18, Ki 0.05, Kd 0.03), slows in curves, throttle up to 1.0 | Fixed thresholds of the old agent; throttle max 0.6; no braking in curves | High |
| Start-up | 60 ticks braked so the car settles | Drives from the first instant | Medium |
🐞 Three Additional Findings
- FPS do not matter. The driver enables sync mode and imposes 0.05 s per tick (20 Hz simulated).
- Speed control was a bug in the previous brain, which copied the old agent. Without braking in curves, the car enters fast and opens the arc.
- Behavior Metrics' sync mode does not work here. With
AsyncMode: Falseit enables sync mode without fixingfixed_delta_seconds, and two processes calltick()at once: the pilot and the world generator.
Solution: brain v2
Sync mode was moved into the brain. The YAML keeps AsyncMode: True; the brain is the only one advancing the simulator. Per step:
- On first play, enables
synchronous_modewithfixed_delta_seconds = 0.05and creates its own 800×600 camera with a frame-indexed queue. - Keeps the car braked for 60 ticks (3 s simulated) so it settles.
- At each decision calls
tick(), takes the image from that same frame and predicts steering. - Smooths steering and computes throttle/brake with the same PID as the own driver.
- Sends all three values in a single
apply_control.
Files added: brain_carla_pilotnet_pth.py (v2, supports ensembles), pilotnet_town02_sync.yml (same parameters as own driver, ClearNoon at 0 % cloudiness), and carla_async.py (restores async mode if CARLA is left frozen after an abrupt close).
📈 Part 3 — Run Results (20261027-012354)
Model run_20261025_233957_011743_1505f6_dataset_w40-62000_Comp0-seed0/model_best.pth. The car covered 428 m in 104.3 simulated seconds at a stable ~15 km/h, then hit a guardrail in a Town02 curve.
| Datum | Value |
|---|---|
| Map and weather | Town02; ClearNoon, sun at 90°, cloudiness 0 %, rain 0 % |
| Simulation step | Exactly 0.05 s in all 2186 rows, consecutive frames |
| Up to first crash | 104.25 s and 428.4 m |
| Speed before crash | Mean 14.8 km/h; percentiles 5–95: 14.2–15.2 km/h |
| Crash | static.guardrail at (191.7, 108.9), at 13.5 km/h |
| Off right lane | 125 ticks (6.3 s) in 2 stretches; broken-line crossings in 45 % of rows |
The last 3 seconds before the crash
| Time before crash | Speed (km/h) | Applied steering | Difference vs. lane heading |
|---|---|---|---|
| 3.0 s | 14.9 | -0.03 | -4° |
| 2.1 s | 14.7 | -0.09 | -6° |
| 1.5 s | 14.9 | 0.00 | 3° |
| 0.9 s | 14.2 | 0.23 | 22° |
| 0.3 s | 14.5 | 0.16 | 38° |
| 0 s | 2.9 | 0.20 | 48° |
📌 Interpretation
- Synchrony works. Every row advances exactly 0.05 s and one frame, same as the own driver.
- Speed control works. Speed stays between 14.2 and 15.2 km/h around the 15 km/h target.
- The car no longer crashes quickly. It covers 428 m and several curves before failing.
- The crash is understeer in a curve. In the last second the car's heading separates 48° from the lane, but applied steering never exceeds 0.23; the model asks for little wheel and the 0.3 smoothing delays the response further.
- The car tends to ride the center line. The broken line appears in 45 % of rows, matching the lateral bias toward the dividing line mentioned when describing
--random-spawn.
🐞 Caveats When Reading the Metrics
- Collisions are inflated. Behavior Metrics does not stop the run on crash, and the PID keeps accelerating against the guardrail; the 101 collision rows are a single crash.
lane_invasionmay carry its value across events. The 45 % is an indication of the bias, not an exact measure;in_right_laneis more reliable.
Next steps
- Run this same model in the own driver on CARLA 0.9.14 and compare meters, seconds and end point with Behavior Metrics: 428 m, 104 s, crash at (191.7, 108.9).
- Run the own driver against the 0.9.15 container, to measure the CARLA version effect.
- Stop the run at the first crash inside the brain, as the own driver does.
- Check whether
PerfectLapshould point to a Town02 route. - Test the 5-seed ensemble with
EnsembleModelsand compare with the ensemble ranking.
| Behavior Metrics | Own script (0.9.14) | Own script (0.9.15) | |
|---|---|---|---|
| Run | 428 m; crash at (191.7, 108.9) | Pending | Pending |
🔀 Part 4 — Upstream Integration Plan
Four independent contributions, each with its issue, branch and PR against noetic-devel, following CONTRIBUTING.md. The two smallest go first; sync mode, the most valuable and largest, goes last.
| Order | Change | Files | Suggested branch | Current state |
|---|---|---|---|---|
| 1 | Fix ROS_VERSION reading (trailing spaces in the variable name) | driver_carla.py | fix/NNN-ros-version-env | Detected, unfixed |
| 2 | Use an external or Docker CARLA without Behavior Metrics closing it (BM_EXTERNAL_CARLA) and without requiring CARLA_ROOT | utils/environment.py | feature/NNN-external-carla | Local patch working |
| 3 | Configurable weather from YAML (Simulation > Weather) | utils/carla_world_generator.py and an example YAML | feature/NNN-weather-config | Local patch working |
| 4 | Real sync mode in Python API: fix fixed_delta_seconds, a single process calling tick(), frame-synchronized image | pilot_carla.py, utils/carla_world_generator.py, utils/traffic.py, utils/configuration.py | fix/NNN-python-api-sync-mode | Solved inside the brain only; still to be moved to the core |
NNN is the GitHub issue number.
Phase 0 — Prepare the dev environment (once)
Work in a clean clone of the fork, separate from ~/BehaviorMetrics, so the working install is not mixed with branches.
- Fork at
https://github.com/JdeRobot/BehaviorMetrics/fork. - Clone it separately and link the original repository:
git clone https://github.com/TU_USUARIO/BehaviorMetrics ~/BehaviorMetrics-dev
cd ~/BehaviorMetrics-dev
git remote add upstream https://github.com/JdeRobot/BehaviorMetrics
git fetch upstream - Check the issue tracker for existing reports on each change.
Phase 1 — Cycle per change (repeat for 1, 2, 3)
- Issue with the six sections required by
CONTRIBUTING.md: expected behavior, current behavior, reproduction steps, environment, possible cause, possible solution. - Branch from the most recent version:
git fetch upstream
git checkout -b feature/NNN-external-carla upstream/noetic-devel - Port the change without copying
.bakor unrelated files. - Test with
default_carla_api.yml(example brain) and withpilotnet_town02_sync.yml, and record the results for the PR. - Commit and push, file by file:
git add behavior_metrics/utils/environment.py
git commit -m "Allow using an external CARLA server without killing it"
git push origin feature/NNN-external-carla - Pull request against
noetic-devel, withFixes #NNN, how it was tested and in which environment (Ubuntu 26.04, Python 3.10, CARLA 0.9.15 in Docker, no ROS). - Answer review in days or weeks, not months.
Phase 2 — Sync mode in the core (change 4)
This change touches central behavior, so agree on the approach with maintainers in the issue before coding.
- Issue with the evidence: with
AsyncMode: Falsefixed_delta_secondsis not fixed, pilot and world generator calltick()at once, and the brain receives the last image instead of the current tick's. Cite PR #719, which addressed asynchrony but not this. - Propose the design: a
FixedDeltakey in YAML; only the pilot callstick(); the world generator stops calling it; the camera delivers the current frame's image. - Implement using the v2 brain logic as reference.
- Validate with the trace: step of exactly 0.05 s, consecutive frames and two identical runs giving the same trajectory.
- Once integrated, simplify the PilotNet brain to use Behavior Metrics' sync mode instead of its own.
🐞 Not Proposed to the Repository
- The PilotNet brain as-is, because it depends on the project's
pilotnet_model.py; at most a generic version could be offered as an example. - Local install adjustments: PyTorch versions for the RTX 5070,
requirements-base.txtand pinned numpy.
⚠️ Limitations
- No ROS on Ubuntu 26.04, so the changes cannot be tested in ROS mode; this must be stated in every PR.
- Review and integration depend on maintainers; meanwhile, the local installation keeps working with the current patches.
🌱 Part 5 — Independence of the Training Seed with EMA
| Property | Meaning | State |
|---|---|---|
| Determinism | Same seed and data give the same model, bit for bit | Solved |
| Seed independence | Different seeds give models that drive the same | Not solved: the seed decides whether the car passes certain points of the route |
The symptom is a lottery: with the same dataset, one model passes point A (~60 m) and another, trained with a different seed, crashes there. The 5-seed ensemble had already attenuated this. Question: could EMA achieve the same for a single model?
Hypothesis: EMA as a filter
EMA keeps an exponential moving average of the weights and validates/saves that average, not the raw weights. It is a first-order IIR low-pass filter with pole at z = β:
The seed injects high-frequency noise; the filter attenuates it. With batch 32 and 52,700 images, one epoch is 1646 steps: β = 0.9999 has a ~6-epoch window. Expected limitation: the filter averages within one trajectory; if two seeds lead to different minima, it does not bring them together.
Experiment 1: β sweep
100 Comp0 datasets × β ∈ {0.99, 0.999, 0.9999} × 1 seed (42), cap 200 epochs; paired comparison dataset by dataset. It measured quality, not seed dependence.
| Metric (v2, common pool) | 0.99 | 0.999 | 0.9999 | 0.9999 vs. 0.999 |
|---|---|---|---|---|
| MAE | 0.0429 | 0.0429 | 0.0424 | −1.2 %, better in 82/100 |
| R² | 0.872 | 0.872 | 0.874 | +0.0025, better in 84/100 |
| Slope | 0.879 | 0.877 | 0.878 | +0.0016, no practical effect |
🐞 Defect Found and Fixed
The average started from the random weights of the freshly created model: with 0.9999 those weights still weighed 85 % after 1 epoch and 19 % after 10, lengthening early stopping and delaying the LR drops of the plateau scheduler. A bias correction was added (Adam-style, dividing by 1 − βᵗ); the average was verified to match Adam's formula and training remains reproducible bit for bit.
Experiment 2: 12 datasets × 5 seeds on the track
With β = 0.9999 and bias correction, 46 of 60 single models reached the gas station; the other 14 failed at point A (~60 m) or at a new point C (~230–250 m). 11 Comp0 datasets of 62,000 images and one of 309,000, seeds 42, 1, 2, 3, 4, cap 300 epochs, each model driven alone in Town02.
📌 Headline
14 of 60 single models do not reach the gas station; in 10 of 12 datasets at least one seed fails. Only two datasets arrived with all 5 seeds: Comp0-seed0 and the 309,000-image one.
| Dataset (Comp0) | s42 | s1 | s2 | s3 | s4 | Reach |
|---|---|---|---|---|---|---|
| seed0 | 431 m | 434 m | 430 m | 427 m | 434 m | 5/5 |
| seed1 | C·229 m | 431 m | 433 m | 435 m | C·228 m | 3/5 |
| seed2 | A·58 m | 437 m | 431 m | 430 m | 433 m | 4/5 |
| seed3 | 435 m | 434 m | 432 m | 433 m | A·57 m | 4/5 |
| seed4 | A·67 m | 432 m | 435 m | 428 m | A·67 m | 3/5 |
| seed5 | 431 m | 433 m | C·249 m | 431 m | 429 m | 4/5 |
| seed6 | 435 m | A·63 m | 433 m | 434 m | 433 m | 4/5 |
| seed7 | 434 m | 434 m | 433 m | 434 m | A·59 m | 4/5 |
| seed8 | 432 m | C·228 m | 431 m | 431 m | C·229 m | 3/5 |
| seed9 | 434 m | 426 m | A·61 m | 433 m | C·253 m | 3/5 |
| seed42 | 431 m | A·67 m | 432 m | 436 m | 434 m | 4/5 |
| 309 k images | 433 m | 437 m | 431 m | 430 m | 435 m | 5/5 |
Same data and parameters; only the seed changes. Unmarked = reaches B (~430 m); C = fails at C (~230–250 m); A = fails at A (~60 m).
Analysis
- Reaching the gas station. A single model reaches B 75 % of the time. The expected distribution if each seed were a fair coin matches the observed one: there are no "bad" datasets (5-of-5: 2.5 expected vs. 1; 4-of-5: 4.36 vs. 3; 3-of-5: 3.04 vs. 4; 2-or-fewer: 1.2 vs. 0).
- Centering. |cte| varies ±0.11 m between seeds of the same dataset and ±0.05 m between dataset means (ANOVA p = 0.56). The seed explains ~77 % of the variation and the dataset ~23 %.
- The large dataset. Its 5 seeds reached B and are well centered (|cte| 0.596 m). Suggests more data reduces seed dependence, but it is a single case and 5 of 5 by chance has a 27 % probability.
- By seed. s3 reached B in all 12 datasets and s4 in 6. With 12 datasets this may be coincidence; a seed is not good in itself.
🐞 Why EMA is not enough
- It attenuates high frequency, not the trajectory. The seed fixes initialization and data order, which decide which minimum training heads to; the filter averages within that trajectory and does not bring two trajectories together.
- Threshold decisions remain. Early stopping and the plateau scheduler cross their thresholds at different moments per seed, and from there the models diverge.
- Minimal offline differences, binary effect on the track. Slope changed ±0.002 between β values; in closed loop, a small difference in a critical curve decides between continuing and crashing.
- Points A and C are weak zones of the data. Any seed variation pushes the model one way or the other.
What follows
- Review the ranking of the 12 ensembles (
ranking_ensembles_run81_ema0.9999bc.csv): if datasets with 2 failing seeds still reach B as ensembles, the ensemble confirms itself as the solution. - Watch the failure videos at A and C to understand what the model sees there.
- Add data from stretches A and C, and from the gas station (B).
- Test whether more data reduces seed dependence with more than one large dataset.
- Try a cosine scheduler, which does not depend on validation and removes one of the threshold decisions.
📌 WEEK 48 SUMMARY – OCTOBER 27, 2026
🎯 Objective: Validate the new Behavior Metrics brain (v2) against the training pipeline, and determine whether EMA can make model quality independent of the training seed.
🧪 Methodologies: Brain v2 replicating the own driver's semantics inside Behavior Metrics (sync mode, frame-indexed camera queue, settle ticks, PID in simulated time, single apply_control), with AsyncMode: True in YAML. Upstream integration planned as four independent issues/branches/PRs against noetic-devel. EMA β sweep over 100 datasets, then 12 datasets × 5 seeds with bias correction.
📈 Key Findings: Brain v2 reproduces the own driver's conditions: exact 0.05 s steps, consecutive frames, correct weather and model. The model covered 428 m in 104.3 s before understeering into a guardrail at 48° heading error, with applied steering never above 0.23. Speed control works (14.2–15.2 km/h around a 15 km/h target). FPS do not influence behavior; sync mode, speed control and start-up settling were the real differences. EMA 0.9999 with bias correction is a minor quality gain (MAE −1.2 %, R² +0.0025) but does not deliver seed independence: 14 of 60 single models fail, and in 10 of 12 datasets at least one seed fails. The seed explains ~77 % of centering variation, the dataset ~23 %.
✅ Achievements: Behavior Metrics now drives under the same conditions as the training pipeline; the remaining gap is model behavior, to be compared with the own driver. Upstream contribution plan defined with four scoped changes. EMA bias-correction defect found and fixed; training remains reproducible bit for bit.
❌ Limitations: Collisions in Behavior Metrics are inflated (run does not stop on crash); lane_invasion may carry across events. The 0.9.14 vs. 0.9.15 comparison is still pending. No ROS on Ubuntu 26.04, so upstream changes cannot be tested in ROS mode. The large-dataset evidence for reduced seed dependence is a single case (5 of 5 by chance has 27 % probability).
🔜 Next Steps: Run the same model in the own driver on 0.9.14 and 0.9.15 and compare distance, time and crash point. Stop the run at first crash inside the brain. Open the first two upstream issues (ROS_VERSION fix, external CARLA). Review the 12-ensemble ranking and the failure videos at A and C. Add data at points A, B and C. Test a cosine scheduler and whether more data reduces seed dependence across multiple large datasets.