Week 48 · October 27, 2026

PilotNet in Behavior Metrics: Brain v2 vs. Own Driver — Seed Independence with EMA

October 27, 2026

📊 Part 1 — Overview

📌 Two Questions Closed This Week

  • PilotNet in Behavior Metrics. Models that drove well in the training pipeline crashed quickly inside Behavior Metrics. A new brain (v2) replicates the own driver's semantics and was validated in Town02.
  • Seed independence with EMA. Whether an EMA of the weights can make single models independent of the training seed, as the 5-seed ensemble had achieved.

Short answer: brain v2 works (428 m before understeering into a guardrail), but EMA 0.9999 does not deliver seed independence (14 of 60 single models still fail).

🛠️ Part 2 — Diagnosis and Brain v2

The initial hypothesis (FPS and sync/async mode) was only partly right. Three differences carried the weight:

AspectOwn driverPrevious brain in BMWeight
Simulation modeSynchronous, fixed_delta_seconds = 0.05, one tick() per decisionAsynchronous; brain decides every 50 ms real timeHigh
Image seen by modelFrom the current tick, via a frame-indexed queueLast image delivered by callback, variable delayHigh
Speed controlPID in simulated time (Kp 0.18, Ki 0.05, Kd 0.03), slows in curves, throttle up to 1.0Fixed thresholds of the old agent; throttle max 0.6; no braking in curvesHigh
Start-up60 ticks braked so the car settlesDrives from the first instantMedium

🐞 Three Additional Findings

  • FPS do not matter. The driver enables sync mode and imposes 0.05 s per tick (20 Hz simulated).
  • Speed control was a bug in the previous brain, which copied the old agent. Without braking in curves, the car enters fast and opens the arc.
  • Behavior Metrics' sync mode does not work here. With AsyncMode: False it enables sync mode without fixing fixed_delta_seconds, and two processes call tick() at once: the pilot and the world generator.

Solution: brain v2

Sync mode was moved into the brain. The YAML keeps AsyncMode: True; the brain is the only one advancing the simulator. Per step:

  1. On first play, enables synchronous_mode with fixed_delta_seconds = 0.05 and creates its own 800×600 camera with a frame-indexed queue.
  2. Keeps the car braked for 60 ticks (3 s simulated) so it settles.
  3. At each decision calls tick(), takes the image from that same frame and predicts steering.
  4. Smooths steering and computes throttle/brake with the same PID as the own driver.
  5. Sends all three values in a single apply_control.

Files added: brain_carla_pilotnet_pth.py (v2, supports ensembles), pilotnet_town02_sync.yml (same parameters as own driver, ClearNoon at 0 % cloudiness), and carla_async.py (restores async mode if CARLA is left frozen after an abrupt close).

📈 Part 3 — Run Results (20261027-012354)

Model run_20261025_233957_011743_1505f6_dataset_w40-62000_Comp0-seed0/model_best.pth. The car covered 428 m in 104.3 simulated seconds at a stable ~15 km/h, then hit a guardrail in a Town02 curve.

DatumValue
Map and weatherTown02; ClearNoon, sun at 90°, cloudiness 0 %, rain 0 %
Simulation stepExactly 0.05 s in all 2186 rows, consecutive frames
Up to first crash104.25 s and 428.4 m
Speed before crashMean 14.8 km/h; percentiles 5–95: 14.2–15.2 km/h
Crashstatic.guardrail at (191.7, 108.9), at 13.5 km/h
Off right lane125 ticks (6.3 s) in 2 stretches; broken-line crossings in 45 % of rows

The last 3 seconds before the crash

Time before crashSpeed (km/h)Applied steeringDifference vs. lane heading
3.0 s14.9-0.03-4°
2.1 s14.7-0.09-6°
1.5 s14.90.003°
0.9 s14.20.2322°
0.3 s14.50.1638°
0 s2.90.2048°

📌 Interpretation

  • Synchrony works. Every row advances exactly 0.05 s and one frame, same as the own driver.
  • Speed control works. Speed stays between 14.2 and 15.2 km/h around the 15 km/h target.
  • The car no longer crashes quickly. It covers 428 m and several curves before failing.
  • The crash is understeer in a curve. In the last second the car's heading separates 48° from the lane, but applied steering never exceeds 0.23; the model asks for little wheel and the 0.3 smoothing delays the response further.
  • The car tends to ride the center line. The broken line appears in 45 % of rows, matching the lateral bias toward the dividing line mentioned when describing --random-spawn.

🐞 Caveats When Reading the Metrics

  • Collisions are inflated. Behavior Metrics does not stop the run on crash, and the PID keeps accelerating against the guardrail; the 101 collision rows are a single crash.
  • lane_invasion may carry its value across events. The 45 % is an indication of the bias, not an exact measure; in_right_lane is more reliable.

Next steps

  • Run this same model in the own driver on CARLA 0.9.14 and compare meters, seconds and end point with Behavior Metrics: 428 m, 104 s, crash at (191.7, 108.9).
  • Run the own driver against the 0.9.15 container, to measure the CARLA version effect.
  • Stop the run at the first crash inside the brain, as the own driver does.
  • Check whether PerfectLap should point to a Town02 route.
  • Test the 5-seed ensemble with EnsembleModels and compare with the ensemble ranking.
Behavior MetricsOwn script (0.9.14)Own script (0.9.15)
Run428 m; crash at (191.7, 108.9)PendingPending

🔀 Part 4 — Upstream Integration Plan

Four independent contributions, each with its issue, branch and PR against noetic-devel, following CONTRIBUTING.md. The two smallest go first; sync mode, the most valuable and largest, goes last.

OrderChangeFilesSuggested branchCurrent state
1Fix ROS_VERSION reading (trailing spaces in the variable name)driver_carla.pyfix/NNN-ros-version-envDetected, unfixed
2Use an external or Docker CARLA without Behavior Metrics closing it (BM_EXTERNAL_CARLA) and without requiring CARLA_ROOTutils/environment.pyfeature/NNN-external-carlaLocal patch working
3Configurable weather from YAML (Simulation > Weather)utils/carla_world_generator.py and an example YAMLfeature/NNN-weather-configLocal patch working
4Real sync mode in Python API: fix fixed_delta_seconds, a single process calling tick(), frame-synchronized imagepilot_carla.py, utils/carla_world_generator.py, utils/traffic.py, utils/configuration.pyfix/NNN-python-api-sync-modeSolved inside the brain only; still to be moved to the core

NNN is the GitHub issue number.

Phase 0 — Prepare the dev environment (once)

Work in a clean clone of the fork, separate from ~/BehaviorMetrics, so the working install is not mixed with branches.

  • Fork at https://github.com/JdeRobot/BehaviorMetrics/fork.
  • Clone it separately and link the original repository:
    git clone https://github.com/TU_USUARIO/BehaviorMetrics ~/BehaviorMetrics-dev
    cd ~/BehaviorMetrics-dev
    git remote add upstream https://github.com/JdeRobot/BehaviorMetrics
    git fetch upstream
  • Check the issue tracker for existing reports on each change.

Phase 1 — Cycle per change (repeat for 1, 2, 3)

  1. Issue with the six sections required by CONTRIBUTING.md: expected behavior, current behavior, reproduction steps, environment, possible cause, possible solution.
  2. Branch from the most recent version:
    git fetch upstream
    git checkout -b feature/NNN-external-carla upstream/noetic-devel
  3. Port the change without copying .bak or unrelated files.
  4. Test with default_carla_api.yml (example brain) and with pilotnet_town02_sync.yml, and record the results for the PR.
  5. Commit and push, file by file:
    git add behavior_metrics/utils/environment.py
    git commit -m "Allow using an external CARLA server without killing it"
    git push origin feature/NNN-external-carla
  6. Pull request against noetic-devel, with Fixes #NNN, how it was tested and in which environment (Ubuntu 26.04, Python 3.10, CARLA 0.9.15 in Docker, no ROS).
  7. Answer review in days or weeks, not months.

Phase 2 — Sync mode in the core (change 4)

This change touches central behavior, so agree on the approach with maintainers in the issue before coding.

  • Issue with the evidence: with AsyncMode: False fixed_delta_seconds is not fixed, pilot and world generator call tick() at once, and the brain receives the last image instead of the current tick's. Cite PR #719, which addressed asynchrony but not this.
  • Propose the design: a FixedDelta key in YAML; only the pilot calls tick(); the world generator stops calling it; the camera delivers the current frame's image.
  • Implement using the v2 brain logic as reference.
  • Validate with the trace: step of exactly 0.05 s, consecutive frames and two identical runs giving the same trajectory.
  • Once integrated, simplify the PilotNet brain to use Behavior Metrics' sync mode instead of its own.

🐞 Not Proposed to the Repository

  • The PilotNet brain as-is, because it depends on the project's pilotnet_model.py; at most a generic version could be offered as an example.
  • Local install adjustments: PyTorch versions for the RTX 5070, requirements-base.txt and pinned numpy.

⚠️ Limitations

  • No ROS on Ubuntu 26.04, so the changes cannot be tested in ROS mode; this must be stated in every PR.
  • Review and integration depend on maintainers; meanwhile, the local installation keeps working with the current patches.

🌱 Part 5 — Independence of the Training Seed with EMA

PropertyMeaningState
DeterminismSame seed and data give the same model, bit for bitSolved
Seed independenceDifferent seeds give models that drive the sameNot solved: the seed decides whether the car passes certain points of the route

The symptom is a lottery: with the same dataset, one model passes point A (~60 m) and another, trained with a different seed, crashes there. The 5-seed ensemble had already attenuated this. Question: could EMA achieve the same for a single model?

Hypothesis: EMA as a filter

EMA keeps an exponential moving average of the weights and validates/saves that average, not the raw weights. It is a first-order IIR low-pass filter with pole at z = β:

θ̄ₜ = β θ̄ₜ₋₁ + (1 − β) θₜ

The seed injects high-frequency noise; the filter attenuates it. With batch 32 and 52,700 images, one epoch is 1646 steps: β = 0.9999 has a ~6-epoch window. Expected limitation: the filter averages within one trajectory; if two seeds lead to different minima, it does not bring them together.

Experiment 1: β sweep

100 Comp0 datasets × β ∈ {0.99, 0.999, 0.9999} × 1 seed (42), cap 200 epochs; paired comparison dataset by dataset. It measured quality, not seed dependence.

Metric (v2, common pool)0.990.9990.99990.9999 vs. 0.999
MAE0.04290.04290.0424−1.2 %, better in 82/100
R²0.8720.8720.874+0.0025, better in 84/100
Slope0.8790.8770.878+0.0016, no practical effect

🐞 Defect Found and Fixed

The average started from the random weights of the freshly created model: with 0.9999 those weights still weighed 85 % after 1 epoch and 19 % after 10, lengthening early stopping and delaying the LR drops of the plateau scheduler. A bias correction was added (Adam-style, dividing by 1 − βᵗ); the average was verified to match Adam's formula and training remains reproducible bit for bit.

Experiment 2: 12 datasets × 5 seeds on the track

With β = 0.9999 and bias correction, 46 of 60 single models reached the gas station; the other 14 failed at point A (~60 m) or at a new point C (~230–250 m). 11 Comp0 datasets of 62,000 images and one of 309,000, seeds 42, 1, 2, 3, 4, cap 300 epochs, each model driven alone in Town02.

📌 Headline

14 of 60 single models do not reach the gas station; in 10 of 12 datasets at least one seed fails. Only two datasets arrived with all 5 seeds: Comp0-seed0 and the 309,000-image one.

Dataset (Comp0)s42s1s2s3s4Reach
seed0431 m434 m430 m427 m434 m5/5
seed1C·229 m431 m433 m435 mC·228 m3/5
seed2A·58 m437 m431 m430 m433 m4/5
seed3435 m434 m432 m433 mA·57 m4/5
seed4A·67 m432 m435 m428 mA·67 m3/5
seed5431 m433 mC·249 m431 m429 m4/5
seed6435 mA·63 m433 m434 m433 m4/5
seed7434 m434 m433 m434 mA·59 m4/5
seed8432 mC·228 m431 m431 mC·229 m3/5
seed9434 m426 mA·61 m433 mC·253 m3/5
seed42431 mA·67 m432 m436 m434 m4/5
309 k images433 m437 m431 m430 m435 m5/5

Same data and parameters; only the seed changes. Unmarked = reaches B (~430 m); C = fails at C (~230–250 m); A = fails at A (~60 m).

Analysis

  • Reaching the gas station. A single model reaches B 75 % of the time. The expected distribution if each seed were a fair coin matches the observed one: there are no "bad" datasets (5-of-5: 2.5 expected vs. 1; 4-of-5: 4.36 vs. 3; 3-of-5: 3.04 vs. 4; 2-or-fewer: 1.2 vs. 0).
  • Centering. |cte| varies ±0.11 m between seeds of the same dataset and ±0.05 m between dataset means (ANOVA p = 0.56). The seed explains ~77 % of the variation and the dataset ~23 %.
  • The large dataset. Its 5 seeds reached B and are well centered (|cte| 0.596 m). Suggests more data reduces seed dependence, but it is a single case and 5 of 5 by chance has a 27 % probability.
  • By seed. s3 reached B in all 12 datasets and s4 in 6. With 12 datasets this may be coincidence; a seed is not good in itself.

🐞 Why EMA is not enough

  • It attenuates high frequency, not the trajectory. The seed fixes initialization and data order, which decide which minimum training heads to; the filter averages within that trajectory and does not bring two trajectories together.
  • Threshold decisions remain. Early stopping and the plateau scheduler cross their thresholds at different moments per seed, and from there the models diverge.
  • Minimal offline differences, binary effect on the track. Slope changed ±0.002 between β values; in closed loop, a small difference in a critical curve decides between continuing and crashing.
  • Points A and C are weak zones of the data. Any seed variation pushes the model one way or the other.

What follows

  • Review the ranking of the 12 ensembles (ranking_ensembles_run81_ema0.9999bc.csv): if datasets with 2 failing seeds still reach B as ensembles, the ensemble confirms itself as the solution.
  • Watch the failure videos at A and C to understand what the model sees there.
  • Add data from stretches A and C, and from the gas station (B).
  • Test whether more data reduces seed dependence with more than one large dataset.
  • Try a cosine scheduler, which does not depend on validation and removes one of the threshold decisions.

📌 WEEK 48 SUMMARY – OCTOBER 27, 2026

🎯 Objective: Validate the new Behavior Metrics brain (v2) against the training pipeline, and determine whether EMA can make model quality independent of the training seed.

🧪 Methodologies: Brain v2 replicating the own driver's semantics inside Behavior Metrics (sync mode, frame-indexed camera queue, settle ticks, PID in simulated time, single apply_control), with AsyncMode: True in YAML. Upstream integration planned as four independent issues/branches/PRs against noetic-devel. EMA β sweep over 100 datasets, then 12 datasets × 5 seeds with bias correction.

📈 Key Findings: Brain v2 reproduces the own driver's conditions: exact 0.05 s steps, consecutive frames, correct weather and model. The model covered 428 m in 104.3 s before understeering into a guardrail at 48° heading error, with applied steering never above 0.23. Speed control works (14.2–15.2 km/h around a 15 km/h target). FPS do not influence behavior; sync mode, speed control and start-up settling were the real differences. EMA 0.9999 with bias correction is a minor quality gain (MAE −1.2 %, R² +0.0025) but does not deliver seed independence: 14 of 60 single models fail, and in 10 of 12 datasets at least one seed fails. The seed explains ~77 % of centering variation, the dataset ~23 %.

✅ Achievements: Behavior Metrics now drives under the same conditions as the training pipeline; the remaining gap is model behavior, to be compared with the own driver. Upstream contribution plan defined with four scoped changes. EMA bias-correction defect found and fixed; training remains reproducible bit for bit.

❌ Limitations: Collisions in Behavior Metrics are inflated (run does not stop on crash); lane_invasion may carry across events. The 0.9.14 vs. 0.9.15 comparison is still pending. No ROS on Ubuntu 26.04, so upstream changes cannot be tested in ROS mode. The large-dataset evidence for reduced seed dependence is a single case (5 of 5 by chance has 27 % probability).

🔜 Next Steps: Run the same model in the own driver on 0.9.14 and 0.9.15 and compare distance, time and crash point. Stop the run at first crash inside the brain. Open the first two upstream issues (ROS_VERSION fix, external CARLA). Review the 12-ensemble ranking and the failure videos at A and C. Add data at points A, B and C. Test a cosine scheduler and whether more data reduces seed dependence across multiple large datasets.