Week 44 ยท September 29, 2026

Driving Metrics Framework, Best Model to Date, and First Steps with Behavior Metrics

September 29, 2026

๐Ÿ“Š Part 1 โ€” Evaluation Metrics Before Behavior Metrics

Before integrating an external evaluation framework (Behavior Metrics), we consolidated a set of driving quality metrics that we had been developing in-house. These metrics are computed by the LaneMonitor class, which updates every frame in both the PyTorch and TensorFlow drivers. All metrics are identical across frameworks, ensuring fair comparison. This section summarizes the exact definitions and formulas behind each one.

๐Ÿ“Œ Why This Matters

Closed-loop driving quality cannot be reduced to a single number. A model can stay on the road but oscillate constantly, or take curves well but at dangerous speeds. The LaneMonitor breaks down driving quality into orthogonal dimensions: lane keeping, smoothness, efficiency, curve handling, speed adaptation, and lateral zigzag.

๐Ÿ›ฃ๏ธ Correct Driving / Inside-Lane State

Every frame, LaneMonitor.update() queries CARLA for the nearest waypoint to the driving lane and evaluates three conditions in order:

  1. Is there a Driving-type waypoint? If not, the frame is marked "outside" (no driving waypoint or wrong lane type).
  2. Distance to lane center: distance_to_center = distance(vehicle, waypoint_center). If it exceeds lane_width * 0.7, the frame is marked outside ("Outside center").
  3. Angle with respect to the lane: angle_diff = normalize(vehicle_yaw - lane_yaw). If |angle_diff| > 80ยฐ, the vehicle is considered crossing or going against traffic, and also counts as outside.

Only if all three conditions pass is the frame marked is_inside = True with reason "Correct driving". Every transition from inside to outside counts as a lane invasion event (lane_invasion_count).

๐Ÿ“‰ Percentage Outside Lane
outside_percentage = (outside_time / total_driving_time) * 100

outside_time accumulates, frame by frame, the real elapsed time (dt) of all frames where is_inside = False. It is not a frame count: it is accumulated real time, so a brief exit at high speed does not weigh the same as a long exit at low speed.

๐ŸŽฏ Driving Smoothness
smoothness_score = (smooth_steering_events / steering_changes) * 100

Each frame computes steering_change = |steering_current - steering_previous|. If that change is less than 0.05, it counts as "smooth" (smooth_steering_events); if it exceeds 0.3, it counts as "aggressive" (aggressive_steering_events, tracked but not directly printed). Smoothness is the percentage of frames where the steering wheel did not make sudden jumps between consecutive frames.

โšก Efficiency (Proposed Metric)
efficiency_ratio = throttle_accumulated / max(1, brake_accumulated + 0.001)

This is a proposed metric that is not yet fully validated for this project. throttle_accumulated and brake_accumulated are the frame-by-frame sums of the throttle and brake values applied (0.0โ€“1.0 each) over the entire run. It is a ratio, not a percentage: the higher, the less the vehicle braked relative to how much it accelerated.

Note: In the current model, a constant speed of 15 km/h is maintained throughout the run, so this metric is not yet discriminative between models. It remains a proposal for future evaluations where speed control becomes a variable of interest.
๐Ÿ”„ Curve-Taking Quality

A curve episode is defined as a contiguous stretch of frames with |steering| > curve_enter_threshold (0.15). To avoid fragmenting a curve due to noise, the episode does not end until |steering| drops below curve_exit_threshold (0.05) โ€” hysteresis.

During each episode, the following are tracked:

  • max_cross_track_error: maximum distance to lane center reached in that stretch.
  • had_lane_exit: whether at any frame of the curve is_inside = False.
  • oscillations: every sign inversion of steering within the same curve (while |steering| > 0.05) โ€” a sign of overcorrection.

The final report computes:

turn_success_rate = (1 - curve_episodes_with_lane_exit / curve_episodes_count) * 100 avg_cte_in_turns = curve_max_cte_sum / curve_episodes_count avg_oscillations_per_turn = curve_oscillations_total / curve_episodes_count
  • turn_success_rate: % of curves taken without leaving the lane at any moment.
  • avg_cte_in_turns: how far from center the vehicle gets, on average, in curves.
  • avg_oscillations_per_turn: how many times, on average, the model overcorrects (zigzag) within the same curve.
๐ŸŽ๏ธ Speed in Sharp Curves vs. Straight
avg_speed_sharp_turns = sharp_turn_speed_sum / sharp_turn_speed_samples # |steering| > 0.6 avg_speed_straight = straight_speed_sum / straight_speed_samples # |steering| < 0.05 speed_reduction_in_turns_pct = (1 - avg_speed_sharp_turns / avg_speed_straight) * 100

Each frame of update_speed_metrics is classified according to the current steering: if it is a sharp curve (same threshold as "sharp turn", 0.6) it adds to the curve speed accumulator; if it is a straight stretch (|steering| < 0.05) it adds to the straight one. The reduction compares both averages: it measures whether the model actually brakes when turning hard, or maintains the same speed regardless of the curve โ€” which would be a problem even if it takes the curve geometrically well.

ใ€ฐ๏ธ Lateral Zigzag (Steering Oscillation)

Unlike "per-curve" oscillations, this measures zigzag over the entire run, including straight stretches:

  • Zero crossings: every sign inversion of steering (ignoring |steering| < 0.05 as normal noise/jitter). The total and the rate are reported: steering_zero_crossings_per_sec = global_zero_crossings / total_driving_time.
  • Steering standard deviation (steering_std): np.std() of the entire steering history of the run โ€” how much the steering wheel varies in general, regardless of sign.
  • Dominant oscillation frequency (FFT): the mean is subtracted from the steering history (to discard constant bias) and np.fft.rfft is applied. The frequency-0 component (DC) is ignored and the one with the largest magnitude is taken:
avg_dt = total_driving_time / (total_frames - 1) spectrum = |FFT(steering - mean(steering))| dominant_frequency = frequency with largest magnitude (excluding 0 Hz) dominant_period = 1 / dominant_frequency

This distinguishes a periodic zigzag (a symptom of something systematic, such as an exponential smoother bouncing or an overcorrection repeating every N seconds) from simple random model noise. A high and consistent dominant_oscillation_hz across runs is more concerning than a high steering_std alone, because it indicates a pattern, not just variability.

๐Ÿ† Part 2 โ€” Best Driving Model to Date

โœ… Best Model Achieved: Composition 1, Seed 456 (Week 40)

We have obtained the best driving model to date using composition 1 with seed 456, created in Week 40. This model, identified as w40_comp9_seed456, completed a 15-minute continuous drive in CARLA with zero collisions and a lane departure percentage of only 3.73%. The full demonstration video is available below.

โ–ถ๏ธ Watch the 15-minute continuous driving demonstration

๐Ÿ“ˆ Quantitative Results

The following table summarizes the key metrics from the best run (run_number 235, September 28, 2026):

Metric Value Interpretation
Total time 900.01 s (15 min) โœ… Continuous driving without interruption
Collisions 0 โœ… No collisions in 15 minutes
Distance traveled 3733.9 m โœ… ~3.7 km covered
Outside lane percentage 3.73% โœ… Low lane departure rate
Smoothness 98.9% โœ… Very smooth steering
Curves taken 84 โœ… Good curve sample
Curves with lane exit 15 โŒ 17.9% of curves had a lane exit
Curve success rate 82.1% โš ๏ธ 82.1% of curves taken cleanly
Avg. max CTE in curves 0.8 m โš ๏ธ Moderate deviation in curves
Avg. oscillations per curve 0.0 โœ… No overcorrection within curves
Steering zero crossings 510 โš ๏ธ Notable lateral oscillation
Zero crossings per second 0.57 /s โš ๏ธ ~1 oscillation every 1.75 s
Steering std 0.083 โš ๏ธ Moderate steering variability
Dominant oscillation frequency 0.02 Hz โš ๏ธ Low-frequency zigzag pattern
Dominant oscillation period 47.37 s โš ๏ธ Periodic pattern every ~47 s
Framework PyTorch โ€”

โš ๏ธ Observed Limitations

The metrics reveal two clear limitations of this model:

  • Oscillations on straight stretches: The model shows 510 steering zero crossings (0.57/s), a steering standard deviation of 0.083, and a dominant oscillation frequency of 0.02 Hz (period ~47 s). This indicates a low-frequency zigzag pattern on straight roads โ€” not random noise, but a systematic periodic correction.
  • Lane exits during curves: Of the 84 curves taken, 15 (17.9%) involved a lane exit. The average maximum cross-track error in curves is 0.8 m, meaning the vehicle deviates noticeably from the lane center during turns. The curve success rate is 82.1%.

โœ… Despite the Limitations: Continuous, Collision-Free Driving

Despite these issues, the model never leaves the road. Even when it exits the lane during a curve, it recovers correctly, achieving continuous driving without collisions. The demonstration video shows a 15-minute uninterrupted drive with 3.7 km traveled and zero collisions โ€” the best result obtained to date.

๐Ÿ”„ New Models Currently in Training

Based on the successful configuration from Week 40 (composition 1, seed 456), new models are currently being trained in search of variants that maintain the collision-free performance of the current best model while improving two specific weaknesses:

  • Reduce lane exits during curves โ€” currently 17.9% of curves involve a lane departure.
  • Decrease straight-line oscillations โ€” currently 0.57 steering zero crossings per second with a dominant 47-second periodic pattern.

The goal is to find a configuration that preserves the 15-minute continuous, collision-free driving capability while delivering smoother, more stable lane-keeping behavior.

๐Ÿš€ Part 3 โ€” Starting Work with Behavior Metrics

๐Ÿ”„ Current Phase: Initial Setup

We have begun the process of integrating Behavior Metrics as an external evaluation framework for our driving models. Work is currently focused on operating system configuration for installation and operation.

โœ… Work Completed So Far

  • OS-level configuration: Preparation of the environment for installing Behavior Metrics dependencies.
  • ROS bridge installation: Initial setup of the ROS bridge to enable communication between Behavior Metrics and the CARLA simulator.

๐Ÿ“Œ Current Bottleneck

No major progress has been made yet due to version compatibility issues across all the packages that need to be installed. The ROS bridge, Behavior Metrics, and CARLA all have specific version requirements that must be aligned. This is the main obstacle at the moment.

๐Ÿ“Œ Next Steps for Behavior Metrics

  • Resolve package version compatibility between ROS bridge, Behavior Metrics, and CARLA.
  • Complete the ROS bridge installation and verify communication with CARLA.
  • Run a first test evaluation of the best model (w40_comp9_seed456) using Behavior Metrics.
  • Compare Behavior Metrics results with the in-house LaneMonitor metrics presented in Part 1.

๐Ÿ“Œ WEEK 44 SUMMARY โ€“ SEPTEMBER 29, 2026

๐ŸŽฏ Objective: Consolidate the driving evaluation metrics, document the best model to date, and begin integration with Behavior Metrics.

๐Ÿงช Methodologies: Frame-by-frame LaneMonitor tracking in both PyTorch and TF drivers; closed-loop CARLA driving with position logging; FFT-based oscillation analysis.

๐Ÿ“ˆ Key Findings: The evaluation framework captures six orthogonal dimensions of driving quality: lane keeping, smoothness, efficiency, curve handling, speed adaptation, and lateral zigzag. The best model to date (w40_comp9_seed456, composition 1, seed 456, Week 40) completed a 15-minute continuous drive with 0 collisions, 3.73% outside-lane time, and 98.9% smoothness, but exhibits periodic straight-line oscillations (0.57 zero crossings/s, 47.37 s dominant period) and 17.9% curve lane exits.

โœ… Achievements: Best model to date validated with a 15-minute continuous, collision-free drive. Full evaluation metrics documented and formalized. Behavior Metrics installation started with ROS bridge setup. New models based on the Week 40 successful configuration are currently in training.

โŒ Limitations: The best model still oscillates on straight stretches and exits the lane in 17.9% of curves (though it always recovers). Efficiency metric is proposed but not yet discriminative due to constant 15 km/h speed. Behavior Metrics integration is blocked by package version compatibility issues.

๐Ÿ”œ Next Steps: Resolve Behavior Metrics dependency versions, complete ROS bridge installation, run first external evaluation, and compare with in-house metrics. Continue training new models to reduce curve lane exits and straight-line oscillations while preserving collision-free driving.