CUDA Training Randomness and Behavior Metrics: First Successful Run with CARLA in Docker
October 6, 2026
π² Part 1 β CUDA Training Randomness: Why the Same Seed Produces Different Models
π The Problem in One Sentence
The training script fixes the PyTorch seed, but the GPU computes convolutions with algorithms that do not guarantee bit-for-bit identical results. That minimal difference grows with every training step until it produces completely different models.
π How the Problem Was Found
While running entrenar_definitivo.py with the same dataset and the same --seed 42, we noticed that each run produced a model with different best_val_loss and different driving behavior. At first this seemed impossible: the seed was fixed, so the weights initialization, dropout, and batch order should all be identical. To investigate, we ran the same command twice with a small number of epochs and compared the per-epoch train_mse and val_mse lines:
python entrenar_definitivo.py --csv datasets/<dataset>/data.csv --epochs 3 --seed 42 > b.txt
diff <(cut -d'(' -f1 a.txt) <(cut -d'(' -f1 b.txt) && echo "IDΓNTICAS"
The cut removes the per-epoch time (which always varies). The result was not "IDΓNTICAS": the losses diverged starting from the very first epochs, even though the seed was the same. This confirmed that the source of randomness was not in the Python-level seeding but deeper, in the GPU computation itself.
β οΈ What Is Fixed and What Is Not
| Source of Randomness | Status | Why |
|---|---|---|
| Train / val split | β Fixed | Uses np.random.RandomState(seed) |
| Initial weights, dropout, batch order | β Fixed | Depend on torch.manual_seed(args.seed) |
| Augmentation (flip, brightness, translation) | β οΈ Fixed but fragile | Uses Python's random, which the script does not seed. It is saved because PyTorch reseeds random in each DataLoader worker. With num_workers=0 it would stop being reproducible. |
| GPU convolutions (cuDNN) | β Not fixed | Some algorithms, especially in gradient computation, sum in an order that varies between runs. |
π§© How a Minimal Difference Ends in a Different Model
1. Rounding
In floating point, adding the same numbers in different order gives results that differ in the last decimal, on the order of 1e-7. cuDNN does not fix that order because not fixing it is faster.
2. Accumulation
That barely different gradient changes the weights slightly, which changes the next gradient. After thousands of batches, the two runs are already on different paths.
3. Thresholds
The script makes yes/no decisions based on val_mse: saving model_best.pth, lowering the LR (ReduceLROnPlateau), and early stopping, all with min_delta = 1e-4. If one run improves by 0.00011 and another by 0.00009, one resets the patience counter and the other does not. From there they diverge completely: different LR, different number of epochs, different saved model.
π Proposed Solution β Three Levels of Determinism
The following is a proposed solution to be applied to the training script. It is structured in three levels of increasing strictness.
β Level 1 β Seed Everything and Request Deterministic cuDNN Algorithms
In main(), replace the line torch.manual_seed(args.seed) with:
np.random.seed(args.seed)
torch.manual_seed(args.seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
For this model, this is usually enough. Training may become somewhat slower.
β Level 2 β Strict Mode, If Level 1 Is Not Enough
At the beginning of the file, before importing torch:
And in main(), after the Level 1 lines:
In this mode PyTorch raises an error if any operation does not have a deterministic version, and the message says which one. If you prefer a warning instead of an error, use torch.use_deterministic_algorithms(True, warn_only=True).
β Level 3 β Shield the DataLoader
Make augmentation not depend on the internal detail of the workers. In train_one_dataset, before creating the loaders:
s = torch.initial_seed() % 2**32
random.seed(s)
np.random.seed(s)
g = torch.Generator()
g.manual_seed(seed)
And add to train_loader these two arguments:
π§ͺ How to Verify It Worked
Run the same command twice with a few epochs and compare the per-epoch lines:
python entrenar_definitivo.py --csv datasets/<dataset>/data.csv --epochs 3 --seed 42 > b.txt
diff <(cut -d'(' -f1 a.txt) <(cut -d'(' -f1 b.txt) && echo "IDΓNTICAS"
If it prints IDΓNTICAS, train_mse and val_mse match to the sixth decimal and training is repeatable. If there are differences, move to Level 2.
π Limits of This Solution
- Only in the same environment. Repeatability holds for the same GPU, the same PyTorch, CUDA, and cuDNN versions. If the environment is reinstalled, the same seed may give a different result.
- Does not recover a previous run. Training done weeks ago was without determinism, so no seed reproduces it exactly. These changes guarantee that runs from now on can be repeated.
- Repeatable does not mean better. Fixing randomness gives you always the same sample of noise. To know if a configuration is good, train it with several seeds (
--seed 1,2,3) and compare.
π Part 2 β Behavior Metrics with CARLA in Docker: First Successful Run
β Behavior Metrics Is Now Running on Ubuntu 26.04 with an RTX 5070
Behavior Metrics now drives a Tesla in CARLA 0.9.15 and saves metrics. Two items remain pending: displaying the image in the frame_0 panel and connecting the PilotNet model.
π§ How It Works
Behavior Metrics is a JdeRobot tool that puts a "brain" to drive in a simulator and measures how well it does, to compare brains under the same conditions.
| Piece | What It Does | In This Installation |
|---|---|---|
| Simulator | Provides the world, car, and sensors | CARLA 0.9.15 in Docker container carla-bm, port 2000 |
| Config YAML | Defines car, sensors, brain, map, and where to save results | configs/CARLA/default_carla_api.yml |
| Driver | Reads the YAML, creates the world, and connects everything; -g opens the interface |
driver_carla.py |
| Brain | Python script that receives sensors and returns throttle, steering, and brake | brain_carla_slow_and_turn_api.py (simple rules, no neural network) |
| Metrics | Step-by-step log (CSV) and summary (JSON) of each recording | Folder with date and time inside behavior_metrics/ |
β Installation Steps Followed
These are the commands that left the environment working; pip check finished without conflicts. They only need to be repeated if reinstalling.
1. Clone the repository (noetic-devel branch, the main one):
2. Environment and base dependencies. PyTorch, TensorFlow, and TensorBoard are excluded because the file versions are incompatible with each other and with the GPU:
python3.10 -m venv .venv && source .venv/bin/activate
pip install -U pip wheel
grep -vE "(torch|torchvision|tensorflow|tensorboard|pathlib)==" requirements.txt > requirements-base.txt
pip install -r requirements-base.txt
3. ML libraries and CARLA client. numpy stays at 1.26.4 because pandas, scipy, and gym from the repo fail with numpy 2:
--index-url https://download.pytorch.org/whl/cu128 \
--extra-index-url https://pypi.org/simple
pip install tensorflow==2.21.0 "numpy==1.26.4"
pip install carla==0.9.15
4. Bird's-eye view and interface resources:
git clone https://github.com/Qi-Zha0/carla-birdeye-view
pyrcc5 -o ui/gui/resources/resources.py ui/gui/resources/resources.qrc
5. Simulator image:
Verification obtained: torch 2.11.0+cu128 True | numpy 1.26.4, tf 2.21.0 and No broken requirements found.
π Patch to Use CARLA in Docker
Without this change, Behavior Metrics closes your simulator on startup. The launch_env function in utils/environment.py does three things:
- Calls
close_ros_and_simulators(), which runskillall -9onCarlaUE4.shandCarlaUE4-Linux-Shipping. The container process is visible from the system, so it kills it. - Requires the
CARLA_ROOTvariable, even though it only uses it if it has to launch CARLA on its own. - Checks if a CARLA is already running and, if so, does not launch another.
The patch adds an escape hatch controlled by the environment variable BM_EXTERNAL_CARLA=1. It was applied once and left a copy of the original in utils/environment.py.bak:
sed -i.bak '/^def close_ros_and_simulators/a\ if os.environ.get("BM_EXTERNAL_CARLA") == "1":\n return' environment.py
With CARLA_ROOT=/tmp, point 2 is satisfied, and point 3 already plays in our favor: the output shows Carla running: True.
Limitation: with closing disabled, Behavior Metrics does not clean the world when finished. Restart the container between sessions to start clean.
β Session Startup
Two terminals are needed: first the simulator, then Behavior Metrics.
Terminal 1: simulator. Stop your CARLA 0.9.14 containers first, because they use conflicting ports. Wait until the CARLA window shows the map.
sudo docker run -d --name carla-bm --privileged --gpus all \
--net=host \
--env="DISPLAY" \
--env="NVIDIA_VISIBLE_DEVICES=all" \
--env="NVIDIA_DRIVER_CAPABILITIES=all" \
--volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" \
carlasim/carla:0.9.15 ./CarlaUE4.sh -quality-level=Epic -nosound -world-port=2000
Terminal 2: Behavior Metrics. Use a new terminal, without venv_py37 active.
export ROS_VERSION=None BM_EXTERNAL_CARLA=1 CARLA_ROOT=/tmp
export OBJECT_PATH=$PWD/configs/CARLA/CARLA_launch_files/CARLA_object_files/parked_car_objects.json
python3 driver_carla.py -c configs/CARLA/default_carla_api.yml -g
Two windows open: pygame window, with the Tesla seen from behind in Town01, and driver_carla.py, the Behavior Metrics interface.
If the simulator does not start, check its status and its last messages:
sudo docker logs --tail 40 carla-bm
β First Test in the Interface
The test consists of letting the example brain drive, recording for a few seconds, and checking that the metrics are saved.
| Section | Purpose |
|---|---|
Panels frame_0 to frame_3 |
Views where the brain displays images; they must be confirmed before use |
| Simulation | The large button starts and pauses the brain; the circular arrow restarts; the CARLA logo shows or hides the simulator |
| Brain | Changes brain with the simulation paused. The one from the config is already loaded |
| Stats | Its button starts and stops metric recording |
| Dataset | Records sensor data |
- In the panel
id: frame_0(bottom left of the four), typeframe_0in Frame name, leave RGB Image, and press Confirm. - In Simulation, press the large play button. No need to choose a brain.
- Check in
pygame windowthat the Tesla moves. - In Stats, press its play button, without touching the paths.
- Let it drive for 30 to 60 seconds.
- Press the Stats button again to compute and save.
- Press the large Simulation button again to pause.
- Check the new folder with
ls -lt | head -n 8insidebehavior_metrics/.
β Metrics Recording Was Generated
The first experiment produced a recording in behavior_metrics/20261006-093952/, containing a .csv (step by step) and a .json (summary). The example brain advanced 14.4 m in 55 s, hit the fence, and stayed there. This is expected: it only accelerates and turns without looking at the road, and serves to validate the chain, not to drive.
π Metrics of the First Experiment
| Metric | Value | Meaning |
|---|---|---|
experiment_total_simulated_time |
55.1 s | Duration of the recording |
completed_distance |
14.4 m | Distance traveled |
average_speed / max_speed |
0.97 / 28.98 | Average and maximum speed |
collisions |
2736 | Collision events, against the fence and the 60 sign |
lane_invasions |
2826 | Lane invasions |
position_deviation_mean |
3.16 m | Average deviation from the reference route |
percentage_completed / completed_laps |
0 / 0 | Progress on the reference circuit |
suddenness_distance_* |
0 | Brusqueness of control commands |
π Two Cautions When Reading These Metrics
- Collisions are counted per event, not per crash. Staying stuck against the fence adds one at every instant, which is why the "per km" figures come out enormous (547,200 collisions per km).
- Deviation and percentage completed are measured against
PerfectLap, the reference route from the config. You must check that it corresponds to the route you want to evaluate.
β οΈ Problems Found
All have an applied solution except the last one, which is still open.
| Symptom | Cause | Solution |
|---|---|---|
virtualenv -p python=3.8 from the guide does not work |
Syntax error in the guide and Python 3.8 not installed | python3.10 -m venv .venv |
requirements.txt cannot be installed as is |
torch==2.13.0 with torchvision==0.16.0 are incompatible; tensorflow==2.14.0 does not use the RTX 5070 |
Install the base without those lines and then the versions from your venv_py310 |
Container exits with Exited (135), Signal 7, Bus error |
Launched without DISPLAY, without NVIDIA_DRIVER_CAPABILITIES=all, and with -RenderOffScreen |
Use your 0.9.14 command changing only version, name, and port |
KeyError: 'CARLA_ROOT' |
The driver requires the variable even if CARLA is already running | export CARLA_ROOT=/tmp |
CARLA closes when launching driver_carla.py |
close_ros_and_simulators() kills CARLA processes |
Patch with BM_EXTERNAL_CARLA=1 |
40: CARLA is not supported sensor |
Warning from the sensor reader | None; it does not affect |
| ROS-free mode is always active | driver_carla.py reads the 'ROS_VERSION' variable with extra spaces and never finds it |
None; it favors this installation |
No image in frame_0 |
Not confirmed; in the first test the panel was renamed to frontal |
β οΈ Pending, see next section |
π Pending Items
Two things remain: displaying the camera in the interface and having your PilotNet drive.
Image in frame_0
In the first test the panel was confirmed with the name frontal, and the log shows it stopped being called frame_0. The hypothesis is that this field is the identifier to which the brain sends the image. It remains to confirm whether typing exactly frame_0 makes it appear. If it stays blank, this command shows which image the brain sends and to which panel:
cat brains/CARLA/brain_carla_slow_and_turn_api.py
grep -rnE "update_frame|frame_0" --include=*.py brains/CARLA/brain_carla_slow_and_turn_api.py ui robot | head -n 30
Brain for Your PilotNet
A brain must be written that loads a .pth model and replicates your top 35% crop and your normalization, so that it receives the same as during training. Needed:
- The content of
brains/CARLA/brain_carla_slow_and_turn_api.pyand the list ofbrains/CARLA/pytorch - Your architecture:
~/carla/proyecto_carla/pilotnet_model.py - Your agent:
~/carla/proyecto_carla/driverAgent_PyTorch_35-crop_simple_pth.py - The path of the
.pthmodel you want to test first - The map on which you trained it, to evaluate on the same one
- Review
PerfectLapso that the reference route corresponds to that route
Your models were trained with CARLA 0.9.14 and will be evaluated in 0.9.15. It does not require retraining, but there may be small visual or physics differences.
π WEEK 45 SUMMARY β OCTOBER 6, 2026
π― Objective: Diagnose the non-reproducibility of training on CUDA and document the full setup of Behavior Metrics with CARLA in Docker, up to the first metrics recording.
π§ͺ Methodologies: Reproducibility testing by running the same training command twice and diffing per-epoch losses; native Python 3.10 environment without ROS; CARLA 0.9.15 in Docker with X11 passthrough; Behavior Metrics driver connected to an external CARLA via a patched launch_env.
π Key Findings: Training is not reproducible even with a fixed seed because cuDNN does not guarantee bit-for-bit identical convolution results; the difference is amplified by floating-point accumulation and by min_delta=1e-4 thresholds in early stopping and LR scheduling. A three-level solution is proposed (seed everything + deterministic cuDNN; strict mode with use_deterministic_algorithms; shielded DataLoader). Behavior Metrics now runs on Ubuntu 26.04 with an RTX 5070, drives a Tesla in CARLA 0.9.15, and produced its first recording in behavior_metrics/20261006-093952/ (CSV + JSON).
β
Achievements: Root cause of CUDA non-reproducibility identified and documented with a concrete, verifiable solution. Behavior Metrics fully installed and operational with an external CARLA in Docker. First metrics recording generated successfully. All installation problems resolved except the frame_0 image display.
β Limitations: The determinism solution only holds within the same GPU, PyTorch, CUDA, and cuDNN environment; it does not recover previous runs. The first Behavior Metrics recording used a trivial rule-based brain (14.4 m in 55 s, 2736 collision events), so it validates the pipeline but not driving quality. The frame_0 panel still shows no image. The PilotNet brain has not yet been connected.
π Next Steps: Apply and verify the three-level determinism solution in entrenar_definitivo.py. Confirm the frame_0 panel name so the camera image appears. Write the PilotNet brain that loads the .pth model with the top 35% crop and the training normalization. Review PerfectLap for the route to evaluate. Run the first Behavior Metrics evaluation of the best model (w40_comp9_seed456) and compare it with the in-house LaneMonitor metrics.