Weeks 3 & 4 - Read the dataset by batches

2 minute read

Meeting summary

  • The main objective of the week of training with more samples of raw images could not be achieved because memory complications arose when reading the dataset.
  • The training code has been adapted to use the new data structure.
  • When there is a lack of coherence in the results obtained so far with the dataset that reflects a linear movement with three degrees of freedom (speed, slope and starting point), the networks and evaluations of them have been regenerated with the dataset of 10000 samples.
  • It is decided to continue with the investigation of adapting the code to be able to read the dataset by batches and to be able to train with the dataset of 100000 samples the networks

To Do

The tasks proposed for this week are

  • Adapt the code to read the dataset by batches.
  • Refactor the evaluation code for the new data structure.

Adapting code

In order to adapt the code developed for network training, it has been necessary to modify several of the scripts that intervene throughout the process. Below are the most relevant changes made in the main scripts.

Network/net_train_config.yml

  batch_data: True #True or False

First, to be able to process the data by batches, a flag has been set in the configuration file.

Network/main_train.py
  
  ...
  if batch_data:
    train_data = utils.get_dirs(data_dir + 'train/raw_samples')
    val_data = utils.get_dirs(data_dir + 'val/raw_samples')
    if channels:
        in_dim = [20, 80, 120, 1]
    else:
        in_dim = [20, 80, 120]
    out_dim = np.prod(in_dim[1:])
  ...

We obtain in an array the routes to each of the samples for training and validation without loading the images to avoid saturating the memory of the server.

Network/Net.py

  ...
  if batch_data:
    steps_per_epoch = np.ceil(len(data_train) / batch_size)
    validation_steps = np.ceil(len(data_val) / batch_size)
    training_batch_generator = frame_utils.batch_generator(data_train, batch_size, steps_per_epoch, channels)
    validation_batch_generator = frame_utils.batch_generator(data_val, batch_size, validation_steps, channels)
    model_history = self.model.fit_generator(training_batch_generator,
                                             epochs=n_epochs, steps_per_epoch=steps_per_epoch,
                                             validation_data=validation_batch_generator,
                                             validation_steps=validation_steps,
                                             callbacks=[early_stopping, checkpoint], verbose=2)
  ...

Batches are set for training and validation, and a sample generator is created that will use the batch_generator function to read and process the images, leaving them ready for training and validation. Then you begin to train the network using the generator.

Utils/frame_utils.py

  def batch_generator(samples, batch_size, steps, channels):
     idx = 1
     while True:
        yield read_batch_data(samples, idx-1, batch_size, channels)
        if idx < steps:
            idx += 1
        else:
            idx = 1
            
  def read_batch_data(samples, idx, batch_size, channels):
     sub_samples = samples[idx * batch_size: (idx * batch_size) + batch_size]
     dataX, dataY = get_samples(sub_samples, channels)

     return dataX, dataY

The function responsible for generating batches uses the read_batch_data function which is responsible for selecting the samples to be used in the batch of the array that we carry to the beginning, and read and process the samples as was done previously by the get_samples function. Finally, it returns the samples of the batch that is being created with the appropriate structure.

You May Also Enjoy