Self-driving Cars — Deep neural networks and convolutional neural networks applied to clone driving behavior

Convolutional Neural Network with Keras/TensorFlow

--

Self Driving Car

Join me on this exciting journey to build, train and validate a new deep neural network to clone driving behavior. The model outputs a steering angle to an autonomous vehicle! Thanks to Udacity Self-driving Car Nanodegree for providing me the basic skills set to get there!

Overview

A simulator where you can steer a car around a track for data collection has been provided by Udacity (which is an amazing open source code and initiative, free to anyone to use!). I have used image data and steering angles to train the neural network and then used the model to drive the car autonomously around the track.

The following steps have been implemented:

  • Used the simulator to collect data on good driving behavior
  • Built a convolution neural network in Keras that predicts steering angles from images
  • Trained and validated the model with a training and validation set
  • Tested that the model successfully drives around track without leaving the road

Model Architecture

My model tries to replicates NVIDIA’s End to End Learning for Self-Driving Cars. The model includes data normalization/zero-mean by 255/-0.5 using a Keras lambda layer, 5x5 and 3x3 convolutions using Keras Convolution2D, RELU layers to introduce nonlinearity, fully connected layers using Keras Flatten and Dense, and overfitting control using Keras Dropout. The loss is compiled using mean square error (mse) and adam optimizer.

Here is the NVIDIA’s target architecture:

NVIDIA’s End to End Learning for Self-Driving Cars

Here is a visualization of the implemented architecture:

My Model

Here is the model summary:

Model Summary — Keras

The model contains dropout layers and it was trained and validated on different data sets to ensure that the model was not overfitting. The model was tested by running it through the simulator and ensuring that the vehicle could stay on the track. Adam optimizer was used, so the learning rate was not tuned manually.

Datasets

The Udacity simulator version I have used captures images from three cameras mounted on the car: a center, right and left camera. That’s because of the issue of recovering from being off-center.

Let’s say the driver is moving forward but wants to turn towards a destination on the left. From the perspective of the left camera, the steering angle would be less than the steering angle from the center camera. From the right camera’s perspective, the steering angle would be larger than the angle from the center camera. The correction used in this project is later presented.

The Udacity simulator generates the following data in the output directory:

  1. IMG folder - this folder contains all the frames of your driving.
  2. driving_log.csv - each row in this sheet correlates the image with the steering angle, throttle, brake, and speed of your car. In this project we'll be using the steering angle.

Two (2) tracks has been recorded and tested, a standard (track 1) and a challenge (track 2). For track 1, I collected a minimum dataset of images, initially with about 10K but after the initial 95% cutoff of high-frequency steering angle = 0, the remaining dataset had 3.3K images only. I did a further augmentation and increased it up to 12.5K. I was surprised by the achievements using such small dataset.

Track 1:

  • I collected my data from ‘fastest’ mode from Udacity simulator, which does not include shadows on the road.
  • I collected images for 3 full cycles driving in the correct direction only. So without proper augmentation, the dataset would be biased for curves on one side.
  • I used the keyboard arrows to drive, so the data is not smooth and there was a lot of s=0 because the key arrow is not held to keep constant steering angle.
  • I tried to keep the car in the center all the time.
  • I did not collect data with the car recovering itself from off the road. That could be future data improvements.

On the other hand, track 2 was a challenging one. I collected a bigger dataset of images, initially with about 53K, after the initial 95% cutoff of high-frequency steering angle = 0, the remaining dataset still had 49.5K images. I did further augmentation and data selection and ended up with a 46.6K dataset.

Track 2:

  • I collected my data from ‘good’ mode from Udacity simulator.
  • I collected images for 2 full cycles driving in the correct direction and then 2 full cycles in the reverse direction.
  • I used the mouse to steer, trying to get smooth data.
  • I tried to keep the car in the right lane all the time, I am not a good racer so in order to achieve that I had to drive the car at 5MPH and bellow speed which I know would be a challenge to train the model and later use it on much higher speed driving.
  • I did not collect data with the car recovering itself off the road. That could be future data improvements.
  • I did not collect data with the car recovering itself from left to right lane. That could be future data improvements.

Main Strategy

The main strategy was to make sure I had a well balanced and not biased dataset, then apply a well know model architecture such as NVIDIA’s End to End Learning for Self-Driving Cars.

My first step was to evaluate the driver log steering histograms for 100 bins and do all required transformation, drop and augmentation to balance it. Here I followed the same methodology as in the well explained pre-processing from Mez Gebre, thanks Mez!

Clean up steps:

  1. Downsample the over-represented examples
  2. Correct right and left steering angles
  3. Augment the dataset by image flipping and horizontal shift.
  4. Get rid of outliers

In image flipping, we “mirror” the image and then reverse the steering angle signal. In horizontal shift, we horizontally move the image and adding or subtract the appropriate angles corresponding to the shift.

Here is Track 1 initial distribution:

Track 1 — Initial Distribution

Track 1 S=0 95% drop:

Track 1–95% drop for S=0

Next, Track 1 Left and right steering angle correction. I did a simplified correction based on the offset dynamic approach suggested by Andrew Hogan, using an offset of 0.08. Thanks Andrew for sharing!

records.append({
'image': left,
'steering': steering + ((steering * steering / max_steer ) + offset)
})
records.append({
'image': right,
'steering': steering - ((steering * steering / max_steer ) + offset)
Track 1 — Steering angle corrections

Track 1 100% Image Flipping:

Track 1 — Image flipping

Track 1 Horizontal shift:

Track 1 — Horizontal shift

Track 1 Outbound angles drop:

Track 1 — Outliers drop

Track 1 Final trimming:

Track 1 — Final dataset

The second step was to take care of images transformation as defined on the new driver log. The images also needed to be cropped to get rid of landscape and car’s hood (image noise) and resize image to same as used by NVIDIA NN Model. Brightness and shadow augmentations as per Vivek Yadav, thanks Vivek!

Track 1 Cropping image size:

Cropping landscape and car’s hood (noise)

Track 1 Apply random brightness:

Brightness augmentation

Track 1 Apply random shadows:

Shadow augmentation

Track 1 Combined image transformation:

Final preprocessed image

Here is the track 2 driver log balance process:

Track 2 initial distribution:

Track 2 — Initial distribution

Track 2 S=0 95% drop:

Track 2–95% drop for S=0

Track 2 Left and right steering angle correction:

Track 2 — Steering angles correction

Track 2 40% Image Flipping:

Track 2 — Image flipping

Track 1 Outbound angles drop:

Trac 2 — Outliers drop

Track 1 Final trimming:

Track 2 — Final dataset

Training results

In order to gauge how well the model was working, I split my image and steering angle data into a training and validation set.

For track 1, after some experimentation, I noted that a loss of about 0.02 for train and validation, in general, would start to diverge (overfitting), so I have used an early stop function. Trained 5 epochs. The model worked great on ‘fastest’ mode of the Udacity simulator, even up to 30MPH speed target!! I let it run several complete cycles to be sure it was a stable solution. I have tried the ‘fantastic’ mode of simulator and it did great as well but presented some ‘drunk’ drive at high speeds, but it managed to stay on the road all the time, the shadow augmentation really helped (remembering I collected data without shadows) for this mode since the detail shadows projected on the road did not interfered much with the driving experience.

Track 1 Loss training history:

For track 2, I noted that a loss of about 0.03~0.05 for train and validation, in general, would start to diverge (overfitting), so I have used an early stop function. For the final model, I started setting up an early stop at 0.05 loss, which barely allowed 2 epochs train. I saved the weights of this model and tested it. It did not work very well getting the car eventually stuck on curves. Then I set up a smaller early stop at 0.04 loss. Trained 3 more epochs. The model worked great on ‘fastest’ mode of the Udacity simulator, even up to 16 MPH speed target which surprised me since the data was recorded at 5MPH and lower!! Stayed in the right lane about 99% of the time!! I let it run several complete cycles to be sure it was a stable solution. But on more detailed graphical modes the car turn to get eventually stuck or changed to the left lane. The relatively small dataset I collected does not include recovering steering from left to the right lane, so the car could not adjust itself back on the right lane and eventually would get stuck somewhere. Then I tried to further improve the training with more epochs, set up a smaller early stop at 0.03 loss and trained 3 more epochs. It did not work, the car was not driving properly even on ‘fastest’ graphical mode of the Udacity simulator. So my final solution is the second training above described. Any future addition of data for off-road and left to right lane recovery would improve the driving experience in higher graphical modes of the simulator. I strongly believe that a better dataset recorded at the right range of speed (I did it at too slow MPH) will generate better driving cloning, but for now, I got a good solution for the challenge!!

Track 1 Loss training history:

Results!

Track 1

Track 2

Acknowledgments / References

--

--

Self-driving Car Engineer Apprentice. Marine and Subsea Engineer. Friendly, creative and proactive.