Comparing Model Performance with Normalised vs standardised input (Traffic Sign Classifier)

Jessica YungData Science, Self-Driving Car ND, StatisticsLeave a Comment

In the previous post, we explained (1) what normalisation and standardisation of data were, (2) why you might want to do it and (3) how you can do it. In this post, we’ll compare the performance of one model on unprocessed, normalised and standardised data.

We’d expect using normalised or standardised input to give us higher accuracy, but how much better performance do we get? And will normalised or standardised data perform better?

Recap: Normalisation and standardisation

Recall that normalisation scales all numeric variables in the range [0,1]. You can implement this with x_{new} = \frac{x-x_{min}}{x_{max}-x_{min}}.

Standardisation transforms your data to have a mean (average) of zero and a variance of one. You can implement this with x_{new} = \frac{x-\mu}{\sigma}.

Experiment specifications

In this example, we train a two-layer feedforward network to classify traffic signs. There are 43 classes (types of traffic signs) in total. We have 39209 training samples and 12630 validation samples.

screenshot.png

 

Here are some examples of traffic sign images we want to classify:

screenshot.png

Carrying out the experiment 

To make our results reproducible, I shuffled the training and test data with random_state=42.

I created copies of the data that were normalised and standardised. I then trained the model for 100 epochs on each version of the data with a batch size of 100.

Results

The training times for the three models did not differ significantly – they were all recorded as 3s per epoch. So we will discuss only accuracy metrics in this post.

There are four accuracy metrics recorded in this experiment. Accuracy is the percentage of examples the model gets completely correct. Loss is a measure of how wrong the model was in its predictions across all the examples for that iteration. Here we use categorical_crossentropy as our measure of loss.

Validation accuracy and validation loss are the accuracy and loss on the validation samples, that is, samples that the model does not see during training. The model is optimising for reducing the training loss, so the validation figures can give us an idea of how well the model generalises and are often the metrics we care about more.

Unprocessed input

First observe that, without preprocessing, the accuracy quickly decreases to under 0.1 and the loss increases to above 10 and gets worse with the number of epochs.

screenshot.png

screenshot.png

Preprocessed input

Now we will compare using normalised and standardised images as input. First note that using either preprocessing method gives clearly better accuracy than using original images:

screenshot.png

screenshot.png

Secondly, note that normalised input seems to perform better here when it comes to both validation and training accuracy. Let’s eliminate the original image series from the plots to better compare using normalised vs standardised input:

screenshot.png

screenshot.png

As expected, because the model is optimising for training loss and not validation loss, the training accuracy is more stable whereas the validation accuracy (and the difference between normalised and standardised input) is more volatile.

The next step would be to test whether there is a statistically significant difference between the accuracy when using normalised as opposed to standardised input.

Further Observations

The graphs of the accuracy and loss for standardised input have a nice shape: the accuracy increases and the loss decreases markedly in the first five epochs.

screenshot.png

Strangely, there is no such marked change with normalised images.

screenshot.png

 

Note: Will post a link to the code soon. Check back later, send me a message or leave a comment if you’re interested!

Related posts:

Posts coming up:

  • Testing whether there is a statistically significant difference between training and validation accuracy when using normalised vs standardised images as input.
  • Explaining TensorFlow code for a multilayer perceptron and a convolutional neural network applied to traffic signs line by line.

 

Leave a Reply