In the previous post, we explained (1) what normalisation and standardisation of data were, (2) why you might want to do it and (3) how you can do it. In this post, we’ll compare the performance of one model on unprocessed, normalised and standardised data.
We’d expect using normalised or standardised input to give us higher accuracy, but how much better performance do we get? And will normalised or standardised data perform better?
Recap: Normalisation and standardisation
Recall that normalisation scales all numeric variables in the range [0,1]. You can implement this with .
Standardisation transforms your data to have a mean (average) of zero and a variance of one. You can implement this with .
In this example, we train a two-layer feedforward network to classify traffic signs. There are 43 classes (types of traffic signs) in total. We have 39209 training samples and 12630 validation samples.
Here are some examples of traffic sign images we want to classify:
Carrying out the experiment
To make our results reproducible, I shuffled the training and test data with random_state=42.
I created copies of the data that were normalised and standardised. I then trained the model for 100 epochs on each version of the data with a batch size of 100.
The training times for the three models did not differ significantly – they were all recorded as 3s per epoch. So we will discuss only accuracy metrics in this post.
There are four accuracy metrics recorded in this experiment. Accuracy is the percentage of examples the model gets completely correct. Loss is a measure of how wrong the model was in its predictions across all the examples for that iteration. Here we use categorical_crossentropy as our measure of loss.
Validation accuracy and validation loss are the accuracy and loss on the validation samples, that is, samples that the model does not see during training. The model is optimising for reducing the training loss, so the validation figures can give us an idea of how well the model generalises and are often the metrics we care about more.
First observe that, without preprocessing, the accuracy quickly decreases to under 0.1 and the loss increases to above 10 and gets worse with the number of epochs.
Now we will compare using normalised and standardised images as input. First note that using either preprocessing method gives clearly better accuracy than using original images:
Secondly, note that normalised input seems to perform better here when it comes to both validation and training accuracy. Let’s eliminate the original image series from the plots to better compare using normalised vs standardised input:
As expected, because the model is optimising for training loss and not validation loss, the training accuracy is more stable whereas the validation accuracy (and the difference between normalised and standardised input) is more volatile.
The next step would be to test whether there is a statistically significant difference between the accuracy when using normalised as opposed to standardised input.
The graphs of the accuracy and loss for standardised input have a nice shape: the accuracy increases and the loss decreases markedly in the first five epochs.
Strangely, there is no such marked change with normalised images.
Note: Will post a link to the code soon. Check back later, send me a message or leave a comment if you’re interested!
Posts coming up:
- Testing whether there is a statistically significant difference between training and validation accuracy when using normalised vs standardised images as input.
- Explaining TensorFlow code for a multilayer perceptron and a convolutional neural network applied to traffic signs line by line.