Deep Learning | Tensor flow | GPU (cuda/cuDNN) Vs CPU w and w/o EarlyStopping

In the previous post we built a neural net and tuned its hyper parameters. The hyper parameters were tuned using GridSearchCV. Here we look at modifying training the same network on GPU and compare that with training on CPU. 

Tools used

Tensorflow-gpu: 2.0.0

Keras version 2.3.1
Pandas version 0.25.3
Scikitlearn version 0.21.3

Nvidia GTX 960M 2GB
Intel i7 6700HQ 16 GB

Now, the number of epochs needed was tuned separately last time. Loss/accuracy Vs epochs curves raises the question of whether those many epochs are needed for this network and data. This has an impact on the time needed to train the network. Given the low end GPU the numbers are as expected.

Could we train the network in a lower duration with an acceptable loss of accuracy?

Early stopping is used here to answer this question. Results are shown below. Instead of going through 55 epochs, it decides to stop when the loss cannot be minimised beyond a certain point around 17-27 epochs. Early stopping parameters used is shown below.

Hyper parameter grid is small to begin with

Results (2 parameters and 3 values each)

	Wall time	Accuracy	learning rate	momentum
CPU	1 min 28 sec	0.978685	0.03	0.43
CPU Early Stopping	35.8 sec	0.973357	0.024	0.39
GPU	9 min 1 sec	0.978685	0.024	0.41
GPU Early Stopping	2 min 54 sec	0.971580	0.024	0.41

Results (2 parameters 5 values each)

Parameter grid is modified with additional ranges.

	Wall time	Accuracy	learning rate	momentum
CPU Early Stopping	1 min 35 sec	0.9822	0.03	0.39
GPU Early Stopping	9 min 37 sec	0.9751	0.033	0.41