8

As the question already suggests, I am new to deep learning. I know that the learning process of the model will be slow without GPU. If I am willing to wait, Will it be OK if i use CPU only ?

Kanu
  • 91
  • 6
  • 3
    of course, it will be ok – Daiver Apr 04 '17 at 07:35
  • It will be OK if you won't be doing convolutions. GPUs really help when working with image datasets that include convolutional layers. For learning purposes a CPU will be enough – Mihail Burduja Apr 04 '17 at 08:49
  • @MihailBurduja GPUs are not required for Deep Learning. They speed things up quite a lot (no matter if you use convolutions or have simple dense layers), but he is not asking about time. – Martin Thoma Apr 05 '17 at 07:33

5 Answers5

13

Many operations which are performed in computing deep learning (and neural networks in general) can be run in parallel, meaning they can be calculated independently then aggregated later. This is, in part, because most of the operations are on vectors.

A typical consumer CPU has between 4 to 8 cores, and hyperthreading allows them to be treated as 8 or 16 respectively. Server CPUs can have between 4 to 24 cores, 8 to 48 threads respectively. Additionally, most modern CPUs have SIMD (single instruction multiple data) extensions which allow them to perform vector operations in parallel on a single thread. Depending on the data type you're working with, an 8 core CPU can perform 8 * 2 * 4 = 64 to 8 * 2 * 8 = 128 vector calculations at once.

Nvidia's new 1080ti has 3584 CUDA cores, which essentially means it can perform 3584 vector calculations at once (hyperthreading and SIMD don't come into play here). That's 56 to 28 times more operations at once than an 8 core CPU. So, whether you're training a single network, or multiples to tune meta-parameters, it will probably be significantly faster on a GPU than a CPU.

KadeG
  • 146
  • 5
4

Depending on what you are doing, it might take a lot longer. I had 20x speedups be using a GPU. If you read some Computer Vision papers, they train their networks on ImageNet for about 1-2 weeks. Now imagine if that took 20x longer...

Having said that: There are much simpler tasks. For example, for my HASY dataset you can train a reasonable network without a GPU in probably 3 hours. Similar small datasets are MNIST, CIFAR-10, CIFAR-100.

Martin Thoma
  • 124,992
  • 159
  • 614
  • 958
  • Thanks Martin, your answer was also helpful. Can you name some computer vision papers in which author tells they trained their network for 1-2 weeks ?? – Kanu Apr 06 '17 at 09:08
2

Computationally intensive part of the neural network is multiple matrix multiplications. And how do we make it faster? We can do this by doing all the operations at the same time instead of doing it one after the other. This is in a nutshell why we use GPU (graphics processing units) instead of a CPU (central processing unit).

Google used to have a powerful system, which they had specially built for training huge nets. This system costs $5 billion, with multiple clusters of CPUs. Few years later, researchers at Stanford built the same system in terms of computation to train their deep nets using GPU. They reduced the costs to $33K. This system was built using GPUs, and it gave the same processing power as Google’s system.

Source: https://www.analyticsvidhya.com/blog/2017/05/gpus-necessary-for-deep-learning/

aerin
  • 20,607
  • 28
  • 102
  • 140
0

Deep learning is all about building a mathematical model of the reality or of some kind of part of reality for some kind of specific use by using a lot of training data so you use a lot of training data from the real world that you have collected and then you can train your model so your mathematical model can predict the other outcomes when you give it new data as input so you basically can train this mathematical model but it needs a lot of data and this training needs a lot of computation. So there are lot of computational heavy operations that need to take place and also you need a lot of data. Therefore, for example companies such as Nvidia who are traditionally have been making gaming GPUs for graphics, now they are also having a huge part of the revenue coming from AI and Machine Learning and all of these scientists who want to train their models and you see companies like Google and Facebook, all of them are using GPUs currently to train their ML models.

Rajesh Prajapati
  • 2,331
  • 1
  • 10
  • 6
0
sogu
  • 2,738
  • 5
  • 31
  • 90