4

I always thought that Hyper-Q technology is nothing but the streams in GPU. Later I found I was wrong(Am I?). So I was doing some reading about Hyper-Q and got confused more. I was going through one article and it had these two statements:

A. Hyper-Q is a flexible solution that allows separate connections from multiple CUDA streams, from multiple Message Passing Interface (MPI) processes, or even from multiple threads within a process

B. Hyper-Q increases the total number of connections (work queues) between the host and the GK110 GPU by allowing 32 simultaneous, hardware-managed connections (compared to the single connection available with Fermi)

In aforementioned points, Point B says that there can be multiple connected created to a single GPU from host. Does it mean I can create multiple context on a simple GPU through different applications? Does it mean that I will have to execute all applications on different streams?What if all my connections are memory and compute resource consuming, who manages the resource (memory/cores) scheduling?

tera
  • 7,080
  • 1
  • 21
  • 32
sandeep.ganage
  • 1,409
  • 2
  • 21
  • 47

1 Answers1

7

Think of HyperQ as streams implemented in hardware on the device side.

Before the arrival of HyperQ, e.g. on Fermi, commands (kernel launches, memory transfers, etc.) from all streams were placed in a single work queue by the driver on the host. That meant that commands could not overtake each other, and you had to be careful issuing them in the right order on the host to achieve best overlap.

On the GK110 GPU and later devices with HyperQ, there are (at least) 32 work queues on the device. This means that commands from different queues can be reordered relative to each other until they start execution. So both orderings in the example linked above lead to good overlap on a GK110 device.

This is particularly important for multithreaded host code, where you can't control the order without additional synchronization between threads.

Note that of the 32 hardware queues only 8 are used by default to save resources. Set the CUDA_​DEVICE_​MAX_​CONNECTIONS environment variable to a higher value if you need more.

tera
  • 7,080
  • 1
  • 21
  • 32
  • (If MPS is not turned on) can multiple CUDA contexts which were created through multiple MPI processes or multiple threads be scheduled concurrently in multiple work queues of hyper-q at the same time? – aka.ecc Jan 17 '21 at 18:38
  • What does "can be reordered relative to each other" mean? Does it imply that we look at the 8 queues and pick the best next task? Or do the elements actually get reordered in the queue? Follow up, if they do get rearranged in the queue, then why couldn't this be done with a single queue in Fermi? – Harsh Mar 23 '22 at 04:35