3

I just had to write a program in which I have to do matrix multiplication using threads, where there's a thread for every multiplication.

Now i'm wondering a few things, Are there really any advantages to using threads for multiplying a 3x2 matrix and a 2x3 matrix? for something small, sequential code is still efficient? If i'm wrong are there any advantages or disadvantages to something so small? I just see the complication too great for something so small.

On the other hand, would having a 10000x10000 matrix have a benefit in using threads? I would assume so, locality comes into play, but I'm still wrapping my head around when multithreading is more efficient, or not.

Thanks!

Nano
  • 73
  • 7
  • 1
    Try asking yourself a different question: would you use a thread to add two integers together? What about 3 or 4 integers? – Tas Mar 01 '18 at 03:08
  • @Tas Well not for 2 integers, but in a matrix it's a different situation isn't it? A lot of threads for the different threads could help in efficiency, but i was under the impression that it could also be inefficient for big chunks of data to use threading... i may be wrong though – Nano Mar 01 '18 at 03:22
  • 1
    A lot of threads is kind of a fuzzy term. Use too many threads and you lose efficiency to task switching. If you have a huge number of computations that are part of an algorithm that can be easily divided-- as in the computations do not count on the results of previous steps--you can divide the job up into a reasonable number of threads and get an almost proportional improvement in performance. Some time will be lost to management of the threads and some assembly work at the end cannot be threaded. Sometimes threads fighting it out over a shared resource renders threading pointless. – user4581301 Mar 01 '18 at 03:29
  • when in doubt, measure. you can find the point where threads are beneficial – kmdreko Mar 01 '18 at 04:27
  • 1
    It takes less time to multiply 3x2 matrices than it does to spawn and join threads. – Pubby Mar 01 '18 at 04:37
  • context switching is more expensive for matrices as small as 3 times 2. – John Z. Li Mar 01 '18 at 06:26
  • I had my FYP around cuda-chill (2014) and our findings were that for smaller matrices(4x5) multi threading didn't add any significant value. As for using more threads for matrix multiplication depends also on architecture. Performance to number of threads usually made a parabola like curve. – Muhammad Umar Farooq Mar 01 '18 at 08:43
  • Thanks for all of the helpful answers, guys! It seems that to some extent it isn't helpful to use threads for small nxn matrix multiplications, but also once the dimensions are too big the threads don't seem to help too much either – Nano Mar 01 '18 at 15:43

1 Answers1

1

Generally, you never want to update values from same cache lines by multiple threads, that would kill performance. You also want to utilize SIMD units within threads. Both are typically achieved due to some kind of processing data in blocks (look for register blocking / cache blocking terms). Also, ideally, you want to create just as many threads as the hardware concurrency is (to prevent expensive context switching). For data parallelism (such as matrix multiplication), this is easier. For task parallelism, thread pools are typically employed.

For small matrices like 3x2, multithreading would be definitely much much slower than sequential processing. For larger matrices, you need to measure to find out the threshold where multithreading will be faster. That threshold depends on too many parameters to provide generic answer.

Also, I don't understand what do you mean by

there's a thread for every multiplication

Do you want to create a single thread for every multiplication of 2 scalars? This would create zillion of threads for large matrices, which would be terribly slow.

Daniel Langr
  • 22,196
  • 3
  • 50
  • 93
  • Thanks for the informative answer. Yes, that is what i mean... so if i were to multiply a 2x3matrix and a 3x2 to create a 3x3 matrix, i would need 10 threads. – Nano Mar 01 '18 at 15:41