1

People have asked similar questions before but none has a satisfactory answer. I'm trying to solve Lindblad Master Equation and the matrix size I'm trying to simulate are of order 10000 x 10000. But the problem is with exponentiation of the matrix, which is consuming a lot of RAM.

The MATLAB and Python expm() function take around 20s and 80s for a matrix of size 1000 x 1000 respectively. The code I've shown below.

pd = makedist('Normal');
N = 1000;
r = random(pd ,[N, N]);
t0 = tic;
r = expm(r);
t_total = toc(t0);

The problem comes when I try to do the same for a matrix of size 10000 x 10000. Whenever I apply expm(), the RAM usage grows and it take all the RAM and SWAP memory on my PC (I've 128 GB RAM and 64 Core CPU) and it's same in case of both MATLAB and Scipy. I don't understand what is taking so much RAM and how can I efficiently rum expm() or if it is not possible at all? Even if I could do it on any other language efficiently it would be really helpful!

Wolfie
  • 27,562
  • 7
  • 28
  • 55
Dev
  • 111
  • 3
  • 1
    It would be good if you actually explained the mathematics you're trying to implement by [edit]ing your question to include a [mcve] to use for benchmarking. Perhaps showing why one of these other answers is not "satisfactory". You ask for efficiency (presumably in terms of RAM) and speed, but you might have to compromise on speed to write a version of the algorithm which has lower memory requirements, worth being explicit about what the most important requirement is – Wolfie Dec 21 '22 at 08:08
  • 1
    Matrix exponential is difficult to compute. In fact still an area of active research. That said, it seems you are dealing with a special case here (GKSL equation, in which the matrix is diagonalizable according to wikipedia, not that I know anything about this) and it may be amenable to approaches that do not apply to the general case. If you post the Python code, there is more chance for others to try that. – harold Dec 21 '22 at 08:39
  • 1
    Could you provide some links to the previous questions, that ones that don't give a satisfactory answer? That way we won't have reinvent the wheel and give yet another unsatisfactory answer (or repeat what has already been given). – hpaulj Dec 21 '22 at 08:44
  • Thanks @Wolfie for the comment. I've added the code snipped in the above question. – Dev Dec 21 '22 at 08:54
  • @harold thank you for help. If possible can you share the Wikipedia page about the diagonalizability of GKSL matrix? I'm sorry but it might not be possible for me to share the code since the results are still unpublished and I am bound to not share them. I hope you understand. Thanks again! – Dev Dec 21 '22 at 08:57
  • https://stackoverflow.com/a/61563153/13202224 here is one example! I'm pretty much doing the same thing, but for a bigger matrix :). I hope this helps! – Dev Dec 21 '22 at 09:00
  • @Optimus what I saw was this [Wikipedia: Lindbladian](https://en.wikipedia.org/wiki/Lindbladian#Definition) where it says "we can rewrite the Lindblad equation in diagonal form". BTW this example with a random matrix is not necessarily representative of what you're dealing with, it's *probably* diagonalizable... – harold Dec 21 '22 at 09:09
  • @harold even after diagonalizing the h matrix, we need to solve the differential equation which contain 'Li" and Hamiltonian H, which are huge in size and this is what is being exponentiated. – Dev Dec 21 '22 at 09:16
  • @Optimus oh right. Is there any chance of diagonalizing those? If not then you're stuck with the slow computation as far as I know. – harold Dec 21 '22 at 09:46
  • 1
    Again, it really matters whether you care more about speed, or processing large matrices. You might only be able to have one of these things. – Wolfie Dec 21 '22 at 09:57
  • 1
    @Wolfie I can manage memory, If there is any alternative for speed. – Dev Dec 21 '22 at 09:59
  • You need a better computer.[Here](https://rextester.com/PMVFT94855) for a `1000x1000` matrix it takes less than `0.5` seconds. – rahnema1 Dec 21 '22 at 15:14
  • @rahnema1 It still crashes for 10000!! – Dev Dec 22 '22 at 07:35
  • The IT admin has deliberately restricted the resources for each computation. They don't allow computations that takes more than 5 seconds. – rahnema1 Dec 22 '22 at 08:39

0 Answers0