Can't import os.system and matlab.engine at the same time in the header

Question

I want to start a shared matlab session then connect to it right away in the same python 3 script. I also want to keep my matlab session opened after the script finishes. Because in the future, I want to write a class to do this, I would like to import all libraries in the header.

The problem is, python 3 script keeps failing to finish if I import os.system or subprocess.run first, then use matlab.engine to connect to matlab. As shown in the code below, my script will be stuck/hanged forever.

# the following code cannot finish running
import os
import matlab.engine
os.system("matlab -r \"matlab.engine.shareEngine\"")

Screenshot:

Oddly, if I start matlab using either of os.system or subprocess.run without importing matlab.engine as the answer of my previous question suggested, I can start it without any issue.

I also noticed that if I import matlab.engine AFTER started matlab (sample code below), the script will finished. Yet this will make my class very ugly...

# the following code can finish running
import os
os.system("matlab -r \"matlab.engine.shareEngine\"")
import matlab.engine

Screenshot:

I tried to bypass this issue by using subprocess.Popen. It did start the matlab and try to use matlab.engine to connect to it without stopping, yet the script can never connect to matlab because matlab.engine will try to connect before matlab finished initializing.

What causes the issue that the script can't finish? How can I import os/subprocess and matlab.engine together in the header? Do I have to make the script stop running for a while in order to wait matlab finished initialization?

For future people: os.system keeps open until called MATLAB exits. That's why the script can't close itself.

Answer 1

Here's a solution which:

• Connects to an existing session if there is one
• Creates a new session and connects to it if there isn't one

When the Python script ends, our MATLAB session will still be open for future use.

---

• For the first case, this is easy. We can simply check for existing sessions with matlab.engine.find_matlab().

• For the second case, the tricky part is that we cannot use matlab.engine.start_matlab function, because as I mentioned in my previous answer, the lifetime of the MATLAB process is tied to that of the Python process. To avoid this, we want to start MATLAB using Python libraries, for instance using either os or subprocess.

  To then connect to the new running MATLAB, we need to first share the engine from MATLAB, and then connect to it from Python. But to do that, we need the pid, which we can't get straightaway because MATLAB takes some time to start up.

  To get around this, in my solution I am using a simple text file where MATLAB dumps it's pid as soon as it starts. When the file exists, we know MATLAB is ready and thus we can connect to it.

---

import subprocess
import matlab.engine
import time
import os

class Test:
    def __init__(self):
        existing_session = self._check_existing_matlab_sessions()
        if existing_session:
            print("Using existing MATLAB session")
            eng = matlab.engine.connect_matlab(existing_session)
            print(f"Connected to {existing_session}")

        else:
            print("Creating new MATLAB session.")
            eng, pid = self._open_new_matlab_session()
            print(f"Connected to MATLAB_{pid}!")

    def _check_existing_matlab_sessions(self):
        sessions = matlab.engine.find_matlab()
        if sessions:
            return sessions[0]
        return ()

    def _open_new_matlab_session(self):
        pid_log = os.path.join(os.getcwd(),'pid_logfile.txt')
        if os.path.exists(pid_log):
            os.remove(pid_log)

        process = subprocess.Popen([r"C:\Program Files\MATLAB\R2020a\bin\matlab.exe","-r","matlab.engine.shareEngine; fid=fopen(fullfile(pwd,'pid_logfile.txt'),'w'); pid=feature('getpid'); fprintf(fid,'%d',pid); fclose(fid)"])

        while not os.path.exists(pid_log):
            time.sleep(1)

        f = open(pid_log, "r")
        pid = f.read(5)

        try:
            eng1 = matlab.engine.connect_matlab(f'MATLAB_{pid}')
            return eng1, pid
        except:
            raise Exception("Could not connect to MATLAB")

if __name__ == "__main__":
    Test()

---

When you run the script, if there is an existing session it will connect to it:

$ python main.py
Using existing MATLAB session
Connected to MATLAB_16480

If there isn't, it will create one and connect to it:

$ python main.py
Creating new MATLAB session.
Connected to MATLAB_39372!