1

I am willing to determine the current CPU usage of a bunch of Linux computers, all available via SSH connexion. As for now I have managed something via the psutil and subprocess modules, but it could be much more straightforward if I could get a list out of subprocess.check_output.

So I defined a cpu_script.py :

import psutil

print(psutil.cpu_percent(interval=1,percpu=True))

to be called in my main script, which is in the same folder, with:

import subprocess
import os
import numpy as np
import psutil

usr = "AA"
computer = "BB"
cpu_script = os.path.join(os.getcwd(),"cpu_script.py")
test = subprocess.check_output("ssh -X " + usr + "@" + computer + 
         "python3 -u - < " + cpu_script,shell=True).strip()

As cpu_percent's outputs are typically in a list, I would expect my "test" variable to be a list, but what I really get is a byte:

input(test)
>> b'[4.0, 5.0, 7.0, 10.9, 18.4, 1.0, 4.0, 3.0]'
input(type(test))
>> <class 'bytes'>

Here I have managed to bend everything to finally get an array (with decode + replace + np.fromstring), but it is far from satisfying:

tmp = test.decode("utf-8")
input(tmp)
>> [4.0, 5.0, 7.0, 10.9, 18.4, 1.0, 4.0, 3.0]
input(type(tmp))
>> <class 'str'>

tmp = (tmp.replace('[','')).replace(']','')
result = np.fromstring(tmp, sep=',')
input(result)
>> [ 4.   5.   7.  10.9 18.4  1.   4.   3. ]
input(type(result))
>> <class 'numpy.ndarray'>

Could'nt I have straightaway a subprocess.check_output variable that is a list or an array ? What am I missing ?

Also if there is a way to avoid defining a cpu_script, hence to pass directly the python command in the subprocess, I would be interested ! I have failed at all my attempts to do so for now, the tricky part here being that there is both a ssh connexion and a python3 console command followed by python commands.

Thanks for your help!

jeannej
  • 1,135
  • 1
  • 9
  • 23
  • "As subprocess's output are typically in a list" this is your fundamental misunderstanding. The output of a process is always merely *bytes*, frequently representing text. That these bytes represent text that are valid Python list-literals is irrelevant - your subprocess script doesn't know that, and it would be a disaster waiting to happen if it tried to run text as arbitrary Python code. So fundamentally, the string representation *that you are printing* represents the underlying data structure, but it is still merely text. – juanpa.arrivillaga Jun 14 '18 at 19:08
  • my bad, I of course meant "cpu_percent outputs are in a list". I eddited that. – jeannej Jun 14 '18 at 19:21
  • Yes, that is what I was talking about as well (the process being your cpu_percent.py script). Again, you are not "outputting a list", you are *"printing the string representation of a list"*. This is a crucial point and once you understand, you'll see why your expectation was not reasonable. – juanpa.arrivillaga Jun 14 '18 at 19:31
  • Well `psutil.cpu_percent(interval=1,percpu=True)` does return a list, but I understand calling the print function translated it in someway. I would have avoided it had I know how to, but without this `print` I only get an empty string-byte `b''` from my `subprocess.check_output`. – jeannej Jun 14 '18 at 19:44

1 Answers1

1

The underlying python program "serialized" the output as text.

check_output returns a bytes object that you need to decode, then "unserialize".

I suggest:

import ast

result = ast.literal_eval(test.decode())

The best way would be to call this sub-python script as a module. You wouldn't have to run a subprocess/serialize/unserialize in that case. Just import the sub-module, call a function and get a real list as a result. Of course, in the case of ssh subprocess call, you cannot.

However you could improve this ugly and code-injection prone/unsafe check_output call needing shell=True by passing the filehandle of the opened python file to execute, feeding the output to the input of check_output, and using a list of strings instead of a string as argument:

with open(cpu_script,"rb") as f:
    test = subprocess.check_output(["ssh","-X",usr + "@" + computer,"python3","-u","-"],stdin=f)

followed by the decode+evaluation code.

Jean-François Fabre
  • 137,073
  • 23
  • 153
  • 219
  • Thanks a lot, the ast part works great and enables me to get a list out of "test" in one line! As for the rest I am affraid I don't really get what you are suggesting... when I replace `script_cpu` by `fct_cpu()` or even directly `"psutil.cpu_percent(interval=1,percpu=True)"` in my subprocess call, I get an error : `subprocess.CalledProcessError: Command 'ssh -X .... ' returned non-zero exit status 2` – jeannej Jun 14 '18 at 18:14
  • I overlooked the ssh part. In that case, you have to stick to `subprocess` (well, there's a `paramiko` module, but I doubt that it can run python calls natively) – Jean-François Fabre Jun 14 '18 at 18:44
  • Yes I figured out. Well thanks a lot for the ast part anyway, I will mark your answer as accepted as it answered my main question here ! Thanks for your swiftness ! – jeannej Jun 14 '18 at 18:49
  • 1
    you're welcome. I"m proposing another better solution which avoids this ugly shell command. Untested but you should try it out. Much simpler anyway. – Jean-François Fabre Jun 14 '18 at 19:01
  • "ugly check_output call", is that so ? ^^" I don't now much about subprocess so I basically went with what I found worked. I will follow you in this. The `with open` lines you suggest work like a charm, thank you very much! Too bad I can't upvote you twice – jeannej Jun 14 '18 at 19:28
  • 1
    I'll explain "ugly": you have to compose your command line, so if some arguments have spaces you have to quote them, etc..., besides `shell=True` is a known security issue since malicious code could be injected as data (ex: add `; rm -rf /` in some user-inputted argument and you could destroy your system by running an uncontrolled shell command). Avoid like the plague except for quick tests. You can find a lot of topics on that on stackoverflow – Jean-François Fabre Jun 14 '18 at 19:31
  • Thanks a lot Jean-François for your kind help and explanations! I am still struggling a bit to have all this nested in a for loop. I created a related question [here](https://stackoverflow.com/questions/50866386/subprocess-popen-inside-a-for-loop-how-to-properly-get-its-output). If it happens that you also have an answer there, your help would also be appreciated :) – jeannej Jun 14 '18 at 21:49
  • I see Charles already answered that one. good (classic mistake :)) – Jean-François Fabre Jun 15 '18 at 05:33