Using a variable containing doublequotes with subprocess

Question

I am having some trouble with the subprocess module. I would like the module to run the shell command equivalent to 'ls -l "/path/to/file/with possible space in directory/or with space in name"'. Subprocess works fine when the filename is not a variable. If the filename is a variable that contains the quotes, then it doesn't work.

Code that doesn't work:

import subprocess

archive_file_list = "/var/tmp/list"
archive = open(archive_file_list, "r")

for line in archive:
    noreturnline = line[:-1]
    quotedline = "\"" + noreturnline + "\""
    if extension == "zip":
        print quotedline
        archivelist = subprocess.check_output(['ls', '-l', quotedline])
        print archivelist

Code that works:

archivelist = subprocess.check_output(['ls', '-l', "/path/to/file/with possible space in directory/or with space in name"])

Here is the output for the code that doesn't work:

"/path/to/file/with possible space in directory/or with space in name"
ls: cannot access "/path/to/file/with possible space in directory/or with space in name" No such file or directory
Traceback (most recent call last):
File "./archive_test.py", line 12, in <module>
archivelist = subprocess.check_output(['ls', '-l', quotedline])
File "/usr/lib64/python2.7/subprocess.py", line 575, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['ls', '-l', '"/path/to/file/with possible space in directory/or with space in name"']' returned non-zero exit status 2

Before you ask - yes, I have already verified that "/path/to/file/with possible space in directory/or with space in name" does in fact exist by running 'ls -l' from the command line.

Any help would be appreciated. Thanks in advance.

Answer 1

in the first command (which is the best option there is):

archivelist = subprocess.check_output(['ls', '-l', "/path/to/file/with possible space in directory/or with space in name"])

the third argument is actually /path/to/file/with possible space in directory/or with space in name (without quotes) which is the filename that exists, and the command works.

Since shell=True isn't even set, the command is directly passed to exec, with the arguments passed as-is: the spaces & other chars are preserved.

If you add more quotes, they're not removed and they're passed literally to ls.

Since there's no such file called "/path/to/file/with possible space in directory/or with space in name" (with quotes), the file/dir isn't found.

There's another (dirty) way of calling a command: passing the full command as a string (not as a list of parameters). In that case, that would work (without shell=True at least on Windows, subprocess seems to handle the argument splitting, shell=True seems to be required on Unix-like systems):

subprocess.check_output('ls -l "/path/to/file/with possible space in directory/or with space in name"')

but your first approach is cleaner, specially if you don't know the directory name because it's a parameter. Let subprocess do the heavy lifting for you.

On Unix-like systems, using this last approach requires shell=True, but then you're exposing your program to malicious attacks like any open system call (appending ;rm -rf / to the filename, evaluating sub-shells for instance)

Final note: if you're really planning to use ls and parse its output, don't do it (http://mywiki.wooledge.org/ParsingLs), use standard os.listdir, os.path.getsize/getmtime & os.stat calls to get the information you need.