0

Success:

>>> scp_cmd = r"sudo scp -i /home/backup/.ssh/id_rsa /opt/backups/*conf backup@a-hostname.local:/opt/backups/"
>>> subprocess.call(scp_cmd, shell=True)
1eadmin1.conf                                                  100%   83KB  83.5KB/s   00:00
1stflr_1.conf                                                  100% 2904     2.8KB/s   00:00
>>> scp_cmd = """sudo scp -i /home/backup/.ssh/id_rsa /opt/backups/*conf backup@a-hostname.local:/opt/backups/"""
>>> os.system(scp_cmd)
1eadmin1.conf                                                  100%   83KB  87.3KB/s   00:00
1stflr_1.conf                                                  100% 2904     3.4KB/s   00:00

Failure:

>>> scp_cmd = r"""sudo scp -i /home/backup/.ssh/id_rsa /opt/backups/*conf backup@a-hostname.local:/opt/backups/"""
>>> subprocess.call(scp_cmd, shell=True)
/opt/backups/*conf: No such file or directory
1
>>> subprocess.call(scp_cmd.split(' '))
/opt/backups/\*conf: No such file or directory
1
>>>
>>> subprocess.call(shlex.split(scp_cmd))
/opt/backups/*conf: No such file or directory
1

I'm confused why the triple quotes fail when I use subprocess.call(), but pass when I use os.system(). Why is there a difference between subprocess.call() and os.system() when handling triple quoted strings?

Mike Pennington
  • 41,899
  • 19
  • 136
  • 174

2 Answers2

6

I am pretty certain you are doing something else different; the triple quoting isn't making a difference here at all:

>>> a = r"sudo scp -i /home/backup/.ssh/id_rsa /opt/backups/*conf backup@a-hostname.local:/opt/backups/"
>>> b = r"""sudo scp -i /home/backup/.ssh/id_rsa /opt/backups/*conf backup@a-hostname.local:/opt/backups/"""
>>> a == b
True
>>> b
'sudo scp -i /home/backup/.ssh/id_rsa /opt/backups/*conf backup@a-hostname.local:/opt/backups/'
>>> a
'sudo scp -i /home/backup/.ssh/id_rsa /opt/backups/*conf backup@a-hostname.local:/opt/backups/'

Using triple quoting is just one way to specify a python string literal. How you specified that literal (with or without the r raw prefix, with single or triple quotes, using single ' or double " quotes) is not preserved.

Where triple quoting does make a difference is when you include a newline:

>>> foo = '''
... '''
>>> foo
'\n'

But your examples do not include any newlines at all.

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • 1
    Indeed, in fact, the first 'Success' example can't work at all. It needs either, `shell=True` or the command needs to be split. – Rob Wouters Oct 11 '12 at 19:58
  • 1
    @MikePennington: I find that hard to believe. *Something else* changed too. – Martijn Pieters Oct 11 '12 at 19:58
  • @Rob, you're correct and I fixed that in the question; however, that doesn't change the reality that I'm getting failures when I try to scp with triple quotes – Mike Pennington Oct 11 '12 at 19:59
  • @MikePennington, either your commands run and/or output is wrong or you found a bug in python. In these cases my experience tells me 99.9% it's the former. Please post an entire (i.e. non-interactive) script that exhibits the problem. – Rob Wouters Oct 11 '12 at 19:59
  • Yeah... I think this has to do with the shell expansion of the *, not the triple-quoting of the string. – mayhewr Oct 11 '12 at 20:00
  • 2
    @MikePennington: I know; the point is that that's the only difference a triple quote makes; it makes it easy to include newlines in your string literal, and it makes it easy to include (single) quote literals. *There is no other difference*, the strings produced are otherwise the same. You are thus barking up the wrong tree. – Martijn Pieters Oct 11 '12 at 20:04
  • If you're so sure that this was a problem with "something else changed", why does `subprocess.call(shlex.split(scp_cmd))` fail? – Mike Pennington Oct 11 '12 at 20:13
  • @MikePennington: Because I know how python string literals work. You asked if a triple quoted string is different, and I can help you with that: No, there is absolutely *no* difference. That's why I am sure. Without more code from you, I cannot see what else could have changed in the meantime, but the triple quoted string *is not the difference*. – Martijn Pieters Oct 11 '12 at 20:14
  • 1
    @MikePennington, that fails because there's no shell-expansion if you run it like that. – Rob Wouters Oct 11 '12 at 20:23
1

+1 to everything Martijn has said, and I think I may know what's going on. This output: 

>>> subprocess.call(scp_cmd.split(' '))
/opt/backups/\*conf: No such file or directory

looks suspicious to me. Why is the backslash there?

Doing my best to match your command, I get something like [suppressing only the username]:

>>> import subprocess, shlex
>>> 
>>> subprocess.call(r"scp -i /home/user/.ssh/id_rsa /tmp/*conf user@localhost:/tmp/save", shell=True)
aconf                                    100%    4     0.0KB/s   00:00    
bconf                                    100%    9     0.0KB/s   00:00    
0
>>> subprocess.call(r"""scp -i /home/user/.ssh/id_rsa /tmp/*conf user@localhost:/tmp/save""", shell=True)
aconf                                    100%    4     0.0KB/s   00:00    
bconf                                    100%    9     0.0KB/s   00:00    
0

which works because as explained, they're the same string. But say you had accidentally put a backslash in there? You'd get

>>> subprocess.call(r"""scp -i /home/user/.ssh/id_rsa /tmp/\*conf user@localhost:/tmp/save""", shell=True)
/tmp/*conf: No such file or directory
1
>>> subprocess.call(r"""scp -i /home/user/.ssh/id_rsa /tmp/\*conf user@localhost:/tmp/save""".split(" "))
/tmp/\*conf: No such file or directory
1
>>> subprocess.call(shlex.split(r"""scp -i /home/user/.ssh/id_rsa /tmp/\*conf user@localhost:/tmp/save"""))
/tmp/*conf: No such file or directory
1

which matches your output exactly -- there's no backslash in the output for shell=True or shlex.split(), but there is for .split(" "). For comparison, if there were no backslash, you should instead get the error message

>>> subprocess.call(r"""scp -i /home/user/.ssh/id_rsa /tmp/*conf user@localhost:/tmp/save""".split(" "))
/tmp/*conf: No such file or directory
1

So it seems to me very likely that the scp_cmd you've posted isn't the one your output actually corresponds to, and that your real one has a backslash.

DSM
  • 342,061
  • 65
  • 592
  • 494
  • of course, when you're troubleshooting something for 20 minutes, it doesn't make sense to post all 20 minutes of debugging in the question... yes I experimented with a backslash escape on the *, no that didn't cause the problem – Mike Pennington Oct 11 '12 at 20:55
  • @MikePennington: respectfully, I don't see how the `scp_cmd` you posted, which does not contain a backslash, is compatible with the output you posted, which does. I have demonstrated above a prima facie case that inserting one generates your output and not inserting one does not. – DSM Oct 11 '12 at 21:02
  • You have demonstrated that the question doesn't reflect every single case I tested, and that I hacked the question together imperfectly. For that I apologize... regardless, your conclusion is flawed. The original script which also had the error never had a backslash. – Mike Pennington Oct 11 '12 at 21:03