getaddrinfow fails if running process from Python

Question

I try to run 3rd party process (nsqd.exe) from my python script but when I do nsqd fails to bind socket. I've no idea why.

The script I'm using:

import subprocess
import sys

proc = subprocess.Popen(['nsqd.exe', '-tcp-address="127.0.0.1:{}"'.format(sys.argv[1]),
    '-http-address="127.0.0.1:{}"'.format(sys.argv[2])])

print("the commandline is {}".format(proc.args))
proc.wait()
sys.exit(proc.returncode)

And the output:

D:\bsm.tar\bsm\final\nsqd>python nsqd.py 4150 4151
the commandline is ['nsqd.exe', '-tcp-address="127.0.0.1:4150"', '-http-address="127.0.0.1:4151"']
[nsqd] 2016/09/26 21:41:51.974681 nsqd v0.3.8 (built w/go1.6.2)
[nsqd] 2016/09/26 21:41:51.975681 ID: 864
[nsqd] 2016/09/26 21:41:51.979675 NSQ: persisting topic/channel metadata to nsqd.864.dat
[nsqd] 2016/09/26 21:41:52.004711 FATAL: listen ("127.0.0.1:4150") failed - listen tcp: lookup "127.0.0.1: getaddrinfow: No such host is known.

If I run that by myself directly everything works fine:

D:\bsm.tar\bsm\final\nsqd>nsqd.exe -tcp-address="127.0.0.1:4150" -http-address="127.0.0.1:4151"
[nsqd] 2016/09/26 21:42:20.093848 nsqd v0.3.8 (built w/go1.6.2)
[nsqd] 2016/09/26 21:42:20.094850 ID: 864
[nsqd] 2016/09/26 21:42:20.095851 NSQ: persisting topic/channel metadata to nsqd.864.dat
[nsqd] 2016/09/26 21:42:20.127984 TCP: listening on 127.0.0.1:4150
[nsqd] 2016/09/26 21:42:20.127984 HTTP: listening on 127.0.0.1:4151
[nsqd] 2016/09/26 21:42:22.111580 NSQ: persisting topic/channel metadata to nsqd.864.dat
[nsqd] 2016/09/26 21:42:22.111580 TCP: closing 127.0.0.1:4150
[nsqd] 2016/09/26 21:42:22.112553 HTTP: closing 127.0.0.1:4151
[nsqd] 2016/09/26 21:42:22.135635 NSQ: closing topics
[nsqd] 2016/09/26 21:42:22.135635 QUEUESCAN: closing
[nsqd] 2016/09/26 21:42:22.135635 LOOKUP: closing
[nsqd] 2016/09/26 21:42:22.135635 ID: closing

D:\bsm.tar\bsm\final\nsqd>

Maybe somebody have idea what is wrong?

Win10, python352. Running as admin does not help.

Thanks.

Greg Bowser · Accepted Answer · 2016-09-27T02:04:34.727

Remove the double quotes in your Popen so it becomes:

proc = subprocess.Popen(['nsqd.exe', 
                         '-tcp-address=127.0.0.1:{}'.format(sys.argv[1]), 
                         '-http-address=127.0.0.1:{}'.format(sys.argv[2])
                       ])

Before passing your command to CreateProcess, Python converts the list to a string using subprocess.list2cmdline:

>>> subprocess.list2cmdline(['nsqd.exe', '-tcp-address="127.0.0.1:4150"', '-http-address="127.0.0.1:4151"'])
'nsqd.exe -tcp-address=\\"127.0.0.1:1234\\" -http-address=\\"127.0.0.1:1234\\"

nsqd.exe thinks "127.0.0.1 is the hostname - hence the failed lookup.

Additional Information

The reason the double quotes work on the command line is that they have special meaning when a function such as CommandLineToArgvW is used to split the command line into individual arguments: normally arguments are delimited by whitespace, but when a quoted string is encountered, the quotes are stripped and the entire string becomes one argument.

This is also why Python is \-escaping the quotes: it expects the resultant line to be parsed in the above manner.

If you pass Popen a string rather than a list, list2cmdline will not be called and you should get the same results as removing the double quotes (i.e. it will be like running it from the command line):

proc = subprocess.Popen('nsqd.exe "-tcp-address=127.0.0.1:{}" '
                        '"-http-address=127.0.0.1:{}"'
                        .format(sys.argv[1], sys.argv[2]))

You can see this illustrated in the following (perhaps contrived) example:

import subprocess
subprocess.Popen('c:\python27\python.exe "--version"')
subprocess.Popen(['c:\python27\python.exe', '"--version"'])

The first Popen will print the python version. The second will look for a file named "--version": can't open file '"--version"': [Errno 22] Invalid argument

Any _Python_ is able to handle a type of quotes inside another. Or is there smth about _nsqd.exe_? — CristiFati, Sep 27 '16 at 00:36
Python handles the nested quotes fine. The exact command line being passed to [CreateProcess](https://msdn.microsoft.com/en-us/library/windows/desktop/ms682425(v=vs.85).aspx) is the one given by [subprocess.list2cmdline](https://github.com/python/cpython/blob/a448fe776a9f2be1f5a9be07756bf61efa7d3fb7/Lib/subprocess.py#L725). Note the `\\"` - the quote is slash escaped and will be taken literally. I'm not a windows person, but I believe the escaping is due to how [CommandLineToArgvW](https://msdn.microsoft.com/en-us/library/windows/desktop/bb776391\(v=vs.85\).aspx) handles quotes. — Greg Bowser, Sep 27 '16 at 01:15
I stupidly assumed that those quotes are needed by nsqd's arg parser. Thank you. — David Skuza, Sep 27 '16 at 09:42

getaddrinfow fails if running process from Python

1 Answers1

Additional Information