Unable to access environment variables inside subprocess

Question

python version: 2.7.12 Environment: MacOS + virtualenv

I am using following code to display environment variable passed to the subprocess

from subprocess import Popen, PIPE
import os


def run_baelish(svc, cfg, dest_dir="/config", cmd=["baelish-client"]):
    """
        Executes baelish command at given directory
    """
    chenv = os.environ.copy()
    chenv["TEST_VAR"] = "test"
    print(chenv["TEST_VAR"])
    try:
        print(cmd)
        p = Popen(cmd, stdout=PIPE, stderr=PIPE, env=chenv)
        (out, err) = p.communicate()
        if out is not None:
            print(out)
        if err is not None:
            print(err)
        return p.returncode
    except OSError as oe:
        print("baelish-client executable not found")
        print(oe)
        return oe.errno
    except ValueError as ve:
        print("Invalid arguments passed to baelish-client")
        print(ve)
        return 3
# end of run baelish function

I am calling this function from test code as follows:

import unittest
import os
from helper.operations import run_baelish

class TestBaelishRun(unittest.TestCase):
    def setUp(self):
        pass

   def testSuccess(self):
        run_baelish("monolith", "master", "", cmd=["echo", "$TEST_VAR"])

# End of class TestBaelish Run

after running tests I am getting following results

.test
['echo', '$TEST_VAR']
$TEST_VAR

Last line is the stdout returned by the subprocess.communicate() call. Instead of returning the actual value of env variable its returning the echo statement as is.

EDIT: I have already tried setting shell=True option. Even that didn't give me variable substitution.

I tried to print the entire env inside the subprocess and it shows the variable TEST_VAR set. So I am guessing this is a problem of bash variable extrapolation inside subprocess, since its not running under any shell.

I am curious if it will affect any cmd which depends on env variables. I will run some more test.

Update: I am closing this ticket as duplicate. The issue was how I was passing the commands to Popen

When I changed my calls from

Popen(["/bin/echo", "$TEST_VAR"], stdout=PIPE, stderr=PIPE, env=curenv, shell=True)

to

Popen(["/bin/echo$TEST_VAR"], stdout=PIPE, stderr=PIPE, env=curenv, shell=True)

I was able to get variable substitution.

`cmd=["echo", "$TEST_VAR"]` is not a valid test. It *never* expands `$TEST_VAR` at all. Run `cmd=['sh', '-c', 'echo "$TEST_VAR"']` — Charles Duffy, Mar 04 '20 at 16:46
...it's the same as if you ran `echo '$TEST_VAR'` at a shell; `$TEST_VAR` is passed as literal data, not parsed as an expansion. — Charles Duffy, Mar 04 '20 at 16:47
Yes. It does. The error I was making was passing the command as list instead of string. This works ``` Popen(["/bin/echo $TEST_VAR"], env=d) ``` This doesn't ``` Popen(["/bin/echo", "$TEST_VAR"], env=d) ``` — pragmaticEngineer, Mar 05 '20 at 01:57
`Popen(["/bin/echo $TEST_VAR"], env=d)` only works if you also add `shell=True`, which adds `['sh', '-c']` as the first two list elements, making it exactly identical to what I suggested in my first comment. You can see that yourself in the Python source at https://github.com/python/cpython/blob/master/Lib/subprocess.py#L1700-L1702 — Charles Duffy, Mar 05 '20 at 02:42
Well, not *quite* identical, because your quotes are wrong, so when you use `echo $TEST_VAR` instead of `echo "$TEST_VAR"` you have the bugs discussed in [BashPitfalls #14](https://mywiki.wooledge.org/BashPitfalls#echo_.24foo). — Charles Duffy, Mar 05 '20 at 02:46

score 1 · Answer 1 · answered Mar 04 '20 at 04:12

The $VAR_NAME syntax is recognized, and expanded, by the UNIX shell executing the equivalent statement. Not the program being run. You are passing the $VAR_NAME string literal directly to the echo program. Which, just as you asked, echoes that text. If you want to do something like this you should include shell=True in your call of subprocess.Popen(). However, doing so creates a large number of problems; most of which are security holes. Doing that should be avoided if you care about software security.

Unable to access environment variables inside subprocess

1 Answers1