How to expand environment variable in python while escaping special characters?

Question

On the host, I have an environment variable with a quote sign in the value, something like:

export VALUE_WITH_QUOTE_FROM_OS='quote"value'

and when I echo on bash, it's fine

#echo $VALUE_WITH_QUOTE_FROM_OS
quote"value

I have a following json string:

json_str = '{"key":"${VALUE_WITH_QUOTE_FROM_OS}"}'

Then I want to expand the environment variable inside a python script before further processing, something like this

json_str = os.path.expandvars(json_str)
json_dict = json.loads(json_str)

However, this expansion would break the json syntax, since the json_str had become

'{"key":"quote"value"}' (<== bad unescaped quote in the value)

instead of

'{"key":"quote\"value"}'

Is there anyway I can inform os.path.expandvars() to escape the double quote when expanding the value? If not, how should I expand the environment variable so that the double quote can be escaped.

Note 1 The value of the environment variable is a security token, so I have to keep the double quote there as it is.

Note 2 Current json interface has already been determined and used extensiely as it is. This json_str is passed to me, therefore I should and must only expand the environment variable denoted by ${} in the json string, no other modification are allowed.

Note 3 This json_str is extremely large with complex, dynamic, nested structure and is consumed by multiple clients who cannot access host os environment variable. Although it's possible for me to load the json_str first, traverse the dictionary to resolve the environment variable, and then dump the dict back to json_str and then dispatch to all the clients, I think it is less efficient compared to processing it just as a string.

Thanks.

Do you need to expand the environment variable in a json string? Could you do something simple like: `{"key": os.env['VALUE_WITH_QUOTE_FROM_OS']}` — mgilson, Mar 09 '16 at 23:57
@mgilson Yes,this json_str is provided by the user, so I **must expand** the json string.and cannot modify the json_str other than that. — cookieisaac, Mar 10 '16 at 00:07

mgilson · Accepted Answer · 2016-03-10T00:24:29.713

I certainly don't guarantee that this will have the portability that you get with os.path, but this should mostly work for systems that use posix I think:

import re
import os
import json

regex = re.compile(r'\$(\w+|\{[^}]*\})')
json_str = '{"key":"${FOO}"}'
def os_expandvar(match):
    v = match.group(1)
    if v.startswith('{') and v.endswith('}'):
        v = v[1:-1]
    return json.dumps(os.environ.get(v, ''))[1:-1]
print(regex.sub(os_expandvar, json_str))

The regular expression (and implementation ideas in general) were borrowed from the implementation of os.path.expandvars in the posixpath module. I removed a lot of the complexity to simplify the answer, but you can put it back in should you find that you need it.

This should handle cases where the substitution strings are $FOO or ${FOO} as is typical on a posix system.

There are other options here too ... With the example you gave, you could decode the json first and then expand all of the values. Depending on the format of the json, you might need a recursive function to get the job done:

# untested
def json_expandvars(o):
    if isinstance(o, dict):
       return {json_expandvars(k): json_expandvars(v) for k, v in o.items()}
    elif isinstance(o, list):
       return [json_expandvars(v) for v in o]
    elif isinstance(o, basestring):
       return os.path.expandvars(o)
    else:
       return o

json_dict = json_expandvars(json.loads(json_str))

thanks for the answer! I guess sometimes I have to reinvent the wheel to have it customized. — cookieisaac, Mar 10 '16 at 18:34

Barmar · Answer 2 · 2016-03-11T05:14:43.880

2

There's no way for os.path.expandvars to know that the result needs to be a valid JSON string, with quotes escaped.

Instead, expand the environment variable after you decode the JSON into a dictionary.

json_dict = json.loads(json_str);
json_dict['key'] = os.path.expandvars(json_dict['key']);

edited Mar 11 '16 at 05:14

answered Mar 10 '16 at 00:31

Barmar

741,623
53
500
612

This is definitely a possibility, but this json_str is extremely large with very complex, unbounded level of nesting (ususally 6 to 7 layers but there is no hard coded limit). This json_str is shared by lots of clients, each of which only cares about its own part and knows exactly how to interpret it. So currently, I want to stay with this design, and only fill in the environment variables in a string processsing fashion for the json_str, then dispatch it still as a string to all the clients who knows exactly how to handle their own part. – cookieisaac Mar 10 '16 at 02:00
I don't think there's any reliable way to do that, other that writing your own version of `expandvars` that escapes quotes that are found in the values. – Barmar Mar 10 '16 at 02:29
I'm pretty sure that `json_dict.key` will raise an `AttributeError` here. I find myself making this mistake when I've been working in Javascript too much... – mgilson Mar 10 '16 at 18:36

How to expand environment variable in python while escaping special characters?

2 Answers2

Linked