Python parse string containing functions, lists and dicts

Question

I'm trying to find way to parse string that can contain variable, function, list, or dict written in python syntax separated with ",". Whitespace should be usable anywhere, so split with "," when its not inside (), [] or {}.

Example string: "variable, function1(1,3), function2([1,3],2), ['list_item_1','list_item_2'],{'dict_key_1': "dict_item_1"}"

Another example string: "variable,function1(1, 3) , function2( [1,3],2), ['list_item_1','list_item_2'],{'dict_key_1': "dict_item_1"}"

Example output ["variable", "function1(1,3)", "function2([1,3],2)", "['list_item_1','list_item_2']", "{'dict_key_1': "dict_item_1"}"]

edit: Reason for the code is to parse string an then run it with exec("var = &s" % list[x]). (yes i know this might not be recommended way to do stuff)

score 2 · Answer 1 · answered Sep 17 '16 at 13:33

I guess the main problem here is that the arrays and dicts also have commas in them, so just using str.split(",") wouldn't work. One way of doing it is to parse the string one character at a time, and keep track of whether all brackets are closed. If they are, we can append the current result to an array when we come across a comma. Here's my attempt:

s = "variable, function1(1,3),function2([1,3],2),['list_item_1','list_item_2'],{'dict_key_1': 'dict_item_1'}"

tokens = []
current = ""
open_brackets = 0

for char in s:
    current += char

    if char in "({[":
        open_brackets += 1
    elif char in ")}]":
        open_brackets -= 1
    elif (char == ",") and (open_brackets == 0):
        tokens.append(current[:-1].strip())
        current = ""

tokens.append(current)

for t in tokens:
    print(t)

"""
    variable
    function1(1,3)
    function2([1,3],2)
    ['list_item_1','list_item_2']
    {'dict_key_1': 'dict_item_1'}
"""

I was thinking of the same idea using a `list` as a stack of brackets, but the `open_brackets` counter works the same and is simpler. — chapelo, Sep 17 '16 at 13:40
Yeah i just thought that regex/python would have had way of doing it instead of writing algorithm my self. I will have to do that then i suppose. — SacredCoconut, Sep 17 '16 at 13:48

score 0 · Answer 2 · answered Sep 17 '16 at 12:57

0

Have you tried using split?

>>> teststring = "variable, function1(1,3), function2([1,3],2), ['list_item_1','list_item_2'],{'dict_key_1': 'dict_item_1'}"
>>> teststring.split(", ")
['variable', 'function1(1,3)', 'function2([1,3],2)', "['list_item_1','list_item_2'],{'dict_key_1': 'dict_item_1'}"]

answered Sep 17 '16 at 12:57

Jack Evans

1,697
3
17
33

Oh yeah i forgot to mention it might or might not have whitespace after ",". For example "['variable', function1(1, 3)'" would not work. – SacredCoconut Sep 17 '16 at 13:00

score 0 · Accepted Answer · answered Sep 17 '16 at 13:08

0

Regular expressions aren't very good for parsing the complexity of arbitrary code. What exactly are you trying to accomplish? You can (unsafely) use eval to just evaluate the string as code. Or if you're trying to understand it without evaling it, you can use the ast or dis modules for various forms of inspection.

answered Sep 17 '16 at 13:08

ShadowRanger

143,180
12
188
271

Reason for the code is to parse string into list and then run it with exec(var = list[x]). – SacredCoconut Sep 17 '16 at 13:17
@SacredCoconut: Is there a reason not to bulk parse it all at once? – ShadowRanger Sep 17 '16 at 13:38
Sorry i don't know what you mean by bulk parse. – SacredCoconut Sep 17 '16 at 13:53
@SacredCoconut: As in, why not just do `allvals = eval(thestring)`, then index the resulting `tuple` for the results? – ShadowRanger Sep 17 '16 at 13:57
Oh that might work, i have most likely misunderstood how eval works – SacredCoconut Sep 17 '16 at 14:01
Only downside is that might want to know what the function, variable name is. But other than that that does seem most simple way. I think that or building the algorithm my self will be enough for now. – SacredCoconut Sep 17 '16 at 14:15

Python parse string containing functions, lists and dicts

3 Answers3

Linked