how to add a list, passed from a subprocess to parent process, to an already existing list in python

Question

I am passing a list from a subprocess to the parent process and in the parent process I want to add this to an already existing list. I did this:

subprocess_script.py:

def func():

    list = []
    list.append('1')
    list.append('2')

    print'Testing the list passing'
    print '>>> list:',list

if __name__ == '__main__':
     func()

parent_script.py:

list1 = []
list1.append('a')
list1.append('b')
ret = subprocess.Popen([sys.executable,"/Users/user1/home/subprocess_script.py"],stdout=subprocess.PIPE)
ret.wait()

return_code = ret.returncode
out, err = ret.communicate()
if out is not None:
        for line in out.splitlines():

            if not line.startswith('>>>'):
                continue
            value = line.split(':',1)[1].lstrip()
            list1.extend(value)
print 'Final List: ',list1

But when I execute this I do not get the desired output. The final list that I want should be : ['a','b','1','2']. But I get ['a', 'b', '[', "'", '1', "'", ',', ' ', "'", '2', "'", ']'] which is wrong. What am I doing wrong here?

Answer 1

The problem is that after your split and lstrip calls, value is still a string, not a list yet. You can stop the script including a pdb.set_trace line and inspect it like this:

if not line.startswith('>>>'):
    continue
import pdb; pdb.set_trace()
value = line.split(':', 1)[1].lstrip()
list1.extend(value)

And then run the code:

❯ python main.py
> /private/tmp/example/main.py(19)<module>()
-> value = line.split(':', 1)[1].lstrip()
(Pdb) line
">>> list: ['1', '2']"
(Pdb) line.split(':', 1)[1].lstrip()
"['1', '2']"

You can evaluate that string into a Python list by using the ast.literal_eval function, like this:

(Pdb) import ast
(Pdb) ast.literal_eval(line.split(':', 1)[1].lstrip())
['1', '2']

Now list1 can be extended with this value.

From the Python 2.7 documentation:

ast.literal_eval(node_or_string)

Safely evaluate an expression node or a Unicode or Latin-1 encoded string containing a Python literal or container display. The string or node provided may only consist of the following Python literal structures: strings, numbers, tuples, lists, dicts, booleans, and None.

This can be used for safely evaluating strings containing Python values from untrusted sources without the need to parse the values oneself. It is not capable of evaluating arbitrarily complex expressions, for example involving operators or indexing.

Answer 2

You are doing it wrongly.

When you do - print '>>> list:',list . It would print -

>>> list: [1, 2]

And when you do - value = line.split(':',1)[1].lstrip() , value would become the string -

'[1, 2]'

And when extending list1 with value , each character in value would be added as a new element in list1 (because it would iterate over each element of the string, which is each character and add them to the list1).

When creating the value , you want to remove the first [ as well as the trailed ] and then split them based on , .

Example code -

value = line.split(':',1)[1].lstrip().lstrip('[').rstrip(']').replace("'",'').replace(" ",'').split(',')

The above code is a very big hack , better would be to use ast.literal_eval as @logc mentioned in his answer -

import ast
value = ast.literal_eval(line.split(":",1)[1].lstrip())

But please be vary, that ast.literal_eval would evalutate the expression and return the result, you should use it with care.

Answer 3

Use a standard serialization data format, like JSON:

script.py

import json


def func():

    lst = []
    lst.append('1')
    lst.append('2')

    print json.dumps(lst)  ## <-- `dumps` dumps to a string

if __name__ == '__main__':
    func()

main.py

import sys
import os
import subprocess
import json

list1 = []
list1.append('a')
list1.append('b')
ret = subprocess.Popen([sys.executable, os.path.join(os.getcwd(), "script.py")], stdout=subprocess.PIPE)
ret.wait()

return_code = ret.returncode
out, err = ret.communicate()
line = next(line for line in out.splitlines())
value = json.loads(line)  ## <-- `loads` loads from a string
list1.extend(map(str, value))
print 'Final List: ', list1

The map(str, value) is just aesthetic: it is there to have a uniform list, because json.dumps will produce Unicode strings by default, and your previous list1 elements are not Unicode strings.

Also, I removed the whole header-printing and line-skipping parts of the code. You are just doing your life more difficult with them :)