http://www.ngcrawford.com/2012/03/29/python-multiprocessing-large-files/ I want to create a dictionary of a large text file using multiprocessing and I found this.But I have some questions about the parameters that author uses in this code:
p = multiprocessing.Pool(4)
what's the parameter of Pool? In other words, what does "4" mean?
for chunk in grouper(10, test_data):
what does "10" mean
Pool(4) means that you start a pool of four worker processes.
The argument of grouper is explained in the function definition on that web page:
def grouper(n, iterable, padvalue=None):
"""grouper(3, 'abcdefg', 'x') -->
('a','b','c'), ('d','e','f'), ('g','x','x')"""
return izip_longest(*[iter(iterable)]*n, fillvalue=padvalue)
The output list will consist of chunks of each n elements.
