Using all cores in Dask

Question

I am working on a google cloud computing instance with 24 vCPUs. The code running is the following

import dask.dataframe as dd
from distributed import Client
client = Client()

#read data
logd = (dd.read_csv('vol/800000test', sep='\t', parse_dates=['Date'])
         .set_index('idHttp')
         .rename(columns={'User Agent Type':'UA'})
         .categorize())

When I run it (and this is also the case for the posterior data analysis I am doing after loading the data) I see 11 cores being used, sometimes 4.

Is there any way to control this better and make full use of cores?

The picture look pretty nice. May I know how to draw the pic above? — Pengju Zhao, Mar 06 '18 at 02:19

score 1 · Accepted Answer · answered Mar 07 '17 at 18:25

read_csv will split your files into chunks according to the chunksize parameter, at least one chunk per input file. You are reading only one file, and it seems you are getting four partitions (i.e., size < 4 * 64MB). This may be reasonable for the amount of data, and extra parallelization of many small tasks maybe would only add overhead.

Nevertheless, you could change the blocksize parameter and see what difference it has for you, or see what happens when you pass multiple files, e.g., read_csv('vol/*test'). Alternatively, you can set the partitioning in the call to set_index.

Using all cores in Dask

1 Answers1