I tried to use ray for crawling some data.
My original code before using ray is as below, and works well.
def download(n):
#download nth data
return downloaded_data
I use ray in reference to ray tutorial, and this makes my kernel dead:
@ray.remote
def download(n):
#download nth data
return downloaded_data
ray.init(num_cpus = 4)
total_data = ray.get([download.remote(x) for x in range(4000)])
I tried the code through jupyter and spyder, but in both cases kernel died.
I do not think the reason is memory lack. It does not use so much memory.
What are the possible causes?
