0

I am using dask (2021.3.0) and rapids(0.18) in my project. In this, I am performing preprocessing task on the CPU, and later the preprocessed data is transferred to GPU for K-means clustering. But in this process, I am getting the following problem:

1 of 1 worker jobs failed: std::bad_alloc: CUDA error: ~/envs/include/rmm/mr/device/cuda_memory_resource.hpp:69: cudaErrorMemoryAllocation out of memory (before using GPU memory completely it gave the error i.e. it is not using GPU memory completely)

I have a single GPU of size 40 GB. Ram size 512 GB.

I am using following snippet of code:

cluster=LocalCluster(n_workers=1, threads_per_worker=1)
cluster.scale(100)
##perform my preprocessing on data and get output on variable A
# convert A varible to cupy
x = A.map_blocks(cp.asarray)
km =KMeans(n_clusters=4)
predict=km.fit_predict(x).compute()

I am also looking for a solution so that the data larger than GPU memory can be preprocessed, and whenever there is a spill in GPU memory the spilled data is transferred into temp directory or CPU (as we do with dask where we define temp directory when there is a spill in RAM).

Any help will be appriciated.

Vivek kala
  • 23
  • 3

1 Answers1

2

There are several ways to run larger than GPU datasets.

  1. Check out Nick Becker's blog, which has a few methods well documented
  2. Check out BlazingSQL, which is built on top of RAPIDS and can perform out of core processings. You can try it at beta.blazingsql.com.
TaureanDyerNV
  • 1,208
  • 8
  • 9
  • Dear TaureanDyerNV, Thank you for your kind attention to my problem. However, I am using CPU-based preprocessing on the inputted dataset by first creating LocalCluster, and then transfer data to GPU, which can be controlled via LocalCudaCluster. In this situation, I am wondering how to transfer data from LocalCluster to LocalCudaCluster. In my code snippet, I directly transfer my data from LocalCluster to GPU without making LocalCudaCluster. Could you please help me in this regard also, as I am not getting any solution for this. – Vivek kala Mar 20 '21 at 05:05