0

I'm using xarray to read in and modify a data set for my analysis.

That's the data repr.:

enter image description here

To plot the data I have to convert it to numpy array:

Z_diff.values()

When doing so I get the error message:

slurmstepd: error: Detected 1 oom-kill event(s) in step 33179783.batch cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler.

I'm using the following settings:

#SBATCH --ntasks-per-node=16
#SBATCH --nodes=4
#SBATCH --mem=250G
Majid Hajibaba
  • 3,105
  • 6
  • 23
  • 55
Martina
  • 21
  • 2
  • 1
    Given that the whole array is only 724 MB, I'd guess you're not showing us a significant part of your workflow. Do you load data and then reduce the size, ending up with this `z_diff` array? [dask evaluates tasks lazily](https://tutorial.dask.org/01x_lazy.html), so `z_diff.values` does not simply access the whole 724 MB, it also executes any tasks which the array's values depend on. check the array & chunk sizes in each step of your workflow and see if you can identify any steps which might be causing the problem. – Michael Delgado Feb 03 '22 at 01:56

1 Answers1

0

It looks like you're just running out of memory. When using dask with xarray (link), the data is split into chunks (183 of them in your case). So only a small portion of the dataset is kept in memory at once. Numpy arrays don't function that way so the whole dataset is being read into memory when you use .values() and you're exceeding your memory.

Depending on what you're trying to plot, you might be able to plot each chunk individually or plot data from each chunk one at a time on the same plot. Or just plot a subset of the data to avoid reading all the data into memory at once. Lastly, if available, you could potentially also request more memory until you no longer get this error.

pasnik
  • 226
  • 1
  • 5