0

I am trying to run some simple operations using Rasterio and GDAL but the memory leaks are just killing my computer. I just wanted to open and plot a tif file and it took about 50 GB!!! of Ram, even though the image itself is only 1.2 GB.

The code is pretty simple. 

import rasterio
from rasterio.plot import show
src = rasterio.open("../temp.tif", mode="r+")
datimg = src.read(1)
show(src.read(1))

Here is a picture of the memory utilization in the middle panel. This is all RAM.

Memory utilization

So the process never actually worked, and now I have to essentially log out and log back into my account on my computer to clear the memory.

I had gdal version 2.3.3 and rasterio version 1.0.21 installed, so I was hoping that person upgrading the gdal version to 3.0.1 might fix the issues.

I use the Anaconda distribution on python 3.7, Ubuntu linux 18.04 LTS. But when I tried to upgrade to the newer gdal version using conda install -c conda-forge gdal gdal==3.0.1 I get errors indicating that there are incompatibilities with other packages. 

conda install -c conda-forge gdal gdal==3.0.1
Collecting package metadata (current_repodata.json): done
Solving environment: failed
Collecting package metadata (repodata.json): done
Solving environment: failed

UnsatisfiableError: The following specifications were found to be incompatible with each other:

  - _anaconda_depends -> mkl-service
  - anaconda -> mkl-service==1.1.2=py36he904b0f_5
  - fiona -> gdal[version='>=2.4.1,<2.5.0a0'] -> libgdal==2.4.2=h0845e09_2
  - gdal==3.0.1 -> libgdal==3.0.1=h0845e09_3
  - geopandas -> fiona -> gdal[version='>=2.4.1,<2.5.0a0'] -> libgdal==2.4.2=h0845e09_2
  - libgdal
  - pkgs/main/linux-64::importlib_metadata==0.8=py37_0
  - pkgs/main/linux-64::mkl-service==1.1.2=py37he904b0f_5
  - pkgs/main/linux-64::path.py==11.5.0=py37_0 -> importlib_metadata[version='>=0.5']
  - rasterio -> libgdal[version='>=2.4.1,<2.5.0a0']

So I am not sure what I am supposed to do here? I can't keep logging out and back in to my account each time I run a command. The memory never releases, so if I run any rasterio or gdal command again, the entire system locks up.

krishnab
  • 9,270
  • 12
  • 66
  • 123

1 Answers1

0

I suggest you to use Google Colab. It allowed almost 20 GB RAM as far as I remember a year ago. Try it in Colab, at least you'll be sure that your computer won't crash anymore. Another advice is, try a smaller map to observe what might go wrong.

Ensar Kaya
  • 21
  • 2