2

I have a Django GIS-related application where users can download shp files. I have the geopandas GeoDataFrame object. I can easily convert it to a zipfile and then read the zipfile to the user when they want to download it:


from django.http import HttpResponse
import geopandas as gpd
import shapely
import os
from zipfile import ZipFile
def download_shp_zip(request):
    # just some random polygon
    geometry = shapely.geometry.MultiPolygon([
        shapely.geometry.Polygon([ (0, 0), (0, 1), (1, 1), (1, 0) ]),
        shapely.geometry.Polygon([ (2, 2), (2, 3), (3, 3), (3, 2) ]),
    ])
    # create GeoDataFrame
    gdf = gpd.GeoDataFrame(data={'geometry':geometry}, crs='epsg:4326')
    # some basename to save under
    basename = 'basename'
    #  create folder for this session
    os.mkdir(f"local_folder/{basename}")
    # export gdf to this folder
    gdf.to_file(f"local_folder/{basename}/{basename}.shp")
    # this file now contains many files. just zip them
    zipObj = ZipFile(f"local_folder/{basename}.zip", 'w')
    # zip everything in the folder to the zip
    for file in os.listdir(f"local_folder/{basename}"):
        zipObj.write(f"local_folder/{basename}/{file}")
    # create zip
    zipObj.close()
    # now delete the original files that were zipped
    shutil.rmtree(f"local_folder/{basename}")

    # now we can server the zip file to the user
    filename = f'local_folder/{basename}.zip'
    # check if file exists (just in case)
    try:
        fsock = open(filename, "rb")
    except:
        return HttpResponse(f"File '{basename}' Does Not Exist!",
            content_type='text/plain')
    # create response
    response = HttpResponse(fsock, content_type='application/zip')
    response['Content-Disposition'] = f'attachment; filename={basename}.zip'
    return response

This method works, but it saves the file to local storage and then serves it. However, I want to save them to Memory and then serve that to the user instead. I've seen people use os.BytesIO to do something similar. I've been playing around, but I can't quite get anything to work for what I'm looking for.

Again, I have the GeoDataFrame. I want to convert it to a shapefile and then zip it and then serve the zipped folder to the user from Memory without writing it to Local Storage.

Heinrich
  • 340
  • 1
  • 4
  • 12

1 Answers1

5

It would be possible to create the file and delete it instantly after serving by using tempfile. So not the answer you were looking for but maybe still of help. According to this answer:

Any file created by tempfile will be deleted once the file handler is closed. In your case, when you exit the with statement.

from django.http import HttpResponse
import geopandas as gpd
import shapely
import os
import tempfile
from zipfile import ZipFile

def download_shp_zip(request):

    # just some random polygon
    geometry = shapely.geometry.MultiPolygon([
        shapely.geometry.Polygon([(0, 0), (0, 1), (1, 1), (1, 0)]),
        shapely.geometry.Polygon([(2, 2), (2, 3), (3, 3), (3, 2)]),
    ])
    # create GeoDataFrame
    gdf = gpd.GeoDataFrame(data={'geometry': geometry}, crs='epsg:4326')

    # some basename to save under
    basename = 'basename'

    # Convert to shapefile and serve it to the user
    with tempfile.TemporaryDirectory() as tmp_dir:

        # Export gdf as shapefile
        gdf.to_file(os.path.join(tmp_dir, f'{basename}.shp'), driver='ESRI Shapefile')

        # Zip the exported files to a single file
        tmp_zip_file_name = f'{basename}.zip'
        tmp_zip_file_path = f"{tmp_dir}/{tmp_zip_file_name}"
        tmp_zip_obj = ZipFile(tmp_zip_file_path, 'w')

        for file in os.listdir(tmp_dir):
            if file != tmp_zip_file_name:
                tmp_zip_obj.write(os.path.join(tmp_dir, file), file)

        tmp_zip_obj.close()

        # Return the file
        with open(tmp_zip_file_path, 'rb') as file:
            response = HttpResponse(file, content_type='application/force-download')
            response['Content-Disposition'] = f'attachment; filename="{tmp_zip_file_name}"'
            return response

Another option would be to use the return statement inside a try-except block and to delete the file in the finally statement:

        # Return the zip file as download response and delete it afterwards
        try:
            with open(tmp_zip_file_path, 'rb') as file:
                response = HttpResponse(file, content_type='application/force-download')
                response['Content-Disposition'] = f'attachment; filename="{tmp_zip_file_name}"'
                return response
        finally:
            os.remove(tmp_zip_file_path)

For a better user experience, you can use a loading gif until the download is starting.

Helge Schneider
  • 483
  • 5
  • 8
  • I see, so in this solution, you are still writing to an actual file and not to memory, but deleting it instantly after use. How would this work on a server that is performing multiple downloads simultaneously? Wouldn't there be file lock issues? I am currently using a similar solution and I fear that once the app grows, this will become an issue. Thus, I hope there is a way to actually server it from memory and not a file – Heinrich Jun 25 '21 at 16:31
  • Sorry, I didn't realize that the file is not created in memory. I dont know how this would perform on a server with simultaneous downloads, but think it should be allright. According to [this answer](https://stackoverflow.com/questions/18550127/how-to-do-virtual-file-processing) you might be able to use a [SpooledTemporaryFile](https://docs.python.org/3/library/tempfile.html#tempfile.SpooledTemporaryFile). However this does not let you create a directory in memory, which is needed to zip the file. Sadly I cannot test this atm as I'm in holidays, but I will come back to this once I'm back. – Helge Schneider Jun 25 '21 at 18:55
  • Good find! I will try to implement it when I start working again (in about two weeks) and let you know. Otherwise if you beat me to it, lemme know or post another answer! – Heinrich Jun 26 '21 at 20:17