Python StringIO memory leak

Question

I have a python program that with time slows to a crawl. I've tested thoroughly, and narrowed it down to a method that downloads an image. The method uses cstringIO and urllib. The problem may also be some sort of infinite download with urllib (the program just freezes after a few hundred downloads).

Any thoughts on where the issue may be?

        foundImages = []

        images = soup.find_all('img')
        print('downloading Images')

        for imageTag in images:
            gc.collect()

            url = None
            try:

                #load image into a file to determine size and width
                url = imageTag.attrs['src']
                imgFile = StringIO(urllib.urlopen(url).read())
                im = Image.open(imgFile)
                width, height = im.size

                #if width and height are both above a threshold, it is a valid image
                #so add to recipe images
                if width > self.minOptimalWidth and height > self.minOptimaHeight:
                    image = MIImage({})
                    image.originalUrl = url.encode('ascii', 'ignore')
                    image.width = width
                    image.height = height

                    foundImages.append(image)

                imgFile = None
                im = None
            except Exception:
                print('failed image download url: ' + url)
                traceback.print_exc()
                continue

        #set the main image to be the first in the array
        if len(foundImages) > 0:
            first = foundImages[0]
            recipe.imageUrl = first.originalUrl

        return foundImages

How thoroughly tested? Have you used e.g. [`memory_profiler`](https://pypi.python.org/pypi/memory_profiler)? — jonrsharpe, Apr 20 '14 at 22:11
used objgraph. I'll give memory_profiler a shot. Thanks for the heads up — William Falcon, Apr 20 '14 at 22:18
@WilliamFalcon I would generally suggest narrowing down your error handling - 12 lines seems a lot for a `try`, and `except Exception` is very broad. What do you expect could go wrong? Perhaps have more `try` blocks, each with only 1-2 lines in, test for specific errors, and put innocuous (e.g. `width, height = im.size`) code outside the `try` or in `else`. — jonrsharpe, Apr 20 '14 at 22:20
@WilliamFalcon and on dstromberg's comment, you never do e.g. `imgFile.close()` - `imgFile = None` is *not* the same. — jonrsharpe, Apr 20 '14 at 22:26

Python StringIO memory leak

0 Answers0