1

I'm running a web app that uses ImageMagick to convert PDFs to PNG.

Some weeks ago it started filling the server's HDD, because it stopped deleting the temp files (/tmp/magick-*). We solved that by using a job to clean old temporary files. This started to happen after we changed from converting PDFs to JPGs to PDFs to PNGs (We needed to keep some transparecy).

In the last few days, we started to see this error in the app's log:

FailedToExecuteCommand `"gs" -q -dQUIET -dSAFER -dBATCH -dNOPAUSE -dNOPROMPT -dMaxBitmap=500000000 -dAlignToPixels=0 -dGridFitTT=2 "-sDEVICE=pngalpha" -dTextAlphaBits=4 -dGraphicsAlphaBits=4 "-r300x300" "-sOutputFile=/tmp/magick-1608EvBhAQ3mK7Lu%d" "-f/tmp/magick-1608aLLJuet4cK9S" "-f/tmp/magick-1608QFcecyiIbnxh"' (-1) @ error/delegate.c/ExternalDelegateCommand/461

I thought that it was a side effect of updating the kernel. Maybe something broke when changing from kernel 3.* to 4.*. So we migrated the web app to a fresh install in a new server, but the error keep appearing.

The question is, how can I be sure that the root of the problem is that the server runs out of memory? Which log should I read and what should I look for in them?

Here is a portion of the code that makes the web app to fail:

    with Image(filename=pdf_name, resolution=300) as img:
        ...
        img.crop(left=xi, top=yi, right=xf, bottom=yf)
        img.background_color = Color('white')
        img.alpha_channel = 'remove'
        ...
        img.save(filename=img_name)

Thanks

EDIT:

A strange thing I noticed is that once the error pop ups for a convert, then all subsequent converts fail until I restart the server. It's like something is hanging.

fmw42
  • 46,825
  • 10
  • 62
  • 80
Martin
  • 660
  • 10
  • 23
  • The error is to do with Ghostscript that Imagemagick uses to process pdf's. You say the problem was changing from jpg to png when saving. But you then go onto say you changed the kernel. So you now have two changed items. Before changing the kernel I would have tried converting back to jpg and checked if there were any problems. You have now changed to a different server. I would assume at some point at the start of this problem either Imagemagick or Ghostscript was updated. Do you know what you were using when everything was OK? – Bonzo Feb 08 '18 at 20:48
  • Just to clarify, the order of the events was: 1- Changed JPG to PNG, 2- Changed Kernel (our VPS provider requested to do so because they were changing some CPUs to mitigate the effects of Spectre and Meltdown) and 3 migrated to a new server. Between steps 1 and 2 there were about 3 months, from step 2 to 3 a week or so. The ideal thing would have been to ckeck by changing that, yes. Ghostscript and Imagemagick versions are the same. – Martin Feb 08 '18 at 21:15
  • Move the conversion process to a managed queue worker on the background. Something that allows python to cleanly shutdown with each completed task. – emcconville Feb 08 '18 at 21:36
  • @emcconville I need the conversion to be done synchronously. If I send the process to a queue I'll need to do some polling until the file is ready, don't I? – Martin Feb 08 '18 at 23:23
  • Ghostscript is failing with a `gs_error_unknownerror` state. It's hard to suggest a solution as countless local/environment variables can contribute. I do know that modern web app frameworks offer some way to interact with a queue / messaging / deferment service, and [FSM](https://en.wikipedia.org/wiki/Finite-state_machine) take little effort to implement. Worth looking into. – emcconville Feb 09 '18 at 16:17
  • 1
    @emcconville I'll look into that. I edited the post to add that after doing some research I noticed that the system isn't freeing RAM after the conversion is successfully done. It starts to fill until it eventually get's full and the app starts to fail. – Martin Feb 09 '18 at 16:50
  • The same thing is happening in my server, same error and same effects, I'm not sure when it happens but once it happens it doesn't work anymore, and I have to restart the service to start work again. @Martin did you find anything new or fixed this issue? I know this is an old post but I'm stuck for over a week – Carlos Barros Apr 13 '21 at 17:23
  • Hi @CarlosBarros, in our case we are using Gunicorn. We "solved" the issue by recycling the workers after some number of requests handled. The setting used is https://docs.gunicorn.org/en/latest/settings.html#max-requests. After some tunning we found out that 50 requests is the number that works for us. Let me know how it goes. – Martin Apr 14 '21 at 18:21
  • Thanks @Martin for your answer, I fixed it! My problem was nothing to do with imagemagick but with another library, I had other library for Deskewing the image after converting the pdf to image and it had a memory leak, so the next time(s) I used imagemagick it would crash and restart the server because it would have the ram full. – Carlos Barros Apr 21 '21 at 14:40

0 Answers0