Dealing with large TIFF images C++/Magick/libtiff

Question

I'm writing C++ software based on OpenCV to analyse a large quantity of large TIFF images - feature extraction and characterisation. My current approach is to attempt to split each image into smaller cropped sections, analyse each one individually and amalgamate the results. I am attempting to use Image Magick convert function to do the cropping, with flags:

convert input.tif -crop 4000x4000 +repage -scene 0 "output%d.tif"

This, however, runs for hours, utilising 100% of RAM and 100% of IO throughput without producing any output.

The image details as given by magick's identify input.tif are:

input.tif TIFF64 63488x49408 63488x49408+0+0 8-bit sRGB 9.4112GB 0.000u 0:00.031

The same instruction ran on a smaller (roughly 20k by 20k) downsampled version of the image returns output within 5 seconds.

The computer has 8GB RAM, windows 8 x64 and a 2.00GHz CPU.

Would anyone be able to advise me on how to establish what is going wrong? Otherwise, could anyone advise me on an alternative way to tackle this problem?

EDIT 1: More info

 c:\test>identify -list resource
Resource limits:
  Width: 214.7MP
  Height: 214.7MP
  Area: 14.888GP
  Memory: 6.9326GiB
  Map: 13.865GiB
  Disk: unlimited
  File: 1536
  Thread: 4
  Throttle: 0
  Time: unlimited

You could maybe have a look at, and vote for ;-) my answer here... http://stackoverflow.com/a/31046397/2836621 — Mark Setchell, Jul 02 '15 at 10:29
Thanks Mark, I'll give that a go in a bit. I'm new on here, so I don't think I can vote just yet. — Oxonon, Jul 02 '15 at 10:43
No problems - don't be put off by the fact it addresses JPEG - it's the same for TIFF, just change the extension. — Mark Setchell, Jul 02 '15 at 10:46

score 1 · Answer 1 · edited May 23 '17 at 12:15

I think my other answer here should help you considerably, but one other thing you could check, is that you are allowing ImageMagick to use your 8GB of RAM. Try this command

identify -list resource
  Resource limits:
  Width: 214.7MP
  Height: 214.7MP
  Area: 4.295GP
  Memory: 2GiB        <---
  Map: 4GiB
  Disk: unlimited
  File: 192
  Thread: 1
  Throttle: 0
  Time: unlimited

and check the Memory option.

If it is low, you can increase it on the command line with something like this

convert -limit memory 6GiB ...

or with an environment variable like this:

export MAGICK_MEMORY_LIMIT=8000000000
identify -list resource
Resource limits:
  Width: 214.7MP
  Height: 214.7MP
  Area: 4.295GP
  Memory: 7.4506GiB     <---
  Map: 4GiB
  Disk: unlimited
  File: 192
  Thread: 1
  Throttle: 0
  Time: unlimited

That's stupid. No sane program should take 8GiB of memory while converting a tiff file. — polkovnikov.ph, Mar 25 '16 at 01:31

Dealing with large TIFF images C++/Magick/libtiff

1 Answers1