0

I'm currently working on a project where I have an image of around 3.9gb. I want to create a google maps like view for this image (which is something LibVIPS can generate) by executing the following command:

vips-dev-8.1.1\bin\vips.exe dzsave testje-131072.tiff mydz

However when doing this some warnings are shown and after that the program crashes:

vips warning: tiff2vips: no resolution information for TIFF image "testje-131072.tiff" -- defaulting to 1 pixel per mm
vips warning: tiff2vips: no resolution information for TIFF image "testje-131072.tiff" -- defaulting to 1 pixel per mm
vips warning: vips_tracked: out of memory --- size == 48MB

Crash

Anyone got a clue what I could do to be able to process an image of this size using Vips? (Or any other library?).

I've done some investigation myself and it seems we need to have BigTiff, I've looked in the VIPS source code and saw the term BigTiff being used a number of times so I suppose it should be supported?

Some information about the image:
Width: 131072
Height: 131072
Chunks: 32x32 (4096x4096 each)
Compression: LZW

When opening the image in a tool like VLIV (Very Large Image Viewer) the image opens fine.

Devedse
  • 1,801
  • 1
  • 19
  • 33
  • how much memory do you have? What windows is this? `x64` ? is the `vips.exe` 32bit ? some reasons: from win Vista and later the Process scheduler changes a lot and now older programs that does not response inside much smaller time limit will be freezed. you can try run the program in XP SP3 compatibility mode wtih turned off advanced gfx stuff that sometime helps (prevents the program has stop working message). If you got 32bit App then you are limited to 4 or 2 GB of memory (depends on used memory manager of the app) and allocation of chunks above 1GB can be problem sometimes. – Spektre Oct 27 '15 at 07:53
  • In that case try to find 64bit version of the App (if you got 64 bit OS of coarse) or cut the image to smaller pieces. As this is not a programing question I vote to close – Spektre Oct 27 '15 at 07:54
  • Thanks for your answer :). There seems to be only a 32 bit version of the program (I have 32gb of memory though), the reason I would like to "not close" this topic is because it's not solved. I posted my question here because LibVIPS is an opensource software package that seems to have problems reading my image. I don't really see a difference between including the library in my project or directly calling VIPS.exe. (To make my question as simple and focussed as possible I chose to use their version of the compiled program instead of using their library in my own project). – Devedse Oct 27 '15 at 08:14
  • 1
    1. 32 bit programs runing on x64 windows are running inside WoW64 emulator and have access only up to 4 GB of memory (does not matter if you have 8GB or 128GB installed). Some memory managers (depends what with your program was compiled) can use only 2GB address space. I also found out that sometimes under WoW64 if you want to allocate single block with more than 1 GB in size that sometimes crashes even if you got that much of memory (from that 4GB). This site is for questions related to programing. I think you should post this in different Stack Exchange site not sure which may be super user – Spektre Oct 27 '15 at 08:22
  • But if LibVIPS is OpenSource than you could try to build it as 64bit app that is perfectly suitable for this site... just download the package source code repository (tortoise is good for this) then install some 64bit compiler compatible with this package (I guess some MSVC++ or GCC) and compile 64 bit binaries ... I am afraid you will need some aditional stuff like frameworks SDK;s as nowdays programmers are unable to do standalone stuff instead you need to include countless crappy libs to make helo world example ... and need tons of crap installed to run exe ... – Spektre Oct 27 '15 at 08:26
  • I think user894763 answered my question by stating that libvips loads 2 lines of tiles in memory which results in the high memory usage. I will try to lower the tile size and see how that works out. Thank you too :) – Devedse Oct 27 '15 at 15:58

1 Answers1

4

I'm the libvips maintainer. The vips.exe binary includes bigtiff support and should be easily able to process an image of this size. It's challenging to build yourself on Windows, perhaps a week's work, I wouldn't try to make your own unless you are very expert.

I think the problem is probably your input image. I think it is using very large tiles (4096 x 4096). libvips is having to keep two complete lines of tiles in memory, so 4096 x 131072 x 3 x 2 pixels, which is 3GB straight away.

I would remake your source image. Use smaller tiles, perhaps 512 x 512, and make sure you are writing a bigtiff image. Please open an issue on the libvips tracker if you still have problems, it's easier to debug stuff there.

https://github.com/jcupitt/libvips/issues

Edit: there's now an official 64-bit Windows build of libvips and vips.exe, it might help:

http://www.vips.ecs.soton.ac.uk/supported/current/win32/vips-dev-w64-8.1.1-2.zip

jcupitt
  • 10,213
  • 2
  • 23
  • 39
  • Ah this explains a LOT. I indeed already heard that compiling libvips on Windows was quite a challenge. I will try to decrease the tile size and recreate the image to see what happens then. Maybe a bit offtopic, but what tiff version does libvips use? Is that 4.0.6 from http://www.remotesensing.org/libtiff/ which follows the Tiff 6.0 spec or is it something completely different? – Devedse Oct 27 '15 at 15:51
  • Hi again, the windows vips is built by this: https://github.com/jcupitt/build-win32 You can see the tiff library here: https://github.com/jcupitt/build-win32/blob/master/8.1/vips.modules#L265 So it's libtiff 4.0.6. – jcupitt Oct 27 '15 at 21:10