0

I have a large image stack with 853 images, each image has the dimensions (11779, 15394) pixels and 3 channels. The images have 8bit depth and are saved in BigTiff format. My data is stored on an external SSD which I connect to my computer. What I want to do is to change the values of some pixels depending on the initial value. Since I cannot load the full stack into memory, I use tifffile.memmap to memory map the pixels I want to investigate and change.

Here is a minimal example of how my code looks like:

import numpy as np
from tifffile import memmap

memmap_img_HSV = memmap('img_HSV.tif', mode='r')    #original image stack I want to investigate
memmap_img_result = memmap('img_result.tif', mode='r+')    #another image which I want to overwrite with the results

img_hue = memmap_img_HSV[:,:,:,0]    #first channels of original image

memmap_img_result[:,:,:] = 0    #set all pixels = 0, before I save the results into it

fill_value = 11    #define the variable "fill_value" which will be used later
sigma_list = []    #create a list where I later save coordinates in

array_label = memmap_img_result[z1:z2, y1:y2, x1:x2]    #only memmap indicated pixels
array_hue = img_hue[z1:z2, y1:y2, x1:x2]

count = array_label == fill_value                       #these are the pixels which have already been investigated and replaced by the "fill_value"

orig_value_hue = np.median(array_hue[count])    #determine the median value of the pixel values from "count"

org = memmap_img_result[zv1:zv2, yv1:yv2, xv1:xv2]    #original values of the investigated pixels

if (orig_value_hue+1) < 10:    #condition, which pixels should be replaced with "fill_value": the rest stays how it was before
    res = np.where((memmap_img_HSV[zv1:zv2, yv1:yv2, xv1:xv2] == 0), fill_value, org)

if (orig_value_hue+1) < 10:    #another condition; if this is True, write the coordinates into list 
    res_sigma = np.argwhere(memmap_img_HSV[zv1:zv2, yv1:yv2, xv1:xv2] == 1)
    if len(res_sigma)>0:
        sigma_list.append([zv1,yv1,xv1])
        sigma_list = np.asarray(sigma_list[1])

        if len(sigma_list[sigma+1])>0:
            np.savetxt('list.txt', np.column_stack([sigma_list[1]]))

memmap_img_result.flush()
del memmap_img_HSV
del memmap_img_result

I apologize if the code looks a bit chaotic; it is just a simplification of the much more complicated code. But I hope, the aim gets clear.

Now my problem is the following: When I run the code, it sometimes happens that the code crashes. When writing the file 'list.txt' to the disk, I either get the error message "No space left on device", although there has been a lot of free space on disk (enough for saving the list). Or I get the error message "Invalid argument: 'list.txt'". Additionally, the SSD shows 0 kB size after the error message. I first have to repair the disk and reboot before I can use the disk again. I think it's not a problem with the memmap, but somehow with saving the list as a text-file.

I could already run the code with other (smaller) data without any error message, but it crashes with this data. Does anybody have an idea what might cause the crash or what I could try out?

JaBu
  • 1
  • 3
  • Shouldnt you add from tifffile import memmap, and specify wich version of tifffile you are using ? – pippo1980 Jan 14 '21 at 16:21
  • https://stackoverflow.com/users/453463/cgohlke is on stack https://www.lfd.uci.edu/~gohlke/ send him a message – pippo1980 Jan 14 '21 at 16:24
  • You are right, I added it. My tifffile version is 2019.7.2. I guess that the reason for crashing has something to do with saving the text-file, not with the memmap. – JaBu Jan 15 '21 at 17:38
  • sorry to bother I dont know anything about tiff and menmmap, tried to test your script but wasnt able to find online any tiff compatible with menmmap. I get an error like my tiff is not valid. Any chance you can point me to a small size tif file compatible (I guess a 3 channel type but not sure). I have very limited respurces to chance to try with a 8Gb BigTiff – pippo1980 Jan 15 '21 at 17:46
  • Thank you for your support! I created two test stacks, one with 1 channel and the other with 3 channels. You can download the two stacks here: https://filebin.net/eezqwxutznlnvye8 When you test the code, you will have to replace the coordinates by some concrete numbers, I think [0:100, 0:100, 0:3] should work out. – JaBu Jan 15 '21 at 18:35
  • thanks a lot !!!!!!!!!!!!!!! – pippo1980 Jan 15 '21 at 18:52
  • thanks for the file, too difficult to me, back to data = numpy.random.randint(0, 255, (256, 256, 3), 'uint8') imwrite('temp.tif', data, photometric='rgb') .... nice pic !!!!! – pippo1980 Jan 16 '21 at 16:08
  • Hi, I am asking you since you were very kind last time. I am struggling with numpy array right now, but wondering if you could point me to an easy to understand tiff file format specification for dummies ? – pippo1980 Jan 18 '21 at 08:14
  • trying to understand tiff to numpy array is there any convention about the shape of the array ? I mean is data.shape(400,400,3) equal to data.shape(3,400,400) ? data is the array created from tif file ? And if it is exactly the same, is memmap accessing the elements of the array in the same way ? – pippo1980 Jan 19 '21 at 08:56
  • The shape of the tiff file is [z,y,x,ch], this is also how memmap is accessing the array. Please note that I'm using image stacks, this is why I have the z coordinate (this coordinate addresses the respective image number in the stack). – JaBu Jan 19 '21 at 09:21
  • Just out of curiosity are them fluorescent microscopy images or ? – pippo1980 Jan 19 '21 at 09:38
  • Yes, the images are from a fluorescence microscope. – JaBu Jan 20 '21 at 15:09
  • hi thanks a lot, by the way tiffinfo gives: tiffinfo Teststack_3ch.tif TIFFReadDirectory: Warning, Unknown field with tag 50838 (0xc696) encountered. TIFFReadDirectory: Warning, Unknown field with tag 50839 (0xc697) encountered. ... I lost myself trying to find a way to visualize the array data (both 1ch and 3ch) spyder4 variable explorer supports only 1ch visualization, array with more than 3 dimension are not supported – pippo1980 Jan 20 '21 at 15:39
  • memmap_img_HSV.shape : (4, 3, 300, 400) dim : 4 is that :''The shape of the tiff file is [z,y,x,ch] ''? – pippo1980 Jan 20 '21 at 19:24
  • If it says (4, 3, 300, 400), that would be [z, ch, x, y]. Can you try to run it with these files: https://filebin.net/805eglwmollwbp9s There are 10 images in z-direction. Sometimes there occur so errors with the dimensions, if z=4. – JaBu Jan 21 '21 at 09:26
  • was using Teststack_3ch.tif for memmap_img_HSV.shape : (4, 3, 300, 400) dim : 4 – pippo1980 Jan 21 '21 at 10:39
  • what about the strange tags ? – pippo1980 Jan 21 '21 at 10:40
  • tiffinfo gives images=30 channels=3 slices=10 meaaning 10*3 = rgb = 30 images and z=10 slices? – pippo1980 Jan 21 '21 at 12:32
  • Sorry, I don't understand the tags you mentioned above... Can you open the files with ImageJ? In total there are 10 RGB images, so z=10 and channels=3 is correct. – JaBu Jan 21 '21 at 15:00
  • tiffinfo from Libtiff: tiffinfo - print information about TIFF files – pippo1980 Jan 21 '21 at 15:16
  • at same point my entire system crashed too, I am low on disk space , dont know if is it or the tiffile or your script – pippo1980 Jan 21 '21 at 15:17
  • with memmap_img_HSV.shape : (10, 3, 300, 400) dim : 4 , your img_hue.shape : (10, 3, 300) dim : 3 so its not doing what it should ? or not ? – pippo1980 Jan 21 '21 at 15:56
  • here https://filebin.net/805eglwmollwbp9s I uploaded img_result.tif just your Teststack_3ch.tif opened with memapp (memmap_img_HSV = memmap('Teststack_3ch.tif', mode='r') and resaved with with TiffWriter('img_result.tif') as memmap_img_result: memmap_img_result.write(memmap_img_HSV, photometric='rgb') can you see the differences next to the c=00 , is it related on how memapp reads multipage (or multiple image in tiff I dont jknow) or how it writes them out ? – pippo1980 Jan 22 '21 at 09:11
  • I am lost here . array_label = memmap_img_result[z1:z2, y1:y2, x1:x2] #only memmap indicated pixels cant understand what is trying to accomplish. – pippo1980 Jan 22 '21 at 09:18
  • Sorry, I can't tell you why the file you uploaded looks like this. You can read the documentation about tifffile.memmap here: https://pypi.org/project/tifffile/ Maybe you can find an answer there? To your last question: By replacing the coordinates by "real" numbers (e.g. [0:2,0:100,0:100]), you can just memmap the indicated pixels of the image and not the whole image. But I think this discussion leads a bit away from the initial question. My problem was that when writing the txt. file, the program crashes. The memmap works well so far. – JaBu Jan 22 '21 at 10:36
  • https://filebin.net/805eglwmollwbp9s untitled5.py is what I got so far from tif into np arrays and tifffile, the documentation it's like a lot of examples, not a lot to me that started 6days ago, I'll try to figure out how a numpy.memmap object works its pronbably different from an array representation of a tiff and strange ways to acess channels or pixel – pippo1980 Jan 22 '21 at 11:13

0 Answers0