0

Tensorflow: 2.6.0, Ubuntu 20.04.3 LTS, GPU: GeForce MX130, CUDA version: 11.2

I have a dataset that contains 32-bit image files and 8-bit mask files, both in tiff format. It is a huge dataset. Therefore I would like to load my data in tf.data.Dataset format instead of numpy arrays (since this makes the loading faster and avoids memory issues). Decoding my tiff files using tf.io.read_file(img_path) and then tfio.experimental.image.decode_tiff(img) doesn't work here since it would throw the following error: TIFFReadDirectory: Warning, Unknown field with tag 42113 (0xa481) encountered. TIFFReadDirectory: Warning, Unknown field with tag 42113 (0xa481) encountered. memory: Sorry, can not handle images with 32-bit samples.

Therefore I decided to use tiffile library to decode the data as following: img = tiff.imread(img_path)

Here is a piece of my code:

full_dataset = tf.data.Dataset.list_files(dataset_path + "*.tif", shuffle=False)
full_dataset = full_dataset.shuffle(buffer_size=100, seed=42)

train_dataset = full_dataset.take(train_size)
test_dataset = full_dataset.skip(train_size)
val_dataset = test_dataset.skip(val_size)
test_dataset = test_dataset.take(test_size)

AUTOTUNE = tf.data.experimental.AUTOTUNE

train_dataset = train_dataset.map(lambda x: tf.py_function(preprocess, inp=[x], Tout= [tf.float32, tf.uint8]))
val_dataset = val_dataset.map(lambda x: tf.py_function(preprocess, inp=[x], Tout=[tf.float32, tf.uint8]))
test_dataset = test_dataset.map(lambda x: tf.py_function(preprocess, inp=[x], Tout=[tf.float32, tf.uint8]))

I'm using tf.py_function here because I would like to use tiff library inside the map function to read the image. For that, I need the image path to be in python string format. I noticed that if I use the usual map function, the path is sent to the function as a tensor of string type.

Here is the preprocess function:

def preprocess(img_path: str):
  f_name = bytes.decode(img_path.numpy())     # for this you need tf.py_function
  img = tiff.imread(f_name)

  img = img[:, :, [0, 2, 3]]  # The image has 4 channels, I need only 3 of them. Therefore extracting those using indices
  img[:, :, 0] = img[:, :, 0] / img[:, :, 0].max() # normaliing values in all 3 channels
  img[:, :, 1] = img[:, :, 1] / img[:, :, 1].max()
  img[:, :, 2] = img[:, :, 2] / img[:, :, 2].max()


  mask_path = tf.strings.regex_replace(img_path, "Images/", "Masks/mask_")
  mask = tf.io.read_file(mask_path)
  mask = tfio.experimental.image.decode_tiff(mask)
  mask = mask[:, :, 0:1]  # 1 channel for mask

  img = tf.image.resize(img, (128, 128))
  mask = tf.image.resize(mask, (128, 128))

  mask = tf.cast(mask, tf.float32) / 255.0  # normalizing mask

  img = tf.image.convert_image_dtype(img, tf.float32)
  mask = tf.image.convert_image_dtype(mask, tf.uint8) # since the tf.py_function is expecting Tout to be tf.float32 and tf.uint8 

  return img, mask

After this I'm going to start training my model:

train_dataset = train_dataset.batch(10)
train_dataset = train_dataset.prefetch(buffer_size=AUTOTUNE)

val_dataset = val_dataset.batch(10)
val_dataset = val_dataset.prefetch(buffer_size=AUTOTUNE)

test_dataset = test_dataset.batch(10)
test_dataset = test_dataset.prefetch(buffer_size=AUTOTUNE)

To confirm that my dataset isn't empty I'm plotting an image and mask from it after this. That works fine.

    for images_batch, masks_batch in train_dataset.take(1):
      fig, arr = plt.subplots(1, 2, figsize=(14, 10))
      print(images_batch.shape)
      print(masks_batch.shape)
      arr[0].imshow(images_batch[0], interpolation='nearest')
      arr[1].imshow(masks_batch[0], cmap='gray')
      plt.show()

It also prints the size of images and masks batches as: (10, 128, 128, 3) (10, 128, 128, 1)

Printing the datasets:

print(train_dataset)
print(val_dataset)
print(test_dataset)

gives the following output:

<PrefetchDataset shapes: ((None, 128, 128, None), (None, 128, 128, None)), types: (tf.float32, tf.uint8)>

<PrefetchDataset shapes: ((None, 128, 128, None), (None, 128, 128, None)), types: (tf.float32, tf.uint8)>

<PrefetchDataset shapes: ((None, 128, 128, None), (None, 128, 128, None)), types: (tf.float32, tf.uint8)>

Now fitting the model:

model.fit(train_dataset, epochs=20,
          steps_per_epoch=STEPS_PER_EPOCH,
          validation_steps=VALIDATION_STEPS,
          batch_size=10,
          validation_data=val_dataset,
          callbacks=callbacks)

gives the following error:

line 39, in train_using_tf_data
    history = model.fit(train_dataset, epochs=attributes.EPOCHS,
  File "/home/user/TfProjects/venv/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 1204, in fit
    raise ValueError('Expect x to be a non-empty array or dataset.')
ValueError: Expect x to be a non-empty array or dataset.

I thought that I am returning a tf.data.Dataset after the preprocess function. I can also plot data from the returned dataset. Where could possibly the dataset go empty here? My guess is that tf.py_fucntion is somehow messing it up. But I don't understand where and how to fix it. Does anyone have any idea?

theNerdyGirl
  • 11
  • 3

1 Answers1

1

Alright, I have found the solution to my problem and I'm posting the answer here for anyone who is stuck with the same! I had forgotten to update the STEPS_PER_EPOCH variable that I pass into the model.fit function. It was taking the value as zero. I updated it with STEPS_PER_EPOCH = train_size // BATCH_SIZE. And then it worked like a charm! The problem is not with the wrapper tf.py_function. It was a carelessness from my side. But it took so long to find out where it was going wrong! I recommend everyone who has a similar error to re-check every attributes that you pass into your model.fit function.

kağan hazal koçdemir
  • 715
  • 4
  • 18
theNerdyGirl
  • 11
  • 3