RandomCrop causing INVALID_ARGUMENT: required broadcastable shapes

Question

I'm training a neural network with Keras, and trying to use RandomCrop layer. I'm using a dynamic sized dataset (varying resolution), but I've found it's not currently the cause of this issue.

When I run model.fit(), after a short while, I receive the above mentioned error INVALID_ARGUMENT: required broadcastable shapes. I am able to get a summary of my model, so it's not some mismatch there.

My model works fine when I remove this layer, but I need it to reduce the size of my inputs (hence using RandomCrop).

full traceback + tensorflow status

2022-03-23 13:27:28.772937: W tensorflow/core/framework/op_kernel.cc:1733] INVALID_ARGUMENT: required broadcastable shapes

Traceback (most recent call last):
  File "c:\Users\samue\Desktop\rcrop\main.py", line 37, in <module>
    conv_model.fit(
  File "C:\Users\samue\AppData\Roaming\Python\Python310\site-packages\keras\utils\traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "C:\Users\samue\AppData\Roaming\Python\Python310\site-packages\tensorflow\python\eager\execute.py", line 54, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError: Graph execution error:

Detected at node 'mean_squared_error/SquaredDifference' defined at (most recent call last):
    File "C:\Program Files\Python310\lib\threading.py", line 966, in _bootstrap
      self._bootstrap_inner()
    File "C:\Program Files\Python310\lib\threading.py", line 1009, in _bootstrap_inner
      self.run()
    File "C:\Users\samue\AppData\Roaming\Python\Python310\site-packages\keras\engine\training.py", line 1000, in run_step
      outputs = model.train_step(data)
    File "C:\Users\samue\AppData\Roaming\Python\Python310\site-packages\keras\engine\training.py", line 860, in train_step
      loss = self.compute_loss(x, y, y_pred, sample_weight)
    File "C:\Users\samue\AppData\Roaming\Python\Python310\site-packages\keras\engine\training.py", line 918, in compute_loss
      return self.compiled_loss(
    File "C:\Users\samue\AppData\Roaming\Python\Python310\site-packages\keras\engine\compile_utils.py", line 201, in __call__
      loss_value = loss_obj(y_t, y_p, sample_weight=sw)
    File "C:\Users\samue\AppData\Roaming\Python\Python310\site-packages\keras\losses.py", line 141, in __call__
      losses = call_fn(y_true, y_pred)
    File "C:\Users\samue\AppData\Roaming\Python\Python310\site-packages\keras\losses.py", line 245, in call
      return ag_fn(y_true, y_pred, **self._fn_kwargs)
    File "C:\Users\samue\AppData\Roaming\Python\Python310\site-packages\keras\losses.py", line 1329, in mean_squared_error
      return backend.mean(tf.math.squared_difference(y_pred, y_true), axis=-1)
Node: 'mean_squared_error/SquaredDifference'
Detected at node 'mean_squared_error/SquaredDifference' defined at (most recent call last):
    File "C:\Program Files\Python310\lib\threading.py", line 966, in _bootstrap
      self._bootstrap_inner()
    File "C:\Program Files\Python310\lib\threading.py", line 1009, in _bootstrap_inner
      self.run()
    File "C:\Users\samue\AppData\Roaming\Python\Python310\site-packages\keras\engine\training.py", line 1000, in run_step
      outputs = model.train_step(data)
    File "C:\Users\samue\AppData\Roaming\Python\Python310\site-packages\keras\engine\training.py", line 860, in train_step
      loss = self.compute_loss(x, y, y_pred, sample_weight)
    File "C:\Users\samue\AppData\Roaming\Python\Python310\site-packages\keras\engine\training.py", line 918, in compute_loss
      return self.compiled_loss(
    File "C:\Users\samue\AppData\Roaming\Python\Python310\site-packages\keras\engine\compile_utils.py", line 201, in __call__
      loss_value = loss_obj(y_t, y_p, sample_weight=sw)
    File "C:\Users\samue\AppData\Roaming\Python\Python310\site-packages\keras\losses.py", line 141, in __call__
      losses = call_fn(y_true, y_pred)
    File "C:\Users\samue\AppData\Roaming\Python\Python310\site-packages\keras\losses.py", line 245, in call
      return ag_fn(y_true, y_pred, **self._fn_kwargs)
    File "C:\Users\samue\AppData\Roaming\Python\Python310\site-packages\keras\losses.py", line 1329, in mean_squared_error
      return backend.mean(tf.math.squared_difference(y_pred, y_true), axis=-1)
Node: 'mean_squared_error/SquaredDifference'
2 root error(s) found.
  (0) INVALID_ARGUMENT:  required broadcastable shapes
         [[{{node mean_squared_error/SquaredDifference}}]]
         [[div_no_nan/ReadVariableOp/_84]]
  (1) INVALID_ARGUMENT:  required broadcastable shapes
         [[{{node mean_squared_error/SquaredDifference}}]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_1308]

How to reproduce

I've create a minimal reproducible example, with only two images with resolution of [10, 10] both saved as .png with rgb colorspace.

Running main.py loads these images and tries to start training (failing with an error).

When I exclude the RandomCrop layer, it works just fine.

folder structure

/main_folder
--main.py
--/data
   --001.png
   --002.png

main.py

import cv2, os
import keras
import tensorflow as tf
from keras import layers


strategy = tf.distribute.MirroredStrategy()

with strategy.scope():
    input_layer = keras.Input(shape=(None, None, 3))
    cropped = layers.RandomCrop(32, 32)(input_layer)
    out = layers.Conv2D(3, (3, 3), activation='sigmoid', padding='same')(cropped)

    conv_model = keras.Model(input_layer, out)
    conv_model.compile(
        optimizer='adam', 
        loss=tf.keras.losses.MeanSquaredError()
    )

conv_model.summary()

path = "data"
data = [cv2.imread(os.path.join(path, f)) / 255 for f in os.listdir(os.path.join(path))]


def data_generator():
    for i in range(len(data)):
        yield data[i], data[i]


dataset = tf.data.Dataset.from_generator(
    data_generator, 
    output_types=(tf.float32, tf.float32), 
    output_shapes=((None, None, 3), (None, None, 3))
).batch(1)

conv_model.fit(
    dataset,
    epochs=1,
    validation_data=dataset
)

Why is your X identical to your y? are you trying to build an autoencoder? — LeonardoVaz, Mar 24 '22 at 06:15
The output of your model expects a shape of size `(1, 32, 32, 3)` but your label will not have this shape unless you crop it outside of the model. — LeonardoVaz, Mar 24 '22 at 06:18
@LeonardoVaz My `X` is identical to `y` because it's a minimal reproducible example. `RandomCrop` says, that if the size is greater than the image provided, it will resize it to fit. I've tried `5, 5` as parameters as well, didn't work either. The example I've provided indeed is an autoencoder, and I was hoping the `RandomCrop` also crops the image it compares with, but thinking about it again, this is not likely possible - upscaling model, how would the layer know to crop the comparing image at twice the scale or other. — Mahrkeenerh, Mar 24 '22 at 09:54
If you build your model from scratch, you can build a custom loss function and calculate the loss between the output of the RandomCrop layer and the output of the UpSampling2D layer. However, I don't think you can do this using Keras Sequential or Functional APIs. Have a look at this post, it might help you: https://stackoverflow.com/questions/56478454/in-tensorflow-2-0-with-eager-execution-how-to-compute-the-gradients-of-a-networ/56567364#56567364 — LeonardoVaz, Mar 24 '22 at 12:01
I thought of another solution for my problem - since I'm using a custom generator, I can extract the random patch in this step. I didn't post this yet, as I want to first test that it works properly, and I'll add the answer afterwards — Mahrkeenerh, Mar 24 '22 at 13:42

score 0 · Answer 1 · answered Mar 24 '22 at 19:57

So, I wanted to use this for an autoencoder (in the example). That means, I'd have to have the same crop done on both the input and compare image. This doesn't sound like something the RandomCrop could do, but since I'm already using a custom generator, I can implement it right there:

def data_generator():
    for i in range(len(data)):
        # Custom function to determine the patch size
        x, x1, y, y1 = randomly_choose(data[i].shape)
        yield data[i][x: x1, y: y1], data[i][x: x1, y: y1]

This gives me full power over the generation process, allowing me to include image flipping, rotating and other alterations.

RandomCrop causing INVALID_ARGUMENT: required broadcastable shapes

full traceback + tensorflow status

How to reproduce

folder structure

main.py

1 Answers1