Image Classification, 3D black and white MRI data. Data Dimensionality Issues

Question

I'm trying to build a CNN that can classify 3D MRI files as being one of two classes, basically with disease or without. I've done a lot of googling and it seems like the consensus is that a 2D CNN would be best because the data is black and white and therefore has reduced dimensionality.

I'm working on Google Colab before I transfer everything to a remote workstation to run it all, and I'm using a subset of the data to get it working before I run it all but i have 20 B&W files of (189, 233, 197). Below is the code I have so far:

import numpy as np
import glob
import os
import tensorflow as tf
import pandas as pd
import glob

pip install SimpleITK
import SimpleITK as sitk

from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split
from keras.preprocessing.image import ImageDataGenerator


from keras.utils import plot_model
from keras.utils import to_categorical
from keras.utils import np_utils

from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout

from google.colab import drive
drive.mount('/content/gdrive')

datapath = ('/content/gdrive/My Drive/DirectoryTest/All Data/')
patients = os.listdir(datapath)
labels_df = pd.read_csv('/content/Data_Index.csv', index_col = 0 )

labelset = []

for i in patients:
  label = labels_df.loc[i, 'Group']
  if label is 'AD':
    np.char.replace(label, ['AD'], [0])
  if label is 'CN':
    np.char.replace(label, ['CN'], [1])
  labelset.append(label)

label_encoder = LabelEncoder()
labelset = label_encoder.fit_transform(labelset)

labelset = np_utils.to_categorical(labelset, num_classes= 2)


FullDataSet = []

for i in patients:
  a = sitk.ReadImage(datapath + i)
  b = sitk.GetArrayFromImage(a)
  #c = np.reshape(b, (197,233,189, 1))
  FullDataSet.append(b)

training_data, testing_data, training_labels, testing_labels = train_test_split(FullDataSet, labelset, train_size=0.70,test_size=0.30)

dataset_train = tf.data.Dataset.from_tensor_slices((training_data, training_labels))
dataset_test = tf.data.Dataset.from_tensor_slices((testing_data, testing_labels))

# 2D CNN

CNN_model = tf.keras.Sequential(
  [
      #tf.keras.layers.Input(shape=(189, 233, 197, 1), batch_size=2),
      #tf.keras.layers.Reshape((197, 233, 189, 1)),   
                              
      tf.keras.layers.Conv2D(kernel_size=(7, 7), data_format='channels_last', filters=64, activation='relu',
                             padding='same', strides=( 3, 3), input_shape=(189, 233, 197)),
      #tf.keras.layers.BatchNormalization(center=True, scale=False),
      tf.keras.layers.MaxPool2D(pool_size=(3, 3), padding='same'),
      tf.keras.layers.Dropout(0.20),
      
      tf.keras.layers.Conv2D(kernel_size=( 7, 7), filters=128, activation='relu', padding='same', strides=( 3, 3)),
      #tf.keras.layers.BatchNormalization(center=True, scale=False),
      tf.keras.layers.MaxPool2D(pool_size=(3, 3), padding='same'),
      tf.keras.layers.Dropout(0.20),      

      tf.keras.layers.Conv2D(kernel_size=( 7, 7), filters=256, activation='relu', padding='same', strides=( 3, 3)),
      #tf.keras.layers.BatchNormalization(center=True, scale=False),
      tf.keras.layers.MaxPool2D(pool_size=(3, 3), padding = 'same'),
      tf.keras.layers.Dropout(0.20), 

      # last activation could be either sigmoid or softmax, need to look into this more. Sig for binary output, Soft for multi output 
      tf.keras.layers.Flatten(),
      tf.keras.layers.Dense(256, activation='relu'),   
      tf.keras.layers.Dense(64, activation='relu'),
      tf.keras.layers.Dropout(0.20),
      tf.keras.layers.Dense(2, activation='softmax')

  ])
# Compile the model
CNN_model.compile(optimizer=tf.keras.optimizers.Adam(lr=0.00001), loss='categorical_crossentropy', metrics=['accuracy'])

# print model layers
CNN_model.summary()

CNN_history = CNN_model.fit(dataset_train, epochs=10, validation_data=dataset_test)

This produces the error

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-89-a8b210ec2e72> in <module>()
      1 #running of the model
      2 #CNN_history = CNN_model.fit(dataset_train, epochs=100, validation_data =dataset_test, validation_steps=1)
----> 3 CNN_history = CNN_model.fit(dataset_train, epochs=10, validation_data=dataset_test)
      4 
      5 

10 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/func_graph.py in wrapper(*args, **kwargs)
    971           except Exception as e:  # pylint:disable=broad-except
    972             if hasattr(e, "ag_error_metadata"):
--> 973               raise e.ag_error_metadata.to_exception(e)
    974             else:
    975               raise

ValueError: in user code:

    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:806 train_function  *
        return step_function(self, iterator)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:796 step_function  **
        outputs = model.distribute_strategy.run(run_step, args=(data,))
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:1211 run
        return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:2585 call_for_each_replica
        return self._call_for_each_replica(fn, args, kwargs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:2945 _call_for_each_replica
        return fn(*args, **kwargs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:789 run_step  **
        outputs = model.train_step(data)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:747 train_step
        y_pred = self(x, training=True)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer.py:976 __call__
        self.name)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/input_spec.py:196 assert_input_compatibility
        str(x.shape.as_list()))

    ValueError: Input 0 of layer sequential_24 is incompatible with the layer: : expected min_ndim=4, found ndim=3. Full shape received: [189, 233, 197]

So I'm missing one dimension, I think because it isn't an rgb image so the channel data is missing. So I go back to the following line of code and add a dimension using np.reshape:

  a = sitk.ReadImage(datapath + i)
  b = sitk.GetArrayFromImage(a)
  c = np.reshape(b, (197,233,189, 1))
  FullDataSet.append(b)

When I run this through everything I get the following error:

Epoch 1/10
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-13-a8b210ec2e72> in <module>()
      1 #running of the model
      2 #CNN_history = CNN_model.fit(dataset_train, epochs=100, validation_data =dataset_test, validation_steps=1)
----> 3 CNN_history = CNN_model.fit(dataset_train, epochs=10, validation_data=dataset_test)
      4 
      5 

10 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/func_graph.py in wrapper(*args, **kwargs)
    971           except Exception as e:  # pylint:disable=broad-except
    972             if hasattr(e, "ag_error_metadata"):
--> 973               raise e.ag_error_metadata.to_exception(e)
    974             else:
    975               raise

ValueError: in user code:

    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:806 train_function  *
        return step_function(self, iterator)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:796 step_function  **
        outputs = model.distribute_strategy.run(run_step, args=(data,))
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:1211 run
        return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:2585 call_for_each_replica
        return self._call_for_each_replica(fn, args, kwargs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:2945 _call_for_each_replica
        return fn(*args, **kwargs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:789 run_step  **
        outputs = model.train_step(data)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:747 train_step
        y_pred = self(x, training=True)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer.py:976 __call__
        self.name)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/input_spec.py:216 assert_input_compatibility
        ' but received input with shape ' + str(shape))

    ValueError: Input 0 of layer sequential_4 is incompatible with the layer: expected axis -1 of input shape to have value 197 but received input with shape [197, 233, 189, 1]

I've been at this for a couple days and any help would be really appreciated, Thanks!

jkr · Answer 1 · 2020-08-12T21:39:10.040

Welcome to Stack Overflow. For a 2D network, your input should have the shape (batch_size, height, width, channels). You are correct in adding an extra dimension at the end of the array. That represents the grayscale color channel.

You get the error

ValueError: Input 0 of layer sequential_4 is incompatible with the layer: expected axis -1 of input shape to have value 197 but received input with shape [197, 233, 189, 1]

because you defined your input_shape incorrectly in the first layer of your network. It should be input_shape=(height, width, 1), and you need to choose the correct height and width. Notice that we omit the batch dimension here.

The features need to be reshaped into an array of shape (n_slices, height, width, 1). In other words, you need to stack 2D slices of volumes. The dimensions of your volumes probably have axial, coronal, and sagittal planes, so you need to decide which views you want to train your model on. Also, you can stack all of the views, and in effect triple the size of your dataset. Though to stack all of the slices, the three planes would need to be the same shape. In your case, the planes are not the same shape.

Here is one way to stack the slices of your data.

import numpy as np

volumes = []
for patient in patients:
    image = sitk.ReadImage(datapath + patient)
    volume = sitk.GetArrayFromImage(image)
    volumes.append(volume)

# Choose the plane you want by changing the `axis` parameter.
axis = 1
features = np.concatenate(volumes, axis=axis)

# Move the concatenation axis to the first position. We will take
# batches along this dimension, so each batch contains several slices.
features = np.moveaxis(slices, source=axis, destination=0)

# Add the grayscale channel dimension.
features = features[..., np.newaxis]

Don't forget to shuffle your inputs along the batch dimension!

Thanks for the help @jakub! So n_slices = batchsize then? I did notice that my loss function was incorrect, so changed it to Binary Crossentropy, but now I'm getting another error: ``` ValueError: slice index 0 of dimension 0 out of bounds. for '{{node strided_slice}} = StridedSlice[Index=DT_INT32, T=DT_INT32, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1](Shape, strided_slice/stack, strided_slice/stack_1, strided_slice/stack_2)' with input shapes: [0], [1], [1], [1] and with computed input tensors: input[1] = <0>, input[2] = <1>, input[3] = <1>. ``` — John Newberry, Aug 12 '20 at 21:50
I just realized you edited your previous comment, going through that now — John Newberry, Aug 12 '20 at 22:03

Image Classification, 3D black and white MRI data. Data Dimensionality Issues

1 Answers1