4

I have CT images from patients and applied a CNN to those images to predict diseases. I would like to combine my clinical data with my image data in a CNN approach, is that possible? My clinical data has information like age, sex, dates, smoker, all of them are numbers, like 1 for smoker and 0 not.

Lucas Ramos
  • 459
  • 4
  • 12

2 Answers2

11

Have a look at, for example, this paper where they combine features from a CNN with text data. In that paper, the CNN is already pre-trained (i.e., the CNN is essentially a featurizer), but you could clearly learn all in one go. The idea would always be to

  • Run the image in your input through the convolution/subsampling layers
  • Just before your final fully connected (decision) layer, concatenate the other features you have available
  • Feed all (pre-processed image and other features) into the decision layer.

So the answer is "yes, certainly", the details depend on which framework you are using.

Anton Schwaighofer
  • 3,119
  • 11
  • 24
2

As far as I know CNN is extremely suitable for image data, but not for other data.

A solution to your problem would be to 'color' your images with the clinical data. (In image recognition CNNs, usually an input image is split into 3 color layers: red, grey and blue. See: http://cs231n.github.io/convolutional-networks/)

Let's say your input data is a 32x32 pixel 8-bit greyscale image (so 1 color layer). I propose to add each clinical data variable as a 'color' layer. All input values in the same color layer should be the same.

Whether each layer should be the same size as the image, or if you can get away with a single pixel, I'm not sure, but at least you can treat the clinical data as an 'image' alongside the CT images.

keepitwiel
  • 122
  • 7
  • One can certainly train images of any type. What you are referring is the fact that most of the pre-trained networks are trained using RGB channels. Even there folks have copied the gray channel 3 times to plug into the RGB trained models (with some limited success). However, once you have featurized the image into a vector one can train a network on top of it with other features. – Sayan Pathak Jan 26 '17 at 00:58