How to Implement the Conv1DTranspose in keras?

Question

I Know there is the Conv2DTranspose in keras which can be used in Image. We need to use it in NLP, so the 1D deconvolution is needed.

How do we implement the Conv1DTranspose in keras?

Answer 1

Use keras backend to fit the input tensor to 2D transpose convolution. Do not always use transpose operation for it will consume a lot of time.

import keras.backend as K
from keras.layers import Conv2DTranspose, Lambda


def Conv1DTranspose(input_tensor, filters, kernel_size, strides=2, padding='same'):
    """
        input_tensor: tensor, with the shape (batch_size, time_steps, dims)
        filters: int, output dimension, i.e. the output tensor will have the shape of (batch_size, time_steps, filters)
        kernel_size: int, size of the convolution kernel
        strides: int, convolution step size
        padding: 'same' | 'valid'
    """
    x = Lambda(lambda x: K.expand_dims(x, axis=2))(input_tensor)
    x = Conv2DTranspose(filters=filters, kernel_size=(kernel_size, 1), strides=(strides, 1), padding=padding)(x)
    x = Lambda(lambda x: K.squeeze(x, axis=2))(x)
    return x

Answer 2

In my answer, I suppose you are previously using Conv1D for the convolution.

Conv2DTranspose is new in Keras2, it used to be that what it does was done by a combination of UpSampling2D and a convolution layer. In StackExchange[Data Science] there is a very interesting discussion about what are deconvolutional layers (one answer includes very usefull animated gifs).

Check this discussion about "Why all convolutions (no deconvolutions) in "Building Autoencoders in Keras" interesting. Here is an excerpt: "As Francois has explained multiple times already, a deconvolution layer is only a convolution layer with an upsampling. I don't think there is an official deconvolution layer. The result is the same." (The discussion goes on, it might be that they are approximately, not exactly the same - also, since then, Keras 2 introduced Conv2DTranspose)

The way I understand it, a combination of UpSampling1D and then Convolution1D is what you are looking for, I see no reason to go to 2D.

If however you want to go with Conv2DTranspose, you will need to first Reshape the input from 1D to 2D e.g.

model = Sequential()
model.add(
    Conv1D(
        filters = 3, 
        kernel_size = kernel_size, 
        input_shape=(seq_length, M),#When using this layer as the first layer in a model, provide an input_shape argument 
    )
)
model.add(
    Reshape( ( -1, 1, M) )
)
model.add(
    keras.layers.Conv2DTranspose(
        filters=M,
        kernel_size=(10,1),
        data_format="channels_last"
    )
)

The inconvenient part for using Conv2DTranspose is that you need to specify seq_length and cannot have it as None (arbitrary length series) Unfortunately, the same is true with UpSampling1D for TensorFlow back-end (Theano seems to be once again better here - too bad its not gonna be around)

Answer 3

In TensorFlow v2.2.0 the Conv1DTranspose layer has been implemented in the tf.keras.layers API. Check it out!

Answer 4

You can reshape it to occupy an extra dimension, run the deconvolution, and then reshape it back. In practice, this works. But I've not really thought very hard if it has any theoretical implications (but it seems to theoretically also be fine as you are not going to "convolve" over that dimension

x = Reshape( ( -1, 1 ) )( x )
x = Permute( ( 3, 1, 2 ) )( x )
x = Conv2DTranspose( filters, kernel )( x )
x = Lambda( K.squeeze, arguments={"axis":1} )( x )