0

I have successfully trained a Keras/TensorFlow model consisting of layers SimpleRNN→Conv1D→GRU→Dense. The model is meant to run on an Apple Watch for real time inference, which means I want to feed it with a new feature vector and predict a new output for each time step. My problem is that I don't know how to feed data into it such that the convolutional layer receives the latest k outputs from the RNN layer.

I can see three options:

  1. Feed it with one feature vector at a time, i.e. (1,1,6). In this case I assume that the convolutional layer will receive only one time step and hence zero pad for all the previous samples.
  2. Feed it with the last k feature vectors for each time step, i.e. (1,9,6), where k = 9 is the CNN kernel length. In this case I assume that the state flow in the recurrent layers will not work.
  3. Feed it with the last k feature vectors every k:th time step, again where k = 9 is the CNN kernel length. I assume this would work, but introduces unnecessary latency that I wish to avoid.

What I want is a model that I can feed with a new single feature vector for each time step, and it will automatically feed the last k outputs of the SimpleRNN layer into the following Conv1D layer. Is this possible with my current model? If not, can I work with the layer arguments, or can I introduce some kind of FIFO buffer layer between the SimpleRNN and Conv1D layer?

Here is my current model:

feature_vector_size = 6
model = tf.keras.models.Sequential([
    Input(shape=(None, feature_vector_size)),
    SimpleRNN(16, return_sequences=True, name="rnn"),
    Conv1D(16, 9, padding="causal", activation="relu"),
    GRU(12, return_sequences=True, name="gru"),
    Dropout(0.2),
    Dense(1, activation=tf.nn.sigmoid, name="dense")
])

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
rnn (SimpleRNN)              (None, None, 16)          368       
_________________________________________________________________
conv1d (Conv1D)              (None, None, 16)          2320      
_________________________________________________________________
gru (GRU)                    (None, None, 12)          1080      
_________________________________________________________________
dropout (Dropout)            (None, None, 12)          0         
_________________________________________________________________
dense (Dense)                (None, None, 1)           13        
=================================================================

Edit:

After having researched the problem a bit, I have realized:

  • The Conv1D layer will zero pad in all three cases that I described, so option 3 won't work either. Setting padding="valid" solves this particular problem.
  • The SimpleRNN and GRU layers must have stateful=True. I found this description of how to make a model stateful after it has been trained stateless: How to implement a forward pass in a Keras RNN in real-time?
  • Keras sequence models seem to be made for complete, finite sequences only. The infinite streaming use case with one time step at a time isn't really supported.

However, the original question remains open: How can I build and/or feed new feature vectors into the model such that the convolutional layer receives the latest k outputs from the RNN layer?

jerha202
  • 187
  • 1
  • 9
  • I see a fundamental issue with your model. The `conv1d` layer expects a sequence output by the `rnn` below. But at inference time it doesn't make sense, as you don't have the full sequence available (no matter how you try to recreate it in your several options - it's always 1 time step available to you). Why do you need this `conv1d` layer here? – thushv89 Jul 12 '22 at 21:21
  • The short answer is that this model performs and generalizes better than other models I've tried. Also, many recent papers propose the combination of CNN and RNN layers. I'm not sure I understand why it's fundamentally impossible, if that's what you mean - for example, a simple FIFO queue that remembers the last 9 outputs of the SimpleRNN layer could provide the 9 time steps that the Conv1D layer needs. Or have I fundamentally misunderstood something? – jerha202 Jul 13 '22 at 08:18
  • I have researched a bit and edited the question. – jerha202 Jul 14 '22 at 16:22

1 Answers1

0

For anyone else with the same problem: I couldn't solve the SimpleRNN to Conv1D data flow easily, so I ended up replacing the SimpleRNN layer with another Conv1D layer and setting padding="valid" on both Conv1D layers. The resulting model outputs exactly one time step when fed with a sequence of c * k - 1 time steps, where c is the number of Conv1D layers and k is the convolutional kernel length (c = 2 and k = 9 in my case):

feature_vector_size = 6
model = tf.keras.models.Sequential([
    Input(shape=(None, feature_vector_size)),
    Conv1D(16, 9, padding="valid", name="conv1d1"),
    Conv1D(16, 9, padding="valid", name="conv1d2"),
    GRU(12, return_sequences=True, name="gru"),
    Dropout(0.2),
    Dense(1, activation=tf.nn.sigmoid, name="dense")
])

After training, I make the GRU layer stateful according to How to implement a forward pass in a Keras RNN in real-time?. For real-time inference I keep a FIFO queue of the 17 latest feature vectors and feed all these 17 vectors into the model as an input sequence for each new time step.

I don't know if this is the best possible solution, but at least it works.

jerha202
  • 187
  • 1
  • 9