I have been trying to model nonlinear dynamic systems with LSTM networks using Keras. I have had success by simply using the Keras LSTM networks, where I define my input/output relationship something like the following pseudo-code:
x[t] = NN(y[t-200:t],x[t-200-1:t-1])
Where y would be my forcing function and x is the variable I'm after. So I use the past 200 points to estimate the next point. I do this recursively by adding the newly predicted point to my "past outputs" vector.
Now I would like to add some information about the PDE that I'm solving to the loss function, so I need to compute derivatives with respect to time. I have read this answer and the related answers to get started but I can't seem to get that workflow to work with LSTMs. First of all, time is not an explicit variable in my workflow, so I would need to add it as an input to accommodate the workflow in that answer.
So I could add the time vector to the list of inputs, and then try to compute derivatives of the output with respect to the input:
_df1 = grad(model.output,model.input)
df1 = tf.Print( _df1, [ _df1 ], message = "df1" )
For reference, my input dimension is (?,200,3) and my output dimension is simply (?,1). The code above works and I get a (?,200,3) tensor. But when I try to compute the second derivative like so:
_df2 = grad(df1,model.input)
df2 = tf.Print( _df2, [ _df2 ], message = "df2" )
Then I get the error: TypeError: Second-order gradient for while loops not supported.
Since I only need the derivatives at the current timestep (t), I have tried slicing the tensor, but that doesn't work either.
_df2 = grad(df1[:,-1,-1],model.input)
df2 = tf.Print( _df2, [ _df2 ], message = "df2" )
Even if I could do something like that, I am not too comfortable adding the time vector as an explicit input. I have considered computing the derivative numerically with diff() (given a constant dt), but I am not sure how to do that here when dealing with tensors.
So I'd appreciate any suggestions or ideas to help me solve this problem. Ultimately, I'd like to add the homogeneous portion of the PDE to the loss function. At this point, my equation only has derivatives with respect to time.
Thanks.
