To further my understanding of RNN and LSTM I am trying to implement a simple LSTM to estimate the frequency and phase of a sinewave. This is proving to be surprisingly difficult to converge. The MSE is quite high (in the thousands) The only thing that seems to work a little bit is if I generate sinewaves all having the same phase (starting at the same time) AND the training samples are passed as a vector and not as samples one-at-a-time as in a RNN. Meanwhile, here is the code that does not converge. In this code I have removed the different phase for every frequency Any thoughts on what is going wrong here
I have looked at this Keras : How should I prepare input data for RNN? and tried to modify my input for this as well, but no luck.
from keras.models import Sequential
from keras.layers.core import Activation, Dropout ,Dense
from keras.layers.recurrent import GRU, LSTM
import numpy as np
from sklearn.cross_validation import train_test_split
np.random.seed(0) # For reproducability
TrainingNums = 12000 #Number of Trials
numSampleInEach = 200 #Length of each sinewave
numPhaseExamples = 1 #for each freq, so many different phases
X = np.zeros((TrainingNums,numSampleInEach))
Y = np.zeros((TrainingNums,2))
#create sinewaves here
for iii in range(0, TrainingNums//numPhaseExamples):
freq = np.round(np.random.randn()*100)
for kkk in range(0,numPhaseExamples):
#set timeOffset below to 0, if you want the same phase every run
timeOffset = 0# 0 for now else np.random.randint(0,90)
X[iii*numPhaseExamples+kkk,:] = np.sin(2*3.142*freq*np.linspace(0+timeOffset,numSampleInEach-1+timeOffset,numSampleInEach)/10000)
Y[iii*numPhaseExamples+kkk,0] = freq
Y[iii*numPhaseExamples+kkk,1] = timeOffset
X = np.reshape(X,(TrainingNums, numSampleInEach,1))
#This below works when there is no phase variation
#X = np.reshape(X,(TrainingNums, numSampleInEach,1))
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33)
#Now create the RNN
model = Sequential()
#batch_input_shape = [batch_size,timeStep,dataDimension]
model.add(LSTM(128,input_shape= (numSampleInEach,1),return_sequences=True))
#For things to work for freq estimation only the following change helps
#model.add(LSTM(128,input_shape=(1,numSampleInEach),return_sequences=True))
model.add(Dropout(0.2))
model.add(Activation("relu"))
#second layer of RNN
model.add(LSTM(128,return_sequences=False))
model.add(Dropout(0.2))
model.add(Activation("relu"))
model.add(Dense(2,activation="linear"))
model.compile(loss="mean_squared_error", optimizer="Nadam")
print model.summary()
print "Model compiled."
model.fit(X_train, y_train, batch_size=16, nb_epoch=150,
validation_split=0.1)
result = model.evaluate(X_test, y_test, verbose=0)
print 'mse: ', result
So questions are:
- Is it right to expect an RNN to estimate frequency and phase?
- I have tried a couple of architectures (multi layer LSTM, single layer with more nodes etc). I have tried different architectures as well.
