I am trying to create a bi-lstm crf to assign labels in a sequence modeling scenario. I have a host of sentences, wherein the words are tagged in a BIO (Begin, Inside, outside) scheme and I would like the deep neural net to learn from the sequence of words and the assigned tags, with forward and backward propagation and use conditional random fields.
I am following the python implementation shown in https://www.depends-on-the-definition.com/sequence-tagging-lstm-crf/ and trying to implement it in R.
I got to the point where I have sequenced and padded the sequences and created X_Train, Y_Train to feed into the Deep Learning network.
the Python implementation of the model is below (from the above link)
input = Input(shape=(max_len,))
model = Embedding(input_dim=n_words + 1, output_dim=20,
input_length=max_len, mask_zero=True)(input) # 20-dim embedding
model = Bidirectional(LSTM(units=50, return_sequences=True,
recurrent_dropout=0.1))(model) # variational biLSTM
model = TimeDistributed(Dense(50, activation="relu"))(model) # a dense layer as suggested by neuralNer
crf = CRF(n_tags) # CRF layer
out = crf(model) # output
model = Model(input, out)
model.compile(optimizer="rmsprop", loss=crf.loss_function, metrics=[crf.accuracy])
history = model.fit(X_tr, np.array(y_tr), batch_size=32, epochs=5,
validation_split=0.1, verbose=1)
HOW IS THIS TO BE IMPLEMENTED IN R ? What I have so far is below, but does not seem correct.
input <- layer_input(shape = c(max_len,)) ## --- Does not seem correct --
output <- input %>%
layer_embedding(input_dim = n_words + 1, output_dim = 20, input_length=max_len, mask_zero = TRUE) %>%
bidirectional(layer_lstm(units=50, return_sequences = TRUE,recurrent_dropout = 0.1)) %>%
time_distributed(layer_dense(units = 50, activation = "relu"))
## -- Dont know how to create a CRF layer
model <- keras_model(input, output)
model %>% compile(
optimizer = "rmsprop",
loss = "categorical_crossentropy"
)
summary(model)
model %>% fit(S1S2S3XTrain, S1S2S3YTrain, epochs = 5, batch_size = 32)
model %>% evaluate(x_train, y_train)
Please help.
