2

Some utterances I deal with are made from several sentences, and I have noticed that existence / non existence of periods are often changing the intent detected.

For the sake of example - "Hi. I'd like to order." may result with different intent than "Hi I'd like to order".

How does LUIS tokenize utterances with periods? Are periods treated as other tokens, or there is some processing done to cleanup periods in some cases?

Amotz Hoshen
  • 65
  • 3

1 Answers1

0

LUIS uses CNTK to build LSTMs and the utterances are going through various NLP pipeline steps like POS tagging and sentence tockenizing.

When there's periods, pipeline inside LUIS may get it as separate sentences. Better to train with the both utterances.

Haritha Thilakarathne
  • 880
  • 8
  • 19