5

I'm building a chatbot and I'm new to NLP.

(api.ai & AlchemyAPI are too expensive for my use case. And wit.ai seems to be buggy and constantly changing at the moment.)

For the NLP experts, how easily can I replicate their services locally?

My vision so far (with node, but open to Python):

  • entity extraction via StanfordNER
  • intent via NodeNatural's LogisticRegressionClassifier
  • training UI with text and validate/invalidate buttons (any prebuilt tools for this?)

Are entities and intents all I'll need for a chatbot? How good will NodeNatural/StanfordNER be compared to NLP-as-a-service? What headaches am I not seeing?

Kyle Cureau
  • 19,028
  • 23
  • 75
  • 104
  • 1
    You almost answered your own question: if those API you mention are too expensive, chances are they provide functionality that is not easy to implement (at least optimally). However, you probably do not need all of the functionality provided by them. Finally, why not use the open source Python NLP library, namely http://www.nltk.org/ ?? –  Apr 20 '16 at 14:03
  • @William, yea I like nltk. But being new to chatbots and NLP in general, I'd like to know what I'm in for. My optimistic sentiment is "oh, ok I'll just use some nltk modules and be done by dinner!" I must be missing something, and I want to know what specifically. – Kyle Cureau Apr 20 '16 at 14:20

3 Answers3

3

You seems to have done your home work. Like you said,following things will help you along the way,

  • any classification algorithm for identifying the intent such as sklearn's LinearSVC or LogisticRegression Classifier
  • Any good NER tool such as StanfordNER or CRFsuite. CRFsuite has easy to use pthon wrapping called pycrfsuite.
  • A sentiment analysis tool for more human like conversations. its better if you go with python because python has lot of free libraries for the same.

Only benefit of services such as wit.ai or api.ai is their heavily trained ready to use intents and models.You too will be able to achive similar accuracy if you are able to provide decent amount of training to your bot.

It's better if you build on some existing opensource libraries rather than building everything from scratch. Please check my opensource project on github for wit.ai/api.ai similar interface. Happy coding!

Alfred Francis
  • 451
  • 1
  • 6
  • 20
0

Have a look at Luis.ai (from Microsoft). It will help you build a Natural Langauge model that identifies intents. You can map intents to actions. It allows you to include pre-configured NLP models so that your bot can decipher text and return entities (names, time, geography, etc) and Luis supports many languages. You can of course add your own entities.

And then you train it with utterances. And when users talk to your bot, those utterances are stored online and Luis.ai which will suggest that you add some to further refine your model via its suggest feature (this allows incremental improvements without coding). Now for the coding stuff...

Visit Luis.ai and when you build your model (free) you can test it via the testing console here: https://dev.projectoxford.ai/docs/services and then you could use their Bot Framework; however I am building my own in JQuery/Bootstrap (have a look at this to see what I am building here http://onlinebotbuilder.com) and handling the json from the server via a post to the Cognitive Services API (which you would do in Python). Currently they allow 10k utterances per month; which for testing is plenty. Tip: When creating your model start with 2 intents and train your model to identify those. Keep it simple, and then expand scope as you go. Hope this helps.

Rob
  • 1,226
  • 3
  • 23
  • 41
0

Two things to think about are: How are you planning on handling the generation side of things? Entity extraction and classification are going to be useful for the Natural language understanding (NLU) side of things, but generation can be tricky in itself.

Another thing to think about is that the training and development of the pipeline of these models is often a separate problem form the deployment. The fact that you want to use node suggests that you already know about deploying software, I think. But remember that deploying large machine learning models in a pipeline can be complicated, and I suspect that these API's may offer neatly packaged pipelines for you.

adamits
  • 73
  • 1
  • 7