7

I installed elasticsearch and kibana, and I'm following the tutorial. https://www.elastic.co/guide/en/elasticsearch/reference/current/_index_and_query_a_document.html And I'm perfectly inserting and reading data, e.g.:

PUT /customer/external/1?pretty
{
  "name": "John Doe"
}

So, that makes me wonder, what do I need logstash or filebeats for? My plan is to log each web request on a website to elasticsearch for analytics.

Do I need to install logstash? I don't understand what would I need it for. (I don't plan to store it on a file)I will read the request info(e.g. ip address, time, user_id, etc) from a PHP script and simply send it through a HTTP REST REQUEST...as the example above to the elasticsearch server which will save the data anyway. So, I don't see any reason to store the data on the webserver(that is data duplicity), and If I wanted to, why would I need logstash anyway...I can just read a .log file and send it to elasticsearch....like this example: https://www.elastic.co/guide/en/elasticsearch/reference/current/_exploring_your_data.html

Gabriel Rodriguez
  • 1,163
  • 10
  • 23
  • 1
    From where will you read each web request (are they stored in a file?) and how will you send them to Elasticsearch? – Val Feb 25 '17 at 05:52
  • (I don't plan to store it on a file)I will read the request info(e.g. ip address, time, user_id, etc) from a PHP script and simply send it through a HTTP REST REQUEST as the example above to the elasticsearch server which will save the data anyway. So, I don't see any reason to store the data on the webserver(that is data duplicity), and If I wanted to, why would I need logstash anyway...I can just read a .log file and send it to elasticsearch....like this example: https://www.elastic.co/guide/en/elasticsearch/reference/current/_exploring_your_data.html – Gabriel Rodriguez Feb 25 '17 at 22:31
  • In the end it all depends on whether you want to index 100% of your data or you're ok to lose some of it. The question you need to ask yourself is what happens when you have network issues between your PHP script and ES and/or what will happen if ES is down for some reason (maintenance, etc)? – Val Feb 26 '17 at 05:35
  • 1
    To sum up, you don't need Logstash if: 1) you can guarantee that everything will always be 100% online and working (which you reasonably can't) or 2) you accept to lose some data from time to time. – Val Feb 27 '17 at 05:22
  • @Val what you think about this: http://stackoverflow.com/questions/43080745/convert-any-elasticsearch-response-to-simple-field-value-format – Gabriel Rodriguez Mar 29 '17 at 14:10
  • Opinion: Logstash is perhaps the most useless thing I have ever learned. I mean, seriously, do you need such heavyloaded process (at least 1 GB RAM for the Java VM) for such a trivial task. An average programmer like me can write a few lines of code to grab the logging data and send it wherver I want, instead of wasting hours in learn->trial-and-error->apply. Such a fuzz! And at the univeryity we created a Lucene like (VSM, TF, IDF and inverted index) search engine from scratch and it was just a few hundred lines of code. Such a fuzz! Such a fuzz!! – sajid Jan 15 '23 at 18:09

1 Answers1

6

No, you do not have to install Logstash, if you plan to collect, normalize and write your application data yourself. As you correctly assumed, Logstash would be a replacement for your PHP script.

Nevertheless, you might still consider to have a look at Logstash. Since it is developed and maintained by same company taking care of Elastic Search, you could benefit from upcoming changes and optimizations.

As you can read from the introduction, Logstash is a tool to read data from multiple sources, normalize it and write the result to multiple destinations. For more details on which sources, filters and oputputs Logstash offers, you should also take a look at the pipeline documentation.

Christian Häckh
  • 512
  • 6
  • 11