1

My system logs requests.

If and when a response arrives, I want to have it logged with the request object.

Meaning, when I query ElasticSearch for activity, I'll get a row for each request, either containing a response or not. This is similar to an SQL LEFT OUTER JOIN.

I am open to solutions everywhere along the Logstash-ElasticSearch pipeline. However, since this is time-series data, aggregation buckets are not a possible solution since they prevent paging.

Thanks.

Dor Rotman
  • 1,511
  • 1
  • 15
  • 27
  • What "system" is logging? Can you provide a sample of your logs? – Alain Collins Aug 06 '15 at 19:23
  • Proprietary system that logs HTTP activity, let's assume there is a request ID field that can connect the request and response. – Dor Rotman Aug 07 '15 at 08:14
  • If the request document has your ID field, and the response document has the ID field, then you can query where myIdField="myIdValue" to get them both (or only one if the response is not found). Assuming you want more, can you perhaps provide an example? – Alain Collins Aug 08 '15 at 08:53
  • 1
    I want to query for all requests during the last X minutes and get their rows to contain their responses as well. – Dor Rotman Aug 09 '15 at 07:11

1 Answers1

1

Your best bet would be to store the request with the response every time you log them. You can also have scripted queries in Elasticsearch - complicated solution but can work.

user229044
  • 232,980
  • 40
  • 330
  • 338
Asaf Yigal
  • 104
  • 2