0

In kibana, I have below type of log entries in the "message" field. I want to search for all the unique/distinct URLs from the log.

My URL format is like web.cluster.test.com/api/*

123.456.78.90 - a898fur6x5b10hd5 phoenix-xml login_unified  "POST web.cluster.test.com/api/login_unified.php HTTP/1.1"  "python-requests/2.21.0"    -   {"output":"xml","domain":"test","reusesess":"0","userid":"abc@test.net","pass":"<REDACTED>"}    {"domain":"test","userid":"abc@test.net","pass":"<REDACTED>","mdpass":null,"reseller":null,"resellerpass":null,"reusesess":false,"output":"xml","apiuser":null,"apipass":"<REDACTED>","logout":null}

123.456.78.90 - a65oaby6x5b10hd5 phoenix-xml user_query "POST web.cluster.test.com/api/user_query.php HTTP/1.1" "python-requests/2.21.0"    -   {"output":"xml","domain":"test","reusesess":"0","userid":"abc@test.net","pass":"<REDACTED>"}    {"domain":"test","userid":"abc@test.net","pass":"<REDACTED>","mdpass":null,"reseller":null,"resellerpass":null,"reusesess":false,"output":"xml","apiuser":null,"apipass":"<REDACTED>","logout":null}

If I search with "web.cluster.test.com/api/*" in the search bar, it gives me that patterns URLs in the search result, but they are not unique.

How can we get all the unique URL's from it? Appriciate your help and thank you.

Nilesh
  • 1,047
  • 1
  • 12
  • 20
  • Are you analyzing the logs on the way to elasticsearch in logstash or a ingest pipeline? Do you then have field, where just the url is stored in? If so, you could add a keyword (sub) mapping for this field. Assuming you have that field, properly mapped as keyword, then you need just to issue a query and aggregate (count) the documents on this field. This aggregation will yield also urls with only on document and this is what you are looking for. In Kibana you can also use a table visualization for unique urls... – ibexit Jun 13 '19 at 15:37
  • Only url is not stored in any field, it's part of "message" fields content – Nilesh Jun 14 '19 at 03:08
  • So consider extracting the url and the described path above will work. – ibexit Jun 14 '19 at 06:47

0 Answers0