Logstash azure plugin

Question

I am very new to logstash, and am kind of stuck how to do this part.

I copied logs from azure blob container, and was able to parse via logstash and put the records in elastic search on my local machine. Now I want to change the input so that instead of file, it gets the logs directly from the azure blob storage. I looked for plugins, but did not find anything for azure.

I am hoping I can do something like this in the input. Any ideas how to get this?

input {
  azureblob {
    storage_name => "abcd"
    storage_key => "key"
    container => "logLocation"
  }
}

score 1 · Accepted Answer · answered Sep 21 '15 at 16:46

After playing around, and reading a bunch of stuff, I got it to work on my local windows machine.

Install ruby. Install jruby. command line -> gem install bundle.

Above is needed to build gems

Then I got these files. https://github.com/juliusl/logstash-input-azurewadtable/tree/0.9.2

Replaced lib/logstash/inputs/azurewadtable.rb with https://github.com/mspnp/semantic-logging/blob/elk/ELK/logstash-extension/inputs/azureblob.rb

Replaced all places with azurewadtable with azureblob in all files.

Via command line gem install "localtion_Of_gemSpec" This will give you the gem file.

Logstash\bin\plugin install "location_of_gem_file" Logstash\bin\plugin install --no-verify

Verify a couple of things: Logstash\gemfile has this entry and Logstash\vendor\local_gems has the files

Setup config:

input {
  azureblob {
    storage_account_name => "STORAGE ACCOUNT NAME"
    storage_access_key => "STORAGE ACCESS KEY"
    container => "CONTAINER NAME"
  }

}

Logstash/bin/logstash -f "location_of_conf_File"

And it works :)

Hi @Dhrumil Can i send data content from azure blob storage to ES with this logstash input config eventhough the data present in blob isn't in JSON format? — Private, Jul 18 '17 at 18:27

score 0 · Answer 2 · answered May 28 '20 at 11:53

0

I faced issue while using lostash-input-azureblob plugin to fetch azure logs and send them to the desired location. There is still and open issue on github for the same : https://github.com/Azure/azure-diagnostics-tools/issues/170
I have found a workaround, in the latest version 0.9.13 there is some files missing which can be downloaded from version 0.9.12, Please follow the below link for detail instructions :
https://github.com/Azure/azure-diagnostics-tools/issues/170#issuecomment-635286539

answered May 28 '20 at 11:53

Deepak Yadav

632
7
14

I switched to lostash-input-azure_blob_storage years ago, after I noticed that ingestion would stop for a period, so it wouldn't process all my files. azureblob doesn't seem to be supported anymore – sleepyhead Dec 24 '21 at 08:03

Logstash azure plugin

2 Answers2