0

Additional info:

  • Logstash version: 6.3.1
  • OS: macOS 10.13.4
  • Elastic Search: 6.2.24 on Elastic Cloud
  • Kibana: 6.2.24 on Elastic Cloud

Problem:

Hi,

I'm trying send data from logstash to Elastic Cloud, but i'm receiving the follow error when logstash run:

"Empty configuration for pipeline_id: artist_profile_views"

But if i try run logstash without define xpack configurations in logststah.yml, this work fine and the stdout {} send collected data to output.

Please, see my configurations files:

logststash.yml

cloud.id: "[...]"
cloud.auth: "[user]:[password]"
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: https://elasticCloudUrl
xpack.monitoring.elasticsearch.username: [user]
xpack.monitoring.elasticsearch.password: [password]
xpack.management.enabled: true
xpack.management.pipeline.id: ["artist_profile_views", "searched"]
xpack.management.elasticsearch.username: [user]
xpack.management.elasticsearch.password: [password]
xpack.management.elasticsearch.url: ["https://elasticCloudUrl"]

pipelines.yml

- pipeline.id: artist_profile_views
  path.config: "pipelines/artist_profile_views.conf"

- pipeline.id: searched
  path.config: "pipelines/searched.conf"

artist_profile_views

input { 
    file {
        path => "/Users/zabaala/Sites/cna/stats/artist_profile_views/artist_profile_views_*.log"
        codec => json
        start_position => "beginning"
    }
}

filter {
    geoip {
        source => "[ip]"
    }   
    useragent {
        source => "[headers][user_agent]"
        target => "[headers][request]"
    }
    mutate {
        remove_field => ["[headers][user_agent]"]
    }
}


output {
    elasticsearch { 
        # hosts => ["https://ElasticCloudUrl"] 
        index => "stats"
    }
    stdout { 
        codec => rubydebug
    }
}

Sample data:

{"artist_profile_views":{"id":"510","type":"ARTIST","area":"PHOTOS"},"env":"local","ip":"172.18.0.1","index":"stats","doc":"artist_profile_views","when":{"date":"2018-07-06T17:20:48-0300"},"viewer":{"id":null,"context":"GUEST","by_himself":true},"headers":{"user_agent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/67.0.3396.99 Safari\/537.36"}}
{"artist_profile_views":{"id":"510","type":"ARTIST","area":"EVENTS"},"env":"local","ip":"172.18.0.1","index":"stats","doc":"artist_profile_views","when":{"date":"2018-07-06T17:20:50-0300"},"viewer":{"id":null,"context":"GUEST","by_himself":true},"headers":{"user_agent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/67.0.3396.99 Safari\/537.36"}}
{"artist_profile_views":{"id":"510","type":"ARTIST","area":"AREA_AUDIOS"},"env":"local","ip":"172.18.0.1","index":"stats","doc":"artist_profile_views","when":{"date":"2018-07-06T17:20:52-0300"},"viewer":{"id":null,"context":"GUEST","by_himself":true},"headers":{"user_agent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/67.0.3396.99 Safari\/537.36"}}
{"artist_profile_views":{"id":"510","type":"ARTIST","area":"VIDEOS"},"env":"local","ip":"172.18.0.1","index":"stats","doc":"artist_profile_views","when":{"date":"2018-07-06T17:20:55-0300"},"viewer":{"id":null,"context":"GUEST","by_himself":true},"headers":{"user_agent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/67.0.3396.99 Safari\/537.36"}}
{"artist_profile_views":{"id":"510","type":"ARTIST","area":"HOME"},"env":"local","ip":"172.18.0.1","index":"stats","doc":"artist_profile_views","when":{"date":"2018-07-06T17:31:32-0300"},"viewer":{"id":null,"context":"GUEST","by_himself":true},"headers":{"user_agent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/67.0.3396.99 Safari\/537.36"}}
{"artist_profile_views":{"id":"510","type":"ARTIST","area":"AREA_AUDIOS"},"env":"local","ip":"172.18.0.1","index":"stats","doc":"artist_profile_views","when":{"date":"2018-07-06T17:31:43-0300"},"viewer":{"id":null,"context":"GUEST","by_himself":true},"headers":{"user_agent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/67.0.3396.99 Safari\/537.36"}}

What's wrong?

Thanks.

Mauricio Rodrigues
  • 757
  • 6
  • 9

1 Answers1

2

I found the answer.

This problem occur because the configuration xpack.managed.* use the Centralized Pipeline feature.

With centralized pipeline you don't need create any pipeline.yml file to configure a local logstash pipeline. With centralized pipeline, you put you own pipelines into elasticsearch. This is a resource of the xpack.

When you configure your logstash to be managed by elasticsearch, the pipelines will be loaded from logstash by id.

More about centralized pipeline can be found here: https://www.elastic.co/guide/en/logstash/current/logstash-centralized-pipeline-management.html

Mauricio Rodrigues
  • 757
  • 6
  • 9