3

I'm having a strange problem I can't work out as my problem, when searching for this error, is different. People seem to have experienced this when trying to connect Filebeat to Logstash.

However, I am trying to write logs directly to Elasticsearch but I am getting Logstash related errors even though I am not even spinning up a container in Docker Compose??

Main Docker Compose File:

version: '2.2'
services:
  filebeat:
    container_name: filebeat
    build:
      context: .
      dockerfile: filebeat.Dockerfile
    volumes:
      - ./logs:/var/log
    networks:
      - esnet
  elasticsearch:
    container_name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:7.5.2
    ports:
      - 9200:9200
      - 9300:9300
    environment:
      - discovery.type=single-node
      - cluster.name=docker-
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    networks:
      - esnet
  elastichq:
    container_name: elastichq
    image: elastichq/elasticsearch-hq
    ports:
      - 8080:5000
    environment:
      - HQ_DEFAULT_URL=http://elasticsearch:9200
      - HQ_ENABLE_SSL=False
      - HQ_DEBUG=FALSE
    networks:
      - esnet  
networks:
  esnet:

DockerFile for Filebeat

FROM docker.elastic.co/beats/filebeat:7.5.2
COPY filebeat/filebeat.yml /usr/share/filebeat/filebeat.yml
USER root
RUN chown root:filebeat /usr/share/filebeat/filebeat.yml
RUN chmod 644 /usr/share/filebeat/filebeat.yml
USER filebeat

I am trying to read json logs that are already in Elasticsearch format, so after reading the docs I decided to try and write directly to Elasticsearch which seems to be valid depending on the application.

My Sample.json file:

{"@timestamp":"2020-02-10T09:35:20.7793960+00:00","level":"Information","messageTemplate":"The value of i is {LoopCountValue}","message":"The value of i is 0","fields":{"LoopCountValue":0,"SourceContext":"WebAppLogger.Startup","Environment":"Development","ApplicationName":"ELK Logging Demo"}}

My Filebeat.yml:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.json
  json.keys_under_root: true
  json.add_error_key: true
  json.message_key: log  

#----------------------------- Elasticsearch output --------------------------------

output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  index: "sample-%{+YYYY.MM.dd}"

As stated in the title of this post, I get this message in the console:

filebeat | 2020-02-10T09:38:24.438Z ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://logstash:5044)): lookup logstash on 127.0.0.11:53: no such host

Then when I eventually try and visualize the data in ElasticHq, inevitably, nothing is there.

So far, I've tried using commands like docker prune just in case theres something funny going on with Docker.

Is there something I'm missing?

user1574598
  • 3,771
  • 7
  • 44
  • 67

1 Answers1

1

You have misconfigured your filebeat.yml file. Look at this error:

Failed to connect to backoff(async(tcp://logstash:5044))

Filebeat tries to connect to logstash, beacause this is the default configuration. In fact on one hand you show a filebeat.yml file and on the other hand, you haven't mounted it on /usr/share/filebeat/filebeat.yml - look at your volumes settings

  filebeat:
    container_name: filebeat
    build:
      context: .
      dockerfile: filebeat.Dockerfile
    volumes:
      - ./logs:/var/log
    networks:
      - esnet

You should mount it. If you try to copy it inside a docker container with dockerfile - why?????is necessary reinvent the wheel and add complexity? - you should use the root user:

USER root

and add root user to your service in docker-compose.yml:

user: root
Lupanoide
  • 3,132
  • 20
  • 36
  • I've added `./filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml` to my `volume` and kept the Filebeat `Dockerfile` the same (see in my post `filebeat.Dockerfile`). Where does `user: root` go in the `docker-compose`? Within Filebeat, or somewhere else? – user1574598 Feb 10 '20 at 11:10
  • yep, at the same level of networks or volumes, or container_name – Lupanoide Feb 10 '20 at 11:11
  • Thanks, I've just put it under `container_name` and I get: `Exiting: error loading config file: config file ("filebeat.yml") can only be writable by the owner but the permissions are "-rw-rw-r--" (to fix the permissions use: 'chmod go-w /usr/share/` – user1574598 Feb 10 '20 at 11:18
  • run the command suggested chmod go-w /path/to/your/filebeat.yml on your desktop, then rerun docker-compose – Lupanoide Feb 10 '20 at 11:24
  • 1
    I have a new error now: `2020-02-10T12:05:41.619Z ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://elasticsearch:9200)): Get http://elasticsearch:9200` – user1574598 Feb 10 '20 at 12:15
  • this is strange.. elastic is up? please post output of docker ps – Lupanoide Feb 10 '20 at 12:48
  • Yeah, I've googled it and found no solution, I even made sure Elasticsearch was spun up first in `docker-compose`. Just done `docker ps` and its running on port `9200`. I also tried `localhost:9200/_cat/indices?v` via the browser and it works. `docker.elastic.co/elasticsearch/elasticsearch:7.5.2 "/usr/local/bin/dock…" 2 minutes ago Up 2 minutes 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp elasticsearch` – user1574598 Feb 10 '20 at 13:03
  • there is something interesting in the elasticsearch logs? – Lupanoide Feb 10 '20 at 13:09
  • I've just tried putting the logs in my post but there is too much text and I've crashed the page. – user1574598 Feb 10 '20 at 13:18
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/207528/discussion-between-user1574598-and-lupanoide). – user1574598 Feb 10 '20 at 13:23