-1

Describe the bug OTEL Collector not able to connect to elastic-apm

Steps to reproduce

Please follow the instructions here, it is simple App https://github.com/sathishsoundharajan/otel-elastic-apm

What did you expect to see? i'm expecting to see the tracing logs to be sent both to zipkin and elastic-apm server.

What did you see instead?

otel-collector    | 2022-07-29T14:39:44.207Z    error   exporterhelper/queued_retry.go:149  Exporting failed. Try enabling retry_on_failure config option to retry on retryable errors  {"kind": "exporter", "data_type": "traces", "name": "elastic", "error": "sending event request failed: Post \"http://elastic-apm:8200/intake/v2/events\": dial tcp 192.168.32.6:8200: connect: connection refused", "errorVerbose": "Post \"http://elastic-apm:8200/intake/v2/events\": dial tcp 192.168.32.6:8200: connect: connection refused\nsending event request failed\ngo.elastic.co/apm/transport.(*HTTPTransport).sendStreamRequest\n\tgo.elastic.co/apm@v1.15.0/transport/http.go:292\ngo.elastic.co/apm/transport.(*HTTPTransport).SendStream\n\tgo.elastic.co/apm@v1.15.0/transport/http.go:282\ngithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticexporter.(*elasticExporter).sendEvents\n\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticexporter@v0.56.0/exporter.go:197\ngithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticexporter.(*elasticExporter).ExportResourceSpans\n\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticexporter@v0.56.0/exporter.go:149\ngithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticexporter.newElasticTracesExporter.func1\n\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticexporter@v0.56.0/exporter.go:53\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesRequest).export\n\tgo.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/traces.go:70\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send\n\tgo.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/common.go:225\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send\n\tgo.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/queued_retry.go:147\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send\n\tgo.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/traces.go:134\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).send\n\tgo.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/queued_retry.go:83\ngo.opentelemetry.io/collector/exporter/exporterhelper.NewTracesExporter.func2\n\tgo.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/traces.go:113\ngo.opentelemetry.io/collector/consumer.ConsumeTracesFunc.ConsumeTraces\n\tgo.opentelemetry.io/collector@v0.56.0/consumer/traces.go:36\ngo.opentelemetry.io/collector/service/internal/fanoutconsumer.(*tracesConsumer).ConsumeTraces\n\tgo.opentelemetry.io/collector@v0.56.0/service/internal/fanoutconsumer/traces.go:75\ngo.opentelemetry.io/collector/processor/batchprocessor.(*batchTraces).export\n\tgo.opentelemetry.io/collector@v0.56.0/processor/batchprocessor/batch_processor.go:262\ngo.opentelemetry.io/collector/processor/batchprocessor.(*batchProcessor).sendItems\n\tgo.opentelemetry.io/collector@v0.56.0/processor/batchprocessor/batch_processor.go:176\ngo.opentelemetry.io/collector/processor/batchprocessor.(*batchProcessor).startProcessingCycle\n\tgo.opentelemetry.io/collector@v0.56.0/processor/batchprocessor/batch_processor.go:143\nruntime.goexit\n\truntime/asm_amd64.s:1571"}
otel-collector    | go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
otel-collector    |     go.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/queued_retry.go:149
otel-collector    | go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send
otel-collector    |     go.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/traces.go:134
otel-collector    | go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).send
otel-collector    |     go.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/queued_retry.go:83
otel-collector    | go.opentelemetry.io/collector/exporter/exporterhelper.NewTracesExporter.func2
otel-collector    |     go.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/traces.go:113
otel-collector    | go.opentelemetry.io/collector/consumer.ConsumeTracesFunc.ConsumeTraces
otel-collector    |     go.opentelemetry.io/collector@v0.56.0/consumer/traces.go:36
otel-collector    | go.opentelemetry.io/collector/service/internal/fanoutconsumer.(*tracesConsumer).ConsumeTraces
otel-collector    |     go.opentelemetry.io/collector@v0.56.0/service/internal/fanoutconsumer/traces.go:75
otel-collector    | go.opentelemetry.io/collector/processor/batchprocessor.(*batchTraces).export
otel-collector    |     go.opentelemetry.io/collector@v0.56.0/processor/batchprocessor/batch_processor.go:262
otel-collector    | go.opentelemetry.io/collector/processor/batchprocessor.(*batchProcessor).sendItems
otel-collector    |     go.opentelemetry.io/collector@v0.56.0/processor/batchprocessor/batch_processor.go:176
otel-collector    | go.opentelemetry.io/collector/processor/batchprocessor.(*batchProcessor).startProcessingCycle
otel-collector    |     go.opentelemetry.io/collector@v0.56.0/processor/batchprocessor/batch_processor.go:143

Docker Compose

version: '3'
services:

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.8.23
    container_name: elasticsearch
    environment:
      - xpack.security.enabled=false
      - discovery.type=single-node
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    cap_add:
      - IPC_LOCK
    volumes:
      - elasticsearch-data:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
      - 9300:9300
    healthcheck:
      test: ["CMD-SHELL", "curl -s http://localhost:9200/_cluster/health | grep -vq '\"status\":\"red\"'"]
      retries: 10
      interval: 20s
    networks:
      - opentelemetry

  kibana:
    container_name: kibana
    image: docker.elastic.co/kibana/kibana:6.8.23
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    ports:
      - 5601:5601
    depends_on:
      elasticsearch:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "curl", "--write-out", "'HTTP %{http_code}'", "--silent", "--output", "/dev/null", "http://localhost:5601/"]
      retries: 10
      interval: 10s
    networks:
      - opentelemetry

  elastic-apm:
    container_name: elastic-apm
    build:
      context: ../
      dockerfile: ./build/elastic-apm/Dockerfile
    ports:
      - 8200:8200
    cap_drop:
      - ALL
    cap_add:
      - CHOWN
      - DAC_OVERRIDE
      - SETGID
      - SETUID
    depends_on:
      elasticsearch:
        condition: service_healthy
      kibana:
        condition: service_healthy
    healthcheck:
      interval: 10s
      retries: 12
      test: curl --write-out 'HTTP %{http_code}' --fail --silent --output /dev/null http://localhost:8200/
    networks:
      - opentelemetry

  zipkin:
    image: openzipkin/zipkin
    container_name: zipkin
    ports:
      - 9411:9411
    networks:
      - opentelemetry

  otel-collector:
    container_name: otel-collector
    build:
      context: ../
      dockerfile: ./build/collector/Dockerfile
    ports:
      - 4318:4318
      - 13133:13133
    networks:
      - opentelemetry

volumes:
  elasticsearch-data:
    driver: local

networks:
  opentelemetry:
    driver: bridge

apm-server.yml

apm-server:
  rum:
    enabled: true
  kibana:
    enabled: true
    host: "kibana:5601"

output.elasticsearch:
  hosts: ["http://elasticsearch:9200"]

setup.kibana.host: "kibana:5601"
setup.kibana.ssl.enabled: false

Environment Docker

Additional context What I'm trying to do is setup small node.js application

  1. Collect the traces via OTEL Collector
  2. Make sure OTEL Collector sends data to Zipkin and as well as Elastic APM

Curl from Host Machine

otel-elastic-apm/build on  main [!] on  v20.10.17 took 3s
➜ curl http://localhost:8200/
curl: (52) Empty reply from server

Curl inside the apm-server

otel-elastic-apm/build on  main [!] on  v20.10.17
➜ docker ps
CONTAINER ID   IMAGE                                                  COMMAND                  CREATED         STATUS                   PORTS                                                                         NAMES
576b44ad7b96   build_elastic-apm                                      "/usr/local/bin/dock…"   3 minutes ago   Up 3 minutes (healthy)   0.0.0.0:8200->8200/tcp                                                        elastic-apm
25e1dd38fb85   docker.elastic.co/kibana/kibana:6.8.23                 "/usr/local/bin/kiba…"   4 minutes ago   Up 4 minutes (healthy)   0.0.0.0:5601->5601/tcp                                                        kibana
a7b72d802380   docker.elastic.co/elasticsearch/elasticsearch:6.8.23   "/usr/local/bin/dock…"   4 minutes ago   Up 4 minutes (healthy)   0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp                                elasticsearch
76f5e0700e10   openzipkin/zipkin                                      "start-zipkin"           4 minutes ago   Up 4 minutes (healthy)   9410/tcp, 0.0.0.0:9411->9411/tcp                                              zipkin
09c318ae9118   build_otel-collector                                   "/otelcol-contrib --…"   4 minutes ago   Up 4 minutes             0.0.0.0:4318->4318/tcp, 4317/tcp, 55678-55679/tcp, 0.0.0.0:13133->13133/tcp   otel-collector

otel-elastic-apm/build on  main [!] on  v20.10.17
➜ docker exec -it 576b44ad7b96 sh
sh-4.2$ curl http://localhost:8200/
{"ok":{"build_date":"2020-07-09T18:01:36Z","build_sha":"dda44e48cd9c3bac7e5a359a11974594d1d10347","version":"6.8.11"}}sh-4.2$

Another Weird Thing:

I can see the docker-compose logs for Health check is working.

otel-elastic-apm/build on  main [!] on  v20.10.17
➜ docker ps
CONTAINER ID   IMAGE                                                  COMMAND                  CREATED         STATUS                   PORTS                                                                         NAMES
576b44ad7b96   build_elastic-apm                                      "/usr/local/bin/dock…"   5 minutes ago   Up 5 minutes (healthy)   0.0.0.0:8200->8200/tcp                                                        elastic-apm
25e1dd38fb85   docker.elastic.co/kibana/kibana:6.8.23                 "/usr/local/bin/kiba…"   6 minutes ago   Up 6 minutes (healthy)   0.0.0.0:5601->5601/tcp                                                        kibana
a7b72d802380   docker.elastic.co/elasticsearch/elasticsearch:6.8.23   "/usr/local/bin/dock…"   6 minutes ago   Up 6 minutes (healthy)   0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp                                elasticsearch
76f5e0700e10   openzipkin/zipkin                                      "start-zipkin"           6 minutes ago   Up 6 minutes (healthy)   9410/tcp, 0.0.0.0:9411->9411/tcp                                              zipkin
09c318ae9118   build_otel-collector                                   "/otelcol-contrib --…"   6 minutes ago   Up 6 minutes             0.0.0.0:4318->4318/tcp, 4317/tcp, 55678-55679/tcp, 0.0.0.0:13133->13133/tcp   otel-collector

otel-elastic-apm/build on  main [!] on  v20.10.17
➜ docker logs 576b44ad7b96
2022-07-30T07:40:18.218Z    INFO    instance/beat.go:611    Home path: [/usr/share/apm-server] Config path: [/usr/share/apm-server] Data path: [/usr/share/apm-server/data] Logs path: [/usr/share/apm-server/logs]
2022-07-30T07:40:18.221Z    INFO    instance/beat.go:618    Beat UUID: 95087cff-e948-4969-90fc-ddc9d925f877
2022-07-30T07:40:18.222Z    INFO    [seccomp]   seccomp/seccomp.go:116  Syscall filter successfully installed
2022-07-30T07:40:18.222Z    INFO    [beat]  instance/beat.go:931    Beat info   {"system_info": {"beat": {"path": {"config": "/usr/share/apm-server", "data": "/usr/share/apm-server/data", "home": "/usr/share/apm-server", "logs": "/usr/share/apm-server/logs"}, "type": "apm-server", "uuid": "95087cff-e948-4969-90fc-ddc9d925f877"}}}
2022-07-30T07:40:18.223Z    INFO    [beat]  instance/beat.go:940    Build info  {"system_info": {"build": {"commit": "dda44e48cd9c3bac7e5a359a11974594d1d10347", "libbeat": "6.8.11", "time": "2020-07-09T18:01:36.000Z", "version": "6.8.11"}}}
2022-07-30T07:40:18.223Z    INFO    [beat]  instance/beat.go:943    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.10.8"}}}
2022-07-30T07:40:18.224Z    INFO    [beat]  instance/beat.go:947    Host info   {"system_info": {"host": {"architecture":"x86_64","boot_time":"2022-07-29T19:44:59Z","containerized":false,"name":"576b44ad7b96","ip":["127.0.0.1/8","192.168.112.6/20"],"kernel_version":"5.10.104-linuxkit","mac":["02:42:c0:a8:70:06"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":8,"patch":2003,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2022-07-30T07:40:18.224Z    INFO    [beat]  instance/beat.go:976    Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":null,"effective":null,"bounding":["chown","dac_override","setgid","setuid"],"ambient":null}, "cwd": "/usr/share/apm-server", "exe": "/usr/share/apm-server/apm-server", "name": "apm-server", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2022-07-30T07:40:17.130Z"}}}
2022-07-30T07:40:18.224Z    INFO    instance/beat.go:280    Setup Beat: apm-server; Version: 6.8.11
2022-07-30T07:40:18.225Z    INFO    elasticsearch/client.go:164 Elasticsearch url: http://elasticsearch:9200
2022-07-30T07:40:18.225Z    INFO    [publisher] pipeline/module.go:110  Beat name: 576b44ad7b96
2022-07-30T07:40:18.225Z    INFO    [beater]    beater/beater.go:91 Falling back to elasticsearch output for sourcemap storage
2022-07-30T07:40:18.226Z    INFO    [beater]    beater/beater.go:112    No pipeline callback registered
2022-07-30T07:40:18.226Z    INFO    instance/beat.go:402    apm-server start running.
2022-07-30T07:40:18.226Z    INFO    [beater]    beater/beater.go:241    self instrumentation is disabled
2022-07-30T07:40:18.228Z    INFO    [beater]    beater/beater.go:148    host resolved from localhost:8200 to 127.0.0.1:8200
2022-07-30T07:40:18.228Z    INFO    [handler]   beater/common_handlers.go:163   Path /v1/metrics added to request handler
2022-07-30T07:40:18.228Z    INFO    [handler]   beater/common_handlers.go:163   Path /assets/v1/sourcemaps added to request handler
2022-07-30T07:40:18.229Z    INFO    elasticsearch/client.go:164 Elasticsearch url: http://elasticsearch:9200
2022-07-30T07:40:18.229Z    INFO    [handler]   beater/common_handlers.go:163   Path /v1/rum/sourcemaps added to request handler
2022-07-30T07:40:18.229Z    INFO    [handler]   beater/common_handlers.go:163   Path /v1/rum/errors added to request handler
2022-07-30T07:40:18.229Z    INFO    [handler]   beater/common_handlers.go:163   Path /v1/client-side/sourcemaps added to request handler
2022-07-30T07:40:18.229Z    INFO    [handler]   beater/common_handlers.go:163   Path /v1/transactions added to request handler
2022-07-30T07:40:18.229Z    INFO    [handler]   beater/common_handlers.go:163   Path /v1/client-side/transactions added to request handler
2022-07-30T07:40:18.229Z    INFO    [handler]   beater/common_handlers.go:163   Path /v1/rum/transactions added to request handler
2022-07-30T07:40:18.229Z    INFO    [handler]   beater/common_handlers.go:163   Path /v1/errors added to request handler
2022-07-30T07:40:18.229Z    INFO    [handler]   beater/common_handlers.go:163   Path /v1/client-side/errors added to request handler
2022-07-30T07:40:18.230Z    INFO    [handler]   beater/common_handlers.go:169   Path /intake/v2/events added to request handler
2022-07-30T07:40:18.230Z    INFO    [handler]   beater/common_handlers.go:169   Path /intake/v2/rum/events added to request handler
2022-07-30T07:40:18.230Z    INFO    [server]    beater/server.go:76 Starting apm-server [dda44e48cd9c3bac7e5a359a11974594d1d10347 built 2020-07-09 18:01:36 +0000 UTC]. Hit CTRL-C to stop it.
2022-07-30T07:40:18.231Z    INFO    [server]    beater/server.go:77 Listening on: 127.0.0.1:8200
2022-07-30T07:40:18.231Z    INFO    [server]    beater/server.go:80 RUM endpoints enabled!
2022-07-30T07:40:18.231Z    INFO    [server]    beater/server.go:97 SSL disabled.
2022-07-30T07:40:18.231Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "9da73838-38e5-473b-b89a-807a151b8d09", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "", "response_code": 200}
2022-07-30T07:40:18.232Z    INFO    [onboarding]    beater/onboarding.go:29 Publishing onboarding document
2022-07-30T07:40:19.233Z    INFO    pipeline/output.go:95   Connecting to backoff(elasticsearch(http://elasticsearch:9200))
2022-07-30T07:40:19.305Z    INFO    elasticsearch/client.go:739 Attempting to connect to Elasticsearch version 6.8.23
2022-07-30T07:40:19.325Z    INFO    template/load.go:128    Template already exists and will not be overwritten.
2022-07-30T07:40:19.325Z    INFO    instance/beat.go:889    Template successfully loaded.
2022-07-30T07:40:19.325Z    INFO    pipeline/output.go:105  Connection to backoff(elasticsearch(http://elasticsearch:9200)) established
2022-07-30T07:40:28.266Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "82fe1be3-5aaf-42d7-9315-02a677a6e8e1", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:40:38.378Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "3657ba8c-624f-4a54-bbea-c9448d6960a0", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:40:48.448Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "0b8d1559-524e-41ef-9ee2-afb869b4ef7b", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:40:58.550Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "88ec8af0-fa90-4977-9157-e72a472034b1", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:41:08.626Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "fd44355c-372a-41b8-a36b-bdaadbb36baa", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:41:18.725Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "726154c7-8b95-4170-8de1-d6901d534346", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:41:28.821Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "62373e63-f71f-4cc0-9c30-05e185526b98", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:41:38.907Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "2e196be1-46a1-4805-8b90-d2884196b530", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:41:49.012Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "e8c2c571-4ea8-477e-9581-5b7d1014c526", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:41:59.127Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "c6bc6637-6d4e-4d13-85e7-e21ae965dfdc", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:42:09.195Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "76bc029f-2831-410d-9069-a49f78b0de17", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:42:19.300Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "eee825b5-3545-496b-b15a-aeb876ca6f19", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:42:29.406Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "ada4dec2-45df-4517-9417-24d9ac96bb23", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:42:39.486Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "4be275b8-c8fa-4129-b92d-69f460b1a5ea", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:42:49.601Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "59756a31-d187-44ec-ac54-d9041c18b1da", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:42:59.715Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "734cdf3d-4e10-4810-9b69-41936de27788", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:43:09.807Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "69afa8e6-4f85-40c6-aab1-c9515d4bcf67", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:43:19.924Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "66110393-e208-4de3-97f1-470049a87a93", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:43:30.046Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "49bd7707-6071-44ff-92fe-0471b46f861c", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:43:40.140Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "96f06d42-c2ef-420f-b26c-80bf604d4baa", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:43:50.242Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "6a1e5703-bbc9-4280-b278-78e0b5a9922d", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:44:00.339Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "08709ff6-bb09-49f5-aee1-469443bab827", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:44:08.800Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "96201795-cc5d-4726-b1d6-87594036f7b5", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:44:10.414Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "40848199-1d34-4c1f-9292-207a2158dba1", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:44:20.524Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "cbd6b50c-20b0-460a-9653-327f7c975604", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:44:30.618Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "36d94913-76c6-4800-9429-a46dd8758a24", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:44:40.694Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "8cd3b68d-73ce-4d8f-a126-3b01145a935d", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:44:50.789Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "4ae17fad-65de-4719-b3c6-e707239b2ca0", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:45:00.893Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "9c181bba-509b-4fd1-81c8-5e474681c8e3", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:45:10.989Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "b70adb0b-7bd7-4f2f-871e-612e276c5256", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:45:21.091Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "639b21fd-9bb4-4d7e-b51a-13c93c30a5c6", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:45:31.187Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "ee7ceff9-8863-4e70-8a99-98135a81e99a", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:45:41.264Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "b1dcd689-3c03-4e83-b0ed-9ade4bbda7d9", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:45:51.361Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "c82e0fc0-f9d4-4097-a158-317d3427784a", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:46:01.478Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "6ad63614-a1d0-416d-9a9d-636a68210c00", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}
2022-07-30T07:46:11.549Z    INFO    [request]   beater/common_handlers.go:272   handled request {"request_id": "f65fcc6f-8100-46e8-bdee-1f859722e5e8", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "curl/7.29.0", "response_code": 200}

After fixing the host, now i'm running into new issue when i use otel/elastic exporter

otel-collector    | 2022-08-01T10:04:18.540Z    info    service/collector.go:215    Starting otelcol-contrib... {"Version": "0.56.0", "NumCPU": 4}
otel-collector    | 2022-08-01T10:04:18.540Z    info    service/collector.go:128    Everything is ready. Begin running and processing data.
otel-collector    | 2022-08-01T10:04:19.330Z    warn    zapgrpc/zapgrpc.go:191  [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
otel-collector    |   "Addr": "elastic-apm:8200",
otel-collector    |   "ServerName": "elastic-apm:8200",
otel-collector    |   "Attributes": null,
otel-collector    |   "BalancerAttributes": null,
otel-collector    |   "Type": 0,
otel-collector    |   "Metadata": null
otel-collector    | }. Err: connection error: desc = "transport: Error while dialing dial tcp: lookup elastic-apm on 127.0.0.11:53: no such host"   {"grpc_log": true}
Sathish
  • 2,056
  • 3
  • 26
  • 40
  • 1. What's the configuration of the OTel collector? Because IMO the failure is going from the collector to the APM server. – xeraa Jul 31 '22 at 15:17
  • 2. Is there a reason for using such an ancient Elastic Stack version? 6.x isn't supported any more. Also more current versions of the APM server support the OTel protocol natively (which should simplify the config and also make the OTel collector optional) – xeraa Jul 31 '22 at 15:18
  • @xeraa 1. Let's leave OTEL Collector. My Problem is APM Server running in docker not accessible outside docker ( host machine, trying to curl from host machine itself not working ) – Sathish Jul 31 '22 at 17:21
  • @xeraa 2. Our production systems are running in 6.x i cannot simply upgrade the system. Also i don't want to use to Elastic APM alone, like i mentioned above i would like to use both Zipkin and as well Elastic APM. So OTEL Collector is best way – Sathish Jul 31 '22 at 17:22

1 Answers1

2

With the custom Dockerfile and config, did you overwrite the default bind to all interfaces? Source from 6.8: https://github.com/elastic/apm-server/blob/6.8/apm-server.docker.yml#L7

This looks suspicious: Listening on: 127.0.0.1:8200

PS: You still want to upgrade to a later version of the APM server that supports the OTel protocol natively. You can keep using the OTel collector, but then you won't have to depend on the custom exporter (that we're IMO also not maintaining any more).

xeraa
  • 10,456
  • 3
  • 33
  • 66
  • Thank you for your answer. I got similar solution in discussion forum. https://discuss.elastic.co/t/elastic-apm-not-accessible-outside-docker-container/311026/2?u=sathishsoundharajan – Sathish Aug 01 '22 at 05:06
  • We are running elasticsearch version 6.8.x in production and kibana also similar to that version. So can i run APM latest version 8.3 and elastic 6.x ? Also can i run APM version 6.8 itself and use oltp/elastic exporter ? Can i independently upgrade APM server alone ? – Sathish Aug 01 '22 at 05:07
  • Seems after upgrading to otel/elastic exporter.. I'm now running into new issue. Updated the post on new issue – Sathish Aug 01 '22 at 10:56
  • 1
    No, the versions should be in sync. This will otherwise only lead to more trouble – xeraa Aug 01 '22 at 23:13
  • Thank you @xeraa will keep the version in sync for now.. Can you give me some pointers `transport: Error while dialing dial tcp: lookup elastic-apm on 127.0.0.11:53: no such host` when using otlp/elastic exporter ? – Sathish Aug 02 '22 at 06:40
  • Your app is running on the same network as the Docker Compose stuff from above? – xeraa Aug 02 '22 at 08:44
  • My Node.js Application is running outside the docker, but OTEL Collector, Kibana, Elastisearch, Zipkin, Elastic APM all of them are running inside docker-container in same network. You can check this link for more info https://github.com/sathishsoundharajan/otel-elastic-apm – Sathish Aug 02 '22 at 16:54
  • The error is from OTEL Collector, Collector tries initiate gGRPC connection with Elastic APM Server, but it says it not able to do dns lookup. But i know for sure Elastic APM is running and accessible. – Sathish Aug 02 '22 at 16:55