0

Am trying to run Elastic APM in GKE cluster. I have installed elastic-search, kibana and apm-server. All services are up and running. All these components has been through helm charts. Below are the configuration for each component.

apmConfig:
  apm-server.yml: |
    apm-server:
      host: "0.0.0.0:8200"

    queue: {}

    output.elasticsearch:
      hosts: ["http://elasticsearch-master.monitoring.svc.cluster.local:9200"]
      username: "elastic"
      password: "password"

kibanaConfig: 
  kibana.yml: |
    server.host: 0.0.0.0
    server.port: 5601
    elasticsearch.hosts: "http://elasticsearch-master.monitoring.svc.cluster.local:9200"
    kibana.index: ".kibana"
    server.basePath: "/kibana"
    server.rewriteBasePath: true
    server.publicBaseUrl: "https://mydomain/kibana"
    elasticsearch:
      username: "kibana_system"
      password: "password"

I have tried to add APM integration to one of my service by using the below config:

var apm = require('elastic-apm-node').start({

    // Override the service name from package.json
    // Allowed characters: a-z, A-Z, 0-9, -, _, and space
    serviceName: 'shopping',
    
    // Use if APM Server requires a secret token
    secretToken: '',
    
    // Set the custom APM Server URL (default: http://localhost:8200)
    serverUrl: 'https://mydomain/apm',
    
    // Set the service environment
    environment: 'production'
    })

When I start the service, I get the below error in logs:

{"log.level":"error","@timestamp":"2022-08-18T10:08:31.584Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (301): Unexpected APM Server response"}

301 - Moved permanently. It would be great , if I could get any help?

Sunil
  • 553
  • 1
  • 12
  • 30
  • What is the exact value you have in `serverUrl`? It should be the same as the one you have in `apm-server.host` (at least the port 8200 should match) – Val Aug 18 '22 at 11:17
  • My `apm-server.host` value is `"0.0.0.0:8200"`. Will it take `https://mydomain/apm` value, which is `serverUrl` value? am using istio ingress – Sunil Aug 18 '22 at 14:05
  • Yeah, somehow you need your APM client to really point to your APM server on port 8200 – Val Aug 18 '22 at 14:09
  • OK, I will try the same – Sunil Aug 18 '22 at 14:12
  • It does seem to be working. The error is `"Exiting: listen tcp: address tcp///mydomain/apm: unknown port"` – Sunil Aug 18 '22 at 14:26
  • You need to figure out what concrete public domain name, resp. public IP address, your APM server resolves to and use that from your APM client whose code runs in a browser outside of GKE – Val Aug 18 '22 at 14:28

1 Answers1

0

The problem have been solved. The reason was because I was using a proxy and hitting port 80, while in the APM server service I was using port as 8200.

After changing the port to 80 and targetPort to 8200 value in apm server service, I was able to correctly instrument Elastic APM with my services.

Sunil
  • 553
  • 1
  • 12
  • 30