Using ELK service from Swisscom cloud from outside

Question

We'd like to use the ELK service provided by the Swisscom cloud. Because the applications we want to log are not hosted with Swisscom, but externally, we'd like to connect to the ELK service from outside. Is this possible at all? Or is the ELK service only available to Apps hosted in the Swisscom cloud?

Answer 1

When you create and bind ELK service you will receive connection string and credentials like this

$ cf env $APP
Getting env variables for app $APP in org $ORC / space $SPACE as $USER...
OK

System-Provided:
{
 "VCAP_SERVICES": {
  "elk": [
   {
    "credentials": {
     "elasticSearchHost": "9zz2ulprvgzlepa5.service.consul",
     "elasticSearchPassword": "$PASSWORD",
     "elasticSearchPort": 48783,
     "elasticSearchUsername": "$USERNAME",
     "kibanaPassword": "$PASSWORD",
     "kibanaUrl": "http://xjcv9zh0jer2s44q.service.consul:59664",
     "kibanaUsername": "$USERNAME",
     "logstashHost": "gew5qn71sxcz49gd.service.consul",
     "logstashPort": 46611,
     "syslog": "syslog://uew5qn71sxcz49gd.service.consul:46611"
    },
    "label": "elk",
    "name": "example-so",
    "plan": "small",
    "provider": null,
    "syslog_drain_url": "syslog://gew5qn71sxcz49gd.service.consul:46611",
    "tags": []
   }
  ],

You can't reach the addresses *.service.consul from outside (DNS is only available in Swisscom Cloud). You can only reach those addresses from your App (running in Cloud Foundry container).

There is a workaround, but I recommend only for development purposes.

You can create from your local desktop a tunnel to Elasticsearch or Kibana web interface.

See Administrating Service Instances with Service Connector. This is a CF CLI plugin developed by Swisscom.

After creating a service instance, you’ll eventually need to administrate the service. For example you might need to create data tables in a database or backup/restore your data. For these use cases, we created the Cloud Foundry CLI Plugin Service Connector which is a local proxy app through which you can connect to your service instances using your preferred locally installed tools.

example for Kibana web interface.

cf service-connector 80 xjcv9zh0jer2s44q.service.consul:59664

you can also access Elasticsearch from your desktop and use API for inserting or query documents.

Answer 2

The ELK stack has three components:

• Elastic Search - storage, index
• Logstash - receive and process log messages (like syslog, JSON, text)
• Kibana - Web UI to search and visualize

Like written by @Fydor, you cannot reach ELK's service endpoints from the outside. This is also an issue, if you want to access the logs of you CF hosted apps. You do not always want to have to use Swisscom's service connector to access Kibana.

Thus normally, you deploy a small proxy application. Swisscom has a sample for that one.

Alternatively there is the possiblity to use a proxy app like the Swisscom Kibana Proxy to make your Kibana dashboard publicly available.

As Elastic Search uses a REST interface, you can use the proxy to publish the Elastic Search endpoint. Eventually, you should also take the chance, to put some security measures into the proxy app.

There are already many logging frameworks, which directly support forwarding to Elastic Search.

If you need to integrate into existing logging solutions (like Syslog, text logs, ...) then you might want to use logstash.

As Cloud Foundry currently supports only publishing HTTP and HTTPS endpoints, you cannot use Swisscom's provided instance for that, but must deploy your own instance and configure this to use your published Elastic Search endpoint.