Unable to push vespa metrics to cloudwatch

Question

Basically I need to monitor vespa metrics and for that I am trying to implement method to push metrics to cloudwatch.

This is the document that I am referring to https://docs.vespa.ai/documentation/monitoring.html

I have added the credentials file and putMetricData permission in the IAM role attached. The service.xml file that I am using in my code looks like this:

      <admin version="2.0">
          <adminserver hostalias="admin0"/>
          <configservers>
              <configserver hostalias="admin0"/>
          </configservers>
          <monitoring>
          </monitoring>
          <metrics>
              <consumer id="my-cloudwatch">
                  <metric-set id="vespa" />
                  <cloudwatch region="ap-south-1" namespace="vespa">
                      <shared-credentials file="~/.aws/credentials" profile="default" />
                  </cloudwatch>
              </consumer>
          </metrics>
  </admin> 

I have deployed the code using vespa-deploy prepare application.zip && vespa-deploy activatebut I am still not seeing any metrics updated on my cloudwatch.

Also, I have tried to add:

<monitoring>
  <interval>1</interval>
  <systemname>vespa</systemname>
</monitoring>

But getting this error when deploying:

Request failed. HTTP status code: 400
Invalid application package: default.default: Error loading model: XML error in services.xml: element "interval" not allowed here; expected the element end-tag [9:16], input:

How can I fix this issue. Or atleast debug the issue that I am facing.

Answer 1

I suggest to use absolute path to the credentials file, as the ~ may not resolve to the directory you intended at runtime.

A couple more things:

• I recommend using the default metric set, as vespa contains a lot of metrics, which will drive your CloudWatch cost higher. If you need additional metrics, you can add them with the metric tag inside consumer.
• The monitoring element doesn't do anything useful in this context, so you should just drop it.

If you still don't see any metrics, please check for warnings or errors in the vespa log file (use vespa-logfmt) and the Telegraf log file: /opt/vespa/logs/telegraf/telegraf.log. (Vespa uses Telegraf internally to emit metrics to CloudWatch.)