Error - "INVALID" is not a valid start token

Question

I can see the target status as down and Error "INVALID" is not a valid start token in prometheus console.

I followed the steps below:

Install Prometheus on linux1 machine.
Install weblogic on linux2 machine.
Deploy jar file on weblogic server
Verify Gauge
Add weblogic server entry in prometheus.yml
Re-start prometheus service.

Below are detailed information -

prometheus logs :

level=warn ts=2019-09-06T11:42:42.187Z caller=scrape.go:937 component="scrape manager" scrape_pool=weblogic1 target=http://**********.*.****.*:7001/wls-exporter msg="append failed" err="\"INVALID\" is not a valid start token"

curl output1 :-
-bash-4.2$ curl http://**********.***.****.***:7001/wls-exporter | promtool check metrics
-bash: promtool: command not found
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1783  100  1783    0     0   323k      0 --:--:-- --:--:-- --:--:--  348k
(23) Failed writing body
-bash-4.2$

curl output2 :-
-bash-4.2$ curl -v --noproxy '*' 'http://**********.***.****.***:7001/wls-exporter'
* About to connect() to **********.***.****.*** port 7001 (#0)
*   Trying **.**.***.***...
* Connected to **********.***.****.*** (**.**.***.***) port 7001 (#0)
> GET /wls-exporter HTTP/1.1
> User-Agent: curl/7.29.0
> Host: **********.***.****.***:7001
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Fri, 06 Sep 2019 11:27:23 GMT
< Content-Length: 1783
<
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Weblogic Monitoring Exporter</title>
</head>
<body>
<h2>This is the WebLogic Monitoring Exporter.</h2>
<p>The metrics are found at <a href="/wls-exporter/metrics">
metrics</a> relative to this location.
</p>
<h2>Configuration</h2>
<p>To change the configuration:</p>
<form action="/wls-exporter/configure" method="post" enctype="multipart/form-data">
    <input type="radio" name="effect" value="append">Append
    <input type="radio" name="effect" value="replace" checked="checked">Replace
    <br><input type="file" name="configuration">
    <br><input type="submit">
</form>
<p>Current Configuration</p>
<p><code><pre>
host: **********.***.****.***
port: 7001
query_sync:
  url: http://coordinator:8999/
  refreshInterval: 5
metricsNameSnakeCase: true
domainQualifier: true
restPort: 7001
queries:
- key: name
  keyName: server
  applicationRuntimes:
    key: name
    keyName: app
    componentRuntimes:
      type: WebAppComponentRuntime
      prefix: webapp_config_
      key: name
      values: [deploymentState, contextRoot, sourceInfo, openSessionsHighCount, openSessionsCurrentCount, sessionsOpenedTotalCount, sessionCookieMaxAgeSecs, sessionInvalidationIntervalSecs, sessionTimeoutSecs, singleThreadedServletPoolSize, sessionIDLength, servletReloadCheckSecs, jSPPageCheckSecs]
      servlets:
        prefix: weblogic_servlet_
        key: servletName
        values: [invocationTotalCount, reloadTotal, executionTimeAverage, poolMaxCapacity, executionTimeTotal, reloadTotalCount, executionTimeHigh, executionTimeLow]
- JVMRuntime:
    prefix: jvm_
    key: name
    values: [heapFreeCurrent, heapFreePercent, heapSizeCurrent, heapSizeMax, uptime, processCpuLoad]
</pre></code></p>
* Connection #0 to host **********.***.****.*** left intact
-bash-4.2$

Thank you Michael for the comment. Please see below oputput :- -bash-4.2$ curl http://xxxxxxxxx.xxxx.xxxxx.xxx:7001/wls-exporter/metrics | promtool check metrics -bash: promtool: command not found % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 23 100 23 0 0 2073 0 --:--:-- --:--:-- --:--:-- 2300 (23) Failed writing body — satya, Sep 09 '19 at 03:46
-bash-4.2$ curl -v --noproxy '*' 'http://x.x.xx.x:7001/wls-exporter/metrics' * About to connect() to x.x.xx.x port 7001 (#0) * Trying x.x.x.x... * Connected to x.x.xx.x (x.x.x.x) port 7001 (#0) > GET /wls-exporter/metrics HTTP/1.1 > User-Agent: curl/7.29.0 > Host: x.x.xx.x:7001 > Accept: */* > < HTTP/1.1 401 Unauthorized < Date: Mon, 09 Sep 2019 03:44:02 GMT < Content-Length: 23 < Content-Type: text/html; charset=UTF-8 < WWW-Authenticate: Basic realm="weblogic" < * Connection #0 to host x.x.xx.x left intact Authentication required-bash-4.2$ — satya, Sep 09 '19 at 03:48
Here are your clues: error code 401 and message `Authentication required`. Did you provide the exporter the authorizations to access WebLogic API ? — Michael Doubez, Sep 09 '19 at 08:54
Michael - Where I should provide authorization input (in prometheus.yml or while generating jar file in config-file)? — satya, Sep 10 '19 at 05:33
I expect the WebLogic API requires authentication. From the code, the basic authentication provided in Prometheus config is forwarded to the API. Let me write an answer in this sense. — Michael Doubez, Sep 10 '19 at 06:52

score 7 · Answer 1 · answered Sep 10 '19 at 07:04

The error ""INVALID" is not a valid start token" is usually encountered when Prometheus expects an OpenMetric format but get something else. In this occurrence, the exporter landing page if you omit the /metrics at the en of the URL or if the exporter reports an error page (401 - Authentication required.

Looking into the relevant source code, it seems the authentication token sent by Prometheus to the exporter are forwarded to the Weblogic API. The prometheus config should look like:

- job_name: 'weblogic'
  ...
  basic_auth:
    username: weblogic
    password: friend

You can test it using curl with the relevant parameters:

curl -u 'weblogic:friend' http://**********.***.****.***:7001/wls-exporter/metrics | promtool check metrics

score 2 · Answer 2 · answered Sep 30 '20 at 05:43

In my case, my /metrics endpoint accidentally had a \ufeff byte-order mark (because in C# .Net I called new StreamWriter(httpListenerResponse.OutputStream, Encoding.UTF8) rather than new UTF8Encoding(false)). So double check that when you fetch the metrics endpoint (e.g. curl localhost:8080/metrics | hexdump -c), it’s all ASCII and the document looks like the prometheus exposition format

Error - "INVALID" is not a valid start token

2 Answers2

Linked