7

I've been struggling to understand why some HTTP endpoints return '0' for 'probe_success' and 'probe_http_status_code', while being perfectly able to get a "valid" response with curl.

Example: curl -s "localhost:9115/probe?target=http://linux.org&module=http_2xx" | grep -v '^#'

Output:

probe_dns_lookup_time_seconds 0.003712821
probe_duration_seconds 0.212811871
probe_failed_due_to_regex 0
probe_http_content_length 0
probe_http_duration_seconds{phase="connect"} 0.002263513
probe_http_duration_seconds{phase="processing"} 0.196389853
probe_http_duration_seconds{phase="resolve"} 0.006723945
probe_http_duration_seconds{phase="tls"} 0
probe_http_duration_seconds{phase="transfer"} 2.6001e-05
probe_http_redirects 1
probe_http_ssl 0
probe_http_status_code 0
probe_http_version 0
probe_ip_protocol 4
probe_success 0

Here is the job definition:

  - job_name: 'blackbox'
    scrape_interval: 30s
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        - http://linux.org
    relabel_configs:
      - source_labels: [__address__]
        regex: '(.*)(:80)?'
        target_label: __param_target
      - source_labels: [__param_target]
        regex: '(.*)'
        target_label: instance
        replacement: '${1}'
      - source_labels: []
        regex: '.*'
        target_label: __address__
        replacement: 'blackbox:9115'

and the module definition:

modules:
  http_2xx:
    prober: http
    timeout: 15s
    http:
      valid_status_codes: []
      method: GET
bulkmoustache
  • 1,875
  • 3
  • 20
  • 24
  • Anything in the blackbox exporter logs? Where/how do you run the blackbo exporter? in a docker container? did you update/install the `ca-certificates` package? – Oliver Feb 19 '19 at 20:36
  • @Oliver nothing on the blackbox-exporter logs. This is running on docker. Yes, I also installed ca-certificates under /etc/ssl/certs but the result is the same. – bulkmoustache Feb 20 '19 at 09:03
  • It would seem like you're getting one redirect (as per `probe_http_redirects 1`, likely to `https://linux.org`) but then that request fails. The `probe_http_duration_seconds{phase="tls"} 0` seems to support that. The short `probe_duration_seconds` suggests that HTTPS connection is not even attempted (for me it takes about 1+ seconds to get a successful probe). Firewall? – Alin Sînpălean Feb 20 '19 at 12:53
  • Did you try the `curl` request from within the blackbox_exporter container? (using docker exec ... /bin/sh) – Oliver Feb 20 '19 at 16:02
  • @Oliver yes, I did. – bulkmoustache Feb 20 '19 at 16:28
  • Maybe we're all missing something obvious, but at first sight this seems to be a blackbox exporter bug. You should probably file an issue: https://github.com/prometheus/blackbox_exporter/issues – Alin Sînpălean Feb 21 '19 at 09:12
  • sorry guys! I had 2 blackbox instances running and was testing against the wrong one... Nonetheless I have another couple of domains which are failing with the same output as above. I'll let you know as soon as I figure out why. Thanks! – bulkmoustache Feb 22 '19 at 14:52

5 Answers5

11

If you append &debug=true you'll get some details on the error.

My guess is blackbox_exporter cannot connect because it defaults to ipv6 connections and that's not working.

In my case (same problem), the debug output says

level=error msg="Resolving target address" ip_protocol=ip6
level=error msg="Resolution with IP protocol failed (fallback_ip_protocol is false): err"
level=error msg="Error resolving address" err="address apple.com: no suitable address found"
level=error msg="Probe failed" duration_seconds=0.003648031

From the blackbox documentation:

  # The IP protocol of the HTTP probe (ip4, ip6).
  [ preferred_ip_protocol: <string> | default = "ip6" ]
  [ ip_protocol_fallback: <boolean> | default = true ]

So, if you modify your module configuration, it should work:

modules:
  http_2xx:
    prober: http
    timeout: 15s
    http:
      valid_status_codes: []
      method: GET
      preferred_ip_protocol: "ip4" # <---- !
Rafa
  • 1,397
  • 15
  • 21
  • + for ` &debug=true ` – Tag Wint Jan 15 '22 at 13:21
  • But if ip_protocol_fallback is true per default, shouldn't it just fallback to ipv4 in case of an ipv6 error? – Tim Jan 04 '23 at 21:00
  • @Tim good point! I guess it should...`¯\_(ツ)_/¯` I now see that in my error log it said `fallback_ip_protocol is false`. I have no idea why my fallback was set to `false`; I guess if it's properly set to `true` it will simply fallback to ipv4 as you said. – Rafa Jan 23 '23 at 16:38
3

In my error was: "x509: certificate signed by unknown authority"

Change option in http section

 tls_config:
    insecure_skip_verify: true
Max
  • 4,292
  • 1
  • 20
  • 14
3

Blackbox_exporter fetching endpoints routinely and provides crucial matrices regarding performance.

I have faced a similar issue. For me issue was typing error. But if you follow these simple debugging steps you can find your issue. right.

  1. curl -s "localhost:9115/probe?target=http://linux.org&module=http_2xx"&debug=true" for getting logs data blackbox logs
  2. for me iptables rules not allowing
dhyanio
  • 71
  • 3
1

In my case, I was to add in the valid_http_versions section the HTTP/1.0 entry. My endpoint was Odoo 13. The &debug=true suggested by @Rafa got me the information to realize that.

JuanMoreno
  • 2,498
  • 1
  • 25
  • 34
0

In my case I specified an invalid http version (HTTP 1.1, but a response from a URL was HTTP 1.0).

D. Suprun
  • 1
  • Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Feb 20 '23 at 03:07