0

I have created a custom health indicator to check availability of an external service. This should be executed by a job every 15 minutes AFAIK. The problem is that I see on the logs that methodToCheckAvailability is being called multiple times within the same second. It is flooding our logs.

Custom Healh Indicator

import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.exception.ExceptionUtils;
import org.springframework.boot.actuate.health.Health;
import org.springframework.boot.actuate.health.HealthIndicator;
import org.springframework.boot.actuate.health.Status;
import org.springframework.stereotype.Component;

import java.util.Optional;
import java.util.function.Consumer;

import static java.util.Collections.emptyMap;
import static java.util.Collections.emptySet;
import static java.util.function.Predicate.not;
import static java.util.stream.Collectors.toList;

@Slf4j
@Component
@RequiredArgsConstructor
public class BananaHealthIndicator implements HealthIndicator {

    public static final Status PARTIAL_DOWN = new Status("PARTIAL_DOWN");

    @Override
    public Health health() {
        var builder = Health.up();

        var someValues = SomeService.getMap();


        doHealthCheck(builder, value, "ExternalService", ExternalService::methodToCheckAvailability);


        return builder.build();
    }

    private void doHealthCheck(Health.Builder builder, String value, String component, Consumer<String> healthCheck) {
        try {
            healthCheck.accept(value);
        } catch (Exception e) {
            log.atError().setCause(e)
                    .log("Health check failed for {}", component);

            builder.status(PARTIAL_DOWN)
                    .withDetail(value, component + ": " + ExceptionUtils.getMessage(e));
        }
    }
}

properties:

management:
  endpoint:
    health:
      status:
        http-mapping:
          partial_down: 500
      show-details: when_authorized
      probes:
        enabled: true
      group:
        banana:
          include: banana
          show-details: when_authorized
          show-components: when_authorized
          status:
            order: "partial_down,unknown,up"

/actuator/health response

{
    "status": "UP",
    "components": {
        "clientConfigServer": {
            "status": "UP",
            "details": {
                "propertySources": [
                    "configClient"
                ]
            }
        },
        "banana": {
            "status": "UP"
        },
        "livenessState": {
            "status": "UP"
        },
        "ping": {
            "status": "UP"
        },
        "readinessState": {
            "status": "UP"
        },
        "redis": {
            "status": "UP",
            "details": {
                "cluster_size": 2,
                "slots_up": 16384,
                "slots_fail": 0
            }
        },
        "refreshScope": {
            "status": "UP"
        }
    },
    "groups": [
        "banana",
        "liveness",
        "readiness"
    ]
}

When /actuator/health gets called, methodToCheckAvailability gets called within the created BananaHealthIndicator. My guess is maybe Kubernetes is calling health endpoint multiple times to check if the service is running and that is making methodToCheckAvailability get called multiple times.

Any help is appreciated

Mokz
  • 124
  • 1
  • 16
  • Did you try to configure `periodSeconds` in `livenessProbe` of your deployment spec? – Jude Niroshan Jul 27 '23 at 13:30
  • @JudeNiroshan I don't have that information. I am trying to make a fix from the code side. I have ran `/actuator/health/liveness` and its not running the `methodToCheckAvailability` method – Mokz Jul 27 '23 at 13:39
  • The ideal solution is to control the frequency of health endpoint calls. But if you don't want to follow that path, you can keep an AtomicBoolean which acts as the flag for whether to call the external service or not. But note that you will face problems when you have your service with more than 1 replica. – Jude Niroshan Jul 27 '23 at 13:58
  • @JudeNiroshan when calling /actuator/health it is running the check under `BananaHealthIndicator`. Do you know if that the expected behaviour? I thought the check would only execute when calling /actuator/health/banana specifically – Mokz Jul 27 '23 at 14:17
  • 1
    Yes that is expected behavior. As the overall health status is derived from all health indicators in the application. Hence it needs to call your custom `HealthIndicator` to have that included. – M. Deinum Jul 27 '23 at 14:17

0 Answers0