21

Our ELB is intermittently failing health checks on instances and throws errors "Environment health has transitioned from Ok to Warning. 2 out of 2 instances are impacted. See instance health for details" and after a minute it throws "Environment health has transitioned from Warning to Severe. ELB health is failing or not available for all instances"

It happens everyday on same time. Also I suspect that it happens whenever our CRON runs but I am not sure as it doesn't happens on other ELB instances.

I would like to know the root cause of this kind of problem as I searched through the forum and everyone had some different situations and cannot figure out what problem it would be for us.

This is our Production instance.

halfer
  • 19,824
  • 17
  • 99
  • 186
Imran Sheikh
  • 341
  • 1
  • 2
  • 6

4 Answers4

43

Application load balancer(a part of ELB) looking default "/" endpoint for health check and ELB gets 404 status code so ElasticBeanstalk instances turn red. You should create an endpoint for health check and send 200 status code. İf you want to use another path you should modify load balancer for example: "default 80 HTTP /" to "default 80 HTTP /healthCheck"

Gökhan Ayhan
  • 1,184
  • 11
  • 12
  • 4
    This answer helped me so much! I spent at least 10 hours trying different (increasingly desperate) things to figure out my severe health status and creating a 200 response at "/" fixed the issue. – person13 Mar 17 '20 at 03:48
  • Please, where should I put that configuration? could you guys share an example or link ? – ray Mar 21 '22 at 16:46
  • @ray in your Beanstalk app environment > configuration > load balancer > processes > edit – ServletException Aug 12 '22 at 08:45
  • But when i deploy the previous version the health check passes! Why would it be happening only for new version? –  Sep 12 '22 at 08:49
3

I had the same problem as you and followed Gökhan Ayhan's approach.

Modifying the load balancer from "default 80 HTTP /" to "default 80 HTTP /healthCheck" did not help, but you can also set the expected response code. I set it to 404 and it works, but this is not how this process is intended. Nevertheless I am only using a Load Balancer to get SSL on my Application, so it does not bother me.

a Load balancer is intended to balance out the load on the connected nodes, therefore it needs to be aware of the status of the individual nodes. If i understood correctly, this is done by the Health Checker process. You can configure that process as described above, or you give the balancer an actual endpoint to evaluate the health properly.

anderwald
  • 139
  • 3
  • 9
  • But when i deploy the previous version the health check passes! Why would it be happening only for new version? –  Sep 12 '22 at 08:50
2

I have created an end point in my spring boot application as follows and deployed app to AWS Elastic Beanstalk. But still ELB health check fails.

package com.sme.controller;

import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class BaseController {
    @GetMapping("/")
    public ResponseEntity getServiceName(){
        ResponseEntity responseEntity = new ResponseEntity(HttpStatus.OK);
        return responseEntity;
    }

}

Since I've configured spring security, does it affect on this ELB health check? Please check the following security configurations.

  @Override
protected void configure(HttpSecurity http) throws Exception {
    http.cors().and()
            .csrf().disable()
            .exceptionHandling().authenticationEntryPoint(unauthorizedHandler).and()
            .sessionManagement()
            .sessionCreationPolicy(SessionCreationPolicy.STATELESS)  // Since this is REST API no need to hold the sessions
            .and()
            .authorizeRequests()
            //.antMatchers(SecurityConstants.SIGN_UP_URLS).permitAll()
            .antMatchers("/**").permitAll()
            .anyRequest().authenticated();

    http.addFilterBefore(jwtAuthenticationFilter(), UsernamePasswordAuthenticationFilter.class);
}
Ashoka
  • 41
  • 1
  • 1
  • 5
  • If you have a new question, please ask it by clicking the [Ask Question](https://stackoverflow.com/questions/ask) button. Include a link to this question if it helps provide context. - [From Review](/review/late-answers/29880019) – m4n0 Sep 21 '21 at 16:49
  • But when i deploy the previous version the health check passes! Why would it be happening only for new version? –  Sep 12 '22 at 08:50
0

create an endpoint for health check and send 200 status code. İf you want to use another path you should modify load balancer. example: "default 80 HTTP /" to default 80 HTTP "/ping"

Sumit Kumar
  • 1
  • Please provide a detailed explanation to your answer, in order for the next user to understand your answer better. – Elydasian Jul 22 '21 at 06:32
  • But when i deploy the previous version the health check passes! Why would it be happening only for new version? –  Sep 12 '22 at 08:50