"failed_to_connect_to_backend" - 502 website temportatily down

Question

I have an issue where when attempting to access the website on a vm through load balancer I receive "Failed to connect to backend" get "502" and an error on the client side suggesting the server is down. I can see quite a few similar issues but not a fix for my own. To try and provide as much info as possible please see log below.

{
insertId: "mrasyug16eoo14"
jsonPayload: {
cacheId: "LHR-1bd33b0f"
@type: "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry"
statusDetails: "failed_to_connect_to_backend"
}
httpRequest: {
requestMethod: "GET"
requestUrl: "http://wedotest.uk/"
requestSize: "456"
status: 502
responseSize: "488"
userAgent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36"
remoteIp: "82.47.240.28"
cacheLookup: true
serverIp: "10.154.0.11"
latency: "9.012451s"
}
resource: {
type: "http_load_balancer"
labels: {
zone: "global"
project_id: "heroic-footing-308515"
target_proxy_name: "wedo-lb-target-proxy"
backend_service_name: "wedo-backend"
url_map_name: "wedo-lb"
forwarding_rule_name: "wedo-frontend-http"
}
}
timestamp: "2021-04-11T11:08:58.919182Z"
severity: "WARNING"
logName: "projects/heroic-footing-308515/logs/requests"
trace: "projects/heroic-footing-308515/traces/bf011eb0460e8a2bf092571e66236e76"
receiveTimestamp: "2021-04-11T11:09:08.668903322Z"
spanId: "270d670900c4811f"
}

I just wanted to keep this ticket updated. I have now recreated the entire conifg. new vm's, healthchecks, load balancer, the whole thing to see if I had just missed some config somewhere. I believe the moment where these issues begin is the moment where I change to "https" in the wordpress.

the health check reports "HTTP response: Moved Permanently" with "UNHEALTHY" state. The latest 502 log is now "failed_to_pick_backend"

Any thoughts/ideas would be welcomed and attempted.

Cheers

Answer 1

If the website stops working after switching to https, it would seem there is an issue with the certificate. Health check result indicates the website is redirecting HTTP traffic to HTTPS but does so incorrectly.
Certificate itself seems to be fine (only it has been set up with www).

Take a look at this tutorial and check if your certificate has been applied correctly.

If the instance is set up correctly but loadbalancer's IP still doesn't redirect to the website, you can go through this how-to and finally this troubleshooting guide to check your loadbalancer.

Answer 2

I just wanted to come back to outline some steps I took to solve my problem. The honest truth is I never found out what exactly was causing my issue but these combined steps did allow me to resolve.

Firstly there was nothing wrong with my SSL certificate and I did nothing to change it in order to fix my issue. My certificate was set up through google cloud managed SSL and applied to the front end of my load balancer. All the checks I did confirmed that the certificate was working perfectly.

I took 2 steps that I believe solved this and I have 2 pieces of advice;

SSL

Initially even though my SSL certificate had been applied the wordpress still needs some love and care. This isn't a hosting issue this is a wordpress issue. I added a wordpress SSL plugin such as "SSL insecure content fixer" or similar. This basically ensures that all of the wordpress code is HTTPS compliant and means you can load the site without any issues through HTTPS. However this was only one part of the problem.

Remember also when creating your google managed SSL certificate that you only use example.com do not use www.example.com. When adding the www. you will receive issues with your set up. I came across this issue the other day and thought it worth pointing out here.

Health check

As soon as you start making changes to the wordpress site. It does something, I am not sure what this something is but all of a sudden you get the errors/logs as shown in my original post. This could be 502's but I have also seen 302's. This is really what confused me. As soon as any change was made to wordpress backend, server was unavailable or a notice to that effect. I left my healthcheck path set to default "/" as advised. It was actually a piece of AWS content that saved me here. It recommended the load balancer path be set to "/wp-login.php". This actually works perfectly. You can also create a file and ping that instead. I chose the easier route. I am sure there are technical con's to pinging this particular file. Please feel free to share. However, would you believe I now no longer get these error messages. I have replicated a few times now and I am satisfied that healthchecks were part of the problem.

HTTP re-direct advice

A lot of documentation I found suggests adding redirects via ssh through the VM/Apache. I disagree with this approach. I have set up a load balancer and my load balancer can handle these re-directs. Again happy to open up the technical merits of doing this/not doing this to discussion. But I personally don't care. This was also a key step though, as by applying the re-direct I now had my site back, up and running and now always directing correctly to HTTPS and wp-admin is now available. Follow this guide.. https://cloud.google.com/load-balancing/docs/https/setting-up-http-https-redirect

Wordpress instance Google click to deploy Vs Bitnami click to deploy opinion

Its a personal opinion. I just found that even though google click to deploy is meant to be easier, I kept finding issues. I switched to the bitnami click to deploy, I am finding life to be much easier. Its arguably a bit more complex to configure in some areas but I am just not encountering as many problems. Add to this the bitnami documentation is so good. I would go with this instance of wp.

Summary

Ultimately, I think my issue was both a wordpress and health check issue throwing up different symptoms depending on the changes I made making it difficult for me to diagnose. I hope this is helpful to someone else later down the line.