4

I'm currently facing some random 503-errors on my Varnish and I can't figure out where they come from. The FetchError is the following: HTC Status -1, which means HTC_S_EOF see here

Somehow, the bereq instantly failes and there is not even a single request sent to the backend server.

Do you guys have an idea how to locate and fix this error? I'm using the latest Varnish 5.2 Version available, based on Alpine.

Thank you in advance!

Here is a docker logs file of the corresponding bereq:

{"log":"-   Begin          bereq 5145026 pass\n","stream":"stdout","time":"2017-12-11T21:38:52.636374999Z"}
{"log":"-   Timestamp      Start: 1513028321.532661 0.000000 0.000000\n","stream":"stdout","time":"2017-12-11T21:38:52.636377235Z"}
{"log":"-   BereqMethod    POST\n","stream":"stdout","time":"2017-12-11T21:38:52.63637951Z"}
{"log":"-   BereqURL       /wp-admin/post.php\n","stream":"stdout","time":"2017-12-11T21:38:52.636384624Z"}
{"log":"-   BereqProtocol  HTTP/1.1\n","stream":"stdout","time":"2017-12-11T21:38:52.636387074Z"}
{"log":"-   BereqHeader    Host: www.example.com\n","stream":"stdout","time":"2017-12-11T21:38:52.636389318Z"}
{"log":"-   BereqHeader    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0\n","stream":"stdout","time":"2017-12-11T21:38:52.636391598Z"}
{"log":"-   BereqHeader    Content-Length: 6225\n","stream":"stdout","time":"2017-12-11T21:38:52.636393963Z"}
{"log":"-   BereqHeader    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\n","stream":"stdout","time":"2017-12-11T21:38:52.636396303Z"}
{"log":"-   BereqHeader    Accept-Encoding: gzip, deflate, br\n","stream":"stdout","time":"2017-12-11T21:38:52.636398632Z"}
{"log":"-   BereqHeader    Accept-Language: de\n","stream":"stdout","time":"2017-12-11T21:38:52.636400931Z"}
{"log":"-   BereqHeader    Content-Type: application/x-www-form-urlencoded\n","stream":"stdout","time":"2017-12-11T21:38:52.636403196Z"}
{"log":"-   BereqHeader    Cookie: scrollPositionX=undefined; scrollPositionY=undefined; wp-saving-post=45779-check; wordpress_sec_2bdc2b00e491d84e1c4be3840dfdb408=Joerg+Maire%7C1513151186%7CUhvUAmySXCqogpTK2a3VXswRI27TT1uZ4JyFuE
c2fdb1687cdfe93d1e69ee67280286479cbbec709af1\n","stream":"stdout","time":"2017-12-11T21:38:52.636406516Z"}
{"log":"-   BereqHeader    Dnt: 1\n","stream":"stdout","time":"2017-12-11T21:38:52.636409503Z"}
{"log":"-   BereqHeader    Referer: https://www.example.com/wp-admin/post.php?post=45779\u0026action=edit\n","stream":"stdout","time":"2017-12-11T21:38:52.636411823Z"}
{"log":"-   BereqHeader    Upgrade-Insecure-Requests: 1\n","stream":"stdout","time":"2017-12-11T21:38:52.636414276Z"}
{"log":"-   BereqHeader    X-Forwarded-Host: www.example.com\n","stream":"stdout","time":"2017-12-11T21:38:52.636416562Z"}
{"log":"-   BereqHeader    X-Forwarded-Port: 443\n","stream":"stdout","time":"2017-12-11T21:38:52.636431001Z"}
{"log":"-   BereqHeader    X-Forwarded-Proto: https\n","stream":"stdout","time":"2017-12-11T21:38:52.636433369Z"}
{"log":"-   BereqHeader    X-Forwarded-Server: lb-proxy-1\n","stream":"stdout","time":"2017-12-11T21:38:52.636435511Z"}
{"log":"-   BereqHeader    X-Real-Ip: XX.XX.XX.XXX\n","stream":"stdout","time":"2017-12-11T21:38:52.636437721Z"}
{"log":"-   BereqHeader    X-Forwarded-For: XX.XX.XX.XXX, 10.42.176.203\n","stream":"stdout","time":"2017-12-11T21:38:52.636439886Z"}
{"log":"-   BereqHeader    X-Varnish: 5145027\n","stream":"stdout","time":"2017-12-11T21:38:52.636442148Z"}
{"log":"-   VCL_call       BACKEND_FETCH\n","stream":"stdout","time":"2017-12-11T21:38:52.636444268Z"}
{"log":"-   VCL_return     fetch\n","stream":"stdout","time":"2017-12-11T21:38:52.636446498Z"}
{"log":"-   BackendOpen    29 boot.default XX.XX.XX.XXX 80 10.42.160.172 55142\n","stream":"stdout","time":"2017-12-11T21:38:52.636448594Z"}
{"log":"-   BackendStart   XX.XX.XX.XXX 80\n","stream":"stdout","time":"2017-12-11T21:38:52.636450886Z"}
{"log":"-   Timestamp      Bereq: 1513028321.551239 0.018578 0.018578\n","stream":"stdout","time":"2017-12-11T21:38:52.636453022Z"}
{"log":"-   FetchError     HTC status -1\n","stream":"stdout","time":"2017-12-11T21:38:52.636455259Z"}
{"log":"-   BackendClose   29 boot.default\n","stream":"stdout","time":"2017-12-11T21:38:52.636457381Z"}
{"log":"-   Timestamp      Beresp: 1513028332.632241 11.099580 11.081002\n","stream":"stdout","time":"2017-12-11T21:38:52.636459871Z"}
{"log":"-   Timestamp      Error: 1513028332.632249 11.099587 0.000008\n","stream":"stdout","time":"2017-12-11T21:38:52.636462044Z"}
{"log":"-   BerespProtocol HTTP/1.1\n","stream":"stdout","time":"2017-12-11T21:38:52.636464306Z"}
{"log":"-   BerespStatus   503\n","stream":"stdout","time":"2017-12-11T21:38:52.636466426Z"}
{"log":"-   BerespReason   Service Unavailable\n","stream":"stdout","time":"2017-12-11T21:38:52.636468611Z"}
{"log":"-   BerespReason   Backend fetch failed\n","stream":"stdout","time":"2017-12-11T21:38:52.636472903Z"}
{"log":"-   BerespHeader   Date: Mon, 11 Dec 2017 21:38:52 GMT\n","stream":"stdout","time":"2017-12-11T21:38:52.636475145Z"}
{"log":"-   BerespHeader   Server: Varnish\n","stream":"stdout","time":"2017-12-11T21:38:52.636477305Z"}
{"log":"-   VCL_call       BACKEND_ERROR\n","stream":"stdout","time":"2017-12-11T21:38:52.636479423Z"}
{"log":"-   BerespHeader   Content-Type: text/html; charset=utf-8\n","stream":"stdout","time":"2017-12-11T21:38:52.636481577Z"}
{"log":"-   BerespHeader   Retry-After: 5\n","stream":"stdout","time":"2017-12-11T21:38:52.636483718Z"}
{"log":"-   VCL_return     deliver\n","stream":"stdout","time":"2017-12-11T21:38:52.636485834Z"}
{"log":"-   Storage        malloc Transient\n","stream":"stdout","time":"2017-12-11T21:38:52.636487925Z"}
{"log":"-   ObjProtocol    HTTP/1.1\n","stream":"stdout","time":"2017-12-11T21:38:52.63649004Z"}
{"log":"-   ObjStatus      503\n","stream":"stdout","time":"2017-12-11T21:38:52.63649216Z"}
{"log":"-   ObjReason      Backend fetch failed\n","stream":"stdout","time":"2017-12-11T21:38:52.636494252Z"}
{"log":"-   ObjHeader      Date: Mon, 11 Dec 2017 21:38:52 GMT\n","stream":"stdout","time":"2017-12-11T21:38:52.63649637Z"}
{"log":"-   ObjHeader      Server: Varnish\n","stream":"stdout","time":"2017-12-11T21:38:52.636498539Z"}
{"log":"-   ObjHeader      Content-Type: text/html; charset=utf-8\n","stream":"stdout","time":"2017-12-11T21:38:52.63650064Z"}
{"log":"-   ObjHeader      Retry-After: 5\n","stream":"stdout","time":"2017-12-11T21:38:52.63650281Z"}
{"log":"-   Length         284\n","stream":"stdout","time":"2017-12-11T21:38:52.636525843Z"}
{"log":"-   BereqAcct      1770 6225 7995 0 0 0\n","stream":"stdout","time":"2017-12-11T21:38:52.636528516Z"}
{"log":"-   End            \n","stream":"stdout","time":"2017-12-11T21:38:52.636530668Z"}
Hermsi1337
  • 241
  • 2
  • 9
  • Having the issue here. Did you ever solve this?! – George Mar 21 '19 at 10:35
  • Hey @George, in which environment are you running the Varnish-Instance? I did not experience this error again since I've moved away from Rancher-Cattle to Kubernetes. – Hermsi1337 Mar 31 '19 at 13:07
  • Hi, unfortunately not the same environment. AWS Ubuntu custom install instance... Thanks for replying! – George Mar 31 '19 at 18:53
  • argh, I'm sorry that I can't give you any advice. But I'd really love to hear if you located and solved the issue. I failed horribly on investigating the whole thing. Good luck! – Hermsi1337 Mar 31 '19 at 19:06
  • No luck still. Looking into trying to figure out if Apache is at fault but can confirm no entries in Apache logs... – George Mar 31 '19 at 20:03

1 Answers1

0

Unfortunately, that "problem" has not reoccured since I've moved away from my Rancher-Cattle-Cluster to Kubernetes. I guess it was a kind of networking-issue.

Hermsi1337
  • 241
  • 2
  • 9