0

It is hard to fully describe our problem. However, I will try my best to provide full information. I spent many hours and still stuck. Hope that anyone here can help me.

Currently, our container randomly hangs on response to network request.

I have a nodejs application running at port 8082, this port is also exposed to the host machine. I created a bridge network to link it with a nginx proxy container.

I wrote a script to do followings

  • stop the nodejs container
  • run a new nodejs container (from same or newer image)
  • remove the network
  • stop the nginx container
  • create new network
  • link nodejs to the new network
  • run another nginx container using with the new network

Sometimes, this design works perfectly. However, sometimes it is broken randomly. The break happens right after the container starts, and once it is broken, it is broken for all requests.

Here is how it is broken

  • From the host machine, if I run curl localhost:8082, it returns Found. Redirecting to /en. This is correct.
  • From the host machine, if I run curl localhost:8082/en, it hangs a long time and returns curl: (52) Empty reply from server.
  • From the host machine, I run docker exec 041fe684edfb curl localhost:8082/en (041fe684edfb is container name), it returns 99% of the response and hangs at printing <script defer src="https://d3l80sdjn9d1ye.cloudfront.net/bundles/admin-car-calendar~admin-host~admin-orders~admin-station~admin-stations~author-postView~holder-car-c~b1a1af4b.05ea0cbf23227e7a323a.js"></script>, then returns curl: (18) transfer closed with 1 bytes remaining to read.
  • If I run docker exec -it 041fe684edfb bash, then run curl localhost:8082/en in the container terminal, it prints response, hangs but at a different (latter) position
    <script defer src="https://d3l80sdjn9d1ye.cloudfront.net/bundles/runtime.93923d84c7f89f7844ae.js"></script>
    <script defer src="https://d3l80sdjn9d1ye.cloudfront.net/bundles/admin-car-calendar~admin-host~admin-orders~admin-station~admin-stations~author-postView~holder-car-c~b1a1af4b.05ea0cbf23227e7a323a.js"></script>
    <script defer src="https://d3l80sdjn9d1ye.cloudfront.net/bundles/admin-car-calendar~admin-car-orders~admin-cars~holder-car-calendar~holder-car-simple-calendar~holder~9dc2a71c.5714564e356df216ebb8.js"></script>
    <script defer src="https://d3l80sdjn9d1ye.cloudfront.net/bundles/user-vs-home.dc7ac409e63e95c891ec.js"></script>
    <script defer src="https://d3l80sdjn9d1ye.cloudfront.net/bundles/vendors.284ffc3f329d46200747.js"></script>
    <script defer src="https://d3l80sdjn9d1ye.cloudfront.net/bundles/client.16b9bdc0c6df3c59afb6.js"></script>
    <script defer src="https://d3l80sdjn9d1ye.cloudfront.net/bundles/styles.91c5d85ab779777de531.js"></script>
</body>

</html> is not printed out.

When I checked my nodejs app log, the request fully processed the request (until </html>).

Further information

  • If I restart the container with docker restart 041fe684edfb, the problem disappears
  • If I run the script commands one by one, the problem disappears
  • My nodejs app connects to external mongo and redis servers at 10.0.81.135 (AWS EC2 private IP). Initially, these servers run locally in other containers, there were absolutely no problem at all. After I migrated them to a separated server, the problem happened.
  • Chunk size is 221773. Nodejs app log prints this size and curl interpret the Content-Length header correctly.
  • At the end, the nodejs throw this error several times (6 times), then exits

12:07:00
AbortError: Redis connection lost and command aborted. It might have been processed.

12:07:00
at RedisClient.flush_and_error (/carstay/node_modules/redis/index.js:362:23)

12:07:00
at RedisClient.connection_gone (/carstay/node_modules/redis/index.js:664:14)

12:07:00
at RedisClient.on_error (/carstay/node_modules/redis/index.js:410:10)

12:07:00
at Socket.<anonymous> (/carstay/node_modules/redis/index.js:279:14)

12:07:00
at Socket.emit (events.js:223:5)

12:07:00
at Socket.EventEmitter.emit (domain.js:475:20)

12:07:00
at emitErrorNT (internal/streams/destroy.js:92:8)

12:07:00
at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)

12:07:00
at processTicksAndRejections (internal/process/task_queues.js:81:21)

12:07:00
AbortError: Redis connection lost and command aborted. It might have been processed.

12:07:00
at RedisClient.flush_and_error (/carstay/node_modules/redis/index.js:362:23)

12:07:00
at RedisClient.connection_gone (/carstay/node_modules/redis/index.js:664:14)

12:07:00
at RedisClient.on_error (/carstay/node_modules/redis/index.js:410:10)

12:07:00
at Socket.<anonymous> (/carstay/node_modules/redis/index.js:279:14)

12:07:00
at Socket.emit (events.js:223:5)

12:07:00
at Socket.EventEmitter.emit (domain.js:475:20)

12:07:00
at emitErrorNT (internal/streams/destroy.js:92:8)

12:07:00
at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)

12:07:00
at processTicksAndRejections (internal/process/task_queues.js:81:21)
Sang
  • 101
  • 4

1 Answers1

0

Ran into a similar issue and found this article: https://medium.com/grano/the-fault-in-our-responses-bytes-remaining-to-read-91efe0553676

A custom error handler that calls next() although the response had been sent already caused this as the last next() is the default handler from Express that will disconnect the socket immediately. Redis in the meantime may still try to save your session and die.

Try stepping through your code and see if a request is served multiple times under certain circumstances, e.g. if an error is thrown at some point. Even res.send() may throw an error, e.g. if you try to send data that can't be serializeed.

Thiemo Müller
  • 1