36

I have a Node server running on AWS Elastic Beanstalk. One of my endpoints accepts a huge payload and the function itself is pretty slow and lengthy, and can take upwards of 10+ minutes.

Due to business requirements it must remain as a single HTTP POST and cannot be split up to be any smaller.

On larger calls I am getting a 504 GATEWAY TIMEOUT, always around the 60 second mark. I have tried toying with the timeout settings in Elastic Beanstalk Load Balancer section to no avail, it seems the longest timeout duration is 60 second anyway.

I did see one promising solution in the docs at https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/config-idle-timeout.html

To ensure that lengthy operations such as file uploads have time to complete, send at least 1 byte of data before each idle timeout period elapses

This sounds like exactly what I need, but I do not know how to accomplish

1) How can I "send at least 1 byte of data" from my Node app to ensure the session is kept alive and doesn't timeout after one minutes

Joshua Ohana
  • 5,613
  • 12
  • 56
  • 112

3 Answers3

50

504 Timeouts on ElasticBeanstalk environments can be resolved by settings your ELB policies. Updating Nginx timeout configurations may also be required.

  • ELB policies: Set the Idle Timeout of your Elastic Load Balancer to the value of your choice (defaults to 60s). To do this, create a .ebextensions folder in the root of your project. In this folder create another file with a .config file extension, and set the ELB Idle Timeout to the value of your choice (e.g. 300 seconds):

     option_settings:
   - namespace: aws:elb:policies
     option_name: ConnectionSettingIdleTimeout
     value: 300

    Or if you are using an application load balancer:

     option_settings:
   - namespace: aws:elbv2:loadbalancer
     option_name: IdleTimeout
     value: 300

  • Nginx configuration: Set Nginx the with the desired value of timeout: send_timeout, proxy_connect_timeout, proxy_read_timeout, proxy_Send_timeout all default to 60s (additional specification to check may be: client_header_timeout, client_body_timeout, keepalive_timeout). Default timeout values in Nginx are specified in the specifications and configurations files (e.g. the .config files under /etc/nginx). In the .ebextensions folder create a new file (e.g. 01-nginx-configuration.config or update in the .config file above), and append the following content inside the file (add or remove relevant settings according to the timeouts observed):

     files:
   "/etc/nginx/conf.d/nginx.custom.conf":
       mode: "644"
       owner: "root"
       group: "root"
       content: |
         client_header_timeout   300;
         client_body_timeout     300;
         send_timeout            300;
         proxy_connect_timeout   300;
         proxy_read_timeout      300;
         proxy_send_timeout      300;

 container_commands:
   01_restart_nginx:
     command: "sudo service nginx reload"

There are several more ways to add this configuration. Read more here and here.

Edit (06/11/2020): Some people indicated that the keep_alive 300; configuration crushed their server. Therefore, I've removed that from the nginx.custom.conf configuration.

asaf am
  • 1,345
  • 15
  • 28
  • This solution renders the environment not accessible, it happened to me, and while I was searching for a solution I found someone else who followed the instruction above and getting the exact same error as am getting. https://stackoverflow.com/questions/51421457/customizing-nginx-conf-on-elastic-beanstalk?noredirect=1&lq=1 – Bilal Oct 11 '19 at 18:55
  • @Bilal From the answer posted for the question in the link you added, it seems that the error is originated from another container command (`01_update_nginx`). I would like to help you but I'd need more information - maybe open a new question – asaf am Oct 12 '19 at 21:21
  • I had to comment out keepalive_timeout from the nginx config, it crashed my deployment. otherwise everything seemed to work, thank you – Mathieu J. Apr 24 '20 at 07:41
  • I created and uploaded this file, and it completely disconfigured my server. – Leon Grin Jun 10 '20 at 16:21
  • @LeonGrin I'm really sorry about it. It very much depends on what other configurations you have there. I've listed several timeout options - you can take what fits your requirements. Can you pls let us know if you were able to identify what has broken your server? Plus, please also notice MathieuJ. comment on commenting out the keepalive_timeout. – asaf am Jun 11 '20 at 17:13
  • these config didn't worked for me, i have tried adding these in my nginx.conf file in my .ebextension folder – Manak Bhardwaj Dec 17 '20 at 15:14
15

In order to update the nginx config on the Amazon Linux 2 platform I successfully managed to increase the timeout limits by extending the nginx config following these instructions: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/platforms-linux-extend.html

/.platform/nginx/conf.d/myconf.conf:

client_header_timeout   300;
client_body_timeout     300;
send_timeout            300;
proxy_connect_timeout   300;
proxy_read_timeout      300;
proxy_send_timeout      300;
staffe
  • 151
  • 2
  • 4
  • 1
    You just saved me :D. I was struggling to get my nginx config built for EBS linux 2 and this is exactly what I was missing. – Sam Munroe May 19 '22 at 02:14
  • We are using Elastic Beanstalk with Linux 2 platform and an application load balancer. When we include a .platform/nginx/conf.d/proxy.conf with the above commands we get an unknown directive (send_timeout) error when deploying. The file is UTF-8 with LF line endings. Any ideas whey we get this error? – Phil Jenson Mar 05 '23 at 16:16
  • worked for me, but had to edit a bit config (remove send_timeout, add fastcgi) fastcgi_read_timeout 300; proxy_read_timeout 300; proxy_connect_timeout 300; proxy_send_timeout 300; – Stefano Pochet Aug 06 '23 at 23:53
1

In my case error was happening because there was not enough memory for application container.

Increasing memory in containerDefinitions in Dockerrun.aws.json file solved my problem easily

{
  "AWSEBDockerrunVersion": 2,
  "volumes": [],
  "containerDefinitions": [
    {
      "name": "###",
      "image": "###",
      "essential": true,
      "memory": 521,         # formerly 128
...
Mustafa Candan
  • 76
  • 7