31

I have a full web application using NodeJS, MongoDB (Mongoose as the driver) and ExpressJS.

The project works perfectly on my local machine. Today I decided to move everything to production. I'm using Google App Engine to host my application, and Compose (formally MongoHQ) to host my database.

App Engine servers my application perfectly, although my API does not seem to work. My API is served from example.com/api, and each request (GET, POST, DELETE and PUT) all returns a 502 (Bad Gateway) error.

I tried running my application on my local machine while connected to my remote MongoDB database and that worked perfectly fine. So it must be a problem with App Engine or NodeJS, not with MongoDB.

I have tried checking all error logs within Google Cloud, although there are no errors.

Why is App Engine/NodeJS serving my application's static content perfectly fine, although not allowing any requests to my API?

Fizzix
  • 23,679
  • 38
  • 110
  • 176

4 Answers4

33

just make sure that your server listens on 8080 port https://cloud.google.com/appengine/docs/flexible/custom-runtimes/build#listen_to_port_8080

Deepak Patil
  • 3,419
  • 3
  • 25
  • 22
  • 1
    That was exactly the problem I was having! Thanks – Joan Salvatella Dec 06 '19 at 13:19
  • 1
    Finally, up and running! Would have been useful if google's nginx had returned some hint about this, 502 could be anything when you're unfamiliar with the platform. Also the docs to get a node serve up and running are so many and scattered – andersand Jan 07 '20 at 18:02
23

502 Bad Gateway is usually an error on the Nginx side. Unfortunately those logs are not surfaced to Cloud Logging, yet.

A lot of times the problem is that your HTTP packets are too big for the buffers or something similar. The way you can see the nginx log is something like this:

  • Use just 1 VM. This isn't strictly necessary, but a lot of times it makes it easier to debug your application if you know that your requests on the one machine. You can accomplish this by adding this to your app.yaml:

manual_scaling: instances: 1 then re-deploy

  • Switch the VM from "Google owned" to self-managed. This can be done in the Cloud Console. Go to Compute Engine, instances, click on the instance name that matches the App Engine version, and you should see an option to switch it to self-managed.

  • gcloud compute ssh <instance name> to SSH to the machine

  • docker ps to see your running containers. Look for the container named nginx and grab its id.

  • Once you have a container ID, you should be able to docker exec -it <container id> -- cat /var/log/nginx/error.log. You might want to ls that whole log directory.

You will likely see an error there which will be a bigger hint as to what's going wrong.

I know this is way more complicated than it should be :-\ If you have any problems with the steps above, leave a comment. If you do find an error and you're not sure what to do with it, also leave a comment.

Bill Prin
  • 2,498
  • 1
  • 20
  • 27
  • 1
    Well here's a plot twist. Just adding `manual_scaling: instances: 5` to my `app.yaml` file and deploying it fixed the entire problem. Any idea why that would be? – Fizzix Jun 25 '16 at 01:09
  • Although, when trying to access one of my more advanced APIs, such as image uploading, it still fails. This is probably because I am sending a large Base64 string to my API. Is there a way to increase how much data can be sent to my servers API? – Fizzix Jun 25 '16 at 01:35
  • Wow, just realized I meant manual_scaling: instances set to 1, not 5! That way you know for sure what request you're looking at. I would still follow the rest of those steps to see if you can't find the error logs. Anyway, if your packets are too big for the buffer, there is no way to directly increase so your best bet is to figure out how to break the data you need to transmit into multiple packets. – Bill Prin Jun 27 '16 at 17:33
  • ok.. I did all this for my project https://vplan-147418.appspot.com. See https://gist.github.com/sureshvv/4d7adbd495ba17fb4c468a120f55ec74 – sureshvv Nov 07 '16 at 09:15
  • 1
    awesome explanation. But you can debug inside the google cloud web site using their terminals. Follow the tutorial "Debug an Instance" from google : https://cloud.google.com/appengine/docs/flexible/nodejs/debugging-an-instance#connecting_to_the_instance – Christophe Chenel Jan 16 '21 at 21:09
5

I had the same problem, I was getting "nginx 502 bad gateway" error on GAE standard environment. There are many reasons for this but I finally got it working. Try these:

1) Run the app on the correct port. Google will set the PORT environment variable. I was running on port 8080, in the stackdriver logs I was getting this warning:

App is listening on port 8080. We recommend your app listen on the port defined by the PORT environment variable to take advantage of an NGINX layer on port 8080.

The code below gets the port from environment, if PORT is set otherwise defaults to 8080:

const PORT = process.env.PORT || 8080;

2) Go to google cloud console -> logging -> logs viewer. Select Google App Engine and then your service from the down and check you logs. Are you getting the requests at all or does it look like the requests do not react to your server. In my case, I was not getting them even after I fixed the port:

2020-03-02 21:50:07 backend[20200302t232314] Server listening on port 8081! 2020-03-02 21:50:08 backend[20200302t232314] "GET /create-user HTTP/1.1" 502

Fix any error if it looks like your application is failing to start, throwing exceptions etc..

3) Don't pass an IP when you are running your server. It seems Google runs the app at a pre-defined IP address and do not want you to modify it:

server.listen(PORT);

4) Don't try to run on https! Google is running an nginx server in front of your app and it is handling the SSL and redirects to your app over http. You can use the environment variable NODE_ENV(it is set to "production" in GAE environment) to run on http on production and https elsewhere, like this:

let https = require('https');
let http = require('http');

if (process.env.NODE_ENV == "production") {
    http.createServer(app).listen(PORT, function () {
        console.log(`Server listening on port ${PORT}!`)
    });
} else {
    https.createServer({
        key: fs.readFileSync('host.key'),
        cert: fs.readFileSync('host.cert')
    }, app).listen(PORT, function () {
        console.log(`Server listening on port ${PORT}!`)
    });
}

5) I didn't need to set any handlers in my yaml file, it might be causing errors for you if you have incorrect configuration. My yaml file is pretty straightforward:

runtime: nodejs12
env: standard
instance_class: F1
Caner
  • 57,267
  • 35
  • 174
  • 180
0

If 502s are rare (e.g. <1% of requests), this can be a result of a race condition between nginx reusing the connection and your app closing the connection, because your app has a similar or smaller keepAliveTimeout than that configured by Google in their nginx server.

You can fix this by setting your server.keepAliveTimeout to 700 seconds (or at least 650 seconds, plus a good buffer for network latency). For example:

const server = http.createServer({ keepAliveTimeout: 700_000 }, app)
server.listen(port, () => console.log('Server listening'));

There's more detailed analysis as to why this happens in https://stackoverflow.com/a/76044099.

domdomegg
  • 1,498
  • 11
  • 20