How to get MQTT client to work on Appengine (python)

Question

Trying to run a simple MQTT client (not the broker) on a paid GAE app. However a on_connect callback never occurs in the following:

worker.py

import webapp2
import paho.mqtt.client as paho

class WorkerHandler(webapp2.RequestHandler):

    def on_subscribe(client, userdata, mid, granted_qos):
        print("Subscribed: "+str(mid)+" "+str(granted_qos))

    def on_message(client, userdata, msg):
        print(msg.topic+" "+str(msg.qos)+" "+str(msg.payload))

    def on_connect(client, userdata, flags, rc):
        client.subscribe("$SYS/#")
        print "Subscribed to wildcard"

    def get(self):
        client = paho.Client()
        client.on_connect = self.on_connect
        client.on_subscribe = self.on_subscribe
        client.on_message = self.on_message
        client.connect("iot.eclipse.org")
        print "connected to broker"
        client.loop_forever()

app = webapp2.WSGIApplication([
    (r'/_ah/start', WorkerHandler),
])

In the dev environment it fails silently with just a message after a minute or so

INFO     2017-04-04 01:51:40,958 module.py:813] worker: "GET /_ah/start HTTP/1.1" 500 220
INFO     2017-04-04 01:51:41,869 module.py:1759] New instance for module "worker" serving on: http://localhost:8080

connected to broker
WARNING  2017-04-04 01:52:10,860 module.py:1927] All instances may not have restarted

This is configured as a "backend"/service and the yaml looks like this:

worker.yaml

service: worker
runtime: python27
api_version: 1
threadsafe: true

instance_class: B8

manual_scaling:
  instances: 1

handlers:
# If a service has an _ah/start handler, it should be listed first.
- url: /_ah/start
  script: worker.app

Note: In the dev environment, socket.py is being imported directly from python install .../2.7/lib/python2.7/socket.py

Dan Cornilescu · Answer 1 · 2017-04-04T02:11:50.517

You're attempting to run a standalone script as your GAE app worker service. It won't work.

Your worker.py needs to contain a WSGI application called app to match your worker.yaml configuration.

From the script row in the Handlers element table:

A script: directive must be a python import path, for example, package.module.app that points to a WSGI application. The last component of a script: directive using a Python module path is the name of a global variable in the module: that variable must be a WSGI app, and is usually called app by convention.

The error you get most likely indicates that the attempt to start the worker's module WSGI app fails.

After your update to bring back the WSGI app the reason for the error message becomes even clearer: the WorkerHandler.get() method doesn't respond to the /_ah/start request, because it's stuck in client.loop_forever().

From Startup:

Each service instance is created in response to a start request, which is an empty HTTP GET request to /_ah/start. App Engine sends this request to bring an instance into existence; users cannot send a request to /_ah/start. Manual and basic scaling instances must respond to the start request before they can handle another request.

...

When an instance responds to the /_ah/start request with an HTTP status code of 200–299 or 404, it is considered to have successfully started and can handle additional requests. Otherwise, App Engine terminates the instance. Manual scaling instances are restarted immediately, while basic scaling instances are restarted only when needed for serving traffic.

Yes it was originally running as an app. The script still runs since it doesn't need any http handlers. I've changed it back to how it was with no change in results. — Fakeer, Apr 04 '17 at 01:54
But at least now you see your code executing to some degree. You just need to somehow respond with an acceptable return code before entering the loop. Not sure if it's possible on the same request, you might need to do it differently. — Dan Cornilescu, Apr 04 '17 at 02:16

How to get MQTT client to work on Appengine (python)

1 Answers1