0

I am building a lightweight proxy server using Tronado which logs the request+response into a Kafka queue. My client request -> Tornado proxy(I am developing) -> TensorFlow serving API back to client response time cannot be more than 20 milliseconds.

Tornado version 6.0.4 Kafka-python

With just the Tornado proxy server without logging to Kafka the response time is coming under 14ms, with Kafka logging the response time goes to 20000ms which is not acceptable. How can I reduce the response time with Kafka publish? I am a newbie with the Tornado framework.

Any help with this appreciated.

I did see this post here and the Kiel python library seems to have updates made 4 years back is that still good to use? OR do we have better ways now to achieve this?

Main Server code:

from tornado.web import Application, RequestHandler
from tornado.ioloop import IOLoop
from model_proxy_server.tfs_request import request
from tornado import gen
from model_proxy_server.model_request_response_publisher import publish_message
import json

class PostToTfs(RequestHandler):

@gen.coroutine
def post (self, *args, **kwargs):
    kafka_msg = {}

    model_url = 'http://localhost:8501' +  self.request.path

    response = yield request(self.request.body, model_url)

    resp = json.loads(response)
    yield self.write(resp)

    kafka_msg['request_url'] = model_url
    kafka_msg['request'] = json.loads(self.request.body)
    kafka_msg['response'] = resp
    msg = json.dumps(kafka_msg)

    # Publish the request and response to Kafka topic
    yield publish_message(msg, 'dev-ml-model-logs')


def make_app():
    # TODO Get to know on how to host this in PROD with common URL for this APP
    urls = [(r"/.*", PostToTfs)]
    return Application(urls, debug=True)

def main():
    app = make_app()
    app.listen(3000)
    IOLoop.instance().start()

if __name__ == '__main__':
    main()

Request Handler:

import tornado.httpclient
from tornado.ioloop import IOLoop
from tornado import gen
import tornado.options
import logging

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

@gen.coroutine
def json_fetch(http_client, body, model_url):
    try:
        response = yield http_client.fetch(f"{model_url}", method='POST', body=body)
    except Exception as e:

        logging.info("Exception on http_client fetch from url")
        logging.error("Error: " + str(e))

    raise gen.Return(response)

@gen.coroutine
def request(body, model_url):

    http_client = tornado.httpclient.AsyncHTTPClient()
    http_response = yield json_fetch(http_client, body, model_url)
    return http_response.body

if __name__ == "__main__":
    tornado.options.parse_command_line()
    IOLoop.instance().run_sync(request)

Kafka Publishers:

from kafka import KafkaProducer
from tornado import gen
import logging

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

@gen.coroutine
def publish_message(msg, topic_name):
    producer_instance = _kafka_connect()
    msg_bytes = bytes(msg, encoding='utf-8')

    try:
        yield producer_instance.send(f'{topic_name}', msg_bytes)
        logging.info("Message published successfully")
    except Exception as e:
        logging.info("Exception in publishing message")
        logging.error(str(e))

@gen.coroutine
def _kafka_connect():

    try:
         producer = yield KafkaProducer(bootstrap_servers='ec2prdkafka01:9093', api_version=(0, 10))

    except Exception as e:
        logging.info("Exception while connecting to Kafka")
        logging.error(str(e))

    return producer
Krish
  • 390
  • 4
  • 15

1 Answers1

0

Okay, so this is how I resolved the issue and brought the response time from 20000ms of my proxy server to 20ms.

The Kafka Publishers creating a connection to Kafka is being done every time that module is called, which was the bottleneck on the response time.

So I established the connection to Kafka while starting the proxy server like below

def make_app():
    producer = KafkaProducer(bootstrap_servers='ec2prdkafka01:9093',api_version=(0, 10))
    urls = [(r"/.*", PostToTfs), dict(producer=producer)]
    return Application(urls, debug=True)

Once its passed here on we need to initialize the producer so it can be used directly

def post (self, *args, **kwargs):

def initilize(self, producer):
    self.producer = producer

Hope this helps someone looking for this kind of info.

Krish
  • 390
  • 4
  • 15