3

I'm looking for an easy and dependency-light way of wrapping a python library to expose it over:

a) The network, either via HTTP or some other custom protocol, it doesn't matter that much, and encryption isn't required. b) The local machine, the main purpose here is to avoid library import overhead, ideally, this would happen via an efficient mechanism ala pipes or shared memory to minimize the number of data copies and [de]serialization.

It seems like it's an easy enough job to just create a class that runs constantly, with e.g. an HTTP interface mirroring the library functionality and returning e.g. pickled objects corresponding to the answers. But getting it to work efficiently and cover various edge-cases seems tedious and I'm wondering if there's a better way to do this, ideally one that's built into python itself.

Ray seems to have some functionality for this using Actors, but it seems rather heavy-weight and prone to fail when being installed, so I'm curios what alternatives exist.

Also, might be too much of a "library question", if you think it's better suited for another stack exchange website please tell me which and I'll remove it from here.

George
  • 3,521
  • 4
  • 30
  • 75
  • 2
    what did you try? How about modules like [FastAPI](https://fastapi.tiangolo.com/) or [Hug](https://www.hug.rest/). And if you expose using HTTP then you can use it also locally. But if want to run it locally then problem can be access from network if your Internet Provider changes your IP every 24h and it blocks some ports for your security. – furas Feb 10 '21 at 23:58
  • 1
    @furas These are HTTP and, the main problem is that I have to re-deinfe an API over all my function. However these APIs should be accessed from python only, so it seems reasonable for a library to exist that just takes all the functions the library exports and auto-generates the API, without the user (me) having to handle re-writing every endpoint (coupled with all the data-wrangling required and having to do data transformations on the caller side when sending and receiving stuff from those endpoints) – George Feb 11 '21 at 01:09

2 Answers2

2

Your best options are Flask or FASTAPi. Both are lightweight and very resilient web frameworks, that are also pretty easy to start working with (it may be a matter of adding one decorator to your function to achieve your goal). You may also couple your API with Swagger UI, to have visual interaction with your API resources.

P.S. Ray actors has nothing to do with your request (they are simply statefull objects meant to be run/used in a distributed fashion).

diman82
  • 702
  • 8
  • 11
  • > P.S. Ray actors has nothing to do with your request (they are simply statefull objects meant to be run/used in a distributed fashion). -- How comes? They seem to behave like separate processes, not objects, from a resource consumption perspective. Which is exactly what I want, something that behaves like a python object/module but from a resource perspective is essentially a separate process – George Feb 25 '21 at 13:51
  • @George You didn't mention (in your original question) anything regarding running some code in a separate process. Also, I think you're mixing some concepts: ray actors are first of all - a development paradigm in the wider Ray framework, with a class API that you can extend/implement. Re resources allocation you're right - with each new ray actor that is instantiated, a new worker is created, and each worker is allocated a separate process. – diman82 Mar 01 '21 at 20:28
1

A relatively simple and lightweight solution is to use RPC and expose your existing functions.

For example:

server:

from xmlrpc.server import SimpleXMLRPCServer

def is_even(n):
    return n % 2 == 0

server = SimpleXMLRPCServer(("localhost", 8000))
print("Listening on port 8000...")
server.register_function(is_even, "is_even")
server.serve_forever()

Functions can also be registered in a decorator way instead of calling server_function(is_even, "is_even"):

@server.register_function
def is_even(n):
    return n % 2 == 0

client:

import xmlrpc.client

with xmlrpc.client.ServerProxy("http://localhost:8000/") as proxy:
    print("3 is even: %s" % str(proxy.is_even(3)))
    print("100 is even: %s" % str(proxy.is_even(100)))

Further information: https://docs.python.org/3/library/xmlrpc.server.html#

momo
  • 3,313
  • 2
  • 19
  • 37
  • This is probably the closest to what I need. RPC is not ideal in several ways, but I completely forgot it was a thing *and* the bount is expiring today, so thanks a lot for the answer :) – George Mar 04 '21 at 14:32