How to properly work with third-party libraries in Pyro v.4.63

Question

I'm a bit confused with working with remote third-party libraries:

1) For example, I have server code:

import Pyro4
import Pyro4.naming
import Pyro4.utils.flame
Pyro4.config.REQUIRE_EXPOSE = False
Pyro4.config.FLAME_ENABLED = True
Pyro4.config.SERIALIZERS_ACCEPTED = set(["pickle"])
Pyro4.config.SERIALIZER = 'pickle'  # for flameserver


def _main():
    """Start RPC server."""
    ip_address = Pyro4.socketutil.getIpAddress(None, workaround127=True)
    ns_daemon = Pyro4.naming.NameServerDaemon(host=ip_address, port=RPC_PORT)
    Pyro4.utils.flame.start(ns_daemon)

    from remote_management.rpc.component.shell import Shell
    shell_uri = ns_daemon.register(Shell())
    ns_daemon.nameserver.register("shell", shell_uri)
    from remote_management.rpc.component.cpu import CPU
    cpu_uri = ns_daemon.register(CPU())
    ns_daemon.nameserver.register("cpu", cpu_uri)

    ns_daemon.requestLoop()
    ns_daemon.close()

if __name__ == "__main__":
    _main()

The CPU component is like this:

import psutil

class CPU(object):
    def ps(self):
        return psutil.cpu_times()

When I use it in client like this:

import Pyro4
import Pyro4.utils.flame
import Pyro4.errors

Pyro4.config.SERIALIZER = "pickle"

proxy = Pyro4.Proxy("PYRONAME:cpu@myhost:47976")
print proxy.ps()

I get an error:

Traceback (most recent call last):
  File "t.py", line 145, in <module>
    print proxy.ps()
  File "/home/korolev/.envs/auto/lib/python2.7/site-packages/Pyro4/core.py", line 187, in __call__
    return self.__send(self.__name, args, kwargs)
  File "/home/korolev/.envs/auto/lib/python2.7/site-packages/Pyro4/core.py", line 464, in _pyroInvoke
    data = serializer.deserializeData(msg.data, compressed=msg.flags & message.FLAGS_COMPRESSED)
  File "/home/korolev/.envs/auto/lib/python2.7/site-packages/Pyro4/util.py", line 170, in deserializeData
    return self.loads(data)
  File "/home/korolev/.envs/auto/lib/python2.7/site-packages/Pyro4/util.py", line 451, in loads
        return pickle.loads(data)

ImportError: No module named psutil._pslinux

As you see it cannot be deserialized. And if I change return to:

return tuple(psutil.cpu_times()), then it works.

2) I think this next problem has the same nature: The server code is almost the same except component

from remote_management.rpc.component.registrator import ModuleRegistrator
reg_uri = ns_daemon.register(ModuleRegistrator())
ns_daemon.nameserver.register("module_registrator", reg_uri)

If I need remote module, but don't have it locally, why I can't do just like this:

Registrator component:

class ModuleRegistrator(object):
    def register(self, module):
        module = importlib.import_module(module)
        self._pyroDaemon.register(module)
        return module

so when I use it in client:

proxy = Pyro4.Proxy("PYRONAME:module_registrator@myhost:47976")
print proxy.register("psutil").cpu_times()

I get an error:

Traceback (most recent call last):
  File "t.py", line 145, in <module>
    print proxy.register('psutil').cpu_times()
  File "/home/korolev/.envs/auto/lib/python2.7/site-packages/Pyro4/core.py", line 279, in __getattr__
    raise AttributeError("remote object '%s' has no exposed attribute or method '%s'" % (self._pyroUri, name))
AttributeError: remote object 'PYRONAME:module_registrator@myhost:47976' has no exposed attribute or method 'register'

Please help me understand this errors and where I'm missing something? And what is the best approach to handle things like this when working with third-party remotely? Thanks in advance!

Answer 1

Firstly, you seem to be using the 'flame' feature of Pyro4, which requires the pickle serialization protocol. This is powerful, but not without problems. Are you sure you want/need to use flame? I hope you are aware of its security implications. http://pyro4.readthedocs.io/en/latest/flame.html

Now, your actual question consists of three parts (try to make your future questions more about just a single issue, that makes answering them a lot simpler)

To answer the first part of your question:

As you already found out yourself, you cannot assume that all types can just be transported over the network and magically just work on the other side. Especially with pickle this only works if the module in which the type is defined, is available on both sides. In your case, it seems that the psutil module is not available on your client side, hence the pickle error when it tries to deserialize the object returned by psutil.cpu_times().

The solution could be to install psutil on your client side as well. You also discovered an alternative solution: don't pass objects of custom types over the wire, but stick to builtin types as much as possible. When you convert the result to a tuple or list or dict, these can be serialized and deserialized without any issues (if all the objects they contain are of a builtin type as well).

To answer the second part of your question:

This code looks like you're trying to find a workaround for the problem in the first question? Anyway, the error should be clear; it means what it says: that the remote object your proxy is connected to, doesn't have the method exposed that you're trying to call. You should add a @Pyro4.expose decoration to properly expose it.

That said, you're already using Flame, so why aren't you using its remote module feature? See http://pyro4.readthedocs.io/en/latest/flame.html#flame-object-and-examples

For example (I assume you installed psutil on the client side):

import Pyro4.utils.flame

Pyro4.config.SERIALIZER = "pickle"
flame = Pyro4.utils.flame.connect("yourhost:9999")

psutilmodule = flame.module("psutil")
print("remote cpu:", psutilmodule.cpu_times())

Finally,

to address your question "what is the best approach to handle things like this when working with third-party remotely":

• try not to expose the details and types of the API you're working with, reduce it to builtin types as much as possible. http://pyro4.readthedocs.io/en/latest/tipstricks.html#consider-using-basic-data-types-instead-of-custom-classes If you're doing this correctly, you can also simply use Pyro's default serializer (serpent) instead of pickle, and not have any security issues. (that does require a solution that does not depend on flame, see next bullet)
• don't use Flame unless you're absolutely certain that you are okay with its security implications and that you can't solve your issue another way. Instead create a normal Pyro server class with a dedicated custom API that you wrote yourself.
• only expose the remote methods that you actually need to have accessible remotely.