Python Twisted Defer returnValue incompatible with dict

Question

I am using twisted.internet.defer extensively in my package , I have encountered a problem which I am unable to solve after spending 2 days on it. Below is my problem scenario.

# all imports done and correct
class infrastructure: # line 1

  @inlineCallbacks
  def dict_service(self):
    client = MyClient()
    services = yield client.listServices() # line 5
    ret = (dict(service.name, [cont.container_id for cont in service.instances]) for service in dockerServices)
    returnValue(ret) # line 7

I am calling my client which returns me list of Services. listServices() return type is twisted.internet.defer.ReturnValue .

class myinterface:
   # has infrastructure

  def init:
     data = dict(
        container_services=self._infrastructure.dict_service(),
        )

when executing this I get below error which I am unable to comprehend. can someone please help.

raise TypeError(repr(o) + \" is not JSON serializable\")\nexceptions.TypeError: <DeferredWithContext at 0x4bfbb48 current result: <twisted.python.failure.Failure <type 'exceptions.NameError'>>> is not JSON serializable\n"

Is it because that wrapping dict with returnValue creating problem?

score 1 · Accepted Answer · answered Feb 10 '17 at 15:56

There's no problem using returnValue with a dict instance:

$ python -m twisted.conch.stdio
>>> from __future__ import print_function
>>> from twisted.internet.defer import inlineCallbacks, returnValue
>>> @inlineCallbacks
... def f():
...     d = yield {"foo": "bar"} # Yield *something* or it's not a generator
...     returnValue(d)
... 
>>> f().addCallback(print)
{'foo': 'bar'}
<Deferred at 0x7f84c51847a0 current result: None>

The error you reported:

raise TypeError(repr(o) + \" is not JSON serializable\")\nexceptions.TypeError: <DeferredWithContext at 0x4bfbb48 current result: <twisted.python.failure.Failure <type 'exceptions.NameError'>>> is not JSON serializable\n"

Makes it appear as though you have some code that triggers a NameError. This seems to happen in a Deferred callback (or otherwise makes its way into a Deferred) and gets wrapped in a Failure:

<twisted.python.failure.Failure <type 'exceptions.NameError'>>

That leaves:

raise TypeError(repr(o) + \" is not JSON serializable\")\nexceptions.TypeError: <DeferredWithContext at 0x4bfbb48 current result: ...> is not JSON serializable\n"

I don't know what DeferredWithContext is. I'm guessing it's a subclass of Deferred with some additional behavior. It'd be nice if you could link to the library that provides this (it seems like a bad idea to me but I'd like to understand more).

If that's so, then the error is talking about a DeferredWithContext instance that has the above-described Failure as its result:

<DeferredWithContext at 0x4bfbb48 current result: ...>

That leaves:

raise TypeError(repr(o) + \" is not JSON serializable\")\nexceptions.TypeError: ... is not JSON serializable\n"

This seems to come from the json module, either the dump or dumps function. This is claiming something that is not JSON serializable was passed in. DeferredWithContext is almost certainly not JSON serializable, so that explains the problem.

This would result from something like:

json.dumps(function_that_returns_deferred())

which should instead be:

function_that_returns_deferred().addCallback(json.dumps)

I do not have explicitly any json.dumps. but total stacktrace looks like this. — chaosguru, Feb 13 '17 at 08:34
Something in your program must be doing _something_ with JSON. Twisted itself certainly isn't trying to JSON serialize any values you yield from an inlineCallbacks-decorated function. — Jean-Paul Calderone, Feb 13 '17 at 13:03
yes got it in-fact i was missing the returnValue in the callback. hence the problem. No json.dump anywhere provided in my code. Thanks. — chaosguru, Feb 14 '17 at 08:00
Accepting this Answer since it helped me to solve my problem though solution was slightly different. — chaosguru, Feb 14 '17 at 08:01

Python Twisted Defer returnValue incompatible with dict

1 Answers1