I have problems with jsonpickle.decode and a dictionary

Question

I have issues decoding a JSON to a dictionary using Python 3.9.9 and jsonpickle 2.0.0.

The JSON looks like follows:

{
  "py/object" : "game.game.GameData",
  "origin" : {
    "py/object" : "player.coordinates.BasicCoordinate",
    "pos_x" : 67,
    "pos_y" : 44
  },
  "top_right" : {
    "py/object" : "player.coordinates.BasicCoordinate",
    "pos_x" : 1134,
    "pos_y" : -2
  },
  "bottom_right" : {
    "py/object" : "player.coordinates.BasicCoordinate",
    "pos_x" : 1125,
    "pos_y" : 623
  },
  "bottom_left" : {
    "py/object" : "player.coordinates.BasicCoordinate",
    "pos_x" : -2,
    "pos_y" : 604
  },
  "player_coordinates" : {
    "json://{\"py/reduce\": [{\"py/function\": \"numpy.core.multiarray.scalar\"}, {\"py/tuple\": [{\"py/reduce\": [{\"py/type\": \"numpy.dtype\"}, {\"py/tuple\": [\"i4\", false, true]}, {\"py/tuple\": [3, \"<\", null, null, null, -1, -1, 0]}]}, {\"py/b64\": \"BAAAAA==\"}]}]}" : {
      "py/object" : "player.coordinates.PlayerCoordinate",
      "player_id" : 4,
      "alpha" : 178,
      "pos_x" : 749,
      "pos_y" : 485
    },
    "json://{\"py/reduce\": [{\"py/function\": \"numpy.core.multiarray.scalar\"}, {\"py/tuple\": [{\"py/id\": 7}, {\"py/b64\": \"AgAAAA==\"}]}]}" : {
      "py/object" : "player.coordinates.PlayerCoordinate",
      "player_id" : 2,
      "alpha" : -4,
      "pos_x" : 290,
      "pos_y" : 421
    },
    "json://{\"py/reduce\": [{\"py/function\": \"numpy.core.multiarray.scalar\"}, {\"py/tuple\": [{\"py/id\": 7}, {\"py/b64\": \"BQAAAA==\"}]}]}" : {
      "py/object" : "player.coordinates.PlayerCoordinate",
      "player_id" : 5,
      "alpha" : 180,
      "pos_x" : 1048,
      "pos_y" : 304
    },
    "json://{\"py/reduce\": [{\"py/function\": \"numpy.core.multiarray.scalar\"}, {\"py/tuple\": [{\"py/id\": 7}, {\"py/b64\": \"AQAAAA==\"}]}]}" : {
      "py/object" : "player.coordinates.PlayerCoordinate",
      "player_id" : 1,
      "alpha" : 2,
      "pos_x" : 109,
      "pos_y" : 301
    },
    "json://{\"py/reduce\": [{\"py/function\": \"numpy.core.multiarray.scalar\"}, {\"py/tuple\": [{\"py/id\": 7}, {\"py/b64\": \"AwAAAA==\"}]}]}" : {
      "py/object" : "player.coordinates.PlayerCoordinate",
      "player_id" : 3,
      "alpha" : 0,
      "pos_x" : 296,
      "pos_y" : 165
    },
    "json://{\"py/reduce\": [{\"py/function\": \"numpy.core.multiarray.scalar\"}, {\"py/tuple\": [{\"py/id\": 7}, {\"py/b64\": \"BgAAAA==\"}]}]}" : {
      "py/object" : "player.coordinates.PlayerCoordinate",
      "player_id" : 6,
      "alpha" : -178,
      "pos_x" : 779,
      "pos_y" : 116
    }
  },
  "ball_coordinates" : {
    "py/object" : "player.coordinates.BasicCoordinate",
    "pos_x" : 617,
    "pos_y" : 344
  },
  "init_coordinates" : { },
  "goal_1" : {
    "py/object" : "game.goal.Goal",
    "post_1" : {
      "py/object" : "player.coordinates.BasicCoordinate",
      "pos_x" : 18,
      "pos_y" : 196
    },
    "post_2" : {
      "py/object" : "player.coordinates.BasicCoordinate",
      "pos_x" : 17,
      "pos_y" : 412
    }
  },
  "goal_2" : {
    "py/object" : "game.goal.Goal",
    "post_1" : {
      "py/object" : "player.coordinates.BasicCoordinate",
      "pos_x" : 1113,
      "pos_y" : 199
    },
    "post_2" : {
      "py/object" : "player.coordinates.BasicCoordinate",
      "pos_x" : 1111,
      "pos_y" : 423
    }
  },
  "game_status" : {
    "py/reduce" : [ {
      "py/type" : "game.game.GameStatus"
    }, {
      "py/tuple" : [ 1 ]
    } ]
  },
  "center" : {
    "py/object" : "player.coordinates.BasicCoordinate",
    "pos_x" : 562,
    "pos_y" : 311
  },
  "goals_team1" : 0,
  "goals_team2" : 0,
  "game_time" : 153.95,
  "should_log" : true
}

It is encoded with this command jsonpickle.encode(data, keys=True), where data is a dict(). Decoding the JSON on a Windows machine works, same Python, same jsonpickle package. On Raspbian it does not work with the same versions.

The error from jsonpickle.decode is as follows (however, other JSON data can be decoded using the same environment and calls without issues, but those dictionaries are not that complex):

   m = jsonpickle.decode(msg.payload, keys=True)
  File "/home/user/venv/lib/python3.9/site-packages/jsonpickle/unpickler.py", line 50, in decode
    return context.restore(data, reset=reset, classes=classes)
  File "/home/user/venv/lib/python3.9/site-packages/jsonpickle/unpickler.py", line 163, in restore
    value = self._restore(obj)
  File "/home/user/venv/lib/python3.9/site-packages/jsonpickle/unpickler.py", line 196, in _restore
    return restore(obj)
  File "/home/user/venv/lib/python3.9/site-packages/jsonpickle/unpickler.py", line 359, in _restore_object
    return self._restore_object_instance(obj, cls)
  File "/home/user/venv/lib/python3.9/site-packages/jsonpickle/unpickler.py", line 421, in _restore_object_instance
    instance = self._restore_object_instance_variables(obj, instance)
  File "/home/user/venv/lib/python3.9/site-packages/jsonpickle/unpickler.py", line 477, in _restore_object_instance_variables
    instance = self._restore_from_dict(obj, instance)
  File "/home/user/venv/lib/python3.9/site-packages/jsonpickle/unpickler.py", line 446, in _restore_from_dict
    value = self._restore(v)
  File "/home/user/venv/lib/python3.9/site-packages/jsonpickle/unpickler.py", line 196, in _restore
    return restore(obj)
  File "/home/user/venv/lib/python3.9/site-packages/jsonpickle/unpickler.py", line 570, in _restore_dict
    k = self._restore_pickled_key(k)
  File "/home/user/venv/lib/python3.9/site-packages/jsonpickle/unpickler.py", line 613, in _restore_pickled_key
    key = decode(
  File "/home/user/venv/lib/python3.9/site-packages/jsonpickle/unpickler.py", line 50, in decode
    return context.restore(data, reset=reset, classes=classes)
  File "/home/user/venv/lib/python3.9/site-packages/jsonpickle/unpickler.py", line 163, in restore
    value = self._restore(obj)
  File "/home/user/venv/lib/python3.9/site-packages/jsonpickle/unpickler.py", line 196, in _restore
    return restore(obj)
  File "/home/user/venv/lib/python3.9/site-packages/jsonpickle/unpickler.py", line 266, in _restore_reduce
    reduce_val = list(map(self._restore, obj[tags.REDUCE]))
  File "/home/user/venv/lib/python3.9/site-packages/jsonpickle/unpickler.py", line 196, in _restore
    return restore(obj)
  File "/home/user/venv/lib/python3.9/site-packages/jsonpickle/unpickler.py", line 537, in _restore_tuple
    return tuple([self._restore(v) for v in obj[tags.TUPLE]])
  File "/home/user/venv/lib/python3.9/site-packages/jsonpickle/unpickler.py", line 537, in <listcomp>
    return tuple([self._restore(v) for v in obj[tags.TUPLE]])
  File "/home/user/venv/lib/python3.9/site-packages/jsonpickle/unpickler.py", line 196, in _restore
    return restore(obj)
  File "/home/user/venv/lib/python3.9/site-packages/jsonpickle/unpickler.py", line 278, in _restore_reduce
    stage1 = f(*args)
TypeError: 'dict' object is not callable

Unfortunately I don't know where to post such issues (jsonpickle) elsewhere, that's why i try my luck here. On GitHub there is only the possibility to report a bug.

I would consider opening a ticket to report a bug. Without delving into the source code, the fact that a variable named `f` is referencing a dict and being called with arguments inside the library source code seems like unexpected behaviour. Regardless of whatever issue exists with your input JSON (if any), this is likely not the way it is intended to handle it. — zr0gravity7, Dec 05 '21 at 07:55
@zr0gravity7 ok, i already did yesterday after i found out how to do it on GitHub. Thank you for the edits and hint :) I debugged the jsonpickle on the Windows machine and the call on f() seems to be a cast operation. Maybe i can manage to debug that stuff directly on Raspbian to see whats going on there. I especially don't understand why `numpy.core.multiarray.scalar` is used on a class which only contains `int` member variables. — Joysn, Dec 05 '21 at 12:41
After i updated the operating system i was able to install newer Python packages and then the issue valished. — Joysn, Dec 06 '21 at 08:58

score 1 · Answer 1 · answered Sep 09 '22 at 17:35

I ran into this problem today (jsonpickle 2.2.0 on Linux) and was able to track it back. Every py/object type declared in the exported JSON must be a public type. In my example, I had a namedtuple that was within a class and not the top-level of the module. jsonpickle recorded the object type as SystemStatus.MyTuple and once I moved that MyTuple definition to the top of the module, jsonpickle was able to restore that object by passing the recorded arguments to it.

My hint was this still-open bug about namedtuples which made me start poking around.

(I know the OP noted in a comment this problem fixed itself, but here I am nearly a year later hitting the same problem and this is one of the top Google hits, so here's my answer.)

Thank you for the answer and effort. Unfortunately I am not able to verify the approach anymore. The environment is completely gone. — Joysn, Oct 23 '22 at 15:06

I have problems with jsonpickle.decode and a dictionary

1 Answers1