Retrieve JSON string from pydantic Json type

Question

I have a pydantic model as follows.

from pydantic import Json, BaseModel
class Foo(BaseModel):
    id: int
    bar: Json

Foo.bar can parse JSON string as input and store it as a dict which is nice.

foo = Foo(id=1, bar='{"foo": 2, "bar": 3}')
type(foo.bar) #outputs dict

And if i want the entire object to be a dict I can do

foo.dict()
#outputs
{'id': 1, 'bar': {'foo': 2, 'bar': 3}}

But how can I export bar as JSON string as following

{'id': 1, 'bar': '{"foo": 2, "bar": 3}'}

I want to write JSON back into the database.

score 12 · Answer 1 · answered May 28 '21 at 08:27

12

Pydantic author here.

There's currently no way to do that without calling json.dumps(foo.bar). You could make that a method on Foo if you liked, which would be easier to use but require the same processing.

If performance is critical, or you need the exact same JSON string you started with (same spaces etc.) You could do one of the following:

Make bar a string field but add a validator to check it's valid JSON
create a custom data type to parse the JSON but also keep a reference to the raw JSON string

answered May 28 '21 at 08:27

SColvin

11,584
6
57
71

Thanks for the answer, Performance is not super critical. Right now I am using `bar` as string with validation. I wish `foo.fields` would give me `'bar': ModelField(name='bar', type=Json, required=False, default=None)` so I can identify the fields which are Json and override `dict()` method and do `json.dumps(self.bar)`. I think it just makes it easier to read and write it back to database without having to manually change Json types in all my models. Right now when i try to get field types i get this `'bar': ModelField(name='bar', type=Optional[Any], required=False, default=None)` – Jey May 28 '21 at 08:34
I will try out this `create a custom data type to parse the JSON but also keep a reference to the raw JSON string` – Jey May 28 '21 at 08:36
look at `foo.__fields__` it'll give you that data albeit, quite as obvious as that to inspect. – SColvin May 28 '21 at 15:59
1

Hey, I've solved this problem with `foo.schema()` instead of __fields__ as `__fields__ ` specify `Json` as `Any`. Thanks for this wonderful library! https://stackoverflow.com/a/67907251/11061699 – Jey Jun 09 '21 at 15:30

score 0 · Answer 2 · answered May 28 '21 at 08:17

0

Do you want to convert the nest to string?

x = {'id': 1, 'bar': str({'foo': 2, 'bar': 3})}

gives

{'id': 1, 'bar': "{'foo': 2, 'bar': 3}"}

answered May 28 '21 at 08:17

Rutger

593
5
11

This is kinda my current approach. I kept `bar` as string type. and handling the `json.dumps` and `json.loads` outside pydantic. – Jey May 28 '21 at 08:35
Maybe I don't understand what you desire then. Do you want to dict values to be converted to strings if it is a dict? In that case you can do some dictionary iterator that coverts to string if the value is a dict. – Rutger May 28 '21 at 08:38
To say it simply. I'm reading Foo() from database. And its stored as string in database. After I retrieve it. I convert the JSON into python object (This can be done in pydantic now). But when I try to write to database. the field `bar` has a python object instead of JSON string. In other words. I want to work with dict object in python and store and retrieve JSON in database – Jey May 28 '21 at 09:11

score 0 · Answer 3 · answered Jun 09 '21 at 15:27

The workaround that solved the problem in my scenario was to inherit BaseModel and override dict method.

class ExtendedModel(BaseModel):
    def dict(self, json_as_string=False, **kwargs) -> dict:
        __dict = super().dict(**kwargs)
        if json_as_string:
            for field, _ in self.schema()['properties'].items():
                if _.get('format') == 'json-string' and field in __dict :
                    __dict[field] = json.dumps(getattr(self, field))
        return __dict

self.__field__ wouldn't specify if the field's type is actually Json. It was returning as Any. So I used this approch.

score 0 · Answer 4 · answered Oct 29 '22 at 20:02

I faced same problem so here comes a little hacky solution with class factory (some elegant one with something like foo: JsonVal[T] is almost impossible to implement due to numerous internal pydantic hacks like how it works with generic type annotations). Unfortunately type inference is not working, but validation and access to parsed value are ok, considering that Fields are always stored/serialized as str.


from abc import ABC
from typing import TypeVar, Generic, Type

from pydantic import Json, parse_obj_as, BaseModel

T = TypeVar('T')


class JsonVal(Generic[T], str, ABC):
    @property
    def parsed(self) -> T:
        return None


def json_val(t: Type) -> Type[JsonVal[T]]:
    class _JsonVal(JsonVal, str):
        _t: Type

        @classmethod
        def __get_validators__(cls):
            yield cls.validate

        @classmethod
        def validate(cls, v):
            parse_obj_as(Json[cls._t], v)
            return cls(v)

        @property
        def parsed(self):
            return parse_obj_as(Json[self._t], self)

    _JsonVal._t = t
    return _JsonVal


class Bar(BaseModel):
    bar: str


class Model(BaseModel):
    foo: json_val(list[int])
    bar: json_val(Bar)


m = Model.parse_obj({"foo": '[1,2,3]', "bar": '{"bar":"baz"}'})

print(m)
print(m.foo, type(m.foo), m.foo.parsed, type(m.foo.parsed))
print(m.bar, type(m.bar), m.bar.parsed, type(m.bar.parsed))
print(m.json())

# foo='[1,2,3]' bar='{"bar":"baz"}'
# f[1,2,3] <class '__main__.json_val.<locals>._JsonVal'> [1, 2, 3] <class 'list'>
# f{"bar":"baz"} <class '__main__.json_val.<locals>._JsonVal'> bar='baz' <class '__main__.Bar'>
# f{"foo": "[1,2,3]", "bar": "{\"bar\":\"baz\"}"}

Retrieve JSON string from pydantic Json type

4 Answers4