18

I have defined a pydantic Schema with extra = Extra.allow in Pydantic Config.

Is it possible to get a list or set of extra fields passed to the Schema separately.

For ex:

from pydantic import BaseModel as pydanticBaseModel
class BaseModel(pydanticBaseModel):
    name: str

    class Config:
        allow_population_by_field_name = True
        extra = Extra.allow

I pass below JSON:

   {
    "name": "Name",
    "address": "bla bla",
    "post": "post"
   }

I need a function from pydantic, if available which will return all extra fields passed. like: {"address", "post"}

alex_noname
  • 26,459
  • 5
  • 69
  • 86
Jyotirmay
  • 1,533
  • 3
  • 21
  • 41
  • 2
    @YaakovBressler Could you elaborate on what you are unsatisfied with for the existing answer? – aaron Feb 08 '23 at 01:11
  • I guess I just wanted to bring more attention to the existing answers. I was trigger happy with creating the bounty @aaron – Yaakov Bressler Feb 08 '23 at 16:09

3 Answers3

18

From the pydantic docs:

extra

whether to ignore, allow, or forbid extra attributes during model initialization. Accepts the string values of 'ignore', 'allow', or 'forbid', or values of the Extra enum (default: Extra.ignore). 'forbid' will cause validation to fail if extra attributes are included, 'ignore' will silently ignore any extra attributes, and 'allow' will assign the attributes to the model.

This can either be included in the model Config class, or as arguments when inheriting BaseModel.

from pydantic import BaseModel, Extra

class BaseModel(BaseModel, extra=Extra.allow):
    name: str


model = Model.parse_obj(
   {"name": "Name", "address": "bla bla", "post": "post"}
)

print(model)
# name='Name' post='post' address='bla bla'

To get the extra values you could do something simple, like comparing the set of __fields__ defined on the class to the values in __dict__ on an instance:

class Model(BaseModel, extra=Extra.allow):
    python_name: str = Field(alias="name")

    @property
    def extra_fields(self) -> set[str]:
        return set(self.__dict__) - set(self.__fields__)

>>> Model.parse_obj({"name": "Name", "address": "bla bla", "post": "post"}).extra_fields
{'address', 'post'}
>>> Model.parse_obj({"name": "Name", "foobar": "fizz"}).extra_fields
{'foobar'}
>>> Model.parse_obj({"name": "Name"}).extra_fields
set()
flakes
  • 21,558
  • 8
  • 41
  • 88
  • Would this work on recursive objects? – Yaakov Bressler Feb 08 '23 at 16:13
  • 2
    @YaakovBressler What do you mean by "work" and what do you mean by "recursive" objects? Each nested object would load their json section independently as defined by their model class. So if you wanted this to work on nested hierarchies, each model in the composition would need to supply the same trick. – flakes Feb 08 '23 at 16:48
  • That answers it. Additional Q, would it be possible for the parent to be aware of `extra_fields` across all children? Or, that would need to be accessed dynamically? I guess you can look over `extra_fields` and do a check if a value has a property `extra_fields`... – Yaakov Bressler Feb 08 '23 at 17:10
  • 1
    @YaakovBressler Could be one way to do it yeah. Would probably need a search function to walk the tree (accounting for potential cycles as well). You'd probably want to not define it on the class, but as a top level function, in the case that you have a hierarchy like `A -> B -> C` where A and C support extra fields, but `B` does not. – flakes Feb 08 '23 at 17:34
  • This behaviour was changed in pydantic 2.0 with this answer no longer being the best solution. Instead of modifying this answer which is correct for 1.x, I posted the updated solution below. – robcxyz Sep 01 '23 at 08:44
15

As far as I know, there is no out-of-the-box solution for this. But since you have access to all "raw" passed data, including extra fields, in the pre root validator, you can put them there in a separate dictionary.

An example is taken from here. Thanks to @PrettyWood for providing it:

from typing import Any, Dict, Optional

from pydantic import BaseModel, Field, root_validator


class ComposeProject(BaseModel):
    versio: Optional[str] = Field(alias='version')  # just to show that it supports alias too
    extra: Dict[str, Any]

    @root_validator(pre=True)
    def build_extra(cls, values: Dict[str, Any]) -> Dict[str, Any]:
        all_required_field_names = {field.alias for field in cls.__fields__.values() if field.alias != 'extra'}  # to support alias

        extra: Dict[str, Any] = {}
        for field_name in list(values):
            if field_name not in all_required_field_names:
                extra[field_name] = values.pop(field_name)
        values['extra'] = extra
        return values


project = ComposeProject(**{
  "version": "3",
  "services": ...
})
print(project.version)  # => '3'
print(project.extra)  # => {'services': Ellipsis}
alex_noname
  • 26,459
  • 5
  • 69
  • 86
  • Thank you very much @alex_noname , Could you please also have a look to this: https://stackoverflow.com/questions/69617702/is-it-possible-to-change-unknow-fields-to-camelcase-in-pydantic That was the main issue for me. Thank you. – Jyotirmay Oct 18 '21 at 15:28
  • 1
    As for your other question you can call `camelcase` for `field_name` before saving fields to extra dict, but i'.m afraid the is no direct way to put extra fields on the same level as declared fields in exporting model dict. – alex_noname Oct 18 '21 at 15:46
  • This behaviour was changed in pydantic 2.0 with this answer no longer being the best solution. Instead of modifying this answer which is correct for 1.x, I posted the updated solution below. – robcxyz Sep 01 '23 at 08:45
0

Pydantic extra fields behaviour was updated in their 2.0 release. Per their docs, you now don't need to do anything but set the model_config extra field to allow and then can use the model_extra field or __pydantic_extra__ instance attribute to get a dict of extra fields.

from pydantic import BaseModel, Field, ConfigDict


class Model(BaseModel):
    python_name: str = Field(alias="name")

    model_config = ConfigDict(
        extra='allow',
    )

m = Model(**{"name": "Name", "address": "bla bla", "post": "post"}).model_extra
assert m == {'address': 'bla bla', 'post': 'post'}

m = Model(**{"name": "Name", "foobar": "fizz"}).__pydantic_extra__
assert m == {'foobar': 'fizz'}

m = Model(**{"name": "Name"}).__pydantic_extra__
assert m == {}
robcxyz
  • 74
  • 2
  • 11