ignore custom fields (with a value) when converting to json #11124

aleivag · 2024-12-16T15:37:21Z

aleivag
Dec 16, 2024

hi folks. I want to ignore some fields when they have a particular value, when converting a model into json, and others keep them... for instance this code

class Step(BaseModel):
    name: str
    command: str
    required: bool = True # dont serialize if true
    timeout: int = 3_600

class Args(BaseModel):
    steps: list[Step] = []
    timeout: int = 3600 * 3
    env: dict[str, str] = {} # dont serialize if empty
    require: list[str] = [] # dont serialize if empty


class Job(BaseModel):
    name: str
    commit: str = "main" # dont serialize if main
    priority: int = 0
    args: Args = Args()


print(Job(name="foo", args=Args(steps=[Step(name="hello", command="echo 'hi!'")])).model_dump_json(indent=2))

prints

{
  "name": "foo",
  "commit": "main",
  "priority": 0,
  "args": {
    "steps": [
      {
        "name": "hello",
        "command": "echo 'hi!'",
        "required": true,
        "timeout": 3600
      }
    ],
    "timeout": 10800,
    "env": {},
    "require": []
  }
}

but i would like it to print

{
  "name": "foo",
  "priority": 0,
  "args": {
    "steps": [
      {
        "name": "hello",
        "command": "echo 'hi!'",
        "timeout": 3600
      }
    ],
    "timeout": 10800,
  }
}

because when those list are huge, those extra bites does count!.

I though of using any of exclude_unset, exclude_defaults, exclude_none from https://docs.pydantic.dev/latest/api/base_model/#pydantic.BaseModel.model_dump_json, but not all the default should be ignored, and all fields are technically set...

i also played around with overwriting dict but, even though it worked, seems very ugly and not really reliable.

i think my issue is that calling model_dump_json does on a parent model, does not call model_dump_json on the childs... and i can really customize child attributes... maybe there is something that can be done in the json_schema_extra param?

janas-adam · 2024-12-16T21:07:25Z

janas-adam
Dec 16, 2024

hi @aleivag 👋

according to your example, u can override model_dump_json method with exclude definitions:

class Step(BaseModel):
    name: str
    command: str
    required: bool = True  # dont serialize if true
    timeout: int = 3_600


class Args(BaseModel):
    steps: list[Step] = []
    timeout: int = 3600 * 3
    env: dict[str, str] = {}  # dont serialize if empty
    require: list[str] = []  # dont serialize if empty


class Job(BaseModel):
    name: str
    commit: str = "main"  # dont serialize if main
    priority: int = 0
    args: Args = Args()

    def model_dump_json(self, **kwargs):
        exclude = kwargs.pop("exclude", {})

        if self.commit == "main":
            exclude["commit"] = True

        nested_exclude = {"args": {"steps": {}}}

        if not self.args.env:
            nested_exclude["args"]["env"] = True

        if not self.args.require:
            nested_exclude["args"]["require"] = True

        for n, step in enumerate(self.args.steps):
            if step.required:
                nested_exclude["args"]["steps"] = {n: {"required": True}}

        exclude = {**exclude, **nested_exclude}

        return super().model_dump_json(exclude=exclude, **kwargs)

which gives:

# your example

print(
    Job(
        name="foo", args=Args(steps=[Step(name="hello", command="echo 'hi!'")])
    ).model_dump_json(indent=2)
)

# result
{
  "name": "foo",
  "priority": 0,
  "args": {
    "steps": [
      {
        "name": "hello",
        "command": "echo 'hi!'",
        "timeout": 3600
      }
    ],
    "timeout": 10800
  }
}

and other example:

print(
    Job(
        name="11111",
        args=Args(
            steps=[
                Step(name="step-1111", command="echo 'hi!'", required=False),
                Step(name="step-2222", command="echo 'hi!'", required=True),
                Step(name="step-3333", command="echo 'hi!'", required=False),
            ],
            env={"not-empty": "123"},
            require=["not-empty"],
        ),
        commit="not-main",
    ).model_dump_json(indent=2)
)

# result
{
  "name": "11111",
  "commit": "not-main",
  "priority": 0,
  "args": {
    "steps": [
      {
        "name": "step-1111",
        "command": "echo 'hi!'",
        "required": false,
        "timeout": 3600
      },
      {
        "name": "step-2222",
        "command": "echo 'hi!'",
        "timeout": 3600
      },
      {
        "name": "step-3333",
        "command": "echo 'hi!'",
        "required": false,
        "timeout": 3600
      }
    ],
    "timeout": 10800,
    "env": {
      "not-empty": "123"
    },
    "require": [
      "not-empty"
    ]
  }
}

helpful links:
pydantic-serialization-concepts-of-advanced-include-and-exclude

hope it helped

0 replies

aleivag · 2024-12-16T23:29:22Z

aleivag
Dec 16, 2024
Author

hi @janas-adam , thanks for the reply... the problem is that the logic for what to exclude its on the Job object not in the actual object that needs to implement them, so if for instance i want to serialize a list of Job i cant do

class ListOfJobs(RootModel):
    root: list[Job]

then ListOfJobs. model_dump_json() would not do the right thing... I'm looking to see if there is a way to do this in the class that implements the fields that gets ignored

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ignore custom fields (with a value) when converting to json #11124

{{title}}

Replies: 2 comments

{{title}}

{{title}}

Select a reply

ignore custom fields (with a value) when converting to json #11124

aleivag Dec 16, 2024

Replies: 2 comments

janas-adam Dec 16, 2024

aleivag Dec 16, 2024 Author

aleivag
Dec 16, 2024

janas-adam
Dec 16, 2024

aleivag
Dec 16, 2024
Author