Personalkollen developer blog

Typed Django forms using Pydantic

2025-01-13T13:40:00+01:00

Update 2026-03-03: The post has been updated with a new version that resolve the run time types from the generic types rather than via class attributes.

Django Forms is a great and easy way to render, clean, and validate HTML form inputs. It is easy to create custom fields and widgets for most needs. But one of the downsides we have been struggling with since moving to static type checking (using MyPy and django-stubs) is the lack of type support for the validated data coming from cleaned_data. Knowing what types the validated data was would help to prevent unnecessary bugs and allows for faster development when you don't have to second guess the underlying data.

We have created a wrapper around Django forms that together with Pydantic gives us more type information, as well as some runtime protection if the data somehow would not match the expected data.

from __future__ import annotations

import functools
import types
from typing import Any, get_args

from django import forms
from pydantic import BaseModel, ConfigDict


class Valid(BaseModel):
    model_config = ConfigDict(arbitrary_types_allowed=True, frozen=True, strict=True)


class Invalid(BaseModel):
    model_config = ConfigDict(arbitrary_types_allowed=True, frozen=True, strict=True)


def _resolve_generic_type_args(cls: type) -> dict[type, tuple[type, ...]]:
    result = {}

    def _collect(c: type, type_map: dict[Any, type]) -> None:
        for base in types.get_original_bases(c):
            origin = getattr(base, "__origin__", base)
            args = get_args(base)

            resolved_args = tuple(type_map.get(a, a) for a in args)

            if origin in result:
                continue
            result[origin] = resolved_args

            if isinstance(origin, type) and hasattr(origin, "__type_params__"):
                next_map = dict(zip(origin.__type_params__, resolved_args, strict=False))
                _collect(origin, next_map)

    _collect(cls, {})
    return result


class TypedForm[T_Valid: Valid, T_Invalid: Invalid = Invalid](forms.Form):
    def result(self) -> T_Valid | T_Invalid:
        if not self.is_bound:
            return self._get_invalid_type()(**dict.fromkeys(self.fields))

        if self.is_valid():
            return self._get_valid_type()(**self.cleaned_data)

        # The non valid fields needs to be supplied to the invalid class.
        missing_data = dict.fromkeys(self.errors)

        return self._get_invalid_type()(**self.cleaned_data, **missing_data)

    @classmethod
    def _get_valid_type(cls) -> type[T_Valid]:
        return cls._get_types()[0]

    @classmethod
    def _get_invalid_type(cls) -> type[T_Invalid]:
        return cls._get_types()[1]

    @classmethod
    @functools.cache
    def _get_types(cls) -> tuple[type[T_Valid], type[T_Invalid]]:
        return _resolve_generic_type_args(cls)[TypedForm]

Basically what we do is we leverage fields and validation from Django forms to determine if the form is even valid to begin with. Then depending on if the form is valid or invalid we pass the cleaned data to one of two Pydantic models, one representing the valid data and one representing the invalid data (which can be partially validated). Pydantic is used as a sanity check of our cleaned_data to enforce that it matches the type definition. We solely rely on Django forms for our validations.

It might not be apparent exactly how to use this or why it is even helpful. Lets put this into practice using a simple example.

forms.py

from pydantic import BaseModel
from typed_form import TypedForm


class ValidRegistrationData(BaseModel):
    name: str
    email: str
    age: int


class MyRegistrationForm(TypedForm[ValidRegistrationData]):
    name = forms.CharField()
    email = forms.EmailField()
    age = forms.IntegerField()

views.py

from .forms import MyRegistrationForm

def create_user(*, name: str, email: str, age: int) -> User:
    pass

def user_registration(request: HttpRequest) -> HttpResponse:
    form = MyRegistrationForm(request.POST or None)

    if request.POST:
        result = form.result()
        if isinstance(result, ValidRegistrationData):
            # At this point we know that 'result' only contains valid data and
            # Pydantic has verified that all the data is present.
            # We can now access the validated fields and create our user.
            user = create_user(
                name=result.name,
                email=result.email,
                age=result.age
            )

            return render(
                request,
                "registration_success.html.dj",
                {"user": user}
            )

    return render(
        request,
        "registration_form.html.dj",
        {"form": form}
    )

So why do all of this? Let's say that in a few months we change the form to use DateField instead of a IntegerField for age. If we change our MyRegistrationForm form as we normally would, our tooling would not complain about the difference in data types since cleaned_data (which we would normally use) is typed as dict[str, Any]. But with the above solution MyPy will inform us, before runtime, that there is a mismatch between our Pydantic definition and the expected type of age on create_user.

This is a bit of a forced example. But hopefully you can see the benefits. Especially when you are dealing with more complex views and templates.

Why the Invalid type?

This is a small tangent away from what the blog post is really about.

With the above solution you would never have to touch the Invalid definition except for setting it to TypedForm.NoData. But one of the nice features of Django forms is that even when the form doesn't pass the validation we might still get some useful data from it. Any field that passes the validation will have its cleaned data accessible through cleaned_data even while the form is not valid. This, together with us moving to use HTMX for many of our more complex forms and views, helps us when we want to rerender our forms, based on the content of the form, even when it is still partially filled/valid.

In these cases we can use the Invalid type to still get some minor help with typing. We declare a new model/definition with the fields we are interested in in case of a failed validation. But the static typing won't know what fields or what type they are so we have to add all of these fields as the specific type and None in case the field was not valid. So when doing the following we would still get some type safety where MyPy informs us that result.birthday.year might be None and we will have to handle it.

# forms.py
class InvalidRegistrationData(BaseModel):
    birthday: datetime.date | None

class MyRegistrationForm(TypedForm[ValidRegistrationData, InvalidRegistrationData]):
    birthday = forms.DateField()
    # The rest of the form implementation

# views.py
def user_registration(request: HttpRequest) -> HttpResponse:
    form = MyRegistrationForm(request.POST or None)
    result = form.result()

    if isinstance(result, form.Invalid):
        print(result.birthday.year) # Mypy tells us results.birthday can be datetime.date | None.

Tangent over! But we might talk about this and how we use it with HTMX more in-depth in a later blog post.

Downsides

One of the potential downsides with this approach is the need to specify the expected/clean types as Pydantic models which results in more lines of code, and depending on the size of the form the models can grow quite large.

If you need full coverage of the fields between Valid and Invalid you will have some extra fields to declare since Invalid would have to be a copy of Valid but with | None for all fields. In TypeScript you could use Partial<T> to make all fields of Valid optional, but there doesn't exist a counterpart in Python yet, so we have to handle this extra type ourself. We have concluded that this is a downside we can live with.

Something similar to TypeScripts Partial<T> is being discussed and when/if it arrives we could easily remove the Invalid type altogether.

Improvements

This implementation requires us to keep the Valid and Invalid types in sync. There is a chance that, when adding or removing a field, the Valid and Invalid definitions can become "out of sync" where a field exists in one of the definitions and not in the other or their types don't match. To prevent this we have added validation that checks that all the fields of Invalid exists in Valid and that they are of the same types (excluding None). But I excluded that implementation from the blog post since there has been enough tangents already.

Load translations directly from source in Django

2024-12-18T00:00:00+01:00

Django's translation system is built on top of GNU gettext tooling. This tooling works with two types of translation files:

Portable Objects (POs), plain-text message catalogues. These are the files that developers and translators work with. Their plain-text form lends itself to source control, allowing generic tools for diffs and merges to work without further effort.
Message Objects (MOs), binary-compiled catalogues. These are built from the POs, typically at package build or deployment time. These are loaded by the application at runtime to provide the translations to the user. In Django, they are built using django-admin compilemessages.

Django currently loads translations only from MO files at runtime, meaning that django-admin compilemessages has to be rerun every time they are updated. This isn't very costly, but does represent an extra step that slows iterating on translations, checking them in the UI, and running tests that rely on them -- or can lead to confusion when the developer (especially me!) forgets to rerun it.

We've introduced a hack in our codebase that monkey-patches Django's translation code in order to have the development environment load the PO files at runtime instead -- and reload them if modified. This avoids the need for rerunning the compilation step manually:

License information

Copyright (c) Django Software Foundation and individual contributors.
Copyright (c) Personalkollen
All rights reserved.

Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:

    1. Redistributions of source code must retain the above copyright notice,
       this list of conditions and the following disclaimer.

    2. Redistributions in binary form must reproduce the above copyright
       notice, this list of conditions and the following disclaimer in the
       documentation and/or other materials provided with the distribution.

    3. Neither the name of Django nor the names of its contributors may be used
       to endorse or promote products derived from this software without
       specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

from gettext import GNUTranslations
from io import BytesIO
from pathlib import Path

from asgiref.local import Local
from django.utils.translation import trans_real
from django.utils.translation.trans_real import DjangoTranslation

# Package containing our code
import pk

SOURCES = [
    Path(pk.__file__).parent / "locale",
]


class PoTranslation(DjangoTranslation):
    """
    Hacky class which compiles .po files on the fly and reloads them
    when updated, allowing us to avoid compiling messages during
    development.
    """

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self._reload_cache = {}

    def check_cache(self):
        import polib

        """
        Reload translations if the PO files in question have changed
        (according to filesystem metadata) since the last reload.
        """
        for source in SOURCES:
            path = source / Path(f"{self.language()}/LC_MESSAGES/{self.domain}.po")
            if not path.exists():
                continue
            modified = path.stat().st_mtime
            if self._reload_cache.get(path, None) != modified:
                po = polib.pofile(path)
                mo = po.to_binary()
                catalog = GNUTranslations(BytesIO(mo))
                self.merge(catalog)
                self._reload_cache[path] = modified

    def gettext(self, message):
        self.check_cache()
        return super().gettext(message)


def install() -> None:
    """
    Monkey-patch Django's translation machinery to use the
    PoTranslation class which loads and compiles PO files (and
    recompiles them on-the-fly when updated) instead of loading
    precompiled .mo files once.
    """
    # Some inspiration taken from django.utils.translation.reloader.
    trans_real.DjangoTranslation = PoTranslation
    trans_real._translations.clear()
    trans_real._default = None
    trans_real._active = Local()

We then enable this in our development settings file (not in production!):

from pk.utils import dev_translation
dev_translation.install()

This has a number of rough edges, which we are fine with or have mitigations for, but are worth considering if you want to use this:

Paths to the PO files for auto-reload need to be specified manually. This is not a problem for our use case, as we know exactly where they live and this won't be changing.
This performs an unnecessary amount of filesystem access (a stat on the PO files for each translation string lookup). There's plenty of room for improvement here, but so far this has not made a measurable difference in how long our tests take to run or web application performance -- and we're only using this in development environment, so it won't affect production.
There is some reliance on implementation details of Django's translation machinery, as is to be expected with monkey-patching. If an update to Django breaks our assumptions, this should be caught by our test suite.
There is more divergence between dev and production code paths.

Overall, we think the reduction in mental load and hidden context makes these tradeoffs worth it (especially when onboarding new developers) -- each step we can remove between "clone our repo" and "have a working development environment" is valuable.

Drop fields safely in production

2024-11-19T14:55:00+01:00

Since Personalkollen runs on multiple servers/pods our users might experience some errors when dropping a field from production. This is because the migration is performed before the new pods have had time to deploy, so the production environment is technically running "old" code, that might be referencing the dropped field.

Here is what a a generated Django migration when dropping a field would look like.

# Generated by Django 5.0.9 on 2024-11-25 08:56

from django.db import migrations


class Migration(migrations.Migration):
    dependencies = [
        ("staff", "0005_add_staff_field_that_will_be_dropped"),
    ]

    operations = [
        migrations.RemoveField(
            model_name="staff",
            name="field_that_will_be_dropped",
        ),
    ]

Deploying this to production will almost certainly cause some errors for our users.

In order to counteract this, we always separate these types of migrations into two migrations and, most importantly, apply the migrations on different deploys.

The first migration drops the field from Django.

# Generated by Personalkollen staff on 2024-11-25 08:57

from django.db import migrations, models


class Migration(migrations.Migration):
    dependencies = [
        ("staff", "0005_add_staff_field_that_will_be_dropped"),
    ]

    operations = [
        migrations.SeparateDatabaseAndState(
            state_operations=[
                migrations.RemoveField(
                    model_name="staff",
                    name="field_that_will_be_dropped",
                ),
            ]
        ),
    ]

The second migration drops the field from the database.

# Generated by Personalkollen staff on 2024-11-25 08:58

from django.db import migrations


class Migration(migrations.Migration):
    dependencies = [
        ("salary", "0006_remove_staff_field_that_will_be_dropped_from_django"),
    ]

    operations = [
        migrations.SeparateDatabaseAndState(
            database_operations=[
                migrations.RunSQL('ALTER TABLE "staff_staff" DROP COLUMN "field_that_will_be_dropped" CASCADE;')
            ]
        ),
    ]