Django dumpdata and loaddata roundtrip does not work with EnumField

[kharybdys]

We're using EnumField with pure Enums, whose values are all strings.

This worked fine for the half year since we adopted it, but now we had a reason to try and make database snapshots.

Django's dumpdata command dumps the value of such a field as e.g. "PlyState.CANCELLED", which guarantees an error on using loaddata.

I would assume the dumped value is wrong. For now we're manually (bash script) adjusting these dumped values (removing everything before the .) and then loaddata works.

If you need more information let me know.

[bckohan]

Just a stab in the dark without trying to reproduce:

Try using a StrEnum if you are on python 3.11+. Or otherwise add a __str__ function that returns self.value. I bet that would fix it.

Please try and let me know.

I suspect you are encountering this because the default json serializer that dumpdata uses serializes Enum objects by calling str() or repr(). So the above suggestion should work for you. Alternatively I think the most appropriate place to fix this would be to provide a custom serializer that understands your objects. The suggestion above is certainly cleaner, but you might have to implement a serializer if you don't control the enum implementation.

I don't think there's a way to modify django-enum to make serialization work for arbitrary enum types. #126 might fix this, but glancing through the code it doesn't appear to me that it obviously would. Serialization should work out of the box when using IntEnum/StrEnum which should satisfy most use cases.

I would consider this mostly an upstream python issue. The default json serializer should serialize Enum types by calling enum.value. The fact that it doesn't has created problems for me before.

Here's a stab at a Django json serializer that understands enums:

# myapp/serializers/json.py
import json
import enum
from django.core.serializers.json import DjangoJSONEncoder
from django.core.serializers.json import Serializer as JSONSerializer


class EnumJSONEncoder(DjangoJSONEncoder):
    """
    Extends Django's JSON encoder to support Python Enum objects by
    returning their .value rather than their name or repr().
    """
    def default(self, obj):
        if isinstance(obj, enum.Enum):
            return super().default(obj.value)
        return super().default(obj)


class Serializer(JSONSerializer):
    """
    JSON serializer that uses EnumJSONEncoder to handle Enum values.
    """
    def end_serialization(self):
        json.dump(
            self.objects,
            self.stream,
            cls=EnumJSONEncoder,
            **self.json_kwargs,
        )

Alternatively, there's a handle_field method on Django's JSONSerializer that could also be overridden. That might be cleaner.

You would have to reference it directly when calling dumpdata:

python manage.py dumpdata --serializer myapp.serializers.json

[bckohan]

Actually it would be possible to fix this in django-enum by overriding value_from_object but I would want to fully understand the side effects before I did that. So in the meantime, see above.

#150

[kharybdys]

Adding a __str__ function that returns self.value fixes it. It's also an acceptable workaround to me.

Agreed that the basic problem is a serializer that doesn't use the fact something is an Enum.
If that's in pure Python, what is the path to try and get them to change this?

In that case I'm not sure you should add complexity to your library only to provide a workaround.

I really appreciate your fast response and constructive thinking!

[bckohan]

the python path would involve opening an issue with whoever owns the json encoder code. I suspect they may be very hesitant to change it at this point - but having looked through django's serialization code i see that the string conversion happens before the json encoder step so this wouldnt actually fix it - even though it would serialize it the same way Django is doing.

[kharybdys]

Which makes sense.

And convince Django to use a customized JSON encoder? Would only makes sense if django-enums gets merged into Django?

It's so annoying ;)

[bckohan]

Yeah - Django could update their encoder with a couple of lines that would make this corner case always work. I've thought about trying to merge django-enum into core but I suspect its doing more automagic stuff than many folks would be comfortable with. I may take a stab at getting true Enum database type field into core. The wrinkle there is that sqlite does not support them.

Also @kharybdys would you be so kind as to accept the answer at the top so other users may easily see it?