Allow turning a few optimisations off for Kryo 5.4.0

[chrisr3]

We are upgrading our code-base from Kryo 4.0.2 to 5.4.0, and have hit an issue when serialising a Collection that only contains instances of Collections$UnmodifiableRandomAccessList.

We have sub-classed Kryo to support serialising classes with writeResolve and readReplace methods; UnmodifiableRandomAccessList implements writeResolve like this:

private Object writeReplace() {
    return new UnmodifiableList<>(list);
}

Kryo 5's CollectionSerializer seems to have an optimisation for the case when all of a collection's elements are of the same type, whereby Kryo serializes the element type once, and then serializes all of the elements' object data afterwards. Unfortunately, UnmodifiableRandomAccessList.writeResolve transforms the elements into instances of UnmodifiableList, but Kryo is unable to discover this until it has invoked writeResolve at least once. This happens after it has incorrectly written UnmodifiableRandomAccessList as the element type to the output stream, of course.

I cannot see a way of working around this apart from ignoring UnmodifiableRandomAccessList.writeResolve, although I cannot ignore every writeResolve method because that would break lambda serialization. Ideally, I'd like to be able to configure CollectionsSerializer not to apply this optimisation at all.

[theigl]

Hi @chrisr3. Could you create a minimal reproducer (ideally a unit test) for the issue you are facing?

I'm open to adding a flag to CollectionSerializer, but there might be other solutions.

[chrisr3]

This sample project should demonstrate what is happening: kryo-example.tar.gz

[theigl]

OK thanks!

I quickly glanced over the code:

Do you really need to globally override Kryo's behavior? Or would it be enough to create a custom serializer that uses writeResolve/readReplace internally and register it only for the classes where you actually want this behavior?

Lambda serialization is handled by ClosureSerializer that uses readResolve/writeReplace internally. So lambda serialization won't break if you change your global serialization logic.

[chrisr3]

I think the problem there is that any class (including unknown "third party" ones) can potentially implement readReplace or writeResolve, so we cannot possibly know them all in advance.

[theigl]

So you are serializing arbitrary classes in your case and have no control over which classes are serialized?

We could add a flag to CollectionSerializer that disables the optimization, but I'm not sure if any other Kryo user will ever need it.

If you want, you can create a PR for this and we can discuss it further.

[chrisr3]

We could add a flag to CollectionSerializer that disables the optimization, but I'm not sure if any other Kryo user will ever need it.

Maybe a better and more general solution would be to allow us to override Kryo's built-in CollectionSerializer with one of our own?

[theigl]

Can you try this:

addDefaultSerializer(Collection.class, MyCollectionSerializer.class);

I'm not 100% sure about the ordering of serializers right now, but this might work.

[chrisr3]

Hmm, interesting. I wouldn't expect that to work because the built-in CollectionSerializer is already registered for Collection.class and so wouldn't MyCollectionSerialzier have lower precedence?

Is there a detailed explanation anywhere of how Kryo determines serializer precedence please?

[theigl]

Default serializers registered after the built-in serializers should take priority. I just did a quick test and it looks like MyCollectionSerializer is chosen over CollectionSerializer. Please give it a try.

[chrisr3]

Yes, I agree. However, I have now discovered that my example code is actually testing the ObjectArraySerializer and not the CollectionSerializer 🤦. It looks like both of these default serializers have this optimisation, and hence will need overriding. Are there any others I need to worry about please?

---

I have successfully added MyCollectionSerializer, but I also needed to re-add some other built-in serializers in order to prevent it from serializing all collection types:

addDefaultSerializer(EnumSet.class, EnumSetSerializer.class);
addDefaultSerializer(Collections.EMPTY_LIST.getClass(), CollectionsEmptyListSerializer.class);
addDefaultSerializer(Collections.EMPTY_SET.getClass(), CollectionsEmptySetSerializer.class);
addDefaultSerializer(Collections.singletonList(null).getClass(), CollectionsSingletonListSerializer.class);
addDefaultSerializer(Collections.singleton(null).getClass(), CollectionsSingletonSetSerializer.class);
addDefaultSerializer(TreeSet.class, TreeSetSerializer.class);
addDefaultSerializer(Collection.class, MyCollectionSerializer.class);

That's a little bit arcane... 🧙

[theigl]

Yes, I agree. However, I have now discovered that my example code is actually testing the ObjectArraySerializer and not the CollectionSerializer 🤦. It looks like both of these default serializers have this optimisation, and hence will need overriding. Are there any others I need to worry about please?

I think these two are the only serializers that use this optimization.

I have successfully added MyCollectionSerializer, but I also needed to re-add some other built-in serializers in order to prevent it from serializing all collection types:

OK this is unfortunate. It might sense to add something like Kryo.replaceDefaultSerializer().

In the meantime, you could try this:

Kryo kryo = new Kryo() {
    public Serializer getDefaultSerializer(Class type) {
        final Serializer serializer = super.getDefaultSerializer(type);
        if (serializer instanceof CollectionSerializer<?>) {
            return new MyCollectionSerializer();
        }
        return serializer;
    }
};

[chrisr3]

Oh dear, the CollectionSerializer has a few sub-classes!

• ArraysAsListSerializer
• CopyForIterateCollectionSerializer
• JdkImmutableListSerializer
• JdkImmutableSetSerializer
• PriorityQueueSerializer
• TreeSetSerializer

All of these will need extra handling too if I replace CollectionSerializer with MyCollectionSerializer... 👷‍♂️.

Maybe explicit re-registration isn't so much worse after all?

[theigl]

OK, this looks like a dead end.

It seems that in order to support your use case, we would indeed need some configuration options for both CollectionSerializer and ObjectArraySerializer.

Then you could override getDefaultSerializer and set the flag on these serializers and their sub-classes.

Do you want to create a PR? I'm still not sure about this, because you are probably the only one who needs this feature. But if we can think of a really good name for the flags and the code changes are minimal, I might merge it.

[chrisr3]

OK, I'll try that. I've also experimented with a CollectionSerializerAdapter that can be applied via:

Kryo kryo = new Kryo() {
    public Serializer getDefaultSerializer(Class type) {
        final Serializer serializer = super.getDefaultSerializer(type);
        if (serializer instanceof CollectionSerializer<?>) {
            return new CollectionSerializerAdapter((CollectionSerializer<?>) serializer);
        } else {
            return serializer;
        }
    }
};

although this has the following drawbacks:

• The writeHeader and create methods are protected, and so must be invoked via reflection.
• Some CollectionSerializer sub-classes override the read and write methods too.

I'll push this idea a bit further, just to see how terrible it really is...