"System.Runtime.Serialization.SerializationException: Failed to deserialize object of type [Akka.Actor.Status+Failure] from the stream. Cause: Index was outside the bounds of the array." After Akka Upgrade from 1.3.14 to 1.4.48 #6760
Description
Version Information
Version of Akka.NET?
1.4.48
Which Akka.NET Modules?
Akka, Akka.Cluster, Akka.Remote, Akka.Cluster.Tools, Akka.Serialization.Hyperion, Hyperion
Describe the bug
After Akka upgrade from
Akka, Akka.Cluster, Akka.Remote, Akka.Cluster.Tools 1.3.14
Akka.Serialization.Hyperion 1.3.14-beta
Hyperion 0.9.12
To
_Akka, Akka.Cluster, Akka.Cluster.Tools, Akka.Remote 1.4.48
Akka.Serialization.Hyperion 1.4.21
Hyperion 0.10.2_
Service is getting this:
Deserialization failed for message with serializer id [-5] and manifest [:]. Transient association error (association remains live). Failed to deserialize object of type [Akka.Actor.Status+Failure] from the stream. Cause: Index was outside the bounds of the array. Could not find any internal Akka.NET serializer with Id [-5]. Please create an issue in our GitHub at [https://github.com/akkadotnet/akka.net]
System.Runtime.Serialization.SerializationException: Failed to deserialize object of type [Akka.Actor.Status+Failure] from the stream. Cause: Index was outside the bounds of the array.
---> System.IndexOutOfRangeException: Index was outside the bounds of the array.
at lambda_method(Closure , Stream , DeserializerSession )
at Hyperion.ValueSerializers.ObjectSerializer.ReadValue(Stream stream, DeserializerSession session)
--- End of inner exception stack trace ---
at Hyperion.ValueSerializers.ObjectSerializer.ReadValue(Stream stream, DeserializerSession session)
at Hyperion.Serializer.Deserialize[T](Stream stream)
at Akka.Serialization.HyperionSerializer.FromBinary(Byte[] bytes, Type type)
at Akka.Serialization.Serialization.Deserialize(Byte[] bytes, Int32 serializerId, String manifest)
at Akka.Remote.MessageSerializer.Deserialize(ExtendedActorSystem system, Payload messageProtocol)
at Akka.Remote.DefaultMessageDispatcher.Dispatch(IInternalActorRef recipient, Address recipientAddress, Payload message, IActorRef senderOption)
at Akka.Remote.EndpointReader.b__11_0(InboundPayload inbound)
To Reproduce
Steps to reproduce the behavior:
The service is in a cluster with another service, below is the second service Akka version
_Akka, Akka.Cluster, Akka.Cluster.Tools, Akka.Remote (1.4.10)
Akka.Serialization.Hyperion (1.3.11-beta)
Hyperion (0.9.16)_
The service is sending request to the second service Via Ask
Expected behavior
When the service send request to the second service it should get back a result
Actual behavior
it get back results 80% of the time, 25% of the time its ask timeout and the exception above("System.Runtime.Serialization.SerializationException: Failed to deserialize object of type [Akka.Actor.Status+Failure] from the stream. Cause: Index was outside the bounds of the array."). Before the upgrade it was getting the back result 99% of the time and the exception above is new
Environment
Docker, .NET Core 3.1
Additional context
Akka Configuration of the service
akka {
stdout-loglevel = INFO
loglevel = INFO
log-config-on-start = on
loggers=["Akka.Logger.Serilog.SerilogLogger, Akka.Logger.Serilog"]
extensions = ["Akka.Cluster.Tools.PublishSubscribe.DistributedPubSubExtensionProvider,Akka.Cluster.Tools"]
actor.provider = cluster
remote {
dot-netty.tcp
{
message-frame-size = 6MiB
send-buffer-size = 6MiB
receive-buffer-size = 6MiB
maximum-frame-size = 6MiB
port = *****
}
}
cluster {
roles = ["receptionist"]
allow-weakly-up-members = on
shutdown-after-unsuccessful-join-seed-nodes = 30s
downing-provider-class = "Akka.Cluster.SplitBrainResolver, Akka.Cluster"
split-brain-resolver {
active-strategy = keep-majority
keep-majority {
role = receptionist
}
stable-after = 20s
}
down-removal-margin = 20s
}
actor.serializers {
hyperion = "Akka.Serialization.HyperionSerializer, Akka.Serialization.Hyperion"
}
actor.serialization-bindings {
"Akka.IO.Client.ClientMessage, Akka.IO.Client" = hyperion
}
io.tcp.direct-buffer-pool.buffer-size = 262144
}