Redis Connection Error running the scheduler after a certain while

[mallowigi]

We are using the resque scheduler for a long time in our company and for a long time we ran without any issues.

However since around two weeks we are unable to run the scheduler and are getting this error:

Redis::ConnectionError: Server closed the connection (redis://****) (Redis::ConnectionError)
/usr/local/bundle/gems/hiredis-client-0.23.2/lib/redis_client/hiredis_connection.rb:92:in 'RedisClient::HiredisConnection#_read'
/usr/local/bundle/gems/hiredis-client-0.23.2/lib/redis_client/hiredis_connection.rb:92:in 'RedisClient::HiredisConnection#read'
/usr/local/bundle/gems/redis-client-0.23.2/lib/redis_client/connection_mixin.rb:52:in 'block in RedisClient::ConnectionMixin#call_pipelined'
/usr/local/bundle/gems/redis-client-0.23.2/lib/redis_client/connection_mixin.rb:50:in 'RedisClient::ConnectionMixin#call_pipelined'
/usr/local/bundle/gems/redis-client-0.23.2/lib/redis_client.rb:481:in 'block (2 levels) in RedisClient#multi'
/usr/local/bundle/gems/redis-client-0.23.2/lib/redis_client/middlewares.rb:16:in 'RedisClient::BasicMiddleware#call'
/usr/local/bundle/gems/opentelemetry-instrumentation-redis-0.25.7/lib/opentelemetry/instrumentation/redis/middlewares/redis_client.rb:37:in 'block in OpenTelemetry::Instrumentation::Redis::Middlewares::RedisClientInstrumentation#call_pipelined'
/usr/local/bundle/gems/opentelemetry-api-1.4.0/lib/opentelemetry/trace/tracer.rb:37:in 'block in OpenTelemetry::Trace::Tracer#in_span'
/usr/local/bundle/gems/opentelemetry-api-1.4.0/lib/opentelemetry/trace.rb:70:in 'block in OpenTelemetry::Trace#with_span'
/usr/local/bundle/gems/opentelemetry-api-1.4.0/lib/opentelemetry/context.rb:87:in 'OpenTelemetry::Context.with_value'
/usr/local/bundle/gems/opentelemetry-api-1.4.0/lib/opentelemetry/trace.rb:70:in 'OpenTelemetry::Trace#with_span'
/usr/local/bundle/gems/opentelemetry-api-1.4.0/lib/opentelemetry/trace/tracer.rb:37:in 'OpenTelemetry::Trace::Tracer#in_span'
/usr/local/bundle/gems/opentelemetry-instrumentation-redis-0.25.7/lib/opentelemetry/instrumentation/redis/middlewares/redis_client.rb:36:in 'OpenTelemetry::Instrumentation::Redis::Middlewares::RedisClientInstrumentation#call_pipelined'
/usr/local/bundle/bundler/gems/prometheus-ruby-exporter-bc9c2c546fac/lib/prometheus_exporter/instrumentation/method_profiler.rb:63:in 'PrometheusExporter::Middleware::RedisInstrumenter#call_pipelined'

We are using Redis Labs but I don't think this is related, as this doesn't happen all the time, but only after a certain time.

After a lot of investigation I found out that what was causing this error was the zset delayed_queue_schedule which was filling up and never expired. Once I removed the key everything went back as expected, only for it to happen randomly a few hours later.

I was wondering if I can get any sort of help. Obviously I don't want to have to trigger the scheduler manually every few hours, it would be against the point.

Thanks in advance

[mallowigi]

Is there a way to force that key to have a TTL maybe?

[mallowigi]

Update (for whoever cares): the problem was that scheduler was saving in the timestamps key the serialization of the Job alongside it's args. Since we have jobs whose args can be large objects, in some instances the key becomes too big and the scheduler fails to parse the job, thus preventing any other jobs to proceed. Since it's too big, you can't even delete it forcibly.