Redis subscriber connect on disconnect handler doesn't work (or at least wouldn't handle the current subscribed callbacks)

[sirilvk]

Hi,

I am heavily dependent on the redis subscriber and recently while listening to a publish from python script my redis subscriber gets disconnected and the re-connection (from the logs) seems going fine but I stop receiving any callbacks from the subscriber. I tried applying the changes from #73 but still it doesn't work. Please help.

Thanks,
Siril.

[Cylix]

Hi,

May I know which version of cpp_redis you are using?
Seems like you are not using the latest version if you had to apply fix of #73 ?

Technically latest version ensure resubscription of subscribed channels on reconnect (see here).

If you are using the latest version, there might be a bug under the hood.
Otherwise, you may need to upgrade or to manually resubscribe to channels on reconnection.

Best

[sirilvk]

HI Cylix,

Thanks for the reply. I tried a simple program with the master code :

#define __CPP_REDIS_LOGGING_ENABLED
#include <cpp_redis/cpp_redis>
#include <iostream>
#include <fstream>

int main()
{
    cpp_redis::active_logger = std::unique_ptr<cpp_redis::logger>(new cpp_redis::logger(cpp_redis::logger::log_level::debug));

    __CPP_REDIS_LOG(debug, "Testing the logging");
    cpp_redis::subscriber sub;
    sub.connect("192.168.10.24", 6379, nullptr, 1, 20, 2);

    sub.subscribe("summary", [&](const std::string& ch, const std::string& msg){
            std::cout << msg << std::endl;
        }).commit();

    while(true);
}

and started pumping some messages using pyredis over redis:

import redis
from time import sleep
import signal
import sys

exitProg = True

def signal_handler(signal, frame):
    global exitProg
    print('Exiting loop')
    exitProg = False

signal.signal(signal.SIGINT, signal_handler)
vslist = []
with open('summary.json') as vs:
    vslist = vs.readlines()
    
l = redis.StrictRedis(host='localhost', port=6379, socket_keepalive=True)
print("Publishing summary now")
while exitProg:
	for vs in vslist:
		l.publish('summary', vs)
	sleep(2)

And another pyredis for subscribe which works well:

import redis
from time import sleep
import signal
import sys
import common
import time
import json

if len(sys.argv) != 2:
    print("Usage : " + sys.argv[0] + " <host ip>" )
    sys.exit(1)

exitProg = True
hostIP = sys.argv[1]
chname = sys.argv[2]

def signal_handler(signal, frame):
    print('Exiting loop')
    common.exitProg = False

def subscribe_channel(rRedis, channelname):
    p = rRedis.pubsub()
    p.subscribe(channelname)
    while common.exitProg:
        try:
            response = p.get_message()
            if response != None:
                if response['type'] == 'message':
                    jval = json.loads(response['data'])
                    print(json.dumps(jval, sort_keys=True, indent=4))
        except:
            continue

signal.signal(signal.SIGINT, signal_handler)

# persist tables now
r = redis.StrictRedis(host=hostIP, port=6379)
subscribe_channel(r, 'summary')

The include common.py is :

exitProg = True

The thing is the python script receives all the messages and doesn't reconnect. Where as the cpp one which is built over your library (g++ -Iincludes/ -Itacopie/includes/ main.cpp -Llib/ -lcpp_redis -ltacopie -lpthread) gets only the first set of messages (broadcasted from the file) and doesn't get anymore. It doesn't even log any messages. Can you please help. The summary.json file can be any random json text for this test.

[sirilvk]

Any update on this?

[Cylix]

Hi @sirilvk,

I didn't have the time to look into yet.
I'll post some updates here when I will dig into it, but I can't give you any timeline.
Sorry for the delay.

Best

[Cylix]

Hi,

Sorry for the very long time to check this issue.
I am splitting my time between different open source projects and kind of hard to give the appropriate time to each project.

I tried your example on my side (mac os, cpp11, latest version) and didn't get any issue.
I force the disconnection and got reconnected successfully.

From your description, it seems that the subscriber of cpp_redis got disconnected while the python one does not. That would explain why part of the messages are not received (if you are not subscribing while a message is published, there is no way to retrieve it later on when reconnecting).

The thing I am more wondering is the reason of the disconnection.
Either the client decided to disconnect or either the redis-server closed the connection.
From your description, the python client is not disconnected, so there might be two cases:

• are the python and the cpp_redis client running on the same machine?
• if so, is it disconnecting every time at the same message (let's say you PUBLISH A, B, C and D and it always disconnect after receiving B)?
• if so, can you share your summary.json file? There is maybe a parsing issue causing the client to think that the data published by the server is invalid. In such case, the client will close the connection.

Thanks and sorry again for the delay,

Best

[sirilvk]

Hi Cylix,

Thanks for the reply and I greatly appreciate your trying to reproduce the same. I think the problem is with the size of the json message thats being published. For smaller messages it works just fine. I built a huge json message from https://www.json-generator.com/ mentioning a repeat of 1000 times. And also in the python publish script publishing it over a range of 100 (repeat 100 times). For this one the Cpp just prints once and then totally stops getting any messages where as python subscribe script gets all the messages. So is there any limitations for the message size that cpp_redis can handle?

The modified python publishing script:

import redis
from time import sleep
import signal
import sys

exitProg = True

def signal_handler(signal, frame):
    global exitProg
    print('Exiting loop')
    exitProg = False

signal.signal(signal.SIGINT, signal_handler)
vslist = []
with open('summary.json') as vs:
    vslist = vs.readlines()

l = redis.StrictRedis(host='localhost', port=6379, socket_keepalive=True)
print("Publishing summary now")
while exitProg:
        for vs in vslist:
            for i in range(100):
                l.publish('summary', vs)
        sleep(2)

Actually we use huge json objects inside our application and couldn't get hold of a sample message currently. Sorry about that. But the issue is happening for json from that link.

Btw am running this on the same host (Ubuntu Server 17.10). And also I don't see any log messages once the cpp process stops receiving (it just goes blank) and so am not sure if its still connected.

However Thanks once again for the reply.

[Cylix]

Thanks, there might be an issue under thehood.

I had a try using redis-cli, but turns out redis-cli limits the size of the published message, so I am unable to put the entire json data.
I'll have a try with python script ASAP.

Best

[Cylix]

Using your script & a huge json, I was able to reproduce a disconnection from the redis client.
I will investigate the reason.

EDIT:
From the log, a format error is received:
[ERROR][cpp_redis][/Users/cylix/personal_projects/cpp_redis/sources/builders/bulk_string_builder.cpp:71] cpp_redis::builders::bulk_string_builder receives invalid ending sequence

I would need to see if there is indeed a format error (which does not sound to be the case if the python client can handle it).

[Cylix]

Ok, so concerning this issue:

Concerning the disconnection, this is triggered by the redis-server. I am not sure why it does so.

When reconnecting, the subsequent request is malformed due to the internal buffer to build server replies not being fully cleaned up.
When cleaning up accordingly, it reconnects and retrieves the additional published messages perfectly fine.
Still, there is some disconnection happening after receiving some of the messages leading to a reconnection.

I will push a fix concerning the buffer cleanup tonight (CA time).

Thanks again for reporting this issue!

[Cylix]

I pushed a fix concerning the reconnection process.

For the cause of the disconnection, I think the reason is that the server socket buffer is overflowed by the amount of data published and the client does not keep the pace.
This is highly visible: if you do not print the entire message received but just something like std::cout << "receive message" << std::endl;, the number of disconnection is lot less frequent.

If you want to limit the number of disconnection, for now a solution would be to do your processing in a background thread in order to release the callback and to quickly go back read new data from the socket.

Additionally, I would need to see what improvement could be made on our side. Right now, when the response is being built, no data is read until it is done processing. I can moving that to the background would solve the issue, but that would require some critical changes to the library, so this may take longer to implement.

I close this issue for now and will create another one concerning the remaning problem itself.
However, feel free to re-open this one if necessary.

Best

[sirilvk]

Thanks. It makes sense to handle it on a different thread.