RWPinger unable to find a RW server when KazooClient is connected to a read only server.

[linsite]

There is a 3-nodes ZK cluster, i.e. A B C. A is in read only mode because of network problem, B C are working well.
And a kazoo client is configured with A B C, and it's connected to A.

Expected Behavior

The instance can find B or C, recover its ability to write something.

Actual Behavior

It can't find B C. According to the DEBUG log, it stopped to try B or C when it found A was 'ro'.

Snippet to Reproduce the Problem

Accoding the client.py code, the _ro_mode property only is recreated when a connection retry takes place. If the current connection connected to a read only server is table, it will never retry for a new connection, so the POC code below will show that it will not find the RW servers.

# -*- coding: utf-8 -*-
# Time:2021-08-10 19:36

import sys
import eventlet
eventlet.monkey_patch()

from contextlib import contextmanager
import socket
import select

import logging

LOG_FILE_NAME = 'test.log'
LOG_FORMAT = ('%(asctime)s %(filename)s[line:%(lineno)d]'
              ' %(levelname)s %(message)s')
DATE_FORMAT = '%Y-%m-%d  %H:%M:%S %a'
stream = logging.StreamHandler()
logging.basicConfig(level=logging.DEBUG, format=LOG_FORMAT, datefmt=DATE_FORMAT,
                    stream=sys.stderr)


console = logging.StreamHandler()
console.setLevel(logging.DEBUG)
formatter = logging.Formatter(LOG_FORMAT)
console.setFormatter(formatter)

LOG = logging.getLogger(__name__)
LOG.addHandler(console)

from kazoo.protocol.connection import RWPinger

from kazoo.handlers.eventlet import SequentialEventletHandler


class ErrorHandler(object):

    @contextmanager
    def _socket_error_handling(self):
        try:
            yield
        except (socket.error, select.error) as e:
            err = getattr(e, 'strerror', e)
            raise ConnectionDropped("socket connection error: %s" % (err,))

handler = SequentialEventletHandler()
eh = ErrorHandler()

hosts = [('zk1', 12181), ('zk2',12181), ('zk3', 12181)]

def test():

    handler.start()
    pinger = RWPinger(hosts, handler.create_connection, eh._socket_error_handling)
    p = iter(pinger)

    while True:
        ret = next(p)
        if ret:
            print(ret)
        else:
            print("not find writable %s" % ret)

        eventlet.sleep(1)


eventlet.spawn(test)

while True:
    eventlet.sleep(1)

Logs with logging in DEBUG mode

not find writable False
2021-08-10  20:00:27 Tue connection.py[line:121] DEBUG Pinging server for r/w: zk3:12181
not find writable False
2021-08-10  20:00:28 Tue connection.py[line:121] DEBUG Pinging server for r/w: zk3:12181
not find writable False
2021-08-10  20:00:29 Tue connection.py[line:121] DEBUG Pinging server for r/w: zk3:12181
not find writable False
2021-08-10  20:00:30 Tue connection.py[line:121] DEBUG Pinging server for r/w: zk3:12181
not find writable False
2021-08-10  20:00:31 Tue connection.py[line:121] DEBUG Pinging server for r/w: zk3:12181
not find writable False

Specifications

• Kazoo version: 2.7
• Result of pip list command:
• Zookeeper version: 3.5.9
• Zookeeper configuration: put here any useful ZK configuration (authentication, encryption, number of ZK members, number of (concurrent?) clients, Java version, krb5 version, etc.) 3 nodes
• Python version: 2.7.12
• OS: Linux