Repository navigation
What is the best way to implement automatic lost connection recovery with socketio.AsyncClient ? #232
Description
I'm developing a system where a socketio.AsyncClient client needs to be "always" connected to it's server (an socketio.AsyncServer).
Of course, the connection will crash once in a while, because there is a network (and an Nginx proxy) between them, and these things have interruptions or crashes once in a while.
I'm actualy noticing that interruptions are not at all rare, and it's also a possibility that my client will sometimes get starved of resources and not be able to "ping" the server for 25 seconds (as described #83 ), from looking at the logs it appears that it is happening at least once a day.
So I would like to have my client recover from connection failures by detecting them and reconnecting, and I was wondering what was the best way to do this, (i.e. which exceptions should be caught etc). The client looks more or less like this :
async def trigger_event(self, event, *args):
"...process the incoming event"
def start(self):
async def start_client():
await self.socketio_client.connect(self.url)
while True:
for m in self.ui_janitor.messages_for_round():
await self.client.emit('broadcast', m)
await asyncio.sleep("...x seconds, based on some throttling logic")
loop = asyncio.get_event_loop()
loop.run_until_complete(start_client())
As can be seen in this pseudo code, the client is both reacting to events that gets sent to it, and emiting messages from an infinite janitor / deamon loop.
I'm thinking that something like the pseudo code below could do, but this kind of approach will only work if I catch the right exceptions, and I should also not be catching the ones that are not recoverable, or not raised as a result of connection failures, and I should probably intruduce some backoff, sleep and retry logic.
Any advice on how to properly implement a "quasi permenant" / "recovering" connection would be appreciated.
def start(self):
async def start_client():
await self.socketio_client.connect(self.url)
while True:
for m in self.ui_janitor.messages_for_round():
await self.client.emit('broadcast', m)
await asyncio.sleep("...x seconds, based on some throttling logic")
while True:
try:
loop = asyncio.get_event_loop()
loop.run_until_complete(start_client())
except engineio.exceptions.ConnectionError ex:
logger.info(f"will reconnect...")
Activity
miguelgrinberg commentedon Jun 25, 2021
The client includes logic to reconnect. I don't understand why you need to build another re-connection layer on top of that.
You can't have an async client that blocks. Use the standard client if you are doing blocking operations.
Also, note that current releases of the Socket.IO client and server have reversed the ping/pong mechanism. Now it is the server sending pings and the clients responding with pongs.
max-l commentedon Jun 25, 2021
I wasn't aware that there was a reconnection mechanism.
I guess it's not working for me then !
Is reading from the file system a blocking operation ? Cause that would be the only blocking going on in my code, and I only read small files, could it cause problems ?
miguelgrinberg commentedon Jun 25, 2021
Reading from files is blocking, but small files is usually fine as long as you aren't constantly doing it. I suggest you try some of the examples in the python-socketio repository, and test the reconnection by killing the server and then restarting it. Once you verify that this is working with the official examples, do the same with your project and if it doesn't work capture logs and add them here.