channels_redis & async_to_sync performance problem

Hello,

I have a worker process that uses async_to_sync(group_send) to send messages to my Channels consumers. The worker process is a simple long-running synchronous loop (not based on Django Channels) that is started via a Django management command.

When I use runserver and the in-memory channel layer, performance is great. But when I use channels_redis in production, each call to group_send takes 2 seconds, because each time it needs to create a new redis connection. Here is the call stack:

  File "c:\my_project\core\my_project\channels\utils.py", line 14, in sync_group_send_wrapper
    return _sync_group_send(group, {'type': type, **event})
  File "c:\my_project\ve-dj2\lib\site-packages\asgiref\sync.py", line 79, in __call__
    return call_result.result()
  File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\concurrent\futures\_base.py", line 428, in result
    return self.__get_result()
  File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\concurrent\futures\_base.py", line 384, in __get_result
    raise self._exception
  File "c:\my_project\ve-dj2\lib\site-packages\asgiref\sync.py", line 98, in main_wrap
    result = await self.awaitable(*args, **kwargs)
  File "c:\my_project\ve-dj2\lib\site-packages\channels_redis\core.py", line 625, in group_send
    async with self.connection(self.consistent_hash(group)) as connection:
  File "c:\my_project\ve-dj2\lib\site-packages\channels_redis\core.py", line 839, in __aenter__
    self.conn = await self.pool.pop()
  File "c:\my_project\ve-dj2\lib\site-packages\channels_redis\core.py", line , in pop
    conns.append(await aioredis.create_redis(**self.host, loop=loop))

This is happening because the connection is deleted after each call to async_to_sync:

  File "c:\my_project\core\my_project\channels\utils.py", line 14, in sync_group_send_wrapper
    return _sync_group_send(group, {'type': type, **event})
  File "c:\my_project\ve-dj2\lib\site-packages\asgiref\sync.py", line 71, in __call__
    loop.close()
  File "c:\my_project\ve-dj2\lib\site-packages\channels_redis\core.py", line 32, in _wrapper
    self.run_until_complete(pool.close_loop(self))
  File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\asyncio\base_events.py", line 579, in run_until_complete
    return future.result()
  File "c:\my_project\ve-dj2\lib\site-packages\channels_redis\core.py", line , in close_loop
    del self.conn_map[loop]

When async_to_sync is called from a synchronous Django view, the connection is NOT deleted. That's because in my worker process, self.main_event_loop.is_running() is False, but in my Django views, it's True. That affects this if-statement in async_to_sync.__call__:

        if not (self.main_event_loop and self.main_event_loop.is_running()):
            # Redis connection gets deleted inside here...

So, how could I solve this? This may be an obvious question but I don't know asyncio well; I just want to get some guidance before I spend too much time looking in the wrong direction.

Thank you!

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/afa1c36a-bdb5-40b7-a602-ee20d7949ac2%40googlegroups.com.