Re: Django Channels 2 poor performance and high CPU usage

Here is a quick and dirty example which is basically the core of what I'm trying to achieve.

https://github.com/jamesfoley/django-channels-test/

Only other requirements other than those in the requirements.txt is redis server. Clone repo, install python 3 requirements using pip, runserver, open two browser windows pointing to the server. You should be presented with a magenta cube which you can rotate, and the view will sync with the other open window.

For some reason it starts to bog down and either turn into a slideshow, or just be extremely delayed. It also occasionally just stops working and none of the websockets work anymore. Restarting the server does not bring back the websockets either, which I guess points to the redis layer.

Here is an example: https://imgur.com/a/qRJv8

It does run a hell of a lot smoother when on its own clean Ubuntu VM which is how I tested it, but I can also run it pretty well on the same Vagrant VM my main project is being developed on.

It seems like there is something project specific that is causing it to be unusable but I'm basically out of ideas on what to try next. The code in my project is almost identical to that in the example.

On Wednesday, 28 March 2018 09:27:01 UTC+1, James Foley wrote:

The project I'm working on is a little sensitive. 

I'll put together a test project and try and reproduce my issues in the same environment, then push it to a repo for you to take a look at.

On Tuesday, 27 March 2018 17:49:19 UTC+1, Andrew Godwin wrote:

Not getting past HANDSHAKING with the in-memory layer is a bit weird, and scope is just a normal Python dictionary. Is it possible to put your code up somewhere in a simple form so I can look over it?

Andrew

On Tue, Mar 27, 2018 at 4:11 AM, 'James Foley' via Django users <django...@googlegroups.com> wrote:

It looks like accessing scope for url kwargs is a big hit on performance. In fact hitting scope for anything seems to introduce some form of delay.

I'm getting very mixed results so I'm unsure if this is now an issue with redis. It is definitely the python process using 100% of the CPU though.

On Tuesday, 27 March 2018 09:51:44 UTC+1, James Foley wrote:

Apologies for the double post, I've removed the other one.

PYTHONASYNCIODEBUG doesn't appear to give me any warnings.

This test I am running is only between two users, each user belonging to the same group. All messages received are pushed back out to all users and filtered clientside.

Assuming I use 'channels.layers.InMemoryChannelLayer' as my backend for the memory layer, none of my websocket connections get past HANDSHAKING so I'm not exactly sure whats up there.

On Monday, 26 March 2018 17:35:34 UTC+1, Andrew Godwin wrote:

(You double-posted this so I'm just going to reply to this one)

I need to know a bit more information about what the slowdown is - in particular:

* Have you run with PYTHONASYNCIODEBUG=1 set as an environment variable to check for non-yielding coroutines?

* What sort of messages per second are we talking about? You say every 20ms, but how many listening on the group (so what does it multiply out to?)

* Do you get the same CPU usage issues if you try using the in-memory channel layer rather than the Redis one?

The last point would be especially interesting to check, as the node relay server you built does not, I imagine, use Redis as a cross-server transport in the middle and so that would be the major difference. 200 requests a second should still be fine, though, so the others are worth knowing about as well.

One further thing you could do is build a simple echo server with no use of groups or the channel layer at all and see how that performs, to narrow down where the performance issue lies.

Andrew

On Mon, Mar 26, 2018 at 6:15 AM, James <jamesrich...@gmail.com> wrote:

I'm using Channels 2 to build a shared 3D model viewing tool, but I'm running into performance issues where Channels can't keep up and uses 100% of a single core. This results in clients just receiving a slow trickle of messages rather than the fast stream I was expecting.

I ended up stripping my consumer all the way back to basically a relay server, so a client sends a message, server broadcasts the exact data received to a group. I am sending a lot of data though (a message every 20ms or so) so not sure if that is causing my issues.

Using node I built the same relay server and I have zero issues with speed or server performance.

Is this down to how many workers I have running vs the data I'm sending, or perhaps the overhead of Django?

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/fecbabcc-2692-4d7f-ac6f-6de7c3631354%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/a18867f0-23b2-49fc-a2f3-becc88971052%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/69306254-1640-42e8-bdf5-68aca5e8c8a5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.