Monday, March 2, 2020

Re: How to debug Daphne CPU 100% issue

Thanks very much for your reply.

Which make me confused now the CPU of Daphne is normal, keep 8% ~20%. I update the new info and new check here. I think it will return to 100% because I have not found the reason and do nothing.

Environment:
  • OS: Ubuntu 18.04.1 LTS x86_64 GNU/Linux
  • Python: 3.6.5
  • Django: 2.1.1
  • Daphne: 2.3.0
  • Channels: 2.2.0
  • channels-redis: 2.4.0
Check Info:
  1. check the thread numbers with ps -o thcount 22549, which shows 45 ( > 40)
  2. check the websocket connected numbers with redis cli: ZCOUNT asgi::group:USER_MESSAGE -inf +inf, which shows 160  ( < 190)
  3. check the system call with sudo strace -c -p 22549 for a minute, the result as the following:

daphne_trace.png

New Check Info:
  1. check the system files open limits with : sudo sysctl -p | grep file, which shows: 65535
  2. check process files open limits with: cat /proc/22549/limits | grep -E 'files | Soft', which shows: 

files_limits.png


           In fact, I set the system and process open files(with supervisord conf) limit long long ago.
       3. check the process open files number with ls -l /proc/22549/fd |  wc -l: 145
       4. check the cpu and io stat with iostat -ctz:

iostat.png



       5. check the net stat with netstat -ano | grep daphne.sock (because ubuntu do not support the b parameter): which show 133, 1 is listening status, 132 is connected status.



在 2020年3月2日星期一 UTC+8下午7:40:11,Integr@te System写道:
Hi Fogmoon,

Plz show again your image file's result and addition check 
iostat -ctz
netstat -anob

to see more root cause.

Dont for get update python version.

Nice.


On Mon, Mar 2, 2020, 13:32 Fogmoon <ifog...@gmail.com> wrote:
Hi, 

Could you please give some suggestions to inspect the high CPU (most time above 90% and some time above 100%) consumed by Daphne. 

Environment:
  • OS: Ubuntu 18.04.1 LTS x86_64 GNU/Linux
  • Django: 2.1.1
  • Daphne: 2.3.0
  • Channels: 2.2.0
  • channels-redis: 2.4.0
Just use the Daphne as the websocket server with redis as the Channel layer. 

Now I do the following steps:
  1. top command show the daphne process consume high cpu
  2. check the thread numbers with ps -o thcount 22549, which shows 40. 
  3. check the websocket connected numbers with redis cli: ZCOUNT asgi::group:USER_MESSAGE -inf +inf, which shows 190
  4. check the system call with sudo strace -c -p 22549 for a minute, the result as the following:

daphne_cpu.png
Best Wishes.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/913143a8-c8c8-47ce-a96d-6c9d349eac6a%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/9120a9df-4879-4769-bf91-c2c6c0bf3642%40googlegroups.com.

No comments:

Post a Comment