Connection timeouts on high load

Hello,

I have an application made with Django 1.7 and the Django Rest
Framework and I'm in the phase of load testing it. My setup is made of
3 servers:

- Nginx + gunicorn
- Gunicorn
- Postgresql + Memcached

Nginx is configured as a loadbalancer so I can add more gunicorn
instances if needed. In the current setup it already balances the load
between its own gunicorn instance and the other server with the
gunicorn instance. The project code is on the loadbalancer and it's
shared on the gunicorn server via an NFS share.

I'm using Locustio to simulate about 4000 users, that make a request
every 30-60 seconds, with about 200 new users per second. The servers
are handling the load until I get to about 2000 users, and then nginx
starts returning 502 and 504 errors with the following in the logs:

upstream timed out (110: Connection timed out) while connecting to upstream
recv() failed (104: Connection reset by peer) while reading response
header from upstream

I know most of the load is caused by the (high) hatch rate, since
every new simulated user goes through an account creation phase, which
I guess is quite slow because of the password hashing. But even when
all users have been created and are just doing regular requests, I
keep getting a failure rate of about 6% (with the same errors as
before), even though the gunicorn workers have plenty of CPU to use. I
checked the I/O (both network and disk) and there's no problem here.

Do you have any advice on where I could look or what tool I could use
to have an idea of what's going on?

Here's my gunicorn config:

bind = "0.0.0.0:8000"
workers = 8
preload_app = True
loglevel = 'info'
pidfile = '/home/myproject/gunicorn.pid'

And my nginx config:

upstream myproject {
    server 127.0.0.1:8000 weight=2 fail_timeout=0;
    server ip_of_gunicorn_server:8000 fail_timeout=0;
}

proxy_pass http://myproject;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;

proxy_read_timeout 300;
proxy_redirect off;

Thank you for your help,
Sylvain

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/3fcb77d5-b1b5-4958-abe6-0ac80fa62970%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.