Re: Scaling django (nginx + apache + mod_wsgi + postgresql)

Thank you so much for tips. I will keep them when I start to test environment for performance.

Cheers

Isaac

On 10/29/2012 06:44 PM, Cal Leeming [Simplicity Media Ltd] wrote:

Hi Isaac,

If there is one thing I have learnt about scaling apps, it's about trying things out for yourself.

Sure there are some best practice guidelines (i.e. serving files from nginx, or using apache's X-SendFile rather than streaming out via the webapp), but if someone comes along and tells you to use X instead of Y, then you don't get the advantages of learning "the hard way".

Another important note is that scaling rarely goes up on a 1:1 ratio, i.e. the configuration and resources required to handle X number of requests/sec, may be completely different if you need to handle Y number of requests/sec.

And often what works for one person, won't work for another (scaling is entirely dependant on your application, despite what any of these cloud providers might tell you!)

In my own experience, I've found that;

* SSDs with Percona MySQL, resolves a LOT of performance problems - but don't abuse it

* Lots and lots and lots of query tuning and InnoDB tuning

* New Relic to identify bottlenecks

* IO contention is a big thing

* Snowball prevention (i.e. you set max clients to X, your backend can't handle it, your requests stack up, the load balancer forces time out, and your database gets smashed - or you set max memory too high, server goes into swap etc).

* uWSGI + nginx is amazing

* Identify where your bottlenecks are (in my own experience, IO/memory tends to come up more often than CPU)

Sadly I haven't tried PSQL so I can't offer any advice on this - Percona are dragging MySQL kicking and screaming into the 21st century and really doing some amazing things, but it's by no means perfect!

The above has helped us grow past 8k-12k requests/minute, the largest database we manage is around 1.1 billion rows weighing in at 160GB+, and we maintain around 60+ servers.

I should reiterate, the above is purely based on my own experience and use cases - I am by no means an expert on the subject and I'm still learning approaches on a daily basis - so this is really meant as "food for thought" rather than a "this is how you should do things".

Hope this helps a bit!

Cal

On Mon, Oct 29, 2012 at 2:42 PM, Isaac XXX <vyrphan@gmail.com> wrote:

Hi folks,

I'm developing a new application that should get high traffic. Right now, I've other projects with the follow architecture:

Nginx on front: serving static content and redirecting to apache for dynamic data
Apache+mod_wsgi: serving dynamic pages
PostgreSQL: backend for data storage (RDBM)
Memcache: for caching purposes :)

All my deployments use a single server, with single frontend/backend (1 nginx, 1 apache, 1 postgresql). The requirements for this new project are really large, and I think I will need to scale all system. Can anyone suggest me an all-in-one tutorial, discussing the main points on scale a system?

I know there are different alternatives for DB (master-slave, clustering...), nginx can serve as a reverse proxy or not... and I need to merge all this information in a single scalable system, but I can't find an unified source of information.

Can anyone help me on it?

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.