Re: Scaling django installation

To me, the biggest bottleneck in a "Django Application Installation" (not application) is not going to be Django at all. It's going to be I/O -- typically to the database and/or file system. These are used heavily (from my personal experience) by all sorts of django functions. As for the database -- it's used by every model, which comprises a huge chunk of the Django Application Installation. Another large part are the templates (and rendering system) which typically read from the filesystem.

Write good code with a focus on minimizing DB queries.
Save your templates in a RAM disk.
Save your DB to a RAM disk with hard-disk persistence on Writes.
Cache everything as much as possible (ORM-Level, Template-Level, View-Level, Sessions, etc..) in memory (memcache)
Let a completely different server serve all static and user media (e.g. S3) over CDN

Then, I'd imagine the Installation would be about as optimized as possible without going through the code and carefuly anylizing every Class, method, and data structure to see what you can pull a bit more out of. Of course you should spend time *before* you even start writing a single line of code to do some careful software "engineering". Some sites using Open-up-everything-with-APIs (think Amazon) -- but that's not exactly a tight code-base.

Of course the problem with this is -- developers are expensive! (or underpaid in some cases). It's a lot easier to just rely on distributed systems so that you can easily throw more resources at the problem until you reach some critical point where optimizations are truly required. If you pay a developer 60k/year to do something like this and it takes 3 times as long (180k) -- why not just pay 60k for a year and throw whatever is needed money-wise at the problem (probably much less than the remaining 120k)

On Fri, Jun 1, 2012 at 10:38 AM, Subhranath Chunder <subhranath@gmail.com> wrote:

Yup. the application performance has been kept in mind from the ground work. Zero or one query, with extensive use of cache is what we try to achieve at the app level.

Just to keep the thread a bit more focused on it's purpose, I would like to remind ourselves that, the discussion is on "Scaling django installation" and not "Scaling django application".

For e.g. (with random number representations)

- Setup 1: Single server setup:

 x1 Computation Units, x2 GB memory, n Geographical location, Max Serves 2000 requests/sec

- Setup 2: 2 servers cluster setup (1 server serving django, other media):

 y1 Computation Units, y2 GB memory, n Geographical location, Max Serves 3000 requests/sec

- Setup 3: 2 servers cluster setup (with a single load balancer):

 z1 Computation Units, z2 GB memory, n Geographical location, Max Serves 2800 requests/sec

Hope, I was able to put things in the right direction.

On Fri, Jun 1, 2012 at 7:43 PM, Javier Guerra Giraldez <javier@guerrag.com> wrote:

On Fri, Jun 1, 2012 at 3:56 AM, Subhranath Chunder <subhranath@gmail.com> wrote:
> how should we measure response complexity?

a simple first approximation is the number of DB queries per page. 

the debug toolbar nicely gives that figure while developing. 

--
Javier

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

--
Thanks,

Subhranath Chunder.

www.subhranath.com

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.