Monday, November 28, 2011

Re: Simple task dispatching (How heavy is celery + RabbitMQ)

On 11/28/2011 12:14 PM, Gelonida N wrote:
> Hi,
> I'd like to use a light weight dispatching system for a web server.
> Basically some django post requests might require processing, which
> should be done in the back ground due to its run times.
> The results would be added to the django data base.
> The browser could verify via AJAX requests whether the task is finished.
> The server would be running on a rather weak virtual machine with rather
> low memory (nginx / uwsgi / django )
> ( For testing I run the server on windows with one of the following
> setups (depending on what I'd like to test)
> - django runserver
> - twisted - django
> - cygwin/nginx - fastcgi - django
> Most people seem to recommend celery with RabbitMQ.
> If I understood well, rabbit MQ requires Erlang to be installed and I
> found some posts indicating that RabbitMQ requires quite some memory
> So I wondered whether celery / RabbitMq wouldn't be a little on the
> heavy side and eat away a little too much from my meory.
> Is there any good light weight dispatching alternative to celery or
> would this be one of these 'roll your own dispatcher' tasks?
Thanks a lot for all your answers.

To summarize quickly
RabbitMQ doe snot seem to be as greedy as I expected (probably around
19MB for my expected load)

Django-ztask is a small brokerless solution

Celery supports multiple brokers with difference performance. the dd
broker would not require an additional process (apart from celery)

uwsgi has a spooler which might be what I'm looking for I just would
like to execute some tasks, which are too slow to be treated directly
within an HTTP request sequentially one after the other.

However if I used uwsgi I had to look for an alternative implementation
and a small wrapper such, that the system would still be working on a
windows host without uwsgi. (performance on windows is not crucial, but
it should work)

Now I just need some time to test some of these options on windows PC
and on my tiny virtual linux host.

You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to
To unsubscribe from this group, send email to
For more options, visit this group at

No comments:

Post a Comment