Re: Simple task dispatching (How heavy is celery + RabbitMQ)

There are plenty of tutorials about memory management with RabbitMQ. There are mechanisms for setting the high watermark for memory to throttle producers. You could look at the database backend for django-kombu. It might have a smaller footprint. django-ztask looks interesting, but you don't seem to get reliable message delivery, but maybe I misread the site.

Brian Schott
bfschott@gmail.com

On Nov 28, 2011, at 8:34 PM, Gelonida N wrote:

> On 11/28/2011 12:14 PM, Gelonida N wrote:
>> Hi,
>>
>> I'd like to use a light weight dispatching system for a web server.
>>
>> Basically some django post requests might require processing, which
>> should be done in the back ground due to its run times.
>>
>> The results would be added to the django data base.
>>
>> The browser could verify via AJAX requests whether the task is finished.
>>
>>
>> The server would be running on a rather weak virtual machine with rather
>> low memory (nginx / uwsgi / django )
>>
>> ( For testing I run the server on windows with one of the following
>> setups (depending on what I'd like to test)
>> - django runserver
>> - twisted - django
>> - cygwin/nginx - fastcgi - django
>>
>> Most people seem to recommend celery with RabbitMQ.
>> If I understood well, rabbit MQ requires Erlang to be installed and I
>> found some posts indicating that RabbitMQ requires quite some memory
>>
>> So I wondered whether celery / RabbitMq wouldn't be a little on the
>> heavy side and eat away a little too much from my meory.
>>
>>
>> Is there any good light weight dispatching alternative to celery or
>> would this be one of these 'roll your own dispatcher' tasks?
>>
>>
> Thanks a lot for all your answers.
>
>
> To summarize quickly
> RabbitMQ doe snot seem to be as greedy as I expected (probably around
> 19MB for my expected load)
>
> Django-ztask is a small brokerless solution
>
> Celery supports multiple brokers with difference performance. the dd
> broker would not require an additional process (apart from celery)
>
>
> uwsgi has a spooler which might be what I'm looking for I just would
> like to execute some tasks, which are too slow to be treated directly
> within an HTTP request sequentially one after the other.
>
> However if I used uwsgi I had to look for an alternative implementation
> and a small wrapper such, that the system would still be working on a
> windows host without uwsgi. (performance on windows is not crucial, but
> it should work)
>
> Now I just need some time to test some of these options on windows PC
> and on my tiny virtual linux host.
>
>
> --
> You received this message because you are subscribed to the Google Groups "Django users" group.
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
>

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.