Re: Trying to run jobs in the background using multiprocessing

On Thu, Nov 28, 2013 at 2:42 AM, Pau Creixell <paucreixell@gmail.com> wrote:
> I guess if there isn't another simpler way, I'll have to dig into Celery.


there _are_ simpler alternatives to Celery:

- worker threads: similar to your proposal, but with very well-thought
process control. pros: no extra process. cons: very hard to get it
even half right, very easy to kill your web workers with the lightest
load. probably unsupported by hostings.

- ghetto queues: a db table + a cron process. pros: simple, easy to
debug. cons: has a hard ceiling, if the queue grows over some (hard
to estimate) limit, it grinds to a halt.

- uWSGI spool: extra uWSGI containers that work on a queue of tasks.
pros: easy to use, you get the process management by uWSGI, no extra
tools. cons: limited semantics, might not be supported by hosts.

- simple brokers: for example pub/sub or other structures in Redis.
pros: easier to use and understand than celery, might be lighter on
memory. cons: no standarization, needs a broker process (at least if
you're not already running Redis), might not be supported by hostings.

as you can see, in most cases the only reasonable answers are Celery
or ghetto queues. the last ones are an easy way out, and if done well
can serve to a surprising load level; but if you hit gold, it will
fall down. this scheme was a big part of the Twitter growing
problems.


--
Javier

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAFkDaoRZJEiEMc2cJP2OzXFAouViNDNY%3DwdqkwOVoWbr2tvsKw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.