Re: Kind'a TL, but please DR - Need your thoughts

I suggest that you use Celery.

If people are making HTTP requests of you, that is reason enough to choose Django.

But do not wait for long calculations to complete before returning an HTTP result.  Instead redirect to a page containing simple JavaScript that will poll for a result.

PostgreSQL is my favorite SQL DB, and I think that you are in good shape there.  Another popular free DB is MariaDB, but I prefer PostgreSQL.

Record the request data in the DB invoke a Celery task to send the Primary Key of the new entry to a worker process.  This simply queues a message in RabbitMQ, so it is quite fast.  (Other MQs are possible, but RabbitMQ is best tested with Celery.)  This lets you return the HTTP response very quickly.

Any additional daemons needed to poll for or listen for request coming by other than HTTP request can also store to the DB and call a Celery task.

Rather that polling the DB for work, the Celery worker system, when a worker process is ready, takes a message from the queue (RabbitMQ) and assigns it to that worker.  Multiple workers can be handling separate messages in parallel, but do size the worker pool according to the size of your machine.  The worker fetches the request from the DB, and can run for as long as necessary to perform the calculation, even, if necessary, reaching out to other resources over the net.  When it is done, it stores the response in the DB, for the polling by the requester's JavaScript to find. and/or send copies via other mechanisms, such as an SMS interface.  Then the worker becomes available to handle another message.

I can assure you that this works well on Linux (you don't mention the platform).  I have not used Celery (or Django, for that matter) on Windows or Mac, but I'll bet that it runs fine, modulo the usual surprises about file system differences and the way that Windows processes are "special".

Pretty much you just code in Python.  The exception is startup scripts to boot time start/manage  the celery works, Apache/nginx front end for Django, and any additional required communications processes.  I guess there is also that small JavaScript to poll for a result.

An alternative to the JavaScript is a button for the user to push to see if the results are ready, and you should probably implement that anyway (and use JavaScript to hide it) for those with JavaScript disabled.

On Mon, Feb 1, 2016 at 1:28 AM, Avraham Serour <tovmeod@gmail.com> wrote:

if a process takes too long to complete it won't be able to process new requests, so you will be limited to the number of workers you told uwsgi to use.

http requests should be short lived, if you have some heavy processing  to do the http request should return with something like 'accepted' and send the job to a queue (you can use celery)

On Mon, Feb 1, 2016 at 6:51 AM, Mario R. Osorio <nimbiotics@gmail.com> wrote:

I need comments on an application I have been recently proposed. The way it is being envisioned at this moment is this:

One python daemon will be listening for different communications media such as email, web and SMS (web also). IMHO, it is necessary to have a daemon per each media. This daemon(s) will only make sure the messages are received from a validated source and put such messages in a DB

A second(?) python daemon would be waiting for those messages to be in the DB, process them, act accordingly to the objective of the application, and update the DB as expected. This process(es) might included complicated and numerous mathematical calculations, which might take seconds and even minutes to process.

A third(?) python daemon would be in charge of replying to the original message with the obtained results, but there might be other media channels involved, eg the message was received from a given email or SMS user, but the results have to be sent to multiple other email/SMS users.

The reason I want to do the application using Django is that all this HAS to have multiple web interfaces and, at the end of the day most media will come through web, and have to be processed as http requests. Also, Django gives me a frame to make this work better organized and clean and I can make the application(s) DB agnostic.

Wanting the application to be DB agnostic does not mean that I don't have a choice: I know I have many options to communicate among different python processes, but I prefer to leave that to the DBMS. Of the open source DBMS I know of, only Firebird and PostgreSQL have event that can provide the communication between all the processes involved. I was able to create a very similar application in 2012 with Firebird, but this time I am being restricted to PostgreSQL, which I don't to oppose at all. That application did not involve http requests.

My biggest concern at this point is this:

If most (if not all) requests to the application are going to be processed as http requests, what will happen to pending requests when one of them takes too long to reply? Is this something to be solved at the application level or at the server level?

This is as simple as I can put it. Any thoughts, comments, criticism or recommendations are welcome.

Thanks a lot in advanced!

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/802693b7-c00c-46bf-9902-688b27e21bbd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAFWa6tKZcKySJdG4YcnHh38oTuTTVEfaz6WBTOyKDmDiV0_sFA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAB%2BAj0v3S77xvZR9_wPyooYQ5CgQwx3ZNo2SFcNSvkA_Z_gfPw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.