Re: Need to process uploaded files into database entries(takes hours), how do I return a progress bar?

You really should look at django-celery and RabbitMQ. The upload submit can initiate a task that is defined in tasks.py. There are separate worker processes that pick up the task from the message queue and do the actual work. These workers don't even have to be on the same machine with rabbitMQ so you get instant scalability. Then your AJAX job status view can poll a job status table that is updated by the task and you don't have to worry about threads.
https://github.com/ask/django-celery

Brian Schott
bfschott@gmail.com

On Nov 27, 2011, at 8:54 PM, Nathan McCorkle wrote:

> I'm using Django 1.3 and am processing 3 files into genes, proteins,
> and genomes for a tool I built... this processing usually takes
> between 5 minutes to a few hours, depending on the genome size.
>
> After uploading the files (10-100 MB), the upload view begins
> processing the files, without returning for a long time (causing my
> browser to ask me if I want to kill the plupload script).
>
> For the user of this app, they don't know if their upload failed or is
> processing, etc... so I'd think forking the processing function (or
> making it a new thread) is what needs to happen, then the view can
> return to the user, and the upload page can start doing AJAX requests
> to check to processing progress.
>
> I feel like I want to do this:
> http://www.artfulcode.net/articles/threading-django/
> but am scared of crashing because I've heard Django isn't threadsafe
> with transactions. (and my processing function is making DB
> transactions)
>
> --
> You received this message because you are subscribed to the Google Groups "Django users" group.
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
>

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.