Re: Need to process uploaded files into database entries(takes hours), how do I return a progress bar?

On Sun, Nov 27, 2011 at 11:21 PM, Nathan McCorkle <nmz787@gmail.com> wrote:
> Yeah I've seen the djcelery solution before... I've tried implementing
> it, but I'm getting an error on the celeryd console:
> TypeError: processXLS1() got an unexpected keyword argument 'task_is_eager'

I couldn't find anything on this error via google... but once I
removed the manual transaction stuff (I was trying to speed things up
by doing transactions only after I did a bunch of database operations)
including the decorator above the function, it started working.

>
> when I try running processXLS1.delay(dataObjectID=someInteger) from a
> function in views.py
>
> here's what's in my tasks.py:
> "
> from celery.decorators import task
> from enzymeFilter.models import *
> from django.db import transaction
>
> @task
> def printGene(y):
>        print "ppppppppppppppppppp"
>        fil=open('/var/www/media/testFile','w')
>        fil.write('coming from background')
>        fil.close()
>
>        print Gene.objects.get(id=y+1)
>        return True
>
> @task
> @transaction.commit_manually
> def processXLS1(datasetObjectID):
>        print "processing XLS as task"
>        datasetObject = Dataset.objects.get(id=datasetObjectID)
>        try:
>            ... more processing code
> "
>
> thanks
> -Nathan
>
> On Sun, Nov 27, 2011 at 9:52 PM, Brian Schott <bfschott@gmail.com> wrote:
>> You really should look at django-celery and RabbitMQ.  The upload submit can initiate a task that is defined in tasks.py.  There are separate worker processes that pick up the task from the message queue and do the actual work.  These workers don't even have to be on the same machine with rabbitMQ so you get instant scalability.   Then your AJAX job status view can poll a job status table that is updated by the task and you don't have to worry about threads.
>> https://github.com/ask/django-celery
>>
>> Brian Schott
>> bfschott@gmail.com
>>
>>
>>
>> On Nov 27, 2011, at 8:54 PM, Nathan McCorkle wrote:
>>
>>> I'm using Django 1.3 and am processing 3 files into genes, proteins,
>>> and genomes for a tool I built... this processing usually takes
>>> between 5 minutes to a few hours, depending on the genome size.
>>>
>>> After uploading the files (10-100 MB), the upload view begins
>>> processing the files, without returning for a long time (causing my
>>> browser to ask me if I want to kill the plupload script).
>>>
>>> For the user of this app, they don't know if their upload failed or is
>>> processing, etc... so I'd think forking the processing function (or
>>> making it a new thread) is what needs to happen, then the view can
>>> return to the user, and the upload page can start doing AJAX requests
>>> to check to processing progress.
>>>
>>> I feel like I want to do this:
>>> http://www.artfulcode.net/articles/threading-django/
>>> but am scared of crashing because I've heard Django isn't threadsafe
>>> with transactions. (and my processing function is making DB
>>> transactions)
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "Django users" group.
>>> To post to this group, send email to django-users@googlegroups.com.
>>> To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
>>> For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups "Django users" group.
>> To post to this group, send email to django-users@googlegroups.com.
>> To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
>> For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
>>
>>
>
>
>
> --
> Nathan McCorkle
> Rochester Institute of Technology
> College of Science, Biotechnology/Bioinformatics
>

--
Nathan McCorkle
Rochester Institute of Technology
College of Science, Biotechnology/Bioinformatics

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.