Tuesday, November 29, 2011

Re: Bulk import of data

On 11/27/2011 07:56 PM, Andre Terra wrote:
> This should be run asynchronously (i.e. celery) when importing large files.
>
> If you have a lot of categories/subcategories, you will need to bulk
> insert them instead of looping through the data and just using
> get_or_create. A single, long transaction will definitely bring great
> improvements to speed.
>
> One tool is DSE, which I've mentioned before.

Hi Andre, I didn't forget tips you gave me in a previous thread about
Celery and DSE. Actually I've been studying them for a while. At the
moment things doesn't seem to heavy. I think I'll use them in future
develpment.

Thank you very much, Fabio.

--
Fabio Natali

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

No comments:

Post a Comment