Re: Bulk import of data

This should be run asynchronously (i.e. celery) when importing large files.

If you have a lot of categories/subcategories, you will need to bulk insert them instead of looping through the data and just using get_or_create. A single, long transaction will definitely bring great improvements to speed.

One tool is DSE, which I've mentioned before.

Good luck!

Cheers,

AT

On Sat, Nov 26, 2011 at 8:44 PM, Petr Přikryl <prikryl@atlas.cz> wrote:

>>> import csv
>>> data = csv.reader(open('/path/to/csv', 'r'), delimiter=';')
>>> for row in data:
>>> category = Category.objects.get_or_create(name=row[0])
>>> sub_category = SubCategory.objects.get_or_create(name=row[1],
>>> defaults={'parent_category': category})
>>> product = Product.objects.get_or_create(name=row[2],
>>> defaults={'sub_category': sub_category})

There are few potential problems with the cvs as used here.

Firstly, the file should be opened in binary mode.  In Unix-based
systems, the binary mode is technically similar to text mode.
However, you may once observe problems when you move
the code to another environment (Windows).

Secondly, the opened file should always be closed -- especially
when building application (web) that may run for a long time.
You can do it like this:

...
f = open('/path/to/csv', 'rb')
data = csv.reader(f, delimiter=';')
for ...
...
f.close()

Or you can use the new Python construct "with".

P.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.