Re: Bulk Creating Django Models from CSV File?

The process of bulk does not insert the keys in the table since before starting a bulktriggers are disabled and the database functions that perform the process these data.It is not up to Django fill the primary keys, but the database.

I think we have some ways to work around this problem:

1. If you only have a bulk process running at a time, you can implement a process to copy the records from table to table to the bulk being used by django and during thisprocess to perform the key assignment.

2. You can implement a trigger or a procedure for the database to assign a line afterinserting a row.
This is the way it usually creates a database primary keys, that will affect the performance.

In bulk processes is common to have another process responsible for analyzing and
filter data.

Did you have any idea how many records are we talking?
More than 15 million records justify a bulk process with primary keys disabled, in other cases I think it will have less work if you let the database take care of things for you. The worst that can happen is to take a while.

If you need any help, contact me!

Ivo Marcelo Leonardi Zaniolo

http://imarcelolz.blogspot.com.br/

2012/4/2 Victor Hooi <victorhooi@gmail.com>

Hi,

I have several large CSV files that I'm hoping to parse, and use to create Django objects. Each line looks might look something like this:

"Server Hostname", "Classification", "Country", "Operating System"

"foo.bar.com", "Prod", "Australia", "Solaris"

"alex.john.com", "Dev", "UK", "Linux"

"bob.hope.com", "UAT, "Japan", "Windows

For performance reasons, it would be nice to use something like 1.4's bulk_create:

https://docs.djangoproject.com/en/dev/ref/models/querysets/#django.db.models.query.QuerySet.bulk_create 

However, I noticed the caveat about it not populating the auto-incrementing PK field. What's the recommended way to populate the PK field then? How do you use it?

Secondly, the models will have FK relationships to other models (for instance, Classification, Country and Operating System would be stored in a lookup table). What's the best way to look these up, and assign the appropriate FK, whilst still preserving good performance and using bulk inserts?

Cheers,

Victor

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/django-users/-/bP0RG4NXJjEJ.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.