Re: how to do a django data migration on a massive Postgres data table (solved)

On Fri, May 22, 2015 at 3:09 PM, Vernon D. Cole <vernondcole@gmail.com> wrote:
> Django migrations are run in a single transaction (if your database is
> capable of doing a rollback of DDL, as PostgreSQL is) so data migrations of
> large tables become impossible.
>
> I was able to solve this problem by having the migration run a no-wait
> subprocess which performs the data migration using raw SQL without the
> django ORM. In order to keep things together, the subprocess is contained
> in the same source file as the migration... when run as a module it runs
> itself as a main program. In my case, it ran for about two days, converting
> my 10 million row table 1000 rows at a time. [Actually, I had to restart it
> three times by pretending to remove it and then re-applying it.]
>
> In case I need to do it some other time, I am putting the code into a GIST.
> I am posting here so that others may find it when searching for the right
> keywords...
> https://gist.github.com/9adedbab1899224a4eaf.git

Thanks very much Vern for doing this. I'm sure it will be very useful
to a lot of people.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CACwCsY53TK56_4nZb105qdC7QNnMOW44K2x2nBf6rCSom53G_A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.