Re: Write Multiple Text Files into a Single CSV File

On Tuesday, December 30, 2014 9:54:10 PM UTC-8, Martin Mirero wrote:

Hi folks,

I'm just cutting my teeth on Python/Django and need some assistance on something I've been grappling with for a few days:

• I have a bunch of text files on disk that all have the same basic format: first line is the title and the rest is the body
• I want to create one CSV file with the first column being populated from the first line of each text file (title) and the second column being populated from the rest of the text file contents (body)
• Once I have the CSV file, I'm good to import that into my django model

Depending on your use case, there may be an easier (or more efficient way) to skin this cat. Both mysql and postgres have syntax that will let you load an entire csv into a db table as a one-liner. To do that from within Django, you need to be able to run an external shell command, and the best way to do that is (often) with Fabric. And since this is a function you'll likely want to do on a regular basis, it makes sense to wrap it all up in a Django management command.

Here's an excerpt of some code I use to do something similar:

http://pastebin.com/crpnNVrx

This is located in someapp/management/commands/import_courses.py . With this in place I can run

./manage.py import_courses

at any time. On each run, it:

- Drops a temporary table 

- Re-creates that temporary table according to a predefined postgres schema (contained in importer_courses.sql)

- Imports the CSV into that temporary table

In my case, I needed to be able to perform queries across the CSV data *before* importing some of it into my real models, but this turned out to be a wonderfully efficient way of handling CSV data in general. You don't have to drop and recreate the temp table each time - I chose to do it that way for my use case but season to taste.

To use this technique you'll need to `pip install fabric`, set a directory location where your CSVs are located (as IMPORTER_DATA_DIR in your settings), and read up a bit on Fabric and Django management commands. Then you'll need to modify it to handle multiple CSVs rather than just one.

./s

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/178e2096-a9b7-4f92-8d7a-f375371a527e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.