Re: importing csv data into database

Hi Andrew,

Thanks heaps. do you mine explaining how to fix that.

cheers,

On Mon, Apr 27, 2015 at 4:26 AM, Andrew Farrell <armorsmith42@gmail.com> wrote:

What I'm saying that you are defining data.LastName as a tuple and when it gets turned into a string. So if a row is

44, Barack, Obama, 1600 Pennsylvania Avenue

Then data.FirstName is going to be 

('Barack',)

Which is a single-element tuple. When it gets turned into a string, it becomes ('Barack',)

On Sun, Apr 26, 2015 at 4:25 AM, sum abiut <suabiut@gmail.com> wrote:

i am importing data from csv file to the model test, so i am taking data as an instance of test(). are you saying  i should define data as a string?? and have the parameters as personid, firstname,lastname,address. Please advise

Cheers

On Sun, Apr 26, 2015 at 2:36 AM, Andrew Farrell <armorsmith42@gmail.com> wrote:

Ah, I suppose that bit didn't get sent. Google groups only lets mail from my old gmail address through and bounces mail from the MIT address I use by default.

I noticed something else about your code: you are assigning 2-tuples to all of your data types rather than strings when you do

   data.PersonID=row[0],
   data.FirstName=row[1],
   data.LastName=row[2],
   data.Address=row[3],

You might actually be aware of this, but I once spent 4 hours debugging before I realized this on a previous project.

weapons = ('fear', 'surprise', 'fanatical devotion to the pope',) n

clearly creates a 3-item tuple. But so does

weapons = 'fear', 'surprise', 'fanatical devotion to the pope',

and in fact

weapons = 'fear',

creates a 1-item tuple.

On Sat, Apr 25, 2015 at 7:27 AM, sum abiut <suabiut@gmail.com> wrote:

Ok , i manage to figure out where the error was coming from in my view.py on line  readata=csv.reader(csvfile,delimiter=';', quotechar='|'). The delimiter =';' i change the semicolon to  a comma The delimiter =',' and it works.

However when the data are imported from the csv file to the model it also write some funny character to the model as shown on the image below. How to import data to model without writing this funny characters?  i want the data to be clean

view.py
def readcsvfile(request):
    with open('/var/www/html/webapp/csvupload/data.csv','rb') as csvfile:
        readata=csv.reader(csvfile,delimiter=';', quotechar='|')
        #readata=csv.reader(csvfile)
        for row in readata:
            data=test()
            data.PersonID=row[0],
            data.FirstName=row[1],
            data.LastName=row[2],
            data.Address=row[3],
            data.save()
        return render_to_response('csv_test.html',locals())

On Fri, Apr 24, 2015 at 4:36 PM, Andrew Farrell <armorsmith42@gmail.com> wrote:

You might have to make sure that my indentation there is correct.

Also, I noticed something else about your code: you are assigning 2-tuples to all of your data types rather than strings when you do

   data.PersonID=row[0],
   data.FirstName=row[1],
   data.LastName=row[2],
   data.Address=row[3],

You might actually be aware of this, but I once spent 4 hours debugging before I realized this on a previous project.

weapons = ('fear', 'surprise', 'fanatical devotion to the pope',) 
clearly creates a 3-item tuple. But so does

weapons = 'fear', 'surprise', 'fanatical devotion to the pope',

and in fact

weapons = 'fear',

creates a 1-item tuple.

On Fri, Apr 24, 2015 at 12:33 AM, Andrew Farrell <armorsmith42@gmail.com> wrote:

could you replace your function with 

def readcsvfile(request):
    with open('/var/www/html/webapp/csvupload/data.csv','rb') as csvfile:
        readata=csv.reader(csvfile,delimiter=';', quotechar='|')
        #readata=csv.reader(csvfile)
        for i, row in enumerate(readata):

            try:
                data=test()
                data.PersonID=row[0],
                data.FirstName=row[1],
                data.LastName=row[2],
                data.Address=row[3],
                data.save()

            except Exception, e:

                import pdb;pdb.set_trace()

                print i

                print row

                print row[1]

                raise e
        return render_to_response('csv_test.html',locals())

And then re-run this again? When you hit the offending row of data, it should pop you into the python debugger. type 'print row<enter>' and you should see what the problem is. If not type 'next<enter>' and step though the code. 

You can execute arbitrary python from the debugger and once you figure out the problem, I encourage you to play around with it. Aside from pedantically explaining to yourself what you think going on, pdb is one of the most generally useful tools.

On Fri, Apr 24, 2015 at 12:01 AM, sum abiut <suabiut@gmail.com> wrote:

Hi,

could you please help me figure out why i am getting this error i am trying to import data from csv file to model buty i am getting the error: list index out of range and if refering to line 20 on view which is: data.FirstName=row[1]

view.py
def readcsvfile(request):
    with open('/var/www/html/webapp/csvupload/data.csv','rb') as csvfile:
        readata=csv.reader(csvfile,delimiter=';', quotechar='|')
        #readata=csv.reader(csvfile)
        for row in readata:
            data=test()
            data.PersonID=row[0],
            data.FirstName=row[1],
            data.LastName=row[2],
            data.Address=row[3],
            data.save()
        return render_to_response('csv_test.html',locals())

model.py

class test(models.Model):
    PersonID = models.CharField(max_length=45)
    FirstName = models.CharField(max_length=45)
    LastName = models.CharField(max_length=45)
    Address = models.CharField(max_length=45)

here is the traceback:

• /var/www/html/env/local/lib/python2.7/site-packages/django/core/handlers/base.py in get_response
  response = wrapped_callback(request, *callback_args, **callback_kwargs
  
  
  /var/www/html/webapp/csvapp/views.py in readcsvfile
  data.FirstName=row[1],
  
  

Cheers,

Sum

On Wed, Apr 1, 2015 at 1:51 PM, sum abiut <suabiut@gmail.com> wrote:

Thanks guys,

appreciate your help. Yes i am looking to have a form that accepts CSV files. I will have a read on the link that you guys have provided and see how it goes.

Cheers

On Wed, Apr 1, 2015 at 11:44 AM, Andrew Farrell <armorsmith42@gmail.com> wrote:

Hello Sum,

There are two approaches you could take.

One is to install a plugin like django-import-export and then use it's import functionality. Its been a while since I've done this, but when I did under django-1.4 it Just Worked. This is the best approach if you know your data is clean and doesn't require any sort of processing before saving it.

The other is to write a custom manage.py command. This would let you load the data using a csv.DictReader, do some logic on it, and then bulk-create objects in the database. This approach has the disadvantage that if you have a bug in your processing script, you will end up adding messy data to your database and having to clean it up. To avoid this, you could write a unit test case which calls your import function in its setUp() method. This testcase would create a separate test database, import the data into it, run your test functions, and then clear the test database. This approach would also let you, for example, deliberately insert malformed info into the data and verify that your import function would catch those errors.

On Tue, Mar 31, 2015 at 6:46 PM, sum abiut <suabiut@gmail.com> wrote:

Hi,

Could someone please advise this i am trying to import data from csv file or write to database but i am confuse on how to start. i am using sqllite database.

Cheers,

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAPCf-y5Nk_ATFgYk-hduiLsrxBasS%3Dzu4Qj8Gp5F9vo%3DNRwF%2BA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CA%2By5TLbaBcs8%2BrJsNmXFknRGNTC5%2BOr3URs53eiFBM8GLekmcA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAPCf-y74u7NDyxxG_c7rvpver7GKmg-X9CNv-ZJpmug%3Dop3LFA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CA%2By5TLY_BPTQieu%3D%3Dc_hg4ahAGJevp%2B5zb%3D1PWOw5GJ-4ULZ3w%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAPCf-y7ky9f7PrpnHvK70yLAMiHaw4s3U500RAdyJV25bdpbag%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CA%2By5TLab49_vfB%3D8ysOLm22tXXXXrOsaKYGoRQvzis_1PbsX3Q%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAPCf-y4zOzVi_uw9jQcmmHy3GVXWukYvu-bKP3HK-G0JB_Zt0Q%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CA%2By5TLbAx_e_LexuC6AOe3LBXWoV2O9GQZpYygSHJrTFF2xagQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAPCf-y4-TXxS9kPQQYUpTzmPP4ts7u__4V_FdPviOE6kYvzoKg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.