Re: Can anyone share code for uploading csv or excel file to sqlite3 database.

Hi,

Which error(s) did you get exactly ? It is not clear in you message whether your problem is at the CSV data processing level or at the Django one.

In case this could help, using the standard csv module you can obtain a reader wich takes care of parsing and decoding thanks to the reader() module function. You pass it the file object directly, without reading its content in memory. Since it provides an iterator over the data, yielding rows one at a time as tuples, this lets you load big files without any problem.

The DictReader class of the same module works more or less the same way, but returns rows as dicts which keys are the CSV columns names, making the mapping code more readable and self-documented.

Best

Eric

---

From: django-users@googlegroups.com <django-users@googlegroups.com> on behalf of BBG <bhavik.gorajiya@gmail.com>
Sent: Thursday, September 27, 2018 8:41:42 PM
To: Django users
Subject: Re: Can anyone share code for uploading csv or excel file to sqlite3 database.

 

I have tried this but not working

def upload_csv(request):      data = {}      if "GET" == request.method:          return render(request, "add_student/bulk.html", data)      # if not GET, then proceed      try:          csv_file = request.FILES["csv_file"]          if not csv_file.name.endswith('.csv'):              messages.error(request,'File is not CSV type')              return HttpResponseRedirect(reverse("add_student:upload_csv"))          #if file is too large, return          if csv_file.multiple_chunks():              messages.error(request,"Uploaded file is too big (%.2f MB)." % (csv_file.size/(1000*1000),))              return HttpResponseRedirect(reverse("add_student:upload_csv"))            file_data = csv_file.read().decode("utf-8")            lines = file_data.split("\n")          #loop over the lines and save them in db. If error , store as string and then display          for line in lines:              fields = line.split(",")              data_dict = {}              data_dict["enrollment_no"] = fields[0]              data_dict["student_name"] = fields[1]              data_dict["gender"] = fields[2]              data_dict["course"] = fields[3]              data_dict["category"] = fields[4]              data_dict["admission_year"] = fields[5]              data_dict["branch"] = fields[6]              data_dict["current_semester"] = fields[7]              data_dict["address"] = fields[8]              data_dict["city"] = fields[9]              data_dict["district"] = fields[10]              data_dict["state"] = fields[11]              data_dict["student_contact"] = fields[12]              data_dict["parent_contact"] = fields[13]              try:                  form = EventsForm(data_dict)                  if form.is_valid():                      form.save()                  else:                      logging.getLogger("error_logger").error(form.errors.as_json())              except Exception as e:                  logging.getLogger("error_logger").error(repr(e))                  pass      except Exception as e:          logging.getLogger("error_logger").error("Unable to upload file. "+repr(e))          messages.error(request,"Unable to upload file. "+repr(e))        return HttpResponseRedirect(reverse("add_student:upload_csv"))

On Thursday, September 27, 2018 at 1:59:24 PM UTC+5:30, Eric Pascual wrote:

Hi,

Have you studied the documentation of the CSV module included in Python standard library ?

You'll find there all the needed information for reading and interpreting CSV files without having to implement the raw parsing, and have there rows in a form ready to use for inserting objects in your model.

For performance's sake, it is advised to use bulk inserts or updates instead on individual saves on the Django side if your CSV files contain a lot of data.

Best

Eric

---

From: django...@googlegroups.com <django...@googlegroups.com> on behalf of BBG <bhavik....@gmail.com>
Sent: Wednesday, September 26, 2018 6:04:09 PM
To: Django users
Subject: Can anyone share code for uploading csv or excel file to sqlite3 database.

 

I want to create upload bulk data. Can anyone share code to upload csv or excel file. 

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to djang...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/fc5736a8-3396-491b-b265-853f46fdad87%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/88ea4342-f61d-4282-99d2-511f7e9f14a7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.