Friday, August 24, 2018

Re: Django performance issue that's troubling

Thanks much, Tim. I'll check it out - probably will need even more tuning down the road...

On Aug 24, 2018, at 1:00 PM, Tim Vogt (Tim Vogt) <timtvogt@gmail.com> wrote:

Also found this perhaps helpfull
https://blog.labdigital.nl/working-with-huge-data-sets-in-django-169453bca049

Op 24 aug. 2018, om 21:28 heeft Jim Illback <subaru_87@hotmail.com> het volgende geschreven:

Vijay, your suggestion worked. I had tried this previously but it was using a FBV and it didn't work. But with this CBV app, it works perfectly. Thanks.

Matthew, I'll definitely keep your suggestion in mind - maybe back on my FBV application if I refactor!

Thanks much to both of you!
Jim

On Aug 24, 2018, at 11:19 AM, Matthew Pava <Matthew.Pava@iss.com> wrote:

It's important that you set the widget to a HiddenInput.  It only has to render it with the current value.  If you are keeping the widget to be a Select widget and then setting a CSS class to make it hidden, the system will still have to load all 37K values.  I guess I'm missing something in your explanation.
 
From: django-users@googlegroups.com [mailto:django-users@googlegroups.com] On Behalf Of Jim Illback
Sent: Friday, August 24, 2018 1:16 PM
To: Django users
Subject: Re: Django performance issue that's troubling
 
Matthew, thanks - that was my original code, but even hidden, it has to fill in all possible values - 37K - so it takes multiple seconds.
 
Vijay, I'll try your suggestion.
 
Thanks much,
Jim


On Aug 24, 2018, at 11:01 AM, Matthew Pava <Matthew.Pava@iss.com> wrote:
 
You can keep the field, but change its widget to a HiddenInput.
 
 
 
From: django-users@googlegroups.com [mailto:django-users@googlegroups.com] On Behalf Of Vijay Khemlani
Sent: Friday, August 24, 2018 12:59 PM
To: django-users@googlegroups.com
Subject: Re: Django performance issue that's troubling
 
Please reply to the user group
 
"But, in order to have the update work, it needs the ForeignKey to the parent in the form"
 
As far as I know, it doesn't need to (if the entry already exists in the database and you are just editing it), you just need to set up the fields that you want to be editable by the user
 
 
On Fri, Aug 24, 2018 at 2:06 PM Jim Illback <subaru_87@hotmail.com> wrote:
Vijay, think of this is a parent - child relationship. I have 37K parents. Now I want to update a child of one of the parents. But, in order to have the update work, it needs the ForeignKey to the parent in the form. However, to add that key, Django must then allow it to update that field and thus has to fill in all 37K entries. Is there no way to tell Django that the FK relationship can not be changed? If I leave it off of the form, the update fails because it is a required field. If I add it to the form, Django has to allow all potential update values to be shown. What's the way out of this mess?
 
Thanks,
Jim
 



Begin forwarded message:
 
From: Vijay Khemlani <vkhemlan@gmail.com>
Subject: Re: Django performance issue that's troubling
Date: August 24, 2018 at 5:07:07 AM PDT
 
To tell you the truth I couldn't understand some parts of your message, but if your AddictionForm is a ModelForm then you should be able to limit its fields to just the ones you want your users to be able to update.
 
 
Regards!
 
 
 
On Thu, Aug 23, 2018 at 8:19 PM Jim Illback <subaru_87@hotmail.com> wrote:
Vijay, you are exactly correct. Thank you much! 
 
Now, knowing the problem, I still don't have a great solution. Here's how it is supposed to work.
 
Someone chooses a client from the list or a search. All the possible support records are shown, whether they exist or not, on a screen to hyperlink to one of them. If the record exists, the URL includes the PK and transfers to the UpdateView. This is similar for an AddView if the record does not exist. However, in the Update or Add, I don't want the primary key changed. So, I hid the field (and thus all the records were read). However, now I can't limit that field to only one value. If I format the entire SELECT tag with only one OPTION, Django gives an invalid expression for the ID or Name because it is based on my query set I passed in that fills out the values (ID and name). If I use the form's field value, all the fields are read. Especially on an Add, I don't know a way to fill the client ID field in such a way that Django will take it as valid.
 
My thoughts on possible solutions:
1. Override the get query set method to only pass in the single record - will that work?
2. I've been trying to show the field in a <p> tag and override the View's post method by aftering is_valid(), but it always returns invalid expression errors.
 
What am I missing?
 
Thanks much again -
Jim
 



On Aug 23, 2018, at 3:48 PM, Vijay Khemlani <vkhemlan@gmail.com> wrote:
 
Maybe your update form has a field related to a model with thousands of records? like rendering a select combo box with thousands of choices?
 
On Thu, Aug 23, 2018 at 5:30 PM Jim Illback <subaru_87@hotmail.com> wrote:
I have a Django app in development that performed perfectly with about 20 client records. Each client could have one or more support records and there are 40 potential support records, connected to the client with a ForeignKey (client_id). Then, I loading about 37K client records. Search queries perform almost instantaneously. A paginated list of support records with 40 pages and 10 clients per page comes back in half a second. Great performance.
 
However, when I click on any record in the list to go to update a specific support record for that specific client (it uses the PK in the URL so is a direct call to the UpdateView routine), I get about 10 second response times and these kind of statistics from Django-debug-toolbar:
SQL: 5 queries, 20ms (so clearly not a DB related performance issue)
TIME:  Resource usage:
User CPU time 8827 msec
System CPU time 137 msec
Elapsed time 9003 msec
Context switches 1 voluntary, 9639 involuntary
Browser timing:
domLoading 9026 (+52130) msec
 
CACHE: 0 calls in 0.00ms (so clearly not a cache issue either)
 
Here are some details: Django 2.1, python 3.6.3
Hardware: iMac with 32GB, i7 quad core
Installed apps: Bootstrap3 v9.1.0, crispy forms v1.7.0, easyPDF v0.1.1, registration-redux v2.2, Pillow v4.3.0
 
Here's one of the specific UpdateView routines being executed - there are only 5 fields in the AddictionForm plus the prime key (no image or file attachments).
class AddictionUpdate(PermissionRequiredMixin, UpdateView):
    model = Addiction
    form_class = AddictionForm
    template_name = 'addiction_update.html'
    permission_required = 'clientapp.change_addiction'
The slow response appears to be on UpdateView pages only, even though all updates are called with the prime key (ID) in the URL. Can anyone give me guidance where to look for improving the performance? Why all the involuntary context switches? Why the large domLoading time?
 
Thanks in advance -
Jim
 
-- 
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/76F251D2-A429-4700-9F69-053673BDB91F%40hotmail.com.
For more options, visit https://groups.google.com/d/optout.
 
-- 
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CALn3ei2korG5V3dDcCUQuKiAhG6tzAz%3DJZssNpGJpkAOyKAXpQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
 
 
-- 
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CE5F73B1-38EC-435B-A7B8-61F143B5C1F2%40hotmail.com.
For more options, visit https://groups.google.com/d/optout.
 
-- 
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CALn3ei3%2B5NV3Hb5rxO-VYYGbfxRzo1Fi8-mJbXLDFFVpYTrAEg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
 
-- 
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CALn3ei2LYt6JFBXUUCVNRAer-LdL69q9y2_Muxk-mmah1z0bAw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
 
-- 
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/2aafdf9d6360491a83da7d5839234e68%40ISS1.ISS.LOCAL.
For more options, visit https://groups.google.com/d/optout.
 
-- 
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/25286DDB-941D-408E-96E4-85D634A9EC5C%40hotmail.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/84e8c6364ac8455285b5a13f75e0044c%40ISS1.ISS.LOCAL.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/FF1B4F5D-AE22-400E-9CB9-2820AF131749%40hotmail.com.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/1C757A10-580D-4ECC-B8E9-B1657BB83687%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Posted by at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)