Re: Optimizing admin change view with inlines

Hi Petros,

I think it's because "value_from_object" only passes the primary key of the object and then raw_id_field re-queries it.

https://github.com/django/django/blob/40ccef16/django/contrib/admin/widgets.py#L195

https://github.com/django/django/blob/40ccef16/django/forms/models.py#L151

https://github.com/django/django/blob/40ccef16/django/db/models/fields/__init__.py#L844-L848

Basically, even if you use select_related(), django still queries your object again. It looks like a spot where there could be some optimization.

Collin

On Monday, December 29, 2014 1:45:08 PM UTC-5, Ernest0x wrote:

Hello Collin,

On 12/29/14 07:25, Collin Anderson wrote:

Hi Petros,

I think the raw_id_fields still display the __str__/__unicode__ of the selected object so therefore need to get queried. I wonder if select_related would possibly help?

I do have select_related in the queryset (look at the last version of ExampleBInline in my previous post).

The thing is that if the same fields that are included in raw_id_fields are also included in readonly_fields, the queries are reduced dramatically, despite the fact that the same amount of information is displayed in both cases. __str__()/__unicode__ is still called when the fields are in readonly_fields, so it is really very strange why there is difference between adding and commenting out the readonly_fields line.

Very strange...

Collin

On Wednesday, December 24, 2014 5:53:02 AM UTC-6, Ernest0x wrote:

On 12/24/14 13:30, 'Petros Moisiadis' via Django users wrote:
> On 12/23/14 19:13, 'Petros Moisiadis' via Django users wrote:
>> Hello people :)
>>
>> I am struggling with optimizing an admin with inlines which causes too
>> many database requests.
>>
>> My model structure is like this:
>>
>>     class ExampleA(models.Model):
>>         ...
>>    
>>     class ExampleB(models.Model):
>>         aexample = models.ForeignKey('ExampleA', related_name='bexamples')
>>         cexample = models.ForeignKey('ExampleC')
>>         dexample = models.ForeignKey('ExampleD')
>>         eexample = models.ForeignKey('ExampleE')
>>         ...
>>
>>     class ExampleC(models.Model):
>>         ...
>>    
>>     class ExampleD(models.Model):
>>         ...
>>    
>>     class ExampleE(models.Model):
>>         ...
>>
>> The admin classes:
>>
>>     class ExampleBInline(admin.StackedInline):
>>         model = ExampleB
>>         extra = 0
>>
>>     class ExampleAAdmin(admin.ModelAdmin):
>>         inlines = [ExampleBInline]
>>
>>     admin.site.register(ExampleA, ExampleAAdmin)
>>
>> As I can see with django-debug-toolbar, when rendering the admin
>> template for the inline formset with the forms for ExampleB objects, a
>> new db request is sent to db server for each related field of each
>> ExampleB object. Particularly, I am seeing a lot of queries like these:
>>
>>     SELECT ••• FROM `examplec` WHERE `examplec`.`id` = 1 LIMIT 21
>>     SELECT ••• FROM `examplee` WHERE `examplee`.`id` = 2 LIMIT 21
>>     SELECT ••• FROM `examplec` WHERE `examplec`.`id` = 2 LIMIT 21
>>     SELECT ••• FROM `exampled` WHERE `exampled`.`id` = 2 LIMIT 21
>>     SELECT ••• FROM `examplee` WHERE `examplee`.`id` = 3 LIMIT 21
>>
>> The template context is this (I am using grappelli):
>>    
>>                     21        {% if field.is_readonly %}
>>                     22            <div class="grp-readonly">{{
>> field.contents|linebreaksbr }}</div>
>>                     23        {% else %}
>> the marked line =>  24            {{ field.field }}
>>                     25        {% endif %}
>>                     26    {% endif %}
>>                     27        {% if line.fields|length_is:'1' %}{{
>> line.errors }}{% endif %}
>>
>>
>> I have tried the following optimizations:
>>
>> First try:
>>
>>     class ExampleAAdmin(admin.ModelAdmin):
>>         inlines = [ExampleBInline]
>>
>>         def get_queryset(self, request):
>>             qs = super(ExampleAAdmin, self).get_queryset(request)
>>             return qs.prefetch_related('bexamples',
>> 'bexamples__cexample', 'bexamples__dexample', 'bexamples__example')
>>
>> Second try:
>>
>>     class ExampleBInline(admin.StackedInline):
>>         model = ExampleB
>>         extra = 0
>>      
>>         def get_queryset(self, request):
>>             qs = super(ExampleInline, self).get_queryset(request)
>>             return qs.select_related('cexample', 'dexample', 'eexample')
>>
>> Third try:
>>
>>     class BaseExampleBFormSet(BaseInlineFormSet):
>>         def __init__(self, *args, **kwargs):
>>             super(BaseExampleBFormSet, self).__init__(*args, **kwargs)
>>             qs = self.queryset
>>             self.queryset = qs.select_related('cexample', 'dexample',
>> 'eexample')
>>
>>     ExampleBFormSet = inlineformset_factory(ExampleA, ExampleB,
>> formset=BaseExampleBFormSet)
>>
>>     class ExampleBInline(admin.StackedInline):
>>         model = ExampleB
>>         extra = 0
>>         formset = ExampleBFormSet
>>
>> Unfortunately, none of the above works.
>>
>> So, I would be really grateful if anyone could help on this by giving
>> any hint or pointing out what could be possibly missing. Or, could I
>> have hit a restriction of django admin's internals?
>>
>> Thank you in advance,
>>
>> Petros
>>
> Hello again,
>
> I would like to clarify that the big number of db requests is not caused
> by fetching all the objects for the related model to populace select
> boxes for the related fields in the line, as it is often the case. To
> avoid that, I have included the related fields in raw_id_fields. I have
> also used grappelli's convenient autocomplete lookup feature, but that
> does not add extra queries. So, the inline class actually looks like this:
>
>     class ExampleBInline(admin.StackedInline):
>         model = ExampleB
>         extra = 0
>         raw_id_fields = ['cexample', 'dexample', 'eexample']
>         autocomplete_lookup_fields = {
>             'fk': ['cexample', 'dexample', 'example'],
>             }
>
> The only optimization that works is adding the related fields in
> readonly_fields in combination with a queryset changing method with
> select_related(), like this:
>
>     class ExampleBInline(admin.StackedInline):
>         model = ExampleB
>         extra = 0
>         raw_id_fields = ['cexample', 'dexample', 'eexample']
>         autocomplete_lookup_fields = {
>             'fk': ['cexample', 'dexample', 'eexample'],
>             }
>         readonly_fields = ['cexample', 'dexample', 'eexample']
>
>         def get_queryset(self, request):
>             qs = super(ExampleBInline, self).get_queryset(request)
>             return qs.select_related('cexample', 'dexample', 'eexample')
>
>
> However, I am not satisfied with that solution because I do want to be
> able to edit the related fields inline.
>

So the question merely is why the inline class right above produces so
many queries if the 'readonly_fields' line is commented out?
For my data, according to debug toolbar, the view causes 11 queries with
readonly_fields and 115 queries without readonly_fields.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/778f1f33-3c34-491e-a062-9527ec703bfd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/57965c06-e757-487b-8937-cfd1c4b675d4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.