Re: Django queryset High CPU Usage

On Sunday, March 12, 2017 at 4:53:25 AM UTC+5:30, James Schneider wrote:

On Mar 11, 2017 12:01 PM, "Vijay Khemlani" <vkhe...@gmail.com> wrote:

Am I the only one who thinks that generating a report over a set of
just 10.000 records could be done in 10 - 20 secs unless there are
some serious computations going on with that data?

For a report I have to query around 200.000 records, with
aggregations, and it takes less than a minute using the ORM.

The OP never mentioned a time interval that I can find in this thread, only CPU utilization. I can only imagine that the query is taking long enough to notice the CPU utilization, which would be at least a few seconds.

Querying and aggregating 200K records within the DB is not comparable to pulling 10K individual records and performing processing on each one. An ORM call with aggregation will perform a large majority of the work in the DB, and the ORM simply wraps the response accordingly. 

I agree. The task was mostly to pull data from DB and generate the CSV. The  model structure was as follows:

Main Model A

A has foreignkey to model D

A has foriegnkey to model E

F has foreignkey to model D

Model B having manyTomany relationship with A through B1

Model C having manyTomany relationship with A through C1

It was mostly DB fetches which was the complex part. Something like fetch the latest of B related to each record of A. Fetch the latest of F related to the record of D which is related to each record of A. 

There were no CPU intensive calculations done. Though the overall DB fetches were complex, as mentioned in my response to Vijay, I just broke down the issue and tried the single queryset with select_related and prefetch_related and also "defer" and "only" to make the query lighter. But weren't helping. In fact the prefetch related on B and C were making the overall query heavier and hence, increasing the response time further.

Please note the above example was just the subset of the overall DB relations and models.

-James

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/b169cc33-8120-441c-847d-6977d0e0712a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.