Re: memory leak? Am I taking crazy pills?

Ah ok. So it sounds like if I drop "print(gc.garbage)" into my code at the right points, I will be able to see where the leak is occurring? I did that and then load tested it (not via anything tricky, I'm just using "ab") via the dev server running locally, but I'm not seeing anything, it's always [] even for n=1000 or 10000. (100k is running right now but we were seeing this problem below roughly 10k)

Which all makes me think it could be a problem on heroku's end. I'm going to try emailing them and testing this in production and then devising ways to test each component we're using (mainly pymongo and BigML), but any other thoughts on how I could track this sucker down? I'd love to have complete confidence that it's not leaking running locally at least.

Thanks!

Dan

On Nov 23, 2012, at 12:30 PM, akaariai wrote:

On 22 marras, 08:27, Dan Ancona <afightingfa...@gmail.com> wrote:

python 2.7.3, django 1.4.2

Our app (a fairly simple tastypie API that runs some machine learning foo on mongodb records) is deployed to heroku and is leaking memory at a pretty good clip. Hitting heroku memory errors after about an hour or two, under significant load.

Wandered through some various profiling options and eventually started to get suspicious of tastypie. So I tried ripping that out. And then everything else, until I finally got down to a django view that has nothing but this in it:

def stack(request):

   gc.set_debug(gc.DEBUG_LEAK)

   print "Uncollectable garbage", gc.garbage

   return HttpResponse('hi')

If you set gc.DEBUG_LEAK then the GC will not free anything, instead
it will append garbage to gc.garbage. So, getting everything into
gc.garbage is expected in that case.

You will want to remove the gc.set_debug() call.

Here is an example without the call:

import gc
from django.http import HttpResponse

# Objects which have __del__ defined and are part of reference cycle
will be uncollectable.
class A(object):
   def __del__(self):
       pass
class B(object):
   pass

def stack(request):
   a = A()
   b = B()
   # Lets create a cycle
   a.b = b
   b.a = a
   print(gc.garbage)
   return HttpResponse('hi')


Load the view a couple of times and you will see garbage accumulating.
It will contain only the A objects - they are uncollectable because of
the __del__ method.

- Anssi

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.