Re: Inconsistent Django test results depending upon how the test is called in Django 1.5.1 running on Python 2.7.4

Hi Russ,

Thanks for your detailed response, deserving of this detailed investigation:

I eventually found that it is the simple declaration of a model within the test suites that is causing the problem:

django.contrib.contents.tests causes the problem - even if I removed all of the tests and just imported the file:

I copied and pared the file down to this which still causes the error:

from django.db import models      class ConcreteModel(models.Model):      name = models.CharField(max_length=10)      class ProxyModel(ConcreteModel):      class Meta:  	proxy = True  

If I remove the ProxyModel class the error changes and becomes a recursion depth error.

Renaming the classes does not get rid of the error (and the class names are not used in my apps).

The infinite recursion problem feels somehow related. My models do do something unusual: Much of the application revolves around models belonging to a company so, for convenience, all models have a company property that returns the company that owns them - it often does this by referencing objects on it. In the case of a company, the company property returns self. In the case of a note (the model where we're getting this problem), the content_object is a company and the note's company method accesses its content_object company attribute:

class Note(TimeStampedModel):      content_type = models.ForeignKey(ContentType)      object_id = models.PositiveIntegerField()      content_object = generic.GenericForeignKey('content_type', 'object_id')      ...        @property      def company(self):  	return self.content_object.company    class Company(BaseModel, ModelWithRandomisedToken):      ...        @property      def company(self):  	return self  

I think this is progress. Declaring ANY model class within the test suite causes the infinite recursion issue on the access. So my theory is that something in the django caching somewhere is getting partly updated when the model is declared but not fully sorted so the caches are broken with respect to content types. This has two symptoms - object references being None and infinite recursion, depending upon how the error gets encountered.

I'm guessing that there is some 'tidying up' of content types after all the models are imported that does not get done for a model arbitrarily defined in the test suite.

Does it sound like I am going in the right direction?

Thanks again for your help.

Cheers, Paul

On 2 December 2013 11:10, Russell Keith-Magee <russell@keith-magee.com> wrote:

On Mon, Dec 2, 2013 at 8:24 AM, Paul Whipp <paul.whipp@gmail.com> wrote:

I have test cases that pass when tested individually, pass when the full app is tested but fail when the tests for the entire project are run:

  (lsapi)~ $ django test  Creating test database for alias 'default'...  .................................................................................................................................................s.....................................................................................................................................x..........................................................................................................................................................................................................................................................Exception RuntimeError: 'maximum recursion depth exceeded' in <function remove at 0x13447d0> ignored  Exception RuntimeError: 'maximum recursion depth exceeded' in <function remove at 0x13447d0> ignored  Exception RuntimeError: 'maximum recursion depth exceeded' in <function remove at 0x13447d0> ignored  ...ss........E.....................................................................................................................E.E.Setting content_object to liquid app one  .EEE.EEEE...............................................................................................................  ======================================================================  ERROR: test_get_direct_notes (lsapi.tests.DirectGetTestCase)  ----------------------------------------------------------------------  Traceback (most recent call last):    File "ls-api/lsapi/tests.py", line 869, in _method  ....    File "ls-core/lscore/model/note.py", line 37, in company      return self.content_object.company  AttributeError: 'NoneType' object has no attribute 'company'  ...      (lsapi)~ $ django test lsapi.NotesTestCase.test_wrong_user_cannot_put  Creating test database for alias 'default'...  .  ----------------------------------------------------------------------  Ran 1 test in 0.241s    OK  Destroying test database for alias 'default'...      (lsapi)~ $ django test lsapi  Creating test database for alias 'default'...  ...................................................ss..................................................................................................................................Setting content_object to liquid app one  ...................Exception RuntimeError: 'maximum recursion depth exceeded while calling a Python object' in <_ctypes.DictRemover object at 0x46dac70> ignored  .....................................................................................................  ----------------------------------------------------------------------  Ran 303 tests in 71.469s    OK (skipped=2)  Destroying test database for alias 'default'...  

The 'django' command is an alias for django-admin.py. So the tests pass if run as a single case or as a test suite when testing the entire app but fail with errors if run when running the tests for all of the apps in the project.

Investigating the error in the full suite test: It is the result of an attribute being None when it 'should' have a value. The attribute is a content_object set up in a fixture.

This is an example of a really annoying edge cases that exists sometimes in Django's own test suite. Despite all our best efforts, sometimes unit tests don't clean up after themselves, so the order of execution matters. 

The test passes when it runs by itself because it gets a completely new environment when it runs. However, the test fails when run in the full suite because one of the tests that proceeds it leaves behind some detritus in the test database that effects the execution of the failing test.

Can anyone shed any light on this or let me know how I might address the problem?

There's two tasks here:

1) Find the individual test that, when paired with the failing test, causes test failure.

2) Fix the test that causes the problem.

The first task is laborious, but not especially difficult. Test execution order in Django is predictable, so you just need to do a binary search until you find a pair of tests that causes the problem. Lets say your suite contains apps A, B, C and D, and C.TestModel.test_problem that is causing the problem. Run:

 ./manage.py test A C.TestModel.test_problem

 ./manage.py test B C.TestModel.test_problem

 ./manage.py test D C.TestModel.test_problem

Normally, one of these three test executions will cause the failure, and the other two will pass. Let's B is the causes the failure. Now run:

 ./manage.py test B.TestModel1 C.TestModel.test_problem

 ./manage.py test B.TestModel2 C.TestModel.test_problem

 ./manage.py test B.TestModel3 C.TestModel.test_problem

Where TestModel[n] are the various test cases in the B app. Again, one of these will fail. Repeat again for individual tests; you should then have the problem narrowed down to a single test pair that causes the failure. Sometimes it will actually be every test in a given test case -- this will happen if the problem is actually being caused by the setUp method in a TestCase.

You've now identified the cause of the problem; now you just need to fix it.

This is a harder to answer without knowing the exact problem. However, you've suggested that the problem might be related to content types and fixtures. This is a common cause of the problem in Django's own test suite as well. It's caused because Django has an internal cache of content types to speed up the lookup process, and this cache *isn't* flushed at the start of each test by default. In most situations, it doesn't need to be, because the content types won't change between tests. However, if you're creating new temporary content types, or you've got fixtures that modify content types, the cache values will end up being incorrect, and will cause problems. 

*If* this is the problem, you just need to manually flush the content type cache at the end of the test that is causing the problem.

def tearDown(self):

    ContentType.objects.clear_cache()

The content type cache will be automatically repopulated on next use, so you don't have to worry about setting anything up again.

If content types aren't the problem, then you'll need to go spelunking to find the cause; essentially, you're looking for anything that *isn't* cleaned up at the end of the problem test that will cause the failing test to have a problem. Caches, stale files, and manually executed SQL are all examples of possible causes.

Hope that helps!

Yours,

Russ Magee %-)

--
You received this message because you are subscribed to a topic in the Google Groups "Django users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-users/OedhPc5bXYE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAJxq848%3DnNnpDF7tRbdUXFBkv2dM_1EgSifFvKpjfOw8JKrbng%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAMAB%2BPopyzRbnc-b8r4CmF1K93jY89hfAAeuMnzCdENtxxOHRg%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.