Re: unit tests and 'system' tests with history

On Wed, Nov 30, 2011 at 2:48 AM, Gelonida N <gelonida@gmail.com> wrote:

2.) Stress / System tests
----------------------------------

These are often called 'integration tests', and are used to test whether all of the (properly unit-tested) components of the system actually function properly when assembled together.

 

I'd also like to do some other tests, where I work on a system and where
I am able to suffer from my history.

Example (the example is perhaps too simple to see what I'm aiming it)
- I create an entry
- I rename an entry
- I remove an entry

If I had three independent tests I would never find out if I had a weird
bug, which would make the system fail if I tried to remove an entry
which has been renamed before.

As an aside, I'm not sure that I would use this sort of test framework for these kinds of bugs -- there are just too many ways for the system to fail if you start thinking this way:

• What if it only fails when the entry has been renamed?
• What if it only fails when the entry has been renamed to something longer?
• What if it only fails when the entry has been renamed while someone else was editing it?
• What if it only fails when the entry has a name with no vowels in it, which was also used as another user's password, but not in the last six weeks? :)

You can only simulate so many conditions, and lots of bugs can lurk undetected even with a whole suite of such tests. Even worse, if the code changes, then the exact bug that you were testing for might appear to be fixed, but actually changes very slightly, and doesn't get picked up any more.

In a case like this, I would normally wait to find out that there was a problem in the first place, and then institute a 'regression test' -- figure out the cause of the original bug (say, renaming the entry put the record into an inconsistent state) and then set up a test which:

1. instantiates a correct record directly
2. performs the rename operation that was failing
3. checks that the record is still consistent under every constraint.

And I would do all of these at the lowest level that makes sense -- certainly not by starting at the HTTP level.

 

What would be a recommended way to run such system tests?

'just create a unit test, which is just calling sunb tests explicitely
in the required order?)

For integration testing, that's exactly what I do -- set up a unit test (not really a unit test in this case, but it's a subclass of unittest.TestCase) that uses the django test client to perform a small sequence of operations; usually a GET, checking the returned context, and then a POST, checking the return status (200 for form errors; 30x for successful redirects). I will set up any required objects in the database first (in fixtures, setUp or the test method itself), and test them afterwards.

Usually there aren't too many of these tests, though -- just enough that I know that the GET and POST methods are not failing spectacularly. The actual logic is tested with a bunch of much faster unit tests.

As part of these system tests I would like to add some of my unit tests,
but such, that the data base is NOT reset after each test run.

As a rule, I find it makes much more sense to declare the state of the database before each test, and then just test a few operations. As long as you properly test the consistency of the data after each operation, then you shouldn't run into the "cascading failures" that you can get if each test is dependent on the ones before it (or worse, cascading successes, if one bug cancels out the effects of the another)

If you're actually looking for stress-testing, then there are a number of other frameworks for that (ab, Bees with Machine Guns, etc) that will tell you how your site performs under load. Host-based testing should be there to tell you that your site behaves predictably and correctly under normal operation.

Ian

 

As you can see from my previous questions I'm used to 'non-atomic'
testing (This was for systems which needed a very huge startup time
and where taking snapshots was virtually impossible)

Thus I often combined unit tests / system tests.

These stress tests shall be able to run for several hours / days,
whereas unit tests would run in a few minutes.


Thanks for any suggestions


--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

--
Regards,
Ian Clelland
<clelland@gmail.com>

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.