Thursday 14 May 2009

Reporting assertions as a way for test parametrization

There's been a discussion recently on Testing In Pyton mailing list on introducing test parametrization. Here's my approach to the problem.

The idea, instead of finding a way to generate test methods, is to promote assertions to report the results in similar manner that testmethods do. In order to achieve that, I had to override few methods in TestCase, but the resulting API looks quite clean.
You can grab the source from here. And here's an example. Run this:

import unittest
from reporting_assertion import TestCase, reporting

class MyTestCase(TestCase):
@reporting
def some_assertion(self, a, b):
assert a == b
def test_example(self):
for x in (1, 2, 3):
self.some_assertion(x, 2)

if __name__ == "__main__":
unittest.main()
In order to get this report:
$ python example.py   
FF.
======================================================================
FAIL: test_example.some_assertion(1, 2) (__main__.MyTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/[path]/reporting_assertion/reporting_assertion.py", line 12, in wrapper
ret = assertion(self, *args, **kwargs)
File "example.py", line 8, in some_assertion
assert a == b
AssertionError

======================================================================
FAIL: test_example.some_assertion(3, 2) (__main__.MyTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/[path]/reporting_assertion/reporting_assertion.py", line 12, in wrapper
ret = assertion(self, *args, **kwargs)
File "example.py", line 8, in some_assertion
assert a == b
AssertionError

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (failures=2)


This approach has few drawbacks. You can't collect assertions when doing collect-only run, it's tightly coupled with TestCase implementation (needs to know how to report results), and - for the same reason - it works only for assertions bound to TestCase instance (the decorator has an optional _self= keyword argument for decorating unbound assertions, but the results are not reusable across TestCase instances). The last problem is __str__ evaluation: if the objects under have side-effects in this method, you might get quite unexpected results.

It is, however quite powerful: the data fed into the assertions is generated in the code itself (so the parameters don't need to be static). It's also very easy to use: just decorate your assertion in order to get the reports.