1 """
2 This is a top level package, hosting the entire CSB test framework. It is divided
3 into several major parts:
4
5 - test cases, located under csb.test.cases
6 - test data, in C{/csb/test/data} (not a package)
7 - test console, in C{/csb/test/app.py}
8
9 This module, csb.test, contains all the glue-code functions, classes and
10 decorators you would need in order to write tests for CSB.
11
12 1. Configuration and Tree
13
14 L{Config<csb.test.Config>} is a common config object shared between CSB
15 tests. Each config instance contains properties like:
16
17 - data: the data folder, automatically discovered and loaded in
18 csb.test.Config.DATA at module import time
19 - temp: a default temp folder, which test cases can use
20
21 Each L{Config<csb.test.Config>} provides a convenient way to retrieve
22 files from C{/csb/test/data}. Be sure to check out L{Config.getTestFile}
23 and L{Config.getPickle}. In case you need a temp file, use
24 L{Config.getTempStream} or have a look at L{csb.io.TempFile} and
25 L{csb.io.TempFolder}.
26
27 All test data files should be placed in the C{data} folder. All test
28 modules must be placed in the root package: csb.test.cases. There is
29 a strict naming convention for test modules: the name of a test module
30 should be the same as the name of the CSB API package it tests. For
31 example, if you are writing tests for C{csb/bio/io/__init__.py}, the
32 test module must be C{csb/test/cases/bio/io/__init__.py}. C{csb.test.cases}
33 is the root package of all test modules in CSB.
34
35 2. Writing Tests
36
37 Writing a test is easy. All you need is to import csb.test and then
38 create your own test cases, derived from L{csb.test.Case}:
39
40 >>> import csb.test
41 >>> @csb.test.unit
42 class TestSomeClass(csb.test.Case):
43 def setUp(self):
44 super(TestSomeClass, self).setUp()
45 # do something with self.config here...
46
47 In this way your test case instance is automatically equipped with a
48 reference to the test config, so your test method can be:
49
50 >>> @csb.test.unit
51 class TestSomeClass(csb.test.Case):
52 def testSomeMethod(self):
53 myDataFile = self.config.getTestFile('some.file')
54 self.assert...
55
56 The "unit" decorator marks a test case as a collection of unit tests.
57 All possibilities are: L{csb.test.unit}, L{csb.test.functional}, L{csb.test.custom},
58 and L{csb.test.regression}.
59
60 Writing custom (a.k.a. "data", "slow", "dynamic") tests is a little bit
61 more work. Custom tests must be functions, not classes. Basically a
62 custom test is a function, which builds a unittest.TestSuite instance
63 and then returns it when called without arguments.
64
65 Regression tests are usually created in response to reported bugs. Therefore,
66 the best practice is to mark each test method with its relevant bug ID:
67
68 >>> @csb.test.regression
69 class SomeClassRegressions(csb.test.Case)
70 def testSomeFeature(self)
71 \"""
72 @see: [CSB 000XXXX]
73 \"""
74 # regression test body...
75
76 3. Style Guide:
77
78 - name test case packages as already described
79 - group tests in csb.test.Case-s and name them properly
80 - prefix test methods with "test", like "testParser" - very important
81 - use camelCase for methods and variables. This applies to all the
82 code under csb.test (including test) and does not apply to the rest
83 of the library!
84 - for functional tests it's okay to define just one test method: runTest
85 - for unit tests you should create more specific test names, for example:
86 "testParseFile" - a unit test for some method called "parse_file"
87 - use csb.test decorators to mark tests as unit, functional, regression, etc.
88 - make every test module executable::
89
90 if __name__ == '__main__':
91 csb.test.Console() # Discovers and runs all test cases in the module
92
93 4. Test Execution
94
95 Test discovery is handled by C{test builders} and a test runner
96 C{app}. Test builders are subclasses of L{AbstractTestBuilder}.
97 For every test type (unit, functional, regression, custom) there is a
98 corresponding test builder. L{AnyTestBuilder} is a special builder which
99 scans for unit, regression and functional tests at the same time.
100
101 Test builder classes inherit the following test discovery methods:
102
103 - C{loadTests} - load tests from a test namespace. Wildcard
104 namespaces are handled by C{loadAllTests}
105 - C{loadAllTests} - load tests from the given namespace, and
106 from all sub-packages (recursive)
107 - C{loadFromFile} - load tests from an absolute file name
108 - C{loadMultipleTests} - calls C{loadTests} for a list of
109 namespaces and combines all loaded tests in a single suite
110
111 Each of those return test suite objects, which can be directly executed
112 with python's unittest runner.
113
114 Much simpler way to execute a test suite is to use our test app
115 (C{csb/test/app.py}), which is simply an instance of L{csb.test.Console}::
116
117 $ python csb/test/app.py --help
118
119 The app has two main arguments:
120
121 - test type - tells the app which TestBuilder to use for test dicsovery
122 ("any" triggers L{AnyTestBuilder}, "unit" - L{UnitTestBuilder}, etc.)
123 - test namespaces - a list of "dotted" test modules, for example::
124
125 csb.test.cases.bio.io.* # io and sub-packages
126 csb.test.cases.bio.utils # only utils
127 . # current module
128
129 In addition to running the app from the command line, you can run it
130 also programmatically by instantiating L{csb.test.Console}. You can
131 construct a test console object by passing a list of test namespace(s)
132 and a test builder class to the Console's constructor.
133
134
135 5. Commit Policies
136
137 Follow these guidelines when making changes to the repository:
138
139 - B{no bugs in "trunk"}: after fixing a bug or implementing a new
140 feature, make sure at least the default test set passes by running
141 the test console without any arguments. This is equivalent to:
142 app.py -t any "csb.test.cases.*". (If no test case from this set covers
143 the affected code, create a test case first, as described in the other
144 policies)
145
146 - B{no recurrent issues}: when a bug is found, first write a regression
147 test with a proper "@see: BugID" tag in the docstring. Run the test
148 to make sure it fails. After fixing the bug, run the test again before
149 you commit, as required by the previous policy
150
151 - B{test all new features}: there should be a test case for every new feature
152 we implement. One possible approach is to write a test case first and
153 make sure it fails; when the new feature is ready, run the test again
154 to make sure it passes
155
156 @warning: for compatibility reasons do NOT import and use the unittest module
157 directly. Always import unittest from csb.test, which is guaranteed
158 to be python 2.7+ compatible. The standard unittest under python 2.6
159 is missing some features, that's why csb.test will take care of
160 replacing it with unittest2 instead.
161 """
162 from __future__ import print_function
163
164 import os
165 import sys
166 import imp
167 import types
168 import time
169 import getopt
170 import tempfile
171 import traceback
172
173 import csb.io
174 import csb.core
175
176 try:
177 from unittest import skip, skipIf
178 import unittest
179 except ImportError:
180 import unittest2 as unittest
181
182 from abc import ABCMeta, abstractproperty
191
193 """
194 General CSB Test Config. Config instances contain the following properties:
195
196 - data - path to the CSB Test Data directory. Default is L{Config.DATA}
197 - temp - path to the system's temp directory. Default is L{Config.TEMP}
198 - config - the L{Config} class
199 """
200
201 DATA = os.path.join(os.path.abspath(os.path.dirname(__file__)), 'data')
202 """
203 @cvar: path to the default test data directory: <install dir>/csb/test/data
204 """
205 TEMP = os.path.abspath(tempfile.gettempdir())
206 """
207 @cvar: path to the default system's temp directory
208 """
209
215
216 @staticmethod
218 """
219 Override the default L{Config.DATA} with a new data root directory.
220
221 @param path: full directory path
222 @type path: str
223 """
224 if not os.path.isdir(path):
225 raise IOError('Path not found: {0}'.format(path))
226
227 Config.DATA = os.path.abspath(path)
228
229 @property
231 """
232 Test data directory
233 @rtype: str
234 """
235 return self.__data
236
237 @property
239 """
240 Test temp directory
241 @rtype: str
242 """
243 return self.__temp
244
246 """
247 Search for C{fileName} in the L{Config.DATA} directory.
248
249 @param fileName: the name of a test file to retrieve
250 @type fileName: str
251 @param subDir: scan a sub-directory of L{Config.DATA}
252 @type subDir: str
253
254 @return: full path to C{fileName}
255 @rtype: str
256
257 @raise IOError: if no such file is found
258 """
259 file = os.path.join(self.data, subDir, fileName)
260 if not os.path.isfile(file):
261 raise IOError('Test file not found: {0}'.format(file))
262 return file
263
265 """
266 Same as C{self.getTestFile}, but try to unpickle the data in the file.
267
268 @param fileName: the name of a test file to retrieve
269 @type fileName: str
270 @param subDir: scan a sub-directory of L{Config.DATA}
271 @type subDir: str
272 """
273 file = self.getTestFile(fileName, subDir)
274 return csb.io.Pickle.load(open(file, 'rb'))
275
276 - def getContent(self, fileName, subDir=''):
277 """
278 Same as C{self.getTestFile}, but also read and return the contents of
279 the file.
280
281 @param fileName: the name of a test file to retrieve
282 @type fileName: str
283 @param subDir: scan a sub-directory of L{Config.DATA}
284 @type subDir: str
285 """
286 with open(self.getTestFile(fileName, subDir)) as f:
287 return f.read()
288
290 """
291 Return a temporary file stream::
292
293 with self.getTempStream() as tmp:
294 tmp.write(something)
295 tmp.flush()
296 file_name = tmp.name
297
298 @param mode: file open mode (text, binary), default=t
299 @type mode: str
300 @rtype: file stream
301 """
302 return csb.io.TempFile(mode=mode)
303
305 """
306 Try to deserialize some pickled data files. Call L{Config.updateDataFiles}
307 if the pickles appeared incompatible with the current interpreter.
308 """
309 try:
310 self.getPickle('1nz9.model1.pickle')
311 except:
312 self.updateDataFiles()
313
337
338 -class Case(unittest.TestCase):
339 """
340 Base class, defining a CSB Test Case. Provides a default implementation
341 of C{unittest.TestCase.setUp} which grabs a reference to a L{Config}.
342 """
343
344 @property
346 """
347 Test config instance
348 @rtype: L{Config}
349 """
350 return self.__config
351
353 """
354 Provide a reference to the CSB Test Config in the C{self.config} property.
355 """
356 self.__config = Config()
357 assert hasattr(self.config, 'data'), 'The CSB Test Config must contain the data directory'
358 assert self.config.data, 'The CSB Test Config must contain the data directory'
359
361 """
362 Re-raise the last exception with its full traceback, but modify the
363 argument list with C{addArgs} and the original stack trace.
364
365 @param addArgs: additional arguments to append to the exception
366 @type addArgs: tuple
367 """
368 klass, ex, _tb = sys.exc_info()
369 ex.args = list(ex.args) + list(addArgs) + [''.join(traceback.format_exc())]
370
371 raise klass(ex.args)
372
374
375 if first == second:
376 return
377 if delta is not None and places is not None:
378 raise TypeError("specify delta or places not both")
379
380 if delta is not None:
381
382 if abs(first - second) <= delta:
383 return
384
385 m = '{0} != {1} within {2} delta'.format(first, second, delta)
386 msg = self._formatMessage(msg, m)
387
388 raise self.failureException(msg)
389
390 else:
391 if places is None:
392 places = 7
393
394 return super(Case, self).assertAlmostEqual(first, second, places=places, msg=msg)
395
397 """
398 Fail if it took more than C{duration} seconds to invoke C{callable}.
399
400 @param duration: maximum amount of seconds allowed
401 @type duration: float
402 """
403
404 start = time.time()
405 callable(*args, **kargs)
406 execution = time.time() - start
407
408 if execution > duration:
409 self.fail('{0}s is slower than {1}s)'.format(execution, duration))
410
411 @classmethod
413 """
414 Run this test case.
415 """
416 suite = unittest.TestLoader().loadTestsFromTestCase(cls)
417 runner = unittest.TextTestRunner()
418
419 return runner.run(suite)
420
423
425 """
426 This is a base class, defining a test loader which exposes the C{loadTests}
427 method.
428
429 Subclasses must override the C{labels} abstract property, which controls
430 what kind of test cases are loaded by the test builder.
431 """
432
433 __metaclass__ = ABCMeta
434
435 @abstractproperty
438
440 """
441 Load L{csb.test.Case}s from a module file.
442
443 @param file: test module file name
444 @type file: str
445
446 @return: a C{unittest.TestSuite} ready for the test runner
447 @rtype: C{unittest.TestSuite}
448 """
449 mod = self._loadSource(file)
450 suite = unittest.TestLoader().loadTestsFromModule(mod)
451 return unittest.TestSuite(self._filter(suite))
452
454 """
455 Load L{csb.test.Case}s from the given CSB C{namespace}. If the namespace
456 ends with a wildcard, tests from sub-packages will be loaded as well.
457 If the namespace is '__main__' or '.', tests are loaded from __main__.
458
459 @param namespace: test module namespace, e.g. 'csb.test.cases.bio' will
460 load tests from '/csb/test/cases/bio/__init__.py'
461 @type namespace: str
462
463 @return: a C{unittest.TestSuite} ready for the test runner
464 @rtype: C{unittest.TestSuite}
465 """
466 if namespace.strip() == '.*':
467 namespace = '__main__.*'
468 elif namespace.strip() == '.':
469 namespace = '__main__'
470
471 if namespace.endswith('.*'):
472 return self.loadAllTests(namespace[:-2])
473 else:
474 loader = unittest.TestLoader()
475 tests = loader.loadTestsFromName(namespace)
476 return unittest.TestSuite(self._filter(tests))
477
479 """
480 Load L{csb.test.Case}s from a list of given CSB C{namespaces}.
481
482 @param namespaces: a list of test module namespaces, e.g.
483 ('csb.test.cases.bio', 'csb.test.cases.bio.io') will
484 load tests from '/csb/test/cases/bio.py' and
485 '/csb/test/cases/bio/io.py'
486 @type namespaces: tuple of str
487
488 @return: a C{unittest.TestSuite} ready for the test runner
489 @rtype: C{unittest.TestSuite}
490 """
491 if not csb.core.iterable(namespaces):
492 raise TypeError(namespaces)
493
494 return unittest.TestSuite(self.loadTests(n) for n in namespaces)
495
497 """
498 Load L{csb.test.Case}s recursively from the given CSB C{namespace} and
499 all of its sub-packages. Same as::
500
501 builder.loadTests('namespace.*')
502
503 @param namespace: test module namespace, e.g. 'csb.test.cases.bio' will
504 load tests from /csb/test/cases/bio/*'
505 @type namespace: str
506
507 @return: a C{unittest.TestSuite} ready for the test runner
508 @rtype: C{unittest.TestSuite}
509 """
510 suites = []
511
512 try:
513 base = __import__(namespace, level=0, fromlist=['']).__file__
514 except ImportError:
515 raise InvalidNamespaceError('Namespapce {0} is not importable'.format(namespace))
516
517 if os.path.splitext(os.path.basename(base))[0] != '__init__':
518 suites.append(self.loadTests(namespace))
519
520 else:
521
522 for entry in os.walk(os.path.dirname(base)):
523
524 for item in entry[2]:
525 file = os.path.join(entry[0], item)
526 if extension and item.endswith(extension):
527 suites.append(self.loadFromFile(file))
528
529 return unittest.TestSuite(suites)
530
532 """
533 Import and return the Python module identified by C{path}.
534
535 @note: Module objects behave as singletons. If you import two different
536 modules and give them the same name in imp.load_source(mn), this
537 counts for a redefinition of the module originally named mn, which
538 is basically the same as reload(mn). Therefore, you need to ensure
539 that for every call to imp.load_source(mn, src.py) the mn parameter
540 is a string that uniquely identifies the source file src.py.
541 """
542 name = os.path.splitext(os.path.abspath(path))[0]
543 name = name.replace('.', '-').rstrip('__init__').strip(os.path.sep)
544
545 return imp.load_source(name, path)
546
548 """
549 Extract test cases recursively from a test C{obj} container.
550 """
551 cases = []
552 if isinstance(obj, unittest.TestSuite) or csb.core.iterable(obj):
553 for item in obj:
554 cases.extend(self._recurse(item))
555 else:
556 cases.append(obj)
557 return cases
558
560 """
561 Filter a list of objects using C{self.labels}.
562 """
563 filtered = []
564
565 for test in self._recurse(tests):
566 for label in self.labels:
567 if hasattr(test, label) and getattr(test, label) is True:
568 filtered.append(test)
569
570 return filtered
571
573 """
574 Build a test suite of cases, marked as either unit, functional or regression
575 tests. For detailed documentation see L{AbstractTestBuilder}.
576 """
577 @property
580
582 """
583 Build a test suite of cases, marked as unit tests.
584 For detailed documentation see L{AbstractTestBuilder}.
585 """
586 @property
589
591 """
592 Build a test suite of cases, marked as functional tests.
593 For detailed documentation see L{AbstractTestBuilder}.
594 """
595 @property
598
600 """
601 Build a test suite of cases, marked as regression tests.
602 For detailed documentation see L{AbstractTestBuilder}.
603 """
604 @property
607
609 """
610 Build a test suite of cases, marked as custom tests. CustomTestBuilder will
611 search for functions, marked with the 'custom' test decorator, which return
612 a dynamically built C{unittest.TestSuite} object when called without
613 parameters. This is convenient when doing data-related tests, e.g.
614 instantiating a single type of a test case many times iteratively, for
615 each entry in a database.
616
617 For detailed documentation see L{AbstractTestBuilder}.
618 """
619 @property
622
624
625 mod = self._loadSource(file)
626 suites = self._inspect(mod)
627
628 return unittest.TestSuite(suites)
629
646
648
649 objects = map(lambda n: getattr(module, n), dir(module))
650 return self._filter(objects)
651
653 """
654 Filter a list of objects using C{self.labels}.
655 """
656 filtered = []
657
658 for obj in factories:
659 for label in self.labels:
660 if hasattr(obj, label) and getattr(obj, label) is True:
661 suite = obj()
662 if not isinstance(suite, unittest.TestSuite):
663 raise ValueError('Custom test function {0} must return a '
664 'unittest.TestSuite, not {1}'.format(obj.__name__, type(suite)))
665 filtered.append(suite)
666
667 return filtered
668
670 """
671 A class decorator, used to label unit test cases.
672
673 @param klass: a C{unittest.TestCase} class type
674 @type klass: type
675 """
676 if not isinstance(klass, type):
677 raise TypeError("Can't apply class decorator on {0}".format(type(klass)))
678
679 setattr(klass, Attributes.UNIT, True)
680 return klass
681
683 """
684 A class decorator, used to label functional test cases.
685
686 @param klass: a C{unittest.TestCase} class type
687 @type klass: type
688 """
689 if not isinstance(klass, type):
690 raise TypeError("Can't apply class decorator on {0}".format(type(klass)))
691
692 setattr(klass, Attributes.FUNCTIONAL, True)
693 return klass
694
696 """
697 A class decorator, used to label regression test cases.
698
699 @param klass: a C{unittest.TestCase} class type
700 @type klass: type
701 """
702 if not isinstance(klass, type):
703 raise TypeError("Can't apply class decorator on {0}".format(type(klass)))
704
705 setattr(klass, Attributes.REGRESSION, True)
706 return klass
707
709 """
710 A function decorator, used to mark functions which build custom (dynamic)
711 test suites when called.
712
713 @param function: a callable object, which returns a dynamically compiled
714 C{unittest.TestSuite}
715 @type function: callable
716 """
717 if isinstance(function, type):
718 raise TypeError("Can't apply function decorator on a class")
719 elif not hasattr(function, '__call__'):
720 raise TypeError("Can't apply function decorator on non-callable {0}".format(type(function)))
721
722 setattr(function, Attributes.CUSTOM, True)
723 return function
724
725 -def skip(reason, condition=None):
726 """
727 Mark a test case or method for skipping.
728
729 @param reason: message
730 @type reason: str
731 @param condition: skip only if the specified condition is True
732 @type condition: bool/expression
733 """
734 if isinstance(reason, types.FunctionType):
735 raise TypeError('skip: no reason specified')
736
737 if condition is None:
738 return unittest.skip(reason)
739 else:
740 return unittest.skipIf(condition, reason)
741
743 """
744 Build and run all tests of the specified namespace and kind.
745
746 @param namespace: a dotted name, which specifies the test module
747 (see L{csb.test.AbstractTestBuilder.loadTests})
748 @type namespace: str
749 @param builder: test builder to use
750 @type builder: any L{csb.test.AbstractTestBuilder} subclass
751 @param verbosity: verbosity level for C{unittest.TestRunner}
752 @type verbosity: int
753 @param update: if True, refresh all pickles in csb/test/data
754 @type update: bool
755 """
756
757 BUILDERS = {'unit': UnitTestBuilder, 'functional': FunctionalTestBuilder,
758 'custom': CustomTestBuilder, 'any': AnyTestBuilder,
759 'regression': RegressionTestBuilder}
760
761 USAGE = r"""
762 CSB Test Runner Console. Usage:
763
764 python {0.program} [-u] [-t type] [-v verbosity] namespace(s)
765
766 Options:
767 namespace(s) A list of CSB test dotted namespaces, from which to
768 load tests. '__main__' and '.' are interpreted as the
769 current module. If a namespace ends with an asterisk
770 '.*', all sub-packages will be scanned as well.
771
772 Examples:
773 "csb.test.cases.bio.*"
774 "csb.test.cases.bio.io" "csb.test.cases.bio.utils"
775 "."
776
777 -t type Type of tests to load from each namespace. Possible
778 values are:
779 {0.builders}
780
781 -v verbosity Verbosity level passed to unittest.TextTestRunner.
782
783 -u update-files Force update of the test pickles in csb/test/data.
784 """
785
786 - def __init__(self, namespace=('__main__',), builder=AnyTestBuilder, verbosity=1,
787 update=False, argv=None):
805
806 @property
808 return self._namespace
809 @namespace.setter
811 if csb.core.iterable(value):
812 self._namespace = list(value)
813 else:
814 self._namespace = [value]
815
816 @property
819 @builder.setter
821 self._builder = value
822
823 @property
825 return self._verbosity
826 @verbosity.setter
828 self._verbosity = value
829
830 @property
833
834 @property
837
838 @property
841 @update.setter
843 self._update = bool(value)
844
857
858 - def exit(self, message=None, code=0, usage=True):
866
868
869 try:
870
871 options, args = getopt.getopt(argv, 'hut:v:', ['help', 'update-files', 'type=', 'verbosity='])
872
873 for option, value in options:
874 if option in('-h', '--help'):
875 self.exit(message=None, code=0)
876 if option in('-t', '--type'):
877 try:
878 self.builder = Console.BUILDERS[value]
879 except KeyError:
880 self.exit(message='E: Invalid test type "{0}".'.format(value), code=2)
881 if option in('-v', '--verbosity'):
882 try:
883 self.verbosity = int(value)
884 except ValueError:
885 self.exit(message='E: Verbosity must be an integer.', code=3)
886 if option in('-u', '--update-files'):
887 self.update = True
888
889 if len(args) > 0:
890 self.namespace = list(args)
891
892 except getopt.GetoptError as oe:
893 self.exit(message='E: ' + str(oe), code=1)
894
895
896 if __name__ == '__main__':
897
898 Console()
899