From 13f35da24a4c546d26a00d0ab3d8012bed6523a2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marek=20Marczykowski-G=C3=B3recki?= Date: Sat, 30 Nov 2019 04:38:10 +0100 Subject: [PATCH] doc/tests: extend qubes-specific quirks in tests - per-VM (template) tests - extra tests loader --- doc/qubes-tests.rst | 105 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 104 insertions(+), 1 deletion(-) diff --git a/doc/qubes-tests.rst b/doc/qubes-tests.rst index a5dd06ec..1cea7095 100644 --- a/doc/qubes-tests.rst +++ b/doc/qubes-tests.rst @@ -33,7 +33,11 @@ variables. :py:class:`qubes.tests.QubesTestCase` classes should be named named ``test_xxx_test_name``, where ``xxx`` is three-digit number. You may introduce some structure of your choice in this number. -FIXME: where are placed integration tests? +Integration tests for Qubes core features are stored in :file:`tests/integ/` +directory. Additional tests may be loaded from other packages (see extra test +loader below). Those tests are run only on real Qubes system and are not suitable +for running in VM or in Travis. Test classes of this category inherit from +:py:class:`qubes.tests.SystemTestCase`. Writing tests ------------- @@ -133,6 +137,105 @@ even entire class) to be skipped outside dom0. Use it freely:: # all tests in this class are skipped pass +VM tests +^^^^^^^^ + +Some integration tests verifies not only dom0 part of the system, but also VM +part. In those cases, it makes sense to iterate them for different templates. +Additionally, list of the templates can be dynamic (different templates +installed, only some considered for testing etc). +This can be achieved by creating a mixin class with the actual tests (a class +inheriting just from :py:class:`object`, instead of +:py:class:`qubes.tests.SystemTestCase` or :py:class:`unittest.TestCase`) and +then create actual test classes dynamically using +:py:func:`qubes.tests.create_testcases_for_templates`. +Test classes created this way will have :py:attr:`template` set to the template +name under test and also this template will be set as the default template +during the test execution. +The function takes a test class name prefix (template name will be appended to +it after '_' separator), a classes to inherit from (in most cases the just +created mixin and :py:class:`qubes.tests.SystemTestCase`) and a current module +object (use `sys.modules[__name__]`). The function will return created test +classes but also add them to the appropriate module (pointed by the *module* +parameter). This should be done in two cases: + +* :py:func:`load_tests` function - when test loader request list of tests +* on module import time, using a wrapper + :py:func:`qubes.tests.maybe_create_testcases_on_import` (will call the + function only if explicit list of templates is given, to avoid loading + :file:`qubes.xml` when just importing the module) + +An example boilerplate looks like this:: + + def create_testcases_for_templates(): + return qubes.tests.create_testcases_for_templates('TC_00_AppVM', + TC_00_AppVMMixin, qubes.tests.SystemTestCase, + module=sys.modules[__name__]) + + def load_tests(loader, tests, pattern): + tests.addTests(loader.loadTestsFromNames( + create_testcases_for_templates())) + return tests + + qubes.tests.maybe_create_testcases_on_import(create_testcases_for_templates) + +This will by default create tests for all the templates installed in the system. +Additionally, it is possible to control this process using environment +variables: + +* `QUBES_TEST_TEMPLATES` - space separated list of templates to test +* `QUBES_TEST_LOAD_ALL` - create tests for all the templates (by inspecting + the :file:`qubes.xml` file), even at module import time + +This is dynamic test creation is intentionally made compatible with Nose2 test +runner and its load_tests protocol implementation. + +Extra tests +^^^^^^^^^^^ + +Most tests live in this package, but it is also possible to store tests in other +packages while still using infrastructure provided here and include them in the +common test run. Loading extra tests is implemented in +:py:mod:`qubes.tests.extra`. To write test to be loaded this way, you need to +create test class(es) as usual. You can also use helper class +:py:class:`qubes.tests.extra.ExtraTestCase` (instead of +:py:class:`qubes.tests.SystemTestCase`) which provide few convenient functions +and hide usage of asyncio for simple cases (like `vm.start()`, `vm.run()`). + +The next step is to register the test class(es). You need to do this by defining +entry point for your package. There are two groups: + +* `qubes.tests.extra` - for general tests (called once) +* `qubes.tests.extra.for_template` - for per-VM tests (called for each template + under test) + +As a name in the group, choose something unique, preferably package name. An +object reference should point at the function that returns a list of test +classes. + +Example :file:`setup.py`:: + + from setuptools import setup + + setup( + name='splitgpg', + version='1.0', + packages=['splitgpg'], + entry_points={ + 'qubes.tests.extra.for_template': + 'splitgpg = splitgpg.tests:list_tests', + } + ) + +The test loading process can be additionally controlled with environment +variables: + +* `QUBES_TEST_EXTRA_INCLUDE` - space separated list of tests to include (named + by a name in an entry point, `splitgpg` in the above example); if defined, only + those extra tests will be loaded + +* `QUBES_TEST_EXTRA_EXCLUDE` - space separated list of tests to exclude + Module contents ---------------