Writing tests, discovering and running them with Grok.testing ============================================================= :Author: ulif Prerequisites ------------- You need Grok 0.13 or newer installed. If you are developing on an older version of Grok, have a look at `z3c.testsetup How-To `_ instead. To follow along with this How-To, you can install a blank new project called Sample.:: $ cd /tmp $ grokproject Sample $ cd Sample Step by step ------------ You don't have to download anything else, because Grok already includes all the packages you need. Test that the project's testrunner works when you run ./bin/test. You should see something like this:: $ ./bin/test Running tests at level 1 Total: 0 tests, 0 failures, 0 errors in 0.000 seconds. Now you're ready to start adding your tests. There are several different kinds of tests and we'll scratch the surface of each one. The tests are either doctest or python and the difference is that you either write them in pure python or as text that you embedd in the doc strings of your classes or in separate plain-text (.txt) files. grok.testing is about discovering tests, not writing them or running them. Let's write a very simple unit test for the app.py that has been created.:: $ cd src/sample $ mkdir app_tests; cd app_tests $ touch __init__.py $ emacs test_app.py Notice the importance of creating an __init__.py file in that new directory to make it a valid Python package. Here's some example code that is really silly but at least proves that the test is found during runs of testrunner: .. code-block:: python """ Do a Python test on the app. :Test-Layer: python """ import unittest from sample.app import Sample class SimpleSampleTest(unittest.TestCase): "Test the Sample application" def test1(self): "Test that something works" grokapp = Sample() self.assertEqual(list(grokapp.keys()), []) The next thing is to tell grok.testing where to find this test. That's achieved by creating a file called tests.py in the 'src/sample' directory with the following content: .. code-block:: python import grok test_suite = grok.testing.register_all_tests('sample') The 'sample' string here denotes our ``sample`` package, which we want to be scanned for test files. We could also pass any other package name, which is available at runtime, in 'dotted name' notation. That's it! How cool is that? I love the Just Works'ism of writing "register_all_tests('sample')" and it does it. Let's now crack on with a "doc test". One way to get started on a doc test is to run grok in debug mode (typing './bin/zopectl debug') and when you're done, copy and paste what you've written in the interactive prompt. Here's a really simple doctest copy and paste after having run ./bin/zopectl debug. I call this file doctest.txt and I place it in the app_tests directory too:: Do a simple doctest test on the app. ************************************ :Test-Layer: unit When you create an instance there are no objects in it:: >>> from sample.app import Sample >>> grokapp = Sample() >>> list(grokapp.keys()) [] Make sure it is found by the testrunner:: $ ./bin/test Running tests at level 1 Running unit tests: Running: .. Ran 2 tests with 0 failures and 0 errors in 0.007 seconds. These tests are just making sure that you're code is working but we haven't yet tried to run any tests in a full blown Grok environment. That's called functional testing (or integration testing). The first test we'll make is a functional test in python. The setup of the complete Grok environment is called the "functional layer" Fortunately grokproject sets one up for you automatically which you can use in your functional test cases. The magic you need is the instance object `FunctionalLayer` which you'll find in the file testing.py. Again pay attention to the marker in the doc string is set to `python` and also bare in mind that the test is extremely simple. Save this as functional.py in the app_tests/ directory.: .. code-block:: python """ Do a functional test on the app. :Test-Layer: python """ from sample.app import Sample from sample.testing import FunctionalLayer from zope.app.testing.functional import FunctionalTestCase class SampleFunctionalTest(FunctionalTestCase): layer = FunctionalLayer class SimpleSampleFunctionalTest(SampleFunctionalTest): """ This the app in ZODB. """ def test_simple(self): """ test creating a Sample instance into Zope """ root = self.getRootFolder() root['instance'] = Sample() self.assertEqual(root.get('instance').__class__, Sample) Apologies for the stupidity of the test but at least it's picked up the next time we run bin/test:: $ ./bin/test Running tests at level 1 Running unit tests: Running: .. Ran 2 tests with 0 failures and 0 errors in 0.021 seconds. Running sample.testing.FunctionalLayer tests: Set up sample.testing.FunctionalLayer in 1.756 seconds. Running: . Ran 1 tests with 0 failures and 0 errors in 0.022 seconds. Tearing down left over layers: Tear down sample.testing.FunctionalLayer ... not supported Total: 3 tests, 0 failures, 0 errors in 1.956 seconds. As you can see, running the functional test is a lot slower. The first two tests took 0.007 seconds and now all three tests took 1.96 seconds. This is why developers sometimes refer to unit tests as "fast tests". Choosing to write unit tests versus functional tests depends upon what you want to achieve with your testing. Unit tests verify the correct behaviour of code in isolation, useful for ensuring that your code is loosely coupled. They are a quicker form of checking for simple errors than clicking around in a web browser, and they can help find bugs by exposing unseen edge cases. Functional testing ensures that the parts of your application work together as a whole, and are useful for ensuring that your application behaves as desired. If you are new to testing, try both forms of testing, as you will often find yourself mixing and matching from both test approaches depending upon what you want your tests to do. The final type of test is a functional doc test which is really sexy in its simplicity. Create a file in app_tests/ called functional.txt and let it have the following content:: Do a functional doctest test on the app. **************************************** :Test-Layer: functional Test creating a Sample instance into Grok:: >>> from sample.app import Sample >>> root = getRootFolder() >>> root['instance'] = Sample() >>> root.get('instance').__class__.__name__ 'Sample' We now have one python unit test, a doc test, a python functional test and a function doc test. Let's check that they all run:: $ ./bin/test Running tests at level 1 Running unit tests: Running: .. Ran 2 tests with 0 failures and 0 errors in 0.005 seconds. Running sample.FunctionalLayer tests: Set up sample.FunctionalLayer in 1.755 seconds. Running: . Ran 1 tests with 0 failures and 0 errors in 0.005 seconds. Running sample.testing.FunctionalLayer tests: Tear down sample.FunctionalLayer ... not supported Running in a subprocess. Set up sample.testing.FunctionalLayer in 1.771 seconds. Running: . Ran 1 tests with 0 failures and 0 errors in 0.004 seconds. Tear down sample.testing.FunctionalLayer ... not supported Total: 4 tests, 0 failures, 0 errors in 4.896 seconds. .. sidebar:: ZopeXMLConfigurationError in ftesting.zcml If you run into an error stating:: ZopeXMLConfigurationError: File ".../ftesting.zcml", line 11.2-13.8 ConfigurationError: ('Invalid value for', 'component', "ImportError: Couldn't import zope.app.securitypolicy.zopepolicy, No module named securitypolicy.zopepolicy") chances are, that you are using Grok 0.12 and grokproject. Unfortunately the ftesting.zcml generated by grokproject includes a wrong package. To fix this edit the `ftesting.zcml` in your application root and replace:: with:: i.e.: remove the 'app' package and all should work again. Further information ------------------- This how-to is meant to be an introduction to get you started on writing tests, how those tests are discovered, and how you can run them. To help making this How-to as short as possible, I've skipped... * The discussing the `various options for z3c.testsetup `_ which is the base of grok.testing. * The options on the test runner (./bin/test --help). * Doctests embedded as docstrings inside the Grok classes. * Browser testing with requests (`example here `_) Hopefully this how-to can improve over time to make things even simpler and dummy-proof but looking back you'll have to admit that we managed to get a lot done with very little configuration work.