Automated testing

How and why?

Automated testing

What?

Using a program to verify correctness of a program (library, module, any piece of software).

Automated testing

Why?

• Decisiveness
• Repeatability
• Parallelism
• Coverage

Automated testing

Types

• Black box
• White box

Automated testing

Types

• Unit
• Integration
• Functional

Automated testing

Unit testing

Usually white box

Automated testing

Integration testing

Usually black box

Automated testing

Functional testing

Black box

Automated testing

Why unit test?

• Sanity-check during development
• Help newcomers keep things working
• Document design decision
• Sign of the maturity of the project

Automated testing

Why unit test?

Unit tests do not:

• Guarantee that your code works
• Replace integration testing
• Replace developer's guide
• Have to cover literally everything

Unit test framework

Unit test framework

• Included in the standard library
• Heavily extended in Python 3
• Plenty 3rd party addons

Unit test framework

Case study: quadratic equation

Package layout:

$ find my_utils
my_utils
my_utils/roots.py
my_utils/__init__.py
my_utils/tests
my_utils/tests/__init__.py
my_utils/tests/test_roots.py

Unit test framework

Case study: quadratic equation

import math

def roots(a, b, c):
    if not a:
        raise ValueError("a cannot be zero")

    discriminant = b ** 2 - 4 * a * c
    if discriminant < 0:
        raise ValueError("discriminant below zero")

    return ((- b - math.sqrt(discriminant)) / 2 / a,
            (- b + math.sqrt(discriminant)) / 2 / a)

Writing unit tests

• Split the testing into independent checks
• Group individual checks into cases
• Don't forget negative tests

Unit test framework

Case study: quadratic equation

import unittest
from my_utils import roots

class RootsTest(unittest.TestCase):
    def test_correct(self):
        self.assertEqual(roots.roots(1, -3, 2),
                         (1.0, 2.0))

    def test_negative_a(self):
        self.assertRaises(ValueError,
                          roots.roots, 0, -3, 2)

    def test_negative_discriminant(self):
        self.assertRaises(ValueError,
                          roots.roots, 1, 1, 1)

Unit test framework

Case study: quadratic equation

Running only one test file:

$ python3 -m unittest my_utils.tests.test_roots
...
----------------------------------------------------------------------
Ran 3 tests in 0.000s

OK

Unit test framework

Case study: quadratic equation

Detecting and running all tests:

$ python3 -m unittest discover my_utils
...
----------------------------------------------------------------------
Ran 3 tests in 0.000s

OK

Unit test framework

Case study: quadratic equation

And this is how it fails:

$ python3 -m unittest discover my_utils
F..
======================================================================
FAIL: test_correct (tests.test_roots.RootsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dtantsur/Projects/berlin-python-unittest/my_utils/tests/test_roots.py", line 10, in test_correct
    self.assertEqual(roots(1, -3, 2), (1.0, 2.0))
AssertionError: Tuples differ: (1.0, 1.0) != (1.0, 2.0)

First differing element 1:
1.0
2.0

- (1.0, 1.0)
?       ^

+ (1.0, 2.0)
?       ^


----------------------------------------------------------------------
Ran 3 tests in 0.002s

FAILED (failures=1)
    

Unit test framework

Quadratic equation: imports

import unittest
from my_utils.roots import roots

Importing the standard unittest package and our code that we plan on testing.

Unit test framework

Quadratic equation: test case

class RootsTest(unittest.TestCase):
    def test_correct(self):
        # ...

    def test_negative_a(self):
        # ...

    # ...

• A test case is a logical group of tests
• Each test is a method starting with test_
• Each tests one aspect of the tested entity

Unit test framework

Quadratic equation: equality

def test_correct(self):
    self.assertEqual(roots.roots(1, -3, 2),
                     (1.0, 2.0))

1. Calculate the value using the function under test
2. Compare it with the known correct result

Using convenience methods self.assert<***>

Unit test framework

Quadratic equation: errors

def test_negative_a(self):
    self.assertRaises(ValueError,
                      roots.roots, 0, -3, 2)

def test_negative_discriminant(self):
    self.assertRaises(ValueError,
                      roots.roots, 1, 1, 1)

• Use assertRaises to check that a callable raises an exception on invalid input
• Note that we do not call it ourselves!

Unit test framework

Quadratic equation: errors

def test_negative_a(self):
    self.assertRaisesRegex(ValueError, 'a cannot be zero',
                           roots.roots, 0, -3, 2)

def test_negative_discriminant(self):
    self.assertRaisesRegex(ValueError, 'discriminant below zero',
                           roots.roots, 1, 1, 1)

• Use assertRaisesRegex to validate the error message
• Useful to distinguish between different cases of the same error

Unit test framework

Available checks

assertEqual, assertNotEqual, assertIs, assertIsNot, assertTrue, assertFalse, assertIsInstance, assertNotIsInstance, assertIn, assertNotIn, assertGreater, assertGreaterEqual, assertLess, assertLessEqual, assertRegex, assertNotRegex, assertRaises, assertRaisesRegex, ...

And even more in 3rd party libraries

Unit test framework

Preparing for tests

class RootsTest(unittest.TestCase):

    def setUp(self):
        # Gets run before every test

    def tearDown(self):
        # Gets run after every test

Also use addCleanup to clean up after tests.

Mocking

Handling integration points

Mocking

Problem statement

How to test code that relies on (a lot of) other code?

How to test code that relies on something not available in a regular testing environment?

Mocking

The mock library

Python has the mock library:

• unittest.mock in Python 3 standard library
• just mock on PyPI for Python 2 and 3

Mocking

Case study: quadratic equation

import sys

def main():
    try:
        a = int(sys.argv[1])
        b = int(sys.argv[2])
        c = int(sys.argv[3])
    except IndexError:
        sys.exit('3 arguments required')
    except ValueError:
        sys.exit('all arguments must be integers')
    print(roots(a, b, c))

if __name__ == '__main__':
    main()

Mocking

Case study: quadratic equation

$ python3 -m my_utils.roots
3 arguments required

$ python3 -m my_utils.roots a b c
all arguments must be integers

$ python3 -m my_utils.roots 1 4 4
(-2.0, -2.0)

Mocking

Case study: quadratic equation

Problems:

• provide values for sys.argv
• test how sys.exit is called
• test how print is called

Mocking

patch function

The unittest.mock.patch function allows to replace an object temporary while a test is running and restore it back.

Can be used in many forms, we will only consider some of them.

Mocking

Case study: quadratic equation

from unittest import mock

class MainTest(unittest.TestCase):

    @mock.patch('sys.argv', [None, '1', '-3', '2'])
    def test_correct(self):
        roots.main()

Using patch as decorator to automate patching object and restoring it after the test.

Mocking

Case study: quadratic equation

Rough equivalent:

class MainTest(unittest.TestCase):

    def test_correct(self):
        import sys
        old_argv = sys.argv
        sys.argv = [None, '1', '-3', '2']
        try:
            roots.main()  # our actual test
        finally:
            sys.argv = old_argv

Mocking

patch function

This test does not verify that main function does anything.

To verify that printing is done correctly, we need to replace the print function with something that will track calls.

Mocking

Mock objects

Magic objects that allow any operations on them and record them for future verification.

m = mock.Mock()
r = m.abc(42, cat='meow')
assert isinstance(m.abc, mock.Mock)
assert isinstance(r, mock.Mock)

m.abc.assert_called_once_with(42, cat='meow')
assert r is m.abc.return_value

Mocking

Case study: quadratic equation

class MainTest(unittest.TestCase):

    @mock.patch('sys.argv', [None, '1', '-3', '2'])
    @mock.patch('builtins.print')
    def test_correct(self, mock_print):
        roots.main()
        mock_print.assert_called_once_with((1.0, 2.0))

Mocking

Case study: quadratic equation

If we make a mistake, the test will tell us:

$ python3 -m unittest discover my_utils
F...
======================================================================
FAIL: test_correct (tests.test_roots.MainTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.6/unittest/mock.py", line 1179, in patched
    return func(*args, **keywargs)
  File "/home/dtantsur/Projects/berlin-python-unittest/my_utils/tests/test_roots.py", line 27, in test_correct
    mock_print.assert_called_once_with((2.0, 2.0))
  File "/usr/lib64/python3.6/unittest/mock.py", line 825, in assert_called_once_with
    return self.assert_called_with(*args, **kwargs)
  File "/usr/lib64/python3.6/unittest/mock.py", line 814, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: Expected call: print((2.0, 2.0))
Actual call: print((1.0, 2.0))

----------------------------------------------------------------------
Ran 4 tests in 0.002s

FAILED (failures=1)

Mocking

Case study: quadratic equation

Handling sys.exit:

• We need to replace it with Mock
• But we also make sure to stop function from executing after calling it

We can make it raise an exception!

Mocking

Case study: quadratic equation

@mock.patch('builtins.print')
class MainTest(unittest.TestCase):

    @mock.patch('sys.argv', [None, '1', '-3', '2'])
    def test_correct(self, mock_print):
        roots.main()
        mock_print.assert_called_once_with((1.0, 2.0))

    @mock.patch('sys.exit')
    @mock.patch('sys.argv', [None, '1', '-3'])
    def test_missing_argument(self, mock_exit, mock_print):
        mock_exit.side_effect = RuntimeError
        self.assertRaises(RuntimeError, roots.main)
        mock_exit.assert_called_once_with(
            '3 arguments required')
        mock_print.assert_not_called()

Mocking

Summary

mock.patch:

• Can be used on methods and classes
• Can replace an object with a given object or with a new mock
• Passes the newly created mock to test methods

Mocking

Summary

Mock:

• Any attribute is also a Mock
• Can be called, result is a Mock
• Can raise exceptions
• Can return specified values (via setting return_value)