Mocking

Mocking

Problem statement

How to test code that relies on (a lot of) other code?

How to test code that relies on something not available in a regular testing environment?

Mocking

Example

Mocking

Example

Mocking

Mock objects

Magic objects that allow any operations on them and record them for future verification.

from unittest import mock

m = mock.Mock()
r = m.abc(42, cat='meow')
assert isinstance(m.abc, mock.Mock)
assert isinstance(r, mock.Mock)

m.abc.assert_called_once_with(42, cat='meow')
assert r is m.abc.return_value

Mocking

Mock objects

Mocks can simulate functions that return values or raise exceptions.

from unittest import mock

m = mock.Mock(return_value=42)
assert m() == 42

m = mock.Mock(side_effect=[1, 2])
assert m() == 1
assert m() == 2

m = mock.Mock(side_effect=RuntimeError("boom"))
m()  # raises

Mocking

Case study: quadratic equation

import sys

def main():
    try:
        a = int(sys.argv[1])
        b = int(sys.argv[2])
        c = int(sys.argv[3])
    except IndexError:
        sys.exit('3 arguments required')
    except ValueError:
        sys.exit('all arguments must be integers')
    print(roots(a, b, c))

if __name__ == '__main__':
    main()

Mocking

Case study: quadratic equation

Problems:

• provide values for sys.argv
• test how sys.exit is called
• test how print is called

Mocking

Case study: quadratic equation

class MainTest(unittest.TestCase):

    @mock.patch('sys.argv', [None, '1', '-3', '2'])
    @mock.patch('builtins.print')
    def test_correct(self, mock_print):
        roots.main()
        mock_print.assert_called_once_with((1.0, 2.0))

Forms of mock.patch

Pure replacement

import sys

class MainTest(unittest.TestCase):

    @mock.patch('sys.argv', [None, '1', '-3', '2'])
    def test_patch(self):
        # sys.argv is equal to the replacement value here

    @mock.patch.object(sys, 'argv', [None, '1', '-3', '2'])
    def test_patch_object(self):
        # ...

Forms of mock.patch

Creating of Mock object

import builtins

class MainTest(unittest.TestCase):

    @mock.patch('builtins.print')
    def test_patch(self, mock_print):
        # calling print() here calls mock_print instead

    @mock.patch(builtins, 'print')
    def test_patch_object(self, mock_print):
        # ...

Forms of mock.patch

The same at the class level

import builtins

@mock.patch('builtins.print')
class MainTest(unittest.TestCase):

    def test_patch(self, mock_print):
        # calling print() here calls mock_print instead

    def test_2(self, mock_print):
        # ...

This is equivalent to adding the decorator to each test method.

Forms of mock.patch

Inline as a context manager

import time

class MainTest(unittest.TestCase):

    def test_patch(self):
        with mock.patch('time.time') as mock_time:
            # calling time.time() here calls mock_time
        # but not here

    def test_patch_object(self):
        with mock.patch.object(time, 'time') as mock_time:
            # ...

Spec and autospec

Spec and autospec

Problem statement

Mock objects can simulate anything.

How to make them simulate a specific object or function?

Spot a problem:

mock_print.asset_called_once_with("Hello")

Spec and autospec

Mock specs

A Mock object accepts a spec - a simulated object.

class A:
    """The class we are simulating."""
    def x(self, n):
        return n ** 2

m = mock.Mock(spec=A)
print(m.x)
m.y = 42
print(m.y)
print(m.z)

What will this program output?

Spec and autospec

Mock specs

$ python3 mock_spec_demo.py
<Mock name='mock.x' id='140537902817232'>
42
Traceback (most recent call last):
  File "mock_spec_demo.py", line 13, in <module>
    print(m.z)
  File "/usr/lib64/python3.6/unittest/mock.py", line 582, in __getattr__
    raise AttributeError("Mock object has no attribute %r" % name)
AttributeError: Mock object has no attribute 'z'

Accessing z causes an error.

Spec and autospec

Mock specs

spec_set also enforces setting attributes.

class A:
    """The class we are simulating."""
    def x(self, n):
        return n ** 2

m = mock.Mock(spec_set=A)
print(m.x)
m.y = 42
print(m.y)
print(m.z)

Spec and autospec

Mock specs

$ python3 mock_spec_set_demo.py
<Mock name='mock.x' id='140537902817232'>
Traceback (most recent call last):
  File "mock_spec_set_demo.py", line 11, in <module>
    m.y = 42
  File "/usr/lib64/python3.6/unittest/mock.py", line 688, in __setattr__
    raise AttributeError("Mock object has no attribute '%s'" % name)
AttributeError: Mock object has no attribute 'y'

Setting y causes an error.

Spec and autospec

Mock specs

spec and spec_set can accept a list of attributes.

m = mock.Mock(spec=['x', 'y'])
m2 = mock.Mock(spec_set=['x', 'y'])

Or even wrap a real object:

m = mock.Mock(wraps=A())
assert m.x(2) == 4
m.x.assert_called_once_with(2)

Spec and autospec

patch autospec

The autospec argument of the patch function can even check function signatures:

class A:
    def x(self, y):
        return y ** 2

@mock.patch.object(A, 'x', autospec=True)
def test(mock_x):
    a = A()
    print(a.x(42))
    print(a.x(z=42))

test()

Spec and autospec

Mock specs

$ python3 mock_autospec_demo.py
<MagicMock name='x()' id='140700038039648'>
Traceback (most recent call last):
  File "mock_autospec_demo.py", line 16, in <module>
    test()
  File "/usr/lib64/python3.6/unittest/mock.py", line 1179, in patched
    return func(*args, **keywargs)
  File "mock_autospec_demo.py", line 13, in test
    print(a.x(z=42))
  File "<string>", line 2, in x
  File "/usr/lib64/python3.6/unittest/mock.py", line 171, in checksig
    sig.bind(*args, **kwargs)
  File "/usr/lib64/python3.6/inspect.py", line 2969, in bind
    return args[0]._bind(args[1:], kwargs)
  File "/usr/lib64/python3.6/inspect.py", line 2884, in _bind
    raise TypeError(msg) from None
TypeError: missing a required argument: 'y'

Spec and autospec

Case study: quadratic equation

@mock.patch('builtins.print', autospec=True)
class MainTest(unittest.TestCase):

    @mock.patch('sys.argv', [None, '1', '-3', '2'])
    def test_correct(self, mock_print):
        roots.main()
        mock_print.assert_called_once_with((1.0, 2.0))

    @mock.patch('sys.exit', autospec=True)
    @mock.patch('sys.argv', [None, '1', '-3'])
    def test_missing_argument(self, mock_exit, mock_print):
        mock_exit.side_effect = RuntimeError
        self.assertRaises(RuntimeError, roots.main)
        mock_exit.assert_called_once_with(
            '3 arguments required')
        mock_print.assert_not_called()

Coverage

Coverage

Problem statement

I want to know how much of my code is covered by unit tests.

The answer is the coverage utility coverage.readthedocs.io.

Coverage

Collect coverage

$ coverage3 run -m unittest discover my_utils
.....
----------------------------------------------------------------------
Ran 5 tests in 0.012s

OK

Coverage

Report coverage

$ coverage3 report
Name                           Stmts   Miss  Cover
--------------------------------------------------
my_utils/__init__.py               0      0   100%
my_utils/roots.py                 21      3    86%
my_utils/tests/__init__.py         0      0   100%
my_utils/tests/test_roots.py      21      0   100%
--------------------------------------------------
TOTAL                             42      3    93%

Coverage

Rich report

Collect also branch information:

$ coverage3 run --branch -m unittest discover my_utils
.....
----------------------------------------------------------------------
Ran 5 tests in 0.012s

OK

Coverage

Rich report

Show what is not covered:

$ coverage3 report -m
Name                           Stmts   Miss Branch BrPart  Cover   Missing
--------------------------------------------------------------------------
my_utils/__init__.py               0      0      0      0   100%
my_utils/roots.py                 21      3      8      2    83%   24-25, 30, 22->24, 29->30
my_utils/tests/__init__.py         0      0      0      0   100%
my_utils/tests/test_roots.py      21      0      2      0   100%
--------------------------------------------------------------------------
TOTAL                             42      3     10      2    90%

Additional runners

Additional runners

Running with python3 -m unittest is quite convenient.

But there are more feature-rich runners for Python unit tests.

Additional runners

PyTest

pytest.readthedocs.io

• Completely different approach to writing tests
• Hacks into Python assert statement
• Automagical fixtures based on test method arguments
• Rich plugin architecture
• (Mostly?) compatible with regular tests

Additional runners

PyTest

$ pytest-3
============================= test session starts =============================
platform linux -- Python 3.6.6, pytest-3.4.2, py-1.5.4, pluggy-0.6.0
rootdir: /home/dtantsur/Projects/berlin-python-unittest, inifile:
collected 5 items

my_utils/tests/test_roots.py .....                                      [100%]

========================== 5 passed in 0.02 seconds ===========================

Additional runners

stestr

stestr.readthedocs.io

• Select specific tests to run via regular expressions
• Machine-parseable output in subunit format
• Emphasis on parallel execution and streaming results
• Execution time reporting
• Testing framework agnostic

Additional runners

stestr

$ stestr-3 --test-path my_utils/tests/ run
{2} my_utils.tests.test_roots.RootsTest.test_correct [0.000341s] ... ok
{3} my_utils.tests.test_roots.RootsTest.test_negative_a [0.000647s] ... ok
{0} my_utils.tests.test_roots.MainTest.test_correct [0.011246s] ... ok
{0} my_utils.tests.test_roots.MainTest.test_missing_argument [0.004206s] ... ok
{1} my_utils.tests.test_roots.RootsTest.test_negative_discriminant [0.000660s] ... ok

======
Totals
======
Ran: 5 tests in 0.4467 sec.
 - Passed: 5
 - Skipped: 0
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 0.0171 sec.

==============
Worker Balance
==============
 - Worker 0 (2 tests) => 0:00:00.015929
 - Worker 1 (1 tests) => 0:00:00.000660
 - Worker 2 (1 tests) => 0:00:00.000341
 - Worker 3 (1 tests) => 0:00:00.000647

Additional runners

stestr - real project

======
Totals
======
Ran: 5125 tests in 139.0000 sec.
 - Passed: 5113
 - Skipped: 12
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 519.9569 sec.

==============
Worker Balance
==============
 - Worker 0 (1280 tests) => 0:02:12.778625
 - Worker 1 (1280 tests) => 0:02:11.374847
 - Worker 2 (1281 tests) => 0:02:10.885204
 - Worker 3 (1284 tests) => 0:02:07.915481

Additional runners

Tox - runner for runners

tox simplifies routine task on managing virtual environments and running stuff in them.

Run unit tests on Python 2.7:

$ tox -epy27

Run unit tests on the default Python 3:

$ tox -epy3

Build a generic environment and run some commands there:

$ tox -evenv -- python -m some.package

Alternative: pyenv.

Tools: tox

Case study: quadratic equation

[tox]
envlist = pep8,py3

[testenv]
usedevelop = True
deps =
    # e.g. -r requirements.txt
commands =
    python -m unittest discover my_utils
setenv =
    PYTHONDONTWRITEBYTECODE=1

[testenv:pep8]
basepython = python3
deps =
    flake8
commands =
    flake8 my_utils

[testenv:venv]
commands = {posargs}

Tools: tox

Case study: quadratic equation

$ tox
pep8 develop-inst-noop: /home/dtantsur/Projects/berlin-python-unittest
pep8 installed: flake8==3.5.0,mccabe==0.6.1,-e git+git@github.com:dtantsur/berlin-python-unittest.git@1b9ac65cee9ba031eb5d3bea979ab2be60ffb844#egg=my_utils,pycodestyle==2.3.1,pyflakes==1.6.0
pep8 runtests: PYTHONHASHSEED='2318298329'
pep8 runtests: commands[0] | flake8 my_utils
py3 create: /home/dtantsur/Projects/berlin-python-unittest/.tox/py3
py3 develop-inst: /home/dtantsur/Projects/berlin-python-unittest
py3 installed: -e git+git@github.com:dtantsur/berlin-python-unittest.git@1b9ac65cee9ba031eb5d3bea979ab2be60ffb844#egg=my_utils
py3 runtests: PYTHONHASHSEED='2318298329'
py3 runtests: commands[0] | python -m unittest discover my_utils
.....
----------------------------------------------------------------------
Ran 5 tests in 0.007s

OK
_____________________________________________________________________________________________________ summary _____________________________________________________________________________________________________
  pep8: commands succeeded
  py3: commands succeeded
  congratulations :)

Handly libraries

requests-mock

Problem statement

How to test code that does complex network interations?

requests-mock: requests-mock.readthedocs.io

Enables pre-defined answers to specified requests.

requests-mock

Example

>>> @requests_mock.Mocker()
... def test_function(m):
...     m.get('http://test.com', text='resp')
...     return requests.get('http://test.com').text
...
>>> test_function()
'resp'

fixtures

The fixtures library provides a format for defining and using test fixtures.

Fixtures are self-contained helpers that ensure some state on test start and revert to the initial state after a test finishes.

fixtures

Writing fixtures

import fixtures
import os

class SecretFileFixture(fixtures.Fixture):
    def __init__(self, fname, content='Hello'):
        self.fname = fname
        self.content = content

    def _setUp(self):
        with open(self.fname, 'w') as f:
            f.write(self.content)
        self.addCleanup(lambda: os.unlink(self.fname))

fixtures

Using fixtures

class MyTest(fixtures.TestWithFixtures):
    def setUp(self):
        self.useFixture(SecretFileFixture('/tmp/test'))

    def test_with_file(self):
        assert os.path.exists('/tmp/test')
        # /tmp/test will be present here and deleted after the test ends

The testtools library provides support for useFixture out of box.

fixtures

Existing fixtures

• MockPatch and MockPatchObject
• EnvironmentVariable
• LogHandler and FakeLogger
• FakePopen
• TempDir and TempHomeDir
• PythonPackage and PythonPathEntry
• ... and many more.