TLDR: Upstream function is called with wrong argument order. How do I ensure, this is caught by tests?

Here is a minimal example of my setup:

# functions.py

def inner(age, name):
    if age > 18:
        return f'{name} is an adult.'
    else: 
        return f'{name} is a child.'

def outer(name, age):
    info = inner(name, age)
    return f'This is {name}. {info}'


# tests.py

from functions import inner, outer
from unittest.mock import Mock, patch

def test_inner():
    name, age = "John", 43
    info = inner(age, name)
    expected = "John is an adult."
    assert info == expected

def test_outer():
    name, age = "John", 43

    mock_get_info = Mock()
    mock_get_info.return_value = "Info"
    patch_get_info = patch("functions.inner", new=mock_get_info)

    with patch_get_info:
        info = outer(name, age)
        expected = 'This is John. Info'
        assert info == expected
        mock_get_info.assert_called_once_with(name, age)

Functionality:

• Two functions and their corresponding tests.
• The inner function produces a string which is checked for correctness by the test_inner function.
• The outer function calls the inner function and concatenates it to it's own string. The correct concatenation and inner function call are checked by the test_outer function.
• Additionally, apart from this minimal example, the inner function may produce a very large string which I don't want to check explicitly in the test_outer which is why the inner return value is mocked.

You may have noticed, that the outer function actually passes the arguments to the inner function the wrong way around. This is because I could have decided to change the order of arguments of the inner function and changed the test_inner accordingly, but forgot that the outer function calls the inner function. This is not caught by the test_outer, because it is internally consistent. We only find out in production, that the inner function throws an error.

How do I ensure, that all tests of downstream functions catch modified function definitions?

CodePudding user response：

I think you're on the right track. I believe your test didn't catch it because you're asserting the wrong order.

mock_get_info.assert_called_once_with(name, age) should be: mock_get_info.assert_called_once_with(age, name) to match inner(age, name) signature.

Nevertheless, I think a more robust approach would be to use keyword arguments, then asserting that the call args are the expected dictionary.

For example:

# functions.py

def outer(name, age):
    info = inner(age=age, name=name)
    return f'This is {name}. {info}'

# tests
def test_outer():
    name, age = "John", 43

    mock_get_info = Mock()
    mock_get_info.return_value = "Info"
    patch_get_info = patch("functions.inner", new=mock_get_info)

    with patch_get_info:
        info = outer(name, age)
        expected = 'This is John. Info'
        assert info == expected
        _, actual_kwargs = mock_get_info.call_args
        assert actual_kwargs == {'name': name, 'age': age}

If you want to be more strict, you can enforce inner to only accept keyword arguments by using * [1]. But that also means you'll have to supply default values, which depending on your use-case might not make sense.

Example:

def inner(*, age=0, name='default'):
    if age > 18:
        return f'{name} is an adult.'
    else:
        return f'{name} is a child.'

If anyone calls it without using keyword arguments, Python will raise an exception in runtime.

Traceback (most recent call last):
  File "/home/user/projects/so-test/functions.py", line 14, in <module>
    outer('user', 32)
  File "/home/user/projects/so-test/functions.py", line 9, in outer
    info = inner(age, name)
TypeError: inner() takes 0 positional arguments but 2 were given

IMO, it's much harder to make a mistake like this inner(age=name, name=age) than this inner(age, name).

[1] https://www.python.org/dev/peps/pep-3102/

CodePudding user response：

You may need some integration testing, or functional/end to end testing to catch this sort of errors. Testing single unit mocking everything outside is good, because your unit test is independent from other unit's errors. However as you found, you can have a problem on function interfaces, because you don't test if they are used correctly (i.e. you can say that you don't test units integration). So this is place where you could introduce some sort of integration testing (or end to end, depends on testing strategy, you can read more about it searching test pyramid).

Example test for your case:

import pytest

@pytest.mark.functional
@pytest.mark.parametrize("name,age,expected", [("John", 18, "This is John. John is an adult."), ("Dave", 17, "This is Dave. Dave is a child")])
def test_outer_functionality(name, age, expected):
    assert outer(name, age) == expected

Then you can also think if you would like to run such test always, or e.g. nightly, and during build process run only unit tests (pytest -v -m "not functional").

CodePudding user response：

Patching is about decoupling the test from that objects actual implementation. i.e. replacing inner with a mock ENSURES that any change to inner won't impact that test. This means the mocks must be manually updated whenever inner is changed. Given the initial issue involved not identifying outer as needing to be updated, it is not reasonable to require that the mock inside a unittest for that unidentified method will be updated.

Quick solution: Type hinting

• The test conditions are attached to inners args and follow it wherever it is called. It's effectively "tested" on every invocation. Single source of truth.
• With an IDE like pycharm, developer gets feed back on the bug as they are making it. Even with the strictest unittest policy, this is quicker. Eliminate the chance of the error being made multiple times before check in (assuming that they run unittests then). Increase the chances of the lesson sticking.
• Patching inner in downstream tests for other reasons (i.e avoiding an expensive operation) won't impact this solution.

Strict Solution: Asserts in inner definition and patching lower down

Place asserts at the top of the definition for inner, i.e confirm age arg is an int between 0 and 120. Then the downstream test will patch whatever is in inner that your avoiding (Possibly a network read) to return a value based on the arg. The test to ensure ordering is inside the inner function. Your patching a function that is unlikely to change, or you just don't have responsibility for ensuring doesn't change. Would need more detail in the example to flesh this out more.