Mocking methods called within multiprocessing doesn't work on Mac-CodePudding

I encountered a very weird bug. In my project, I have unit tests where I mock some methods (download methods for instance) that are called within a multiprocessing operation.
Those unit tests work fine on my CI, but when I try to run them locally on Mac OSX, the mock isn't taken into account.

I implemented the following minimal reproducible example:

# test_mock_multiprocessing.py
import multiprocessing
from typing import List
from unittest import mock

import pytest


def _f(x: float) -> float:
    return x**2


def f(x: float) -> float:
    return _f(x)


def map_f(xs: List[float]) -> List[float]:
    return list(map(f, xs))


def multimap_f(xs: List[float]) -> List[float]:
    with multiprocessing.Pool(4) as pool:
        ys = list(pool.map(f, xs))
    return ys


@pytest.mark.parametrize("method", [map_f, multimap_f])
def test_original(method):
    xs = [-2, 3, 1]
    expected_ys = [4, 9, 1]
    ys = method(xs)
    assert ys == expected_ys


def mocked_f(x: float) -> float:
    return -x


@mock.patch("test_mock_multiprocessing._f", side_effect=mocked_f)
@pytest.mark.parametrize("method", [map_f, multimap_f])
def test_mocked(mocker, method):
    xs = [-2, 3, 1]
    expected_ys = [2, -3, -1]
    ys = method(xs)
    assert ys == expected_ys

That I launch with

pytest test_mock_multiprocessing.py

If I launch those tests within the Docker image python3.8-buster (with pytest installed), they are all successful. But if I launch the same tests directly on my host machine (Mac OSX), within a virtualenv with only pytest installed, I have the following output:

$ pytest test_mock_multiprocessing.py
=========================================================================================== test session starts ============================================================================================
platform darwin -- Python 3.8.12, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: ***
plugins: typeguard-2.13.0, cov-3.0.0
collected 4 items

test_mock_multiprocessing.py ...F                                                                                                                                                                    [100%]

================================================================================================= FAILURES =================================================================================================
_________________________________________________________________________________________ test_mocked[multimap_f] __________________________________________________________________________________________

mocker = <MagicMock name='_f' id='4377597216'>, method = <function multimap_f at 0x104e431f0>

    @mock.patch("test_mock_multiprocessing._f", side_effect=mocked_f)
    @pytest.mark.parametrize("method", [map_f, multimap_f])
    def test_mocked(mocker, method):
        xs = [-2, 3, 1]
        expected_ys = [2, -3, -1]
        ys = method(xs)
>       assert ys == expected_ys
E       assert [4, 9, 1] == [2, -3, -1]
E         At index 0 diff: 4 != 2
E         Use -v to get the full diff

test_mock_multiprocessing.py:44: AssertionError
========================================================================================= short test summary info ==========================================================================================
FAILED test_mock_multiprocessing.py::test_mocked[multimap_f] - assert [4, 9, 1] == [2, -3, -1]
======================================================================================= 1 failed, 3 passed in 1.06s ========================================================================================

CodePudding user response：

The difference between the behavior under Linux and MacOs has probably to do with the multiprocessing start method. On Linux, the default start method is fork, while on MacOS and Windows it is spawn.

If using fork, your processes are forked in the current state, including the mocking, while with using spawn, a new Python interpreter is launched, where the mock will not work. Under MacOs (but not under Windows) you can change the start method to fork in your test:


@mock.patch("test_mock_multiprocessing._f", side_effect=mocked_f)
@pytest.mark.parametrize("method", [map_f, multimap_f])
def test_mocked(mocker, method):
    start_method = multiprocessing.get_start_method()
    try:
        multiprocessing.set_start_method('fork', force=True)
        ... do the test
    finally: 
        multiprocessing.set_start_method(start_method, force=True)

For convenience, you could also wrap that into a context manager:

@contextmanager
def set_start_method(method):
    start_method = multiprocessing.get_start_method()
    try:
        multiprocessing.set_start_method(method, force=True)
        yield
    finally:
        multiprocessing.set_start_method(start_method, force=True)

...
def test_something():
    with set_start_method('fork'):
        ... do the test