Advanced pytest techniques I learned while contributing to pandas

In the past months, I contributed quite a few PRs to pandas, the open-source data analysis and manipulation library at the core of Python’s excellent data processing ecosystem.

As pandas is over 10 years old and consists of over half a million lines of code, no one can know all the consequences of a simple code change anymore. Consequently, we need to rely on our test suite to avoid the risk of constantly breaking the code of millions of our users and extensive coverage of corner and edge cases is crucial.

In this article, I want to share a few advanced pytest features I learned during that time. I assume basic knowledge of pytest including fixtures, regular parametrization and testing exceptions. If you aren’t familiar with these, I recommend checking out realpython.com’s great pytest tutorial first.

Parametrize your fixtures

I’m a big fan of parametrization in tests. It’s an elegant and concise way to test multiple configurations in order to unravel bugs and boost confidence in your application. Something I didn’t know is that you can even parametrize fixtures directly, not just test cases.

For instance, we extensively use our beloved index fixture which gives us a variety of different Index instances:

indices_dict = {
    "unicode": tm.makeUnicodeIndex(100),
    "string": tm.makeStringIndex(100),
    "datetime": tm.makeDateIndex(100),
    "datetime-tz": tm.makeDateIndex(100, tz="US/Pacific"),
    "period": tm.makePeriodIndex(100),
    "timedelta": tm.makeTimedeltaIndex(100),
    "int": tm.makeIntIndex(100),
    "uint": tm.makeUIntIndex(100),
    "range": tm.makeRangeIndex(100),
    "float": tm.makeFloatIndex(100),
    "bool": tm.makeBoolIndex(10),
    "categorical": tm.makeCategoricalIndex(100),
    "interval": tm.makeIntervalIndex(100),
    "empty": Index([]),
    "tuples": MultiIndex.from_tuples(zip(["foo", "bar", "baz"], [1, 2, 3])),
    "mi-with-dt64tz-level": _create_mi_with_dt64tz_level(),
    "multi": _create_multiindex(),
    "repeats": Index([0, 0, 1, 1, 2, 2]),
}


@pytest.fixture(params=indices_dict.keys())
def index(request):
    """
    Fixture for many "simple" kinds of indices.
    These indices are unlikely to cover corner cases, e.g.
        - no names
        - no NaTs/NaNs
        - no values near implementation bounds
        - ...
    """
    # copy to avoid mutation, e.g. setting .name
    return indices_dict[request.param].copy()

Let’s put this into action with a little example: Running the following test

def test_indices(index):
    assert isinstance(index, pd.Index)

yields

test_example.py::test_indices[unicode] PASSED
test_example.py::test_indices[string] PASSED
test_example.py::test_indices[datetime] PASSED
test_example.py::test_indices[datetime-tz] PASSED
test_example.py::test_indices[period] PASSED
test_example.py::test_indices[timedelta] PASSED
test_example.py::test_indices[int] PASSED
test_example.py::test_indices[uint] PASSED
test_example.py::test_indices[range] PASSED
test_example.py::test_indices[float] PASSED
test_example.py::test_indices[bool] PASSED
test_example.py::test_indices[categorical] PASSED
test_example.py::test_indices[interval] PASSED
test_example.py::test_indices[empty] PASSED
test_example.py::test_indices[tuples] PASSED
test_example.py::test_indices[mi-with-dt64tz-level] PASSED
test_example.py::test_indices[multi] PASSED
test_example.py::test_indices[repeats] PASSED

Isn’t that convenient? We can write a test as if we’re using a regular fixture, and pytest automatically executes it once per fixture value!

At pandas, this is great because we offer a large variety of different indices that sometimes (for historic or performance reasons) don’t share a common implementation. Parametrized fixtures free us from thinking too much about this. If a test case should work for all indices, we can just use the index fixture and automatically gain coverage for all index types. And it works: By applying more parametrized fixtures in our test suite, I actually found bugs we didn’t have coverage for otherwise.

I don’t think parametrized fixtures are essential to every codebase, but they come in handy in these two situations:

1. You want to ensure common functionality which can’t be guaranteed by inheritance alone (like in the example above).
2. You have common parametrizations which are used on multiple tests, e.g. identical parameters accepted by multiple functions (see here).

Note: Even though parametrized fixtures have multiple values, you should give them singular names.

Set ids in parametrizations

Another (related) feature I didn’t use before is specifying ids in the parametrizations. Most of the time that’s not necessary as pytest will find good defaults. Sometimes, however, these defaults are hard to read or aren’t concise enough. Take the following example:

@pytest.mark.parametrize("obj", [pd.Index([]), pd.Series([])])
def test_with_ids(obj):
    assert obj.empty

Running this test results in

test_example.py::test_with_ids[obj0] PASSED                                                                                                      
test_example.py::test_with_ids[obj1] PASSED

In case one of these runs fails in the future, you’d always need to derive the failed configuration from counting the parametrization values. I.e. if obj1 fails, you’ll have to search for the second parametrization value. While this works, it is pretty annoying. A better alternative is to overwrite ids manually

@pytest.mark.parametrize(
    "obj",
    [pd.Index([]), pd.Series([])],
    ids=["Index", "Series"]
)
def test_with_ids(obj):
    assert obj.empty

or pass a callable to derive the ids implicitly

@pytest.mark.parametrize(
    "obj",
    [pd.Index([]), pd.Series([])],
    ids=lambda x: type(x).__name__
)
def test_with_ids(obj):
    assert obj.empty

Both approaches yield

test_example.py::test_with_ids[Index] PASSED                                                                                                      
test_example.py::test_with_ids[Series] PASSED

Note: The examples above focused on parametrizing tests, but it works the same way when parametrizing fixtures.

Call skip or xfail inside the test

Usually when you want to skip or xfaila test, you make use of pytest‘s elegant decorator syntax. Sometimes, however, the condition you need to check for is only available at run time. In that case, you can call pytest.skip or pytest.xfail inside the test logic as well:

def test_indexing(index):
    if len(index) == 0:
        pytest.skip("Test case is not applicable for empty data.")
    index[0]

which yields

test_example.py::test_indexing[unicode] PASSED
test_example.py::test_indexing[string] PASSED
test_example.py::test_indexing[datetime] PASSED
test_example.py::test_indexing[datetime-tz] PASSED
test_example.py::test_indexing[period] PASSED
test_example.py::test_indexing[timedelta] PASSED
test_example.py::test_indexing[int] PASSED
test_example.py::test_indexing[uint] PASSED
test_example.py::test_indexing[range] PASSED
test_example.py::test_indexing[float] PASSED
test_example.py::test_indexing[bool] PASSED
test_example.py::test_indexing[categorical] PASSED
test_example.py::test_indexing[interval] PASSED
test_example.py::test_indexing[empty] SKIPPED
test_example.py::test_indexing[tuples] PASSED
test_example.py::test_indexing[mi-with-dt64tz-level] PASSED
test_example.py::test_indexing[multi] PASSED
test_example.py::test_indexing[repeats] PASSED

Note: While it’s great that this is possible, it has the downside of setting up a test run that is effectively ignored in the results. Therefore, the decorator syntax should always be prefered (if possible) as it can already be evaluated at test collection.

Indirect parametrization

Sometimes, you only want to use a subset of the parametrized values of your fixture. E.g. what do you do, in case you want to use the index fixture, but you only want to use the MultiIndex values of it? Then you have these options:

1. Use a new parametrization or fixtures (→ redundant)
2. Use explicit pytest.skip’s (→ creates unused test runs)
3. Use indirect parametrization

As we’ve already covered point 2, let’s take a look at option 3:

@pytest.mark.parametrize(
    "index",
    ["mi-with-dt64tz-level", "multi"],
    indirect=True
)
def test_dummy(index):
    assert isinstance(index, pd.MultiIndex)

The syntax can be a bit overwhelming at first, but actually makes a lot of sense (as usual): First, you add your parametrized fixture to the signature. Then, you add a parametrization to the test function where you specify

1. The fixture you want to indirectly parametrize
2. The values you want to select
3. Set the indirect flag to True

This results in the following two test runs:

test_example.py::test_multiindex[mi-with-dt64tz-level] PASSED
test_example.py::test_multiindex[multi] PASSED

The main downside I see with this approach is that it requires you to specify the fixture values manually. So in case, we add a new MultiIndex value to our index fixture, the indirect parametrization can’t pick it up automatically. Therefore, you’ll have to balance which of the three approaches you’re going to use.

In case you want to dive deeper into this topic, I recommend to read this great post at pytest-tricks which gives a bit more context.

pytest.raises vs nullcontext

The final piece in this article deals with redundancy you might face when dealing with parametrization. More specifically, when you’re testing exceptions that should only occur with some parametrization values, but not with others. Take the following example:

@pytest.mark.parametrize(
    "obj, raises",
    [
        ([], False),
        (pd.Index([]), True),
    ],
)
def test_exception(obj, raises):
    if raises:
        with pytest.raises(ValueError):
            # you should also test the error message, using
            # the 'matches' parameter in .raises()
            # I don't have enough space here though...
            bool(obj)
    else:
        bool(obj)

The redundancy of calling bool(obj) twice can be annoying when the test becomes larger than this minimal example.

After a bit of digging, I bumped into this thread in the pytest GitHub page, where someone eventually suggested to make use of contextlib.nullcontext:

from contextlib import nullcontext


@pytest.mark.parametrize(
    "obj, expected_raises",
    [
        ([], nullcontext()),
        (pd.Index([]), pytest.raises(ValueError)),
    ],
)
def test_exception2(obj, expected_raises):
    with expected_raises:
        bool(obj)

Note: contextlib.nullcontext is only available since Python3.7. If you’re on an earlier version, please check out this comment in the mentioned thread.

Please don’t overuse this feature, though. While it can be an elegant way for simple use cases, it can also encourage you to use parametrization for disparate behaviour where separate tests would be more appropriate.

Conclusion

While there are many more awesome pytest features I haven’t talked about, I hope you enjoyed exploring these features as much as I did.

Contributing to open-source projects is typically a great opportunity to discover and learn new concepts or technology while giving back to the community at the same time. So if you haven’t, go ahead and check for unresolved issues on open source projects you find interesting or already use. It’s going to be a win-win for you and everybody else ;)

Coding Interview Questions | Skilled.dev