Testing

With the advent of LLMs, the temptation to churn out a flood of unit tests for a false veneer of productivity and protection is stronger than ever. My colleague Matthias Doepmann recently fired a shot at AI-generated tests that don’t validate the behavior of the subject under test (SUT) but instead create needless ceremony around internal implementations. At best, these tests give a shallow illusion of confidence in the system’s correctness while breaking at the smallest change. At worst, they remain green even when the SUT’s behavior changes. ...

Unlike pytest or JUnit, Go’s standard testing framework doesn’t give you as many knobs for tuning the lifecycle of your tests. By lifecycle I mean the usual setup and teardown hooks or fixtures that are common in other languages. I think this is a good thing because you don’t need to pick up many different framework-specific workflows for something so fundamental. Go gives you enough hooks to handle this with less ceremony. But it can still be tricky to figure out the right conventions for setup and teardown that don’t look odd to other Gophers, especially if you haven’t written Go for a while. This text explores some common ways to do lifecycle management in your Go tests. ...

As your test suite grows, you need ways to toggle certain kinds of tests on or off. Maybe you want to enable snapshot tests, skip long-running integration tests, or switch between real services and mocks. In every case, you’re really saying, “Run this test only if X is true.” So where does X come from? I like to rely on Go’s standard tooling so that integration and snapshot tests can live right beside ordinary unit tests. Because I usually run these heavier tests in testcontainers, I don’t always want them running while I’m iterating on a feature or chasing a bug. So I need to enable them in an optional manner. ...

Ideally, every function that writes to the stdout probably should ask for a io.Writer and write to it instead. However, it’s common to encounter functions like this: func frobnicate() { fmt.Println("do something") } This would be easier to test if frobnicate would ask for a writer to write to. For instance: func frobnicate(w io.Writer) { fmt.Fprintln(w, "do something") } You could pass os.Stdout to frobnicate explicitly to write to the console: ...

While watching Mitchell Hashimoto’s excellent talk on Go testing, I came across this neat technique for deferring teardown to the caller. Let’s say you have a helper function in a test that needs to perform some cleanup afterward. You can’t run the teardown inside the helper itself because the test still needs the setup. For example, in the following case, the helper runs its teardown immediately: func TestFoo(t *testing.T) { helper(t) // Test logic here: resources may already be cleaned up! } func helper(t *testing.T) { t.Helper() // Setup code here. // Teardown code here. defer func() { // Clean up something. }() } When helper is called, it defers its teardown—which executes at the end of the helper function, not the test. But the test logic still depends on whatever the helper set up. So this approach doesn’t work. ...

Here’s a Python snippet that makes an HTTP POST request: # script.py import httpx from typing import Any async def make_request(url: str) -> dict[str, Any]: headers = {"Content-Type": "application/json"} async with httpx.AsyncClient(headers=headers) as client: response = await client.post( url, json={"key_1": "value_1", "key_2": "value_2"}, ) return response.json() The function make_request makes an async HTTP request with the httpx1 library. Running this with asyncio.run(make_request("https://httpbin.org/post")) gives us the following output: { "args": {}, "data": "{\"key_1\": \"value_1\", \"key_2\": \"value_2\"}", "files": {}, "form": {}, "headers": { "Accept": "*/*", "Accept-Encoding": "gzip, deflate", "Content-Length": "40", "Content-Type": "application/json", "Host": "httpbin.org", "User-Agent": "python-httpx/0.27.2", "X-Amzn-Trace-Id": "Root=1-66d5f7b0-2ed0ddc57241f0960f28bc91" }, "json": { "key_1": "value_1", "key_2": "value_2" }, "origin": "95.90.238.240", "url": "https://httpbin.org/post" } We’re only interested in the json field and want to assert in our test that making the HTTP call returns the expected values. ...

I love @pytest.mark.parametrize1—so much so that I sometimes shoehorn my tests to fit into it. But the default style of writing tests with parametrize can quickly turn into an unreadable mess as the test complexity grows. For example: import pytest from math import atan2 def polarify(x: float, y: float) -> tuple[float, float]: r = (x**2 + y**2) ** 0.5 theta = atan2(y, x) return r, theta @pytest.mark.parametrize( "x, y, expected", [ (0, 0, (0, 0)), (1, 0, (1, 0)), (0, 1, (1, 1.5707963267948966)), (1, 1, (2**0.5, 0.7853981633974483)), (-1, -1, (2**0.5, -2.356194490192345)), ], ) def test_polarify(x: float, y: float, expected: tuple[float, float]) -> None: # pytest.approx helps us ignore floating point discrepancies assert polarify(x, y) == pytest.approx(expected) The polarify function converts Cartesian coordinates to polar coordinates. We’re using @pytest.mark.parametrize in its standard form to test different conditions. ...

While reading the second version of Brian Okken’s pytest book1, I came across this neat trick to compose multiple levels of fixtures. Suppose, you want to create a fixture that returns some canned data from a database. Now, let’s say that invoking the fixture multiple times is expensive, and to avoid that you want to run it only once per test session. However, you still want to clear all the database states after each test function runs. Otherwise, a test might inadvertently get coupled with another test that runs before it via the fixture’s shared state. Let’s demonstrate this: ...

I was reading Ned Bachelder’s blog “Why your mock doesn’t work”1 and it triggered an epiphany in me about a testing pattern that I’ve been using for a while without being aware that there might be an aphorism on the practice. Patch where the object is used; not where it’s defined. To understand it, consider the example below. Here, you have a module containing a function that fetches data from some fictitious database. ...

I just found out that you can use Python’s unittest.mock.ANY to make assertions about certain arguments in a mock call, without caring about the other arguments. This can be handy if you want to test how a callable is called but only want to make assertions about some arguments. Consider the following example: # test_src.py import random import time def fetch() -> list[float]: # Simulate fetching data from a database. time.sleep(2) return [random.random() for _ in range(4)] def add(w: float, x: float, y: float, z: float) -> float: return w + x + y + z def procss() -> float: return add(*fetch()) Let’s say we only want to test the process function. But process ultimately depends on the fetch function, which has multiple side effects—it returns pseudo-random values and waits for 2 seconds on a fictitious network call. Since we only care about process, we’ll mock the other two functions. Here’s how unittest.mock.ANY can make life easier: ...

This is the 4th time in a row that I’ve wasted time figuring out how to mock out a function during testing that calls the chained methods of a datetime.datetime object in the function body. So I thought I’d document it here. Consider this function: # src.py from __future__ import annotations import datetime def get_utcnow_isoformat() -> str: """Get UTCnow as an isoformat compliant string.""" return datetime.datetime.utcnow().isoformat() How’d you test it? Mocking out datetime.datetime is tricky because of its immutable nature. Third-party libraries like freezegun1 make it easier to mock and test functions like the one above. However, it’s not too difficult to cover this simple case without any additional dependencies. Here’s one way to achieve the goal: ...

While most of my pytest fixtures don’t react to the dynamically-passed values of function parameters, there have been situations where I’ve definitely felt the need for that. Consider this example: # test_src.py import pytest @pytest.fixture def create_file(tmp_path): """Fixture to create a file in the tmp_path/tmp directory.""" directory = tmp_path / "tmp" directory.mkdir() file = directory / "foo.md" # The filename is hardcoded here! yield directory, file def test_file_creation(create_file): """Check the fixture.""" directory, file = create_file assert directory.name == "tmp" assert file.name == "foo.md" Here, in the create_file fixture, I’ve created a file named foo.md in the tmp folder. Notice that the name of the file foo.md is hardcoded inside the body of the fixture function. The fixture yields the path of the directory and the created file. ...

In Python, even though I adore writing tests in a functional manner via pytest, I still have a soft corner for the tools provided in the unittest.mock module. I like the fact it’s baked into the standard library and is quite flexible. Moreover, I’m yet to see another mock library in any other language or in the Python ecosystem that allows you to mock your targets in such a terse, flexible, and maintainable fashion. ...