<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>API on Redowan&#39;s Reflections</title>
    <link>https://rednafi.com/tags/api/</link>
    <description>Recent content in API on Redowan&#39;s Reflections</description>
    <image>
      <title>Redowan&#39;s Reflections</title>
      <url>https://blob.rednafi.com/static/images/home/cover.png</url>
      <link>https://blob.rednafi.com/static/images/home/cover.png</link>
    </image>
    <generator>Hugo -- 0.154.2</generator>
    <language>en</language>
    <lastBuildDate>Sun, 15 Mar 2026 21:05:03 +0100</lastBuildDate>
    <atom:link href="https://rednafi.com/tags/api/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>Wrapping a gRPC client in Go</title>
      <link>https://rednafi.com/go/wrap-grpc-client/</link>
      <pubDate>Sun, 15 Mar 2026 00:00:00 +0000</pubDate>
      <guid>https://rednafi.com/go/wrap-grpc-client/</guid>
      <description>How to wrap a generated gRPC client behind a clean Go API so users never have to touch protobuf types or connection management directly.</description>
      <category>Go</category>
      <category>gRPC</category>
      <category>API</category>
      <category>Distributed Systems</category>
      <content:encoded><![CDATA[<p>Yesterday I wrote a <a href="/shards/2026/03/etcd-codebase">shard on exploring the etcd codebase</a>. One of the things that stood out
was how the <a href="https://github.com/etcd-io/etcd/tree/main/client/v3" rel="noopener noreferrer" target="_blank">clientv3 package</a> abstracts out the underlying gRPC machinery.</p>
<p>etcd is a distributed key-value store where the server and client communicate over gRPC. But
if you&rsquo;ve only ever used <code>clientv3</code> and never peeked into the internals, you wouldn&rsquo;t know
that. You call <code>resp, err := client.Put(ctx, &quot;key&quot;, &quot;value&quot;)</code> and get back a <code>*PutResponse</code>.
It feels like a regular Go library. The fact that gRPC and protobuf are involved is an
implementation detail that the client wrapper keeps away from you.</p>
<p>I&rsquo;ve been building a few gRPC services at work lately, and I keep running into the same
question: what API do the users of my client library see? The server ships as a binary. The
client ships as a Go package that other teams <code>go get</code>. If I hand them the raw generated
gRPC stubs, they have to import my protobuf types, manage gRPC connections, configure TLS,
and parse <code>codes.NotFound</code> from <code>google.golang.org/grpc/status</code>. That&rsquo;s a lot of protocol
plumbing for someone who just wants to consume my service.</p>
<p>This post walks through wrapping a generated gRPC client behind a higher level Go API,
following the same pattern etcd uses. The idea is to give the user a wrapper client that
abstracts out the generated client.</p>
<p>I&rsquo;ll use a small in-memory KV store as the running example.</p>
<h2 id="layout">Layout</h2>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-txt" data-lang="txt"><span class="line"><span class="cl">kv/
</span></span><span class="line"><span class="cl">├── api/
</span></span><span class="line"><span class="cl">│   ├── kv.proto           # service definition
</span></span><span class="line"><span class="cl">│   ├── kv.pb.go           # generated message types
</span></span><span class="line"><span class="cl">│   └── kv_grpc.pb.go      # generated client and server stubs
</span></span><span class="line"><span class="cl">├── client/
</span></span><span class="line"><span class="cl">│   └── client.go          # the wrapper (what users import)
</span></span><span class="line"><span class="cl">├── server/
</span></span><span class="line"><span class="cl">│   └── main.go            # the server binary
</span></span><span class="line"><span class="cl">└── go.mod
</span></span></code></pre></div><p><code>api/</code> holds the proto and generated code. <code>server/</code> is a binary you deploy. <code>client/</code> is
the library you ship. Other teams add it to their <code>go.mod</code> and never touch proto types
directly.</p>
<h2 id="defining-the-service">Defining the service</h2>
<p>The KV store has three RPCs: put, get, and delete.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-proto" data-lang="proto"><span class="line"><span class="cl"><span class="c1">// api/kv.proto
</span></span></span><span class="line"><span class="cl"><span class="n">syntax</span> <span class="o">=</span> <span class="s">&#34;proto3&#34;</span><span class="p">;</span><span class="err">
</span></span></span><span class="line"><span class="cl"><span class="kn">package</span> <span class="nn">kvpb</span><span class="p">;</span><span class="err">
</span></span></span><span class="line"><span class="cl"><span class="k">option</span> <span class="n">go_package</span> <span class="o">=</span> <span class="s">&#34;example.com/kv/api&#34;</span><span class="p">;</span><span class="err">
</span></span></span><span class="line"><span class="cl"><span class="err">
</span></span></span><span class="line"><span class="cl"><span class="kd">service</span> <span class="n">KV</span> <span class="p">{</span><span class="err">
</span></span></span><span class="line"><span class="cl">  <span class="k">rpc</span> <span class="n">Put</span><span class="p">(</span><span class="n">PutRequest</span><span class="p">)</span> <span class="k">returns</span> <span class="p">(</span><span class="n">PutResponse</span><span class="p">);</span><span class="err">
</span></span></span><span class="line"><span class="cl">  <span class="k">rpc</span> <span class="n">Get</span><span class="p">(</span><span class="n">GetRequest</span><span class="p">)</span> <span class="k">returns</span> <span class="p">(</span><span class="n">GetResponse</span><span class="p">);</span><span class="err">
</span></span></span><span class="line"><span class="cl">  <span class="k">rpc</span> <span class="n">Delete</span><span class="p">(</span><span class="n">DeleteRequest</span><span class="p">)</span> <span class="k">returns</span> <span class="p">(</span><span class="n">DeleteResponse</span><span class="p">);</span><span class="err">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="err">
</span></span></span><span class="line"><span class="cl"><span class="err">
</span></span></span><span class="line"><span class="cl"><span class="kd">message</span> <span class="nc">PutRequest</span>    <span class="p">{</span> <span class="kt">string</span> <span class="n">key</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span> <span class="kt">bytes</span> <span class="n">value</span> <span class="o">=</span> <span class="mi">2</span><span class="p">;</span> <span class="p">}</span><span class="err">
</span></span></span><span class="line"><span class="cl"><span class="kd">message</span> <span class="nc">PutResponse</span>   <span class="p">{}</span><span class="err">
</span></span></span><span class="line"><span class="cl"><span class="kd">message</span> <span class="nc">GetRequest</span>    <span class="p">{</span> <span class="kt">string</span> <span class="n">key</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span> <span class="p">}</span><span class="err">
</span></span></span><span class="line"><span class="cl"><span class="kd">message</span> <span class="nc">GetResponse</span>   <span class="p">{</span> <span class="kt">bytes</span> <span class="n">value</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span> <span class="k">optional</span> <span class="kt">bool</span> <span class="n">found</span> <span class="o">=</span> <span class="mi">2</span><span class="p">;</span> <span class="p">}</span><span class="err">
</span></span></span><span class="line"><span class="cl"><span class="kd">message</span> <span class="nc">DeleteRequest</span> <span class="p">{</span> <span class="kt">string</span> <span class="n">key</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span> <span class="p">}</span><span class="err">
</span></span></span><span class="line"><span class="cl"><span class="kd">message</span> <span class="nc">DeleteResponse</span> <span class="p">{}</span><span class="err">
</span></span></span></code></pre></div><p><code>GetResponse</code> uses <code>optional bool found</code> because proto3 normally can&rsquo;t distinguish &ldquo;field is
zero&rdquo; from &ldquo;field was never set.&rdquo; The <code>optional</code> keyword generates a pointer in Go, which
lets callers tell a missing key apart from an empty value.</p>
<p>Running <code>protoc</code> on this generates a client interface and a server stub. The client side
looks like this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// api/kv_grpc.pb.go (generated)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">KVClient</span><span class="w"> </span><span class="kd">interface</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">Put</span><span class="p">(</span><span class="nx">ctx</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w"> </span><span class="nx">in</span><span class="w"> </span><span class="o">*</span><span class="nx">PutRequest</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">opts</span><span class="w"> </span><span class="o">...</span><span class="nx">grpc</span><span class="p">.</span><span class="nx">CallOption</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="o">*</span><span class="nx">PutResponse</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">Get</span><span class="p">(</span><span class="nx">ctx</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w"> </span><span class="nx">in</span><span class="w"> </span><span class="o">*</span><span class="nx">GetRequest</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">opts</span><span class="w"> </span><span class="o">...</span><span class="nx">grpc</span><span class="p">.</span><span class="nx">CallOption</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="o">*</span><span class="nx">GetResponse</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">Delete</span><span class="p">(</span><span class="nx">ctx</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w"> </span><span class="nx">in</span><span class="w"> </span><span class="o">*</span><span class="nx">DeleteRequest</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">opts</span><span class="w"> </span><span class="o">...</span><span class="nx">grpc</span><span class="p">.</span><span class="nx">CallOption</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="o">*</span><span class="nx">DeleteResponse</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Every method takes a <code>context.Context</code>, a protobuf request struct, and variadic
<code>grpc.CallOption</code>s, and returns a protobuf response plus an error. Anyone calling the
service has to import protobuf types, construct request structs like <code>&amp;api.PutRequest{}</code>,
and understand gRPC call options, even for a simple &ldquo;get this key&rdquo; call.</p>
<p>The server implements the other side with an in-memory map. What we care about for the
wrapper is that it returns a gRPC <code>NOT_FOUND</code> status when a key doesn&rsquo;t exist. The wrapper
translates that into a Go sentinel error. Here&rsquo;s the server code:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// server/main.go</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">server</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">kvpb</span><span class="p">.</span><span class="nx">UnimplementedKVServer</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">data</span><span class="w"> </span><span class="kd">map</span><span class="p">[</span><span class="kt">string</span><span class="p">][]</span><span class="kt">byte</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">s</span><span class="w"> </span><span class="o">*</span><span class="nx">server</span><span class="p">)</span><span class="w"> </span><span class="nf">Get</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">ctx</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">kvpb</span><span class="p">.</span><span class="nx">GetRequest</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="o">*</span><span class="nx">kvpb</span><span class="p">.</span><span class="nx">GetResponse</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">v</span><span class="p">,</span><span class="w"> </span><span class="nx">ok</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">s</span><span class="p">.</span><span class="nx">data</span><span class="p">[</span><span class="nx">r</span><span class="p">.</span><span class="nx">Key</span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="p">!</span><span class="nx">ok</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="nx">status</span><span class="p">.</span><span class="nf">Errorf</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">codes</span><span class="p">.</span><span class="nx">NotFound</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;key %q&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="p">.</span><span class="nx">Key</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">kvpb</span><span class="p">.</span><span class="nx">GetResponse</span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">Value</span><span class="p">:</span><span class="w"> </span><span class="nx">v</span><span class="p">,</span><span class="w"> </span><span class="nx">Found</span><span class="p">:</span><span class="w"> </span><span class="nx">proto</span><span class="p">.</span><span class="nf">Bool</span><span class="p">(</span><span class="kc">true</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">},</span><span class="w"> </span><span class="kc">nil</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Put and Delete follow the same shape.</span><span class="w">
</span></span></span></code></pre></div><p>The server embeds <code>UnimplementedKVServer</code>, the standard gRPC pattern. It provides no-op
implementations for all RPCs so the code compiles even before you&rsquo;ve written the real logic.
The <code>Get</code> method checks the map and returns <code>codes.NotFound</code> when the key isn&rsquo;t there. This
is the status code the wrapper will catch and turn into a Go error. I&rsquo;ve elided <code>Put</code> and
<code>Delete</code> since they follow the same structure.</p>
<h2 id="using-the-generated-client-directly">Using the generated client directly</h2>
<p>Without a wrapper, callers use the generated <code>KVClient</code> directly. Pay attention to the
imports:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// example/main.go (raw usage without wrapper)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kn">import</span><span class="w"> </span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="s">&#34;context&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="s">&#34;google.golang.org/grpc&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="s">&#34;google.golang.org/grpc/credentials/insecure&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="s">&#34;example.com/kv/api&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// ...</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">conn</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">grpc</span><span class="p">.</span><span class="nf">NewClient</span><span class="p">(</span><span class="s">&#34;localhost:9090&#34;</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">grpc</span><span class="p">.</span><span class="nf">WithTransportCredentials</span><span class="p">(</span><span class="nx">insecure</span><span class="p">.</span><span class="nf">NewCredentials</span><span class="p">()))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// ...</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">kv</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">api</span><span class="p">.</span><span class="nf">NewKVClient</span><span class="p">(</span><span class="nx">conn</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">kv</span><span class="p">.</span><span class="nf">Put</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">api</span><span class="p">.</span><span class="nx">PutRequest</span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">Key</span><span class="p">:</span><span class="w"> </span><span class="s">&#34;greeting&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">Value</span><span class="p">:</span><span class="w"> </span><span class="p">[]</span><span class="nb">byte</span><span class="p">(</span><span class="s">&#34;hello&#34;</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">})</span><span class="w">
</span></span></span></code></pre></div><p>Three imports just to put a key. The caller manages the gRPC connection, constructs
<code>&amp;api.PutRequest{}</code> structs for every call, and has to parse gRPC status codes to check if a
key exists. For internal code where everyone knows gRPC, this is fine. For a library you
ship to other teams, it&rsquo;s a lot of ceremony.</p>
<h2 id="calling-the-server-with-the-wrapper">Calling the server with the wrapper</h2>
<p>This is the API we actually want to give our users. Same sequence as before (put a key, get
it back, handle a missing key) but without any gRPC or protobuf leaking through:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// example/main.go (with the wrapper)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kn">import</span><span class="w"> </span><span class="s">&#34;example.com/kv/client&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// ...</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">c</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">client</span><span class="p">.</span><span class="nf">New</span><span class="p">(</span><span class="s">&#34;localhost:9090&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// ...</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">defer</span><span class="w"> </span><span class="nx">c</span><span class="p">.</span><span class="nf">Close</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">c</span><span class="p">.</span><span class="nf">Put</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;greeting&#34;</span><span class="p">,</span><span class="w"> </span><span class="p">[]</span><span class="nb">byte</span><span class="p">(</span><span class="s">&#34;hello&#34;</span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">val</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">c</span><span class="p">.</span><span class="nf">Get</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;greeting&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">c</span><span class="p">.</span><span class="nf">Get</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;missing&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">if</span><span class="w"> </span><span class="nx">errors</span><span class="p">.</span><span class="nf">Is</span><span class="p">(</span><span class="nx">err</span><span class="p">,</span><span class="w"> </span><span class="nx">client</span><span class="p">.</span><span class="nx">ErrNotFound</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="o">...</span><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>One import instead of three. No gRPC or protobuf packages in sight. <code>Put</code> takes a string and
a byte slice. <code>Get</code> returns <code>[]byte</code>. Missing keys come back as <code>client.ErrNotFound</code>,
checked with <code>errors.Is</code> like any other Go error. The caller doesn&rsquo;t need to know that gRPC
is involved at all.</p>
<div class="alert alert-note">
  <p class="alert-title">Note</p>
  <p>Callers never have to build an <code>api.PutRequest</code>, call <code>grpc.NewClient</code>, configure TLS, or
check <code>codes.NotFound</code>. They pass strings and byte slices, get Go errors back, and the
wrapper handles the rest.</p>
</div><p>The rest of this post builds the wrapper that turns the generated <code>KVClient</code> from the
previous section into this API.</p>
<h2 id="building-the-wrapper">Building the wrapper</h2>
<p>The <code>client/</code> package is the only thing users import. It hides the generated <code>api.KVClient</code>
behind a struct and re-exposes the same operations using plain Go types. The whole wrapper
lives in a single file (<code>client/client.go</code>).</p>
<p>The wrapper starts with a sentinel error and a testable interface:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// client/client.go</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">var</span><span class="w"> </span><span class="nx">ErrNotFound</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">errors</span><span class="p">.</span><span class="nf">New</span><span class="p">(</span><span class="s">&#34;key not found&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">KV</span><span class="w"> </span><span class="kd">interface</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">Put</span><span class="p">(</span><span class="nx">ctx</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w"> </span><span class="nx">key</span><span class="w"> </span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="nx">value</span><span class="w"> </span><span class="p">[]</span><span class="kt">byte</span><span class="p">)</span><span class="w"> </span><span class="kt">error</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">Get</span><span class="p">(</span><span class="nx">ctx</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w"> </span><span class="nx">key</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">([]</span><span class="kt">byte</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">Delete</span><span class="p">(</span><span class="nx">ctx</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w"> </span><span class="nx">key</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="kt">error</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p><code>ErrNotFound</code> replaces the gRPC <code>NOT_FOUND</code> status code. Callers check it with <code>errors.Is</code>
and never import <code>google.golang.org/grpc/codes</code>.</p>
<p><code>Client</code> implements <code>KV</code>, and <code>KV</code> uses only standard Go types instead of protobuf or gRPC
types. This is intentionally a producer-side interface: we define it in the same package as
<code>Client</code> because we know the full set of operations the service supports and we want to
offer a ready-made contract for consumers. Other packages that depend on your client can
accept a <code>KV</code> in their function signatures and swap in a simple in-memory fake during tests
without spinning up a gRPC server or importing any gRPC packages.</p>
<div class="alert alert-important">
  <p class="alert-title">Important</p>
  <p><code>KV</code> is a producer-side interface. I wrote about when these make sense in <a href="/go/interface-segregation">Revisiting
interface segregation in Go</a>.</p>
</div><p>Then the struct and constructor:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">Client</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">conn</span><span class="w"> </span><span class="o">*</span><span class="nx">grpc</span><span class="p">.</span><span class="nx">ClientConn</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">kv</span><span class="w">   </span><span class="nx">api</span><span class="p">.</span><span class="nx">KVClient</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">New</span><span class="p">(</span><span class="nx">addr</span><span class="w"> </span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="nx">opts</span><span class="w"> </span><span class="o">...</span><span class="nx">grpc</span><span class="p">.</span><span class="nx">DialOption</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="o">*</span><span class="nx">Client</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nb">len</span><span class="p">(</span><span class="nx">opts</span><span class="p">)</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">opts</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="p">[]</span><span class="nx">grpc</span><span class="p">.</span><span class="nx">DialOption</span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">grpc</span><span class="p">.</span><span class="nf">WithTransportCredentials</span><span class="p">(</span><span class="nx">insecure</span><span class="p">.</span><span class="nf">NewCredentials</span><span class="p">()),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">conn</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">grpc</span><span class="p">.</span><span class="nf">NewClient</span><span class="p">(</span><span class="nx">addr</span><span class="p">,</span><span class="w"> </span><span class="nx">opts</span><span class="o">...</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Errorf</span><span class="p">(</span><span class="s">&#34;connecting to %s: %v&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">addr</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">Client</span><span class="p">{</span><span class="nx">conn</span><span class="p">:</span><span class="w"> </span><span class="nx">conn</span><span class="p">,</span><span class="w"> </span><span class="nx">kv</span><span class="p">:</span><span class="w"> </span><span class="nx">api</span><span class="p">.</span><span class="nf">NewKVClient</span><span class="p">(</span><span class="nx">conn</span><span class="p">)},</span><span class="w"> </span><span class="kc">nil</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">c</span><span class="w"> </span><span class="o">*</span><span class="nx">Client</span><span class="p">)</span><span class="w"> </span><span class="nf">Close</span><span class="p">()</span><span class="w"> </span><span class="kt">error</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">c</span><span class="p">.</span><span class="nx">conn</span><span class="p">.</span><span class="nf">Close</span><span class="p">()</span><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p><code>Client</code> holds the gRPC connection and the generated <code>api.KVClient</code> as unexported fields.
Note that <code>api.KVClient</code> is an interface, not a concrete struct. The gRPC codegen doesn&rsquo;t
expose the actual client struct at all; you get back a <code>KVClient</code> interface from
<code>api.NewKVClient(conn)</code>. We store it as a regular field rather than embedding it. If you
embedded the <code>api.KVClient</code> interface, all its methods like
<code>Put(ctx, *PutRequest, ...CallOption)</code> would be promoted onto <code>Client</code> directly, and callers
could bypass the wrapper to make raw gRPC calls.</p>
<div class="alert alert-warning">
  <p class="alert-title">Warning</p>
  <p>Don&rsquo;t embed the generated client interface. Keep it as a private field so the only way to
talk to the server is through the wrapper methods.</p>
</div><p><code>New</code> creates the gRPC connection and builds the generated client from it. The variadic
<code>grpc.DialOption</code> lets callers pass custom TLS, keepalive, or interceptor config. If they
pass nothing, the default is insecure credentials for local dev. The retries section below
shows what a production setup looks like.</p>
<p>With the types in place, we can look at the wrapper methods. <code>Get</code> shows the pattern all
three follow:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">c</span><span class="w"> </span><span class="o">*</span><span class="nx">Client</span><span class="p">)</span><span class="w"> </span><span class="nf">Get</span><span class="p">(</span><span class="nx">ctx</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w"> </span><span class="nx">key</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">([]</span><span class="kt">byte</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">resp</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">c</span><span class="p">.</span><span class="nx">kv</span><span class="p">.</span><span class="nf">Get</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">api</span><span class="p">.</span><span class="nx">GetRequest</span><span class="p">{</span><span class="nx">Key</span><span class="p">:</span><span class="w"> </span><span class="nx">key</span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">if</span><span class="w"> </span><span class="nx">s</span><span class="p">,</span><span class="w"> </span><span class="nx">ok</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">status</span><span class="p">.</span><span class="nf">FromError</span><span class="p">(</span><span class="nx">err</span><span class="p">);</span><span class="w"> </span><span class="nx">ok</span><span class="w"> </span><span class="o">&amp;&amp;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">s</span><span class="p">.</span><span class="nf">Code</span><span class="p">()</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="nx">codes</span><span class="p">.</span><span class="nx">NotFound</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="nx">ErrNotFound</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Errorf</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="s">&#34;getting key %s: %v&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">key</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">resp</span><span class="p">.</span><span class="nx">Value</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Put and Delete follow the same shape.</span><span class="w">
</span></span></span></code></pre></div><p>Each wrapper method follows the same pattern: take the caller&rsquo;s Go arguments, build the
protobuf request internally, call the generated client, and return plain Go types.</p>
<p>Pay attention to the error handling. When the server returns <code>NOT_FOUND</code>, we catch that gRPC
status and convert it to our own <code>ErrNotFound</code> sentinel so callers can check it with
<code>errors.Is</code> instead of parsing gRPC status codes themselves. For everything else, we wrap
with <code>%v</code> instead of <code>%w</code>. If we used <code>%w</code>, callers could unwrap the error with <code>errors.As</code>
and reach the underlying gRPC status types, which would re-couple them to gRPC internals and
defeat the whole point of having a wrapper. I wrote about this tradeoff in <a href="/go/to-wrap-or-not-to-wrap">Go errors: to
wrap or not to wrap?</a>.</p>
<h2 id="plugging-in-retries-and-metrics">Plugging in retries and metrics</h2>
<p>Since the wrapper owns the <code>grpc.NewClient</code> call, it can bake in retries and observability
without the caller knowing. gRPC interceptors work like HTTP middleware. They wrap every RPC
with extra logic (logging, retries, metrics) without changing the handler code. You register
them as dial options when creating the connection:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// client/client.go (production version of New)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">New</span><span class="p">(</span><span class="nx">addr</span><span class="w"> </span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="nx">opts</span><span class="w"> </span><span class="o">...</span><span class="nx">grpc</span><span class="p">.</span><span class="nx">DialOption</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="o">*</span><span class="nx">Client</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">defaults</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="p">[]</span><span class="nx">grpc</span><span class="p">.</span><span class="nx">DialOption</span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">grpc</span><span class="p">.</span><span class="nf">WithTransportCredentials</span><span class="p">(</span><span class="nx">credentials</span><span class="p">.</span><span class="nf">NewTLS</span><span class="p">(</span><span class="o">&amp;</span><span class="nx">tls</span><span class="p">.</span><span class="nx">Config</span><span class="p">{})),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">grpc</span><span class="p">.</span><span class="nf">WithChainUnaryInterceptor</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">grpc_retry</span><span class="p">.</span><span class="nf">UnaryClientInterceptor</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="nx">grpc_retry</span><span class="p">.</span><span class="nf">WithMax</span><span class="p">(</span><span class="mi">3</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="nx">grpc_retry</span><span class="p">.</span><span class="nf">WithBackoff</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="nx">grpc_retry</span><span class="p">.</span><span class="nf">BackoffExponential</span><span class="p">(</span><span class="mi">100</span><span class="o">*</span><span class="nx">time</span><span class="p">.</span><span class="nx">Millisecond</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">grpcprom</span><span class="p">.</span><span class="nx">UnaryClientInterceptor</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">opts</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nb">append</span><span class="p">(</span><span class="nx">defaults</span><span class="p">,</span><span class="w"> </span><span class="nx">opts</span><span class="o">...</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// ... rest is the same</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p><a href="https://pkg.go.dev/github.com/grpc-ecosystem/go-grpc-middleware/retry" rel="noopener noreferrer" target="_blank">grpc_retry</a> from <a href="https://github.com/grpc-ecosystem/go-grpc-middleware" rel="noopener noreferrer" target="_blank">go-grpc-middleware</a> retries failed RPCs with exponential backoff.
<a href="https://pkg.go.dev/github.com/grpc-ecosystem/go-grpc-middleware/providers/prometheus" rel="noopener noreferrer" target="_blank">grpcprom</a> records latency histograms and error rates. Same <code>client.New</code>, same <code>c.Put</code>, but
now with retries and metrics baked in. Callers who need to override the defaults can pass
their own dial options. This is useful in tests where you might want insecure credentials or
no retries.</p>
<h2 id="try-it-yourself">Try it yourself</h2>
<p>The full code is on <a href="https://github.com/rednafi/examples/tree/main/wrapping-grpc-client" rel="noopener noreferrer" target="_blank">GitHub</a>. Install the server and run the example:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl">go install github.com/rednafi/examples/wrapping-grpc-client/server@latest
</span></span><span class="line"><span class="cl">server <span class="p">&amp;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">go install github.com/rednafi/examples/wrapping-grpc-client/example@latest
</span></span><span class="line"><span class="cl">example
</span></span></code></pre></div><p>Running the example will return:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="cl">put greeting=hello
</span></span><span class="line"><span class="cl">get greeting=hello
</span></span><span class="line"><span class="cl">get missing: not found (expected)
</span></span><span class="line"><span class="cl">deleted greeting
</span></span><span class="line"><span class="cl">get greeting after delete: not found (expected)
</span></span></code></pre></div><p>Or add the client library to your own project:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl">go get github.com/rednafi/examples/wrapping-grpc-client/client@latest
</span></span></code></pre></div><!-- references -->
<!-- prettier-ignore-start -->
<!-- prettier-ignore-end -->
]]></content:encoded>
    </item>
    <item>
      <title>Mutate your locked state inside a closure</title>
      <link>https://rednafi.com/go/mutex-closure/</link>
      <pubDate>Thu, 05 Mar 2026 00:00:00 +0000</pubDate>
      <guid>https://rednafi.com/go/mutex-closure/</guid>
      <description>Why your mutex wrapper should accept a closure for mutation instead of a plain value, with examples from the standard library and Tailscale.</description>
      <category>Go</category>
      <category>Concurrency</category>
      <category>API</category>
      <content:encoded><![CDATA[<p>When multiple goroutines need to read and write the same value, you need a mutex to make sure
they don&rsquo;t step on each other. Without one, concurrent writes can corrupt the state - two
goroutines might read the same value, both modify it, and one silently overwrites the other&rsquo;s
change. The usual approach is to put a <code>sync.Mutex</code> next to the fields it protects:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">var</span><span class="w"> </span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">mu</span><span class="w">      </span><span class="nx">sync</span><span class="p">.</span><span class="nx">Mutex</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">counter</span><span class="w"> </span><span class="kt">int</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">mu</span><span class="p">.</span><span class="nf">Lock</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">counter</span><span class="o">++</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">mu</span><span class="p">.</span><span class="nf">Unlock</span><span class="p">()</span><span class="w">
</span></span></span></code></pre></div><p>This works, but nothing enforces it. The compiler won&rsquo;t stop you from accessing <code>counter</code>
without holding the lock. Forget to lock in one spot and you have a data race. One way to
make this safer is to bundle the value and its mutex into a small generic wrapper that only
exposes locked access through methods:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">Locked</span><span class="p">[</span><span class="nx">T</span><span class="w"> </span><span class="kt">any</span><span class="p">]</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">mu</span><span class="w"> </span><span class="nx">sync</span><span class="p">.</span><span class="nx">Mutex</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">v</span><span class="w">  </span><span class="nx">T</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nx">NewLocked</span><span class="p">[</span><span class="nx">T</span><span class="w"> </span><span class="kt">any</span><span class="p">](</span><span class="nx">initial</span><span class="w"> </span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="o">*</span><span class="nx">Locked</span><span class="p">[</span><span class="nx">T</span><span class="p">]</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">Locked</span><span class="p">[</span><span class="nx">T</span><span class="p">]{</span><span class="nx">v</span><span class="p">:</span><span class="w"> </span><span class="nx">initial</span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">l</span><span class="w"> </span><span class="o">*</span><span class="nx">Locked</span><span class="p">[</span><span class="nx">T</span><span class="p">])</span><span class="w"> </span><span class="nf">Get</span><span class="p">()</span><span class="w"> </span><span class="nx">T</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">l</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nf">Lock</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">defer</span><span class="w"> </span><span class="nx">l</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nf">Unlock</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">l</span><span class="p">.</span><span class="nx">v</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">l</span><span class="w"> </span><span class="o">*</span><span class="nx">Locked</span><span class="p">[</span><span class="nx">T</span><span class="p">])</span><span class="w"> </span><span class="nf">Set</span><span class="p">(</span><span class="nx">v</span><span class="w"> </span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">l</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nf">Lock</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">defer</span><span class="w"> </span><span class="nx">l</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nf">Unlock</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">l</span><span class="p">.</span><span class="nx">v</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">v</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>You keep <code>mu</code> and <code>v</code> unexported, pass around <code>*Locked[T]</code>, and callers use <code>Get</code> to read
and <code>Set</code> to write:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">counter</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">NewLocked</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">counter</span><span class="p">.</span><span class="nf">Set</span><span class="p">(</span><span class="mi">42</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">fmt</span><span class="p">.</span><span class="nf">Println</span><span class="p">(</span><span class="nx">counter</span><span class="p">.</span><span class="nf">Get</span><span class="p">())</span><span class="w"> </span><span class="c1">// 42</span><span class="w">
</span></span></span></code></pre></div><p>Now callers can&rsquo;t touch the underlying value without going through the lock. This doesn&rsquo;t
prevent misuse within the same package, but it makes unprotected access from other packages
impossible.</p>
<p>This works fine when you&rsquo;re replacing the value wholesale - just call <code>counter.Set(42)</code> and
move on. But when your mutation depends on the current value, <code>Get</code> and <code>Set</code> can race
against each other.</p>
<h2 id="the-problem-with-set">The problem with Set</h2>
<p>Say you want to increment the counter instead of replacing it. You&rsquo;d have to do:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">v</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">counter</span><span class="p">.</span><span class="nf">Get</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">v</span><span class="o">++</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">counter</span><span class="p">.</span><span class="nf">Set</span><span class="p">(</span><span class="nx">v</span><span class="p">)</span><span class="w">
</span></span></span></code></pre></div><p>Each individual call is safe - <code>Get</code> holds the lock while reading, <code>Set</code> holds it while
writing. But the three calls together aren&rsquo;t atomic. Between <code>Get</code> and <code>Set</code>, another
goroutine can modify the value, and your increment overwrites theirs. That&rsquo;s the classic
lost-update bug.</p>
<p>It gets worse with compound state. Say the wrapper holds a struct:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">State</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">Count</span><span class="w"> </span><span class="kt">int</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">Name</span><span class="w">  </span><span class="kt">string</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">state</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">NewLocked</span><span class="p">(</span><span class="nx">State</span><span class="p">{})</span><span class="w">
</span></span></span></code></pre></div><p>And you want to conditionally update both fields:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">s</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">state</span><span class="p">.</span><span class="nf">Get</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">if</span><span class="w"> </span><span class="nx">s</span><span class="p">.</span><span class="nx">Count</span><span class="w"> </span><span class="p">&lt;</span><span class="w"> </span><span class="mi">10</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">s</span><span class="p">.</span><span class="nx">Count</span><span class="o">++</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">s</span><span class="p">.</span><span class="nx">Name</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Sprintf</span><span class="p">(</span><span class="s">&#34;item-%d&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">s</span><span class="p">.</span><span class="nx">Count</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">state</span><span class="p">.</span><span class="nf">Set</span><span class="p">(</span><span class="nx">s</span><span class="p">)</span><span class="w">
</span></span></span></code></pre></div><p>Same problem. <code>Get</code> returns a copy, you mutate the copy, then <code>Set</code> writes it back. If
another goroutine modified <code>state</code> between those two calls, your write clobbers it.</p>
<div class="alert alert-important">
  <p class="alert-title">Important</p>
  <p>The race detector (<code>go test -race</code>) won&rsquo;t catch this. It detects data races - two
goroutines accessing the same memory without synchronization. Here, every <code>Get</code> and <code>Set</code>
properly acquires the mutex, so each individual access is synchronized. The bug is a
logical race (lost update), not a data race. The race detector sees nothing wrong.</p>
<p>You can prove this with a simple test. Ten goroutines each increment the counter 1000
times, so the final value should be 10000:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">TestSetValue</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">counter</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">NewLocked</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">var</span><span class="w"> </span><span class="nx">wg</span><span class="w"> </span><span class="nx">sync</span><span class="p">.</span><span class="nx">WaitGroup</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">for</span><span class="w"> </span><span class="k">range</span><span class="w"> </span><span class="mi">10</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">wg</span><span class="p">.</span><span class="nf">Go</span><span class="p">(</span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="k">for</span><span class="w"> </span><span class="k">range</span><span class="w"> </span><span class="mi">1000</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="nx">v</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">counter</span><span class="p">.</span><span class="nf">Get</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="nx">v</span><span class="o">++</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="nx">counter</span><span class="p">.</span><span class="nf">SetValue</span><span class="p">(</span><span class="nx">v</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">wg</span><span class="p">.</span><span class="nf">Wait</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">got</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">counter</span><span class="p">.</span><span class="nf">Get</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">got</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="mi">10000</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">t</span><span class="p">.</span><span class="nf">Errorf</span><span class="p">(</span><span class="s">&#34;got %d, want 10000 (lost %d updates)&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">got</span><span class="p">,</span><span class="w"> </span><span class="mi">10000</span><span class="o">-</span><span class="nx">got</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Running <code>go test -race</code> produces no race warnings, but the test fails:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-txt" data-lang="txt"><span class="line"><span class="cl">=== RUN   TestSetValue
</span></span><span class="line"><span class="cl">    locked_test.go:30: got 1855, want 10000 (lost 8145 updates)
</span></span><span class="line"><span class="cl">--- FAIL: TestSetValue (0.02s)
</span></span></code></pre></div><p>The race detector is silent. The updates are just gone.</p>
</div><h2 id="take-a-function-instead">Take a function instead</h2>
<p>Instead of taking a value, have <code>Set</code> take a function:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">l</span><span class="w"> </span><span class="o">*</span><span class="nx">Locked</span><span class="p">[</span><span class="nx">T</span><span class="p">])</span><span class="w"> </span><span class="nf">Set</span><span class="p">(</span><span class="nx">f</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="o">*</span><span class="nx">T</span><span class="p">))</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">l</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nf">Lock</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">defer</span><span class="w"> </span><span class="nx">l</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nf">Unlock</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">f</span><span class="p">(</span><span class="o">&amp;</span><span class="nx">l</span><span class="p">.</span><span class="nx">v</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Now the counter increment becomes:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">counter</span><span class="p">.</span><span class="nf">Set</span><span class="p">(</span><span class="kd">func</span><span class="p">(</span><span class="nx">v</span><span class="w"> </span><span class="o">*</span><span class="kt">int</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="o">*</span><span class="nx">v</span><span class="o">++</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">})</span><span class="w">
</span></span></span></code></pre></div><p>And the compound mutation:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">state</span><span class="p">.</span><span class="nf">Set</span><span class="p">(</span><span class="kd">func</span><span class="p">(</span><span class="nx">s</span><span class="w"> </span><span class="o">*</span><span class="nx">State</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">s</span><span class="p">.</span><span class="nx">Count</span><span class="w"> </span><span class="p">&lt;</span><span class="w"> </span><span class="mi">10</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">s</span><span class="p">.</span><span class="nx">Count</span><span class="o">++</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">s</span><span class="p">.</span><span class="nx">Name</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Sprintf</span><span class="p">(</span><span class="s">&#34;item-%d&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">s</span><span class="p">.</span><span class="nx">Count</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">})</span><span class="w">
</span></span></span></code></pre></div><p>The lock is held for the entire closure. There&rsquo;s no gap between reading and writing, so no
other goroutine can interfere. Both fields update together or not at all.</p>
<p>The function takes a pointer to <code>T</code> rather than a value of <code>T</code> for two reasons. First, it
lets you mutate the state in place instead of working on a copy. Second, if <code>T</code> is a large
struct, passing a pointer avoids copying the whole thing into the closure on every call.</p>
<h2 id="the-stdlib-already-does-this">The stdlib already does this</h2>
<p>Go&rsquo;s <code>database/sql</code> package has an internal <a href="https://cs.opensource.google/go/go/+/refs/tags/go1.24.0:src/database/sql/sql.go;l=3576" rel="noopener noreferrer" target="_blank">withLock</a> helper that follows the same pattern:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// withLock runs while holding lk.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">withLock</span><span class="p">(</span><span class="nx">lk</span><span class="w"> </span><span class="nx">sync</span><span class="p">.</span><span class="nx">Locker</span><span class="p">,</span><span class="w"> </span><span class="nx">fn</span><span class="w"> </span><span class="kd">func</span><span class="p">())</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">lk</span><span class="p">.</span><span class="nf">Lock</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">defer</span><span class="w"> </span><span class="nx">lk</span><span class="p">.</span><span class="nf">Unlock</span><span class="p">()</span><span class="w"> </span><span class="c1">// in case fn panics</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">fn</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>It&rsquo;s used throughout <code>database/sql</code> to serialize access to the underlying driver connection.
For example, when pinging a connection:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="k">if</span><span class="w"> </span><span class="nx">pinger</span><span class="p">,</span><span class="w"> </span><span class="nx">ok</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">dc</span><span class="p">.</span><span class="nx">ci</span><span class="p">.(</span><span class="nx">driver</span><span class="p">.</span><span class="nx">Pinger</span><span class="p">);</span><span class="w"> </span><span class="nx">ok</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">withLock</span><span class="p">(</span><span class="nx">dc</span><span class="p">,</span><span class="w"> </span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">pinger</span><span class="p">.</span><span class="nf">Ping</span><span class="p">(</span><span class="nx">ctx</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Or when preparing a statement:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nf">withLock</span><span class="p">(</span><span class="nx">dc</span><span class="p">,</span><span class="w"> </span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">si</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nf">ctxDriverPrepare</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span><span class="w"> </span><span class="nx">dc</span><span class="p">.</span><span class="nx">ci</span><span class="p">,</span><span class="w"> </span><span class="nx">query</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">})</span><span class="w">
</span></span></span></code></pre></div><p>Or committing a transaction:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nf">withLock</span><span class="p">(</span><span class="nx">tx</span><span class="p">.</span><span class="nx">dc</span><span class="p">,</span><span class="w"> </span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">tx</span><span class="p">.</span><span class="nx">txi</span><span class="p">.</span><span class="nf">Commit</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">})</span><span class="w">
</span></span></span></code></pre></div><p>There are about 18 call sites in <code>sql.go</code> alone. In those snippets, <code>dc</code> is a
<code>*driverConn</code> - the struct that wraps a database driver connection. It embeds <code>sync.Mutex</code>
directly, so it satisfies <code>sync.Locker</code> and can be passed straight to <code>withLock</code>.</p>
<div class="alert alert-note">
  <p class="alert-title">Note</p>
  <p><code>withLock</code> accepts <code>sync.Locker</code> instead of <code>*sync.Mutex</code>, so it also works with the read
side of an <code>RWMutex</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nf">withLock</span><span class="p">(</span><span class="nx">rs</span><span class="p">.</span><span class="nx">closemu</span><span class="p">.</span><span class="nf">RLocker</span><span class="p">(),</span><span class="w"> </span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">doClose</span><span class="p">,</span><span class="w"> </span><span class="nx">ok</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">rs</span><span class="p">.</span><span class="nf">nextLocked</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">})</span><span class="w">
</span></span></span></code></pre></div><p>Here <code>rs.closemu</code> is a <code>sync.RWMutex</code>, and <code>.RLocker()</code> returns a <code>sync.Locker</code> that
acquires the read lock. The same <code>withLock</code> function handles both cases.</p>
</div><h2 id="the-proposal-to-add-this-to-sync">The proposal to add this to sync</h2>
<p>In 2021, twmb filed <a href="https://github.com/golang/go/issues/49563" rel="noopener noreferrer" target="_blank">proposal #49563</a> to add a <code>Mutex.Locked(func())</code> method to the standard
library:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">m</span><span class="w"> </span><span class="o">*</span><span class="nx">Mutex</span><span class="p">)</span><span class="w"> </span><span class="nf">Locked</span><span class="p">(</span><span class="nx">fn</span><span class="w"> </span><span class="kd">func</span><span class="p">())</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">m</span><span class="p">.</span><span class="nf">Lock</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">defer</span><span class="w"> </span><span class="nx">m</span><span class="p">.</span><span class="nf">Unlock</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">fn</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>The idea was that if <code>sync.Mutex</code> had this method natively, you wouldn&rsquo;t need to write a
wrapper at all for simple cases - you&rsquo;d just call <code>mu.Locked(fn)</code> directly. It also
eliminates forgotten unlocks and guards against panics leaving the mutex locked. esote
pointed out that <code>database/sql</code> already had an internal version of this - the same
<code>withLock</code> helper we saw earlier.</p>
<p>zephyrtronium raised the <code>sync.Locker</code> point:</p>
<blockquote>
  <p>I think there are advantages to making this a function that takes a Locker rather than a
method on Mutex. This would allow using it with either end of an RWMutex, or another
custom Locker.</p>
<p>&ndash; <a href="https://github.com/golang/go/issues/49563#issuecomment-968093753" rel="noopener noreferrer" target="_blank">zephyrtronium on #49563</a></p>

</blockquote><p>rsc declined it on philosophical grounds:</p>
<blockquote>
  <p>In general we try not to have two different ways to do something, and for better or worse
we have the current idioms.</p>
<p>&ndash; <a href="https://github.com/golang/go/issues/49563#issuecomment-983955169" rel="noopener noreferrer" target="_blank">rsc on #49563</a></p>

</blockquote><p>The more interesting pushback came from bcmills, who argued the proposal didn&rsquo;t go far
enough. With generics arriving, he wanted something that also prevents unguarded access to
the protected data, not just forgotten unlocks:</p>
<blockquote>
  <p>Now that we have generics on the way, I would rather see us move in a direction that
<em>also</em> eliminates unlocked-access bugs, not just incrementally update <code>Mutex</code> for
forgotten-<code>defer</code> bugs.</p>
<p>&ndash; <a href="https://github.com/golang/go/issues/49563#issuecomment-984092316" rel="noopener noreferrer" target="_blank">bcmills on #49563</a></p>

</blockquote><p>He sketched out what that could look like:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">Synchronized</span><span class="p">[</span><span class="nx">T</span><span class="w"> </span><span class="kt">any</span><span class="p">]</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">mu</span><span class="w">  </span><span class="nx">Mutex</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">val</span><span class="w"> </span><span class="nx">T</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">s</span><span class="w"> </span><span class="o">*</span><span class="nx">Synchronized</span><span class="p">[</span><span class="nx">T</span><span class="p">])</span><span class="w"> </span><span class="nf">Do</span><span class="p">(</span><span class="nx">fn</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="o">*</span><span class="nx">T</span><span class="p">))</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">s</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nf">Lock</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">defer</span><span class="w"> </span><span class="nx">s</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nf">Unlock</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">fn</span><span class="p">(</span><span class="o">&amp;</span><span class="nx">s</span><span class="p">.</span><span class="nx">val</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>This is essentially the <code>Locked[T]</code> wrapper from the beginning of this post. The proposal
was declined, but bcmills&rsquo; suggestion is the direction the community ended up going
anyway-just outside the standard library.</p>
<h2 id="tailscales-mutexvalue">Tailscale&rsquo;s MutexValue</h2>
<p>Tailscale&rsquo;s <a href="https://github.com/tailscale/tailscale/blob/v1.94.2/syncs/syncs.go#L110" rel="noopener noreferrer" target="_blank">syncs</a> package has a <code>MutexValue[T]</code> type that follows this direction:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">MutexValue</span><span class="p">[</span><span class="nx">T</span><span class="w"> </span><span class="kt">any</span><span class="p">]</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">mu</span><span class="w"> </span><span class="nx">sync</span><span class="p">.</span><span class="nx">Mutex</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">v</span><span class="w">  </span><span class="nx">T</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">m</span><span class="w"> </span><span class="o">*</span><span class="nx">MutexValue</span><span class="p">[</span><span class="nx">T</span><span class="p">])</span><span class="w"> </span><span class="nf">WithLock</span><span class="p">(</span><span class="nx">f</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="nx">p</span><span class="w"> </span><span class="o">*</span><span class="nx">T</span><span class="p">))</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">m</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nf">Lock</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">defer</span><span class="w"> </span><span class="nx">m</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nf">Unlock</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">f</span><span class="p">(</span><span class="o">&amp;</span><span class="nx">m</span><span class="p">.</span><span class="nx">v</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">m</span><span class="w"> </span><span class="o">*</span><span class="nx">MutexValue</span><span class="p">[</span><span class="nx">T</span><span class="p">])</span><span class="w"> </span><span class="nf">Load</span><span class="p">()</span><span class="w"> </span><span class="nx">T</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">m</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nf">Lock</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">defer</span><span class="w"> </span><span class="nx">m</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nf">Unlock</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">m</span><span class="p">.</span><span class="nx">v</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">m</span><span class="w"> </span><span class="o">*</span><span class="nx">MutexValue</span><span class="p">[</span><span class="nx">T</span><span class="p">])</span><span class="w"> </span><span class="nf">Store</span><span class="p">(</span><span class="nx">v</span><span class="w"> </span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">m</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nf">Lock</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">defer</span><span class="w"> </span><span class="nx">m</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nf">Unlock</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">m</span><span class="p">.</span><span class="nx">v</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">v</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>They provide both <code>Store</code> for simple replacements and <code>WithLock</code> for compound mutations.
When you need to read-modify-write, you go through <code>WithLock</code> so the lock covers the whole
operation.</p>
<h2 id="when-a-plain-set-is-fine">When a plain Set is fine</h2>
<p>If <code>T</code> is small and you only ever replace the whole value without reading it first, a plain
<code>Set</code> works. A boolean flag that gets toggled from one place, a config value that gets
swapped wholesale - those are fine.</p>
<p>But most state doesn&rsquo;t stay that simple. You start with a single integer, it becomes a
struct with three fields, and now you need to update two of them based on the third. At that
point, <code>Set(func(*T))</code> is the only safe option.</p>
<div class="alert alert-important">
  <p class="alert-title">Important</p>
  <p>The proposal benchmarks showed about 35% overhead for the closure-based approach (14.65
ns/op vs 10.82 ns/op for direct lock/unlock) due to closures and <code>defer</code> not being
inlineable. In practice this rarely matters. If your critical section does any real work,
the lock overhead dominates.</p>
</div><!-- references -->
<!-- prettier-ignore-start -->
<!-- prettier-ignore-end -->
]]></content:encoded>
    </item>
    <item>
      <title>Your Go tests probably don&#39;t need a mocking library</title>
      <link>https://rednafi.com/go/mocking-libraries-bleh/</link>
      <pubDate>Fri, 23 Jan 2026 00:00:00 +0000</pubDate>
      <guid>https://rednafi.com/go/mocking-libraries-bleh/</guid>
      <description>Practical patterns for mocking in Go without external libraries. Learn to mock functions, methods, interfaces, HTTP calls, and time using only the standard library</description>
      <category>Go</category>
      <category>Testing</category>
      <category>API</category>
      <content:encoded><![CDATA[<blockquote>
  <p>There are frameworks that generate those kind of fakes, and one of them is called
GoMock&hellip; they&rsquo;re fine, but I find that on balance, the handwritten fakes tend to be
easier to reason about and clearer to sort of see what&rsquo;s going on. But I&rsquo;m not an
enterprise Go programmer. Maybe people do need that, so I don&rsquo;t know, but that&rsquo;s my
advice.</p>
<p>&ndash; Andrew Gerrand, <a href="https://www.youtube.com/watch?v=ndmB0bj7eyw&t=2804s" rel="noopener noreferrer" target="_blank">Testing Techniques (46:44)</a></p>

</blockquote><p>No shade against mocking libraries like <a href="https://github.com/uber-go/mock" rel="noopener noreferrer" target="_blank">gomock</a> or <a href="https://github.com/vektra/mockery" rel="noopener noreferrer" target="_blank">mockery</a>. I use them all the time, both
at work and outside. But one thing I&rsquo;ve noticed is that generating mocks often leads to
poorly designed tests and increases onboarding time for a codebase.</p>
<p>Also, since almost no one writes tests by hand anymore and instead generates them with LLMs,
the situation gets more dire. These ghosts often pull in all kinds of third-party libraries
to mock your code, simply because they were trained on a lot of hastily written examples on
the web.</p>
<p>So the idea of this post isn&rsquo;t to discourage using mocking libraries. Rather, it&rsquo;s to show
that even if your codebase already has a mocking library in the dependency chain, not all of
your tests need to depend on it. Below are a few cases where I tend not to use any mocking
library and instead leverage the constructs that Go gives us.</p>
<p>This does require some extra song and dance with the language, but in return, we gain more
control over our tests and reduce the chance of encountering <a href="https://en.wikipedia.org/wiki/Action_at_a_distance_(computer_programming)" rel="noopener noreferrer" target="_blank">spooky action at a distance</a>.</p>
<h2 id="mocking-a-function">Mocking a function</h2>
<p>Say you have a function that creates a database handle:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">OpenDB</span><span class="p">(</span><span class="nx">user</span><span class="p">,</span><span class="w"> </span><span class="nx">pass</span><span class="p">,</span><span class="w"> </span><span class="nx">host</span><span class="p">,</span><span class="w"> </span><span class="nx">dbName</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="o">*</span><span class="nx">sql</span><span class="p">.</span><span class="nx">DB</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">dsn</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Sprintf</span><span class="p">(</span><span class="s">&#34;%s:%s@tcp(%s)/%s&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">user</span><span class="p">,</span><span class="w"> </span><span class="nx">pass</span><span class="p">,</span><span class="w"> </span><span class="nx">host</span><span class="p">,</span><span class="w"> </span><span class="nx">dbName</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">sql</span><span class="p">.</span><span class="nf">Open</span><span class="p">(</span><span class="s">&#34;mysql&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">dsn</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>The problem is that <code>sql.Open</code> hands the DSN directly to the driver. When you call
<code>OpenDB(&quot;admin&quot;, &quot;secret&quot;, &quot;db.internal&quot;, &quot;orders&quot;)</code>, the function formats the DSN string
and hands it to the MySQL driver. You can&rsquo;t intercept that call, you can&rsquo;t control what it
returns, and you probably don&rsquo;t want unit tests leaning on a real driver (or a real MySQL
instance) just to verify DSN formatting.</p>
<p>The fix is to make the database opener injectable:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">SQLOpenFunc</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="nx">driver</span><span class="p">,</span><span class="w"> </span><span class="nx">dsn</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="o">*</span><span class="nx">sql</span><span class="p">.</span><span class="nx">DB</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w">  </span><span class="c1">// (1)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">OpenDB</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">user</span><span class="p">,</span><span class="w"> </span><span class="nx">pass</span><span class="p">,</span><span class="w"> </span><span class="nx">host</span><span class="p">,</span><span class="w"> </span><span class="nx">dbName</span><span class="w"> </span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="nx">openFn</span><span class="w"> </span><span class="nx">SQLOpenFunc</span><span class="p">,</span><span class="w"> </span><span class="c1">// (2)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="o">*</span><span class="nx">sql</span><span class="p">.</span><span class="nx">DB</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">dsn</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Sprintf</span><span class="p">(</span><span class="s">&#34;%s:%s@tcp(%s)/%s&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">user</span><span class="p">,</span><span class="w"> </span><span class="nx">pass</span><span class="p">,</span><span class="w"> </span><span class="nx">host</span><span class="p">,</span><span class="w"> </span><span class="nx">dbName</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nf">openFn</span><span class="p">(</span><span class="s">&#34;mysql&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">dsn</span><span class="p">)</span><span class="w">  </span><span class="c1">// (3)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Here:</p>
<ul>
<li>(1) defines a function type that matches <code>sql.Open</code>&rsquo;s signature</li>
<li>(2) accepts an opener function as a parameter</li>
<li>(3) delegates to that function instead of calling <code>sql.Open</code> directly</li>
</ul>
<p>In production, pass the real <code>sql.Open</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">db</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">OpenDB</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="s">&#34;admin&#34;</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;secret&#34;</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;db.internal&#34;</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;orders&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">sql</span><span class="p">.</span><span class="nx">Open</span><span class="p">,</span><span class="w">  </span><span class="c1">// (1)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// ...</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Here:</p>
<ul>
<li>(1) the real <code>sql.Open</code> is passed as the last argument - no wrapper needed</li>
</ul>
<p>In tests, pass a fake that captures what was passed or returns canned values:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">TestOpenDB</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">var</span><span class="w"> </span><span class="nx">got</span><span class="w"> </span><span class="kt">string</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">fakeOpen</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="nx">driver</span><span class="p">,</span><span class="w"> </span><span class="nx">dsn</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="o">*</span><span class="nx">sql</span><span class="p">.</span><span class="nx">DB</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">got</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">dsn</span><span class="w">  </span><span class="c1">// (1) capture what was passed</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">OpenDB</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="s">&#34;admin&#34;</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;secret&#34;</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;db.internal&#34;</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;orders&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">fakeOpen</span><span class="p">,</span><span class="w">  </span><span class="c1">// (2)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">want</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="s">&#34;admin:secret@tcp(db.internal)/orders&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">got</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="nx">want</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">t</span><span class="p">.</span><span class="nf">Errorf</span><span class="p">(</span><span class="s">&#34;got %q, want %q&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">got</span><span class="p">,</span><span class="w"> </span><span class="nx">want</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Here:</p>
<ul>
<li>(1) the fake captures the DSN for later assertion</li>
<li>(2) the call site looks the same, just with a different opener</li>
</ul>
<p>This pattern works for any function dependency - UUID generators, random number sources,
file openers. Functions are first-class values in Go, so you can pass them around like any
other value.</p>
<p>The downside is that parameter lists can grow quickly. If <code>OpenDB</code> also needed a logger, a
metrics client, and a config loader, the signature becomes unwieldy. When you find yourself
passing more than two or three function dependencies, consider grouping them into a struct
with an interface - see <a href="#mocking-a-method-on-a-type">Mocking a method on a type</a>.</p>
<h2 id="monkey-patching">Monkey patching</h2>
<p>Sometimes you inherit code where refactoring the function signature isn&rsquo;t practical. Maybe
it&rsquo;s called from dozens of places, or it&rsquo;s part of a public API you can&rsquo;t change:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">PublishOrderCreated</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">ctx</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w"> </span><span class="nx">brokers</span><span class="w"> </span><span class="p">[]</span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="nx">id</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="kt">error</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">w</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">kafka</span><span class="p">.</span><span class="nx">Writer</span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">Addr</span><span class="p">:</span><span class="w"> </span><span class="nx">kafka</span><span class="p">.</span><span class="nf">TCP</span><span class="p">(</span><span class="nx">brokers</span><span class="o">...</span><span class="p">),</span><span class="w"> </span><span class="nx">Topic</span><span class="p">:</span><span class="w"> </span><span class="s">&#34;order-events&#34;</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">defer</span><span class="w"> </span><span class="nx">w</span><span class="p">.</span><span class="nf">Close</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">w</span><span class="p">.</span><span class="nf">WriteMessages</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span><span class="w"> </span><span class="nx">kafka</span><span class="p">.</span><span class="nx">Message</span><span class="p">{</span><span class="nx">Key</span><span class="p">:</span><span class="w"> </span><span class="p">[]</span><span class="nb">byte</span><span class="p">(</span><span class="nx">id</span><span class="p">)})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>The Kafka writer is instantiated directly inside the function. There&rsquo;s no seam to inject a
fake without touching every call site. If this function is called from 50 places in your
codebase, changing its signature means updating all 50.</p>
<p>One workaround is a package-level variable that points to the constructor:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">kafkaWriter</span><span class="w"> </span><span class="kd">interface</span><span class="w"> </span><span class="p">{</span><span class="w">  </span><span class="c1">// (1)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">WriteMessages</span><span class="p">(</span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w"> </span><span class="o">...</span><span class="nx">kafka</span><span class="p">.</span><span class="nx">Message</span><span class="p">)</span><span class="w"> </span><span class="kt">error</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">Close</span><span class="p">()</span><span class="w"> </span><span class="kt">error</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">var</span><span class="w"> </span><span class="nx">newWriter</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="nx">brokers</span><span class="w"> </span><span class="p">[]</span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="nx">kafkaWriter</span><span class="w"> </span><span class="p">{</span><span class="w">  </span><span class="c1">// (2)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">kafka</span><span class="p">.</span><span class="nx">Writer</span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">Addr</span><span class="p">:</span><span class="w"> </span><span class="nx">kafka</span><span class="p">.</span><span class="nf">TCP</span><span class="p">(</span><span class="nx">brokers</span><span class="o">...</span><span class="p">),</span><span class="w"> </span><span class="nx">Topic</span><span class="p">:</span><span class="w"> </span><span class="s">&#34;order-events&#34;</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">PublishOrderCreated</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">ctx</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w"> </span><span class="nx">brokers</span><span class="w"> </span><span class="p">[]</span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="nx">id</span><span class="w"> </span><span class="kt">string</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">)</span><span class="w"> </span><span class="kt">error</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">w</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">newWriter</span><span class="p">(</span><span class="nx">brokers</span><span class="p">)</span><span class="w">  </span><span class="c1">// (3)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">defer</span><span class="w"> </span><span class="nx">w</span><span class="p">.</span><span class="nf">Close</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">w</span><span class="p">.</span><span class="nf">WriteMessages</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span><span class="w"> </span><span class="nx">kafka</span><span class="p">.</span><span class="nx">Message</span><span class="p">{</span><span class="nx">Key</span><span class="p">:</span><span class="w"> </span><span class="p">[]</span><span class="nb">byte</span><span class="p">(</span><span class="nx">id</span><span class="p">)})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Here:</p>
<ul>
<li>(1) define an interface with only the methods we need from <code>kafka.Writer</code></li>
<li>(2) the package variable returns the interface type, not the concrete type</li>
<li>(3) the function calls it instead of instantiating directly</li>
</ul>
<p>Production code doesn&rsquo;t change - it calls <code>PublishOrderCreated</code> exactly as before, and the
default <code>newWriter</code> creates real Kafka writers.</p>
<p>Tests swap it out:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">fakeWriter</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">key</span><span class="w"> </span><span class="p">[]</span><span class="kt">byte</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">f</span><span class="w"> </span><span class="o">*</span><span class="nx">fakeWriter</span><span class="p">)</span><span class="w"> </span><span class="nf">WriteMessages</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">_</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w"> </span><span class="nx">msgs</span><span class="w"> </span><span class="o">...</span><span class="nx">kafka</span><span class="p">.</span><span class="nx">Message</span><span class="p">)</span><span class="w"> </span><span class="kt">error</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nb">len</span><span class="p">(</span><span class="nx">msgs</span><span class="p">)</span><span class="w"> </span><span class="p">&gt;</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">f</span><span class="p">.</span><span class="nx">key</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">msgs</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="nx">Key</span><span class="w">  </span><span class="c1">// (1)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">f</span><span class="w"> </span><span class="o">*</span><span class="nx">fakeWriter</span><span class="p">)</span><span class="w"> </span><span class="nf">Close</span><span class="p">()</span><span class="w"> </span><span class="kt">error</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">TestPublishOrderCreated</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">orig</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">newWriter</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">t</span><span class="p">.</span><span class="nf">Cleanup</span><span class="p">(</span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">newWriter</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">orig</span><span class="w"> </span><span class="p">})</span><span class="w">  </span><span class="c1">// (2) restore after test</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">fake</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">fakeWriter</span><span class="p">{}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">newWriter</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="kd">func</span><span class="p">([]</span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="nx">kafkaWriter</span><span class="w"> </span><span class="p">{</span><span class="w">  </span><span class="c1">// (3)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">return</span><span class="w"> </span><span class="nx">fake</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">PublishOrderCreated</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">t</span><span class="p">.</span><span class="nf">Context</span><span class="p">(),</span><span class="w"> </span><span class="p">[]</span><span class="kt">string</span><span class="p">{</span><span class="s">&#34;kafka:9092&#34;</span><span class="p">},</span><span class="w"> </span><span class="s">&#34;ord-1&#34;</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">got</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nb">string</span><span class="p">(</span><span class="nx">fake</span><span class="p">.</span><span class="nx">key</span><span class="p">);</span><span class="w"> </span><span class="nx">got</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="s">&#34;ord-1&#34;</span><span class="w"> </span><span class="p">{</span><span class="w">  </span><span class="c1">// (4)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">t</span><span class="p">.</span><span class="nf">Errorf</span><span class="p">(</span><span class="s">&#34;got %q, want %q&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">got</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;ord-1&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Here:</p>
<ul>
<li>(1) the fake captures the message key for later assertion</li>
<li>(2) <code>t.Cleanup</code> ensures the original is restored even if the test fails</li>
<li>(3) the replacement factory returns the fake - note it returns <code>kafkaWriter</code>, matching the
variable&rsquo;s type</li>
<li>(4) assert the captured key matches the expected value</li>
</ul>
<p>This works, but be aware of the costs. Tests that mutate package state can&rsquo;t run in
parallel - they&rsquo;d stomp on each other&rsquo;s fakes. If you&rsquo;re writing tests from an external
package (<code>package events_test</code>), the variable must be exported, which pollutes your public
API.</p>
<p>Prefer the <a href="#mocking-a-function">function parameter pattern</a> or the <a href="#mocking-a-method-on-a-type">interface pattern</a> over monkey patching.
Reserve this technique for legacy code where changing signatures would be too disruptive.</p>
<h2 id="mocking-a-method-on-a-type">Mocking a method on a type</h2>
<p>This is a pattern you&rsquo;ll see all the time in services that integrate with third-party APIs.
Here&rsquo;s a payment service that charges customers through Stripe (this uses the newer
<code>stripe.Client</code> API, which is the recommended shape in recent stripe-go versions):</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">s</span><span class="w"> </span><span class="o">*</span><span class="nx">Service</span><span class="p">)</span><span class="w"> </span><span class="nf">ChargeCustomer</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">ctx</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w"> </span><span class="nx">custID</span><span class="w"> </span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="nx">cents</span><span class="w"> </span><span class="kt">int64</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">intent</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">s</span><span class="p">.</span><span class="nx">client</span><span class="p">.</span><span class="nx">V1PaymentIntents</span><span class="p">.</span><span class="nf">Create</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="o">&amp;</span><span class="nx">stripe</span><span class="p">.</span><span class="nx">PaymentIntentCreateParams</span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">Amount</span><span class="p">:</span><span class="w">   </span><span class="nx">stripe</span><span class="p">.</span><span class="nf">Int64</span><span class="p">(</span><span class="nx">cents</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">Currency</span><span class="p">:</span><span class="w"> </span><span class="nx">stripe</span><span class="p">.</span><span class="nf">String</span><span class="p">(</span><span class="s">&#34;usd&#34;</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">Customer</span><span class="p">:</span><span class="w"> </span><span class="nx">stripe</span><span class="p">.</span><span class="nf">String</span><span class="p">(</span><span class="nx">custID</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">},</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">return</span><span class="w"> </span><span class="s">&#34;&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">intent</span><span class="p">.</span><span class="nx">ID</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Testing this hits the real Stripe API. That&rsquo;s slow, requires live credentials, and in
production mode charges actual money. The problem is that <code>s.client</code> is a <code>*stripe.Client</code>
from the SDK - there&rsquo;s no way to swap it for a fake without introducing a seam.</p>
<p>The solution is to introduce an interface that describes what you need:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">PaymentIntentCreator</span><span class="w"> </span><span class="kd">interface</span><span class="w"> </span><span class="p">{</span><span class="w">  </span><span class="c1">// (1)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">Create</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w"> </span><span class="o">*</span><span class="nx">stripe</span><span class="p">.</span><span class="nx">PaymentIntentCreateParams</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="o">*</span><span class="nx">stripe</span><span class="p">.</span><span class="nx">PaymentIntent</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">Service</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">intents</span><span class="w"> </span><span class="nx">PaymentIntentCreator</span><span class="w">  </span><span class="c1">// (2)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">s</span><span class="w"> </span><span class="o">*</span><span class="nx">Service</span><span class="p">)</span><span class="w"> </span><span class="nf">ChargeCustomer</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">ctx</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w"> </span><span class="nx">custID</span><span class="w"> </span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="nx">cents</span><span class="w"> </span><span class="kt">int64</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">intent</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">s</span><span class="p">.</span><span class="nx">intents</span><span class="p">.</span><span class="nf">Create</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span><span class="w">  </span><span class="c1">// (3)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="o">&amp;</span><span class="nx">stripe</span><span class="p">.</span><span class="nx">PaymentIntentCreateParams</span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">Amount</span><span class="p">:</span><span class="w">   </span><span class="nx">stripe</span><span class="p">.</span><span class="nf">Int64</span><span class="p">(</span><span class="nx">cents</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">Currency</span><span class="p">:</span><span class="w"> </span><span class="nx">stripe</span><span class="p">.</span><span class="nf">String</span><span class="p">(</span><span class="s">&#34;usd&#34;</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">Customer</span><span class="p">:</span><span class="w"> </span><span class="nx">stripe</span><span class="p">.</span><span class="nf">String</span><span class="p">(</span><span class="nx">custID</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">return</span><span class="w"> </span><span class="s">&#34;&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">intent</span><span class="p">.</span><span class="nx">ID</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Here:</p>
<ul>
<li>(1) the interface has one method matching what we need from the SDK</li>
<li>(2) the service holds the dependency as a field</li>
<li>(3) calls through the interface instead of the client directly</li>
</ul>
<p>In production, inject the real Stripe service client:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">sc</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">stripe</span><span class="p">.</span><span class="nf">NewClient</span><span class="p">(</span><span class="s">&#34;sk_test_...&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">svc</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">Service</span><span class="p">{</span><span class="nx">intents</span><span class="p">:</span><span class="w"> </span><span class="nx">sc</span><span class="p">.</span><span class="nx">V1PaymentIntents</span><span class="p">}</span><span class="w">  </span><span class="c1">// (1)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// ...</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Here:</p>
<ul>
<li>(1) <code>sc.V1PaymentIntents</code> satisfies <code>PaymentIntentCreator</code> (it has a <code>Create</code> method with
the right signature)</li>
</ul>
<p>In tests, you pass a fake that returns canned values:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">fakeIntents</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">id</span><span class="w"> </span><span class="kt">string</span><span class="w">  </span><span class="c1">// (1)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">f</span><span class="w"> </span><span class="o">*</span><span class="nx">fakeIntents</span><span class="p">)</span><span class="w"> </span><span class="nf">Create</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w"> </span><span class="o">*</span><span class="nx">stripe</span><span class="p">.</span><span class="nx">PaymentIntentCreateParams</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="o">*</span><span class="nx">stripe</span><span class="p">.</span><span class="nx">PaymentIntent</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">stripe</span><span class="p">.</span><span class="nx">PaymentIntent</span><span class="p">{</span><span class="nx">ID</span><span class="p">:</span><span class="w"> </span><span class="nx">f</span><span class="p">.</span><span class="nx">id</span><span class="p">},</span><span class="w"> </span><span class="kc">nil</span><span class="w">  </span><span class="c1">// (2)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">TestChargeCustomer</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">fake</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">fakeIntents</span><span class="p">{</span><span class="nx">id</span><span class="p">:</span><span class="w"> </span><span class="s">&#34;pi_123&#34;</span><span class="p">}</span><span class="w">  </span><span class="c1">// (3)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">svc</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">Service</span><span class="p">{</span><span class="nx">intents</span><span class="p">:</span><span class="w"> </span><span class="nx">fake</span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">id</span><span class="p">,</span><span class="w"> </span><span class="nx">_</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">svc</span><span class="p">.</span><span class="nf">ChargeCustomer</span><span class="p">(</span><span class="nx">t</span><span class="p">.</span><span class="nf">Context</span><span class="p">(),</span><span class="w"> </span><span class="s">&#34;cus_abc&#34;</span><span class="p">,</span><span class="w"> </span><span class="mi">5000</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// assert id == &#34;pi_123&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Here:</p>
<ul>
<li>(1) the fake struct holds the canned return value</li>
<li>(2) returns whatever you configured instead of calling Stripe</li>
<li>(3) configure the fake with the expected payment intent ID</li>
</ul>
<p>The service doesn&rsquo;t know or care whether it&rsquo;s talking to Stripe or a test fake. This is the
most common mocking pattern in Go - define an interface for your dependency, accept it in
your constructor, and swap implementations at runtime.</p>
<p>But what happens when the SDK surface area is huge and your code only needs one operation?
That&rsquo;s where the next pattern comes in.</p>
<h2 id="consumer-side-interface-segregation">Consumer-side interface segregation</h2>
<p>The previous pattern works well when you control the interface. But AWS SDK clients have
dozens of methods. The DynamoDB client has over 40 operations - <code>GetItem</code>, <code>PutItem</code>,
<code>Query</code>, <code>Scan</code>, <code>BatchGetItem</code>, and so on. If you write tests against a dependency that
exposes the whole surface area, your fakes become annoying fast.</p>
<p>The solution is to define a minimal interface on the consumer side:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">itemGetter</span><span class="w"> </span><span class="kd">interface</span><span class="w"> </span><span class="p">{</span><span class="w">  </span><span class="c1">// (1)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">GetItem</span><span class="p">(</span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w"> </span><span class="o">*</span><span class="nx">dynamodb</span><span class="p">.</span><span class="nx">GetItemInput</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="o">...</span><span class="kd">func</span><span class="p">(</span><span class="o">*</span><span class="nx">dynamodb</span><span class="p">.</span><span class="nx">Options</span><span class="p">))</span><span class="w"> </span><span class="p">(</span><span class="o">*</span><span class="nx">dynamodb</span><span class="p">.</span><span class="nx">GetItemOutput</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">GetUserByID</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">ctx</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w"> </span><span class="nx">client</span><span class="w"> </span><span class="nx">itemGetter</span><span class="p">,</span><span class="w"> </span><span class="nx">id</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="o">*</span><span class="nx">User</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">out</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">client</span><span class="p">.</span><span class="nf">GetItem</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">dynamodb</span><span class="p">.</span><span class="nx">GetItemInput</span><span class="p">{</span><span class="w">  </span><span class="c1">// (2)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">TableName</span><span class="p">:</span><span class="w"> </span><span class="nx">aws</span><span class="p">.</span><span class="nf">String</span><span class="p">(</span><span class="s">&#34;users&#34;</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">Key</span><span class="p">:</span><span class="w"> </span><span class="kd">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="nx">types</span><span class="p">.</span><span class="nx">AttributeValue</span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="s">&#34;pk&#34;</span><span class="p">:</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">types</span><span class="p">.</span><span class="nx">AttributeValueMemberS</span><span class="p">{</span><span class="nx">Value</span><span class="p">:</span><span class="w"> </span><span class="nx">id</span><span class="p">},</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">},</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// ...</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Here:</p>
<ul>
<li>(1) the interface has exactly one method - just what this function needs</li>
<li>(2) accept the minimal interface and call through it</li>
</ul>
<p>In production, pass the real DynamoDB client - it satisfies <code>itemGetter</code> because it has a
<code>GetItem</code> method. Go interfaces are satisfied implicitly:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">client</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">dynamodb</span><span class="p">.</span><span class="nf">NewFromConfig</span><span class="p">(</span><span class="nx">cfg</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">user</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">GetUserByID</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span><span class="w"> </span><span class="nx">client</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;user-123&#34;</span><span class="p">)</span><span class="w">  </span><span class="c1">// (1)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// ...</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Here:</p>
<ul>
<li>(1) the real client satisfies <code>itemGetter</code> automatically - no adapter or wrapper needed
thanks to implicit interface satisfaction</li>
</ul>
<p>In tests, you only implement the one method you need:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">fakeItemGetter</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">item</span><span class="w"> </span><span class="kd">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="nx">types</span><span class="p">.</span><span class="nx">AttributeValue</span><span class="w">  </span><span class="c1">// (1)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">f</span><span class="w"> </span><span class="o">*</span><span class="nx">fakeItemGetter</span><span class="p">)</span><span class="w"> </span><span class="nf">GetItem</span><span class="p">(</span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w"> </span><span class="o">*</span><span class="nx">dynamodb</span><span class="p">.</span><span class="nx">GetItemInput</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="o">...</span><span class="kd">func</span><span class="p">(</span><span class="o">*</span><span class="nx">dynamodb</span><span class="p">.</span><span class="nx">Options</span><span class="p">))</span><span class="w"> </span><span class="p">(</span><span class="o">*</span><span class="nx">dynamodb</span><span class="p">.</span><span class="nx">GetItemOutput</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">dynamodb</span><span class="p">.</span><span class="nx">GetItemOutput</span><span class="p">{</span><span class="nx">Item</span><span class="p">:</span><span class="w"> </span><span class="nx">f</span><span class="p">.</span><span class="nx">item</span><span class="p">},</span><span class="w"> </span><span class="kc">nil</span><span class="w">  </span><span class="c1">// (2)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">TestGetUserByID</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">fake</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">fakeItemGetter</span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">item</span><span class="p">:</span><span class="w"> </span><span class="kd">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="nx">types</span><span class="p">.</span><span class="nx">AttributeValue</span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="s">&#34;email&#34;</span><span class="p">:</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">types</span><span class="p">.</span><span class="nx">AttributeValueMemberS</span><span class="p">{</span><span class="nx">Value</span><span class="p">:</span><span class="w"> </span><span class="s">&#34;a@b.com&#34;</span><span class="p">},</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">},</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">user</span><span class="p">,</span><span class="w"> </span><span class="nx">_</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">GetUserByID</span><span class="p">(</span><span class="nx">t</span><span class="p">.</span><span class="nf">Context</span><span class="p">(),</span><span class="w"> </span><span class="nx">fake</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;u-1&#34;</span><span class="p">)</span><span class="w">  </span><span class="c1">// (3)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// assert user.Email == &#34;a@b.com&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Here:</p>
<ul>
<li>(1) the fake struct holds the canned response data</li>
<li>(2) returns the configured item - no network call</li>
<li>(3) pass the fake to the function under test</li>
</ul>
<p>This is the <a href="/go/interface-segregation/">Interface Segregation Principle</a> in action - clients shouldn&rsquo;t be forced to
depend on methods they don&rsquo;t use.</p>
<p>But this approach has limits. If you have 20 functions each using different DynamoDB
operations, you&rsquo;d end up with 20 tiny interfaces. And sometimes you&rsquo;re stuck with a
preexisting interface type that has more methods than you want. That&rsquo;s where struct
embedding helps.</p>
<h2 id="struct-embedding-for-partial-implementation">Struct embedding for partial implementation</h2>
<p>Sometimes you can&rsquo;t define your own minimal interface. Maybe a library insists on a specific
interface type, and it&rsquo;s bigger than what your test cares about.</p>
<p>The AWS SDK v2&rsquo;s S3 upload manager is a good example. <code>manager.NewUploader</code> takes a client
interface that supports both single-part uploads and multipart uploads. If your test is
exercising the single-part path and you only want to intercept <code>PutObject</code>, implementing the
multipart methods just to satisfy the interface is pure busywork.</p>
<p>Go&rsquo;s struct embedding provides an escape hatch. Here&rsquo;s the production code:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">UploadReport</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">ctx</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w"> </span><span class="nx">client</span><span class="w"> </span><span class="nx">manager</span><span class="p">.</span><span class="nx">UploadAPIClient</span><span class="p">,</span><span class="w">  </span><span class="c1">// (1)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">bucket</span><span class="p">,</span><span class="w"> </span><span class="nx">key</span><span class="w"> </span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="nx">body</span><span class="w"> </span><span class="nx">io</span><span class="p">.</span><span class="nx">Reader</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">)</span><span class="w"> </span><span class="kt">error</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">up</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">manager</span><span class="p">.</span><span class="nf">NewUploader</span><span class="p">(</span><span class="nx">client</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">up</span><span class="p">.</span><span class="nf">Upload</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">s3</span><span class="p">.</span><span class="nx">PutObjectInput</span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">Bucket</span><span class="p">:</span><span class="w"> </span><span class="nx">aws</span><span class="p">.</span><span class="nf">String</span><span class="p">(</span><span class="nx">bucket</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">Key</span><span class="p">:</span><span class="w">    </span><span class="nx">aws</span><span class="p">.</span><span class="nf">String</span><span class="p">(</span><span class="nx">key</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">Body</span><span class="p">:</span><span class="w">   </span><span class="nx">body</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">err</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Here:</p>
<ul>
<li>(1) accepts the SDK&rsquo;s <code>UploadAPIClient</code> interface - a large interface with many methods</li>
</ul>
<p>In tests, embed the interface in your fake and override only what you need:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">fakeS3</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">manager</span><span class="p">.</span><span class="nx">UploadAPIClient</span><span class="w">  </span><span class="c1">// (1)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">gotKey</span><span class="w">  </span><span class="kt">string</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">gotBody</span><span class="w"> </span><span class="p">[]</span><span class="kt">byte</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">f</span><span class="w"> </span><span class="o">*</span><span class="nx">fakeS3</span><span class="p">)</span><span class="w"> </span><span class="nf">PutObject</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">_</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w"> </span><span class="nx">in</span><span class="w"> </span><span class="o">*</span><span class="nx">s3</span><span class="p">.</span><span class="nx">PutObjectInput</span><span class="p">,</span><span class="w"> </span><span class="nx">_</span><span class="w"> </span><span class="o">...</span><span class="kd">func</span><span class="p">(</span><span class="o">*</span><span class="nx">s3</span><span class="p">.</span><span class="nx">Options</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="o">*</span><span class="nx">s3</span><span class="p">.</span><span class="nx">PutObjectOutput</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">in</span><span class="p">.</span><span class="nx">Key</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">f</span><span class="p">.</span><span class="nx">gotKey</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="o">*</span><span class="nx">in</span><span class="p">.</span><span class="nx">Key</span><span class="w">  </span><span class="c1">// (2)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">in</span><span class="p">.</span><span class="nx">Body</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">f</span><span class="p">.</span><span class="nx">gotBody</span><span class="p">,</span><span class="w"> </span><span class="nx">_</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">io</span><span class="p">.</span><span class="nf">ReadAll</span><span class="p">(</span><span class="nx">in</span><span class="p">.</span><span class="nx">Body</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">s3</span><span class="p">.</span><span class="nx">PutObjectOutput</span><span class="p">{},</span><span class="w"> </span><span class="kc">nil</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">TestUploadReport</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">fake</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">fakeS3</span><span class="p">{}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">UploadReport</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">t</span><span class="p">.</span><span class="nf">Context</span><span class="p">(),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">fake</span><span class="p">,</span><span class="w">  </span><span class="c1">// (3)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="s">&#34;my-bucket&#34;</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="s">&#34;reports/q1.csv&#34;</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">bytes</span><span class="p">.</span><span class="nf">NewReader</span><span class="p">([]</span><span class="nb">byte</span><span class="p">(</span><span class="s">&#34;hi&#34;</span><span class="p">)),</span><span class="w">  </span><span class="c1">// (4)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">t</span><span class="p">.</span><span class="nf">Fatal</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">fake</span><span class="p">.</span><span class="nx">gotKey</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="s">&#34;reports/q1.csv&#34;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">t</span><span class="p">.</span><span class="nf">Errorf</span><span class="p">(</span><span class="s">&#34;got %q, want %q&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">fake</span><span class="p">.</span><span class="nx">gotKey</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;reports/q1.csv&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Here:</p>
<ul>
<li>(1) embedding the interface satisfies the full interface at compile time</li>
<li>(2) capture what you care about - only implement what this test needs</li>
<li>(3) pass the fake to code that expects the full <code>UploadAPIClient</code> interface</li>
<li>(4) use a small body so the upload manager takes the single <code>PutObject</code> path</li>
</ul>
<p>The embedded interface value is <code>nil</code>, so any method you don&rsquo;t override will panic if
called. This is a feature, not a bug. If your code accidentally triggers multipart and calls
<code>CreateMultipartUpload</code>, the test crashes immediately, and you learn that your test setup
(or your assumptions) are wrong.</p>
<h2 id="function-type-as-interface">Function type as interface</h2>
<p>For interfaces with a single method, there&rsquo;s an even more compact approach. Say you have
middleware that validates authentication tokens:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">ctxKey</span><span class="w"> </span><span class="kt">string</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">const</span><span class="w"> </span><span class="nx">userIDKey</span><span class="w"> </span><span class="nx">ctxKey</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="s">&#34;userID&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">TokenValidator</span><span class="w"> </span><span class="kd">interface</span><span class="w"> </span><span class="p">{</span><span class="w">  </span><span class="c1">// (1)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">Validate</span><span class="p">(</span><span class="nx">token</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nx">userID</span><span class="w"> </span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">RequireAuth</span><span class="p">(</span><span class="nx">v</span><span class="w"> </span><span class="nx">TokenValidator</span><span class="p">,</span><span class="w"> </span><span class="nx">next</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">Handler</span><span class="p">)</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">Handler</span><span class="w"> </span><span class="p">{</span><span class="w">  </span><span class="c1">// (2)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">fn</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">userID</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">v</span><span class="p">.</span><span class="nf">Validate</span><span class="p">(</span><span class="nx">r</span><span class="p">.</span><span class="nx">Header</span><span class="p">.</span><span class="nf">Get</span><span class="p">(</span><span class="s">&#34;Authorization&#34;</span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">http</span><span class="p">.</span><span class="nf">Error</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;unauthorized&#34;</span><span class="p">,</span><span class="w"> </span><span class="mi">401</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="k">return</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">ctx</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nf">WithValue</span><span class="p">(</span><span class="nx">r</span><span class="p">.</span><span class="nf">Context</span><span class="p">(),</span><span class="w"> </span><span class="nx">userIDKey</span><span class="p">,</span><span class="w"> </span><span class="nx">userID</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">next</span><span class="p">.</span><span class="nf">ServeHTTP</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="p">.</span><span class="nf">WithContext</span><span class="p">(</span><span class="nx">ctx</span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nf">HandlerFunc</span><span class="p">(</span><span class="nx">fn</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Here:</p>
<ul>
<li>(1) a single-method interface - the perfect candidate for a function type adapter</li>
<li>(2) the middleware accepts the interface as a dependency</li>
</ul>
<p>You could write a fake struct with a <code>Validate</code> method, but Go lets you define a function
type that satisfies the interface:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">TokenValidatorFunc</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w">  </span><span class="c1">// (1)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">f</span><span class="w"> </span><span class="nx">TokenValidatorFunc</span><span class="p">)</span><span class="w"> </span><span class="nf">Validate</span><span class="p">(</span><span class="nx">token</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nf">f</span><span class="p">(</span><span class="nx">token</span><span class="p">)</span><span class="w">  </span><span class="c1">// (2)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Here:</p>
<ul>
<li>(1) define a function type with the right signature</li>
<li>(2) add a method that just calls the function itself</li>
</ul>
<p>This is the same pattern the standard library uses with <code>http.HandlerFunc</code>. Now tests can
pass inline functions:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">TestRequireAuth</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">v</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">TokenValidatorFunc</span><span class="p">(</span><span class="kd">func</span><span class="p">(</span><span class="nx">token</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">if</span><span class="w"> </span><span class="nx">token</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="s">&#34;Bearer valid&#34;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="k">return</span><span class="w"> </span><span class="s">&#34;user-123&#34;</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="w">  </span><span class="c1">// (1)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">return</span><span class="w"> </span><span class="s">&#34;&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">errors</span><span class="p">.</span><span class="nf">New</span><span class="p">(</span><span class="s">&#34;invalid&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">next</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nf">HandlerFunc</span><span class="p">(</span><span class="kd">func</span><span class="p">(</span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">handler</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">RequireAuth</span><span class="p">(</span><span class="nx">v</span><span class="p">,</span><span class="w"> </span><span class="nx">next</span><span class="p">)</span><span class="w">  </span><span class="c1">// (2)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">req</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">httptest</span><span class="p">.</span><span class="nf">NewRequest</span><span class="p">(</span><span class="s">&#34;GET&#34;</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;/protected&#34;</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">req</span><span class="p">.</span><span class="nx">Header</span><span class="p">.</span><span class="nf">Set</span><span class="p">(</span><span class="s">&#34;Authorization&#34;</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;Bearer valid&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">rec</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">httptest</span><span class="p">.</span><span class="nf">NewRecorder</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">handler</span><span class="p">.</span><span class="nf">ServeHTTP</span><span class="p">(</span><span class="nx">rec</span><span class="p">,</span><span class="w"> </span><span class="nx">req</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// assert rec.Code == http.StatusOK</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Here:</p>
<ul>
<li>(1) return a known user ID for a valid token</li>
<li>(2) the middleware accepts it as a <code>TokenValidator</code> interface</li>
</ul>
<p>No extra struct definitions cluttering up your test file.</p>
<h2 id="mocking-http-calls">Mocking HTTP calls</h2>
<p>When your code makes HTTP requests to external services, the <code>net/http/httptest</code> package
provides a test server that runs on localhost. Say you have a client that fetches exchange
rates:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">c</span><span class="w"> </span><span class="o">*</span><span class="nx">Client</span><span class="p">)</span><span class="w"> </span><span class="nf">GetRate</span><span class="p">(</span><span class="nx">from</span><span class="p">,</span><span class="w"> </span><span class="nx">to</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">float64</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">url</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">c</span><span class="p">.</span><span class="nx">baseURL</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="s">&#34;/latest?base=&#34;</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nx">from</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="s">&#34;&amp;symbols=&#34;</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nx">to</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">resp</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">c</span><span class="p">.</span><span class="nx">httpClient</span><span class="p">.</span><span class="nf">Get</span><span class="p">(</span><span class="nx">url</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">return</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">defer</span><span class="w"> </span><span class="nx">resp</span><span class="p">.</span><span class="nx">Body</span><span class="p">.</span><span class="nf">Close</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// decode JSON, return rate...</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>In production, <code>c.baseURL</code> points to the real API. Testing against it is problematic - it&rsquo;s
slow, requires credentials, returns different values each time, and might rate-limit your
CI.</p>
<p>The <code>httptest.Server</code> spins up a real HTTP server on localhost:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">TestGetRate</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">srv</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">httptest</span><span class="p">.</span><span class="nf">NewServer</span><span class="p">(</span><span class="nx">http</span><span class="p">.</span><span class="nf">HandlerFunc</span><span class="p">(</span><span class="w">  </span><span class="c1">// (1)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="kd">func</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Fprint</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="s">`{&#34;base&#34;:&#34;USD&#34;,&#34;rates&#34;:{&#34;EUR&#34;:0.92}}`</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">},</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">defer</span><span class="w"> </span><span class="nx">srv</span><span class="p">.</span><span class="nf">Close</span><span class="p">()</span><span class="w">  </span><span class="c1">// (2)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">client</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">NewClient</span><span class="p">(</span><span class="nx">srv</span><span class="p">.</span><span class="nx">URL</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;key&#34;</span><span class="p">)</span><span class="w">  </span><span class="c1">// (3)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">rate</span><span class="p">,</span><span class="w"> </span><span class="nx">_</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">client</span><span class="p">.</span><span class="nf">GetRate</span><span class="p">(</span><span class="s">&#34;USD&#34;</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;EUR&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// assert rate == 0.92</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Here:</p>
<ul>
<li>(1) spin up a local HTTP server with a handler that returns canned JSON</li>
<li>(2) shut down the server when the test finishes</li>
<li>(3) point your client at <code>srv.URL</code> instead of the real API</li>
</ul>
<p>Your code makes real HTTP calls over TCP, but they never leave the machine. You can return
different responses for different scenarios - rate limits, malformed JSON, network errors -
whatever you need to test.</p>
<h2 id="mocking-time">Mocking time</h2>
<p>This is essentially the same technique as <a href="#mocking-a-function">Mocking a function</a> - we&rsquo;re just applying it to
<code>time.Now</code>. Code that depends on the current time is tricky to test:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">IsExpired</span><span class="p">(</span><span class="nx">expiresAt</span><span class="w"> </span><span class="nx">time</span><span class="p">.</span><span class="nx">Time</span><span class="p">)</span><span class="w"> </span><span class="kt">bool</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">time</span><span class="p">.</span><span class="nf">Now</span><span class="p">().</span><span class="nf">After</span><span class="p">(</span><span class="nx">expiresAt</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Every call to <code>time.Now()</code> returns a different value. You can&rsquo;t write a reliable test
because the result depends on when the test runs.</p>
<p>Make the clock injectable:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">Clock</span><span class="w"> </span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="nx">time</span><span class="p">.</span><span class="nx">Time</span><span class="w">  </span><span class="c1">// (1)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">IsExpired</span><span class="p">(</span><span class="nx">expiresAt</span><span class="w"> </span><span class="nx">time</span><span class="p">.</span><span class="nx">Time</span><span class="p">,</span><span class="w"> </span><span class="nx">clock</span><span class="w"> </span><span class="nx">Clock</span><span class="p">)</span><span class="w"> </span><span class="kt">bool</span><span class="w"> </span><span class="p">{</span><span class="w">  </span><span class="c1">// (2)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nf">clock</span><span class="p">().</span><span class="nf">After</span><span class="p">(</span><span class="nx">expiresAt</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Here:</p>
<ul>
<li>(1) define a function type for getting the current time</li>
<li>(2) accept it as a parameter</li>
</ul>
<p>In production, pass <code>time.Now</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">expired</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">IsExpired</span><span class="p">(</span><span class="nx">token</span><span class="p">.</span><span class="nx">ExpiresAt</span><span class="p">,</span><span class="w"> </span><span class="nx">time</span><span class="p">.</span><span class="nx">Now</span><span class="p">)</span><span class="w">  </span><span class="c1">// (1)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// ...</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Here:</p>
<ul>
<li>(1) pass the real <code>time.Now</code> function - it satisfies the <code>Clock</code> type</li>
</ul>
<p>In tests, pass a function that returns a fixed time:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">TestIsExpired</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">expiry</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">time</span><span class="p">.</span><span class="nf">Date</span><span class="p">(</span><span class="mi">2025</span><span class="p">,</span><span class="w"> </span><span class="mi">6</span><span class="p">,</span><span class="w"> </span><span class="mi">15</span><span class="p">,</span><span class="w"> </span><span class="mi">12</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="nx">time</span><span class="p">.</span><span class="nx">UTC</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">before</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="nx">time</span><span class="p">.</span><span class="nx">Time</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">expiry</span><span class="p">.</span><span class="nf">Add</span><span class="p">(</span><span class="o">-</span><span class="nx">time</span><span class="p">.</span><span class="nx">Hour</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w">  </span><span class="c1">// (1)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">after</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="nx">time</span><span class="p">.</span><span class="nx">Time</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">expiry</span><span class="p">.</span><span class="nf">Add</span><span class="p">(</span><span class="nx">time</span><span class="p">.</span><span class="nx">Hour</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w">   </span><span class="c1">// (2)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nf">IsExpired</span><span class="p">(</span><span class="nx">expiry</span><span class="p">,</span><span class="w"> </span><span class="nx">before</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">t</span><span class="p">.</span><span class="nf">Error</span><span class="p">(</span><span class="s">&#34;should not be expired&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="p">!</span><span class="nf">IsExpired</span><span class="p">(</span><span class="nx">expiry</span><span class="p">,</span><span class="w"> </span><span class="nx">after</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">t</span><span class="p">.</span><span class="nf">Error</span><span class="p">(</span><span class="s">&#34;should be expired&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Here:</p>
<ul>
<li>(1) a clock that returns one hour before expiry</li>
<li>(2) a clock that returns one hour after expiry</li>
</ul>
<p>For code that uses <code>time.Sleep</code>, timers, or tickers, Go 1.25&rsquo;s <a href="https://pkg.go.dev/testing/synctest" rel="noopener noreferrer" target="_blank">testing/synctest</a> provides a
fake clock that advances automatically when goroutines in the bubble are <a href="https://pkg.go.dev/testing/synctest#hdr-Blocking" rel="noopener noreferrer" target="_blank">durably blocked</a>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">TestPeriodicFlush</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">synctest</span><span class="p">.</span><span class="nf">Test</span><span class="p">(</span><span class="nx">t</span><span class="p">,</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">  </span><span class="c1">// (1)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">count</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="mi">0</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">go</span><span class="w"> </span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">ticker</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">time</span><span class="p">.</span><span class="nf">NewTicker</span><span class="p">(</span><span class="mi">10</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="nx">time</span><span class="p">.</span><span class="nx">Second</span><span class="p">)</span><span class="w">  </span><span class="c1">// (2)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="k">defer</span><span class="w"> </span><span class="nx">ticker</span><span class="p">.</span><span class="nf">Stop</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="k">for</span><span class="w"> </span><span class="k">range</span><span class="w"> </span><span class="nx">ticker</span><span class="p">.</span><span class="nx">C</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="nx">count</span><span class="o">++</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="k">if</span><span class="w"> </span><span class="nx">count</span><span class="w"> </span><span class="o">&gt;=</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="k">return</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">time</span><span class="p">.</span><span class="nf">Sleep</span><span class="p">(</span><span class="mi">35</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="nx">time</span><span class="p">.</span><span class="nx">Second</span><span class="p">)</span><span class="w">  </span><span class="c1">// (3)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">synctest</span><span class="p">.</span><span class="nf">Wait</span><span class="p">()</span><span class="w">               </span><span class="c1">// (4)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// assert count == 3</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Here:</p>
<ul>
<li>(1) <code>synctest.Test</code> runs the function in an isolated bubble with fake time starting at
2000-01-01</li>
<li>(2) the ticker uses fake time - no real 10-second waits</li>
<li>(3) <code>time.Sleep</code> inside the bubble uses fake time; time advances when goroutines are
durably blocked, so this returns instantly after the ticker fires 3 times</li>
<li>(4) <code>synctest.Wait</code> is a synchronization point; it blocks until the other goroutines in
the bubble are durably blocked or finished</li>
</ul>
<p>Inside <code>synctest.Test</code>, the framework intercepts time operations. The test completes
instantly rather than waiting for real time to pass.</p>
<h2 id="closing-words">Closing words</h2>
<p>These are the most common ones where I typically avoid opting for mocking libraries. But
there are cases when I still like to generate mocks for an interface. One example that comes
to mind is testing gRPC servers. I&rsquo;m sure I&rsquo;m forgetting some other cases where I regularly
use mocking libraries.</p>
<p>The point is not to discourage the use of mocking libraries or to make a general statement
that &ldquo;all mocking libraries are bad.&rdquo; It&rsquo;s that these mocking libraries have costs
associated with them. Code generation is fun, but it&rsquo;s one extra step that you have to teach
someone who&rsquo;s onboarding to your codebase.</p>
<p>Also, if you&rsquo;re using LLMs to generate tests, you may want to write some tests manually to
give the tool a sense of how you want your tests written, so it doesn&rsquo;t pull in the universe
just to mock something that can be mocked natively using Go constructs.</p>
<p>For more on why handwritten fakes often beat generated mocks, see <a href="/go/test-state-not-interactions/">Test state, not
interactions</a>.</p>
<!-- references -->
<!-- prettier-ignore-start -->
<!-- prettier-ignore-end -->
]]></content:encoded>
    </item>
    <item>
      <title>Revisiting interface segregation in Go</title>
      <link>https://rednafi.com/go/interface-segregation/</link>
      <pubDate>Sat, 01 Nov 2025 00:00:00 +0000</pubDate>
      <guid>https://rednafi.com/go/interface-segregation/</guid>
      <description>Apply SOLID&amp;#39;s Interface Segregation Principle in Go with consumer-defined contracts. Learn why small interfaces and implicit implementation matter.</description>
      <category>Go</category>
      <category>API</category>
      <category>Testing</category>
      <content:encoded><![CDATA[<p>Object-oriented (OO) patterns get a lot of flak in the Go community, and often for good
reason.</p>
<p>Still, I&rsquo;ve found that principles like <a href="https://en.wikipedia.org/wiki/SOLID" rel="noopener noreferrer" target="_blank">SOLID</a>, despite their OO origin, can be useful
guides when thinking about design in Go.</p>
<p>Recently, while chatting with a few colleagues new to Go, I noticed that some of them had
spontaneously rediscovered the Interface Segregation Principle (the &ldquo;I&rdquo; in SOLID) without
even realizing it. The benefits were obvious, but without a shared vocabulary, it was harder
to talk about and generalize the idea.</p>
<p>So I wanted to revisit ISP in the context of Go and show how <a href="https://go.dev/doc/effective_go#interfaces_and_types:~:text=Interfaces%20with%20only%20one%20or%20two%20methods%20are%20common%20in%20Go%20code%2C%20and%20are%20usually%20given%20a%20name%20derived%20from%20the%20method%2C%20such%20as%20io.Writer%20for%20something%20that%20implements%20Write" rel="noopener noreferrer" target="_blank">small interfaces</a>, <a href="https://go.dev/tour/methods/10" rel="noopener noreferrer" target="_blank">implicit
implementation</a>, and <a href="https://go.dev/wiki/CodeReviewComments#interfaces:~:text=Go%20interfaces%20generally,requiring%20extensive%20refactoring" rel="noopener noreferrer" target="_blank">consumer-defined contracts</a> make interface segregation feel natural
and lead to code that&rsquo;s easier to test and maintain.</p>
<blockquote>
  <p>Clients should not be forced to depend on methods they do not use.</p>
<p>&ndash; Robert C. Martin (SOLID, interface segregation principle)</p>

</blockquote><p>Or, put simply: your code shouldn&rsquo;t accept anything it doesn&rsquo;t use.</p>
<p>Consider this example:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">FileStorage</span><span class="w"> </span><span class="kd">struct</span><span class="p">{}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">FileStorage</span><span class="p">)</span><span class="w"> </span><span class="nf">Save</span><span class="p">(</span><span class="nx">data</span><span class="w"> </span><span class="p">[]</span><span class="kt">byte</span><span class="p">)</span><span class="w"> </span><span class="kt">error</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Println</span><span class="p">(</span><span class="s">&#34;Saving data to disk...&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">FileStorage</span><span class="p">)</span><span class="w"> </span><span class="nf">Load</span><span class="p">(</span><span class="nx">id</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">([]</span><span class="kt">byte</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Println</span><span class="p">(</span><span class="s">&#34;Loading data from disk...&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="p">[]</span><span class="nb">byte</span><span class="p">(</span><span class="s">&#34;data&#34;</span><span class="p">),</span><span class="w"> </span><span class="kc">nil</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p><code>FileStorage</code> has two methods: <code>Save</code> and <code>Load</code>. Now suppose you write a function that only
needs to save data:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">Backup</span><span class="p">(</span><span class="nx">fs</span><span class="w"> </span><span class="nx">FileStorage</span><span class="p">,</span><span class="w"> </span><span class="nx">data</span><span class="w"> </span><span class="p">[]</span><span class="kt">byte</span><span class="p">)</span><span class="w"> </span><span class="kt">error</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">fs</span><span class="p">.</span><span class="nf">Save</span><span class="p">(</span><span class="nx">data</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>This works, but there are a few problems hiding here.</p>
<p><code>Backup</code> takes a <code>FileStorage</code> directly, so it only works with that type. If you later want
to back up to memory, a network location, or an encrypted store, you&rsquo;ll need to rewrite the
function. Because it depends on a concrete type, your tests have to use <code>FileStorage</code> too,
which might involve disk I/O or other side effects you don&rsquo;t want in unit tests. And from
the function signature, it&rsquo;s not obvious what part of <code>FileStorage</code> the function actually
uses.</p>
<p>Instead of depending on a specific type, we can depend on an abstraction. In Go, you can
achieve that through an interface. So let&rsquo;s define one:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">Storage</span><span class="w"> </span><span class="kd">interface</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">Save</span><span class="p">(</span><span class="nx">data</span><span class="w"> </span><span class="p">[]</span><span class="kt">byte</span><span class="p">)</span><span class="w"> </span><span class="kt">error</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">Load</span><span class="p">(</span><span class="nx">id</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">([]</span><span class="kt">byte</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Now <code>Backup</code> can take a <code>Storage</code> instead:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">Backup</span><span class="p">(</span><span class="nx">store</span><span class="w"> </span><span class="nx">Storage</span><span class="p">,</span><span class="w"> </span><span class="nx">data</span><span class="w"> </span><span class="p">[]</span><span class="kt">byte</span><span class="p">)</span><span class="w"> </span><span class="kt">error</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">store</span><span class="p">.</span><span class="nf">Save</span><span class="p">(</span><span class="nx">data</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p><code>Backup</code> now depends on behavior, not implementation. You can plug in anything that
satisfies <code>Storage</code>, something that writes to disk, memory, or even a remote service. And
<code>FileStorage</code> still works without any change.</p>
<p>You can also test it with a fake:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">FakeStorage</span><span class="w"> </span><span class="kd">struct</span><span class="p">{}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">FakeStorage</span><span class="p">)</span><span class="w"> </span><span class="nf">Save</span><span class="p">(</span><span class="nx">data</span><span class="w"> </span><span class="p">[]</span><span class="kt">byte</span><span class="p">)</span><span class="w"> </span><span class="kt">error</span><span class="w">         </span><span class="p">{</span><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">FakeStorage</span><span class="p">)</span><span class="w"> </span><span class="nf">Load</span><span class="p">(</span><span class="nx">id</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">([]</span><span class="kt">byte</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">TestBackup</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">fake</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">FakeStorage</span><span class="p">{}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">Backup</span><span class="p">(</span><span class="nx">fake</span><span class="p">,</span><span class="w"> </span><span class="p">[]</span><span class="nb">byte</span><span class="p">(</span><span class="s">&#34;test-data&#34;</span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">t</span><span class="p">.</span><span class="nf">Fatal</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>That&rsquo;s a step forward. It fixes the coupling issue and makes the tests free of side effects.
However, there&rsquo;s still one issue: <code>Backup</code> only calls <code>Save</code>, yet the <code>Storage</code> interface
includes both <code>Save</code> and <code>Load</code>. If <code>Storage</code> later gains more methods, every fake must grow
too, even if those methods aren&rsquo;t used. That&rsquo;s exactly what the ISP warns against.</p>
<p>The above interface is too broad. So let&rsquo;s narrow it to match what the function actually
needs:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">Saver</span><span class="w"> </span><span class="kd">interface</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">Save</span><span class="p">(</span><span class="nx">data</span><span class="w"> </span><span class="p">[]</span><span class="kt">byte</span><span class="p">)</span><span class="w"> </span><span class="kt">error</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Then update the function:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">Backup</span><span class="p">(</span><span class="nx">s</span><span class="w"> </span><span class="nx">Saver</span><span class="p">,</span><span class="w"> </span><span class="nx">data</span><span class="w"> </span><span class="p">[]</span><span class="kt">byte</span><span class="p">)</span><span class="w"> </span><span class="kt">error</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">s</span><span class="p">.</span><span class="nf">Save</span><span class="p">(</span><span class="nx">data</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Now the intent is clear. <code>Backup</code> only depends on <code>Save</code>. A test double can just implement
that one method:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">FakeSaver</span><span class="w"> </span><span class="kd">struct</span><span class="p">{}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">FakeSaver</span><span class="p">)</span><span class="w"> </span><span class="nf">Save</span><span class="p">(</span><span class="nx">data</span><span class="w"> </span><span class="p">[]</span><span class="kt">byte</span><span class="p">)</span><span class="w"> </span><span class="kt">error</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">TestBackup</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">fake</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">FakeSaver</span><span class="p">{}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">Backup</span><span class="p">(</span><span class="nx">fake</span><span class="p">,</span><span class="w"> </span><span class="p">[]</span><span class="nb">byte</span><span class="p">(</span><span class="s">&#34;test-data&#34;</span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">t</span><span class="p">.</span><span class="nf">Fatal</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>The original <code>FileStorage</code> still works fine:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">fs</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">FileStorage</span><span class="p">{}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">_</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nf">Backup</span><span class="p">(</span><span class="nx">fs</span><span class="p">,</span><span class="w"> </span><span class="p">[]</span><span class="nb">byte</span><span class="p">(</span><span class="s">&#34;backup-data&#34;</span><span class="p">))</span><span class="w">
</span></span></span></code></pre></div><p>Go&rsquo;s implicit interface satisfaction makes this less ceremonious. Any type with a <code>Save</code>
method automatically satisfies <code>Saver</code>.</p>
<p>This pattern reflects a broader Go convention: define small interfaces on the consumer side,
close to the code that uses them. The consumer knows what subset of behavior it needs and
can define a minimal contract for it. If you define the interface on the producer side
instead, every consumer is forced to depend on that definition. A single change to the
producer&rsquo;s interface can ripple across your codebase unnecessarily.</p>
<p>From Go <a href="https://go.dev/wiki/CodeReviewComments#interfaces:~:text=Go%20interfaces%20generally,the%20real%20implementation" rel="noopener noreferrer" target="_blank">code review comments</a>:</p>
<blockquote>
  <p>Go interfaces generally belong in the package that uses values of the interface type, not
the package that implements those values. The implementing package should return concrete
(usually pointer or struct) types: that way, new methods can be added to implementations
without requiring extensive refactoring.</p>

</blockquote><p>This isn&rsquo;t a strict rule. The standard library defines producer-side interfaces like
<code>io.Reader</code> and <code>io.Writer</code>, which is fine because they&rsquo;re stable and general-purpose. But
for application code, interfaces usually exist in only two places: production code and
tests. Keeping them near the consumer reduces coupling between multiple packages and keeps
the code easier to evolve.</p>
<p>You&rsquo;ll see this same idea pop up all the time. Take the AWS SDK, for example. It&rsquo;s tempting
to define a big S3 client interface and use it everywhere:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">S3Client</span><span class="w"> </span><span class="kd">interface</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">PutObject</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">ctx</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">input</span><span class="w"> </span><span class="o">*</span><span class="nx">s3</span><span class="p">.</span><span class="nx">PutObjectInput</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">opts</span><span class="w"> </span><span class="o">...</span><span class="kd">func</span><span class="p">(</span><span class="o">*</span><span class="nx">s3</span><span class="p">.</span><span class="nx">Options</span><span class="p">))</span><span class="w"> </span><span class="p">(</span><span class="o">*</span><span class="nx">s3</span><span class="p">.</span><span class="nx">PutObjectOutput</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">GetObject</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">ctx</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">input</span><span class="w"> </span><span class="o">*</span><span class="nx">s3</span><span class="p">.</span><span class="nx">GetObjectInput</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">opts</span><span class="w"> </span><span class="o">...</span><span class="kd">func</span><span class="p">(</span><span class="o">*</span><span class="nx">s3</span><span class="p">.</span><span class="nx">Options</span><span class="p">))</span><span class="w"> </span><span class="p">(</span><span class="o">*</span><span class="nx">s3</span><span class="p">.</span><span class="nx">GetObjectOutput</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">ListObjectsV2</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">ctx</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">input</span><span class="w"> </span><span class="o">*</span><span class="nx">s3</span><span class="p">.</span><span class="nx">ListObjectsV2Input</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">opts</span><span class="w"> </span><span class="o">...</span><span class="kd">func</span><span class="p">(</span><span class="o">*</span><span class="nx">s3</span><span class="p">.</span><span class="nx">Options</span><span class="p">))</span><span class="w"> </span><span class="p">(</span><span class="o">*</span><span class="nx">s3</span><span class="p">.</span><span class="nx">ListObjectsV2Output</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// ...and many more</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Depending on such a large interface couples your code to far more than it uses. Any change
or addition to this interface can ripple through your code and tests for no good reason.</p>
<p>For example, if your code uploads files, it only needs the <code>PutObject</code> method:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">UploadReport</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">ctx</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w"> </span><span class="nx">client</span><span class="w"> </span><span class="nx">S3Client</span><span class="p">,</span><span class="w"> </span><span class="nx">data</span><span class="w"> </span><span class="p">[]</span><span class="kt">byte</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">)</span><span class="w"> </span><span class="kt">error</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">client</span><span class="p">.</span><span class="nf">PutObject</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">ctx</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="o">&amp;</span><span class="nx">s3</span><span class="p">.</span><span class="nx">PutObjectInput</span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">Bucket</span><span class="p">:</span><span class="w"> </span><span class="nx">aws</span><span class="p">.</span><span class="nf">String</span><span class="p">(</span><span class="s">&#34;reports&#34;</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">Key</span><span class="p">:</span><span class="w">    </span><span class="nx">aws</span><span class="p">.</span><span class="nf">String</span><span class="p">(</span><span class="s">&#34;daily.csv&#34;</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">Body</span><span class="p">:</span><span class="w">   </span><span class="nx">bytes</span><span class="p">.</span><span class="nf">NewReader</span><span class="p">(</span><span class="nx">data</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">},</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">err</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>But accepting the full <code>S3Client</code> here ties <code>UploadReport</code> to an interface that&rsquo;s too broad.
A fake must implement all the methods just to satisfy it.</p>
<p>It&rsquo;s better to define a small, consumer-side interface that captures only the operations you
need. This is exactly what the <a href="https://docs.aws.amazon.com/sdk-for-go/v2/developer-guide/unit-testing.html#:~:text=To%20support%20mocking%2C%20use%20Go%20interfaces%20instead%20of%20concrete%20service%20client%2C%20paginators%2C%20and%20waiter%20types%2C%20such%20as%20s3.Client.%20This%20allows%20your%20application%20to%20use%20patterns%20like%20dependency%20injection%20to%20test%20your%20application%20logic." rel="noopener noreferrer" target="_blank">AWS SDK doc</a> recommends for testing.</p>
<blockquote>
  <p>To support mocking, use Go interfaces instead of concrete service client, paginators, and
waiter types, such as s3.Client. This allows your application to use patterns like
dependency injection to test your application logic.</p>

</blockquote><p>Similar to what we&rsquo;ve seen before, you can define a single method interface:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">Uploader</span><span class="w"> </span><span class="kd">interface</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">PutObject</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">ctx</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">input</span><span class="w"> </span><span class="o">*</span><span class="nx">s3</span><span class="p">.</span><span class="nx">PutObjectInput</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">opts</span><span class="w"> </span><span class="o">...</span><span class="kd">func</span><span class="p">(</span><span class="o">*</span><span class="nx">s3</span><span class="p">.</span><span class="nx">Options</span><span class="p">))</span><span class="w"> </span><span class="p">(</span><span class="o">*</span><span class="nx">s3</span><span class="p">.</span><span class="nx">PutObjectOutput</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>And then use it in the function:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">UploadReport</span><span class="p">(</span><span class="nx">ctx</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w"> </span><span class="nx">u</span><span class="w"> </span><span class="nx">Uploader</span><span class="p">,</span><span class="w"> </span><span class="nx">data</span><span class="w"> </span><span class="p">[]</span><span class="kt">byte</span><span class="p">)</span><span class="w"> </span><span class="kt">error</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">u</span><span class="p">.</span><span class="nf">PutObject</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">ctx</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="o">&amp;</span><span class="nx">s3</span><span class="p">.</span><span class="nx">PutObjectInput</span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">Bucket</span><span class="p">:</span><span class="w"> </span><span class="nx">aws</span><span class="p">.</span><span class="nf">String</span><span class="p">(</span><span class="s">&#34;reports&#34;</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">Key</span><span class="p">:</span><span class="w">    </span><span class="nx">aws</span><span class="p">.</span><span class="nf">String</span><span class="p">(</span><span class="s">&#34;daily.csv&#34;</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">Body</span><span class="p">:</span><span class="w">   </span><span class="nx">bytes</span><span class="p">.</span><span class="nf">NewReader</span><span class="p">(</span><span class="nx">data</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">},</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">err</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>The intent is obvious: this function uploads data and depends only on <code>PutObject</code>. The fake
for tests is now tiny:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">FakeUploader</span><span class="w"> </span><span class="kd">struct</span><span class="p">{}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">FakeUploader</span><span class="p">)</span><span class="w"> </span><span class="nf">PutObject</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">_</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">_</span><span class="w"> </span><span class="o">*</span><span class="nx">s3</span><span class="p">.</span><span class="nx">PutObjectInput</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">_</span><span class="w"> </span><span class="o">...</span><span class="kd">func</span><span class="p">(</span><span class="o">*</span><span class="nx">s3</span><span class="p">.</span><span class="nx">Options</span><span class="p">))</span><span class="w"> </span><span class="p">(</span><span class="o">*</span><span class="nx">s3</span><span class="p">.</span><span class="nx">PutObjectOutput</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">s3</span><span class="p">.</span><span class="nx">PutObjectOutput</span><span class="p">{},</span><span class="w"> </span><span class="kc">nil</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>If we distill the workflow as a general rule of thumb, it&rsquo;d look like this:</p>
<blockquote>
  <p>Insert a seam between two tightly coupled components by placing a consumer-side interface
that exposes only the methods the caller invokes.</p>

</blockquote><p>Fin!</p>
<!-- references -->
<!-- prettier-ignore-start -->
<!-- prettier-ignore-end -->
]]></content:encoded>
    </item>
    <item>
      <title>Avoiding collisions in Go context keys</title>
      <link>https://rednafi.com/go/avoid-context-key-collisions/</link>
      <pubDate>Wed, 22 Oct 2025 00:00:00 +0000</pubDate>
      <guid>https://rednafi.com/go/avoid-context-key-collisions/</guid>
      <description>Master Go context keys with custom types, avoid collisions using empty structs, and learn accessor patterns for safe request-scoped values.</description>
      <category>Go</category>
      <category>API</category>
      <category>Performance</category>
      <content:encoded><![CDATA[<p>Along with propagating deadlines and cancellation signals, Go&rsquo;s <code>context</code> package can also
carry request-scoped values across API boundaries and processes.</p>
<p>There are only two public API constructs associated with context values:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">WithValue</span><span class="p">(</span><span class="nx">parent</span><span class="w"> </span><span class="nx">Context</span><span class="p">,</span><span class="w"> </span><span class="nx">key</span><span class="p">,</span><span class="w"> </span><span class="nx">val</span><span class="w"> </span><span class="kt">any</span><span class="p">)</span><span class="w"> </span><span class="nx">Context</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">c</span><span class="w"> </span><span class="nx">Context</span><span class="p">)</span><span class="w"> </span><span class="nf">Value</span><span class="p">(</span><span class="nx">key</span><span class="w"> </span><span class="kt">any</span><span class="p">)</span><span class="w"> </span><span class="kt">any</span><span class="w">
</span></span></span></code></pre></div><p><code>WithValue</code> can take any comparable value as both the key and the value. The key defines how
the stored value is identified, and the value can be any data you want to pass through the
call chain.</p>
<p><code>Value</code>, on the other hand, also returns <code>any</code>, which means the compiler cannot infer the
concrete type at compile time. To use the returned data safely, you must perform a type
assertion.</p>
<p>A naive workflow to store and retrieve values in a context looks like this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">ctx</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nf">Background</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Store some value against a key</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">ctx</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nf">WithValue</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;userID&#34;</span><span class="p">,</span><span class="w"> </span><span class="mi">42</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Retrieve the value</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">v</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">ctx</span><span class="p">.</span><span class="nf">Value</span><span class="p">(</span><span class="s">&#34;userID&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Value returns any, so you need a type assertion</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">id</span><span class="p">,</span><span class="w"> </span><span class="nx">ok</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">v</span><span class="p">.(</span><span class="kt">int</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">if</span><span class="w"> </span><span class="p">!</span><span class="nx">ok</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Println</span><span class="p">(</span><span class="s">&#34;unexpected type&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">fmt</span><span class="p">.</span><span class="nf">Println</span><span class="p">(</span><span class="nx">id</span><span class="p">)</span><span class="w"> </span><span class="c1">// 42</span><span class="w">
</span></span></span></code></pre></div><p><code>WithValue</code> returns a new context that wraps the parent. <code>Value</code> walks up the chain of
contexts and returns the first matching key it finds. Since the return type is <code>any</code>, a type
assertion is required to recover the original type. Without the <code>ok</code> check, a mismatch would
cause a panic.</p>
<p>The issue with this setup is that it risks collision. If another package sets a value
against the same key, one overwrites the other:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kn">package</span><span class="w"> </span><span class="nx">main</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kn">import</span><span class="w"> </span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="s">&#34;context&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="s">&#34;fmt&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">ctx</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nf">WithValue</span><span class="p">(</span><span class="nx">context</span><span class="p">.</span><span class="nf">Background</span><span class="p">(),</span><span class="w"> </span><span class="s">&#34;key&#34;</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;from-main&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">ctx</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nf">foo</span><span class="p">(</span><span class="nx">ctx</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Println</span><span class="p">(</span><span class="nx">ctx</span><span class="p">.</span><span class="nf">Value</span><span class="p">(</span><span class="s">&#34;key&#34;</span><span class="p">))</span><span class="w"> </span><span class="c1">// from-foo</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">foo</span><span class="p">(</span><span class="nx">ctx</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">)</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Accidentally reuse the same key in another package</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nf">WithValue</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;key&#34;</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;from-foo&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>The first value becomes inaccessible because <code>WithValue</code> returns a new derived context that
shadows parent values with the same key. The original value still exists in the parent
context but is unreachable through the reassigned variable.</p>
<p>To understand why this collision occurs, you need to know how Go compares interface values.
When you assign a value to an <code>interface{}</code> (or <code>any</code>), Go boxes that value into an internal
representation made up of two <a href="https://unicminds.com/what-is-a-machine-word-and-its-implications/" rel="noopener noreferrer" target="_blank">machine words</a>: one points to the type information, and the
other points to the underlying data.</p>
<p>For example:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">var</span><span class="w"> </span><span class="nx">a</span><span class="w"> </span><span class="kt">any</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="s">&#34;key&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">var</span><span class="w"> </span><span class="nx">b</span><span class="w"> </span><span class="kt">any</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="s">&#34;key&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">fmt</span><span class="p">.</span><span class="nf">Println</span><span class="p">(</span><span class="nx">a</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="nx">b</span><span class="p">)</span><span class="w"> </span><span class="c1">// true</span><span class="w">
</span></span></span></code></pre></div><p>Each boxed interface here stores two things: a pointer to the type <code>string</code> and a pointer to
the data <code>&quot;key&quot;</code>. Since both type and data pointers match, the comparison returns true.</p>
<p><code>WithValue</code> stores both the key and the value as <code>any</code>. When you later call <code>Value</code>, Go
compares the boxed key you pass in with those stored in the context chain. If two different
packages use the same built-in key type and data, like both passing <code>&quot;key&quot;</code> as a string,
their boxed representations look identical. Go sees them as equal, and the most recent value
shadows the earlier one.</p>
<p>If you want to learn more about how interfaces are represented and compared, <a href="https://research.swtch.com/interfaces" rel="noopener noreferrer" target="_blank">Russ Cox&rsquo;s
post on Go interface internals</a> explains it in detail with pretty pictures.</p>
<p>The fix is to make sure the keys have unique types so their boxed representations differ. If
you define a custom type, the type pointer changes even if the data looks the same. For
example:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">userKey</span><span class="w"> </span><span class="kt">string</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">var</span><span class="w"> </span><span class="nx">a</span><span class="w"> </span><span class="kt">any</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nf">userKey</span><span class="p">(</span><span class="s">&#34;key&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">var</span><span class="w"> </span><span class="nx">b</span><span class="w"> </span><span class="kt">any</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="s">&#34;key&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">fmt</span><span class="p">.</span><span class="nf">Println</span><span class="p">(</span><span class="nx">a</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="nx">b</span><span class="p">)</span><span class="w"> </span><span class="c1">// false</span><span class="w">
</span></span></span></code></pre></div><p>Even though the underlying value is <code>&quot;key&quot;</code>, the two interfaces now hold different type
information, so Go considers them unequal. That difference in type identity is what prevents
collisions.</p>
<p>The <a href="https://pkg.go.dev/context#WithValue" rel="noopener noreferrer" target="_blank">context documentation</a> gives this advice:</p>
<blockquote>
  <p>The provided key must be comparable and should not be of type string or any other built-in
type to avoid collisions between packages using context. Users of WithValue should define
their own types for keys. To avoid allocating when assigning to an interface{}, context
keys often have concrete type struct{}. Alternatively, exported context key variables'
static type should be a pointer or interface.</p>

</blockquote><p>In short:</p>
<ul>
<li>Keys must be comparable (<code>string</code>, <code>int</code>, <code>struct</code>, <code>pointer</code>, etc.)</li>
<li>Define unique key types per package to avoid collisions</li>
<li>Use <code>struct{}</code> keys to avoid allocation when stored as <code>any</code></li>
<li>Exported key variables should have pointer or interface types</li>
</ul>
<p>Here&rsquo;s how defining a unique key type prevents collisions:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">userIDKey</span><span class="w"> </span><span class="kt">string</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Store value</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">ctx</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nf">WithValue</span><span class="p">(</span><span class="nx">context</span><span class="p">.</span><span class="nf">Background</span><span class="p">(),</span><span class="w"> </span><span class="nf">userIDKey</span><span class="p">(</span><span class="s">&#34;id&#34;</span><span class="p">),</span><span class="w"> </span><span class="mi">42</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Retrieve value</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">id</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">ctx</span><span class="p">.</span><span class="nf">Value</span><span class="p">(</span><span class="nf">userIDKey</span><span class="p">(</span><span class="s">&#34;id&#34;</span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">fmt</span><span class="p">.</span><span class="nf">Println</span><span class="p">(</span><span class="nx">id</span><span class="p">)</span><span class="w"> </span><span class="c1">// 42</span><span class="w">
</span></span></span></code></pre></div><p>Even if another package uses the string <code>&quot;id&quot;</code>, the key types differ, so they cannot
collide.</p>
<p>To avoid allocation when <code>WithValue</code> assigns the inbound value to interface <code>any</code>, you can
define an empty struct key. Unlike strings or integers, which allocate when boxed into an
interface, a zero-sized struct occupies no memory and needs no allocation:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">key</span><span class="w"> </span><span class="kd">struct</span><span class="p">{}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Store value</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">ctx</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nf">WithValue</span><span class="p">(</span><span class="nx">context</span><span class="p">.</span><span class="nf">Background</span><span class="p">(),</span><span class="w"> </span><span class="nx">key</span><span class="p">{},</span><span class="w"> </span><span class="s">&#34;value&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Retrieve value</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">v</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">ctx</span><span class="p">.</span><span class="nf">Value</span><span class="p">(</span><span class="nx">key</span><span class="p">{})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">fmt</span><span class="p">.</span><span class="nf">Println</span><span class="p">(</span><span class="nx">v</span><span class="p">)</span><span class="w"> </span><span class="c1">// value</span><span class="w">
</span></span></span></code></pre></div><p>Empty structs are ideal for local, unexported keys. They are unique by type and add no
overhead.</p>
<p>Alternatively, exported keys can use pointers, which also avoid allocation and guarantee
uniqueness. When a pointer is boxed into an interface, no data copy occurs because the
interface just holds the pointer reference. Pointers are also ideal for keys that need to be
shared across packages.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">userIDKey</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">name</span><span class="w"> </span><span class="kt">string</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Struct pointer as key</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">var</span><span class="w"> </span><span class="nx">UserIDKey</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">userIDKey</span><span class="p">{</span><span class="s">&#34;user-id&#34;</span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Store value. No allocation here since userIDKey is a pointer</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// to a struct</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">ctx</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nf">WithValue</span><span class="p">(</span><span class="nx">context</span><span class="p">.</span><span class="nf">Background</span><span class="p">(),</span><span class="w"> </span><span class="nx">UserIDKey</span><span class="p">,</span><span class="w"> </span><span class="mi">42</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Retrieve value</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">id</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">ctx</span><span class="p">.</span><span class="nf">Value</span><span class="p">(</span><span class="nx">UserIDKey</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">fmt</span><span class="p">.</span><span class="nf">Println</span><span class="p">(</span><span class="nx">id</span><span class="p">)</span><span class="w"> </span><span class="c1">// 42</span><span class="w">
</span></span></span></code></pre></div><p>Here, <code>UserIDKey</code> points to a unique struct instance, so equality checks work by pointer
identity. The <code>name</code> field exists only for debugging. This avoids allocation and ensures
exported keys remain unique even when shared between packages.</p>
<p>When exposing context values across APIs, you can approach it in two ways depending on how
much control and safety you want to give your users.</p>
<h2 id="1-expose-keys-directly">1. Expose keys directly</h2>
<p>You can export the key itself and let users interact with it freely:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">APIKey</span><span class="w"> </span><span class="kt">string</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Allow the other packages to directly use this key</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">var</span><span class="w"> </span><span class="nx">APIKeyContextKey</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nf">APIKey</span><span class="p">(</span><span class="s">&#34;api-key&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Store value. An allocation will occur since the key is of type string</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">ctx</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nf">WithValue</span><span class="p">(</span><span class="nx">context</span><span class="p">.</span><span class="nf">Background</span><span class="p">(),</span><span class="w"> </span><span class="nx">APIKeyContextKey</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;secret&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Retrieve value</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">v</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">ctx</span><span class="p">.</span><span class="nf">Value</span><span class="p">(</span><span class="nx">APIKeyContextKey</span><span class="p">).(</span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="c1">// caller must do this assertion</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">fmt</span><span class="p">.</span><span class="nf">Println</span><span class="p">(</span><span class="nx">v</span><span class="p">)</span><span class="w"> </span><span class="c1">// secret</span><span class="w">
</span></span></span></code></pre></div><p>When you export the key directly the caller gains direct access, but they also must:</p>
<ul>
<li>do the type assertion themselves and handle the ok result to avoid panics</li>
<li>ensure they don&rsquo;t accidentally overwrite values using the wrong key</li>
</ul>
<p>The <a href="https://cs.opensource.google/go/go/+/refs/tags/go1.25.3:src/net/http/server.go;l=239-251" rel="noopener noreferrer" target="_blank">net/http</a> package uses this approach for some of its exported context keys:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">contextKey</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">name</span><span class="w"> </span><span class="kt">string</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Notice the exported keys</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">var</span><span class="w"> </span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">ServerContextKey</span><span class="w">    </span><span class="p">=</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">contextKey</span><span class="p">{</span><span class="s">&#34;http-server&#34;</span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">LocalAddrContextKey</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">contextKey</span><span class="p">{</span><span class="s">&#34;local-addr&#34;</span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">)</span><span class="w">
</span></span></span></code></pre></div><p>Each variable points to a distinct struct, making them unique by pointer identity.</p>
<p>The <a href="https://cs.opensource.google/go/go/+/refs/tags/go1.25.3:src/net/http/serve_test.go;l=5132-5144" rel="noopener noreferrer" target="_blank">serve_test.go</a> file uses these keys like this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">ctx</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nf">WithValue</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">context</span><span class="p">.</span><span class="nf">Background</span><span class="p">(),</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ServerContextKey</span><span class="p">,</span><span class="w"> </span><span class="nx">srv</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Type assertion to recover the concrete type</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">srv2</span><span class="p">,</span><span class="w"> </span><span class="nx">ok</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">ctx</span><span class="p">.</span><span class="nf">Value</span><span class="p">(</span><span class="nx">http</span><span class="p">.</span><span class="nx">ServerContextKey</span><span class="p">).(</span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Server</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">if</span><span class="w"> </span><span class="nx">ok</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Println</span><span class="p">(</span><span class="nx">srv</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="nx">srv2</span><span class="p">)</span><span class="w"> </span><span class="c1">// true</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>The server value is stored in the context and later retrieved using the same pointer key.
The user must perform a type assertion and handle it safely.</p>
<h2 id="2-expose-accessor-functions">2. Expose accessor functions</h2>
<p>The other approach is to hide the key and provide accessor functions to set and retrieve
values. This removes the need for users to remember the right key type or perform type
assertions manually.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// Define a private key type to avoid collisions</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">contextKey</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">name</span><span class="w"> </span><span class="kt">string</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Define the key</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">var</span><span class="w"> </span><span class="nx">userIDKey</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">contextKey</span><span class="p">{</span><span class="s">&#34;user-id&#34;</span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Public accessor to store a value to ctx</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">WithUserID</span><span class="p">(</span><span class="nx">ctx</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w"> </span><span class="nx">id</span><span class="w"> </span><span class="kt">int</span><span class="p">)</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// No allocation here since userIDKey is a pointer to a struct</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nf">WithValue</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span><span class="w"> </span><span class="nx">userIDKey</span><span class="p">,</span><span class="w"> </span><span class="nx">id</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Public accessor to fetch a value from ctx</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">UserIDFromContext</span><span class="p">(</span><span class="nx">ctx</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">int</span><span class="p">,</span><span class="w"> </span><span class="kt">bool</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">v</span><span class="p">,</span><span class="w"> </span><span class="nx">ok</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">ctx</span><span class="p">.</span><span class="nf">Value</span><span class="p">(</span><span class="nx">userIDKey</span><span class="p">).(</span><span class="kt">int</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">v</span><span class="p">,</span><span class="w"> </span><span class="nx">ok</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Store value</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">ctx</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">WithUserID</span><span class="p">(</span><span class="nx">context</span><span class="p">.</span><span class="nf">Background</span><span class="p">(),</span><span class="w"> </span><span class="mi">42</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Retrieve value</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">id</span><span class="p">,</span><span class="w"> </span><span class="nx">ok</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">UserIDFromContext</span><span class="p">(</span><span class="nx">ctx</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">if</span><span class="w"> </span><span class="nx">ok</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Println</span><span class="p">(</span><span class="nx">id</span><span class="p">)</span><span class="w"> </span><span class="c1">// 42</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Println</span><span class="p">(</span><span class="s">&#34;no user ID found in context&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>This approach centralizes how values are stored and retrieved from the context. It ensures
the correct key and type are always used, preventing collisions and runtime panics. It also
keeps the calling code shorter since your API users won&rsquo;t need to repeat type assertions
everywhere.</p>
<p><code>WithX</code> / <code>XFromContext</code> accessors appear throughout the Go standard library:</p>
<ul>
<li>
<p><strong><a href="https://github.com/golang/go/blob/39ed968832ad8923a4bd1fb6bc3d9090ddd98401/src/net/http/httptrace/trace.go#L20-L68" rel="noopener noreferrer" target="_blank">net/http/httptrace</a></strong></p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">WithClientTrace</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">ctx</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w"> </span><span class="nx">trace</span><span class="w"> </span><span class="o">*</span><span class="nx">ClientTrace</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">)</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">ContextClientTrace</span><span class="p">(</span><span class="nx">ctx</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">)</span><span class="w"> </span><span class="o">*</span><span class="nx">ClientTrace</span><span class="w">
</span></span></span></code></pre></div></li>
<li>
<p><strong><a href="https://github.com/golang/go/blob/39ed968832ad8923a4bd1fb6bc3d9090ddd98401/src/runtime/pprof/label.go#L60-L63" rel="noopener noreferrer" target="_blank">runtime/pprof</a></strong></p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">WithLabels</span><span class="p">(</span><span class="nx">ctx</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w"> </span><span class="nx">labels</span><span class="w"> </span><span class="nx">LabelSet</span><span class="p">)</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">Labels</span><span class="p">(</span><span class="nx">ctx</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">)</span><span class="w"> </span><span class="nx">LabelSet</span><span class="w">
</span></span></span></code></pre></div></li>
</ul>
<p>You can find similar examples outside of the stdlib. For instance, the <a href="https://github.com/open-telemetry/opentelemetry-go/blob/f0c24571557de839332e48790714a5899c4fd2c6/trace/context.go" rel="noopener noreferrer" target="_blank">OpenTelemetry Go
SDK</a> follows the same model:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">ContextWithSpan</span><span class="p">(</span><span class="nx">parent</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w"> </span><span class="nx">span</span><span class="w"> </span><span class="nx">Span</span><span class="p">)</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">SpanFromContext</span><span class="p">(</span><span class="nx">ctx</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">)</span><span class="w"> </span><span class="nx">Span</span><span class="w">
</span></span></span></code></pre></div><p>This technique standardizes how values are passed across APIs, eliminates redundant type
assertions, and prevents key misuse across packages.</p>
<h2 id="closing-words">Closing words</h2>
<p>I usually use a pointer to a struct as a key and <a href="#2-expose-accessor-functions">expose accessor functions</a> when building
user-facing APIs. Otherwise, in services, I often define empty struct keys and expose them
publicly to avoid the ceremony around accessor functions.</p>
<!-- references -->
<!-- prettier-ignore-start -->
<!-- prettier-ignore-end -->
]]></content:encoded>
    </item>
    <item>
      <title>Let the domain guide your application structure</title>
      <link>https://rednafi.com/go/app-structure/</link>
      <pubDate>Sat, 20 Sep 2025 00:00:00 +0000</pubDate>
      <guid>https://rednafi.com/go/app-structure/</guid>
      <description>Organize Go apps by domain, not technology. Learn why models/controllers structure hurts and how bounded contexts create better separation.</description>
      <category>Go</category>
      <category>API</category>
      <category>Web</category>
      <content:encoded><![CDATA[<p>I like to make the distinction between application structure and architecture. Structure is
how you organize the directories and packages in your app while architecture is how
different components talk to each other. The way your app talks to other services in a fleet
can also be called architecture.</p>
<p>While structure often influences architecture and vice versa, this distinction is important.
This post is strictly about application structure and not library structure. Library
structure is often driven by different design pressures than their app counterparts. There
are a ton of canonical examples of good library structure in the stdlib, but it&rsquo;s app
structure where things get a bit more muddy.</p>
<p>At work, I not only write Go in a distributed system environment but also review potential
candidates&rsquo; assignments in the hiring pipeline. While there is no objectively right or wrong
way to structure an app, I do see a common pitfall in candidates&rsquo; submissions that is
usually frowned upon in a Go application.</p>
<blockquote>
  <p>App structure should be driven by what it does and not what it&rsquo;s built with. Let the
domain guide the structure, not technology or the current language specific zeitgeist.</p>

</blockquote><p>Ben Johnson&rsquo;s <a href="https://www.gobeyond.dev/standard-package-layout/" rel="noopener noreferrer" target="_blank">Standard Package Layout</a> is a good reference for this. He points out why
approaches like monolithic packages, Rails style layouts, or grouping by module don&rsquo;t fit
well in Go. Then he lays out a map where the root package holds domain types, dependencies
are grouped in separate packages, and the main package wires everything together.</p>
<p>While Ben&rsquo;s post is focused on what you should be doing, I want to keep this discussion a
bit more open-ended and just talk about one bad pattern that you probably should avoid. The
rest of the app structure is subjective and should be driven by requirements. Use your
judgement.</p>
<p>The mistake I often see is people making a bunch of generically named packages like
<code>models</code>, <code>controllers</code>, <code>handlers</code> and stuffing everything there. App structure like the
following is quite common:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="cl">mystore/
</span></span><span class="line"><span class="cl">├── controllers/
</span></span><span class="line"><span class="cl">│   ├── order_controller.go
</span></span><span class="line"><span class="cl">│   └── user_controller.go
</span></span><span class="line"><span class="cl">├── models/
</span></span><span class="line"><span class="cl">│   ├── order.go
</span></span><span class="line"><span class="cl">│   └── user.go
</span></span><span class="line"><span class="cl">├── handlers/
</span></span><span class="line"><span class="cl">│   ├── http_handler.go
</span></span><span class="line"><span class="cl">│   └── webhook_handler.go
</span></span><span class="line"><span class="cl">└── main.go
</span></span></code></pre></div><p>In Go there&rsquo;s no file level separation, only package level separation. That means everything
under <code>models</code> like order and product lives in the same namespace. The same is true for
<code>controllers</code> and <code>handlers</code>.</p>
<p>Once you put multiple business domains under a generic umbrella, you tie them together. This
might make sense in a language like Python where file names are prefixed in the fully
qualified import path. In Python you&rsquo;d import them as follows:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># Identifiers live in the order namespace</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">mystore.models</span> <span class="kn">import</span> <span class="n">order</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Identifiers live in the http_handler namespace</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">mystore.handlers</span> <span class="kn">import</span> <span class="n">http_handler</span>
</span></span></code></pre></div><p>But in Go the import path becomes this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// Identifiers from order.go, user.go, product.go</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// all live in the same namespace</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kn">import</span><span class="w"> </span><span class="s">&#34;mystore/models&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Identifiers from http_handler.go &amp; webhook_handler.go</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// all live in the same namespace</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kn">import</span><span class="w"> </span><span class="s">&#34;mystore/handlers&#34;</span><span class="w">
</span></span></span></code></pre></div><p>There is no file level delineation in Go. If you put different domains under the same
<code>models</code> directory, there is no indication at import time what domain a model belongs to.
The only clue is the identifier name. This isn&rsquo;t ideal when you want clear separation
between domains.</p>
<blockquote>
  <p>In Go, packages define your <a href="https://martinfowler.com/bliki/BoundedContext.html" rel="noopener noreferrer" target="_blank">bounded context</a>, not files within a package. Domains should
be delineated by top level packages, not by file names.</p>

</blockquote><p>For your top level business logic, you want package level separation between domains. Order
logic should live in <code>order</code>, user logic should live in <code>user</code>. These packages will be
imported in many places throughout the app, and keeping them separate keeps dependencies
clear.</p>
<p>It could look like this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="cl">mystore/
</span></span><span class="line"><span class="cl">├── order/          &lt;-- business logic related to the order domain
</span></span><span class="line"><span class="cl">│   ├── order.go
</span></span><span class="line"><span class="cl">│   └── service.go
</span></span><span class="line"><span class="cl">├── user/           &lt;-- business logic related to the user domain
</span></span><span class="line"><span class="cl">│   ├── user.go
</span></span><span class="line"><span class="cl">│   └── service.go
</span></span><span class="line"><span class="cl">└── cmd/            &lt;-- wire everything here
</span></span><span class="line"><span class="cl">    └── mystore/
</span></span><span class="line"><span class="cl">        └── main.go
</span></span></code></pre></div><p>Each domain owns its own logic and optional adapters. If you need to find order related
code, you go to <code>order</code>. If you need user code, you go to <code>user</code>. Nothing is smooshed
together under a generic bucket.</p>
<p>The details around how you layer your app can differ based on requirements, but the
important point is that your top level directories shouldn&rsquo;t just be generic buckets
containing all domains. That makes navigation harder. A better approach is letting the
domain guide the structure and only layering in technology when it matters.</p>
<p>You can place your transport concerns alongside the top level packages. A top level <code>http</code>
package can hold handlers that import service functions from the domain packages. You can
put all handlers under <code>http</code> or split them into <code>http/order</code> and <code>http/user</code>. Both are
valid choices. If you put all handlers under <code>http</code>, that&rsquo;s fine because they are usually
imported in one place where you wire routes. The same is true for database adapters. You can
put them all under <code>postgres</code> or split them into <code>postgres/order</code> and <code>postgres/user</code>. Both
patterns are acceptable. The key difference is that domains need package level separation,
while technology packages can be grouped because they are only wired at the edge.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-txt" data-lang="txt"><span class="line"><span class="cl">mystore/
</span></span><span class="line"><span class="cl">├── order/
</span></span><span class="line"><span class="cl">│   ├── order.go
</span></span><span class="line"><span class="cl">│   └── service.go
</span></span><span class="line"><span class="cl">├── user/
</span></span><span class="line"><span class="cl">│   ├── user.go
</span></span><span class="line"><span class="cl">│   └── service.go
</span></span><span class="line"><span class="cl">├── http/                 &lt;-- lumping all the handlers here is fine
</span></span><span class="line"><span class="cl">│   ├── order_handler.go
</span></span><span class="line"><span class="cl">│   └── user_handler.go
</span></span><span class="line"><span class="cl">├── postgres/             &lt;-- this is fine, but you can create sub pkgs too
</span></span><span class="line"><span class="cl">│   ├── order_repo.go
</span></span><span class="line"><span class="cl">│   └── user_repo.go
</span></span><span class="line"><span class="cl">└── cmd/
</span></span><span class="line"><span class="cl">    └── server/
</span></span><span class="line"><span class="cl">        └── main.go
</span></span></code></pre></div><p>But depending on the complexity of your app, this is also absolutely fine:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-txt" data-lang="txt"><span class="line"><span class="cl">mystore/
</span></span><span class="line"><span class="cl">├── order/
</span></span><span class="line"><span class="cl">│   ├── order.go
</span></span><span class="line"><span class="cl">│   └── service.go
</span></span><span class="line"><span class="cl">├── user/
</span></span><span class="line"><span class="cl">│   ├── user.go
</span></span><span class="line"><span class="cl">│   └── service.go
</span></span><span class="line"><span class="cl">├── http/                 &lt;-- handlers are split by domain here
</span></span><span class="line"><span class="cl">│   ├── order/
</span></span><span class="line"><span class="cl">│   │   └── handler.go
</span></span><span class="line"><span class="cl">│   └── user/
</span></span><span class="line"><span class="cl">│       └── handler.go
</span></span><span class="line"><span class="cl">├── postgres/             &lt;-- repos are split by domain here
</span></span><span class="line"><span class="cl">│   ├── order/
</span></span><span class="line"><span class="cl">│   │   └── repo.go
</span></span><span class="line"><span class="cl">│   └── user/
</span></span><span class="line"><span class="cl">│       └── repo.go
</span></span><span class="line"><span class="cl">└── cmd/
</span></span><span class="line"><span class="cl">    └── server/
</span></span><span class="line"><span class="cl">        └── main.go
</span></span></code></pre></div><p>The rule of thumb is that top level domains should never import anything from technology
folders like <code>http</code> or <code>postgres</code>. Instead, <code>http</code> and <code>postgres</code> should always import from
domain packages. You can add a linter to enforce this rule but since Go doesn&rsquo;t allow import
cycles, this is automatically enforced by the compiler.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="cl">   +-----------+     +-----------+
</span></span><span class="line"><span class="cl">   |   order   |     |   user    |
</span></span><span class="line"><span class="cl">   +-----------+     +-----------+
</span></span><span class="line"><span class="cl">          ^                ^
</span></span><span class="line"><span class="cl">          |                |
</span></span><span class="line"><span class="cl">   +------------------------------+
</span></span><span class="line"><span class="cl">   | http              postgres   |
</span></span><span class="line"><span class="cl">   +------------------------------+
</span></span><span class="line"><span class="cl">                  ^
</span></span><span class="line"><span class="cl">                  |
</span></span><span class="line"><span class="cl">             +---------+
</span></span><span class="line"><span class="cl">             |   cmd   |
</span></span><span class="line"><span class="cl">             +---------+
</span></span></code></pre></div><p>Domains sit at the top. Technology packages depend on them, never the other way around. The
<code>cmd</code> package wires everything together. This keeps the graph simple and keeps domains
independent.</p>
<hr>
<p>Astute readers might notice that I have left out any discussion around the <code>internal</code>
directory. This is intentional. Depending on your requirements, you might opt in for an
<code>internal</code> directory or not. This isn&rsquo;t important for our discussion. The main point I
wanted to emphasize is that technology or architecture patterns shouldn&rsquo;t guide your app
structure. It should be based on something more persistent and nothing is more persistent
than your application&rsquo;s domain.</p>
<!-- references -->
<!-- prettier-ignore-start -->
<!-- prettier-ignore-end -->
]]></content:encoded>
    </item>
    <item>
      <title>Lifecycle management in Go tests</title>
      <link>https://rednafi.com/go/lifecycle-management-in-tests/</link>
      <pubDate>Sat, 30 Aug 2025 00:00:00 +0000</pubDate>
      <guid>https://rednafi.com/go/lifecycle-management-in-tests/</guid>
      <description>Master Go test lifecycle with t.Cleanup(), subtests, and TestMain. Learn per-test, grouped, and package-wide setup patterns effectively.</description>
      <category>Go</category>
      <category>Testing</category>
      <category>API</category>
      <content:encoded><![CDATA[<p>Unlike pytest or JUnit, Go&rsquo;s standard testing framework doesn&rsquo;t give you as many knobs for
tuning the lifecycle of your tests.</p>
<p>By lifecycle I mean the usual setup and teardown hooks or fixtures that are common in other
languages. I think this is a good thing because you don&rsquo;t need to pick up many different
framework-specific workflows for something so fundamental.</p>
<p>Go gives you enough hooks to handle this with less ceremony. But it can still be tricky to
figure out the right conventions for setup and teardown that don&rsquo;t look odd to other
Gophers, especially if you haven&rsquo;t written Go for a while. This text explores some common
ways to do lifecycle management in your Go tests.</p>
<p>Before we cover multiple testing scenarios, it&rsquo;s useful to understand how Go&rsquo;s test harness
actually runs your tests.</p>
<h2 id="how-go-discovers-and-runs-your-tests">How Go discovers and runs your tests</h2>
<p>When you type <code>go test</code>, Go doesn&rsquo;t interpret test files directly. It collects all the
<code>_test.go</code> files in a package, compiles them together with the rest of the package, and
produces a temporary binary. That binary contains both your code and your tests, along with
a small harness that drives them. The harness then runs the binary and reports results.</p>
<p>From the <a href="https://pkg.go.dev/cmd/go#hdr-Test_packages" rel="noopener noreferrer" target="_blank">&ldquo;go test&rdquo; command doc</a>:</p>
<blockquote>
  <p>&ldquo;go test&rdquo; automates testing the packages named by the import paths. [&hellip;] recompiles each
package along with any files with names matching the file pattern &ldquo;*_test.go&rdquo;.</p>

</blockquote><h3 id="discovery">Discovery</h3>
<p>Inside each package, the harness looks for test functions. A function qualifies if it has
the form:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">TestXxx</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w">
</span></span></span></code></pre></div><p>where <code>Xxx</code> starts with an uppercase letter. There are no annotations or decorators, just
naming convention. Functions that don&rsquo;t match this signature are ignored.</p>
<h3 id="execution">Execution</h3>
<p>By default, the harness runs tests sequentially. If you want concurrency, you can opt in at
the test level. Calling <code>t.Parallel()</code> inside a test signals that this test may run
alongside others in the same package that also call <code>t.Parallel()</code>. Tests that don&rsquo;t opt in
remain strictly ordered.</p>
<h3 id="scope-of-binaries">Scope of binaries</h3>
<p>Every package with tests produces its own binary, and those binaries are run independently.
There is no global suite that links packages together, so setup and teardown only exist
inside one package&rsquo;s process. If you have ten packages containing tests, you get ten
binaries, each with its own lifecycle.</p>
<p>For example:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-txt" data-lang="txt"><span class="line"><span class="cl">project/
</span></span><span class="line"><span class="cl">├── go.mod
</span></span><span class="line"><span class="cl">├── db/
</span></span><span class="line"><span class="cl">│   ├── db.go
</span></span><span class="line"><span class="cl">│   └── db_test.go
</span></span><span class="line"><span class="cl">└── api/
</span></span><span class="line"><span class="cl">    ├── api.go
</span></span><span class="line"><span class="cl">    └── api_test.go
</span></span></code></pre></div><p>Running <code>go test ./...</code> produces two binaries: one for <code>db</code> and one for <code>api</code>. Each binary
bundles the package code and its tests, and each binary runs on its own. The harness
aggregates the results and prints a combined report, but execution itself is confined to the
package.</p>
<p>It is important to note that there is no file-level scope. All <code>_test.go</code> files in a package
are merged into a single binary, so there is no way to run setup once per file. Similarly,
there is no cross-package scope. Go does not let you set up once for all tests in a module
or tear down after the last package finishes. If you need orchestration across packages, it
has to happen outside of <code>go test</code>, for example in a shell script or a CI pipeline step.</p>
<p>With this background, we can now look at the lifecycle hooks Go does provide. They apply at
three levels: per test function, per group of subtests, and per package.</p>
<h2 id="three-different-scopes">Three different scopes</h2>
<p>Typically you need to perform setup and teardown before and after:</p>
<ul>
<li>each test function is executed (single test function scope)</li>
<li>a group of tests is executed (multiple test function scope)</li>
<li>the full test suite is executed (test package scope)</li>
</ul>
<h3 id="per-test-setup-and-teardown">Per-test setup and teardown</h3>
<p>The smallest scope is the test function itself. You create resources at the start of the
test and clean them up when it ends. This pattern is common when you want each test to run
against a fresh state with no leakage from other tests. The idiomatic way in Go is to wrap
the setup in a helper and register the cleanup with <code>t.Cleanup</code>.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">TestDB</span><span class="w"> </span><span class="kd">struct</span><span class="p">{}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// newTestDB sets up a fresh database for a single test</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">newTestDB</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="o">*</span><span class="nx">TestDB</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">t</span><span class="p">.</span><span class="nf">Helper</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">db</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">TestDB</span><span class="p">{}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// cleanup tied to the function scope</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">t</span><span class="p">.</span><span class="nf">Cleanup</span><span class="p">(</span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">db</span><span class="p">.</span><span class="nf">Close</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">db</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">db</span><span class="w"> </span><span class="o">*</span><span class="nx">TestDB</span><span class="p">)</span><span class="w"> </span><span class="nf">Close</span><span class="p">()</span><span class="w"> </span><span class="p">{}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">db</span><span class="w"> </span><span class="o">*</span><span class="nx">TestDB</span><span class="p">)</span><span class="w"> </span><span class="nf">Insert</span><span class="p">(</span><span class="nx">k</span><span class="p">,</span><span class="w"> </span><span class="nx">v</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="kt">error</span><span class="w">       </span><span class="p">{</span><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">db</span><span class="w"> </span><span class="o">*</span><span class="nx">TestDB</span><span class="p">)</span><span class="w"> </span><span class="nf">Query</span><span class="p">(</span><span class="nx">k</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="s">&#34;value&#34;</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">TestInsert</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">db</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">newTestDB</span><span class="p">(</span><span class="nx">t</span><span class="p">)</span><span class="w"> </span><span class="c1">// new DB created for this test only</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">db</span><span class="p">.</span><span class="nf">Insert</span><span class="p">(</span><span class="s">&#34;foo&#34;</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;bar&#34;</span><span class="p">);</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">t</span><span class="p">.</span><span class="nf">Fatalf</span><span class="p">(</span><span class="s">&#34;insert failed: %v&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>In this example, <code>TestInsert</code> gets its own new database. The cleanup registered with
<code>t.Cleanup</code> makes sure the database is closed when the test finishes. The resource is never
shared with other tests, which gives you strong isolation. The downside is that if your
setup is expensive, it will run before and after every test function, which can slow things
down.</p>
<h3 id="grouped-setup-and-teardown-with-subtests">Grouped setup and teardown with subtests</h3>
<p>The next scope is a group of subtests. Instead of repeating setup for every test, you create
the resource once in the parent test and share it with the children. Teardown runs when the
parent finishes. This works well when you want to test a flow of operations against the same
shared state.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">TestUserFlow</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// new DB created once for this group</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// t.Cleanup() gets called after all the subtests finish and</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// the parent returns</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">db</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">newTestDB</span><span class="p">(</span><span class="nx">t</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">t</span><span class="p">.</span><span class="nf">Run</span><span class="p">(</span><span class="s">&#34;insert user&#34;</span><span class="p">,</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">db</span><span class="p">.</span><span class="nf">Insert</span><span class="p">(</span><span class="s">&#34;user:1&#34;</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;alice&#34;</span><span class="p">);</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">t</span><span class="p">.</span><span class="nf">Fatal</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">t</span><span class="p">.</span><span class="nf">Run</span><span class="p">(</span><span class="s">&#34;query user&#34;</span><span class="p">,</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">val</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">db</span><span class="p">.</span><span class="nf">Query</span><span class="p">(</span><span class="s">&#34;user:1&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">t</span><span class="p">.</span><span class="nf">Fatal</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">if</span><span class="w"> </span><span class="nx">val</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="s">&#34;alice&#34;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">t</span><span class="p">.</span><span class="nf">Fatalf</span><span class="p">(</span><span class="s">&#34;expected alice, got %s&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">val</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Here both subtests share the same database, and the cleanup runs once when <code>TestUserFlow</code>
ends. This is useful when your tests need to act on shared state, like inserting a record
and then querying it. The trade-off is that the tests are no longer fully independent, and
if one subtest leaves the database in a bad state, others may fail in unexpected ways.</p>
<h3 id="package-wide-setup-and-teardown-with-testmain">Package-wide setup and teardown with <code>TestMain</code></h3>
<p>The broadest scope is the package. If you define <code>TestMain</code>, the test harness calls it
instead of running the tests directly. You can perform setup, run all the tests, and then
perform teardown. This allows you to reuse an expensive resource across all tests in the
package.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">var</span><span class="w"> </span><span class="nx">globalDB</span><span class="w"> </span><span class="o">*</span><span class="nx">TestDB</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">TestMain</span><span class="p">(</span><span class="nx">m</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">M</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">globalDB</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">TestDB</span><span class="p">{}</span><span class="w"> </span><span class="c1">// setup once for the entire package</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">code</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">m</span><span class="p">.</span><span class="nf">Run</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">globalDB</span><span class="p">.</span><span class="nf">Close</span><span class="p">()</span><span class="w"> </span><span class="c1">// teardown after all tests</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">os</span><span class="p">.</span><span class="nf">Exit</span><span class="p">(</span><span class="nx">code</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">TestGlobalInsert</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">globalDB</span><span class="p">.</span><span class="nf">Insert</span><span class="p">(</span><span class="s">&#34;k&#34;</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;v&#34;</span><span class="p">);</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">t</span><span class="p">.</span><span class="nf">Fatal</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Here the database is created once and reused by all tests in the package. The teardown runs
when everything is finished. This can make your tests run much faster if setup is expensive,
but you pay for it in global (package wide) state. If one test mutates the shared resource
in an unexpected way, other tests may start failing, and debugging those failures can be
difficult.</p>
<p>Also, remember your setup and teardown are still package bound, meaning each package can
have its own <code>TestMain</code>. Reasoning about their order can get out of hand quickly. Make sure
your tests never depends on the order of <code>TestMain</code> execution. Treat these like <code>init</code>
functions and use them sparingly.</p>
<h3 id="combining-the-levels">Combining the levels</h3>
<p>These three scopes are not mutually exclusive. You can combine them when you need different
levels of control. A typical pattern is to have <code>TestMain</code> start a package-wide service,
create a shared schema or fixture in a parent test for a group of related subtests, and then
still use per-test setup inside individual subtests for fine-grained isolation. Each call to
<code>newTestDB</code> creates a fresh database, so using it at different levels produces different
resources with different lifetimes.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">TestOrders</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">schema</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">newTestDB</span><span class="p">(</span><span class="nx">t</span><span class="p">)</span><span class="w"> </span><span class="c1">// group-level DB shared across subtests</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">t</span><span class="p">.</span><span class="nf">Run</span><span class="p">(</span><span class="s">&#34;create order&#34;</span><span class="p">,</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">db</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">newTestDB</span><span class="p">(</span><span class="nx">t</span><span class="p">)</span><span class="w"> </span><span class="c1">// per-test DB, fresh for this subtest only</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">db</span><span class="p">.</span><span class="nf">Insert</span><span class="p">(</span><span class="s">&#34;order:1&#34;</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;widget&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">t</span><span class="p">.</span><span class="nf">Run</span><span class="p">(</span><span class="s">&#34;query order&#34;</span><span class="p">,</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// uses the group-level DB, so the state persists across subtests</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">schema</span><span class="p">.</span><span class="nf">Insert</span><span class="p">(</span><span class="s">&#34;order:1&#34;</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;widget&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">val</span><span class="p">,</span><span class="w"> </span><span class="nx">_</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">schema</span><span class="p">.</span><span class="nf">Query</span><span class="p">(</span><span class="s">&#34;order:1&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">if</span><span class="w"> </span><span class="nx">val</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="s">&#34;widget&#34;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">t</span><span class="p">.</span><span class="nf">Fatalf</span><span class="p">(</span><span class="s">&#34;expected widget, got %s&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">val</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>In this example, <code>TestMain</code> could be running a package-wide database server. The parent test
<code>TestOrders</code> sets up a schema that is shared across its subtests. Inside, one subtest spins
up its own per-test database to work in isolation, while another uses the shared schema to
test how state persists across operations.</p>
<p>The combination of package, group, and function scopes gives you flexibility: reuse
expensive resources when you need to, and isolate state when correctness depends on it.
However, combining scopes can be hard to reason about when you have many different subtests
under a single parent that are also interacting with some global state. I tend to avoid this
whenever possible.</p>
<h2 id="parting-words">Parting words</h2>
<p>Most of your setup and teardown should happen at the function level. That gives you the
strongest isolation and keeps each test self-contained.</p>
<p>The next most useful pattern is at the subtest group level, where you create a resource once
in a parent test and let its children share it. Cleanup runs when the parent finishes, which
makes sense when you really do want that shared state.</p>
<p>Package-level setup through <code>TestMain</code> should be rare. It is tempting when setup is
expensive, but global state is the fastest way to end up with brittle tests. Mixing
different scopes is possible, but usually creates more confusion than clarity, so reach for
it only when you have no better option.</p>
<!-- references -->
<!-- prettier-ignore-start -->
<!-- prettier-ignore-end -->
]]></content:encoded>
    </item>
    <item>
      <title>You probably don&#39;t need a DI framework</title>
      <link>https://rednafi.com/go/di-frameworks-bleh/</link>
      <pubDate>Sat, 24 May 2025 00:00:00 +0000</pubDate>
      <guid>https://rednafi.com/go/di-frameworks-bleh/</guid>
      <description>Dependency injection in Go doesn&amp;#39;t need Dig or Wire. Learn why manual wiring beats reflection magic and how Go&amp;#39;s design makes DI frameworks overkill.</description>
      <category>Go</category>
      <category>Testing</category>
      <category>API</category>
      <content:encoded><![CDATA[<p>When working with Go in an <a href="https://peter.bourgon.org/go-for-industrial-programming/" rel="noopener noreferrer" target="_blank">industrial programming</a> context, I feel like dependency
injection (DI) often gets a bad rep because of <em>DI frameworks</em>. But DI as a technique is
quite useful. It just tends to get explained with too many OO jargons and triggers PTSD
among those who came to Go to escape GoF theology.</p>
<blockquote>
  <p>Dependency Injection is a 25-dollar term for a 5-cent concept.</p>
<p>&ndash; James Shore</p>

</blockquote><p>DI basically means <em>passing values into a constructor instead of creating them inside it</em>.
That&rsquo;s really it. Observe:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">server</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">db</span><span class="w"> </span><span class="nx">DB</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// NewServer constructs a server instance</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">NewServer</span><span class="p">()</span><span class="w"> </span><span class="o">*</span><span class="nx">server</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">db</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">DB</span><span class="p">{}</span><span class="w">            </span><span class="c1">// The dependency is created here</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">server</span><span class="p">{</span><span class="nx">db</span><span class="p">:</span><span class="w"> </span><span class="nx">db</span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Here, <code>NewServer</code> creates its own <code>DB</code>. Instead, to inject the dependency, build <code>DB</code>
elsewhere and pass it in as a constructor parameter:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">NewServer</span><span class="p">(</span><span class="nx">db</span><span class="w"> </span><span class="nx">DB</span><span class="p">)</span><span class="w"> </span><span class="o">*</span><span class="nx">server</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">server</span><span class="p">{</span><span class="nx">db</span><span class="p">:</span><span class="w"> </span><span class="nx">db</span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Now the constructor no longer decides how a database is built; it simply <em>receives</em> one.</p>
<p>In Go, DI is often done using interfaces. You collate the behavior you care about in an
interface, and then provide different concrete implementations for different contexts. In
production, you pass a real implementation of <code>DB</code>. In unit tests, you pass a fake
implementation that behaves the same way from the caller&rsquo;s perspective but avoids real
database calls.</p>
<p>Here&rsquo;s how that looks:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// behaviour we care about</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">DB</span><span class="w"> </span><span class="kd">interface</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">Get</span><span class="p">(</span><span class="nx">id</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">Save</span><span class="p">(</span><span class="nx">id</span><span class="p">,</span><span class="w"> </span><span class="nx">value</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="kt">error</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">server</span><span class="w"> </span><span class="kd">struct</span><span class="p">{</span><span class="w"> </span><span class="nx">db</span><span class="w"> </span><span class="nx">DB</span><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// NewServer accepts a DB implementation and passes it to server</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">NewServer</span><span class="p">(</span><span class="nx">db</span><span class="w"> </span><span class="nx">DB</span><span class="p">)</span><span class="w"> </span><span class="o">*</span><span class="nx">server</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">server</span><span class="p">{</span><span class="nx">db</span><span class="p">:</span><span class="w"> </span><span class="nx">db</span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>A real implementation of <code>DB</code> might look like this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">RealDB</span><span class="w"> </span><span class="kd">struct</span><span class="p">{</span><span class="w"> </span><span class="nx">url</span><span class="w"> </span><span class="kt">string</span><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">NewDB</span><span class="p">(</span><span class="nx">url</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="o">*</span><span class="nx">RealDB</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">RealDB</span><span class="p">{</span><span class="nx">url</span><span class="p">:</span><span class="w"> </span><span class="nx">url</span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">RealDB</span><span class="p">)</span><span class="w"> </span><span class="nf">Get</span><span class="p">(</span><span class="nx">id</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// pretend we hit Postgres</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="s">&#34;real value&#34;</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">RealDB</span><span class="p">)</span><span class="w"> </span><span class="nf">Save</span><span class="p">(</span><span class="nx">id</span><span class="p">,</span><span class="w"> </span><span class="nx">value</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="kt">error</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>And a fake implementation for unit tests might be:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">FakeDB</span><span class="w"> </span><span class="kd">struct</span><span class="p">{</span><span class="w"> </span><span class="nx">data</span><span class="w"> </span><span class="kd">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="kt">string</span><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">NewFake</span><span class="p">()</span><span class="w"> </span><span class="o">*</span><span class="nx">FakeDB</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">FakeDB</span><span class="p">{</span><span class="nx">data</span><span class="p">:</span><span class="w"> </span><span class="kd">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="kt">string</span><span class="p">{}}</span><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">f</span><span class="w"> </span><span class="o">*</span><span class="nx">FakeDB</span><span class="p">)</span><span class="w"> </span><span class="nf">Get</span><span class="p">(</span><span class="nx">id</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">f</span><span class="p">.</span><span class="nx">data</span><span class="p">[</span><span class="nx">id</span><span class="p">],</span><span class="w"> </span><span class="kc">nil</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">f</span><span class="w"> </span><span class="o">*</span><span class="nx">FakeDB</span><span class="p">)</span><span class="w"> </span><span class="nf">Save</span><span class="p">(</span><span class="nx">id</span><span class="p">,</span><span class="w"> </span><span class="nx">value</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="kt">error</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">f</span><span class="p">.</span><span class="nx">data</span><span class="p">[</span><span class="nx">id</span><span class="p">]</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">value</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Use the fake in unit tests like so:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">TestServerGet</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">fake</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">NewFake</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">_</span><span class="w">    </span><span class="p">=</span><span class="w"> </span><span class="nx">fake</span><span class="p">.</span><span class="nf">Save</span><span class="p">(</span><span class="s">&#34;42&#34;</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;fake&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">srv</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">NewServer</span><span class="p">(</span><span class="nx">fake</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">val</span><span class="p">,</span><span class="w"> </span><span class="nx">_</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">srv</span><span class="p">.</span><span class="nx">db</span><span class="p">.</span><span class="nf">Get</span><span class="p">(</span><span class="s">&#34;42&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">val</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="s">&#34;fake&#34;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">t</span><span class="p">.</span><span class="nf">Fatalf</span><span class="p">(</span><span class="s">&#34;want fake, got %s&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">val</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>The compiler guarantees both <code>RealDB</code> and <code>FakeDB</code> satisfy <code>DB</code>, and during tests, we can
swap out the implementations without much ceremony.</p>
<h2 id="why-frameworks-turn-mild-annoyance-into-actual-pain">Why frameworks turn mild annoyance into actual pain</h2>
<p>Once <code>NewServer</code> grows half a dozen dependencies, wiring them by hand can feel noisy. That&rsquo;s
when a DI framework starts looking tempting.</p>
<p>With Uber&rsquo;s <a href="https://github.com/uber-go/dig" rel="noopener noreferrer" target="_blank">dig</a>, you register each constructor as a <em>provider</em>. <code>Provide</code> takes a
function, uses reflection to inspect its parameters and return type, and adds it as a node
in an internal dependency graph. Nothing is executed yet. Things only run when you call
<code>.Invoke()</code> on the container.</p>
<p>But that reflection-driven magic is also where the pain starts. As your graph grows, it gets
harder to tell which constructor feeds which one. Some constructor takes one parameter, some
takes three. There&rsquo;s no single place you can glance at to understand the wiring. It&rsquo;s all
figured out inside the container at runtime.</p>
<blockquote>
  <p>Let the container figure it out!</p>
<p>&ndash; every DI framework ever</p>

</blockquote><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">BuildContainer</span><span class="p">()</span><span class="w"> </span><span class="o">*</span><span class="nx">dig</span><span class="p">.</span><span class="nx">Container</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">c</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">dig</span><span class="p">.</span><span class="nf">New</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Each Provide call teaches dig about one node in the graph.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">c</span><span class="p">.</span><span class="nf">Provide</span><span class="p">(</span><span class="nx">NewConfig</span><span class="p">)</span><span class="w">     </span><span class="c1">// produces *Config</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">c</span><span class="p">.</span><span class="nf">Provide</span><span class="p">(</span><span class="nx">NewDB</span><span class="p">)</span><span class="w">         </span><span class="c1">// wants *Config, produces *DB</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">c</span><span class="p">.</span><span class="nf">Provide</span><span class="p">(</span><span class="nx">NewRepo</span><span class="p">)</span><span class="w">       </span><span class="c1">// wants *DB, produces *Repo</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">c</span><span class="p">.</span><span class="nf">Provide</span><span class="p">(</span><span class="nx">NewFlagClient</span><span class="p">)</span><span class="w"> </span><span class="c1">// produces *FlagClient</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">c</span><span class="p">.</span><span class="nf">Provide</span><span class="p">(</span><span class="nx">NewService</span><span class="p">)</span><span class="w">    </span><span class="c1">// wants *Repo, *FlagClient, produces *Service</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">c</span><span class="p">.</span><span class="nf">Provide</span><span class="p">(</span><span class="nx">NewServer</span><span class="p">)</span><span class="w">     </span><span class="c1">// wants *Service, produces *server</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">c</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Invoke starts the graph; dig sorts and calls constructors</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">BuildContainer</span><span class="p">().</span><span class="nf">Invoke</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="kd">func</span><span class="p">(</span><span class="nx">s</span><span class="w"> </span><span class="o">*</span><span class="nx">server</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">s</span><span class="p">.</span><span class="nf">Run</span><span class="p">()</span><span class="w"> </span><span class="p">});</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Now try commenting out <code>NewFlagClient</code>. The code <a href="https://go.dev/play/p/Vhimup7ukLo" rel="noopener noreferrer" target="_blank">still compiles</a>. There&rsquo;s no error until
runtime, when dig fails to construct <code>NewService</code> due to a missing dependency. And the error
message you get?</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-txt" data-lang="txt"><span class="line"><span class="cl">dig invoke failed: could not build arguments for function
</span></span><span class="line"><span class="cl">        main.main.func1 (prog.go:87)
</span></span><span class="line"><span class="cl">    : failed to build *main.Server
</span></span><span class="line"><span class="cl">    : could not build arguments for function main.NewServer (prog.go:65)
</span></span><span class="line"><span class="cl">    : failed to build *main.Service: missing dependencies for function
</span></span><span class="line"><span class="cl">        main.NewService (prog.go:55)
</span></span><span class="line"><span class="cl">    : missing type: *main.FlagClient
</span></span></code></pre></div><p>That&rsquo;s five stack frames deep, far from where the problem started. Now you&rsquo;re digging
through dig&rsquo;s internals to reconstruct the graph in your head.</p>
<p>Google&rsquo;s <a href="https://github.com/google/wire" rel="noopener noreferrer" target="_blank">wire</a> takes a different approach: it shifts the graph-building to <em>code
generation</em>. You collect your constructors in a <code>wire.NewSet</code>, call <code>wire.Build</code>, and the
generator writes a <code>wire_gen.go</code> that wires everything up explicitly.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">var</span><span class="w"> </span><span class="nx">serverSet</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">wire</span><span class="p">.</span><span class="nf">NewSet</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">NewConfig</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">NewDB</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">NewRepo</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">NewFlagClient</span><span class="p">,</span><span class="w"> </span><span class="c1">// comment out to see Wire complain</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">NewService</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">NewServer</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">InitializeServer</span><span class="p">()</span><span class="w"> </span><span class="p">(</span><span class="o">*</span><span class="nx">server</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">wire</span><span class="p">.</span><span class="nf">Build</span><span class="p">(</span><span class="nx">serverSet</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="c1">// replaced by generated code</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Comment out <code>NewFlagClient</code> and Wire fails earlier - during generation:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-txt" data-lang="txt"><span class="line"><span class="cl">wire: ../../service/wire.go:13:2: cannot find dependency for *flags.Client
</span></span></code></pre></div><p>It&rsquo;s better than dig&rsquo;s runtime panic, but still comes with its own headaches:</p>
<ul>
<li>You need to remember to run <code>go generate ./...</code> whenever constructor signatures change.</li>
<li>When something breaks, you&rsquo;re stuck reading through hundreds of lines of autogenerated
glue to trace the issue.</li>
<li>You have to teach every teammate Wire&rsquo;s DSL - <code>wire.NewSet</code>, <code>wire.Build</code>, build tags, and
sentinel rules. And if you ever switch to something different like dig, you&rsquo;ll need to
learn a completely different set of concepts: <code>Provide</code>, <code>Invoke</code>, scopes, named values,
etc.</li>
</ul>
<p>While DI frameworks tend to use vocabularies like <em>provider</em> or <em>container</em> to give you an
essense of familiarity, they still reinvent the API surface every time. Switching between
them means relearning a new mental model.</p>
<p>So the promise of &ldquo;just register your providers and forget about wiring&rdquo; ends up trading
clear, compile-time control for either reflection or hidden generator logic - and yet
another abstraction layer you have to debug.</p>
<h2 id="the-boring-alternative-keep-wiring-explicit">The boring alternative: keep wiring explicit</h2>
<p>In Go, you can just wire your own dependencies manually. Like this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">cfg</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">NewConfig</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">db</span><span class="w">    </span><span class="o">:=</span><span class="w"> </span><span class="nf">NewDB</span><span class="p">(</span><span class="nx">cfg</span><span class="p">.</span><span class="nx">DSN</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">repo</span><span class="w">  </span><span class="o">:=</span><span class="w"> </span><span class="nf">NewRepo</span><span class="p">(</span><span class="nx">db</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">flags</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">NewFlagClient</span><span class="p">(</span><span class="nx">cfg</span><span class="p">.</span><span class="nx">FlagURL</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">svc</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">NewService</span><span class="p">(</span><span class="nx">repo</span><span class="p">,</span><span class="w"> </span><span class="nx">flags</span><span class="p">,</span><span class="w"> </span><span class="nx">cfg</span><span class="p">.</span><span class="nx">APIKey</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">srv</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">NewServer</span><span class="p">(</span><span class="nx">svc</span><span class="p">,</span><span class="w"> </span><span class="nx">cfg</span><span class="p">.</span><span class="nx">ListenAddr</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">srv</span><span class="p">.</span><span class="nf">Run</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Longer? Yes. But:</p>
<ul>
<li>
<p>The call order is the dependency graph.</p>
</li>
<li>
<p>Errors are handled right where they happen.</p>
</li>
<li>
<p>If a constructor changes, the compiler points straight at every broken call:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-txt" data-lang="txt"><span class="line"><span class="cl">./main.go:33:39: not enough arguments in call to NewService
</span></span><span class="line"><span class="cl">    have (*Repo, *FlagClient)
</span></span><span class="line"><span class="cl">    want (*Repo, *FlagClient, string)
</span></span></code></pre></div></li>
</ul>
<p>No reflection, no generated code, no global state. Go type-checks the dependency graph early
and loudly, exactly how it should be. And also, it doesn&rsquo;t confuse your LSP, so your IDE
keeps on being useful.</p>
<p>If <code>main()</code> really grows unwieldy, split <em>your</em> code:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">buildInfra</span><span class="p">(</span><span class="nx">cfg</span><span class="w"> </span><span class="o">*</span><span class="nx">Config</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="o">*</span><span class="nx">DB</span><span class="p">,</span><span class="w"> </span><span class="o">*</span><span class="nx">FlagClient</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// ...</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">buildService</span><span class="p">(</span><span class="nx">cfg</span><span class="w"> </span><span class="o">*</span><span class="nx">Config</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="o">*</span><span class="nx">Service</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">db</span><span class="p">,</span><span class="w"> </span><span class="nx">flags</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">buildInfra</span><span class="p">(</span><span class="nx">cfg</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nf">NewService</span><span class="p">(</span><span class="nf">NewRepo</span><span class="p">(</span><span class="nx">db</span><span class="p">),</span><span class="w"> </span><span class="nx">flags</span><span class="p">,</span><span class="w"> </span><span class="nx">cfg</span><span class="p">.</span><span class="nx">APIKey</span><span class="p">),</span><span class="w"> </span><span class="kc">nil</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">cfg</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">NewConfig</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">svc</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">buildService</span><span class="p">(</span><span class="nx">cfg</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nf">Fatal</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">NewServer</span><span class="p">(</span><span class="nx">svc</span><span class="p">,</span><span class="w"> </span><span class="nx">cfg</span><span class="p">.</span><span class="nx">ListenAddr</span><span class="p">).</span><span class="nf">Run</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Each helper is a regular function that anyone can skim without reading a framework manual.
Also, you usually build all of your dependency in one place and it&rsquo;s really not that big of
a deal if your builder function takes in 20 parameters and builds all the dependencies. Just
put each function parameter on their own line and use gofumpt to format the code to make it
readable.</p>
<h2 id="reflection-works-elsewhere-so-why-not-here">Reflection works elsewhere, so why not here?</h2>
<p>Other languages lean on containers because often times constructors cannot be overloaded and
compile times hurt. Go already gives you:</p>
<ul>
<li>First-class functions so constructors are plain values.</li>
<li>Interfaces so implementations swap cleanly in tests.</li>
<li>Fast compilation so feedback loops stay tight.</li>
</ul>
<p>A DI framework often fixes problems Go already solved and trades away readability to do it.</p>
<blockquote>
  <p>The most magical thing about Go is how little magic it allows.</p>
<p>&ndash; Some Gopher on Reddit</p>

</blockquote><h2 id="you-might-still-want-a-framework">You might still want a framework</h2>
<p>It&rsquo;s tempting to make a blanket statement saying that you should <strong>never</strong> pick up a DI
framework, but context matters here.</p>
<p>I was watching <a href="https://www.youtube.com/watch?v=nLskCRJOdxM&t" rel="noopener noreferrer" target="_blank">Uber&rsquo;s GopherCon talk on Go at scale</a> and how their DI framework <a href="https://github.com/uber-go/fx" rel="noopener noreferrer" target="_blank">Fx</a> (which
uses dig underneath) allows them to achieve consistency at scale. If you&rsquo;re Uber and have
all the observability tools in place to get around the downsides, then you&rsquo;ll know.</p>
<p>Also, if you&rsquo;re working in a codebase that&rsquo;s already leveraging a framework and it works
well, then it doesn&rsquo;t make sense to refactor it without any incentives.</p>
<p>Or, you&rsquo;re writing one of those languages where using a DI framework is the norm, and you&rsquo;ll
be called a weirdo if you try to reinvent the wheel there.</p>
<p>However, in my experience, even in organizations that maintain a substantial number of Go
repos, DI frameworks add more confusion than they&rsquo;re worth. If your experience is otherwise,
I&rsquo;d love to be proven wrong.</p>
<hr>
<p>The post got a fair bit of discussion going around the web. You might find it interesting.</p>
<ul>
<li><a href="https://news.ycombinator.com/item?id=44086235" rel="noopener noreferrer" target="_blank">hackernews</a></li>
<li><a href="https://www.reddit.com/r/golang/comments/1kv0y1u/you_probably_dont_need_a_di_framework/" rel="noopener noreferrer" target="_blank">r/golang</a></li>
<li><a href="https://www.reddit.com/r/ExperiencedDevs/comments/1kv0y3n/you_probably_dont_need_a_di_framework/" rel="noopener noreferrer" target="_blank">r/experienceddevs</a></li>
<li><a href="https://www.reddit.com/r/programming/comments/1kv0y2l/you_probably_dont_need_a_di_framework/" rel="noopener noreferrer" target="_blank">r/programming</a></li>
</ul>
<!-- references -->
<!-- prettier-ignore-start -->
<!-- prettier-ignore-end -->
]]></content:encoded>
    </item>
    <item>
      <title>Deferred teardown closure in Go testing</title>
      <link>https://rednafi.com/go/deferred-teardown-closure/</link>
      <pubDate>Fri, 28 Mar 2025 00:00:00 +0000</pubDate>
      <guid>https://rednafi.com/go/deferred-teardown-closure/</guid>
      <description>Return teardown closures from test helpers to manage cleanup elegantly. Learn patterns for temp files, mock servers, and t.Cleanup() usage.</description>
      <category>Go</category>
      <category>Testing</category>
      <category>API</category>
      <content:encoded><![CDATA[<p>While watching <a href="https://www.youtube.com/watch?v=8hQG7QlcLBk" rel="noopener noreferrer" target="_blank">Mitchell Hashimoto&rsquo;s Advanced Testing with Go talk</a>, I came across this neat
technique for deferring teardown to the caller. Let&rsquo;s say you have a helper function in a
test that needs to perform some cleanup afterward.</p>
<p>You can&rsquo;t run the teardown inside the helper itself because the test still needs the setup.
For example, in the following case, the <code>helper</code> runs its teardown immediately:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">TestFoo</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">helper</span><span class="p">(</span><span class="nx">t</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Test logic here: resources may already be cleaned up!</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">helper</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">t</span><span class="p">.</span><span class="nf">Helper</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Setup code here.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Teardown code here.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">defer</span><span class="w"> </span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// Clean up something.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>When <code>helper</code> is called, it defers its teardown - which executes at the end of the helper
function, not the test. But the test logic still depends on whatever the helper set up. So
this approach doesn&rsquo;t work.</p>
<p>The next working option is to move the teardown logic into the test itself:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">TestFoo</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">helper</span><span class="p">(</span><span class="nx">t</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Run the teardown of helper.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">defer</span><span class="w"> </span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// Clean up something.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Test logic here.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">helper</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">t</span><span class="p">.</span><span class="nf">Helper</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Setup code here.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// No teardown here; we move it to the caller.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>This works fine if you have only one helper. But with multiple helpers, it quickly becomes
messy - you now have to manage multiple teardown calls manually, like this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">TestFoo</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">helper1</span><span class="p">(</span><span class="nx">t</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">helper2</span><span class="p">(</span><span class="nx">t</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">defer</span><span class="w"> </span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// Clean up helper2.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">defer</span><span class="w"> </span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// Clean up helper1.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Test logic here.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>You also need to be careful with the order: <code>defer</code> statements are executed in LIFO
(last-in, first-out) order. So if teardown order matters, this can be a problem. Ideally,
your tests shouldn&rsquo;t depend on teardown order - but sometimes they do.</p>
<p>So rather than manually handling cleanup inside the test, have helpers return a teardown
function that the test can <code>defer</code> itself. Here&rsquo;s how:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">TestFoo</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">teardown1</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">helper1</span><span class="p">(</span><span class="nx">t</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">defer</span><span class="w"> </span><span class="nf">teardown1</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">teardown2</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">helper2</span><span class="p">(</span><span class="nx">t</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">defer</span><span class="w"> </span><span class="nf">teardown2</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Test logic here.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">helper1</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">t</span><span class="p">.</span><span class="nf">Helper</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Setup code here.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Maybe create a temp dir, start a mock server, etc.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// Teardown code here.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">helper2</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">t</span><span class="p">.</span><span class="nf">Helper</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Setup code here.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// Teardown code here.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Each helper is self-contained: it sets something up and returns a function to clean up
whatever resource it has spun up. The test controls when teardown happens by calling the
cleanup function at the appropriate time. Another benefit is that the returned teardown
closure has access to the local variables of the helper. So <code>func()</code> can access the helper&rsquo;s
<code>*testing.T</code> without us having to pass it explicitly as a parameter.</p>
<p>Here&rsquo;s how I&rsquo;ve been using this pattern.</p>
<h2 id="creating-a-temporary-file-to-test-file-io">Creating a temporary file to test file I/O</h2>
<p>The <code>setupTempFile</code> helper creates a temporary file, writes some content to it, and returns
the file name along with a teardown function that removes the file.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">setupTempFile</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">,</span><span class="w"> </span><span class="nx">content</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="kd">func</span><span class="p">())</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">t</span><span class="p">.</span><span class="nf">Helper</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">tmpFile</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">os</span><span class="p">.</span><span class="nf">CreateTemp</span><span class="p">(</span><span class="s">&#34;&#34;</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;temp-*.txt&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">t</span><span class="p">.</span><span class="nf">Fatalf</span><span class="p">(</span><span class="s">&#34;failed to create temp file: %v&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">tmpFile</span><span class="p">.</span><span class="nf">WriteString</span><span class="p">(</span><span class="nx">content</span><span class="p">);</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">t</span><span class="p">.</span><span class="nf">Fatalf</span><span class="p">(</span><span class="s">&#34;failed to write to temp file: %v&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">tmpFile</span><span class="p">.</span><span class="nf">Close</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">tmpFile</span><span class="p">.</span><span class="nf">Name</span><span class="p">(),</span><span class="w"> </span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">os</span><span class="p">.</span><span class="nf">Remove</span><span class="p">(</span><span class="nx">tmpFile</span><span class="p">.</span><span class="nf">Name</span><span class="p">());</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">t</span><span class="p">.</span><span class="nf">Errorf</span><span class="p">(</span><span class="s">&#34;failed to remove temp file %s: %v&#34;</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="nx">tmpFile</span><span class="p">.</span><span class="nf">Name</span><span class="p">(),</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">t</span><span class="p">.</span><span class="nf">Logf</span><span class="p">(</span><span class="s">&#34;cleaned up temp file: %s&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">tmpFile</span><span class="p">.</span><span class="nf">Name</span><span class="p">())</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>In the main test:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">TestReadFile</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">path</span><span class="p">,</span><span class="w"> </span><span class="nx">cleanup</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">setupTempFile</span><span class="p">(</span><span class="nx">t</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;hello world&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">defer</span><span class="w"> </span><span class="nf">cleanup</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">data</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">os</span><span class="p">.</span><span class="nf">ReadFile</span><span class="p">(</span><span class="nx">path</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">t</span><span class="p">.</span><span class="nf">Fatalf</span><span class="p">(</span><span class="s">&#34;failed to read file: %v&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">t</span><span class="p">.</span><span class="nf">Logf</span><span class="p">(</span><span class="s">&#34;file contents: %s&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">data</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Running the test displays:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-txt" data-lang="txt"><span class="line"><span class="cl">=== RUN   TestReadFile
</span></span><span class="line"><span class="cl">    prog_test.go:18: file contents: hello world
</span></span><span class="line"><span class="cl">    prog_test.go:38: cleaned up temp file: /tmp/temp-30176446.txt
</span></span><span class="line"><span class="cl">--- PASS: TestReadFile (0.00s)
</span></span><span class="line"><span class="cl">PASS
</span></span></code></pre></div><h2 id="starting-and-stopping-a-mock-http-server">Starting and stopping a mock HTTP server</h2>
<p>Sometimes you want to test code that makes HTTP calls. Here&rsquo;s a helper that starts an
in-memory mock server and returns its URL and a cleanup function that shuts it down:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">setupMockServer</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="kd">func</span><span class="p">())</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">t</span><span class="p">.</span><span class="nf">Helper</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">handler</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nf">HandlerFunc</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="kd">func</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">w</span><span class="p">.</span><span class="nf">WriteHeader</span><span class="p">(</span><span class="nx">http</span><span class="p">.</span><span class="nx">StatusOK</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">w</span><span class="p">.</span><span class="nf">Write</span><span class="p">([]</span><span class="nb">byte</span><span class="p">(</span><span class="s">&#34;mock response&#34;</span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">},</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">server</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">httptest</span><span class="p">.</span><span class="nf">NewServer</span><span class="p">(</span><span class="nx">handler</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">server</span><span class="p">.</span><span class="nx">URL</span><span class="p">,</span><span class="w"> </span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">server</span><span class="p">.</span><span class="nf">Close</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">t</span><span class="p">.</span><span class="nf">Log</span><span class="p">(</span><span class="s">&#34;mock server shut down&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>And in the test:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">TestHTTPRequest</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">url</span><span class="p">,</span><span class="w"> </span><span class="nx">cleanup</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">setupMockServer</span><span class="p">(</span><span class="nx">t</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">defer</span><span class="w"> </span><span class="nf">cleanup</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">resp</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nf">Get</span><span class="p">(</span><span class="nx">url</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">t</span><span class="p">.</span><span class="nf">Fatalf</span><span class="p">(</span><span class="s">&#34;failed to make HTTP request: %v&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">defer</span><span class="w"> </span><span class="nx">resp</span><span class="p">.</span><span class="nx">Body</span><span class="p">.</span><span class="nf">Close</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">body</span><span class="p">,</span><span class="w"> </span><span class="nx">_</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">io</span><span class="p">.</span><span class="nf">ReadAll</span><span class="p">(</span><span class="nx">resp</span><span class="p">.</span><span class="nx">Body</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">t</span><span class="p">.</span><span class="nf">Logf</span><span class="p">(</span><span class="s">&#34;response body: %s&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">body</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Running the test prints:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-txt" data-lang="txt"><span class="line"><span class="cl">=== RUN   TestHTTPRequest
</span></span><span class="line"><span class="cl">    prog_test.go:34: response body: mock response
</span></span><span class="line"><span class="cl">    prog_test.go:20: mock server shut down
</span></span><span class="line"><span class="cl">--- PASS: TestHTTPRequest (0.00s)
</span></span><span class="line"><span class="cl">PASS
</span></span></code></pre></div><h2 id="setting-up-and-tearing-down-a-database-table">Setting up and tearing down a database table</h2>
<p>In tests that hit a real (or test) database, you often need to create and drop tables.
Here&rsquo;s a helper that sets up a test table and returns a teardown function to drop it:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">setupTestTable</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">,</span><span class="w"> </span><span class="nx">db</span><span class="w"> </span><span class="o">*</span><span class="nx">sql</span><span class="p">.</span><span class="nx">DB</span><span class="p">)</span><span class="w"> </span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">t</span><span class="p">.</span><span class="nf">Helper</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">query</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="s">`CREATE TABLE IF NOT EXISTS users (
</span></span></span><span class="line"><span class="cl"><span class="s">        id INTEGER PRIMARY KEY,
</span></span></span><span class="line"><span class="cl"><span class="s">        name TEXT
</span></span></span><span class="line"><span class="cl"><span class="s">    )`</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">db</span><span class="p">.</span><span class="nf">Exec</span><span class="p">(</span><span class="nx">query</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">t</span><span class="p">.</span><span class="nf">Fatalf</span><span class="p">(</span><span class="s">&#34;failed to create table: %v&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">db</span><span class="p">.</span><span class="nf">Exec</span><span class="p">(</span><span class="s">`DROP TABLE IF EXISTS users`</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">t</span><span class="p">.</span><span class="nf">Errorf</span><span class="p">(</span><span class="s">&#34;failed to drop table: %v&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">t</span><span class="p">.</span><span class="nf">Log</span><span class="p">(</span><span class="s">&#34;dropped test table&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>And the test:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">TestInsertUser</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">db</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">getTestDB</span><span class="p">(</span><span class="nx">t</span><span class="p">)</span><span class="w"> </span><span class="c1">// Opens test DB; defined elsewhere</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">cleanup</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">setupTestTable</span><span class="p">(</span><span class="nx">t</span><span class="p">,</span><span class="w"> </span><span class="nx">db</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">defer</span><span class="w"> </span><span class="nf">cleanup</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">db</span><span class="p">.</span><span class="nf">Exec</span><span class="p">(</span><span class="s">`INSERT INTO users (name) VALUES (?)`</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;Alice&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">t</span><span class="p">.</span><span class="nf">Fatalf</span><span class="p">(</span><span class="s">&#34;failed to insert user: %v&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><h2 id="the-tcleanup-method">The t.Cleanup() method</h2>
<p><em>P.S. I learned about this after the blog went live.</em></p>
<p>Go 1.14 added the <code>t.Cleanup()</code> method, which lets you avoid returning the teardown closures
from helper functions altogether. It also runs the cleanup logic in the correct order
(LIFO). So, you could rewrite the first example in this post as follows:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">TestFoo</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// The testing package will ensure that the cleanup runs at the end of</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// this test function.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">helper</span><span class="p">(</span><span class="nx">t</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Test logic here.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">helper</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">t</span><span class="p">.</span><span class="nf">Helper</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// We register the teardown logic with t.Cleanup().</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">t</span><span class="p">.</span><span class="nf">Cleanup</span><span class="p">(</span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// Teardown logic here.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Now the <code>testing</code> package will handle calling the cleanup logic in the correct order. You
can add multiple teardown functions like this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">t</span><span class="p">.</span><span class="nf">Cleanup</span><span class="p">(</span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="p">{})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">t</span><span class="p">.</span><span class="nf">Cleanup</span><span class="p">(</span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="p">{})</span><span class="w">
</span></span></span></code></pre></div><p>The functions will run in LIFO order. Similarly, the database setup example can be rewritten
like this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">setupTestTable</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">,</span><span class="w"> </span><span class="nx">db</span><span class="w"> </span><span class="o">*</span><span class="nx">sql</span><span class="p">.</span><span class="nx">DB</span><span class="p">)</span><span class="w"> </span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">t</span><span class="p">.</span><span class="nf">Helper</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Logic as before.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Instead of returning the teardown function, we register</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// it with t.Cleanup().</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">t</span><span class="p">.</span><span class="nf">Cleanup</span><span class="p">(</span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">db</span><span class="p">.</span><span class="nf">Exec</span><span class="p">(</span><span class="s">`DROP TABLE IF EXISTS users`</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">t</span><span class="p">.</span><span class="nf">Errorf</span><span class="p">(</span><span class="s">&#34;failed to drop table: %v&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">t</span><span class="p">.</span><span class="nf">Log</span><span class="p">(</span><span class="s">&#34;dropped test table&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Then the helper function is used like this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">TestInsertUser</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">db</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">getTestDB</span><span class="p">(</span><span class="nx">t</span><span class="p">)</span><span class="w"> </span><span class="c1">// Opens a test DB connection; defined elsewhere.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// This sets up the DB, and t.Cleanup will execute the teardown</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// logic once this test function finishes.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">setupTestTable</span><span class="p">(</span><span class="nx">t</span><span class="p">,</span><span class="w"> </span><span class="nx">db</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Rest of the test logic.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Fin!</p>
<!-- references -->
<!-- prettier-ignore-start -->
<!-- prettier-ignore-end -->
]]></content:encoded>
    </item>
    <item>
      <title>Stacked middleware vs embedded delegation in Go</title>
      <link>https://rednafi.com/go/middleware-vs-delegation/</link>
      <pubDate>Thu, 06 Mar 2025 00:00:00 +0000</pubDate>
      <guid>https://rednafi.com/go/middleware-vs-delegation/</guid>
      <description>Compare middleware stacking with embedded delegation in Go HTTP servers. Learn when to override ServeHTTP for simpler request handling.</description>
      <category>Go</category>
      <category>API</category>
      <category>Web</category>
      <content:encoded><![CDATA[<p>Middleware is usually the go-to pattern in Go HTTP servers for tweaking request behavior.
Typically, you wrap your base handler with layers of middleware - one might log every
request, while another intercepts specific routes like <code>/special</code> to serve a custom
response.</p>
<p>However, I often find the indirections introduced by this pattern a bit hard to read and
debug. I recently came across the embedded delegation pattern while browsing <a href="https://github.com/gin-gonic/gin/blob/3b28645dc95d58e0df36b8aff7a6c64f7c0ca5e9/gin.go#L94" rel="noopener noreferrer" target="_blank">Gin&rsquo;s HTTP
router source code</a>. Here, I explore both patterns and explain why I usually start with
delegation whenever I need to modify HTTP requests in my Go services.</p>
<h2 id="middleware-stacking">Middleware stacking</h2>
<p>Here&rsquo;s an example where the logging middleware records each request, and the special
middleware intercepts requests to <code>/special</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kn">package</span><span class="w"> </span><span class="nx">main</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kn">import</span><span class="w"> </span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="s">&#34;log&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="s">&#34;net/http&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// loggingMiddleware logs incoming requests.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">loggingMiddleware</span><span class="p">(</span><span class="nx">next</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">Handler</span><span class="p">)</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">Handler</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nf">HandlerFunc</span><span class="p">(</span><span class="kd">func</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">log</span><span class="p">.</span><span class="nf">Println</span><span class="p">(</span><span class="s">&#34;Middleware: received request for&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="p">.</span><span class="nx">URL</span><span class="p">.</span><span class="nx">Path</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">next</span><span class="p">.</span><span class="nf">ServeHTTP</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// specialMiddleware intercepts requests for &#34;/special&#34; and handles them.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">specialMiddleware</span><span class="p">(</span><span class="nx">next</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">Handler</span><span class="p">)</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">Handler</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nf">HandlerFunc</span><span class="p">(</span><span class="kd">func</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">if</span><span class="w"> </span><span class="nx">r</span><span class="p">.</span><span class="nx">URL</span><span class="p">.</span><span class="nx">Path</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="s">&#34;/special&#34;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">w</span><span class="p">.</span><span class="nf">Write</span><span class="p">([]</span><span class="nb">byte</span><span class="p">(</span><span class="s">&#34;Special middleware handling request&#34;</span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="k">return</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">next</span><span class="p">.</span><span class="nf">ServeHTTP</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">mux</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nf">NewServeMux</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">mux</span><span class="p">.</span><span class="nf">HandleFunc</span><span class="p">(</span><span class="s">&#34;/&#34;</span><span class="p">,</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">w</span><span class="p">.</span><span class="nf">Write</span><span class="p">([]</span><span class="nb">byte</span><span class="p">(</span><span class="s">&#34;Hello, world!&#34;</span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Middleware chain: special handling then logging.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">handler</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">loggingMiddleware</span><span class="p">(</span><span class="nf">specialMiddleware</span><span class="p">(</span><span class="nx">mux</span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">http</span><span class="p">.</span><span class="nf">ListenAndServe</span><span class="p">(</span><span class="s">&#34;:8080&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">handler</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>In this setup, every incoming request is first handled by the special middleware, which
checks for the <code>/special</code> route, and then by the logging middleware that logs the request
details. We&rsquo;re effectively stacking the middleware functions.</p>
<p>If you hit the server with:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl">curl localhost:8080/
</span></span><span class="line"><span class="cl">curl localhost:8080/special
</span></span></code></pre></div><p>the server logs will look like this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-txt" data-lang="txt"><span class="line"><span class="cl">2025/03/06 21:24:44 Middleware: received request for /
</span></span><span class="line"><span class="cl">2025/03/06 21:24:47 Middleware: received request for /special
</span></span></code></pre></div><p>Stacking middleware functions like <code>middleware3(middleware2(middleware1(mux)))</code> can get
messy when you have many of them. That&rsquo;s why people usually write a wrapper function to
apply the middlewares to the mux:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">applyMiddleware</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">handler</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">Handler</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">middlewares</span><span class="w"> </span><span class="o">...</span><span class="kd">func</span><span class="p">(</span><span class="nx">http</span><span class="p">.</span><span class="nx">Handler</span><span class="p">)</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">Handler</span><span class="p">)</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">Handler</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Apply middlewares in reverse order to preserve LIFO.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">for</span><span class="w"> </span><span class="nx">i</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nb">len</span><span class="p">(</span><span class="nx">middlewares</span><span class="p">)</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span><span class="w"> </span><span class="nx">i</span><span class="w"> </span><span class="o">&gt;=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"> </span><span class="nx">i</span><span class="o">--</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">handler</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">middlewares</span><span class="p">[</span><span class="nx">i</span><span class="p">](</span><span class="nx">handler</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">handler</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p><code>applyMiddleware</code> takes an <code>http.Handler</code> and a variadic list of middleware functions
(<code>...func(http.Handler) http.Handler</code>). It loops over the middleware in reverse order so
each one wraps the next properly. This avoids deep nesting like
<code>middleware3(middleware2(middleware1(mux)))</code> and keeps the middleware chain tidy.</p>
<p>You&rsquo;d then use it like this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">mux</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nf">NewServeMux</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">mux</span><span class="p">.</span><span class="nf">HandleFunc</span><span class="p">(</span><span class="s">&#34;/&#34;</span><span class="p">,</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">w</span><span class="p">.</span><span class="nf">Write</span><span class="p">([]</span><span class="nb">byte</span><span class="p">(</span><span class="s">&#34;Hello, world!&#34;</span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Middleware chain: special handling then logging.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// specialMiddleware is applied before loggingMiddleware.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">handler</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">applyMiddleware</span><span class="p">(</span><span class="nx">mux</span><span class="p">,</span><span class="w"> </span><span class="nx">loggingMiddleware</span><span class="p">,</span><span class="w"> </span><span class="nx">specialMiddleware</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">http</span><span class="p">.</span><span class="nf">ListenAndServe</span><span class="p">(</span><span class="s">&#34;:8080&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">handler</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>This behaves just like the manual middleware stacking, but it&rsquo;s a bit cleaner.</p>
<p>While this is the canonical way to handle request-response modifications in Go, it can
sometimes be hard to reason about, especially when debugging or dealing with many middleware
layers.</p>
<p>There&rsquo;s another way to achieve the same result without dealing with a soup of nested
functions. The next section talks about that.</p>
<h2 id="embedded-delegation">Embedded delegation</h2>
<p>Embedded delegation (or the delegation pattern) means you embed the standard HTTP
multiplexer inside your own struct and override its <code>ServeHTTP</code> method.</p>
<p>It&rsquo;s a bit like inheritance - overriding a method in a subclass to add extra functionality
and then delegating the call to the original method. Although Go doesn&rsquo;t have a class
hierarchy, you can still delegate responsibilities to the embedded type&rsquo;s method.</p>
<p>The following example implements the same behavior - logging every request and intercepting
the <code>/special</code> route - directly within a custom mux:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kn">package</span><span class="w"> </span><span class="nx">main</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kn">import</span><span class="w"> </span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="s">&#34;log&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="s">&#34;net/http&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// CustomMux embeds http.ServeMux to override ServeHTTP.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">CustomMux</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">ServeMux</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// ServeHTTP logs the request and intercepts &#34;/special&#34; before</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// delegating to the embedded mux.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">cm</span><span class="w"> </span><span class="o">*</span><span class="nx">CustomMux</span><span class="p">)</span><span class="w"> </span><span class="nf">ServeHTTP</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Log all requests.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">log</span><span class="p">.</span><span class="nf">Println</span><span class="p">(</span><span class="s">&#34;CustomMux: received request for&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="p">.</span><span class="nx">URL</span><span class="p">.</span><span class="nx">Path</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Handle &#34;/special&#34; differently.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">r</span><span class="p">.</span><span class="nx">URL</span><span class="p">.</span><span class="nx">Path</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="s">&#34;/special&#34;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">w</span><span class="p">.</span><span class="nf">Write</span><span class="p">([]</span><span class="nb">byte</span><span class="p">(</span><span class="s">&#34;Special handling in CustomMux&#34;</span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">return</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">cm</span><span class="p">.</span><span class="nx">ServeMux</span><span class="p">.</span><span class="nf">ServeHTTP</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">mux</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nf">NewServeMux</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">mux</span><span class="p">.</span><span class="nf">HandleFunc</span><span class="p">(</span><span class="s">&#34;/&#34;</span><span class="p">,</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">w</span><span class="p">.</span><span class="nf">Write</span><span class="p">([]</span><span class="nb">byte</span><span class="p">(</span><span class="s">&#34;Hello, world!&#34;</span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Wrap the standard mux with our custom delegation.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">customMux</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">CustomMux</span><span class="p">{</span><span class="nx">ServeMux</span><span class="p">:</span><span class="w"> </span><span class="nx">mux</span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">http</span><span class="p">.</span><span class="nf">ListenAndServe</span><span class="p">(</span><span class="s">&#34;:8080&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">customMux</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>In this example, the custom mux centralizes both logging and special-case route handling
within one <code>ServeHTTP</code> method. This approach cuts out the extra function calls in a
middleware chain and can simplify tracking the request flow. I find it a bit easier on the
eyes too.</p>
<p>If you have a bunch of extra functionality to add inside <code>cm.ServeHTTP</code>, you can wrap them
in utility functions like this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// logRequest logs incoming HTTP requests.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">logRequest</span><span class="p">(</span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">log</span><span class="p">.</span><span class="nf">Println</span><span class="p">(</span><span class="s">&#34;CustomMux: received request for&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="p">.</span><span class="nx">URL</span><span class="p">.</span><span class="nx">Path</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// handleSpecialRequest handles requests to &#34;/special&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// and returns true if handled.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">handleSpecialRequest</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="kt">bool</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">r</span><span class="p">.</span><span class="nx">URL</span><span class="p">.</span><span class="nx">Path</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="s">&#34;/special&#34;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">return</span><span class="w"> </span><span class="kc">false</span><span class="w"> </span><span class="c1">// Not handled, continue processing.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">w</span><span class="p">.</span><span class="nf">Write</span><span class="p">([]</span><span class="nb">byte</span><span class="p">(</span><span class="s">&#34;Special handling in CustomMux&#34;</span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="kc">true</span><span class="w"> </span><span class="c1">// Handled; no further processing needed.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Then, simply call these functions inside your <code>cm.ServeHTTP</code> method:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">cm</span><span class="w"> </span><span class="o">*</span><span class="nx">CustomMux</span><span class="p">)</span><span class="w"> </span><span class="nf">ServeHTTP</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">logRequest</span><span class="p">(</span><span class="nx">r</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nf">handleSpecialRequest</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">return</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">cm</span><span class="p">.</span><span class="nx">ServeMux</span><span class="p">.</span><span class="nf">ServeHTTP</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>This keeps all the request modifications in a single <code>ServeHTTP</code> method.</p>
<h2 id="mixing-the-two-approaches">Mixing the two approaches</h2>
<p>You can also mix both techniques. For example, you might use direct delegation for special
route handling and then wrap the resulting handler with middleware for logging. Here&rsquo;s how a
hybrid solution might look:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kn">package</span><span class="w"> </span><span class="nx">main</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kn">import</span><span class="w"> </span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="s">&#34;log&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="s">&#34;net/http&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// CustomMux embeds http.ServeMux and intercepts &#34;/special&#34;.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">CustomMux</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">ServeMux</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// ServeHTTP intercepts &#34;/special&#34; and delegates other routes.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">cm</span><span class="w"> </span><span class="o">*</span><span class="nx">CustomMux</span><span class="p">)</span><span class="w"> </span><span class="nf">ServeHTTP</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">r</span><span class="p">.</span><span class="nx">URL</span><span class="p">.</span><span class="nx">Path</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="s">&#34;/special&#34;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">w</span><span class="p">.</span><span class="nf">Write</span><span class="p">([]</span><span class="nb">byte</span><span class="p">(</span><span class="s">&#34;Special handling in CustomMux&#34;</span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">return</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">cm</span><span class="p">.</span><span class="nx">ServeMux</span><span class="p">.</span><span class="nf">ServeHTTP</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// loggingMiddleware logs incoming requests.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">loggingMiddleware</span><span class="p">(</span><span class="nx">next</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">Handler</span><span class="p">)</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">Handler</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nf">HandlerFunc</span><span class="p">(</span><span class="kd">func</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">log</span><span class="p">.</span><span class="nf">Println</span><span class="p">(</span><span class="s">&#34;Middleware: received request for&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="p">.</span><span class="nx">URL</span><span class="p">.</span><span class="nx">Path</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">next</span><span class="p">.</span><span class="nf">ServeHTTP</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">mux</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nf">NewServeMux</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">mux</span><span class="p">.</span><span class="nf">HandleFunc</span><span class="p">(</span><span class="s">&#34;/&#34;</span><span class="p">,</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">w</span><span class="p">.</span><span class="nf">Write</span><span class="p">([]</span><span class="nb">byte</span><span class="p">(</span><span class="s">&#34;Hello, world!&#34;</span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Use direct delegation for special routing.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">customMux</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">CustomMux</span><span class="p">{</span><span class="nx">ServeMux</span><span class="p">:</span><span class="w"> </span><span class="nx">mux</span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Wrap the custom mux with logging middleware.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">handler</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">loggingMiddleware</span><span class="p">(</span><span class="nx">customMux</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">http</span><span class="p">.</span><span class="nf">ListenAndServe</span><span class="p">(</span><span class="s">&#34;:8080&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">handler</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>In this hybrid approach, the specialized behavior (intercepting the <code>/special</code> path) is
handled via direct delegation, while logging stays modular as middleware. This gives you the
best of both worlds.</p>
<p>I usually start with the embedded delegation and gradually introduce the middleware pattern
if I need it later. It&rsquo;s easier to adopt the middleware pattern if you start with delegation
than the other way around.</p>
<!-- references -->
<!-- prettier-ignore-start -->
<!-- prettier-ignore-end -->
]]></content:encoded>
    </item>
    <item>
      <title>Function types and single-method interfaces in Go</title>
      <link>https://rednafi.com/go/func-types-and-smis/</link>
      <pubDate>Sun, 22 Dec 2024 00:00:00 +0000</pubDate>
      <guid>https://rednafi.com/go/func-types-and-smis/</guid>
      <description>Implement single-method interfaces with function types instead of structs. Master http.HandlerFunc patterns for middlewares, mocks, and adapters.</description>
      <category>Go</category>
      <category>API</category>
      <category>Testing</category>
      <content:encoded><![CDATA[<p>People love single-method interfaces (SMIs) in Go. They&rsquo;re simple to implement and easy to
reason about. The standard library is packed with SMIs like <code>io.Reader</code>, <code>io.Writer</code>,
<code>io.Closer</code>, <code>io.Seeker</code>, and more.</p>
<p>One cool thing about SMIs is that you don&rsquo;t always need to create a full-blown struct with a
method to satisfy the interface. You can define a function type, attach the interface method
to it, and use it right away. This approach works well when there&rsquo;s no state to maintain, so
the extra struct becomes unnecessary. However, I find the syntax for this a bit abstruse.
So, I&rsquo;m jotting down a few examples here to reference later.</p>
<h2 id="using-a-struct-to-implement-an-interface">Using a struct to implement an interface</h2>
<p>This is how interfaces are typically implemented. Here, we&rsquo;ll satisfy the <code>io.Writer</code>
interface to create a writer that logs some stats before saving data to an in-memory buffer.</p>
<p>The standard library defines <code>io.Writer</code> like this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">Writer</span><span class="w"> </span><span class="kd">interface</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">Write</span><span class="p">(</span><span class="nx">p</span><span class="w"> </span><span class="p">[]</span><span class="kt">byte</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nx">n</span><span class="w"> </span><span class="kt">int</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>We can implement <code>io.Writer</code> by defining a struct type, <code>LoggingWriter</code>, and attaching a
<code>Write</code> method with the required signature:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// LoggingWriter writes data to an underlying writer and logs stats.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">LoggingWriter</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">w</span><span class="w"> </span><span class="nx">io</span><span class="p">.</span><span class="nx">Writer</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">lw</span><span class="w"> </span><span class="o">*</span><span class="nx">LoggingWriter</span><span class="p">)</span><span class="w"> </span><span class="nf">Write</span><span class="p">(</span><span class="nx">data</span><span class="w"> </span><span class="p">[]</span><span class="kt">byte</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">int</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Printf</span><span class="p">(</span><span class="s">&#34;LoggingWriter: Writing %d bytes\n&#34;</span><span class="p">,</span><span class="w"> </span><span class="nb">len</span><span class="p">(</span><span class="nx">data</span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">lw</span><span class="p">.</span><span class="nx">w</span><span class="p">.</span><span class="nf">Write</span><span class="p">(</span><span class="nx">data</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Here&rsquo;s how to use it:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">var</span><span class="w"> </span><span class="nx">buf</span><span class="w"> </span><span class="nx">bytes</span><span class="p">.</span><span class="nx">Buffer</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">logWriter</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">LoggingWriter</span><span class="p">{</span><span class="nx">w</span><span class="p">:</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">buf</span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">logWriter</span><span class="p">.</span><span class="nf">Write</span><span class="p">([]</span><span class="nb">byte</span><span class="p">(</span><span class="s">&#34;Hello, world!&#34;</span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Println</span><span class="p">(</span><span class="s">&#34;Error writing data:&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">return</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Println</span><span class="p">(</span><span class="s">&#34;Buffer content:&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">buf</span><span class="p">.</span><span class="nf">String</span><span class="p">())</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Running this will log the stats before writing to the buffer:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-txt" data-lang="txt"><span class="line"><span class="cl">LoggingWriter: Writing 13 bytes
</span></span><span class="line"><span class="cl">Buffer content: Hello, world!
</span></span></code></pre></div><h2 id="using-a-function-type-instead">Using a function type instead</h2>
<p>Instead of defining the <code>LoggingWriter</code> struct, you can use a function type to satisfy
<code>io.Writer</code>. This works well for SMIs but doesn&rsquo;t make sense for interfaces with multiple
methods. In those cases, we need to resort to the methods-on-struct approach.</p>
<p>Here&rsquo;s how it looks:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// WriteFunc is a function type that implements io.Writer.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">WriteFunc</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="nx">data</span><span class="w"> </span><span class="p">[]</span><span class="kt">byte</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">int</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Write makes WriteFunc satisfy io.Writer.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">wf</span><span class="w"> </span><span class="nx">WriteFunc</span><span class="p">)</span><span class="w"> </span><span class="nf">Write</span><span class="p">(</span><span class="nx">data</span><span class="w"> </span><span class="p">[]</span><span class="kt">byte</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">int</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nf">wf</span><span class="p">(</span><span class="nx">data</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>You can use <code>WriteFunc</code> like this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">var</span><span class="w"> </span><span class="nx">buf</span><span class="w"> </span><span class="nx">bytes</span><span class="p">.</span><span class="nx">Buffer</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Define a WriteFunc to log stats and write data.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">logWriter</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">WriteFunc</span><span class="p">(</span><span class="kd">func</span><span class="p">(</span><span class="nx">data</span><span class="w"> </span><span class="p">[]</span><span class="kt">byte</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">int</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Printf</span><span class="p">(</span><span class="s">&#34;WriteFunc: Writing %d bytes\n&#34;</span><span class="p">,</span><span class="w"> </span><span class="nb">len</span><span class="p">(</span><span class="nx">data</span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">return</span><span class="w"> </span><span class="nx">buf</span><span class="p">.</span><span class="nf">Write</span><span class="p">(</span><span class="nx">data</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">logWriter</span><span class="p">.</span><span class="nf">Write</span><span class="p">([]</span><span class="nb">byte</span><span class="p">(</span><span class="s">&#34;Hello, world!&#34;</span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Println</span><span class="p">(</span><span class="s">&#34;Error writing data:&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">return</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Println</span><span class="p">(</span><span class="s">&#34;Buffer content:&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">buf</span><span class="p">.</span><span class="nf">String</span><span class="p">())</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p><code>WriteFunc</code> satisfies <code>io.Writer</code> by defining a <code>Write</code> method with the expected signature.
You can adapt any function to match the signature <code>(data []byte) (int, error)</code> using
<code>WriteFunc</code>, so there&rsquo;s no need for a struct when no state is involved.</p>
<p>In <code>main</code>, an anonymous function logs the number of bytes and writes the data to a buffer.
Wrapping this function with <code>WriteFunc</code> lets it implement the <code>io.Writer</code> interface. The
<code>.Write</code> method is called on the wrapped function to log stats and write data to the buffer.
Finally, the buffer&rsquo;s content is printed to verify everything worked.</p>
<div class="alert alert-note">
  <p class="alert-title">Note</p>
  <p>For a simple example like this, using a function type to implement an interface might feel
like overkill. But there are cases where it simplifies things. The next sections explore
real-world examples where function types make interface implementation a bit more
ergonomic.</p>
</div><h2 id="mocking-interfaces-for-testing">Mocking interfaces for testing</h2>
<p>Function types let you mock interfaces without creating dedicated structs. Here&rsquo;s how it
works with an <code>Authenticator</code> interface:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">Authenticator</span><span class="w"> </span><span class="kd">interface</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">Authenticate</span><span class="p">(</span><span class="nx">username</span><span class="p">,</span><span class="w"> </span><span class="nx">password</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">bool</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">AuthFunc</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="nx">username</span><span class="p">,</span><span class="w"> </span><span class="nx">password</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">bool</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">af</span><span class="w"> </span><span class="nx">AuthFunc</span><span class="p">)</span><span class="w"> </span><span class="nf">Authenticate</span><span class="p">(</span><span class="nx">username</span><span class="p">,</span><span class="w"> </span><span class="nx">password</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">bool</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nf">af</span><span class="p">(</span><span class="nx">username</span><span class="p">,</span><span class="w"> </span><span class="nx">password</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>The <code>AuthFunc</code> type implements the <code>Authenticate</code> method by calling itself with the provided
arguments. This lets you create mock implementations inline in your tests.</p>
<p>Here&rsquo;s how to use it in a test:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">TestLogin</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">mockAuth</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">AuthFunc</span><span class="p">(</span><span class="kd">func</span><span class="p">(</span><span class="nx">u</span><span class="p">,</span><span class="w"> </span><span class="nx">p</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">bool</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Printf</span><span class="p">(</span><span class="s">&#34;MockAuth called with username=%s, password=%s\n&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">u</span><span class="p">,</span><span class="w"> </span><span class="nx">p</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">return</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">success</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">PerformLogin</span><span class="p">(</span><span class="s">&#34;john_doe&#34;</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;secret&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">mockAuth</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="p">!</span><span class="nx">success</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">t</span><span class="p">.</span><span class="nf">Fatalf</span><span class="p">(</span><span class="s">&#34;Authentication failed&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>And in application code:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">auth</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">AuthFunc</span><span class="p">(</span><span class="kd">func</span><span class="p">(</span><span class="nx">u</span><span class="p">,</span><span class="w"> </span><span class="nx">p</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">bool</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">return</span><span class="w"> </span><span class="nx">u</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="s">&#34;admin&#34;</span><span class="w"> </span><span class="o">&amp;&amp;</span><span class="w"> </span><span class="nx">p</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="s">&#34;password123&#34;</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">success</span><span class="p">,</span><span class="w"> </span><span class="nx">_</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">auth</span><span class="p">.</span><span class="nf">Authenticate</span><span class="p">(</span><span class="s">&#34;admin&#34;</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;password123&#34;</span><span class="p">);</span><span class="w"> </span><span class="nx">success</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Println</span><span class="p">(</span><span class="s">&#34;Authentication successful!&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><h2 id="building-http-middlewares">Building HTTP middlewares</h2>
<p>The standard library&rsquo;s <code>http.HandlerFunc</code> demonstrates function types in action. Here&rsquo;s how
to build a logging middleware that times requests:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">Handler</span><span class="w"> </span><span class="kd">interface</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">ServeHTTP</span><span class="p">(</span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">LoggingMiddleware</span><span class="p">(</span><span class="nx">next</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">Handler</span><span class="p">)</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">Handler</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nf">HandlerFunc</span><span class="p">(</span><span class="kd">func</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">start</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">time</span><span class="p">.</span><span class="nf">Now</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Printf</span><span class="p">(</span><span class="s">&#34;Started %s %s\n&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="p">.</span><span class="nx">Method</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="p">.</span><span class="nx">URL</span><span class="p">.</span><span class="nx">Path</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">next</span><span class="p">.</span><span class="nf">ServeHTTP</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Printf</span><span class="p">(</span><span class="s">&#34;Completed %s in %v\n&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="p">.</span><span class="nx">URL</span><span class="p">.</span><span class="nx">Path</span><span class="p">,</span><span class="w"> </span><span class="nx">time</span><span class="p">.</span><span class="nf">Since</span><span class="p">(</span><span class="nx">start</span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p><code>http.HandlerFunc</code> converts functions into HTTP handlers. The logging middleware wraps the
next handler and adds timing and logging.</p>
<p>We use it as follows:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">handler</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nf">HandlerFunc</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="kd">func</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Fprintf</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;Hello, World!&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">},</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">http</span><span class="p">.</span><span class="nf">Handle</span><span class="p">(</span><span class="s">&#34;/&#34;</span><span class="p">,</span><span class="w"> </span><span class="nf">LoggingMiddleware</span><span class="p">(</span><span class="nx">handler</span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">http</span><span class="p">.</span><span class="nf">ListenAndServe</span><span class="p">(</span><span class="s">&#34;:8080&#34;</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><h2 id="adapting-function-types-for-database-queries">Adapting function types for database queries</h2>
<p>Function types can abstract database query execution for testing or supporting different
database implementations:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">QueryExecutor</span><span class="w"> </span><span class="kd">interface</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">Execute</span><span class="p">(</span><span class="nx">query</span><span class="w"> </span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="nx">args</span><span class="w"> </span><span class="o">...</span><span class="kt">any</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nx">Result</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">QueryFunc</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="nx">query</span><span class="w"> </span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="nx">args</span><span class="w"> </span><span class="o">...</span><span class="kt">any</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nx">Result</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">qf</span><span class="w"> </span><span class="nx">QueryFunc</span><span class="p">)</span><span class="w"> </span><span class="nf">Execute</span><span class="p">(</span><span class="nx">query</span><span class="w"> </span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="nx">args</span><span class="w"> </span><span class="o">...</span><span class="kt">any</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nx">Result</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nf">qf</span><span class="p">(</span><span class="nx">query</span><span class="p">,</span><span class="w"> </span><span class="nx">args</span><span class="o">...</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p><code>QueryFunc</code> turns regular functions into <code>QueryExecutor</code> implementations, making it easy to
swap implementations or create mocks.</p>
<p>This is how to use it:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">executor</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">QueryFunc</span><span class="p">(</span><span class="kd">func</span><span class="p">(</span><span class="nx">query</span><span class="w"> </span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="nx">args</span><span class="w"> </span><span class="o">...</span><span class="kt">any</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nx">Result</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Printf</span><span class="p">(</span><span class="s">&#34;Executing query: %s with args: %v\n&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">query</span><span class="p">,</span><span class="w"> </span><span class="nx">args</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">return</span><span class="w"> </span><span class="nx">Result</span><span class="p">{</span><span class="nx">RowsAffected</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="p">},</span><span class="w"> </span><span class="kc">nil</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">result</span><span class="p">,</span><span class="w"> </span><span class="nx">_</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">executor</span><span class="p">.</span><span class="nf">Execute</span><span class="p">(</span><span class="s">&#34;SELECT * FROM users WHERE id = ?&#34;</span><span class="p">,</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Printf</span><span class="p">(</span><span class="s">&#34;Rows affected: %d\n&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">result</span><span class="p">.</span><span class="nx">RowsAffected</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><h2 id="implementing-retry-logic">Implementing retry logic</h2>
<p>Function types can encapsulate retry behavior without creating configuration structs:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">Retryer</span><span class="w"> </span><span class="kd">interface</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nf">Retry</span><span class="p">(</span><span class="nx">fn</span><span class="w"> </span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="kt">error</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">RetryFunc</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="nx">fn</span><span class="w"> </span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="kt">error</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">rf</span><span class="w"> </span><span class="nx">RetryFunc</span><span class="p">)</span><span class="w"> </span><span class="nf">Retry</span><span class="p">(</span><span class="nx">fn</span><span class="w"> </span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="kt">error</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nf">rf</span><span class="p">(</span><span class="nx">fn</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p><code>RetryFunc</code> converts functions with the matching signature into a <code>Retryer</code>, letting you
swap retry strategies or create test versions.</p>
<p>Here&rsquo;s how to use it:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">retry</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">RetryFunc</span><span class="p">(</span><span class="kd">func</span><span class="p">(</span><span class="nx">fn</span><span class="w"> </span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="kt">error</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">for</span><span class="w"> </span><span class="nx">i</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"> </span><span class="nx">i</span><span class="w"> </span><span class="p">&lt;</span><span class="w"> </span><span class="mi">3</span><span class="p">;</span><span class="w"> </span><span class="nx">i</span><span class="o">++</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">fn</span><span class="p">();</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">time</span><span class="p">.</span><span class="nf">Sleep</span><span class="p">(</span><span class="nx">time</span><span class="p">.</span><span class="nx">Second</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="nx">time</span><span class="p">.</span><span class="nf">Duration</span><span class="p">(</span><span class="nx">i</span><span class="o">+</span><span class="mi">1</span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">return</span><span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Errorf</span><span class="p">(</span><span class="s">&#34;operation failed after 3 retries&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">retry</span><span class="p">.</span><span class="nf">Retry</span><span class="p">(</span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="kt">error</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="c1">// Your operation here</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Printf</span><span class="p">(</span><span class="s">&#34;Failed to execute operation: %v\n&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Go lets us define methods on custom types, including function types. While this can be handy
for adapting a function type to an interface, it can make the code hard to read at times. So
I don&rsquo;t always reach for it. It&rsquo;s perfectly fine to define an empty struct with a single
method if that makes the code more readable. Nonetheless, it&rsquo;s a neat trick to keep in your
repertoire.</p>
]]></content:encoded>
    </item>
    <item>
      <title>Shades of testing HTTP requests in Python</title>
      <link>https://rednafi.com/python/testing-http-requests/</link>
      <pubDate>Mon, 02 Sep 2024 00:00:00 +0000</pubDate>
      <guid>https://rednafi.com/python/testing-http-requests/</guid>
      <description>Test HTTP requests in Python with pytest-httpx for full mocking, respx for pattern matching, or VCR.py for recording real responses.</description>
      <category>Python</category>
      <category>API</category>
      <category>Testing</category>
      <category>TIL</category>
      <content:encoded><![CDATA[<p>Here&rsquo;s a Python snippet that makes an HTTP POST request:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># script.py</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">httpx</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">Any</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">make_request</span><span class="p">(</span><span class="n">url</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">]:</span>
</span></span><span class="line"><span class="cl">    <span class="n">headers</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&#34;Content-Type&#34;</span><span class="p">:</span> <span class="s2">&#34;application/json&#34;</span><span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">async</span> <span class="k">with</span> <span class="n">httpx</span><span class="o">.</span><span class="n">AsyncClient</span><span class="p">(</span><span class="n">headers</span><span class="o">=</span><span class="n">headers</span><span class="p">)</span> <span class="k">as</span> <span class="n">client</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">response</span> <span class="o">=</span> <span class="k">await</span> <span class="n">client</span><span class="o">.</span><span class="n">post</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="n">url</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="n">json</span><span class="o">=</span><span class="p">{</span><span class="s2">&#34;key_1&#34;</span><span class="p">:</span> <span class="s2">&#34;value_1&#34;</span><span class="p">,</span> <span class="s2">&#34;key_2&#34;</span><span class="p">:</span> <span class="s2">&#34;value_2&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">response</span><span class="o">.</span><span class="n">json</span><span class="p">()</span>
</span></span></code></pre></div><p>The function <code>make_request</code> makes an async HTTP request with the <a href="https://www.python-httpx.org/" rel="noopener noreferrer" target="_blank">HTTPx</a> library. Running
this with <code>asyncio.run(make_request(&quot;https://httpbin.org/post&quot;))</code> gives us the following
output:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;args&#34;</span><span class="p">:</span> <span class="p">{},</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;data&#34;</span><span class="p">:</span> <span class="s2">&#34;{\&#34;key_1\&#34;: \&#34;value_1\&#34;, \&#34;key_2\&#34;: \&#34;value_2\&#34;}&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;files&#34;</span><span class="p">:</span> <span class="p">{},</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;form&#34;</span><span class="p">:</span> <span class="p">{},</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;headers&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="nt">&#34;Accept&#34;</span><span class="p">:</span> <span class="s2">&#34;*/*&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="nt">&#34;Accept-Encoding&#34;</span><span class="p">:</span> <span class="s2">&#34;gzip, deflate&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="nt">&#34;Content-Length&#34;</span><span class="p">:</span> <span class="s2">&#34;40&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="nt">&#34;Content-Type&#34;</span><span class="p">:</span> <span class="s2">&#34;application/json&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="nt">&#34;Host&#34;</span><span class="p">:</span> <span class="s2">&#34;httpbin.org&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="nt">&#34;User-Agent&#34;</span><span class="p">:</span> <span class="s2">&#34;python-httpx/0.27.2&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="nt">&#34;X-Amzn-Trace-Id&#34;</span><span class="p">:</span> <span class="s2">&#34;Root=1-66d5f7b0-2ed0ddc57241f0960f28bc91&#34;</span>
</span></span><span class="line"><span class="cl">  <span class="p">},</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;json&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="nt">&#34;key_1&#34;</span><span class="p">:</span> <span class="s2">&#34;value_1&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="nt">&#34;key_2&#34;</span><span class="p">:</span> <span class="s2">&#34;value_2&#34;</span>
</span></span><span class="line"><span class="cl">  <span class="p">},</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;origin&#34;</span><span class="p">:</span> <span class="s2">&#34;95.90.238.240&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;url&#34;</span><span class="p">:</span> <span class="s2">&#34;https://httpbin.org/post&#34;</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span></code></pre></div><p>We&rsquo;re only interested in the <code>json</code> field and want to assert in our test that making the
HTTP call returns the expected values.</p>
<h2 id="testing-the-http-request">Testing the HTTP request</h2>
<p>Now, how would you test it? One approach is by patching the <code>httpx.AsyncClient</code> instance to
return a canned response and asserting against that. The happy path might be tested as
follows:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># test_script.py</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">unittest.mock</span> <span class="kn">import</span> <span class="n">AsyncMock</span><span class="p">,</span> <span class="n">patch</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">pytest</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">script</span> <span class="kn">import</span> <span class="n">make_request</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd">@pytest.mark.asyncio</span>
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">test_make_request_ok</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">url</span> <span class="o">=</span> <span class="s2">&#34;https://httpbin.org/post&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="n">expected_json</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&#34;key_1&#34;</span><span class="p">:</span> <span class="s2">&#34;value_1&#34;</span><span class="p">,</span> <span class="s2">&#34;key_2&#34;</span><span class="p">:</span> <span class="s2">&#34;value_2&#34;</span><span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1"># Create a mock response object</span>
</span></span><span class="line"><span class="cl">    <span class="n">mock_response</span> <span class="o">=</span> <span class="n">AsyncMock</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="n">mock_response</span><span class="o">.</span><span class="n">json</span><span class="o">.</span><span class="n">return_value</span> <span class="o">=</span> <span class="n">expected_json</span>
</span></span><span class="line"><span class="cl">    <span class="n">mock_response</span><span class="o">.</span><span class="n">status_code</span> <span class="o">=</span> <span class="mi">200</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1"># Patch the httpx.AsyncClient.post method to return the mock_response</span>
</span></span><span class="line"><span class="cl">    <span class="k">with</span> <span class="n">patch</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;script.httpx.AsyncClient.post&#34;</span><span class="p">,</span>  <span class="c1"># Don&#39;t mock what you don&#39;t own</span>
</span></span><span class="line"><span class="cl">        <span class="n">return_value</span><span class="o">=</span><span class="n">mock_response</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span> <span class="k">as</span> <span class="n">mock_post</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">response</span> <span class="o">=</span> <span class="k">await</span> <span class="n">make_request</span><span class="p">(</span><span class="n">url</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="c1"># Await the coroutine that was returned</span>
</span></span><span class="line"><span class="cl">        <span class="n">response</span> <span class="o">=</span> <span class="k">await</span> <span class="n">response</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="c1"># Assertions</span>
</span></span><span class="line"><span class="cl">        <span class="n">mock_post</span><span class="o">.</span><span class="n">assert_called_once_with</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">json</span><span class="o">=</span><span class="n">expected_json</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">assert</span> <span class="n">response</span> <span class="o">==</span> <span class="n">expected_json</span>
</span></span></code></pre></div><p>That&rsquo;s quite a bit of work just to test a simple HTTP request. The mocking gets pretty hairy
as the complexity of your HTTP calls increases. One way to cut down the mess is by using a
library like <a href="https://lundberg.github.io/respx/" rel="noopener noreferrer" target="_blank">respx</a> that handles the patching for you.</p>
<h2 id="simplifying-mocks-with-respx">Simplifying mocks with respx</h2>
<p>For instance:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># test_script.py</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">pytest</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">respx</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">script</span> <span class="kn">import</span> <span class="n">make_request</span><span class="p">,</span> <span class="n">httpx</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd">@pytest.mark.asyncio</span>
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">test_make_request_ok</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">url</span> <span class="o">=</span> <span class="s2">&#34;https://httpbin.org/post&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="n">expected_json</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&#34;key_1&#34;</span><span class="p">:</span> <span class="s2">&#34;value_1&#34;</span><span class="p">,</span> <span class="s2">&#34;key_2&#34;</span><span class="p">:</span> <span class="s2">&#34;value_2&#34;</span><span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1"># Mocking the HTTP POST request using respx</span>
</span></span><span class="line"><span class="cl">    <span class="k">with</span> <span class="n">respx</span><span class="o">.</span><span class="n">mock</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">respx</span><span class="o">.</span><span class="n">post</span><span class="p">(</span><span class="n">url</span><span class="p">)</span><span class="o">.</span><span class="n">mock</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="n">return_value</span><span class="o">=</span><span class="n">httpx</span><span class="o">.</span><span class="n">Response</span><span class="p">(</span><span class="mi">200</span><span class="p">,</span> <span class="n">json</span><span class="o">=</span><span class="n">expected_json</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="c1"># Calling the function</span>
</span></span><span class="line"><span class="cl">        <span class="n">response</span> <span class="o">=</span> <span class="k">await</span> <span class="n">make_request</span><span class="p">(</span><span class="n">url</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="c1"># Assertions</span>
</span></span><span class="line"><span class="cl">        <span class="k">assert</span> <span class="n">response</span> <span class="o">==</span> <span class="n">expected_json</span>
</span></span></code></pre></div><p>Much cleaner. During tests, respx intercepts HTTP requests made by httpx, allowing you to
test against canned responses. The library provides a context manager that acts like an
httpx client, so you can set the expected response. This removes the need to manually patch
methods like <code>post</code> in <code>httpx.AsyncClient</code>.</p>
<h2 id="testing-with-a-stub-client">Testing with a stub client</h2>
<p>The previous strategy wouldn&rsquo;t work if you want to change your HTTP client since respx is
coupled with httpx. As an alternative, you could rewrite <code>make_request</code> to parametrize the
HTTP client, pass a stub object during the test, and assert against it. This eliminates the
need to write fragile mocking sludges or depend on an external mocking library.</p>
<p>Here&rsquo;s how you&rsquo;d change the code:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># script.py</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">httpx</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">asyncio</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">Any</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">make_request</span><span class="p">(</span><span class="n">url</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">client</span><span class="p">:</span> <span class="n">httpx</span><span class="o">.</span><span class="n">AsyncClient</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">]:</span>
</span></span><span class="line"><span class="cl">    <span class="c1"># We don&#39;t want to initiate the ctx manager in every request</span>
</span></span><span class="line"><span class="cl">    <span class="c1"># AsyncClient.__enter__ will be called once and passed here</span>
</span></span><span class="line"><span class="cl">    <span class="n">response</span> <span class="o">=</span> <span class="k">await</span> <span class="n">client</span><span class="o">.</span><span class="n">post</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="n">url</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="n">json</span><span class="o">=</span><span class="p">{</span><span class="s2">&#34;key_1&#34;</span><span class="p">:</span> <span class="s2">&#34;value_1&#34;</span><span class="p">,</span> <span class="s2">&#34;key_2&#34;</span><span class="p">:</span> <span class="s2">&#34;value_2&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">response</span><span class="o">.</span><span class="n">json</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">main</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">headers</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&#34;Content-Type&#34;</span><span class="p">:</span> <span class="s2">&#34;application/json&#34;</span><span class="p">}</span>
</span></span><span class="line"><span class="cl">    <span class="n">url</span> <span class="o">=</span> <span class="s2">&#34;https://httpbin.org/post&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1"># Enter into the context manager and pass the instance to make_request</span>
</span></span><span class="line"><span class="cl">    <span class="k">async</span> <span class="k">with</span> <span class="n">httpx</span><span class="o">.</span><span class="n">AsyncClient</span><span class="p">(</span><span class="n">headers</span><span class="o">=</span><span class="n">headers</span><span class="p">)</span> <span class="k">as</span> <span class="n">client</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">response</span> <span class="o">=</span> <span class="k">await</span> <span class="n">make_request</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">client</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="nb">print</span><span class="p">(</span><span class="n">response</span><span class="p">)</span>
</span></span></code></pre></div><p>Now the tests would look as follows:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">pytest</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">Any</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">httpx</span> <span class="kn">import</span> <span class="n">Response</span><span class="p">,</span> <span class="n">Request</span><span class="p">,</span> <span class="n">AsyncClient</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">script</span> <span class="kn">import</span> <span class="n">make_request</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">StubAsyncClient</span><span class="p">(</span><span class="n">AsyncClient</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">async</span> <span class="k">def</span> <span class="nf">post</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="p">,</span> <span class="n">url</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">json</span><span class="p">:</span> <span class="n">Any</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">:</span> <span class="n">Any</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Response</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">request</span> <span class="o">=</span> <span class="n">Request</span><span class="p">(</span><span class="n">method</span><span class="o">=</span><span class="s2">&#34;POST&#34;</span><span class="p">,</span> <span class="n">url</span><span class="o">=</span><span class="n">url</span><span class="p">,</span> <span class="n">json</span><span class="o">=</span><span class="n">json</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="c1"># Simulate the original response that matches the request</span>
</span></span><span class="line"><span class="cl">        <span class="n">response</span> <span class="o">=</span> <span class="n">Response</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="n">status_code</span><span class="o">=</span><span class="mi">200</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="n">json</span><span class="o">=</span><span class="p">{</span><span class="s2">&#34;key_1&#34;</span><span class="p">:</span> <span class="s2">&#34;value_1&#34;</span><span class="p">,</span> <span class="s2">&#34;key_2&#34;</span><span class="p">:</span> <span class="s2">&#34;value_2&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">            <span class="n">request</span><span class="o">=</span><span class="n">request</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">response</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd">@pytest.mark.asyncio</span>
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">test_make_request_ok</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">url</span> <span class="o">=</span> <span class="s2">&#34;https://httpbin.org/post&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="n">headers</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&#34;Content-Type&#34;</span><span class="p">:</span> <span class="s2">&#34;application/json&#34;</span><span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">async</span> <span class="k">with</span> <span class="n">StubAsyncClient</span><span class="p">(</span><span class="n">headers</span><span class="o">=</span><span class="n">headers</span><span class="p">)</span> <span class="k">as</span> <span class="n">client</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">response_data</span> <span class="o">=</span> <span class="k">await</span> <span class="n">make_request</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">client</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">assert</span> <span class="n">response_data</span> <span class="o">==</span> <span class="p">{</span><span class="s2">&#34;key_1&#34;</span><span class="p">:</span> <span class="s2">&#34;value_1&#34;</span><span class="p">,</span> <span class="s2">&#34;key_2&#34;</span><span class="p">:</span> <span class="s2">&#34;value_2&#34;</span><span class="p">}</span>
</span></span></code></pre></div><p>Much better!</p>
<h2 id="integration-testing-with-a-test-server">Integration testing with a test server</h2>
<p>One thing I&rsquo;ve picked up from writing Go is that it&rsquo;s often just easier to perform
integration tests on these I/O-bound functions. That is, you can spin up a server that
returns a canned response and then test your code against it to assert if it&rsquo;s getting the
expected output.</p>
<p>The test could look as follows. This assumes <code>make_request</code> takes in an <code>AsyncClient</code>
instance as a parameter, as shown in the last example.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">pytest</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">starlette.applications</span> <span class="kn">import</span> <span class="n">Starlette</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">starlette.responses</span> <span class="kn">import</span> <span class="n">JSONResponse</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">starlette.routing</span> <span class="kn">import</span> <span class="n">Route</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">starlette.requests</span> <span class="kn">import</span> <span class="n">Request</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">httpx</span> <span class="kn">import</span> <span class="n">AsyncClient</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">script</span> <span class="kn">import</span> <span class="n">make_request</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">test_endpoint</span><span class="p">(</span><span class="n">request</span><span class="p">:</span> <span class="n">Request</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">JSONResponse</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">JSONResponse</span><span class="p">({</span><span class="s2">&#34;key_1&#34;</span><span class="p">:</span> <span class="s2">&#34;value_1&#34;</span><span class="p">,</span> <span class="s2">&#34;key_2&#34;</span><span class="p">:</span> <span class="s2">&#34;value_2&#34;</span><span class="p">})</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">app</span> <span class="o">=</span> <span class="n">Starlette</span><span class="p">(</span><span class="n">routes</span><span class="o">=</span><span class="p">[</span><span class="n">Route</span><span class="p">(</span><span class="s2">&#34;/post&#34;</span><span class="p">,</span> <span class="n">test_endpoint</span><span class="p">,</span> <span class="n">methods</span><span class="o">=</span><span class="p">[</span><span class="s2">&#34;POST&#34;</span><span class="p">])])</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd">@pytest.mark.asyncio</span>
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">test_make_request</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="c1"># Manually create the AsyncClient</span>
</span></span><span class="line"><span class="cl">    <span class="k">async</span> <span class="k">with</span> <span class="n">AsyncClient</span><span class="p">(</span><span class="n">app</span><span class="o">=</span><span class="n">app</span><span class="p">,</span> <span class="n">base_url</span><span class="o">=</span><span class="s2">&#34;http://test&#34;</span><span class="p">)</span> <span class="k">as</span> <span class="n">client</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">url</span> <span class="o">=</span> <span class="s2">&#34;http://testserver/post&#34;</span>
</span></span><span class="line"><span class="cl">        <span class="n">response</span> <span class="o">=</span> <span class="k">await</span> <span class="n">make_request</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">client</span><span class="o">=</span><span class="n">client</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">assert</span> <span class="n">response</span> <span class="o">==</span> <span class="p">{</span><span class="s2">&#34;key_1&#34;</span><span class="p">:</span> <span class="s2">&#34;value_1&#34;</span><span class="p">,</span> <span class="s2">&#34;key_2&#34;</span><span class="p">:</span> <span class="s2">&#34;value_2&#34;</span><span class="p">}</span>
</span></span></code></pre></div><p>In the above test, we&rsquo;re using <a href="https://www.starlette.io/" rel="noopener noreferrer" target="_blank">Starlette</a> to define a simple ASGI server that returns our
expected response. Then we set up the <code>httpx.AsyncClient</code> so it makes the request against
the test server instead of making an external network call. Finally, we call the
<code>make_request</code> function and assert the expected payload.</p>
<p>Sure, you could set up the server with the standard library&rsquo;s <code>http</code> module, but that code
doesn&rsquo;t look half as pretty.</p>
<!-- references -->
<!-- prettier-ignore-start -->
<!-- prettier-ignore-end -->
]]></content:encoded>
    </item>
    <item>
      <title>Log context propagation in Python ASGI apps</title>
      <link>https://rednafi.com/python/log-context-propagation/</link>
      <pubDate>Tue, 06 Aug 2024 00:00:00 +0000</pubDate>
      <guid>https://rednafi.com/python/log-context-propagation/</guid>
      <description>Automatically tag Python logs with request context using middleware and contextvars for distributed tracing in ASGI web applications.</description>
      <category>Python</category>
      <category>API</category>
      <category>Logging</category>
      <content:encoded><![CDATA[<p>Let&rsquo;s say you have a web app that emits log messages from different layers. Your log shipper
collects and sends these messages to a destination like Datadog where you can query them.
One common requirement is to tag the log messages with some common attributes, which you can
use later to query them.</p>
<p>In distributed tracing, this tagging is usually known as <a href="https://opentelemetry.io/docs/concepts/context-propagation/" rel="noopener noreferrer" target="_blank">context propagation</a>, where you&rsquo;re
attaching some contextual information to your log messages that you can use later for query
purposes. However, if you have to collect the context at each layer of your application and
pass it manually to the downstream ones, that&rsquo;d make the whole process quite painful.</p>
<p>Suppose you have a web view for an endpoint that calls another function to do something:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">view</span><span class="p">(</span><span class="n">request</span><span class="p">:</span> <span class="n">Request</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">JSONResponse</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="c1"># Collect contextual info from the header</span>
</span></span><span class="line"><span class="cl">    <span class="n">user_id</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">headers</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;Svc-User-Id&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">platform</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">headers</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;Svc-Platform&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1"># Log the request with context</span>
</span></span><span class="line"><span class="cl">    <span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;Request started&#34;</span><span class="p">,</span> <span class="n">extra</span><span class="o">=</span><span class="p">{</span><span class="s2">&#34;user_id&#34;</span><span class="p">:</span> <span class="n">user_id</span><span class="p">,</span> <span class="s2">&#34;platform&#34;</span><span class="p">:</span> <span class="n">platform</span><span class="p">}</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">await</span> <span class="n">work</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1"># Log the response too</span>
</span></span><span class="line"><span class="cl">    <span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;Request ended&#34;</span><span class="p">,</span> <span class="n">extra</span><span class="o">=</span><span class="p">{</span><span class="s2">&#34;user_id&#34;</span><span class="p">:</span> <span class="n">user_id</span><span class="p">,</span> <span class="s2">&#34;platform&#34;</span><span class="p">:</span> <span class="n">platform</span><span class="p">}</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">JSONResponse</span><span class="p">({</span><span class="s2">&#34;message&#34;</span><span class="p">:</span> <span class="s2">&#34;Work, work work!&#34;</span><span class="p">})</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">work</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">&#34;Work done&#34;</span><span class="p">)</span>
</span></span></code></pre></div><p>I&rsquo;m using <a href="https://www.starlette.io/" rel="noopener noreferrer" target="_blank">starlette</a> syntax for the above pseudocode, but this is valid for any generic
ASGI web app. The <code>view</code> procedure collects contextual information like <code>user_id</code> and
<code>platform</code> from the request headers. Then it tags the log statements before and after
calling the <code>work</code> function using the <code>extra</code> fields in the logger calls. This way, the log
messages have contextual info attached to them.</p>
<p>However, the <code>work</code> procedure also generates a log message, and that won&rsquo;t get tagged here.
We may be tempted to pass the contextual information to the <code>work</code> subroutine and use them
to tag the logs, but that&rsquo;ll quickly get repetitive and cumbersome. Passing a bunch of
arguments to a function just so it can tag some log messages also makes things unnecessarily
verbose. Plus, it&rsquo;s quite easy to forget to do so, which will leave you with orphan logs
with no way to query them.</p>
<p>It turns out we can write a simple middleware to tag log statements in a way where we won&rsquo;t
need to manually propagate the contextual information throughout the call chain. To
demonstrate that, here&rsquo;s a simple <code>get</code> endpoint server written in Starlette that&rsquo;ll just
return a canned response after logging a few events. The app structure looks as follows:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-txt" data-lang="txt"><span class="line"><span class="cl">svc
</span></span><span class="line"><span class="cl">├── __init__.py
</span></span><span class="line"><span class="cl">├── log.py
</span></span><span class="line"><span class="cl">├── main.py
</span></span><span class="line"><span class="cl">├── middleware.py
</span></span><span class="line"><span class="cl">└── view.py
</span></span></code></pre></div><h2 id="configure-the-logger">Configure the logger</h2>
<p>The first step is to configure the application logger so that it emits structured log
statements in JSON where each message will look as follows:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;message&#34;</span><span class="p">:</span> <span class="s2">&#34;Some log message&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;timestamp&#34;</span><span class="p">:</span> <span class="mi">1722794887376</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;tags&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="nt">&#34;user_id&#34;</span><span class="p">:</span> <span class="s2">&#34;1234&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="nt">&#34;platform&#34;</span><span class="p">:</span> <span class="s2">&#34;ios&#34;</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span></code></pre></div><p>Here&rsquo;s the log configuration logic:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># log.py</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">contextvars</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">json</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">logging</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">time</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Set up the context variable with default values</span>
</span></span><span class="line"><span class="cl"><span class="n">default_context</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&#34;user_id&#34;</span><span class="p">:</span> <span class="s2">&#34;unknown&#34;</span><span class="p">,</span> <span class="s2">&#34;platform&#34;</span><span class="p">:</span> <span class="s2">&#34;unknown&#34;</span><span class="p">}</span>
</span></span><span class="line"><span class="cl"><span class="n">log_context_var</span> <span class="o">=</span> <span class="n">contextvars</span><span class="o">.</span><span class="n">ContextVar</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;log_context&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">default</span><span class="o">=</span><span class="n">default_context</span><span class="o">.</span><span class="n">copy</span><span class="p">(),</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Custom log formatter</span>
</span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">ContextAwareJsonFormatter</span><span class="p">(</span><span class="n">logging</span><span class="o">.</span><span class="n">Formatter</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">format</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">record</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="n">log_data</span> <span class="o">=</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">            <span class="s2">&#34;message&#34;</span><span class="p">:</span> <span class="n">record</span><span class="o">.</span><span class="n">getMessage</span><span class="p">(),</span>
</span></span><span class="line"><span class="cl">            <span class="c1"># Add millisecond precision timestamp</span>
</span></span><span class="line"><span class="cl">            <span class="s2">&#34;timestamp&#34;</span><span class="p">:</span> <span class="nb">int</span><span class="p">(</span><span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span> <span class="o">*</span> <span class="mi">1000</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">            <span class="c1"># Get context from ContextVar (concurrency-safe).</span>
</span></span><span class="line"><span class="cl">            <span class="c1"># Context is set in middleware; .get() returns current</span>
</span></span><span class="line"><span class="cl">            <span class="s2">&#34;tags&#34;</span><span class="p">:</span> <span class="n">log_context_var</span><span class="o">.</span><span class="n">get</span><span class="p">(),</span>
</span></span><span class="line"><span class="cl">        <span class="p">}</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">log_data</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Set up the logger</span>
</span></span><span class="line"><span class="cl"><span class="n">logger</span> <span class="o">=</span> <span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">()</span>
</span></span><span class="line"><span class="cl"><span class="n">logger</span><span class="o">.</span><span class="n">setLevel</span><span class="p">(</span><span class="n">logging</span><span class="o">.</span><span class="n">INFO</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">handler</span> <span class="o">=</span> <span class="n">logging</span><span class="o">.</span><span class="n">StreamHandler</span><span class="p">()</span>
</span></span><span class="line"><span class="cl"><span class="n">formatter</span> <span class="o">=</span> <span class="n">ContextAwareJsonFormatter</span><span class="p">()</span>
</span></span><span class="line"><span class="cl"><span class="n">handler</span><span class="o">.</span><span class="n">setFormatter</span><span class="p">(</span><span class="n">formatter</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="n">logger</span><span class="o">.</span><span class="n">addHandler</span><span class="p">(</span><span class="n">handler</span><span class="p">)</span>
</span></span></code></pre></div><p>The <code>contextvars</code> module manages context information across asynchronous tasks, preventing
context leakage between requests. We use a <code>log_context_var</code> context variable to store user
ID and platform information, ensuring each log entry includes relevant context for the
request.</p>
<p>The <code>ContextAwareJsonFormatter</code> formats log statements to include the message, timestamp in
milliseconds, and context tags. The context is retrieved using <code>log_context_var.get()</code>,
ensuring concurrency-safe access. The context variable is set in the middleware, so
<code>log_context_var.get()</code> always returns the current context for each request.</p>
<p>Next, we set up a <code>StreamHandler</code>, attach the <code>ContextAwareJsonFormatter</code> to it, and add the
handler to the root logger.</p>
<h2 id="write-a-middleware-that-tags-the-log-statements-automatically">Write a middleware that tags the log statements automatically</h2>
<p>With log formatting out of the way, here&rsquo;s how to write the middleware to update the logger
so that all the log messages within a request-response cycle get automatically tagged:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># middleware.py</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">logging</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">collections.abc</span> <span class="kn">import</span> <span class="n">Awaitable</span><span class="p">,</span> <span class="n">Callable</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">starlette.middleware.base</span> <span class="kn">import</span> <span class="n">BaseHTTPMiddleware</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">starlette.requests</span> <span class="kn">import</span> <span class="n">Request</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">starlette.responses</span> <span class="kn">import</span> <span class="n">Response</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">svc.log</span> <span class="kn">import</span> <span class="n">default_context</span><span class="p">,</span> <span class="n">log_context_var</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Middleware for setting log context</span>
</span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">LogContextMiddleware</span><span class="p">(</span><span class="n">BaseHTTPMiddleware</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">async</span> <span class="k">def</span> <span class="nf">dispatch</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="n">request</span><span class="p">:</span> <span class="n">Request</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="n">call_next</span><span class="p">:</span> <span class="n">Callable</span><span class="p">[[</span><span class="n">Request</span><span class="p">],</span> <span class="n">Awaitable</span><span class="p">[</span><span class="n">Response</span><span class="p">]],</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Response</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="c1"># Copy the default context so that we&#39;re not sharing anything</span>
</span></span><span class="line"><span class="cl">        <span class="c1"># between requests</span>
</span></span><span class="line"><span class="cl">        <span class="n">context</span> <span class="o">=</span> <span class="n">default_context</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="c1"># Collect the contextual information from request headers</span>
</span></span><span class="line"><span class="cl">        <span class="n">user_id</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">headers</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;Svc-User-Id&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">platform</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">headers</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;Svc-Platform&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="c1"># Update the context</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">user_id</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="n">context</span><span class="p">[</span><span class="s2">&#34;user_id&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="n">user_id</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">platform</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="n">context</span><span class="p">[</span><span class="s2">&#34;platform&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="n">platform</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="c1"># Set the log_context_var context variable</span>
</span></span><span class="line"><span class="cl">        <span class="n">token</span> <span class="o">=</span> <span class="n">log_context_var</span><span class="o">.</span><span class="n">set</span><span class="p">(</span><span class="n">context</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="c1"># Log before making request</span>
</span></span><span class="line"><span class="cl">            <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">&#34;From middleware: request started&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">            <span class="n">response</span> <span class="o">=</span> <span class="k">await</span> <span class="n">call_next</span><span class="p">(</span><span class="n">request</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">            <span class="c1"># Log after making request</span>
</span></span><span class="line"><span class="cl">            <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">&#34;From middleware: request ended&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">finally</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="c1"># Reset the context after the request is processed</span>
</span></span><span class="line"><span class="cl">            <span class="n">log_context_var</span><span class="o">.</span><span class="n">reset</span><span class="p">(</span><span class="n">token</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">response</span>
</span></span></code></pre></div><p>The <code>LogContextMiddleware</code> class inherits from <code>starlette.BaseHTTPMiddleware</code> and gets
initialized with the application.</p>
<p>The <code>dispatch</code> method is called automatically for each request. It extracts <code>user_id</code> and
<code>platform</code> from the request headers and sets these values in the <code>log_context_var</code> to tag
log messages. Then it logs the incoming request, processes it, logs the outgoing response,
and then clears the context so that we don&rsquo;t leak the context information across requests.
This way, our view function won&rsquo;t need to be peppered with repetitive log statements.</p>
<h2 id="write-the-simplified-view">Write the simplified view</h2>
<p>Setting up the logger and middleware drastically simplifies our endpoint view since we won&rsquo;t
need to tag the logs explicitly or add request-response logs in each view. It looks like
this now:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># view.py</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">asyncio</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">logging</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">starlette.requests</span> <span class="kn">import</span> <span class="n">Request</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">starlette.responses</span> <span class="kn">import</span> <span class="n">JSONResponse</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">view</span><span class="p">(</span><span class="n">request</span><span class="p">:</span> <span class="n">Request</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">JSONResponse</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">await</span> <span class="n">work</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">&#34;From view function: work finished&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">JSONResponse</span><span class="p">({</span><span class="s2">&#34;message&#34;</span><span class="p">:</span> <span class="sa">f</span><span class="s2">&#34;Work work work!!!&#34;</span><span class="p">})</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">work</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">&#34;From work function: work started&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
</span></span></code></pre></div><p>Notice there&rsquo;s no repetitive request-response log statements in the <code>view</code> function, and
we&rsquo;re not passing the log context anywhere explicitly. The middleware will ensure that the
request and response logs are always emitted and all the logs, including the one coming out
of the <code>work</code> function, are tagged with the contextual information.</p>
<h2 id="wire-everything-together">Wire everything together</h2>
<p>The logging configuration and middleware can be wired up like this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># main.py</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">uvicorn</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">starlette.applications</span> <span class="kn">import</span> <span class="n">Starlette</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">starlette.middleware</span> <span class="kn">import</span> <span class="n">Middleware</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">starlette.routing</span> <span class="kn">import</span> <span class="n">Route</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">svc.middleware</span> <span class="kn">import</span> <span class="n">LogContextMiddleware</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">svc.view</span> <span class="kn">import</span> <span class="n">view</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">middlewares</span> <span class="o">=</span> <span class="p">[</span><span class="n">Middleware</span><span class="p">(</span><span class="n">LogContextMiddleware</span><span class="p">)]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">app</span> <span class="o">=</span> <span class="n">Starlette</span><span class="p">(</span><span class="n">routes</span><span class="o">=</span><span class="p">[</span><span class="n">Route</span><span class="p">(</span><span class="s2">&#34;/&#34;</span><span class="p">,</span> <span class="n">view</span><span class="p">)],</span> <span class="n">middleware</span><span class="o">=</span><span class="n">middlewares</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">&#34;__main__&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">uvicorn</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">app</span><span class="p">,</span> <span class="n">host</span><span class="o">=</span><span class="s2">&#34;0.0.0.0&#34;</span><span class="p">,</span> <span class="n">port</span><span class="o">=</span><span class="mi">8000</span><span class="p">)</span>
</span></span></code></pre></div><p>To instantiate the logger config, we import <code>log.py</code> in the <code>__init__.py</code> module:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># __init__.py</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">svc</span> <span class="kn">import</span> <span class="n">log</span>  <span class="c1"># noqa</span>
</span></span></code></pre></div><p>Now the application can be started with:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl">python -m svc.main
</span></span></code></pre></div><p>And then we can make a request to the server:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl">curl http://localhost:8000/ -H <span class="s1">&#39;Svc-User-Id: 1234&#39;</span> -H <span class="s1">&#39;Svc-Platform: ios&#39;</span>
</span></span></code></pre></div><p>On the server, the request will emit the following log messages:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-plaintext" data-lang="plaintext"><span class="line"><span class="cl">INFO:     Started server process [41848]
</span></span><span class="line"><span class="cl">INFO:     Waiting for application startup.
</span></span><span class="line"><span class="cl">INFO:     Application startup complete.
</span></span><span class="line"><span class="cl">INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
</span></span><span class="line"><span class="cl">{
</span></span><span class="line"><span class="cl">  &#34;message&#34;: &#34;From middleware: request started&#34;,
</span></span><span class="line"><span class="cl">  &#34;timestamp&#34;: 1723166008113,
</span></span><span class="line"><span class="cl">  &#34;tags&#34;: {
</span></span><span class="line"><span class="cl">    &#34;user_id&#34;: &#34;1234&#34;,
</span></span><span class="line"><span class="cl">    &#34;platform&#34;: &#34;ios&#34;
</span></span><span class="line"><span class="cl">  }
</span></span><span class="line"><span class="cl">}
</span></span><span class="line"><span class="cl">{
</span></span><span class="line"><span class="cl">  &#34;message&#34;: &#34;From work function: work started&#34;,
</span></span><span class="line"><span class="cl">  &#34;timestamp&#34;: 1723166008113,
</span></span><span class="line"><span class="cl">  &#34;tags&#34;: {
</span></span><span class="line"><span class="cl">    &#34;user_id&#34;: &#34;1234&#34;,
</span></span><span class="line"><span class="cl">    &#34;platform&#34;: &#34;ios&#34;
</span></span><span class="line"><span class="cl">  }
</span></span><span class="line"><span class="cl">}
</span></span><span class="line"><span class="cl">{
</span></span><span class="line"><span class="cl">  &#34;message&#34;: &#34;From view function: work finished&#34;,
</span></span><span class="line"><span class="cl">  &#34;timestamp&#34;: 1723166009114,
</span></span><span class="line"><span class="cl">  &#34;tags&#34;: {
</span></span><span class="line"><span class="cl">    &#34;user_id&#34;: &#34;1234&#34;,
</span></span><span class="line"><span class="cl">    &#34;platform&#34;: &#34;ios&#34;
</span></span><span class="line"><span class="cl">  }
</span></span><span class="line"><span class="cl">}
</span></span><span class="line"><span class="cl">{
</span></span><span class="line"><span class="cl">  &#34;message&#34;: &#34;From middleware: request ended&#34;,
</span></span><span class="line"><span class="cl">  &#34;timestamp&#34;: 1723166009115,
</span></span><span class="line"><span class="cl">  &#34;tags&#34;: {
</span></span><span class="line"><span class="cl">    &#34;user_id&#34;: &#34;1234&#34;,
</span></span><span class="line"><span class="cl">    &#34;platform&#34;: &#34;ios&#34;
</span></span><span class="line"><span class="cl">  }
</span></span><span class="line"><span class="cl">}
</span></span><span class="line"><span class="cl">INFO:     127.0.0.1:54780 - &#34;GET / HTTP/1.1&#34; 200 OK
</span></span></code></pre></div><p>And we&rsquo;re done. You can find the fully working code in this <a href="https://gist.github.com/rednafi/dc2016a8ea0e2405b943f023bfb18142" rel="noopener noreferrer" target="_blank">GitHub gist</a>.</p>
<p><em>Note: The <a href="https://web.archive.org/web/20240806220817/https://rednafi.com/python/log_context_propagation/" rel="noopener noreferrer" target="_blank">previous version</a> of this example wasn&rsquo;t concurrency safe and used a shared
logger filter, leaking context information during concurrent requests. This was pointed out
in this <a href="https://gist.github.com/rednafi/dc2016a8ea0e2405b943f023bfb18142?permalink_comment_id=5148207#gistcomment-5148207" rel="noopener noreferrer" target="_blank">GitHub comment</a>.</em></p>
<!-- references -->
<!-- prettier-ignore-start -->
<!-- prettier-ignore-end -->
]]></content:encoded>
    </item>
    <item>
      <title>Protobuffed contracts</title>
      <link>https://rednafi.com/misc/protobuffed-contracts/</link>
      <pubDate>Fri, 10 May 2024 00:00:00 +0000</pubDate>
      <guid>https://rednafi.com/misc/protobuffed-contracts/</guid>
      <description>Define service contracts with Protocol Buffers for non-gRPC systems. Generate serializers, maintain self-documented APIs, and ensure cross-language compatibility.</description>
      <category>API</category>
      <category>Networking</category>
      <category>Distributed Systems</category>
      <content:encoded><![CDATA[<p>People typically associate Google&rsquo;s <a href="https://protobuf.dev/" rel="noopener noreferrer" target="_blank">Protocol Buffer</a> with <a href="https://grpc.io/" rel="noopener noreferrer" target="_blank">gRPC</a> services, and rightfully
so. But things often get confusing when discussing protobufs because the term can mean
different things:</p>
<ul>
<li>A binary protocol for efficiently serializing structured data.</li>
<li>A language used to specify how this data should be structured.</li>
</ul>
<p>In gRPC services, you usually use both: the protobuf language in proto files defines the
service interface, and then the clients use the same proto files to communicate with the
services.</p>
<p>However, protobuf can be used in non-gRPC contexts for anything that requires a strict
interface. You can optionally choose to use the more compact serialization format that gRPC
tools offer, or just keep using JSON if you prefer. I&rsquo;ve seen this use case in several
organizations over the past few years, though I haven&rsquo;t given it much thought. It definitely
has its benefits!</p>
<p>Defining your service contracts with protobuf:</p>
<ul>
<li>Allows you to generate message serializers and deserializers in almost any language of
your choice.</li>
<li>You can choose from a set of serialization formats.</li>
<li>The service contracts are self-documented, and you can simply hand over the proto files to
your service users.</li>
<li>Different parts of a service or a fleet of services can be written in different languages,
as long as their communication conforms to the defined contracts.</li>
</ul>
<p>For example, consider an event-driven application that sends messages to a message broker
when an event occurs. A consumer then processes these messages asynchronously. Both the
producer and consumer need to agree on a message format, which is defined by a contract. The
workflow usually goes as follows:</p>
<ul>
<li>Define the message contract using the protobuf DSL.</li>
<li>Generate the code for serializing/deserializing the messages in the language of your
choice.</li>
<li>On the publisher side, serialize the message using the generated code.</li>
<li>On the consumer side, generate code from the same contract and deserialize the message
with that.</li>
</ul>
<h2 id="define-the-contract">Define the contract</h2>
<p>You can define your service interface in a <code>.proto</code> file. Let&rsquo;s say we want to emit some
event in a search service when a user queries something. The query message structure can be
defined as follows:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-proto" data-lang="proto"><span class="line"><span class="cl"><span class="c1">// ./search/protos/message.proto
</span></span></span><span class="line"><span class="cl"><span class="err">
</span></span></span><span class="line"><span class="cl"><span class="n">syntax</span> <span class="o">=</span> <span class="s">&#34;proto3&#34;</span><span class="p">;</span><span class="err">
</span></span></span><span class="line"><span class="cl"><span class="err">
</span></span></span><span class="line"><span class="cl"><span class="kd">message</span> <span class="nc">SearchRequest</span> <span class="p">{</span><span class="err">
</span></span></span><span class="line"><span class="cl">  <span class="kt">string</span> <span class="n">query</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span><span class="err">
</span></span></span><span class="line"><span class="cl">  <span class="kt">int32</span> <span class="n">page_number</span> <span class="o">=</span> <span class="mi">2</span><span class="p">;</span><span class="err">
</span></span></span><span class="line"><span class="cl">  <span class="kt">int32</span> <span class="n">results_per_page</span> <span class="o">=</span> <span class="mi">3</span><span class="p">;</span><span class="err">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="err">
</span></span></span></code></pre></div><p>I&rsquo;m using proto3 syntax, and you can find more about that in the <a href="https://protobuf.dev/programming-guides/proto3/" rel="noopener noreferrer" target="_blank">official proto3 guide</a>.
Next, you can install the gRPC tools for your preferred programming language to generate the
interfacing code that&rsquo;ll be used to serialize and deserialize the messages.</p>
<p>Here&rsquo;s how it looks in Python:</p>
<ul>
<li>
<p>Install <code>grpcio-tools</code>.</p>
</li>
<li>
<p>Generate the interface. From the directory where your proto files live, run:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl">python -m grpc_tools.protoc -I. <span class="se">\
</span></span></span><span class="line"><span class="cl">    --python_out<span class="o">=</span>contracts <span class="se">\
</span></span></span><span class="line"><span class="cl">    --grpc_python_out<span class="o">=</span>contracts protos/message.proto
</span></span></code></pre></div></li>
<li>
<p>This will generate the following files in the root directory:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-txt" data-lang="txt"><span class="line"><span class="cl">search
</span></span><span class="line"><span class="cl">├── contracts
</span></span><span class="line"><span class="cl">│   └── protos
</span></span><span class="line"><span class="cl">│       ├── message_pb2.py
</span></span><span class="line"><span class="cl">│       └── message_pb2_grpc.py
</span></span><span class="line"><span class="cl">└── protos
</span></span><span class="line"><span class="cl">    └── message.proto
</span></span></code></pre></div></li>
</ul>
<h2 id="serialize-and-publish">Serialize and publish</h2>
<p>Once you have the contracts in place and have generated the interfacing code, here&rsquo;s how you
can serialize a message payload before publishing it to an event stream:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># ./search/services/publish.py</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">contracts.protos.message_pb2</span> <span class="kn">import</span> <span class="n">SearchRequest</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">serialize</span><span class="p">(</span><span class="n">query</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">page_number</span><span class="p">:</span> <span class="nb">int</span><span class="p">,</span> <span class="n">results_per_page</span><span class="p">:</span> <span class="nb">int</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">search_request</span> <span class="o">=</span> <span class="n">SearchRequest</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="n">query</span><span class="o">=</span><span class="n">query</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="n">page_number</span><span class="o">=</span><span class="n">page_number</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="n">results_per_page</span><span class="o">=</span><span class="n">results_per_page</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1"># Serialize the search request to a compact binary string</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">search_request</span><span class="o">.</span><span class="n">SerializeToString</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">publish</span><span class="p">(</span><span class="n">serialized_message</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="c1"># Publish the message to a message broker</span>
</span></span><span class="line"><span class="cl">    <span class="o">...</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">&#34;__main__&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">serialized_message</span> <span class="o">=</span> <span class="n">serialize</span><span class="p">(</span><span class="s2">&#34;foo bar&#34;</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">5</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">publish</span><span class="p">(</span><span class="n">serialized_message</span><span class="p">)</span>
</span></span></code></pre></div><p>The code is structured in the following manner now:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-txt" data-lang="txt"><span class="line"><span class="cl">search
</span></span><span class="line"><span class="cl">├── contracts
</span></span><span class="line"><span class="cl">│   ├── __init__.py
</span></span><span class="line"><span class="cl">│   └── protos
</span></span><span class="line"><span class="cl">│       ├── message_pb2.py
</span></span><span class="line"><span class="cl">│       └── message_pb2_grpc.py
</span></span><span class="line"><span class="cl">├── protos
</span></span><span class="line"><span class="cl">│   └── message.proto
</span></span><span class="line"><span class="cl">└── services
</span></span><span class="line"><span class="cl">    ├── __init__.py
</span></span><span class="line"><span class="cl">    └── publish.py
</span></span></code></pre></div><h2 id="deserialize-and-consume">Deserialize and consume</h2>
<p>On the consumer side, if you have access to the proto files, you can generate the interface
code again via the same commands as before and use it to deserialize the message</p>
<p>as follows:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># ./search/services/consume.py</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">contracts.protos.message_pb2</span> <span class="kn">import</span> <span class="n">SearchRequest</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">get_message</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="c1"># Let&#39;s say we get the message from a message broker and return it</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="sa">b</span><span class="s2">&#34;</span><span class="se">\n\x04</span><span class="s2">test</span><span class="se">\x10\x01\x18\x02</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">deserialize</span><span class="p">(</span><span class="n">serialized_message</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">SearchRequest</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">search_request</span> <span class="o">=</span> <span class="n">SearchRequest</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="n">search_request</span><span class="o">.</span><span class="n">ParseFromString</span><span class="p">(</span><span class="n">serialized_message</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">search_request</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">consume</span><span class="p">(</span><span class="n">message</span><span class="p">:</span> <span class="n">SearchRequest</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span> <span class="o">...</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">&#34;__main__&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">serialized_message</span> <span class="o">=</span> <span class="n">get_message</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="n">search_request</span> <span class="o">=</span> <span class="n">deserialize</span><span class="p">(</span><span class="n">serialized_message</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">consume</span><span class="p">(</span><span class="n">search_request</span><span class="p">)</span>
</span></span></code></pre></div><p>You can even save the proto files in a common repo, generate the interfacing code
automatically for multiple languages, and package them up via CI whenever some changes are
merged into the main branch. Then the services can just update those protocol packages and
use the serializers and deserializers as needed.</p>
<!-- references -->
<!-- prettier-ignore-start -->
<!-- prettier-ignore-end -->
]]></content:encoded>
    </item>
    <item>
      <title>ETag and HTTP caching</title>
      <link>https://rednafi.com/misc/etag-and-http-caching/</link>
      <pubDate>Wed, 10 Apr 2024 00:00:00 +0000</pubDate>
      <guid>https://rednafi.com/misc/etag-and-http-caching/</guid>
      <description>Implement client-side HTTP caching with ETag headers. Learn If-None-Match, 304 Not Modified responses, and weak validation in Go servers.</description>
      <category>API</category>
      <category>Go</category>
      <category>Web</category>
      <content:encoded><![CDATA[<p>One neat use case for the HTTP <code>ETag</code> header is client-side HTTP caching for <code>GET</code> requests.
Along with the <code>ETag</code> header, the caching workflow requires you to fiddle with other
conditional HTTP headers like <code>If-Match</code> or <code>If-None-Match</code>. However, their interaction can
feel a bit confusing at times.</p>
<p>Every time I need to tackle this, I end up spending some time browsing through the relevant
MDN docs on <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag" rel="noopener noreferrer" target="_blank">ETag</a>, <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-Match" rel="noopener noreferrer" target="_blank">If-Match</a>, and <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-None-Match" rel="noopener noreferrer" target="_blank">If-None-Match</a> to jog my memory. At this point, I&rsquo;ve
done it enough times to justify spending the time to write this.</p>
<h2 id="caching-the-response-of-a-get-endpoint">Caching the response of a <code>GET</code> endpoint</h2>
<p>The basic workflow goes as follows:</p>
<ul>
<li>The client makes a <code>GET</code> request to the server.</li>
<li>The server responds with a <code>200 OK</code> status, including the content requested and an <code>ETag</code>
header.</li>
<li>The client caches the response and the <code>ETag</code> value.</li>
<li>For subsequent requests to the same resource, the client includes the <code>If-None-Match</code>
header with the <code>ETag</code> value it has cached.</li>
<li>The server regenerates the <code>ETag</code> independently and checks if the <code>ETag</code> value sent by the
client matches the generated one.
<ul>
<li>If they match, the server responds with a <code>304 Not Modified</code> status, indicating that
the client&rsquo;s cached version is still valid, and the client serves the resource from
the cache.</li>
<li>If they don&rsquo;t match, the server responds with a <code>200 OK</code> status, including the new
content and a new <code>ETag</code> header, prompting the client to update its cache.</li>
</ul>
</li>
</ul>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-txt" data-lang="txt"><span class="line"><span class="cl">Client                                 Server
</span></span><span class="line"><span class="cl">  |                                       |
</span></span><span class="line"><span class="cl">  |----- GET Request --------------------&gt;|
</span></span><span class="line"><span class="cl">  |                                       |
</span></span><span class="line"><span class="cl">  |&lt;---- Response 200 OK + ETag ----------|
</span></span><span class="line"><span class="cl">  |     (Cache response locally)          |
</span></span><span class="line"><span class="cl">  |                                       |
</span></span><span class="line"><span class="cl">  |----- GET Request + If-None-Match ----&gt;|  (If-None-Match == previous ETag)
</span></span><span class="line"><span class="cl">  |                                       |
</span></span><span class="line"><span class="cl">  |       Does ETag match?                |
</span></span><span class="line"><span class="cl">  |&lt;---- Yes: 304 Not Modified -----------|  (No body sent; Use local cache)
</span></span><span class="line"><span class="cl">  |       No: 200 OK + New ETag ----------|  (Update cached response)
</span></span><span class="line"><span class="cl">  |                                       |
</span></span></code></pre></div><p>We can test this workflow with GitHub&rsquo;s REST API suite via the <a href="https://cli.github.com/" rel="noopener noreferrer" target="_blank">GitHub CLI</a>. If you&rsquo;ve
installed the CLI and authenticated yourself, you can make a request like this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl">gh api -i  /users/rednafi
</span></span></code></pre></div><p>This asks for the data associated with the user <code>rednafi</code>. The response looks as follows:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-txt" data-lang="txt"><span class="line"><span class="cl">HTTP/2.0 200 OK
</span></span><span class="line"><span class="cl">Etag: W/&#34;b8fdfabd59aed6e0e602dd140c0a0ff48a665cac791dede458c5109bf4bf9463&#34;
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">{
</span></span><span class="line"><span class="cl">  &#34;login&#34;:&#34;rednafi&#34;,
</span></span><span class="line"><span class="cl">  &#34;id&#34;:30027932,
</span></span><span class="line"><span class="cl">  ...
</span></span><span class="line"><span class="cl">}
</span></span></code></pre></div><p>I&rsquo;ve truncated the response body and omitted the headers that aren&rsquo;t relevant to this
discussion. You can see that the HTTP status code is <code>200 OK</code> and the server has included an
<code>ETag</code> header.</p>
<p>The <code>W/</code> prefix indicates that a <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Conditional_requests#weak_validation" rel="noopener noreferrer" target="_blank">weak validator</a> is used to validate the content of the
cache. Using a weak validator means when the server compares the response payload to
generate the hash, it doesn&rsquo;t do it bit-by-bit. So, if your response is JSON, then changing
the format of the JSON won&rsquo;t change the value of the <code>ETag</code> header since two JSON payloads
with the same content but with different formatting are semantically the same thing.</p>
<p>Let&rsquo;s see what happens if we make the same request again while passing the value of the
<code>ETag</code> in the <code>If-None-Match</code> header.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl">gh api -i <span class="se">\
</span></span></span><span class="line"><span class="cl">    -H <span class="s1">&#39;If-None-Match: W/&#34;b8fdfabd59aed6e0e602dd140c0a...&#34;&#39;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">    /users/rednafi
</span></span></code></pre></div><p>This returns:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-txt" data-lang="txt"><span class="line"><span class="cl">HTTP/2.0 304 Not Modified
</span></span><span class="line"><span class="cl">Etag: &#34;b8fdfabd59aed6e0e602dd140c0a0ff48a665cac791dede458c5109bf4bf9463&#34;
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">gh: HTTP 304
</span></span></code></pre></div><p>This means that the cached response in the client is still valid and it doesn&rsquo;t need to
refetch that from the server. So, the client can be coded to serve the previously cached
data to the users when asked for.</p>
<p>A few key points to keep in mind:</p>
<ul>
<li>
<p>Always wrap your <code>ETag</code> values in double quotes when sending them with the <code>If-None-Match</code>
header, just as the <a href="https://www.rfc-editor.org/rfc/rfc7232#section-3.2" rel="noopener noreferrer" target="_blank">spec says for conditional header values</a>.</p>
</li>
<li>
<p>Using the <code>If-None-Match</code> header to pass the <code>ETag</code> value means that the client request is
considered successful when the <code>ETag</code> value from the client doesn&rsquo;t match that of the
server. When the values match, the server will return <code>304 Not Modified</code> with no body.</p>
</li>
<li>
<p>If we&rsquo;re writing a compliant server, when it comes to <code>If-None-Match</code>, the <a href="https://www.rfc-editor.org/rfc/rfc7232#section-2.3.2" rel="noopener noreferrer" target="_blank">spec tells us
to use weak comparison for ETags</a>. This means that the client will still be able to
validate the cache with weak ETags, even if there have been slight changes to the
representation of the data.</p>
</li>
<li>
<p>If the client is a browser, it&rsquo;ll automatically manage the cache and send conditional
requests without any extra work.</p>
</li>
</ul>
<h2 id="writing-a-server-that-enables-client-side-caching">Writing a server that enables client-side caching</h2>
<p>If you&rsquo;re serving static content, you can configure your load balancer to enable this
caching workflow. But for dynamic <code>GET</code> requests, the server needs to do a bit more work to
allow client-side caching.</p>
<p>Here&rsquo;s a simple server in Go that enables the above workflow for a dynamic <code>GET</code> request:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kn">package</span><span class="w"> </span><span class="nx">main</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kn">import</span><span class="w"> </span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="s">&#34;crypto/sha256&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="s">&#34;encoding/hex&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="s">&#34;fmt&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="s">&#34;net/http&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="s">&#34;strings&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// calculateETag generates a weak ETag by SHA-256-hashing the content</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// and prefixing it with W/ to indicate a weak comparison</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">calculateETag</span><span class="p">(</span><span class="nx">content</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="kt">string</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">hasher</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">sha256</span><span class="p">.</span><span class="nf">New</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">hasher</span><span class="p">.</span><span class="nf">Write</span><span class="p">([]</span><span class="nb">byte</span><span class="p">(</span><span class="nx">content</span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">hash</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">hex</span><span class="p">.</span><span class="nf">EncodeToString</span><span class="p">(</span><span class="nx">hasher</span><span class="p">.</span><span class="nf">Sum</span><span class="p">(</span><span class="kc">nil</span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Sprintf</span><span class="p">(</span><span class="s">&#34;W/\&#34;%s\&#34;&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">hash</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">http</span><span class="p">.</span><span class="nf">HandleFunc</span><span class="p">(</span><span class="s">&#34;/&#34;</span><span class="p">,</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// Define the content within the handler</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">content</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="s">`{&#34;message&#34;: &#34;Hello, world!&#34;}`</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">eTag</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">calculateETag</span><span class="p">(</span><span class="nx">content</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// Remove quotes and W/ prefix for If-None-Match header comparison</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">ifNoneMatch</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">strings</span><span class="p">.</span><span class="nf">TrimPrefix</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">strings</span><span class="p">.</span><span class="nf">Trim</span><span class="p">(</span><span class="nx">r</span><span class="p">.</span><span class="nx">Header</span><span class="p">.</span><span class="nf">Get</span><span class="p">(</span><span class="s">&#34;If-None-Match&#34;</span><span class="p">),</span><span class="w"> </span><span class="s">&#34;\&#34;&#34;</span><span class="p">),</span><span class="w"> </span><span class="s">&#34;W/&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// Hash content without W/ prefix for comparison</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">contentHash</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">strings</span><span class="p">.</span><span class="nf">TrimPrefix</span><span class="p">(</span><span class="nx">eTag</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;W/&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// Check if the ETag matches; if so, return 304 Not Modified</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">if</span><span class="w"> </span><span class="nx">ifNoneMatch</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="nx">strings</span><span class="p">.</span><span class="nf">Trim</span><span class="p">(</span><span class="nx">contentHash</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;\&#34;&#34;</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">w</span><span class="p">.</span><span class="nf">WriteHeader</span><span class="p">(</span><span class="nx">http</span><span class="p">.</span><span class="nx">StatusNotModified</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="k">return</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// If ETag does not match, return the content and the ETag</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">w</span><span class="p">.</span><span class="nf">Header</span><span class="p">().</span><span class="nf">Set</span><span class="p">(</span><span class="s">&#34;ETag&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">eTag</span><span class="p">)</span><span class="w"> </span><span class="c1">// Send weak ETag</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">w</span><span class="p">.</span><span class="nf">Header</span><span class="p">().</span><span class="nf">Set</span><span class="p">(</span><span class="s">&#34;Content-Type&#34;</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;application/json&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">w</span><span class="p">.</span><span class="nf">WriteHeader</span><span class="p">(</span><span class="nx">http</span><span class="p">.</span><span class="nx">StatusOK</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Fprint</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="nx">content</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Println</span><span class="p">(</span><span class="s">&#34;Server is running on http://localhost:8080&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">http</span><span class="p">.</span><span class="nf">ListenAndServe</span><span class="p">(</span><span class="s">&#34;:8080&#34;</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><ul>
<li>
<p>The server generates a weak <code>ETag</code> for its content by creating a SHA-256 hash and adding
<code>W/</code> to the front, indicating it&rsquo;s meant for weak comparison.</p>
<p>You could make the <code>calculateETag</code> function format-agnostic, so the hash stays the same
if the JSON format changes but the content does not. The current <code>calculateETag</code>
implementation is susceptible to format changes, and I kept it that way to keep the code
shorter.</p>
</li>
<li>
<p>When delivering content, the server includes this weak <code>ETag</code> in the response headers,
allowing clients to cache the content along with the <code>ETag</code>.</p>
</li>
<li>
<p>For subsequent requests, the server checks if the client has sent an <code>ETag</code> in the
<code>If-None-Match</code> header and weakly compares it with the current content&rsquo;s <code>ETag</code> by
independently generating the hash.</p>
</li>
<li>
<p>If the ETags match, indicating no significant content change, the server replies with a
<code>304 Not Modified</code> status. Otherwise, it sends the content again with a <code>200 OK</code> status
and updates the <code>ETag</code>. When this happens, the client knows that the existing cache is
still warm and can be served without any changes to it.</p>
</li>
</ul>
<p>You can spin up the server by running <code>go run main.go</code> and from a different console, start
making requests to it like this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl">curl -i  http://localhost:8080/foo
</span></span></code></pre></div><p>This will return the <code>ETag</code> header along with the JSON response:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-txt" data-lang="txt"><span class="line"><span class="cl">HTTP/1.1 200 OK
</span></span><span class="line"><span class="cl">Content-Type: application/json
</span></span><span class="line"><span class="cl">Etag: W/&#34;1d3b4242cc9039faa663d7ca51a25798e91fbf7675c9007c2b0470b72c2ed2f3&#34;
</span></span><span class="line"><span class="cl">Date: Wed, 10 Apr 2024 15:54:33 GMT
</span></span><span class="line"><span class="cl">Content-Length: 28
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">{&#34;message&#34;: &#34;Hello, world!&#34;}
</span></span></code></pre></div><p>Now, you can make another request with the value of</p>
<p>the <code>ETag</code> in the <code>If-None-Match</code> header:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl">curl -i <span class="se">\
</span></span></span><span class="line"><span class="cl">    -H <span class="s1">&#39;If-None-Match: &#34;1d3b4242cc9039faa663d7ca51a2...&#34;&#39;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">    http://localhost:8080/foo
</span></span></code></pre></div><p>This will return a <code>304 Not Modified</code> response with no body:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-txt" data-lang="txt"><span class="line"><span class="cl">HTTP/1.1 304 Not Modified
</span></span><span class="line"><span class="cl">Date: Wed, 10 Apr 2024 15:57:25 GMT
</span></span></code></pre></div><p>In a real-life scenario, you&rsquo;ll probably factor out the caching part in middleware so that
all of your HTTP <code>GET</code> requests can be cached from the client-side without repetition.</p>
<h2 id="one-thing-to-look-out-for">One thing to look out for</h2>
<p>While writing a cache-enabled server, make sure the system is set up so that the server
always sends back the same <code>ETag</code> for the same content, even when there are multiple servers
working behind a load balancer. If these servers give out different ETags for the same
content, it can mess up how clients cache that content.</p>
<p>Clients use ETags to decide if content has changed. If the <code>ETag</code> value hasn&rsquo;t changed, they
know the content is the same and don&rsquo;t download it again, saving bandwidth and speeding up
access. But if ETags are inconsistent across servers, clients might download content they
already have, wasting bandwidth and slowing things down.</p>
<p>This inconsistency also means servers end up dealing with more requests for content that
clients could have just used from their cache if ETags were consistent.</p>
<!-- references -->
<!-- prettier-ignore-start -->
<!-- prettier-ignore-end -->
]]></content:encoded>
    </item>
    <item>
      <title>Dysfunctional options pattern in Go</title>
      <link>https://rednafi.com/go/dysfunctional-options-pattern/</link>
      <pubDate>Wed, 06 Mar 2024 00:00:00 +0000</pubDate>
      <guid>https://rednafi.com/go/dysfunctional-options-pattern/</guid>
      <description>Discover a simpler alternative to functional options: method chaining with builder-style configuration that&amp;#39;s 76x faster and easier to understand.</description>
      <category>Go</category>
      <category>API</category>
      <category>Performance</category>
      <content:encoded><![CDATA[<p>Ever since Rob Pike published the text on the <a href="https://commandcenter.blogspot.com/2014/01/self-referential-functions-and-design.html" rel="noopener noreferrer" target="_blank">functional options pattern</a>, there&rsquo;s been no
shortage of blogs, talks, or comments on how it improves or obfuscates configuration
ergonomics.</p>
<p>While the necessity of such a pattern is quite evident in a language that lacks default
arguments in functions, more often than not, it needlessly complicates things. The situation
gets worse if you have to maintain a public API where multiple configurations are controlled
in this manner.</p>
<p>However, the pattern solves a valid problem and definitely has its place. Otherwise, it
wouldn&rsquo;t have been picked up by pretty much every other library, whether it&rsquo;s <a href="https://github.com/ngrok/ngrok-api-go/blob/ec1a3e91cae94c70f0e5c31b95aed5a1d6dd65b7/client_config.go" rel="noopener noreferrer" target="_blank">Ngrok</a> or the
<a href="https://github.com/elastic/elastic-agent/blob/4aeba5b3fcf0d72924c70ff2127996a817b83a23/pkg/testing/fetcher_http.go" rel="noopener noreferrer" target="_blank">Elasticsearch agent</a>.</p>
<p>If you have no idea what I&rsquo;m talking about, you might want to give my previous write-up on
<a href="/go/configure-options/">configuring options</a> a read.</p>
<h2 id="functional-options-pattern">Functional options pattern</h2>
<p>As a recap, here&rsquo;s how the functional options pattern works. Let&rsquo;s say, you need to allow
the users of your API to configure something. You can expose a struct from your package
that&rsquo;ll be passed to some other function to tune its behavior. For example:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kn">package</span><span class="w"> </span><span class="nx">src</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">Config</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Required</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">Foo</span><span class="p">,</span><span class="w"> </span><span class="nx">Bar</span><span class="w"> </span><span class="kt">string</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Optional</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">Fizz</span><span class="p">,</span><span class="w"> </span><span class="nx">Bazz</span><span class="w"> </span><span class="kt">int</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">Do</span><span class="p">(</span><span class="nx">config</span><span class="w"> </span><span class="o">*</span><span class="nx">Config</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Do something with the config values</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Then the <code>Config</code> struct will be imported, initialized, and passed to the <code>Do</code> function by
your API users:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kn">package</span><span class="w"> </span><span class="nx">main</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kn">import</span><span class="w"> </span><span class="s">&#34;.../src&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Initialize the config and pass it to the Do function</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">config</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">src</span><span class="p">.</span><span class="nx">Config</span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">Foo</span><span class="p">:</span><span class="w"> </span><span class="s">&#34;hello&#34;</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">Bar</span><span class="p">:</span><span class="w"> </span><span class="s">&#34;world&#34;</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">Fizz</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">Bazz</span><span class="p">:</span><span class="w"> </span><span class="mi">42</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Call Do with the initialized Config struct</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">src</span><span class="p">.</span><span class="nf">Do</span><span class="p">(</span><span class="nx">config</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>This is one way of doing that, but it&rsquo;s generally discouraged since it requires you to
expose the internals of your API to the users. So instead, a library usually exposes a
factory function that&rsquo;ll do the struct initialization while keeping the struct and fields
private. For instance:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kn">package</span><span class="w"> </span><span class="nx">src</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">config</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="c1">// Notice that the struct and fields are now private</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Required</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">foo</span><span class="p">,</span><span class="w"> </span><span class="nx">bar</span><span class="w"> </span><span class="kt">string</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Optional</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">fizz</span><span class="p">,</span><span class="w"> </span><span class="nx">bazz</span><span class="w"> </span><span class="kt">int</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Public factory function</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">NewConfig</span><span class="p">(</span><span class="nx">foo</span><span class="p">,</span><span class="w"> </span><span class="nx">bar</span><span class="w"> </span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="nx">fizz</span><span class="p">,</span><span class="w"> </span><span class="nx">bazz</span><span class="w"> </span><span class="kt">int</span><span class="p">)</span><span class="w"> </span><span class="nx">config</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">config</span><span class="p">{</span><span class="nx">foo</span><span class="p">,</span><span class="w"> </span><span class="nx">bar</span><span class="p">,</span><span class="w"> </span><span class="nx">fizz</span><span class="p">,</span><span class="w"> </span><span class="nx">bazz</span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">Do</span><span class="p">(</span><span class="nx">c</span><span class="w"> </span><span class="o">*</span><span class="nx">config</span><span class="p">){}</span><span class="w">
</span></span></span></code></pre></div><p>The API consumers will now use <code>NewConfig</code> to produce a configuration and then pass the
struct instance to the <code>Do</code> function as follows:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kn">package</span><span class="w"> </span><span class="nx">main</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kn">import</span><span class="w"> </span><span class="s">&#34;.../src&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Initialize the config with the NewConfig factory</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">c</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">src</span><span class="p">.</span><span class="nf">NewConfig</span><span class="p">(</span><span class="s">&#34;hello&#34;</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;world&#34;</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="mi">42</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Call Do with the initialized Config struct</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">src</span><span class="p">.</span><span class="nf">Do</span><span class="p">(</span><span class="nx">c</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>This approach is better as it keeps the internal machinery hidden from users. However, it
doesn&rsquo;t allow for setting default values for some configuration attributes; all must be set
explicitly. What if your users want to override the values of multiple attributes? This
leads your configuration struct to be overloaded with options, making the <code>NewConfig</code>
function demands numerous positional arguments.</p>
<p>This setup isn&rsquo;t user-friendly, as it forces API users to explicitly pass all these options
to the <code>NewConfig</code> factory. Ideally, you&rsquo;d initialize <code>config</code> with some default values,
offering users a chance to override them. But, Go doesn&rsquo;t support default values for
function arguments, which compels us to be creative and come up with different workarounds.
Functional options pattern is one of them.</p>
<p>Here&rsquo;s how you can build your API to leverage the pattern:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kn">package</span><span class="w"> </span><span class="nx">src</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">config</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Required</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">foo</span><span class="p">,</span><span class="w"> </span><span class="nx">bar</span><span class="w"> </span><span class="kt">string</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Optional</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">fizz</span><span class="p">,</span><span class="w"> </span><span class="nx">bazz</span><span class="w"> </span><span class="kt">int</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">option</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="o">*</span><span class="nx">config</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Each optional config attribute can be overridden with</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// an associated function</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">WithFizz</span><span class="p">(</span><span class="nx">fizz</span><span class="w"> </span><span class="kt">int</span><span class="p">)</span><span class="w"> </span><span class="nx">option</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="nx">c</span><span class="w"> </span><span class="o">*</span><span class="nx">config</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">c</span><span class="p">.</span><span class="nx">fizz</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">fizz</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">WithBazz</span><span class="p">(</span><span class="nx">bazz</span><span class="w"> </span><span class="kt">int</span><span class="p">)</span><span class="w"> </span><span class="nx">option</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="nx">c</span><span class="w"> </span><span class="o">*</span><span class="nx">config</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">c</span><span class="p">.</span><span class="nx">bazz</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">bazz</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">NewConfig</span><span class="p">(</span><span class="nx">foo</span><span class="p">,</span><span class="w"> </span><span class="nx">bar</span><span class="w"> </span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="nx">opts</span><span class="w"> </span><span class="o">...</span><span class="nx">option</span><span class="p">)</span><span class="w"> </span><span class="nx">config</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// First fill in the options with default values</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">c</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">config</span><span class="p">{</span><span class="nx">foo</span><span class="p">,</span><span class="w"> </span><span class="nx">bar</span><span class="p">,</span><span class="w"> </span><span class="mi">10</span><span class="p">,</span><span class="w"> </span><span class="mi">100</span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Now allow users to override the optional configuration attributes</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">for</span><span class="w"> </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">opt</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">range</span><span class="w"> </span><span class="nx">opts</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nf">opt</span><span class="p">(</span><span class="o">&amp;</span><span class="nx">c</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">c</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">Do</span><span class="p">(</span><span class="nx">c</span><span class="w"> </span><span class="o">*</span><span class="nx">config</span><span class="p">)</span><span class="w"> </span><span class="p">{}</span><span class="w">
</span></span></span></code></pre></div><p>Then you&rsquo;d use it as follows:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kn">package</span><span class="w"> </span><span class="nx">main</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kn">import</span><span class="w"> </span><span class="s">&#34;.../src&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">c</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">src</span><span class="p">.</span><span class="nf">NewConfig</span><span class="p">(</span><span class="s">&#34;hello&#34;</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;world&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">src</span><span class="p">.</span><span class="nf">WithFizz</span><span class="p">(</span><span class="mi">1</span><span class="p">),</span><span class="w"> </span><span class="nx">src</span><span class="p">.</span><span class="nf">WithBazz</span><span class="p">(</span><span class="mi">2</span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">src</span><span class="p">.</span><span class="nf">Do</span><span class="p">(</span><span class="nx">c</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Functional options pattern relies on functions that modify the configuration struct&rsquo;s state.
These modifier functions, or option functions, are defined to accept a pointer to the
configuration struct <code>*config</code> and then directly alter its fields. This direct manipulation
is possible because the option functions are closures, which means they capture and modify
the variables from their enclosing scope, in this case, the <code>config</code> instance.</p>
<p>In the <code>NewConfig</code> factory, the variadic parameter <code>opts ...option</code> allows for an arbitrary
number of option functions to be passed. Here, <code>opts</code> represents the optional configurations
that the users can override if they want to.</p>
<p>The <code>NewConfig</code> function iterates over this slice of option functions, invoking each one
with the <code>&amp;c</code> argument, which is a pointer to the newly created <code>config</code> instance. The
config instance is created with default values, and the users can use the <code>With*</code> functions
to override them.</p>
<h2 id="curse-of-indirection">Curse of indirection</h2>
<p>That&rsquo;s a fair bit of indirection just to allow API users to configure some options. I don&rsquo;t
know about you, but multi-layered higher-order functions hurt my brain. It&rsquo;s quite slow as
well.</p>
<p>All this complexity could have been avoided if Go allowed default arguments in functions.
Your configuration factory could simply grab the default values from the keyword arguments
and pass them to the underlying struct. The idea that supporting default arguments in
functions would lead to a parameter explosion seems unfounded, especially when the
alternative requires gymnastics like the functional option pattern.</p>
<p>Also, the multiple layers of indirection hinder API discoverability. Trying to discover
modifier functions by hovering your cursor over the factory function&rsquo;s return value in the
IDE won&rsquo;t be very helpful, as these functions are defined at the package level.</p>
<p>So, if you need to configure multiple structs in this manner, the explosion of their
respective package-level modifiers make it even harder for the user to know which function
they&rsquo;ll need to use to update a certain configuration attribute.</p>
<p>Recently, I&rsquo;ve spontaneously stumbled upon a fluent-style API to manage configurations that
don&rsquo;t require so many layers of indirection and lets you expose optional configuration
attributes. Let&rsquo;s call it <strong>dysfunctional options pattern</strong>.</p>
<h2 id="dysfunctional-options-pattern">Dysfunctional options pattern</h2>
<p>The idea is quite similar to how the API with functional options pattern is constructed.
Here&rsquo;s the complete implementation:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kn">package</span><span class="w"> </span><span class="nx">src</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">config</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Required</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">foo</span><span class="p">,</span><span class="w"> </span><span class="nx">bar</span><span class="w"> </span><span class="kt">string</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Optional</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">fizz</span><span class="p">,</span><span class="w"> </span><span class="nx">bazz</span><span class="w"> </span><span class="kt">int</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Each optional configuration attribute will have its own public method</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">c</span><span class="w"> </span><span class="o">*</span><span class="nx">config</span><span class="p">)</span><span class="w"> </span><span class="nf">WithFizz</span><span class="p">(</span><span class="nx">fizz</span><span class="w"> </span><span class="kt">int</span><span class="p">)</span><span class="w"> </span><span class="o">*</span><span class="nx">config</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">c</span><span class="p">.</span><span class="nx">fizz</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">fizz</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">c</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">c</span><span class="w"> </span><span class="o">*</span><span class="nx">config</span><span class="p">)</span><span class="w"> </span><span class="nf">WithBazz</span><span class="p">(</span><span class="nx">bazz</span><span class="w"> </span><span class="kt">int</span><span class="p">)</span><span class="w"> </span><span class="o">*</span><span class="nx">config</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">c</span><span class="p">.</span><span class="nx">bazz</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">bazz</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">c</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// This only accepts the required options as params</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">NewConfig</span><span class="p">(</span><span class="nx">foo</span><span class="p">,</span><span class="w"> </span><span class="nx">bar</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="o">*</span><span class="nx">config</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// First fill in the options with default values</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">config</span><span class="p">{</span><span class="nx">foo</span><span class="p">,</span><span class="w"> </span><span class="nx">bar</span><span class="p">,</span><span class="w"> </span><span class="mi">10</span><span class="p">,</span><span class="w"> </span><span class="mi">100</span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">Do</span><span class="p">(</span><span class="nx">c</span><span class="w"> </span><span class="o">*</span><span class="nx">config</span><span class="p">)</span><span class="w"> </span><span class="p">{}</span><span class="w">
</span></span></span></code></pre></div><p>You&rsquo;d use the API as follows:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kn">package</span><span class="w"> </span><span class="nx">main</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kn">import</span><span class="w"> </span><span class="s">&#34;.../src&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Initialize the struct with only the required options and then chain</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// the option methods to update the optional configuration attributes</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">c</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">src</span><span class="p">.</span><span class="nf">NewConfig</span><span class="p">(</span><span class="s">&#34;hello&#34;</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;world&#34;</span><span class="p">).</span><span class="nf">WithFizz</span><span class="p">(</span><span class="mi">0</span><span class="p">).</span><span class="nf">WithBazz</span><span class="p">(</span><span class="mi">42</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">src</span><span class="p">.</span><span class="nf">Do</span><span class="p">(</span><span class="nx">c</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Similar to the previous pattern, we have modifiers here too. However, instead of being
higher order functions, the modifiers are methods on <code>config</code> and return a pointer to the
struct.</p>
<p>The <code>NewConfig</code> factory function instantiates the <code>config</code> struct with some default values
and returns the struct pointer like the modifiers. This enables us to chain the <code>WithFizz</code>
and <code>WithBazz</code> modifiers on the returned value of <code>NewConfig</code> and update the values of the
optional configuration attributes.</p>
<p>Apart from simplicity and the lack of magic, you can hover over the return type of the
factory and immediately know about the supported modifier methods.</p>
<p>I did a <a href="https://gist.github.com/rednafi/08fe371ed31072ab0bd96bf51611660a" rel="noopener noreferrer" target="_blank">rudimentary benchmark</a> of the two approaches and was surprised that the second one
was roughly ~76x faster on Go 1.22!</p>
<p>Here&rsquo;s an example in <a href="https://github.com/rednafi/fork-sweeper/blob/80e1f7c76a2efcb7d1b65d6b12303c590bb74c2c/src/cli.go#L172" rel="noopener noreferrer" target="_blank">fork-sweeper&rsquo;s CLI code</a>.</p>
<p><em>P.S. This is indeed a lightweight spin on what OO languages call the builder pattern.
However, I didn&rsquo;t call it that because there&rsquo;s no mandatory <code>.Build()</code> method to be called
at the end of the method chain.</em></p>
<!-- references -->
<!-- prettier-ignore-start -->
<!-- functional options pattern in ngrok -->
<!-- functional options pattern in elasticsearch agent -->
<!-- configuring options in go - rednafi -->
<!-- benchmarking functional vs dysfunctional options pattern -->
<!-- prettier-ignore-end -->
]]></content:encoded>
    </item>
    <item>
      <title>Configuring options in Go</title>
      <link>https://rednafi.com/go/configure-options/</link>
      <pubDate>Tue, 05 Sep 2023 00:00:00 +0000</pubDate>
      <guid>https://rednafi.com/go/configure-options/</guid>
      <description>Compare three Go option patterns: exposed structs, option constructors, and functional options. Learn when to use each for clean APIs.</description>
      <category>Go</category>
      <category>API</category>
      <category>Design Patterns</category>
      <content:encoded><![CDATA[<p>Suppose, you have a function that takes an option struct and a message as input. Then it
stylizes the message according to the option fields and prints it. What&rsquo;s the most sensible
API you can offer for users to configure your function? Observe:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// app/src</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kn">package</span><span class="w"> </span><span class="nx">src</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Option struct</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">Style</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">Fg</span><span class="w"> </span><span class="kt">string</span><span class="w"> </span><span class="c1">// ANSI escape codes for foreground color</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">Bg</span><span class="w"> </span><span class="kt">string</span><span class="w"> </span><span class="c1">// Background color</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Display the message according to Style</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">Display</span><span class="p">(</span><span class="nx">s</span><span class="w"> </span><span class="o">*</span><span class="nx">Style</span><span class="p">,</span><span class="w"> </span><span class="nx">msg</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">{}</span><span class="w">
</span></span></span></code></pre></div><p>In the <code>src</code> package, the function <code>Display</code> takes a pointer to a <code>Style</code> instance and a
<code>msg</code> string as parameters. Then it decorates the <code>msg</code> and prints it according to the style
specified in the option struct. In the wild, I&rsquo;ve seen 3 main ways to write APIs that let
users configure options:</p>
<ul>
<li>Expose the option struct directly</li>
<li>Use the option constructor pattern</li>
<li>Apply functional option constructor pattern</li>
</ul>
<p>Each comes with its own pros and cons.</p>
<h2 id="expose-the-option-struct">Expose the option struct</h2>
<p>In this case, you&rsquo;d export the <code>Style</code> struct with all its fields and let the user configure
them directly. The previous snippet already made the struct and fields public. From another
package, you could import the <code>src</code> package and instantiate <code>Style</code> like this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kn">package</span><span class="w"> </span><span class="nx">main</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kn">import</span><span class="w"> </span><span class="s">&#34;app/src&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Users instantiate the option struct</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">c</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">src</span><span class="p">.</span><span class="nx">Style</span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="s">&#34;\033[31m&#34;</span><span class="p">,</span><span class="w"> </span><span class="c1">// Maroon</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="s">&#34;\033[43m&#34;</span><span class="p">,</span><span class="w"> </span><span class="c1">// Yellow</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Then pass the struct to the function</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nf">Display</span><span class="p">(</span><span class="nx">c</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;Hello, World!&#34;</span><span class="p">)</span><span class="w">
</span></span></span></code></pre></div><p>To configure option fields, mutate the values in place:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">c</span><span class="p">.</span><span class="nx">Fg</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="s">&#34;\033[35m&#34;</span><span class="w"> </span><span class="c1">// Magenta</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">c</span><span class="p">.</span><span class="nx">Bg</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="s">&#34;\033[40m&#34;</span><span class="w"> </span><span class="c1">// Black</span><span class="w">
</span></span></span></code></pre></div><p>This works but will break users&rsquo; code if new fields are added to the option struct. But your
users can instantiate the struct with named parameters to avoid breakage:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">c</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">src</span><span class="p">.</span><span class="nx">Style</span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">Fg</span><span class="p">:</span><span class="w"> </span><span class="s">&#34;\033[31m&#34;</span><span class="p">,</span><span class="w"> </span><span class="c1">// Maroon</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                   </span><span class="c1">// Bg will be implicitly set to an empty string</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>In this case, the field that wasn&rsquo;t passed would assume the corresponding zero value. For
instance, <code>Bg</code> will be initialized as an empty string. However, this pattern puts the
responsibility of retaining API compatibility on the users&rsquo; shoulders. So if your code is
meant for external use, there are better ways to achieve option configurability.</p>
<h2 id="option-constructor">Option constructor</h2>
<p>Go standard library extensively uses this pattern. Instead of letting the users instantiate
<code>Style</code> directly, you expose a <code>NewStyle</code> constructor function that constructs the struct
instance for them:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kn">package</span><span class="w"> </span><span class="nx">src</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// same as before</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// NewStyle option constructor instantiates a Style instance</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">NewStyle</span><span class="p">(</span><span class="nx">fg</span><span class="p">,</span><span class="w"> </span><span class="nx">bg</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="o">*</span><span class="nx">Style</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">Style</span><span class="p">{</span><span class="nx">fg</span><span class="p">,</span><span class="w"> </span><span class="nx">bg</span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>It&rsquo;ll be used as follows:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kn">package</span><span class="w"> </span><span class="nx">main</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kn">import</span><span class="w"> </span><span class="s">&#34;app/src&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// The users will now use NewStyle to instantiate Style</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">c</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">src</span><span class="p">.</span><span class="nf">NewStyle</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="s">&#34;\033[31m&#34;</span><span class="p">,</span><span class="w"> </span><span class="c1">// Maroon</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="s">&#34;\033[43m&#34;</span><span class="p">,</span><span class="w"> </span><span class="c1">// Yellow</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nf">Display</span><span class="p">(</span><span class="nx">c</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;Hello, World!&#34;</span><span class="p">)</span><span class="w">
</span></span></span></code></pre></div><p>If a new field is added to <code>Style</code>, update <code>NewStyle</code> to have a sensible default value for
it or initialize the struct with named parameters to set the optional fields to their
respective zero values. This avoids breaking users&rsquo; code as long as the constructor
function&rsquo;s signature doesn&rsquo;t change.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kn">package</span><span class="w"> </span><span class="nx">src</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">Style</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">Fg</span><span class="w"> </span><span class="kt">string</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">Bg</span><span class="w"> </span><span class="kt">string</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">Und</span><span class="w"> </span><span class="kt">bool</span><span class="w"> </span><span class="c1">// Underline or not</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Function signature unchanged though new option field added</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Set sensible default in constructor function</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">NewStyle</span><span class="p">(</span><span class="nx">fg</span><span class="p">,</span><span class="w"> </span><span class="nx">bg</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="o">*</span><span class="nx">Style</span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">Style</span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">Fg</span><span class="p">:</span><span class="w"> </span><span class="nx">fg</span><span class="p">,</span><span class="w"> </span><span class="nx">Bg</span><span class="p">:</span><span class="w"> </span><span class="nx">bg</span><span class="p">,</span><span class="w"> </span><span class="c1">// Und will be implicitly set to false</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>In <code>NewStyle</code>, we implicitly set the value of <code>Und</code> to <code>false</code> but you can be explicit there
depending on your needs. The struct fields can be updated in the same manner as before:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kn">package</span><span class="w"> </span><span class="nx">main</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">c</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">src</span><span class="p">.</span><span class="nf">NewStyle</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="s">&#34;\033[31m&#34;</span><span class="p">,</span><span class="w"> </span><span class="c1">// Maroon</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="s">&#34;\033[43m&#34;</span><span class="p">,</span><span class="w"> </span><span class="c1">// Yellow</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">c</span><span class="p">.</span><span class="nx">Und</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="kc">true</span><span class="w"> </span><span class="c1">// Default is false, we&#39;re setting it to true</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">src</span><span class="p">.</span><span class="nf">Display</span><span class="p">(</span><span class="nx">c</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;Hello, World!&#34;</span><span class="p">)</span><span class="w">
</span></span></span></code></pre></div><p>This should cover most use cases. However, if you don&rsquo;t want to export the underlying option
struct, or your struct has tons of optional fields requiring extensibility, you&rsquo;ll need an
extra layer of indirection to avoid the need to accept a zillion config parameters in your
option constructor.</p>
<h2 id="functional-option-constructor">Functional option constructor</h2>
<p>As mentioned at the tail of the last section, this approach works better when your struct
contains many optional fields and you need your users to be able to configure them if they
want. Go doesn&rsquo;t allow setting non-zero default values for struct fields. So an extra level
of indirection is necessary to let the users configure them. This approach also allows us to
make the option struct private so that there&rsquo;s no ambiguity around API usage.</p>
<p>Let&rsquo;s say <code>style</code> now has two optional fields <code>und</code> and <code>zigzag</code> that allows users to
decorate the message string with underlines or zigzagged lines:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kn">package</span><span class="w"> </span><span class="nx">src</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">style</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">fg</span><span class="w"> </span><span class="kt">string</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">bg</span><span class="w"> </span><span class="kt">string</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">und</span><span class="w"> </span><span class="kt">bool</span><span class="w"> </span><span class="c1">// Optional field</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">zigzag</span><span class="w"> </span><span class="kt">bool</span><span class="w"> </span><span class="c1">// Optional field</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Now, we&rsquo;ll define a new type called <code>styleoption</code> like this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// package src</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">styleoption</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="o">*</span><span class="nx">style</span><span class="p">)</span><span class="w">
</span></span></span></code></pre></div><p>The <code>styleoption</code> function accepts a pointer to the option struct and updates a particular
field with a user-provided value. The implementation of this type would look as such:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">s</span><span class="w"> </span><span class="o">*</span><span class="nx">style</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="nx">s</span><span class="p">.</span><span class="nx">fieldName</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">fieldValue</span><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Next, we&rsquo;ll need to define a higher order config function for each optional field in the
struct where the function will accept the field value and return another function with the
<code>styleoption</code> signature. The <code>WithUnd</code> and <code>WithZigzag</code> wrapper functions will be a part of
the public API that the users will use to configure <code>style</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// We only define config functions for the optional fields</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">WithUnd</span><span class="p">(</span><span class="nx">und</span><span class="w"> </span><span class="kt">bool</span><span class="p">)</span><span class="w"> </span><span class="nx">styleoption</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="nx">s</span><span class="w"> </span><span class="o">*</span><span class="nx">style</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">s</span><span class="p">.</span><span class="nx">und</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">und</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">WithZigzag</span><span class="p">(</span><span class="nx">zigzag</span><span class="w"> </span><span class="kt">bool</span><span class="p">)</span><span class="w"> </span><span class="nx">styleoption</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="nx">s</span><span class="w"> </span><span class="o">*</span><span class="nx">style</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">s</span><span class="p">.</span><span class="nx">zigzag</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">zigzag</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Finally, our option constructor function needs to be updated to accept variadic options.
Observe how we&rsquo;re looping through the <code>options</code> slice and applying the field config
functions to the struct pointer:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">NewStyle</span><span class="p">(</span><span class="nx">fg</span><span class="p">,</span><span class="w"> </span><span class="nx">bg</span><span class="w"> </span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="nx">options</span><span class="w"> </span><span class="o">...</span><span class="nx">styleoption</span><span class="p">)</span><span class="w"> </span><span class="o">*</span><span class="nx">style</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">s</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">style</span><span class="p">{</span><span class="nx">fg</span><span class="p">:</span><span class="w"> </span><span class="nx">fg</span><span class="p">,</span><span class="w"> </span><span class="nx">bg</span><span class="p">:</span><span class="w"> </span><span class="nx">bg</span><span class="p">}</span><span class="w"> </span><span class="c1">// und and zigzag are set to false</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Apply all the styleoption functions returned from</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// field config functions.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">for</span><span class="w"> </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">opt</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">range</span><span class="w"> </span><span class="nx">options</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nf">opt</span><span class="p">(</span><span class="nx">s</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">s</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>The users will use the code like this to instantiate <code>style</code> and update the optional fields:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">c</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">src</span><span class="p">.</span><span class="nf">NewStyle</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="s">&#34;\033[31m&#34;</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="s">&#34;\033[43m&#34;</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">src</span><span class="p">.</span><span class="nf">WithUnd</span><span class="p">(</span><span class="kc">true</span><span class="p">),</span><span class="w"> </span><span class="c1">// Default is false, but we&#39;re setting it to true</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">src</span><span class="p">.</span><span class="nf">WithZigzag</span><span class="p">(</span><span class="kc">true</span><span class="p">),</span><span class="w"> </span><span class="c1">// Default is false</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">)</span><span class="w">
</span></span></span></code></pre></div><p>The required fields <code>fg</code> and <code>bg</code> must be passed while constructing the option struct. The
optional fields can be configured with the field config functions like <code>WithUnd</code> and
<code>WithZigzag</code>.</p>
<p>The complete snippet looks as follows:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kn">package</span><span class="w"> </span><span class="nx">src</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// We can keep the option struct private</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">style</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">fg</span><span class="w"> </span><span class="kt">string</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">bg</span><span class="w"> </span><span class="kt">string</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">und</span><span class="w"> </span><span class="kt">bool</span><span class="w"> </span><span class="c1">// Optional field</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">zigzag</span><span class="w"> </span><span class="kt">bool</span><span class="w"> </span><span class="c1">// Optional field</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// This can be private too since the users won&#39;t need it directly</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">styleoption</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="o">*</span><span class="nx">style</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// We only define public config functions for the optional fields</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">WithUnd</span><span class="p">(</span><span class="nx">und</span><span class="w"> </span><span class="kt">bool</span><span class="p">)</span><span class="w"> </span><span class="nx">styleoption</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="nx">s</span><span class="w"> </span><span class="o">*</span><span class="nx">style</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">s</span><span class="p">.</span><span class="nx">und</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">und</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">WithZigzag</span><span class="p">(</span><span class="nx">zigzag</span><span class="w"> </span><span class="kt">bool</span><span class="p">)</span><span class="w"> </span><span class="nx">styleoption</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="nx">s</span><span class="w"> </span><span class="o">*</span><span class="nx">style</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">s</span><span class="p">.</span><span class="nx">zigzag</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">zigzag</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Options are variadic but the required fiels must be passed</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">NewStyle</span><span class="p">(</span><span class="nx">fg</span><span class="p">,</span><span class="w"> </span><span class="nx">bg</span><span class="w"> </span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="nx">options</span><span class="w"> </span><span class="o">...</span><span class="nx">styleoption</span><span class="p">)</span><span class="w"> </span><span class="o">*</span><span class="nx">style</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// You can also initialize the optional values explicitly</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">s</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">style</span><span class="p">{</span><span class="nx">fg</span><span class="p">:</span><span class="w"> </span><span class="nx">fg</span><span class="p">,</span><span class="w"> </span><span class="nx">bg</span><span class="p">:</span><span class="w"> </span><span class="nx">bg</span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">for</span><span class="w"> </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">opt</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">range</span><span class="w"> </span><span class="nx">options</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nf">opt</span><span class="p">(</span><span class="nx">s</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">return</span><span class="w"> </span><span class="nx">s</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>I first came across this pattern in <a href="https://commandcenter.blogspot.com/2014/01/self-referential-functions-and-design.html" rel="noopener noreferrer" target="_blank">Rob Pike&rsquo;s post on self-referential functions</a>.</p>
<h2 id="verdict">Verdict</h2>
<p>While the functional constructor pattern is the most intriguing one among the three, I
almost never reach for it unless I need my users to be able to configure large option
structs with many optional fields. It&rsquo;s rare and the extra indirection makes the code
inscrutable. Also, it renders the IDE suggestions useless.</p>
<p>In most cases, you can get away with exporting the option struct <code>Stuff</code> and a companion
function <code>NewStuff</code> to instantiate it. For another canonical example, see <code>bufio.Read</code> and
<code>bufio.NewReader</code> in the standard library.</p>
<h2 id="further-reading">Further reading</h2>
<ul>
<li><a href="https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis" rel="noopener noreferrer" target="_blank">Functional options for friendly APIs - Dave Cheney</a></li>
<li><a href="https://twitter.com/MattJamesBoyle/status/1698605808517288428" rel="noopener noreferrer" target="_blank">Functional options pattern in Go - Matt Boyle</a></li>
</ul>
<!-- references -->
<!-- prettier-ignore-start -->
<!-- prettier-ignore-end -->
]]></content:encoded>
    </item>
    <item>
      <title>Interface guards in Go</title>
      <link>https://rednafi.com/go/interface-guards/</link>
      <pubDate>Fri, 18 Aug 2023 00:00:00 +0000</pubDate>
      <guid>https://rednafi.com/go/interface-guards/</guid>
      <description>Use compile-time interface guards to verify type conformity in Go without runtime overhead. Learn the var _ Interface = (*Type)(nil) pattern.</description>
      <category>Go</category>
      <category>TIL</category>
      <category>API</category>
      <content:encoded><![CDATA[<p>I love Go&rsquo;s implicit interfaces. While convenient, they can also introduce subtle bugs
unless you&rsquo;re careful. Types expected to conform to certain interfaces can fluidly add or
remove methods. The compiler will only complain if an identifier anticipates an interface,
but is passed a type that doesn&rsquo;t implement that interface. This can be problematic if you
need to export types that are required to implement specific interfaces as part of their API
contract.</p>
<p>However, there&rsquo;s a way you can statically check interface conformity at compile time with
zero runtime overhead. Turns out, this was always buried in <a href="https://go.dev/doc/effective_go#interfaces" rel="noopener noreferrer" target="_blank">Effective Go</a>. Observe:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kn">import</span><span class="w"> </span><span class="s">&#34;io&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Interface guard</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">var</span><span class="w"> </span><span class="nx">_</span><span class="w"> </span><span class="nx">io</span><span class="p">.</span><span class="nx">ReadWriter</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="p">(</span><span class="o">*</span><span class="nx">T</span><span class="p">)(</span><span class="kc">nil</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">T</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="cp">//...</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="nf">Read</span><span class="p">(</span><span class="nx">p</span><span class="w"> </span><span class="p">[]</span><span class="kt">byte</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nx">n</span><span class="w"> </span><span class="kt">int</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// ...</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="o">*</span><span class="nx">T</span><span class="p">)</span><span class="w"> </span><span class="nf">Write</span><span class="p">(</span><span class="nx">p</span><span class="w"> </span><span class="p">[]</span><span class="kt">byte</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nx">n</span><span class="w"> </span><span class="kt">int</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// ...</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>We&rsquo;re checking if struct <code>T</code> implements the <code>io.ReadWriter</code> interface. It needs to have both
<code>Read</code> and <code>Write</code> methods defined. The type conformity is explicitly checked via
<code>var _ io.ReadWriter = (*T)(nil)</code>. It verifies that a <code>nil</code> pointer to a value of type <code>T</code>
conforms to the <code>io.ReadWriter</code> interface. The code will fail to compile if the type ever
stops matching the interface.</p>
<p>This is only possible because <code>nil</code> values in Go can assume many <a href="https://go101.org/article/nil.html" rel="noopener noreferrer" target="_blank">different types as
explained in Go 101</a>. In this case, <code>var _ io.ReadWriter = T{}</code> will also work, but then
you&rsquo;ll have to fiddle with different zero values if the type isn&rsquo;t a struct. One important
thing to point out is that we&rsquo;re using <code>_</code> because we don&rsquo;t want to accidentally refer to
this <code>nil</code> pointer anywhere in our code. Also, trying to access any method on it will cause
runtime panic.</p>
<p>Here&rsquo;s another example borrowed from <a href="https://github.com/uber-go/guide/blob/master/style.md#verify-interface-compliance" rel="noopener noreferrer" target="_blank">Uber&rsquo;s style guide</a>:</p>
<p>No check:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">Handler</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="cp">//...</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">h</span><span class="w"> </span><span class="o">*</span><span class="nx">Handler</span><span class="p">)</span><span class="w"> </span><span class="nf">ServeHTTP</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="cp">//...</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Check:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span><span class="w"> </span><span class="nx">Handler</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="c1">// ...</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Interface guard</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">var</span><span class="w"> </span><span class="nx">_</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">Handler</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="p">(</span><span class="o">*</span><span class="nx">Handler</span><span class="p">)(</span><span class="kc">nil</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">h</span><span class="w"> </span><span class="o">*</span><span class="nx">Handler</span><span class="p">)</span><span class="w"> </span><span class="nf">ServeHTTP</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="cp">//...</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Neat, but don&rsquo;t abuse this. <a href="https://go.dev/doc/effective_go#interfaces:~:text=The%20appearance%20of,a%20rare%20event" rel="noopener noreferrer" target="_blank">Effective Go warns</a>:</p>
<blockquote>
  <p>Don&rsquo;t do this for every type that satisfies an interface, though. By convention, such
declarations are only used when there are no static conversions already present in the
code, which is a rare event.</p>

</blockquote><h2 id="further-reading">Further reading</h2>
<ul>
<li><a href="https://caddyserver.com/docs/extending-caddy#interface-guards" rel="noopener noreferrer" target="_blank">Interface guards - Caddy docs</a></li>
<li><a href="https://twitter.com/MattJamesBoyle/status/1692428212058403251?s=20" rel="noopener noreferrer" target="_blank">Tweet by Matt Boyle</a></li>
</ul>
<!-- references -->
<!-- prettier-ignore-start -->
<!-- interfaces in effective go -->
<!-- check for interface compliance - uber style guide -->
<!-- effective go warns about not abusing interface guards -->
<!-- prettier-ignore-end -->
]]></content:encoded>
    </item>
    <item>
      <title>Bulk request Google search indexing with API</title>
      <link>https://rednafi.com/javascript/bulk-request-google-search-index/</link>
      <pubDate>Fri, 26 May 2023 00:00:00 +0000</pubDate>
      <guid>https://rednafi.com/javascript/bulk-request-google-search-index/</guid>
      <description>Learn how to programmatically request Google search indexing for multiple URLs using the Indexing API and NodeJS for faster reindexing.</description>
      <category>JavaScript</category>
      <category>API</category>
      <category>Web</category>
      <content:encoded><![CDATA[<p>Recently, I purchased a domain for this blog and migrated the content from
<a href="https://rednafi.github.io" rel="noopener noreferrer" target="_blank">rednafi.github.io</a> to <a href="/">rednafi.com</a>. This turned out to be a much bigger hassle than I
originally thought it&rsquo;d be, mostly because, despite setting redirection for almost all the
URLs from the previous domain to the new one and submitting the new <a href="/sitemap.xml">sitemap.xml</a> to the
Search Console, Google kept indexing the older domain. To make things worse, the search
engine selected the previous domain as canonical, and no amount of manual requests were
changing the status in the last 30 days. Strangely, I didn&rsquo;t encounter this issue with Bing,
as it reindexed the new site within a week after I submitted the sitemap file via their
webmaster panel.</p>
<p>While researching this, one potential solution suggested that along with submitting the
sitemap via <a href="https://search.google.com/search-console/about" rel="noopener noreferrer" target="_blank">Google Search Console</a>, I&rsquo;d have to make individual indexing requests for each
URL to encourage faster indexing. The problem is, I&rsquo;ve got quite a bit of content on this
site, and it&rsquo;ll take forever for me to click through all the links and request indexing that
way. Naturally, I looked for a way to do this programmatically. Luckily, I found out that
there&rsquo;s an <a href="https://developers.google.com/search/apis/indexing-api/v3/quickstart" rel="noopener noreferrer" target="_blank">indexing API</a> that allows you to make bulk indexing requests programmatically.
This has one big advantage - Google <a href="https://developers.google.com/search/apis/indexing-api/v3/quickstart#sitemaps" rel="noopener noreferrer" target="_blank">responds to API requests faster</a> than indexing requests
with sitemap submission.</p>
<p>All you&rsquo;ve to do is:</p>
<ul>
<li>
<p>List out the URLs that need to be indexed.</p>
</li>
<li>
<p>Fulfill the <a href="https://developers.google.com/search/apis/indexing-api/v3/quickstart" rel="noopener noreferrer" target="_blank">prerequisites</a> and download the private key JSON file required to make
requests to the API. From the docs:</p>
<blockquote>
  <p><em>Every call to the Indexing API must be authenticated with an OAuth token that you get
in exchange for your private key. Each token is good for a span of time. Google
provides API client libraries to get OAuth tokens for a number of languages.</em></p>

</blockquote><p>The private key file will look like this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;service_account&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;project_id&#34;</span><span class="p">:</span> <span class="s2">&#34;...&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;private_key_id&#34;</span><span class="p">:</span> <span class="s2">&#34;...&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;private_key&#34;</span><span class="p">:</span> <span class="s2">&#34;...&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;client_email&#34;</span><span class="p">:</span> <span class="s2">&#34;...&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;client_id&#34;</span><span class="p">:</span> <span class="s2">&#34;...&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;auth_uri&#34;</span><span class="p">:</span> <span class="s2">&#34;...&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;token_uri&#34;</span><span class="p">:</span> <span class="s2">&#34;...&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;auth_provider_x509_cert_url&#34;</span><span class="p">:</span> <span class="s2">&#34;...&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;client_x509_cert_url&#34;</span><span class="p">:</span> <span class="s2">&#34;...&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;universe_domain&#34;</span><span class="p">:</span> <span class="s2">&#34;...&#34;</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span></code></pre></div></li>
<li>
<p>Use an API client to make the requests.</p>
</li>
</ul>
<p>In my case, this site&rsquo;s <a href="/sitemap.xml">sitemap.xml</a> lists out all the URLs as follows:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-xml" data-lang="xml"><span class="line"><span class="cl"><span class="nt">&lt;urlset</span> <span class="na">xmlns=</span><span class="s">&#34;http://www.sitemaps.org/schemas/sitemap/0.9&#34;</span>
</span></span><span class="line"><span class="cl"><span class="na">xmlns:xhtml=</span><span class="s">&#34;http://www.w3.org/1999/xhtml&#34;</span><span class="nt">&gt;</span>
</span></span><span class="line"><span class="cl"><span class="nt">&lt;url&gt;</span>
</span></span><span class="line"><span class="cl">    <span class="nt">&lt;loc&gt;</span>https://rednafi.com/tags/github/<span class="nt">&lt;/loc&gt;</span>
</span></span><span class="line"><span class="cl">    <span class="nt">&lt;lastmod&gt;</span>2023-05-21T00:00:00+00:00<span class="nt">&lt;/lastmod&gt;</span>
</span></span><span class="line"><span class="cl"><span class="nt">&lt;/url&gt;</span>
</span></span><span class="line"><span class="cl"><span class="nt">&lt;url&gt;</span>
</span></span><span class="line"><span class="cl">    <span class="nt">&lt;loc&gt;</span>https://rednafi.com/tags/javascript/<span class="nt">&lt;/loc&gt;</span>
</span></span><span class="line"><span class="cl">    <span class="nt">&lt;lastmod&gt;</span>2023-05-21T00:00:00+00:00<span class="nt">&lt;/lastmod&gt;</span>
</span></span><span class="line"><span class="cl"><span class="nt">&lt;/url&gt;</span>
</span></span><span class="line"><span class="cl">...
</span></span></code></pre></div><p>Here&rsquo;s a NodeJS script that collects the URLs from <code>sitemap.xml</code> and makes requests to the
indexing API:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-js" data-lang="js"><span class="line"><span class="cl"><span class="c1">// ES6 import
</span></span></span><span class="line"><span class="cl"><span class="kr">import</span> <span class="p">{</span> <span class="nx">google</span> <span class="p">}</span> <span class="nx">from</span> <span class="s2">&#34;googleapis&#34;</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="kr">import</span> <span class="p">{</span> <span class="nx">parseString</span> <span class="p">}</span> <span class="nx">from</span> <span class="s2">&#34;xml2js&#34;</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="kr">import</span> <span class="nx">fetch</span> <span class="nx">from</span> <span class="s2">&#34;node-fetch&#34;</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">import</span> <span class="nx">pkey</span> <span class="nx">from</span> <span class="s2">&#34;./google-api-pkey.json&#34;</span> <span class="nx">assert</span> <span class="p">{</span> <span class="nx">type</span><span class="o">:</span> <span class="s2">&#34;json&#34;</span> <span class="p">};</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Parse the sitemap.xml file and extract the URLs.
</span></span></span><span class="line"><span class="cl"><span class="kr">async</span> <span class="kd">function</span> <span class="nx">getUrls</span><span class="p">(</span><span class="nx">url</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="k">try</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="kr">const</span> <span class="nx">response</span> <span class="o">=</span> <span class="kr">await</span> <span class="nx">fetch</span><span class="p">(</span><span class="nx">url</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">    <span class="kr">const</span> <span class="nx">xml</span> <span class="o">=</span> <span class="kr">await</span> <span class="nx">response</span><span class="p">.</span><span class="nx">text</span><span class="p">();</span>
</span></span><span class="line"><span class="cl">    <span class="kd">let</span> <span class="nx">urls</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">    <span class="nx">parseString</span><span class="p">(</span><span class="nx">xml</span><span class="p">,</span> <span class="p">(</span><span class="nx">err</span><span class="p">,</span> <span class="nx">result</span><span class="p">)</span> <span class="p">=&gt;</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="k">if</span> <span class="p">(</span><span class="nx">err</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">        <span class="nx">console</span><span class="p">.</span><span class="nx">error</span><span class="p">(</span><span class="s2">&#34;Error parsing XML:&#34;</span><span class="p">,</span> <span class="nx">err</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">      <span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">      <span class="nx">urls</span> <span class="o">=</span> <span class="nx">result</span><span class="p">.</span><span class="nx">urlset</span><span class="p">.</span><span class="nx">url</span><span class="p">.</span><span class="nx">map</span><span class="p">((</span><span class="nx">url</span><span class="p">)</span> <span class="p">=&gt;</span> <span class="nx">url</span><span class="p">.</span><span class="nx">loc</span><span class="p">[</span><span class="mi">0</span><span class="p">]);</span>
</span></span><span class="line"><span class="cl">    <span class="p">});</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="nx">urls</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span> <span class="k">catch</span> <span class="p">(</span><span class="nx">error</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="nx">console</span><span class="p">.</span><span class="nx">error</span><span class="p">(</span><span class="s2">&#34;Error fetching sitemap:&#34;</span><span class="p">,</span> <span class="nx">error</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Initialize auth client
</span></span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">jwtClient</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">google</span><span class="p">.</span><span class="nx">auth</span><span class="p">.</span><span class="nx">JWT</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">  <span class="nx">pkey</span><span class="p">.</span><span class="nx">client_email</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="kc">null</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nx">pkey</span><span class="p">.</span><span class="nx">private_key</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="p">[</span><span class="s2">&#34;https://www.googleapis.com/auth/indexing&#34;</span><span class="p">],</span>
</span></span><span class="line"><span class="cl">  <span class="kc">null</span>
</span></span><span class="line"><span class="cl"><span class="p">);</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Perfrom auth and make multiple API calls
</span></span></span><span class="line"><span class="cl"><span class="nx">jwtClient</span><span class="p">.</span><span class="nx">authorize</span><span class="p">(</span><span class="kr">async</span> <span class="kd">function</span> <span class="p">(</span><span class="nx">err</span><span class="p">,</span> <span class="nx">tokens</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="k">if</span> <span class="p">(</span><span class="nx">err</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">err</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="kr">const</span> <span class="nx">options</span> <span class="o">=</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="nx">url</span><span class="o">:</span> <span class="s2">&#34;https://indexing.googleapis.com/v3/urlNotifications:publish&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="nx">method</span><span class="o">:</span> <span class="s2">&#34;POST&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="nx">headers</span><span class="o">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="s2">&#34;Content-Type&#34;</span><span class="o">:</span> <span class="s2">&#34;application/json&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nx">Authorization</span><span class="o">:</span> <span class="sb">`Bearer </span><span class="si">${</span><span class="nx">tokens</span><span class="p">.</span><span class="nx">access_token</span><span class="si">}</span><span class="sb">`</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="nx">json</span><span class="o">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nx">url</span><span class="o">:</span> <span class="s2">&#34;&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nx">type</span><span class="o">:</span> <span class="s2">&#34;URL_UPDATED&#34;</span><span class="p">,</span> <span class="c1">// Means we want to request indexing
</span></span></span><span class="line"><span class="cl">    <span class="p">},</span>
</span></span><span class="line"><span class="cl">  <span class="p">};</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">try</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="kr">const</span> <span class="nx">urls</span> <span class="o">=</span> <span class="kr">await</span> <span class="nx">getUrls</span><span class="p">(</span><span class="s2">&#34;https://www.rednafi.com/sitemap.xml&#34;</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1">// There&#39;s a bulk endpoint but looping through the list
</span></span></span><span class="line"><span class="cl">    <span class="c1">// and making multiple requests is just as easy
</span></span></span><span class="line"><span class="cl">    <span class="k">for</span> <span class="p">(</span><span class="kr">const</span> <span class="nx">url</span> <span class="k">of</span> <span class="nx">urls</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nx">options</span><span class="p">.</span><span class="nx">json</span><span class="p">.</span><span class="nx">url</span> <span class="o">=</span> <span class="nx">url</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">      <span class="kr">const</span> <span class="nx">response</span> <span class="o">=</span> <span class="kr">await</span> <span class="nx">fetch</span><span class="p">(</span><span class="nx">options</span><span class="p">.</span><span class="nx">url</span><span class="p">,</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">        <span class="nx">method</span><span class="o">:</span> <span class="nx">options</span><span class="p">.</span><span class="nx">method</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="nx">headers</span><span class="o">:</span> <span class="nx">options</span><span class="p">.</span><span class="nx">headers</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="nx">body</span><span class="o">:</span> <span class="nx">JSON</span><span class="p">.</span><span class="nx">stringify</span><span class="p">(</span><span class="nx">options</span><span class="p">.</span><span class="nx">json</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">      <span class="p">});</span>
</span></span><span class="line"><span class="cl">      <span class="kr">const</span> <span class="nx">body</span> <span class="o">=</span> <span class="kr">await</span> <span class="nx">response</span><span class="p">.</span><span class="nx">json</span><span class="p">();</span>
</span></span><span class="line"><span class="cl">      <span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">body</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span> <span class="k">catch</span> <span class="p">(</span><span class="nx">error</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="nx">console</span><span class="p">.</span><span class="nx">error</span><span class="p">(</span><span class="s2">&#34;Error:&#34;</span><span class="p">,</span> <span class="nx">error</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span>
</span></span><span class="line"><span class="cl"><span class="p">});</span>
</span></span></code></pre></div><p>Before executing the script, npm install <code>googleapis</code> and <code>xml2js</code>. Now running the script
will give you an output similar to this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-txt" data-lang="txt"><span class="line"><span class="cl">{
</span></span><span class="line"><span class="cl">  urlNotificationMetadata: {
</span></span><span class="line"><span class="cl">    url: &#39;https://rednafi.com/categories/&#39;,
</span></span><span class="line"><span class="cl">    latestUpdate: {
</span></span><span class="line"><span class="cl">      url: &#39;https://rednafi.com/categories/&#39;,
</span></span><span class="line"><span class="cl">      type: &#39;URL_UPDATED&#39;,
</span></span><span class="line"><span class="cl">      notifyTime: &#39;2023-05-27T01:02:35.537421311Z&#39;
</span></span><span class="line"><span class="cl">    }
</span></span><span class="line"><span class="cl">  }
</span></span><span class="line"><span class="cl">}
</span></span><span class="line"><span class="cl">{
</span></span><span class="line"><span class="cl">  urlNotificationMetadata: {
</span></span><span class="line"><span class="cl">    url: &#39;https://rednafi.com/search/&#39;,
</span></span><span class="line"><span class="cl">    latestUpdate: {
</span></span><span class="line"><span class="cl">      url: &#39;https://rednafi.com/search/&#39;,
</span></span><span class="line"><span class="cl">      type: &#39;URL_UPDATED&#39;,
</span></span><span class="line"><span class="cl">      notifyTime: &#39;2023-05-27T01:02:35.789809492Z&#39;
</span></span><span class="line"><span class="cl">    }
</span></span><span class="line"><span class="cl">  }
</span></span><span class="line"><span class="cl">},
</span></span><span class="line"><span class="cl">...
</span></span></code></pre></div><p>Here, the <code>getUrls</code> function is defined to fetch the sitemap content from a specified URL,
parse the XML content and extract the URLs. It uses the fetch function to retrieve the file,
then uses <code>xml2js</code> to parse the XML and extract the URLs from the result.</p>
<p>The script then initializes an authentication client using the imported private key and
specifies the required API scope. The <code>authorize</code> function is called to authenticate the
client and obtain access tokens. Inside the authorization callback, the script prepares the
necessary <code>options</code> for making API requests to the Google indexing API. It then calls the
<code>getUrls</code> function to fetch the URLs from the <code>sitemap.xml</code> file. For each URL, it updates
the <code>options</code> with the URL and makes a POST request to the Indexing API to request indexing.
The response from the API is then logged into the console.</p>
<p>One thing to keep in mind is that by default, the daily request quota per project is 200.
But you can <a href="https://developers.google.com/search/apis/indexing-api/v3/quota-pricing" rel="noopener noreferrer" target="_blank">request more quota</a> if you need it.</p>
<!-- references -->
<!-- prettier-ignore-start -->
<!-- prettier-ignore-end -->
]]></content:encoded>
    </item>
    <item>
      <title>Verifying webhook origin via payload hash signing</title>
      <link>https://rednafi.com/python/verify-webhook-origin/</link>
      <pubDate>Sun, 18 Sep 2022 00:00:00 +0000</pubDate>
      <guid>https://rednafi.com/python/verify-webhook-origin/</guid>
      <description>Secure webhooks by verifying payload authenticity using HMAC hash signatures with shared secrets, preventing man-in-the-middle attacks.</description>
      <category>Python</category>
      <category>API</category>
      <category>Security</category>
      <content:encoded><![CDATA[<p>While working with GitHub webhooks, I discovered a common <a href="https://docs.github.com/en/developers/webhooks-and-events/webhooks/securing-your-webhooks" rel="noopener noreferrer" target="_blank">webhook security pattern</a> a
receiver can adopt to verify that the incoming webhooks are indeed arriving from GitHub; not
from some miscreant trying to carry out a man-in-the-middle attack. After some amount of
digging, I found that it&rsquo;s quite a common practice that many other webhook services employ
as well. Also, check out how <a href="https://docs.sentry.io/product/integrations/integration-platform/webhooks/#sentry-hook-resource" rel="noopener noreferrer" target="_blank">Sentry handles webhook verification</a>.</p>
<p>Moreover, GitHub&rsquo;s documentation demonstrates the pattern in Ruby. So I thought it&rsquo;d be a
good idea to translate that into Python in a more platform-agnostic manner. The core idea of
the pattern goes as follows:</p>
<ul>
<li>
<p>The webhook sender will hash the JSONified webhook payload with a well-known hashing
algorithm like MD5, SHA-1, or SHA-256. A secret token known to the receiver will be used
to sign the calculated hash of the payload.</p>
</li>
<li>
<p>The sender will include the payload hash digest prefixed by the name of the hash algorithm
to the header of the webhook request. For example, the GitHub webhook&rsquo;s request header has
a key like the following. Notice how the digest is prefixed with the name of the algorithm
<code>sha256</code>:</p>
</li>
</ul>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-txt" data-lang="txt"><span class="line"><span class="cl">X-Hub-Signature-256=\
</span></span><span class="line"><span class="cl">    sha-256=e863e1f6370b60981bbbcbc2da3313321e65eaaac36f9d1262af415965df9320
</span></span></code></pre></div><ul>
<li>The webhook receiver is then expected to hash the received JSON payload with the same
algorithm found in the prefix of the header and sign with the common secret token known to
both the sender and the receiver. Afterward, the receiver compares the calculated hash
with the incoming hash in the request header. If the two digests match, that ensures that
the payload hasn&rsquo;t been tampered with. Otherwise, the receiver should reject the incoming
payload. This provides a second layer of protection over the usual authentication that the
receiver might have in place.</li>
</ul>
<p>To demonstrate the workflow, here&rsquo;s an example of how the webhook sender might be
implemented:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># sender.py</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">annotations</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">hashlib</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">json</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">http</span> <span class="kn">import</span> <span class="n">HTTPStatus</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">httpx</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">starlette.applications</span> <span class="kn">import</span> <span class="n">Starlette</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">starlette.requests</span> <span class="kn">import</span> <span class="n">Request</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">starlette.responses</span> <span class="kn">import</span> <span class="n">JSONResponse</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">starlette.routing</span> <span class="kn">import</span> <span class="n">Route</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">send_webhook</span><span class="p">(</span><span class="n">request</span><span class="p">:</span> <span class="n">Request</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">JSONResponse</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="c1"># Get the request body as bytes.</span>
</span></span><span class="line"><span class="cl">    <span class="n">raw_body</span> <span class="o">=</span> <span class="k">await</span> <span class="n">request</span><span class="o">.</span><span class="n">body</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1"># Disallow empty body.</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="n">raw_body</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">JSONResponse</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="p">{</span><span class="s2">&#34;error&#34;</span><span class="p">:</span> <span class="s2">&#34;Empty body&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">            <span class="n">status_code</span><span class="o">=</span><span class="n">HTTPStatus</span><span class="o">.</span><span class="n">BAD_REQUEST</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1"># Check that the request body is a valid JSON payload.</span>
</span></span><span class="line"><span class="cl">    <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">body</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">raw_body</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">except</span> <span class="n">json</span><span class="o">.</span><span class="n">JSONDecodeError</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">JSONResponse</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="p">{</span><span class="s2">&#34;error&#34;</span><span class="p">:</span> <span class="s2">&#34;Invalid JSON body&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">            <span class="n">status_code</span><span class="o">=</span><span class="n">HTTPStatus</span><span class="o">.</span><span class="n">BAD_REQUEST</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1"># Hash the body and sign it with a secret.</span>
</span></span><span class="line"><span class="cl">    <span class="n">x_payload_signature</span> <span class="o">=</span> <span class="n">hashlib</span><span class="o">.</span><span class="n">sha256</span><span class="p">(</span><span class="n">raw_body</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">x_payload_signature</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="sa">b</span><span class="s2">&#34;some-secret&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">x_payload_signature</span> <span class="o">=</span> <span class="n">x_payload_signature</span><span class="o">.</span><span class="n">hexdigest</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1"># Send the webhook.</span>
</span></span><span class="line"><span class="cl">    <span class="k">async</span> <span class="k">with</span> <span class="n">httpx</span><span class="o">.</span><span class="n">AsyncClient</span><span class="p">()</span> <span class="k">as</span> <span class="n">client</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">response</span> <span class="o">=</span> <span class="k">await</span> <span class="n">client</span><span class="o">.</span><span class="n">post</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="s2">&#34;http://localhost:6000/receive-webhook&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="n">json</span><span class="o">=</span><span class="n">body</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="n">headers</span><span class="o">=</span><span class="p">{</span>
</span></span><span class="line"><span class="cl">                <span class="s2">&#34;X-Payload-Signature-256&#34;</span><span class="p">:</span> <span class="sa">f</span><span class="s2">&#34;sha256=</span><span class="si">{</span><span class="n">x_payload_signature</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">                <span class="s2">&#34;Content-Type&#34;</span><span class="p">:</span> <span class="s2">&#34;application/json&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="p">},</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">response</span><span class="o">.</span><span class="n">status_code</span> <span class="o">!=</span> <span class="n">HTTPStatus</span><span class="o">.</span><span class="n">ACCEPTED</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="n">JSONResponse</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">                <span class="p">{</span><span class="s2">&#34;error&#34;</span><span class="p">:</span> <span class="s2">&#34;Could not sent webhook&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">                <span class="n">status_code</span><span class="o">=</span><span class="n">HTTPStatus</span><span class="o">.</span><span class="n">BAD_REQUEST</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">JSONResponse</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="p">{</span>
</span></span><span class="line"><span class="cl">            <span class="s2">&#34;message&#34;</span><span class="p">:</span> <span class="s2">&#34;Webhook sent&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="s2">&#34;response_payload&#34;</span><span class="p">:</span> <span class="n">response</span><span class="o">.</span><span class="n">json</span><span class="p">(),</span>
</span></span><span class="line"><span class="cl">        <span class="p">},</span>
</span></span><span class="line"><span class="cl">        <span class="n">status_code</span><span class="o">=</span><span class="n">HTTPStatus</span><span class="o">.</span><span class="n">OK</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">app</span> <span class="o">=</span> <span class="n">Starlette</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="n">debug</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">routes</span><span class="o">=</span><span class="p">[</span>
</span></span><span class="line"><span class="cl">        <span class="n">Route</span><span class="p">(</span><span class="s2">&#34;/send-webhook&#34;</span><span class="p">,</span> <span class="n">send_webhook</span><span class="p">,</span> <span class="n">methods</span><span class="o">=</span><span class="p">[</span><span class="s2">&#34;POST&#34;</span><span class="p">]),</span>
</span></span><span class="line"><span class="cl">    <span class="p">],</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span>
</span></span></code></pre></div><p>Here, I&rsquo;ve implemented a simple POST API that:</p>
<ul>
<li>Accepts a payload from the user.</li>
<li>Hashes the payload with <code>sha-256</code> algorithm and signs it with a <code>some-secret</code> token.</li>
<li>Adds the digest to the request header to the receiver. The header has a key called
<code>X-Payload-Signature-256</code> that contains the prefixed payload digest:</li>
</ul>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-txt" data-lang="txt"><span class="line"><span class="cl">X-Payload-Signature-256: \
</span></span><span class="line"><span class="cl">    sha-256=e863e1f6370b60981bbbcbc2da3313321e65eaaac36f9d1262af415965df9320
</span></span></code></pre></div><ul>
<li>After hashing, the sender sends the payload to the receiver via HTTP POST request. Here,
I&rsquo;m using HTTPx to send the request to the receiver. For demonstration purposes, I&rsquo;m
assuming that the receiver endpoint is <code>localhost:6000/receive-webhook</code>.</li>
</ul>
<p>The receiver will:</p>
<ul>
<li>Accept the incoming request from the sender.</li>
<li>Parse the header and store the value of <code>X-Payload-Signature-256</code>.</li>
<li>Calculate the hash value of the incoming payload in the same manner as the sender.</li>
<li>Sign the payload with the common secret that&rsquo;s known to both parties.</li>
<li>Compare the newly calculated signed-hash with the digest value of the
<code>X-Payload-Signature-256</code> attribute.</li>
<li>Only accept and process the payload if the incoming and the computed hashes match.</li>
</ul>
<p>Here&rsquo;s how you can implement the receiver:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># receiver.py</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">annotations</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">hashlib</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">json</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">secrets</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">http</span> <span class="kn">import</span> <span class="n">HTTPStatus</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">starlette.applications</span> <span class="kn">import</span> <span class="n">Starlette</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">starlette.requests</span> <span class="kn">import</span> <span class="n">Request</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">starlette.responses</span> <span class="kn">import</span> <span class="n">JSONResponse</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">starlette.routing</span> <span class="kn">import</span> <span class="n">Route</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">receive_webhook</span><span class="p">(</span><span class="n">request</span><span class="p">:</span> <span class="n">Request</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">JSONResponse</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="c1"># Get the payload signature from the request headers.</span>
</span></span><span class="line"><span class="cl">    <span class="n">x_payload_signature_256</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">headers</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;X-Payload-Signature-256&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1"># Disallow empty signature.</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">x_payload_signature_256</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">JSONResponse</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="p">{</span><span class="s2">&#34;error&#34;</span><span class="p">:</span> <span class="s2">&#34;Missing X-Payload-Signature header&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">            <span class="n">status_code</span><span class="o">=</span><span class="n">HTTPStatus</span><span class="o">.</span><span class="n">BAD_REQUEST</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1"># Check that the signature is valid.</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="n">x_payload_signature_256</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&#34;sha256=&#34;</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">JSONResponse</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="p">{</span><span class="s2">&#34;error&#34;</span><span class="p">:</span> <span class="s2">&#34;Invalid X-Payload-Signature header&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">            <span class="n">status_code</span><span class="o">=</span><span class="n">HTTPStatus</span><span class="o">.</span><span class="n">BAD_REQUEST</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1"># Get x_payload_signature_256 without the &#34;sha256=&#34; prefix.</span>
</span></span><span class="line"><span class="cl">    <span class="n">x_payload_signature</span> <span class="o">=</span> <span class="n">x_payload_signature_256</span><span class="o">.</span><span class="n">removeprefix</span><span class="p">(</span><span class="s2">&#34;sha256=&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="n">raw_body</span> <span class="o">=</span> <span class="k">await</span> <span class="n">request</span><span class="o">.</span><span class="n">body</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1"># Disallow empty body.</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="n">raw_body</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">JSONResponse</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="p">{</span><span class="s2">&#34;error&#34;</span><span class="p">:</span> <span class="s2">&#34;Empty body&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">            <span class="n">status_code</span><span class="o">=</span><span class="n">HTTPStatus</span><span class="o">.</span><span class="n">BAD_REQUEST</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1"># Check that the request body is a valid JSON payload.</span>
</span></span><span class="line"><span class="cl">    <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">body</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">raw_body</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">except</span> <span class="n">json</span><span class="o">.</span><span class="n">JSONDecodeError</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">JSONResponse</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="p">{</span><span class="s2">&#34;error&#34;</span><span class="p">:</span> <span class="s2">&#34;Invalid JSON body&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">            <span class="n">status_code</span><span class="o">=</span><span class="n">HTTPStatus</span><span class="o">.</span><span class="n">BAD_REQUEST</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1"># Hash the incoming body with the secret.</span>
</span></span><span class="line"><span class="cl">    <span class="n">expected_signature</span> <span class="o">=</span> <span class="n">hashlib</span><span class="o">.</span><span class="n">sha256</span><span class="p">(</span><span class="n">raw_body</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">expected_signature</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="sa">b</span><span class="s2">&#34;some-secret&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">expected_signature</span> <span class="o">=</span> <span class="n">expected_signature</span><span class="o">.</span><span class="n">hexdigest</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1"># Compare the expected signature with the incoming signature.</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="n">secrets</span><span class="o">.</span><span class="n">compare_digest</span><span class="p">(</span><span class="n">x_payload_signature</span><span class="p">,</span> <span class="n">expected_signature</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="ow">is</span> <span class="kc">False</span>
</span></span><span class="line"><span class="cl">    <span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">JSONResponse</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="p">{</span><span class="s2">&#34;error&#34;</span><span class="p">:</span> <span class="s2">&#34;Invalid signature&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">            <span class="n">status_code</span><span class="o">=</span><span class="n">HTTPStatus</span><span class="o">.</span><span class="n">UNAUTHORIZED</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">JSONResponse</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="p">{</span><span class="s2">&#34;message&#34;</span><span class="p">:</span> <span class="s2">&#34;Webhook accepted&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">        <span class="n">status_code</span><span class="o">=</span><span class="n">HTTPStatus</span><span class="o">.</span><span class="n">ACCEPTED</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">app</span> <span class="o">=</span> <span class="n">Starlette</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="n">debug</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">routes</span><span class="o">=</span><span class="p">[</span>
</span></span><span class="line"><span class="cl">        <span class="n">Route</span><span class="p">(</span><span class="s2">&#34;/receive-webhook&#34;</span><span class="p">,</span> <span class="n">receive_webhook</span><span class="p">,</span> <span class="n">methods</span><span class="o">=</span><span class="p">[</span><span class="s2">&#34;POST&#34;</span><span class="p">]),</span>
</span></span><span class="line"><span class="cl">    <span class="p">],</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span>
</span></span></code></pre></div><blockquote>
  <p>In the receiver, instead of using plain string comparison to compare the payload hashes,
leverage <code>secrets.compare_digest</code> to mitigate the possibility of <a href="https://en.wikipedia.org/wiki/Timing_attack" rel="noopener noreferrer" target="_blank">timing attacks</a>.</p>

</blockquote><p>To test the end-to-end workflow, you&rsquo;ll need to pip install <a href="https://www.python-httpx.org/" rel="noopener noreferrer" target="_blank">httpx</a> and <a href="https://www.uvicorn.org/" rel="noopener noreferrer" target="_blank">uvicorn</a>. Then on
your console, you can run the two scripts in the background with the following command:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl">nohup uvicorn sender:app --reload --port <span class="m">5000</span> &gt; /dev/null <span class="se">\
</span></span></span><span class="line"><span class="cl">    <span class="p">&amp;</span> nohup uvicorn receiver:app --reload --port <span class="m">6000</span> &gt; /dev/null <span class="p">&amp;</span>
</span></span></code></pre></div><p>This will spin up two uvicorn servers in the background where the sender and the receiver
can be accessed via ports 5000 and 6000 respectively. Now if you make a request to the
sender service, you&rsquo;ll see that the sender sends the webhook payload to the receiver service
and returns an HTTP 200 code only if the receiver has been able to verify the signed-hash of
the payload:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl">curl -si POST http://localhost:5000/send-webhook -d <span class="s1">&#39;{&#34;hello&#34;: &#34;world&#34;}&#39;</span>
</span></span></code></pre></div><p>This will return:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-txt" data-lang="txt"><span class="line"><span class="cl">HTTP/1.1 200 OK
</span></span><span class="line"><span class="cl">date: Tue, 20 Sep 2022 06:31:07 GMT
</span></span><span class="line"><span class="cl">server: uvicorn
</span></span><span class="line"><span class="cl">content-length: 76
</span></span><span class="line"><span class="cl">content-type: application/json
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">{&#34;message&#34;:&#34;Webhook sent&#34;,&#34;response_payload&#34;:{&#34;message&#34;:&#34;Webhook accepted&#34;}}
</span></span></code></pre></div><p>The reciver will return a HTTP 400 error code if it can&rsquo;t verify the payload. Once you&rsquo;re
done, kill the running servers with <code>sudo pkill uvicorn</code> command.</p>
<!-- references -->
<!-- prettier-ignore-start -->
<!-- prettier-ignore-end -->
]]></content:encoded>
    </item>
    <item>
      <title>Return JSON error payload instead of HTML text in DRF</title>
      <link>https://rednafi.com/python/return-json-error-payload-in-drf/</link>
      <pubDate>Wed, 13 Apr 2022 00:00:00 +0000</pubDate>
      <guid>https://rednafi.com/python/return-json-error-payload-in-drf/</guid>
      <description>Fix Django REST Framework to return JSON error responses for 403, 404, 500 errors using middleware instead of default HTML pages.</description>
      <category>Python</category>
      <category>API</category>
      <category>Django</category>
      <content:encoded><![CDATA[<p>At my workplace, we have a large Django monolith that powers the main website and works as
the primary REST API server at the same time. We use Django Rest Framework (DRF) to build
and serve the API endpoints. This means, whenever there&rsquo;s an error, based on the incoming
request header - we&rsquo;ve to return different formats of error responses to the website and API
users.</p>
<p>The default DRF configuration returns a JSON response when the system experiences an HTTP
400 (bad request) error. However, the server returns an HTML error page to the API users
whenever HTTP 403 (forbidden), HTTP 404 (not found), or HTTP 500 (internal server error)
occurs. This is suboptimal; JSON APIs should never return HTML text whenever something goes
wrong. On the other hand, the website needs those error text to appear accordingly.</p>
<p>This happens because 403, 404, and 500 are handled by Django&rsquo;s default handlers for those
errors and not by DRF&rsquo;s exception handlers. As the DRF doc suggests on <a href="https://www.django-rest-framework.org/api-guide/exceptions/#generic-error-views" rel="noopener noreferrer" target="_blank">generic error
views</a>, overriding the error handlers is one way of solving it. But this will only work if
the application is an API-only backend or if you haven&rsquo;t already overridden the error
handlers for custom error pages.</p>
<p>In our case, we already had to override the default error handlers to display custom error
pages on the website. These custom pages would bleed into the API endpoints occasionally
when errors occur. So, I thought, if I could handle this in the middleware layer, that&rsquo;d be
cleaner than most of the solutions that I&rsquo;d seen at that point.</p>
<h2 id="solution">Solution</h2>
<p>To fix the dilemma, I wrote a middleware called <code>JSONErrorMiddleware</code> that returns the
expected response based on the content type in the request header. If the header has
<code>Content-Type: html/text</code> and it experiences an error, the server returns an appropriate
HTML page. On the contrary, if the incoming request header has
<code>Content-Type: application/json</code> and the server sees an error, it responds with a JSON error
payload instead. Here&rsquo;s how the middleware looks:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># &lt;app&gt;/middleware.py</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">http</span> <span class="kn">import</span> <span class="n">HTTPStatus</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">JSONErrorMiddleware</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;&#34;&#34;Without this middleware, APIs would respond with
</span></span></span><span class="line"><span class="cl"><span class="s2">    html/text whenever there&#39;s an error.&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">get_response</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">get_response</span> <span class="o">=</span> <span class="n">get_response</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">status_code_description</span> <span class="o">=</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">            <span class="n">v</span><span class="o">.</span><span class="n">value</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">description</span> <span class="k">for</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">HTTPStatus</span>
</span></span><span class="line"><span class="cl">        <span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="fm">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">request</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="n">response</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_response</span><span class="p">(</span><span class="n">request</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="c1"># If the content_type isn&#39;t &#39;application/json&#39;, do nothing.</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="ow">not</span> <span class="n">request</span><span class="o">.</span><span class="n">content_type</span> <span class="o">==</span> <span class="s2">&#34;application/json&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="n">response</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="c1"># If there&#39;s no error, let Django and DRF&#39;s default views deal</span>
</span></span><span class="line"><span class="cl">        <span class="c1"># with it.</span>
</span></span><span class="line"><span class="cl">        <span class="n">status_code</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">status_code</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="ow">not</span> <span class="n">HTTPStatus</span><span class="o">.</span><span class="n">BAD_REQUEST</span>
</span></span><span class="line"><span class="cl">            <span class="o">&lt;</span> <span class="n">status_code</span>
</span></span><span class="line"><span class="cl">            <span class="o">&lt;=</span> <span class="n">HTTPStatus</span><span class="o">.</span><span class="n">INTERNAL_SERVER_ERROR</span>
</span></span><span class="line"><span class="cl">        <span class="p">):</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="n">response</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="c1"># Return a JSON error response if any of 403, 404, or 500 occurs.</span>
</span></span><span class="line"><span class="cl">        <span class="n">r</span> <span class="o">=</span> <span class="n">JsonResponse</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="p">{</span>
</span></span><span class="line"><span class="cl">                <span class="s2">&#34;error&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">                    <span class="s2">&#34;status_code&#34;</span><span class="p">:</span> <span class="n">status_code</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">                    <span class="s2">&#34;message&#34;</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">status_code_description</span><span class="p">[</span><span class="n">status_code</span><span class="p">],</span>
</span></span><span class="line"><span class="cl">                    <span class="s2">&#34;detail&#34;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&#34;url&#34;</span><span class="p">:</span> <span class="n">request</span><span class="o">.</span><span class="n">get_full_path</span><span class="p">()},</span>
</span></span><span class="line"><span class="cl">                <span class="p">}</span>
</span></span><span class="line"><span class="cl">            <span class="p">},</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">r</span><span class="o">.</span><span class="n">status_code</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">status_code</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">r</span>
</span></span></code></pre></div><p>You&rsquo;ll have to add this middleware to the list of middlewares in the <code>settings.py</code> file:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">MIDDLEWARE</span> <span class="o">=</span> <span class="p">[</span><span class="o">...</span><span class="p">,</span> <span class="s2">&#34;&lt;app&gt;.middleware.JSONErrorMiddleware&#34;</span><span class="p">]</span>
</span></span></code></pre></div><p>And voila, now the API and non-API errors will be handled differently as expected!</p>
<h2 id="test">Test</h2>
<p>Here&rsquo;s how you can unit test the behavior of the middleware:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">json</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">unittest.mock</span> <span class="kn">import</span> <span class="n">MagicMock</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">django.http</span> <span class="kn">import</span> <span class="n">JsonResponse</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">django.test</span> <span class="kn">import</span> <span class="n">RequestFactory</span><span class="p">,</span> <span class="n">TestCase</span><span class="p">,</span> <span class="n">override_settings</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">main.middleware</span> <span class="kn">import</span> <span class="n">JSONErrorMiddleware</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd">@override_settings</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="n">MIDDLEWARE_CLASSES</span><span class="o">=</span><span class="p">(</span><span class="s2">&#34;main.middleware.JSONErrorMiddleware&#34;</span><span class="p">,),</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">TestJSONErrorMiddleware</span><span class="p">(</span><span class="n">TestCase</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">setUp</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">setUp</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">factory</span> <span class="o">=</span> <span class="n">RequestFactory</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="k">def</span> <span class="nf">get_response</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">            <span class="n">response</span> <span class="o">=</span> <span class="n">MagicMock</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">            <span class="n">response</span><span class="o">.</span><span class="n">status_code</span> <span class="o">=</span> <span class="n">HTTPStatus</span><span class="o">.</span><span class="n">FORBIDDEN</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="n">response</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">middleware</span> <span class="o">=</span> <span class="n">JSONErrorMiddleware</span><span class="p">(</span><span class="n">get_response</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">test_json_error_middleware</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="c1"># Arrange</span>
</span></span><span class="line"><span class="cl">        <span class="n">corrupted_url</span> <span class="o">=</span> <span class="s2">&#34;/account&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="c1"># Act</span>
</span></span><span class="line"><span class="cl">        <span class="n">request</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">factory</span><span class="o">.</span><span class="n">get</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="n">path</span><span class="o">=</span><span class="n">corrupted_url</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">request</span><span class="o">.</span><span class="n">content_type</span> <span class="o">=</span> <span class="s2">&#34;application/json&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="n">response</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">middleware</span><span class="o">.</span><span class="fm">__call__</span><span class="p">(</span><span class="n">request</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="c1"># Assert</span>
</span></span><span class="line"><span class="cl">        <span class="c1"># Assert 404 no longer returns html/text.</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">assertTrue</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">response</span><span class="p">,</span> <span class="n">JsonResponse</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="c1"># Assert json format.</span>
</span></span><span class="line"><span class="cl">        <span class="n">json_data</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">content</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">expected_json_data</span> <span class="o">=</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">            <span class="s2">&#34;error&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">                <span class="s2">&#34;status_code&#34;</span><span class="p">:</span> <span class="n">HTTPStatus</span><span class="o">.</span><span class="n">FORBIDDEN</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">                <span class="s2">&#34;message&#34;</span><span class="p">:</span> <span class="n">HTTPStatus</span><span class="o">.</span><span class="n">FORBIDDEN</span><span class="o">.</span><span class="n">description</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">                <span class="s2">&#34;detail&#34;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&#34;url&#34;</span><span class="p">:</span> <span class="s2">&#34;/account&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">            <span class="p">}</span>
</span></span><span class="line"><span class="cl">        <span class="p">}</span>
</span></span><span class="line"><span class="cl">        <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">json_data</span><span class="p">[</span><span class="s2">&#34;error&#34;</span><span class="p">]</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
</span></span><span class="line"><span class="cl">            <span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">v</span><span class="p">,</span> <span class="n">expected_json_data</span><span class="p">[</span><span class="s2">&#34;error&#34;</span><span class="p">][</span><span class="n">k</span><span class="p">])</span>
</span></span></code></pre></div><h2 id="breadcrumbs">Breadcrumbs</h2>
<p>This workflow has been tested on Django 3.2, 4.0, and DRF 3.13.</p>
<h2 id="further-reading">Further reading</h2>
<ul>
<li><a href="https://github.com/encode/django-rest-framework/issues/3362" rel="noopener noreferrer" target="_blank">HTML sometimes returned when Accept: application/json is provided #3362</a></li>
<li><a href="https://github.com/encode/django-rest-framework/pull/5904" rel="noopener noreferrer" target="_blank">Added generic 500 and 400 JSON error handlers #5904</a></li>
</ul>
<!-- references -->
<!-- prettier-ignore-start -->
<!-- prettier-ignore-end -->
]]></content:encoded>
    </item>
    <item>
      <title>Disallow large file download from URLs in Python</title>
      <link>https://rednafi.com/python/disallow-large-file-download/</link>
      <pubDate>Wed, 23 Mar 2022 00:00:00 +0000</pubDate>
      <guid>https://rednafi.com/python/disallow-large-file-download/</guid>
      <description>Prevent excessive file downloads in Python by streaming with HTTPX and limiting file size with chunk-based validation and memory-safe processing.</description>
      <category>Python</category>
      <category>API</category>
      <category>Security</category>
      <content:encoded><![CDATA[<p>I was working on a DRF POST API endpoint where the consumer is expected to add a URL
containing a PDF file and the system would then download the file and save it to an S3
bucket. While this sounds quite straightforward, there&rsquo;s one big issue. Before I started
working on it, the core logic looked like this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># src.py</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">annoatations</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">urllib.request</span> <span class="kn">import</span> <span class="n">urlopen</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">tempfile</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">shutil</span> <span class="kn">import</span> <span class="n">copyfileobj</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">save_to_s3</span><span class="p">(</span><span class="n">src_url</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">dest_url</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">with</span> <span class="n">tempfile</span><span class="o">.</span><span class="n">NamedTemporaryFile</span><span class="p">()</span> <span class="k">as</span> <span class="n">file</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">with</span> <span class="n">urlopen</span><span class="p">(</span><span class="n">src_url</span><span class="p">)</span> <span class="k">as</span> <span class="n">response</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="c1"># This stdlib function saves the content of the file</span>
</span></span><span class="line"><span class="cl">            <span class="c1"># in &#39;file&#39;.</span>
</span></span><span class="line"><span class="cl">            <span class="n">copyfileobj</span><span class="p">(</span><span class="n">response</span><span class="p">,</span> <span class="n">file</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="c1"># Logic to save file in s3.</span>
</span></span><span class="line"><span class="cl">        <span class="n">_save_to_s3</span><span class="p">(</span><span class="n">des_url</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">&#34;__main__&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">save_to_s3</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;https://citeseerx.ist.psu.edu/viewdoc/download?&#34;</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;doi=10.1.1.92.4846&amp;rep=rep1&amp;type=pdf&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;https://s3-url.com&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span>
</span></span></code></pre></div><p>In the above snippet, there&rsquo;s no guardrail against how large the target file can be. You
could bring the entire server down to its knees by posting a link to a ginormous file. The
server would be busy downloading the file and keep consuming resources.</p>
<p>I didn&rsquo;t want to use <code>urllib</code> at all for this purpose and went for <a href="https://www.python-httpx.org/" rel="noopener noreferrer" target="_blank">HTTPx</a>. It exposes a
neat API to perform streaming file download. Also, I didn&rsquo;t want to peek into the
<code>Content-Length</code> header to assess the file size since the file server can choose not to
include that header key. I was looking for something more dependable than that. Here&rsquo;s how I
solved it:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># src</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">annotations</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">httpx</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">tempfile</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">save_to_s3</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="n">src_url</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">dest_url</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">chunk_size</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">1024</span> <span class="o">*</span> <span class="mi">1024</span><span class="p">,</span>  <span class="c1"># 1 MB buffer.</span>
</span></span><span class="line"><span class="cl">    <span class="n">max_size</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">10</span> <span class="o">*</span> <span class="mi">1024</span> <span class="o">*</span> <span class="mi">1024</span><span class="p">,</span>  <span class="c1"># 10 MB</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="c1"># Keep track of the already downloaded byte length.</span>
</span></span><span class="line"><span class="cl">    <span class="n">downloaded_content_length</span> <span class="o">=</span> <span class="mi">0</span>  <span class="c1"># bytes</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">with</span> <span class="n">tempfile</span><span class="o">.</span><span class="n">NamedTemporaryFile</span><span class="p">()</span> <span class="k">as</span> <span class="n">file</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">with</span> <span class="n">httpx</span><span class="o">.</span><span class="n">stream</span><span class="p">(</span><span class="s2">&#34;GET&#34;</span><span class="p">,</span> <span class="n">src_url</span><span class="p">)</span> <span class="k">as</span> <span class="n">response</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">for</span> <span class="n">chunk</span> <span class="ow">in</span> <span class="n">response</span><span class="o">.</span><span class="n">iter_bytes</span><span class="p">(</span><span class="n">chunk_size</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">                <span class="n">downloaded_content_length</span> <span class="o">+=</span> <span class="nb">len</span><span class="p">(</span><span class="n">chunk</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">                <span class="k">if</span> <span class="n">downloaded_content_length</span> <span class="o">&gt;</span> <span class="n">max_size</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">                    <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">                        <span class="sa">f</span><span class="s2">&#34;File size too large. Make sure your linked &#34;</span>
</span></span><span class="line"><span class="cl">                        <span class="s2">&#34;file is not larger than 10 MB.&#34;</span>
</span></span><span class="line"><span class="cl">                    <span class="p">)</span>
</span></span><span class="line"><span class="cl">                <span class="n">file</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">chunk</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="c1"># logic to save file in s3.</span>
</span></span><span class="line"><span class="cl">        <span class="n">_save_to_s3</span><span class="p">(</span><span class="n">dest_url</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">&#34;__main__&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">save_to_s3</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;https://citeseerx.ist.psu.edu/viewdoc/download?&#34;</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;doi=10.1.1.92.4846&amp;rep=rep1&amp;type=pdf&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span>
</span></span></code></pre></div><p>The <code>chunk_size</code> parameter explicitly dictates the buffer size of the file being downloaded.
This means the entire file won&rsquo;t be loaded into memory while being downloaded. The
<code>max_size</code> parameter defines the maximum file size that&rsquo;ll be allowed. In this example,
we&rsquo;re keeping track of the size of the already downloaded bytes in the
<code>downloaded_content_length</code> variable and raising an error if the size exceeds 10MB. Sweet!</p>
<h2 id="further-reading">Further reading</h2>
<ul>
<li><a href="https://www.python-httpx.org/advanced/#monitoring-download-progress" rel="noopener noreferrer" target="_blank">Streaming download with HTTPx</a></li>
</ul>
<!-- references -->
<!-- prettier-ignore-start -->
<!-- prettier-ignore-end -->
]]></content:encoded>
    </item>
    <item>
      <title>Declarative payloads with TypedDict in Python</title>
      <link>https://rednafi.com/python/declarative-payloads-with-typedict/</link>
      <pubDate>Fri, 11 Mar 2022 00:00:00 +0000</pubDate>
      <guid>https://rednafi.com/python/declarative-payloads-with-typedict/</guid>
      <description>Use Python TypedDict to declaratively define API payload structures. Get type safety for nested dictionaries and improve code maintainability.</description>
      <category>Python</category>
      <category>Typing</category>
      <category>API</category>
      <content:encoded><![CDATA[<p>While working with microservices in Python, a common pattern that I see is - the usage of
dynamically filled dictionaries as payloads of REST APIs or message queues. To understand
what I mean by this, consider the following example:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># src.py</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">annotations</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">json</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">Any</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">redis</span>  <span class="c1"># Do a pip install.</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">get_payload</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">]:</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;&#34;&#34;Get the &#39;zoo&#39; payload containing animal names and attributes.&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="n">payload</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&#34;name&#34;</span><span class="p">:</span> <span class="s2">&#34;awesome_zoo&#34;</span><span class="p">,</span> <span class="s2">&#34;animals&#34;</span><span class="p">:</span> <span class="p">[]}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="n">names</span> <span class="o">=</span> <span class="p">(</span><span class="s2">&#34;wolf&#34;</span><span class="p">,</span> <span class="s2">&#34;snake&#34;</span><span class="p">,</span> <span class="s2">&#34;ostrich&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">attributes</span> <span class="o">=</span> <span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="p">{</span><span class="s2">&#34;family&#34;</span><span class="p">:</span> <span class="s2">&#34;Canidae&#34;</span><span class="p">,</span> <span class="s2">&#34;genus&#34;</span><span class="p">:</span> <span class="s2">&#34;Canis&#34;</span><span class="p">,</span> <span class="s2">&#34;is_mammal&#34;</span><span class="p">:</span> <span class="kc">True</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">        <span class="p">{</span><span class="s2">&#34;family&#34;</span><span class="p">:</span> <span class="s2">&#34;Viperidae&#34;</span><span class="p">,</span> <span class="s2">&#34;genus&#34;</span><span class="p">:</span> <span class="s2">&#34;Boas&#34;</span><span class="p">,</span> <span class="s2">&#34;is_mammal&#34;</span><span class="p">:</span> <span class="kc">False</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">attr</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="n">names</span><span class="p">,</span> <span class="n">attributes</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="n">payload</span><span class="p">[</span><span class="s2">&#34;animals&#34;</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span>  <span class="c1"># type: ignore</span>
</span></span><span class="line"><span class="cl">            <span class="p">{</span><span class="s2">&#34;name&#34;</span><span class="p">:</span> <span class="n">name</span><span class="p">,</span> <span class="s2">&#34;attribute&#34;</span><span class="p">:</span> <span class="n">attr</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">payload</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">save_to_cache</span><span class="p">(</span><span class="n">payload</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">])</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="c1"># You&#39;ll need to spin up a Redis db before instantiating</span>
</span></span><span class="line"><span class="cl">    <span class="c1"># a connection here.</span>
</span></span><span class="line"><span class="cl">    <span class="n">r</span> <span class="o">=</span> <span class="n">redis</span><span class="o">.</span><span class="n">Redis</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="nb">print</span><span class="p">(</span><span class="s2">&#34;Saving to cache...&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">r</span><span class="o">.</span><span class="n">set</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;zoo:</span><span class="si">{</span><span class="n">payload</span><span class="p">[</span><span class="s1">&#39;name&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">,</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">payload</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">&#34;__main__&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">payload</span> <span class="o">=</span> <span class="n">get_payload</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="n">save_to_cache</span><span class="p">(</span><span class="n">payload</span><span class="p">)</span>
</span></span></code></pre></div><p>Here, the <code>get_payload</code> function constructs a payload that gets stored in a Redis DB in the
<code>save_to_cache</code> function. The <code>get_payload</code> function returns a dict that denotes a contrived
payload containing the data of an imaginary zoo. To execute the above snippet, you&rsquo;ll need
to spin up a Redis database first. You can use <a href="https://www.docker.com/" rel="noopener noreferrer" target="_blank">Docker</a> to do so. Install and configure
Docker on your system and run:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl">docker run -d -p 6379:6379 redis:alpine
</span></span></code></pre></div><p>If you run the above snippet after instantiating the Redis server, it&rsquo;ll run without raising
any error. You can inspect the content saved in Redis with the following command (assuming
you&rsquo;ve got <code>redis-cli</code> and <code>jq</code> installed in your system):</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-txt" data-lang="txt"><span class="line"><span class="cl">echo &#34;get zoo:awesome_zoo&#34; | redis-cli | jq
</span></span></code></pre></div><p>This will return the following payload to your console:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;name&#34;</span><span class="p">:</span> <span class="s2">&#34;awesome_zoo&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;animals&#34;</span><span class="p">:</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;name&#34;</span><span class="p">:</span> <span class="s2">&#34;wolf&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;attribute&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">        <span class="nt">&#34;family&#34;</span><span class="p">:</span> <span class="s2">&#34;Canidae&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="nt">&#34;genus&#34;</span><span class="p">:</span> <span class="s2">&#34;Canis&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="nt">&#34;is_mammal&#34;</span><span class="p">:</span> <span class="kc">true</span>
</span></span><span class="line"><span class="cl">      <span class="p">}</span>
</span></span><span class="line"><span class="cl">    <span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;name&#34;</span><span class="p">:</span> <span class="s2">&#34;snake&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;attribute&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">        <span class="nt">&#34;family&#34;</span><span class="p">:</span> <span class="s2">&#34;Viperidae&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="nt">&#34;genus&#34;</span><span class="p">:</span> <span class="s2">&#34;Boas&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="nt">&#34;is_mammal&#34;</span><span class="p">:</span> <span class="kc">false</span>
</span></span><span class="line"><span class="cl">      <span class="p">}</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span></code></pre></div><p>Although this workflow is functional in runtime, there&rsquo;s a big gotcha here! It&rsquo;s really
difficult to picture the shape of the <code>payload</code> from the output of the <code>get_payload</code>
function; as it dynamically builds the dictionary. First, it declares a dictionary with two
fields - <code>name</code> and <code>animals</code>. Here, <code>name</code> is a string value that denotes the name of the
zoo. The other field <code>animals</code> is a list containing the names and attributes of the animals
in the zoo. Later on, the for-loop fills up the dictionary with nested data structures. This
charade of operations makes it difficult to reify the final shape of the resulting <code>payload</code>
in your mind.</p>
<p>In this case, you&rsquo;ll have to inspect the content of the Redis cache to fully understand the
shape of the data. Writing code in the above manner is effortless but it makes it really
hard for the next person working on the codebase to understand how the payload looks without
tapping into the data storage. There&rsquo;s a better way to declaratively communicate the shape
of the payload that doesn&rsquo;t involve writing unmaintainably large docstrings. Here&rsquo;s how you
can leverage <code>TypedDict</code> and <code>Annotated</code> to achieve the goals:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># src.py</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">annotations</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">json</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># In &lt; Python 3.8, import &#39;TypedDict&#39; from &#39;typing_extensions&#39;.</span>
</span></span><span class="line"><span class="cl"><span class="c1"># In &lt; Python 3.9, import &#39;Annotated&#39; from &#39;typing_extensions&#39;.</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">Annotated</span><span class="p">,</span> <span class="n">Any</span><span class="p">,</span> <span class="n">TypedDict</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">redis</span>  <span class="c1"># Do a pip install.</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">Attribute</span><span class="p">(</span><span class="n">TypedDict</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">family</span><span class="p">:</span> <span class="nb">str</span>
</span></span><span class="line"><span class="cl">    <span class="n">genus</span><span class="p">:</span> <span class="nb">str</span>
</span></span><span class="line"><span class="cl">    <span class="n">is_mammal</span><span class="p">:</span> <span class="nb">bool</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">Animal</span><span class="p">(</span><span class="n">TypedDict</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">name</span><span class="p">:</span> <span class="nb">str</span>
</span></span><span class="line"><span class="cl">    <span class="n">attribute</span><span class="p">:</span> <span class="n">Attribute</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">Zoo</span><span class="p">(</span><span class="n">TypedDict</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">name</span><span class="p">:</span> <span class="nb">str</span>
</span></span><span class="line"><span class="cl">    <span class="n">animals</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="n">Animal</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">get_payload</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="n">Zoo</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;&#34;&#34;Get the &#39;zoo&#39; payload containing animal names and attributes.&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="n">payload</span><span class="p">:</span> <span class="n">Zoo</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&#34;name&#34;</span><span class="p">:</span> <span class="s2">&#34;awesome_zoo&#34;</span><span class="p">,</span> <span class="s2">&#34;animals&#34;</span><span class="p">:</span> <span class="p">[]}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="n">names</span> <span class="o">=</span> <span class="p">(</span><span class="s2">&#34;wolf&#34;</span><span class="p">,</span> <span class="s2">&#34;snake&#34;</span><span class="p">,</span> <span class="s2">&#34;ostrich&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">attributes</span><span class="p">:</span> <span class="nb">tuple</span><span class="p">[</span><span class="n">Attribute</span><span class="p">,</span> <span class="o">...</span><span class="p">]</span> <span class="o">=</span> <span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="p">{</span><span class="s2">&#34;family&#34;</span><span class="p">:</span> <span class="s2">&#34;Canidae&#34;</span><span class="p">,</span> <span class="s2">&#34;genus&#34;</span><span class="p">:</span> <span class="s2">&#34;Canis&#34;</span><span class="p">,</span> <span class="s2">&#34;is_mammal&#34;</span><span class="p">:</span> <span class="kc">True</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">        <span class="p">{</span><span class="s2">&#34;family&#34;</span><span class="p">:</span> <span class="s2">&#34;Viperidae&#34;</span><span class="p">,</span> <span class="s2">&#34;genus&#34;</span><span class="p">:</span> <span class="s2">&#34;Boas&#34;</span><span class="p">,</span> <span class="s2">&#34;is_mammal&#34;</span><span class="p">:</span> <span class="kc">False</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">attr</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="n">names</span><span class="p">,</span> <span class="n">attributes</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="n">payload</span><span class="p">[</span><span class="s2">&#34;animals&#34;</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">({</span><span class="s2">&#34;name&#34;</span><span class="p">:</span> <span class="n">name</span><span class="p">,</span> <span class="s2">&#34;attribute&#34;</span><span class="p">:</span> <span class="n">attr</span><span class="p">})</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">payload</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">save_to_cache</span><span class="p">(</span><span class="n">payload</span><span class="p">:</span> <span class="n">Annotated</span><span class="p">[</span><span class="n">Zoo</span><span class="p">,</span> <span class="nb">dict</span><span class="p">])</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="c1"># You&#39;ll need to spin up a Redis db before instantiating</span>
</span></span><span class="line"><span class="cl">    <span class="c1"># a connection here.</span>
</span></span><span class="line"><span class="cl">    <span class="n">r</span> <span class="o">=</span> <span class="n">redis</span><span class="o">.</span><span class="n">Redis</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="nb">print</span><span class="p">(</span><span class="s2">&#34;Saving to cache...&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">r</span><span class="o">.</span><span class="n">set</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;zoo:</span><span class="si">{</span><span class="n">payload</span><span class="p">[</span><span class="s1">&#39;name&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">,</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">payload</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">&#34;__main__&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">payload</span><span class="p">:</span> <span class="n">Zoo</span> <span class="o">=</span> <span class="n">get_payload</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="n">save_to_cache</span><span class="p">(</span><span class="n">payload</span><span class="p">)</span>
</span></span></code></pre></div><p>Notice, how I&rsquo;ve used <code>TypedDict</code> to declare the nested structure of the payload <code>Zoo</code>. In
runtime, instances of typed-dict classes behave the same way as normal dicts. Here, <code>Zoo</code>
contains two fields - <code>name</code> and <code>animals</code>. The <code>animals</code> field is annotated as
<code>list[Animal]</code> where <code>Animal</code> is another typed-dict. The <code>Animal</code> typed-dict houses another
typed-dict called <code>Attribute</code> that defines various properties of the animal.</p>
<p>Taking a look at the typed-dict <code>Zoo</code> and following along its nested structure, the final
shape of the payload becomes clearer without us having to look for example payloads. Also,
Mypy can check whether the payload conforms to the shape of the annotated type. I used
<code>Annotated[Zoo, dict]</code> in the input parameter of <code>save_to_cache</code> function to communicate
with the reader that an instance of the class <code>Zoo</code> is a dict that conforms to the contract
laid out in the type itself. The type <code>Annotated</code> can be used to add any arbitrary metadata
to a particular type.</p>
<p>In runtime, this snippet will exhibit the same behavior as the previous one. Mypy also
approves this.</p>
<h2 id="handling-missing-key-value-pairs">Handling missing key-value pairs</h2>
<p>By default, the type checker will structurally validate the shape of the dict annotated with
a <code>TypedDict</code> class and all the key-value pairs expected by the annotation must be present
in the dict. It&rsquo;s possible to lax this behavior by specifying <em>totality</em>. This can be
helpful to deal with missing fields without letting go of type safety. Consider this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">annotations</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">TypedDict</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">Attribute</span><span class="p">(</span><span class="n">TypedDict</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">family</span><span class="p">:</span> <span class="nb">str</span>
</span></span><span class="line"><span class="cl">    <span class="n">genus</span><span class="p">:</span> <span class="nb">str</span>
</span></span><span class="line"><span class="cl">    <span class="n">is_mammal</span><span class="p">:</span> <span class="nb">bool</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">animal_attribute</span><span class="p">:</span> <span class="n">Attribute</span> <span class="o">=</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;family&#34;</span><span class="p">:</span> <span class="s2">&#34;Hominidae&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;genus&#34;</span><span class="p">:</span> <span class="s2">&#34;Homo&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>  <span class="c1"># Mypy will complain about the missing &#39;is_mammal&#39; key.</span>
</span></span></code></pre></div><p>Mypy will complain about the missing key:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-txt" data-lang="txt"><span class="line"><span class="cl">src.py:12: error: Missing key &#34;is_mammal&#34; for TypedDict &#34;Attribute&#34;
</span></span><span class="line"><span class="cl">    animal_attribute: Attribute = {
</span></span><span class="line"><span class="cl">                                  ^
</span></span><span class="line"><span class="cl">Found 1 error in 1 file (checked 1 source file)
</span></span></code></pre></div><p>You can relax this behavior like this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="o">...</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">Attribute</span><span class="p">(</span><span class="n">TypedDict</span><span class="p">,</span> <span class="n">total</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">family</span><span class="p">:</span> <span class="nb">str</span>
</span></span><span class="line"><span class="cl">    <span class="n">genus</span><span class="p">:</span> <span class="nb">str</span>
</span></span><span class="line"><span class="cl">    <span class="n">is_mammal</span><span class="p">:</span> <span class="nb">bool</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="o">...</span>
</span></span></code></pre></div><p>Now Mypy will no longer complain about the missing field in the annotated dict. However,
this will still disallow arbitrary keys that isn&rsquo;t defined in the <code>TypedDict</code>. For example:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="o">...</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Mypy will complain as the key &#39;species&#39; doesn&#39;t exist in the TypedDict.</span>
</span></span><span class="line"><span class="cl"><span class="n">animal_attribute</span><span class="p">[</span><span class="s2">&#34;species&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="s2">&#34;Sapiens&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="o">...</span>
</span></span></code></pre></div><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-txt" data-lang="txt"><span class="line"><span class="cl">src.py:17: error: TypedDict &#34;Attribute&#34; has no key &#34;species&#34;
</span></span><span class="line"><span class="cl">    animal_attribute[&#34;species&#34;] = &#34;Sapiens&#34;
</span></span><span class="line"><span class="cl">                    ^
</span></span><span class="line"><span class="cl">Found 1 error in 1 file (checked 3 source files)
</span></span><span class="line"><span class="cl">make: *** [Makefile:134: mypy] Error 1
</span></span></code></pre></div><p>Sweet type safety without being too strict about missing fields!</p>
<h2 id="further-reading">Further reading</h2>
<ul>
<li><a href="https://peps.python.org/pep-0589/" rel="noopener noreferrer" target="_blank">PEP 589 – TypedDict: Type hints for dictionaries with a fixed set of keys</a></li>
</ul>
<!-- references -->
<!-- prettier-ignore-start -->
<!-- prettier-ignore-end -->
]]></content:encoded>
    </item>
    <item>
      <title>Uniform error response in Django Rest Framework</title>
      <link>https://rednafi.com/python/uniform-error-response-in-drf/</link>
      <pubDate>Thu, 20 Jan 2022 00:00:00 +0000</pubDate>
      <guid>https://rednafi.com/python/uniform-error-response-in-drf/</guid>
      <description>Standardize Django REST Framework error responses using custom exception handlers to match Microsoft&amp;#39;s REST API guidelines format.</description>
      <category>Python</category>
      <category>Django</category>
      <category>TIL</category>
      <category>API</category>
      <content:encoded><![CDATA[<p>Django Rest Framework exposes a neat hook to customize the response payload of your API when
errors occur. I was going through <a href="https://github.com/microsoft/api-guidelines" rel="noopener noreferrer" target="_blank">Microsoft&rsquo;s REST API guideline</a> and wanted to make the
error response of my APIs more uniform and somewhat similar to <a href="https://github.com/microsoft/api-guidelines/blob/vNext/Guidelines.md#examples" rel="noopener noreferrer" target="_blank">this example</a>.</p>
<p>I&rsquo;ll use a modified version of the <a href="https://www.django-rest-framework.org/#example" rel="noopener noreferrer" target="_blank">quickstart example</a> in the DRF docs to show how to
achieve that. Also, we&rsquo;ll need a POST API to demonstrate the changes better. Here&rsquo;s the same
example with the added POST API. Place this code in the project&rsquo;s <code>urls.py</code> file.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># urls.py</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">django.urls</span> <span class="kn">import</span> <span class="n">path</span><span class="p">,</span> <span class="n">include</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">django.contrib.auth.models</span> <span class="kn">import</span> <span class="n">User</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">rest_framework</span> <span class="kn">import</span> <span class="n">routers</span><span class="p">,</span> <span class="n">serializers</span><span class="p">,</span> <span class="n">viewsets</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Serializers define the API representation.</span>
</span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">UserSerializer</span><span class="p">(</span><span class="n">serializers</span><span class="o">.</span><span class="n">HyperlinkedModelSerializer</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">class</span> <span class="nc">Meta</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">model</span> <span class="o">=</span> <span class="n">User</span>
</span></span><span class="line"><span class="cl">        <span class="n">fields</span> <span class="o">=</span> <span class="p">[</span><span class="s2">&#34;url&#34;</span><span class="p">,</span> <span class="s2">&#34;username&#34;</span><span class="p">,</span> <span class="s2">&#34;email&#34;</span><span class="p">,</span> <span class="s2">&#34;is_staff&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">validate_username</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">username</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">username</span><span class="p">)</span> <span class="o">&lt;</span> <span class="mi">10</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">raise</span> <span class="n">serializers</span><span class="o">.</span><span class="n">ValidationError</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">                <span class="s2">&#34;Username must be at least 10 characters long.&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">username</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">validate_email</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">email</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="n">validate_email</span><span class="p">(</span><span class="n">email</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">except</span> <span class="n">ValidationError</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">raise</span> <span class="n">serializers</span><span class="o">.</span><span class="n">ValidationError</span><span class="p">(</span><span class="s2">&#34;Invalid email format.&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">email</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">create</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">validated_data</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">User</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">User</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">create</span><span class="p">(</span><span class="o">**</span><span class="n">validated_data</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># ViewSets define the view behavior.</span>
</span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">UserViewSet</span><span class="p">(</span><span class="n">viewsets</span><span class="o">.</span><span class="n">ModelViewSet</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">queryset</span> <span class="o">=</span> <span class="n">User</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">all</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="n">serializer_class</span> <span class="o">=</span> <span class="n">UserSerializer</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Routers provide an easy way of automatically determining the URL conf.</span>
</span></span><span class="line"><span class="cl"><span class="n">router</span> <span class="o">=</span> <span class="n">routers</span><span class="o">.</span><span class="n">DefaultRouter</span><span class="p">()</span>
</span></span><span class="line"><span class="cl"><span class="n">router</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="sa">r</span><span class="s2">&#34;users&#34;</span><span class="p">,</span> <span class="n">UserViewSet</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Wire up our API using automatic URL routing.</span>
</span></span><span class="line"><span class="cl"><span class="c1"># Additionally, we include login URLs for the browsable API.</span>
</span></span><span class="line"><span class="cl"><span class="n">urlpatterns</span> <span class="o">=</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">    <span class="n">path</span><span class="p">(</span><span class="s2">&#34;&#34;</span><span class="p">,</span> <span class="n">include</span><span class="p">(</span><span class="n">router</span><span class="o">.</span><span class="n">urls</span><span class="p">)),</span>
</span></span><span class="line"><span class="cl">    <span class="n">path</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;api-auth/&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="n">include</span><span class="p">(</span><span class="s2">&#34;rest_framework.urls&#34;</span><span class="p">,</span> <span class="n">namespace</span><span class="o">=</span><span class="s2">&#34;rest_framework&#34;</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">    <span class="p">),</span>
</span></span><span class="line"><span class="cl"><span class="p">]</span>
</span></span></code></pre></div><p>If you make a POST request to <code>/users</code> endpoint with the following payload where it&rsquo;ll
intentionally fail email and username validation:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;username&#34;</span><span class="p">:</span> <span class="s2">&#34;hello&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;email&#34;</span><span class="p">:</span> <span class="s2">&#34;email..&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;is_staff&#34;</span><span class="p">:</span> <span class="kc">false</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span></code></pre></div><p>you&rsquo;ll see the following response:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;username&#34;</span><span class="p">:[</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;Username must be at least 10 characters long.&#34;</span>
</span></span><span class="line"><span class="cl">  <span class="p">],</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;email&#34;</span><span class="p">:[</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;Enter a valid email address.&#34;</span>
</span></span><span class="line"><span class="cl">  <span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span></code></pre></div><p>While this is okay, there&rsquo;s one gotcha here. The error payload isn&rsquo;t consistent. Depending
on the type of error, the shape of the response payload will change. This can be a problem
if your system has custom error handling logic that expects a consistent response.</p>
<p>I wanted the error payload to have a predictable shape while carrying more information like</p>
<ul>
<li>HTTP error code, error message, etc. You can do it by wrapping the default
<code>rest_framework.views.exception_handler</code> function in a custom exception handler function.
Let&rsquo;s write the <code>api_exception_handler</code>:</li>
</ul>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># urls.py</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">rest_framework.views</span> <span class="kn">import</span> <span class="n">exception_handler</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">http</span> <span class="kn">import</span> <span class="n">HTTPStatus</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">Any</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">rest_framework.views</span> <span class="kn">import</span> <span class="n">Response</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="o">...</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">api_exception_handler</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="n">exc</span><span class="p">:</span> <span class="ne">Exception</span><span class="p">,</span> <span class="n">context</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Response</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;&#34;&#34;Custom API exception handler.&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1"># Call REST framework&#39;s default exception handler first,</span>
</span></span><span class="line"><span class="cl">    <span class="c1"># to get the standard error response.</span>
</span></span><span class="line"><span class="cl">    <span class="n">response</span> <span class="o">=</span> <span class="n">exception_handler</span><span class="p">(</span><span class="n">exc</span><span class="p">,</span> <span class="n">context</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">response</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="c1"># Using the description&#39;s of the HTTPStatus class as error message.</span>
</span></span><span class="line"><span class="cl">        <span class="n">http_code_to_message</span> <span class="o">=</span> <span class="p">{</span><span class="n">v</span><span class="o">.</span><span class="n">value</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">description</span> <span class="k">for</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">HTTPStatus</span><span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="n">error_payload</span> <span class="o">=</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">            <span class="s2">&#34;error&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">                <span class="s2">&#34;status_code&#34;</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">                <span class="s2">&#34;message&#34;</span><span class="p">:</span> <span class="s2">&#34;&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">                <span class="s2">&#34;details&#34;</span><span class="p">:</span> <span class="p">[],</span>
</span></span><span class="line"><span class="cl">            <span class="p">}</span>
</span></span><span class="line"><span class="cl">        <span class="p">}</span>
</span></span><span class="line"><span class="cl">        <span class="n">error</span> <span class="o">=</span> <span class="n">error_payload</span><span class="p">[</span><span class="s2">&#34;error&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">        <span class="n">status_code</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">status_code</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="n">error</span><span class="p">[</span><span class="s2">&#34;status_code&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="n">status_code</span>
</span></span><span class="line"><span class="cl">        <span class="n">error</span><span class="p">[</span><span class="s2">&#34;message&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="n">http_code_to_message</span><span class="p">[</span><span class="n">status_code</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">        <span class="n">error</span><span class="p">[</span><span class="s2">&#34;details&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">data</span>
</span></span><span class="line"><span class="cl">        <span class="n">response</span><span class="o">.</span><span class="n">data</span> <span class="o">=</span> <span class="n">error_payload</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">response</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="o">...</span>
</span></span></code></pre></div><p>Now, you&rsquo;ll have to register this custom exception handler in the <code>settings.py</code> file. Head
over to the <code>REST_FRAMEWORK</code> section and add the following key:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">REST_FRAMEWORK</span> <span class="o">=</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="o">...</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;EXCEPTION_HANDLER&#34;</span><span class="p">:</span> <span class="s2">&#34;&lt;project&gt;.urls.api_exception_handler&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span></code></pre></div><p>If you make a POST request to <code>/users</code> endpoint with an invalid payload as before, you&rsquo;ll
see this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;error&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="nt">&#34;status_code&#34;</span><span class="p">:</span><span class="mi">400</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="nt">&#34;message&#34;</span><span class="p">:</span><span class="s2">&#34;Bad request syntax or unsupported method&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="nt">&#34;details&#34;</span><span class="p">:{</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;username&#34;</span><span class="p">:[</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;Username must be at least 10 character long.&#34;</span>
</span></span><span class="line"><span class="cl">      <span class="p">],</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;email&#34;</span><span class="p">:[</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;Enter a valid email address.&#34;</span>
</span></span><span class="line"><span class="cl">      <span class="p">]</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span></code></pre></div><p>Much nicer!</p>
<h2 id="further-reading">Further reading</h2>
<ul>
<li><a href="https://www.django-rest-framework.org/api-guide/exceptions/#custom-exception-handling" rel="noopener noreferrer" target="_blank">Custom Exception Handling - DRF docs</a></li>
</ul>
<!-- references -->
<!-- prettier-ignore-start -->
<!-- prettier-ignore-end -->
]]></content:encoded>
    </item>
    <item>
      <title>Effortless API response caching with Python &amp; Redis</title>
      <link>https://rednafi.com/python/redis-cache/</link>
      <pubDate>Mon, 25 May 2020 00:00:00 +0000</pubDate>
      <guid>https://rednafi.com/python/redis-cache/</guid>
      <description>Cache API responses with Redis in Python to reduce redundant requests, improve response times, and handle expiring key-value pairs efficiently.</description>
      <category>Python</category>
      <category>API</category>
      <category>Redis</category>
      <content:encoded><![CDATA[<p><strong><em>Updated on 2023-09-11</em></strong>: <em>Fix broken URLs.</em></p>
<p>Recently, I was working with Mapbox&rsquo;s <a href="https://docs.mapbox.com/api/navigation/#optimization" rel="noopener noreferrer" target="_blank">Route optimization API</a>. It tries to solve the
<a href="https://en.wikipedia.org/wiki/Travelling_salesman_problem" rel="noopener noreferrer" target="_blank">traveling salesman problem</a> where you provide the API with coordinates of multiple places
and it returns a duration-optimized route between those locations. This is a perfect usecase
where <a href="https://redis.io/" rel="noopener noreferrer" target="_blank">Redis</a> caching can come handy. Redis is a fast and lightweight in-memory database
with additional persistence options; making it a perfect candidate for the task at hand.
Here, caching can save you from making redundant API requests and also, it can dramatically
improve the response time as well.</p>
<p>I found that in my country, the optimized routes returned by the API do not change
dramatically for at least for a couple of hours. So the workflow will look something like
this:</p>
<ul>
<li>Caching the API response in Redis using the key-value data structure. Here the requested
coordinate-string will be the key and the response will be the corresponding value.</li>
<li>Setting a timeout on the records.</li>
<li>Serving new requests from cache if the records exist.</li>
<li>Only send a new request to MapBox API if the response is not cached and then add that
response to cache.</li>
</ul>
<h2 id="setting-up-redis--redisinsight">Setting up Redis &amp; RedisInsight</h2>
<p>To proceed with the above workflow, you&rsquo;ll need to install and setup Redis database on your
system. For monitoring the database, I&rsquo;ll be using <a href="https://redislabs.com/redisinsight/" rel="noopener noreferrer" target="_blank">RedisInsight</a>. The easiest way to setup
Redis and RedisInsight is through <a href="https://www.docker.com/" rel="noopener noreferrer" target="_blank">Docker</a>. Here&rsquo;s a docker-compose that you can use to
setup everything with a single command.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yml" data-lang="yml"><span class="line"><span class="cl"><span class="c"># docker-compose.yml</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">version</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;3.9&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">services</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">redis</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">container_name</span><span class="p">:</span><span class="w"> </span><span class="l">redis-cont</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;redis:alpine&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">environment</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="l">REDIS_PASSWORD=ubuntu</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="l">REDIS_REPLICATION_MODE=master</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">ports</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="s2">&#34;6379:6379&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">volumes</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># save redisearch data to your current working directory</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="l">./redis-data:/data</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">command</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># Save if 100 keys are added in every 10 seconds</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="s2">&#34;--save 10 100&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># Set password</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="s2">&#34;--requirepass ubuntu&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">redisinsight</span><span class="p">:</span><span class="w"> </span><span class="c"># redis db visualization dashboard</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">container_name</span><span class="p">:</span><span class="w"> </span><span class="l">redisinsight-cont</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">redislabs/redisinsight</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">ports</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="m">8001</span><span class="p">:</span><span class="m">8001</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">volumes</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="l">redisinsight:/db</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">volumes</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">redis-data</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">redisinsight</span><span class="p">:</span><span class="w">
</span></span></span></code></pre></div><p>The above <code>docker-compose</code> file has two services, <code>redis</code> and <code>redisinsight</code>. I&rsquo;ve set up
the database with a dummy password <code>ubuntu</code> and made it persistent using a folder named
<code>redis-data</code> in the current working directory. The database listens in localhost&rsquo;s port
<code>6379</code>. You can monitor the database using <code>redisinsight</code> in port 8000. To spin up Redis and
RedisInsight containers, run:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl">docker compose up -d
</span></span></code></pre></div><p>This command will start the database and monitor accordingly. You can go to this
<code>localhost:8000</code> link using your browser and connect redisinsight to your database. After
connecting your database, you should see a dashboard like this in your redisinsight panel:</p>
<p><img alt="RedisInsight dashboard showing connected Redis database overview" loading="lazy" src="https://blob.rednafi.com/static/images/redis_cache/img_1.png"></p>
<h2 id="preparing-python-environment">Preparing Python environment</h2>
<p>For local development, you can set up your python environment and install the dependencies
using pip. Here, I&rsquo;m on a Linux machine and using virtual environment for isolation. The
following commands will work if you&rsquo;re on a *nix based system and have <code>python 3.12</code>
installed on your system. This will install the necessary dependencies in a virtual
environment:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl">python3.12 -m venv .venv
</span></span><span class="line"><span class="cl"><span class="nb">source</span> .venv/bin/activate
</span></span><span class="line"><span class="cl">pip install redis httpx
</span></span></code></pre></div><h2 id="workflow">Workflow</h2>
<h3 id="connecting-python-client-to-redis">Connecting Python client to Redis</h3>
<p>Assuming the database server is running and you&rsquo;ve installed the dependencies, the following
snippet connects <code>redis-py</code> client to the database.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">redis</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">sys</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">redis_connect</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="n">redis</span><span class="o">.</span><span class="n">client</span><span class="o">.</span><span class="n">Redis</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">client</span> <span class="o">=</span> <span class="n">redis</span><span class="o">.</span><span class="n">Redis</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="n">host</span><span class="o">=</span><span class="s2">&#34;localhost&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="n">port</span><span class="o">=</span><span class="mi">6379</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="n">password</span><span class="o">=</span><span class="s2">&#34;ubuntu&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="n">db</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="n">socket_timeout</span><span class="o">=</span><span class="mi">5</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">ping</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">ping</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">ping</span> <span class="ow">is</span> <span class="kc">True</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="n">client</span>
</span></span><span class="line"><span class="cl">    <span class="k">except</span> <span class="n">redis</span><span class="o">.</span><span class="n">AuthenticationError</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="nb">print</span><span class="p">(</span><span class="s2">&#34;AuthenticationError&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">client</span> <span class="o">=</span> <span class="n">redis_connect</span><span class="p">()</span>
</span></span></code></pre></div><p>The above excerpt tries to connect to the <code>Redis</code> database server using the port <code>6379</code>.
Notice, how I&rsquo;m providing the password <code>ubuntu</code> via the <code>password</code> argument. Here,
<code>client.ping()</code> helps you determine if a connection has been established successfully. It
returns <code>True</code> if a successful connection can be established or raises specific errors in
case of failures. The above function handles <code>AuthenticationError</code> and prints out an error
message if the error occurs. If everything goes well, running the <code>redis_connect()</code> function
will return an instance of the <code>redis.client.Redis</code> class. This instance will be used later
to set and retrieve data to and from the redis database.</p>
<h3 id="getting-route-data-from-mapbox-api">Getting route data from MapBox API</h3>
<p>The following function strikes the MapBox Route Optimization API and collects route data.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">httpx</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">get_routes_from_api</span><span class="p">(</span><span class="n">coordinates</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">dict</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;&#34;&#34;Data from mapbox api.&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">with</span> <span class="n">httpx</span><span class="o">.</span><span class="n">Client</span><span class="p">()</span> <span class="k">as</span> <span class="n">client</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">base</span> <span class="o">=</span> <span class="s2">&#34;https://api.mapbox.com/optimized-trips/v1/mapbox/driving&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="n">geometries</span> <span class="o">=</span> <span class="s2">&#34;geojson&#34;</span>
</span></span><span class="line"><span class="cl">        <span class="n">access_token</span> <span class="o">=</span> <span class="s2">&#34;Your-MapBox-API-token&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="n">url</span> <span class="o">=</span> <span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">base</span><span class="si">}</span><span class="s2">/</span><span class="si">{</span><span class="n">coordinates</span><span class="si">}</span><span class="s2">?geometries=</span><span class="si">{</span><span class="n">geometries</span><span class="si">}</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">            <span class="sa">f</span><span class="s2">&#34;&amp;access_token=</span><span class="si">{</span><span class="n">access_token</span><span class="si">}</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">response</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">url</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">response</span><span class="o">.</span><span class="n">json</span><span class="p">()</span>
</span></span></code></pre></div><p>The above code uses Python&rsquo;s <a href="https://github.com/encode/httpx" rel="noopener noreferrer" target="_blank">HTTPx</a> library to make the get request. HTTPx is almost a
drop-in replacement for the ubiquitous <a href="https://github.com/psf/requests" rel="noopener noreferrer" target="_blank">requests</a> library but way faster and has async
support. Here, I&rsquo;ve used context manager <code>httpx.Client()</code> for better resource management
while making the <code>get</code> request. You can read more about <a href="/python/contextmanager">context managers</a> and how to use
them for hassle free resource management.</p>
<p>The <code>base_url</code> is the base url of the route optimization API and the you&rsquo;ll need to provide
your own access token in the <code>access_token</code> field. Notice, how the <code>url</code> variable builds up
the final request url. The <code>coordinates</code> are provided using the
<code>lat0,lon0;lat1,lon1;lat2,lon2...</code> format. Rest of the function sends the http requests and
converts the response into a native dictionary object using the <code>response.json()</code> method.</p>
<h3 id="setting--retrieving-data-to--from-redis-database">Setting &amp; retrieving data to &amp; from Redis database</h3>
<p>The following two functions retrieves data from and sets data to redis database
respectively.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">timedelta</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">get_routes_from_cache</span><span class="p">(</span><span class="n">key</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;&#34;&#34;Get data from redis.&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="n">val</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">val</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">set_routes_to_cache</span><span class="p">(</span><span class="n">key</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">value</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">bool</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;&#34;&#34;Set data to redis.&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="n">state</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">setex</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="n">key</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="n">timedelta</span><span class="p">(</span><span class="n">seconds</span><span class="o">=</span><span class="mi">3600</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">        <span class="n">value</span><span class="o">=</span><span class="n">value</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">state</span>
</span></span></code></pre></div><p>Here, both the keys and the values are strings. In the second function,
<code>set_routes_to_cache</code>, the <code>client.setex()</code> method sets a timeout of 1 hour on the key.
After that the key and its associated value get deleted automatically.</p>
<h3 id="the-central-orchestration">The central orchestration</h3>
<p>The <code>route_optima</code> function is the primary agent that orchestrates and executes the caching
and returning of responses against requests. It roughly follows the execution flow shown
below.</p>
<p><img alt="Flowchart showing cache check and API request decision flow" loading="lazy" src="https://blob.rednafi.com/static/images/redis_cache/img_2.png"></p>
<p>When a new request arrives, the function first checks if the return-value exists in the
Redis cache. If the value exists, it shows the cached value, otherwise, it sends a new
request to the MapBox API, cache that value and then shows the result.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">route_optima</span><span class="p">(</span><span class="n">coordinates</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">dict</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="c1"># First it looks for the data in redis cache</span>
</span></span><span class="line"><span class="cl">    <span class="n">data</span> <span class="o">=</span> <span class="n">get_routes_from_cache</span><span class="p">(</span><span class="n">key</span><span class="o">=</span><span class="n">coordinates</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1"># If cache is found then serves the data from cache</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">data</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">data</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">data</span><span class="p">[</span><span class="s2">&#34;cache&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="kc">True</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">data</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="c1"># If cache is not found then sends request to the MapBox API</span>
</span></span><span class="line"><span class="cl">        <span class="n">data</span> <span class="o">=</span> <span class="n">get_routes_from_api</span><span class="p">(</span><span class="n">coordinates</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="c1"># This block sets saves the respose to redis and serves it directly</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">data</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;code&#34;</span><span class="p">)</span> <span class="o">==</span> <span class="s2">&#34;Ok&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="n">data</span><span class="p">[</span><span class="s2">&#34;cache&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="kc">False</span>
</span></span><span class="line"><span class="cl">            <span class="n">data</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">            <span class="n">state</span> <span class="o">=</span> <span class="n">set_routes_to_cache</span><span class="p">(</span><span class="n">key</span><span class="o">=</span><span class="n">coordinates</span><span class="p">,</span> <span class="n">value</span><span class="o">=</span><span class="n">data</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">            <span class="k">if</span> <span class="n">state</span> <span class="ow">is</span> <span class="kc">True</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">                <span class="k">return</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">data</span>
</span></span></code></pre></div><h3 id="exposing-as-an-api">Exposing as an API</h3>
<p>This part of the code wraps the original Route Optimization API and exposes that as a new
endpoint. I&rsquo;ve used <a href="https://fastapi.tiangolo.com/" rel="noopener noreferrer" target="_blank">FastAPI</a> to build the wrapper API. Doing this also hides the underlying
details of authentication and the actual endpoint of the MapBox API.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">fastapi</span> <span class="kn">import</span> <span class="n">FastAPI</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">app</span> <span class="o">=</span> <span class="n">FastAPI</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd">@app.get</span><span class="p">(</span><span class="s2">&#34;/route-optima/</span><span class="si">{coordinates}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">view</span><span class="p">(</span><span class="n">coordinates</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;&#34;&#34;This will wrap our original route optimization API and
</span></span></span><span class="line"><span class="cl"><span class="s2">    incorporate Redis Caching. You&#39;ll only expose this API to
</span></span></span><span class="line"><span class="cl"><span class="s2">    the end user.&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1"># coordinates = &#34;90.3866,23.7182;90.3742,23.7461&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">route_optima</span><span class="p">(</span><span class="n">coordinates</span><span class="p">)</span>
</span></span></code></pre></div><h3 id="putting-it-all-together">Putting it all together</h3>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># app.py</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">json</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">sys</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">timedelta</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">httpx</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">redis</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">fastapi</span> <span class="kn">import</span> <span class="n">FastAPI</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">redis_connect</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="n">redis</span><span class="o">.</span><span class="n">client</span><span class="o">.</span><span class="n">Redis</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">client</span> <span class="o">=</span> <span class="n">redis</span><span class="o">.</span><span class="n">Redis</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="n">host</span><span class="o">=</span><span class="s2">&#34;localhost&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="n">port</span><span class="o">=</span><span class="mi">6379</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="n">password</span><span class="o">=</span><span class="s2">&#34;ubuntu&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="n">db</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="n">socket_timeout</span><span class="o">=</span><span class="mi">5</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">ping</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">ping</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">ping</span> <span class="ow">is</span> <span class="kc">True</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="n">client</span>
</span></span><span class="line"><span class="cl">    <span class="k">except</span> <span class="n">redis</span><span class="o">.</span><span class="n">AuthenticationError</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="nb">print</span><span class="p">(</span><span class="s2">&#34;AuthenticationError&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">client</span> <span class="o">=</span> <span class="n">redis_connect</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">get_routes_from_api</span><span class="p">(</span><span class="n">coordinates</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">dict</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;&#34;&#34;Data from mapbox api.&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">with</span> <span class="n">httpx</span><span class="o">.</span><span class="n">Client</span><span class="p">()</span> <span class="k">as</span> <span class="n">client</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">base</span> <span class="o">=</span> <span class="s2">&#34;https://api.mapbox.com/optimized-trips/v1/mapbox/driving&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="n">geometries</span> <span class="o">=</span> <span class="s2">&#34;geojson&#34;</span>
</span></span><span class="line"><span class="cl">        <span class="n">access_token</span> <span class="o">=</span> <span class="s2">&#34;Your-MapBox-API-token&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="n">url</span> <span class="o">=</span> <span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">base</span><span class="si">}</span><span class="s2">/</span><span class="si">{</span><span class="n">coordinates</span><span class="si">}</span><span class="s2">?geometries=</span><span class="si">{</span><span class="n">geometries</span><span class="si">}</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">            <span class="sa">f</span><span class="s2">&#34;&amp;access_token=</span><span class="si">{</span><span class="n">access_token</span><span class="si">}</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">response</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">url</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">response</span><span class="o">.</span><span class="n">json</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">get_routes_from_cache</span><span class="p">(</span><span class="n">key</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;&#34;&#34;Data from redis.&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="n">val</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">val</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">set_routes_to_cache</span><span class="p">(</span><span class="n">key</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">value</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">bool</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;&#34;&#34;Data to redis.&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="n">state</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">setex</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="n">key</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="n">timedelta</span><span class="p">(</span><span class="n">seconds</span><span class="o">=</span><span class="mi">3600</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">        <span class="n">value</span><span class="o">=</span><span class="n">value</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">state</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">route_optima</span><span class="p">(</span><span class="n">coordinates</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">dict</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="c1"># First it looks for the data in redis cache</span>
</span></span><span class="line"><span class="cl">    <span class="n">data</span> <span class="o">=</span> <span class="n">get_routes_from_cache</span><span class="p">(</span><span class="n">key</span><span class="o">=</span><span class="n">coordinates</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1"># If cache is found then serves the data from cache</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">data</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">data</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">data</span><span class="p">[</span><span class="s2">&#34;cache&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="kc">True</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">data</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="c1"># If cache is not found then sends request to the MapBox API</span>
</span></span><span class="line"><span class="cl">        <span class="n">data</span> <span class="o">=</span> <span class="n">get_routes_from_api</span><span class="p">(</span><span class="n">coordinates</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="c1"># This block sets saves the respose to redis and serves it directly</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">data</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;code&#34;</span><span class="p">)</span> <span class="o">==</span> <span class="s2">&#34;Ok&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="n">data</span><span class="p">[</span><span class="s2">&#34;cache&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="kc">False</span>
</span></span><span class="line"><span class="cl">            <span class="n">data</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">            <span class="n">state</span> <span class="o">=</span> <span class="n">set_routes_to_cache</span><span class="p">(</span><span class="n">key</span><span class="o">=</span><span class="n">coordinates</span><span class="p">,</span> <span class="n">value</span><span class="o">=</span><span class="n">data</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">            <span class="k">if</span> <span class="n">state</span> <span class="ow">is</span> <span class="kc">True</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">                <span class="k">return</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">data</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">app</span> <span class="o">=</span> <span class="n">FastAPI</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd">@app.get</span><span class="p">(</span><span class="s2">&#34;/route-optima/</span><span class="si">{coordinates}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">view</span><span class="p">(</span><span class="n">coordinates</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">dict</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;&#34;&#34;This will wrap our original route optimization API and
</span></span></span><span class="line"><span class="cl"><span class="s2">    incorporate Redis Caching. You&#39;ll only expose this API to
</span></span></span><span class="line"><span class="cl"><span class="s2">    the end user.&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1"># coordinates = &#34;90.3866,23.7182;90.3742,23.7461&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">route_optima</span><span class="p">(</span><span class="n">coordinates</span><span class="p">)</span>
</span></span></code></pre></div><p>You can copy the complete code to a file named <code>app.py</code> and run the app using the command
below (assuming redis, redisinsight is running and you&rsquo;ve installed the dependencies
beforehand):</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl">uvicorn app.app:app --host 0.0.0.0 --port <span class="m">5000</span> --reload
</span></span></code></pre></div><p>This will run a local server where you can send new request with coordinates.</p>
<p>Go to your browser and hit the endpoint with a set of new coordinates. For example:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-txt" data-lang="txt"><span class="line"><span class="cl">http://localhost:5000/route-optima/90.3866,23.7182;90.3742,23.7461
</span></span></code></pre></div><p>This should return a response with the coordinates of the optimized route.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl">   <span class="nt">&#34;code&#34;</span><span class="p">:</span><span class="s2">&#34;Ok&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">   <span class="nt">&#34;waypoints&#34;</span><span class="p">:[</span>
</span></span><span class="line"><span class="cl">      <span class="p">{</span>
</span></span><span class="line"><span class="cl">         <span class="nt">&#34;distance&#34;</span><span class="p">:</span><span class="mf">26.041809241776583</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">         <span class="nt">&#34;name&#34;</span><span class="p">:</span><span class="s2">&#34;&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">         <span class="nt">&#34;location&#34;</span><span class="p">:[</span>
</span></span><span class="line"><span class="cl">            <span class="mf">90.386855</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="mf">23.718213</span>
</span></span><span class="line"><span class="cl">         <span class="p">],</span>
</span></span><span class="line"><span class="cl">         <span class="nt">&#34;waypoint_index&#34;</span><span class="p">:</span><span class="mi">0</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">         <span class="nt">&#34;trips_index&#34;</span><span class="p">:</span><span class="mi">0</span>
</span></span><span class="line"><span class="cl">      <span class="p">},</span>
</span></span><span class="line"><span class="cl">      <span class="p">{</span>
</span></span><span class="line"><span class="cl">         <span class="nt">&#34;distance&#34;</span><span class="p">:</span><span class="mf">6.286653078791968</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">         <span class="nt">&#34;name&#34;</span><span class="p">:</span><span class="s2">&#34;&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">         <span class="nt">&#34;location&#34;</span><span class="p">:[</span>
</span></span><span class="line"><span class="cl">            <span class="mf">90.374253</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="mf">23.746129</span>
</span></span><span class="line"><span class="cl">         <span class="p">],</span>
</span></span><span class="line"><span class="cl">         <span class="nt">&#34;waypoint_index&#34;</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">         <span class="nt">&#34;trips_index&#34;</span><span class="p">:</span><span class="mi">0</span>
</span></span><span class="line"><span class="cl">      <span class="p">}</span>
</span></span><span class="line"><span class="cl">   <span class="p">],</span>
</span></span><span class="line"><span class="cl">   <span class="nt">&#34;trips&#34;</span><span class="p">:[</span>
</span></span><span class="line"><span class="cl">      <span class="p">{</span>
</span></span><span class="line"><span class="cl">         <span class="nt">&#34;geometry&#34;</span><span class="p">:{</span>
</span></span><span class="line"><span class="cl">            <span class="nt">&#34;coordinates&#34;</span><span class="p">:[</span>
</span></span><span class="line"><span class="cl">               <span class="p">[</span>
</span></span><span class="line"><span class="cl">                  <span class="mf">90.386855</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">                  <span class="mf">23.718213</span>
</span></span><span class="line"><span class="cl">               <span class="p">],</span>
</span></span><span class="line"><span class="cl">               <span class="s2">&#34;...
</span></span></span><span class="line"><span class="cl"><span class="s2">...&#34;</span>
</span></span><span class="line"><span class="cl">            <span class="p">],</span>
</span></span><span class="line"><span class="cl">            <span class="nt">&#34;type&#34;</span><span class="p">:</span><span class="s2">&#34;LineString&#34;</span>
</span></span><span class="line"><span class="cl">         <span class="p">},</span>
</span></span><span class="line"><span class="cl">         <span class="nt">&#34;legs&#34;</span><span class="p">:[</span>
</span></span><span class="line"><span class="cl">            <span class="p">{</span>
</span></span><span class="line"><span class="cl">               <span class="nt">&#34;summary&#34;</span><span class="p">:</span><span class="s2">&#34;&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">               <span class="nt">&#34;weight&#34;</span><span class="p">:</span><span class="mf">3303.1</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">               <span class="nt">&#34;duration&#34;</span><span class="p">:</span><span class="mf">2842.8</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">               <span class="nt">&#34;steps&#34;</span><span class="p">:[</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">               <span class="p">],</span>
</span></span><span class="line"><span class="cl">               <span class="nt">&#34;distance&#34;</span><span class="p">:</span><span class="mf">5250.2</span>
</span></span><span class="line"><span class="cl">            <span class="p">},</span>
</span></span><span class="line"><span class="cl">            <span class="p">{</span>
</span></span><span class="line"><span class="cl">               <span class="nt">&#34;summary&#34;</span><span class="p">:</span><span class="s2">&#34;&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">               <span class="nt">&#34;weight&#34;</span><span class="p">:</span><span class="mf">2536.5</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">               <span class="nt">&#34;duration&#34;</span><span class="p">:</span><span class="mi">2297</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">               <span class="nt">&#34;steps&#34;</span><span class="p">:[</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">               <span class="p">],</span>
</span></span><span class="line"><span class="cl">               <span class="nt">&#34;distance&#34;</span><span class="p">:</span><span class="mf">4554.8</span>
</span></span><span class="line"><span class="cl">            <span class="p">}</span>
</span></span><span class="line"><span class="cl">         <span class="p">],</span>
</span></span><span class="line"><span class="cl">         <span class="nt">&#34;weight_name&#34;</span><span class="p">:</span><span class="s2">&#34;routability&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">         <span class="nt">&#34;weight&#34;</span><span class="p">:</span><span class="mf">5839.6</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">         <span class="nt">&#34;duration&#34;</span><span class="p">:</span><span class="mf">5139.8</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">         <span class="nt">&#34;distance&#34;</span><span class="p">:</span><span class="mi">9805</span>
</span></span><span class="line"><span class="cl">      <span class="p">}</span>
</span></span><span class="line"><span class="cl">   <span class="p">],</span>
</span></span><span class="line"><span class="cl">   <span class="nt">&#34;cache&#34;</span><span class="p">:</span><span class="kc">false</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span></code></pre></div><p>If you&rsquo;ve hit the above URL for the first time, the <code>cache</code> attribute of the json response
should show <code>false</code>. This means that the response is being served from the original MapBox
API. However, hitting the same URL with the same coordinates again will show the cached
response and this time the <code>cache</code> attribute should show <code>true</code>.</p>
<h2 id="inspection">Inspection</h2>
<p>Once you&rsquo;ve got everything up and running you can inspect the cache via redis insight. To do
so, go to the link below while your app server is running:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-txt" data-lang="txt"><span class="line"><span class="cl">http://localhost:8000/
</span></span></code></pre></div><p>Select the <code>Browser</code> panel from the left menu and click on a key of your cached data. It
should show something like this:</p>
<p><img alt="RedisInsight browser panel displaying cached coordinate data as key-value pairs" loading="lazy" src="https://blob.rednafi.com/static/images/redis_cache/img_3.png"></p>
<p>Also you can play around with the API in the swagger UI. To do so, go to the following link:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-txt" data-lang="txt"><span class="line"><span class="cl">http://localhost:5000/docs
</span></span></code></pre></div><p>This will take you to the swagger dashboard. Here you can make requests using the
interactive UI. Go ahead and inspect how the caching works for new coordinates.</p>
<p><img alt="FastAPI Swagger UI showing route optimization API endpoint" loading="lazy" src="https://blob.rednafi.com/static/images/redis_cache/img_4.png"></p>
<h2 id="remarks">Remarks</h2>
<p>You can find the complete source code of the app in my <a href="https://github.com/rednafi/redis-request-caching" rel="noopener noreferrer" target="_blank">HTTP request caching with Redis</a>
repository.</p>
<h2 id="disclaimer">Disclaimer</h2>
<p>This app has been made for demonstration purpose only. So it might not reflect the best
practices of production ready applications.</p>
<!-- references -->
<!-- prettier-ignore-start -->
<!-- prettier-ignore-end -->
]]></content:encoded>
    </item>
  </channel>
</rss>
