Misc on Redowan's Reflections

Migrating from GNU stow to chezmoi

Redowan Delowar — Fri, 12 Jun 2026 00:00:00 +0200

I’ve been managing my dotfiles with GNU stow for a few years. I even wrote a piece with a corny title about that setup back in 2023. Stow served me well, but managing symlinks across multiple devices slowly became a pain in the butt.

So I started looking around for a better tool and even considered writing my own. Then a colleague pointed me to chezmoi, and so far I’m liking it a lot. It does everything I need, and I’ve started tracking my agent skill files with it too.

The machines #

I run three Macs: a MacBook Pro for work, a MacBook Air for personal use, and a Mac Mini that acts as a small personal server. The Mini mostly gets SSHed into from the other two. It’s still a Mac with my shell on it, so the same dotfiles apply.

I also keep a few Linux VMs around, but I rarely need my dotfiles on servers. Ansible provisions those. This workflow is strictly for the desktop machines.

When I outgrew stow #

Stow’s model is symlinking. The config files live in a git repo, grouped into directories that stow calls packages, and stowing a package links its files into the home directory. For a single machine it still holds up. The commands are idempotent and there’s almost nothing to learn.

The trouble is that symlinks cut both ways. Every edit on every machine writes straight through the link into that machine’s clone of the repo. Months later I’d find dirty working trees on the Air with changes I had no memory of making. Half of them conflicted with whatever the Pro had already pushed. Keeping three clones converged turned into a chore.

Fresh machines were the other half of the problem. Stow won’t link over a real file. By the time Homebrew and a couple of tools have run on a new Mac, files like ~/.zprofile and ~/.gitconfig already exist. Bootstrapping meant cloning the repo, deleting the conflicting files by hand, and restowing every package while trying to remember what I’d named them. And stow only does files. Homebrew packages and macOS settings lived in separate scripts that I had to remember to run in the right order.

How chezmoi works #

Chezmoi keeps a source directory at ~/.local/share/chezmoi, which is a regular git repo. chezmoi add ~/.zshrc copies the live file into it and names the copy dot_zshrc. Adding ~/.config/gh/config.yml creates dot_config/gh/config.yml, parent directories included. I never create those names by hand since chezmoi add derives them from the real paths. The tree ends up mirroring the home directory, with every leading dot spelled out as a dot_ prefix.

dot_ is one of several attributes that chezmoi encodes into file names. A private_ prefix strips group and world permissions from the file. A .tmpl suffix turns the file into a Go template that can read per-machine data. I use templates sparingly, and every one of them shows up later in this post.

chezmoi apply goes the other way. It writes every tracked file back to the home path its name spells out, so dot_zshrc lands at ~/.zshrc. The copies are real files, not symlinks. The source directory is the single source of truth. When a file in the home directory stops matching its source copy, chezmoi diff shows the difference and the next apply puts it back.

Losing the automatic write-through of symlinks turned out to be the thing I like most. Nothing changes in the repo unless I deliberately put the change there.

What I track #

All of it sits in that source directory. chezmoi cd drops me into a subshell there, and here’s the entire tree:

~/.local/share/chezmoi
├── .chezmoi.toml.tmpl
├── .chezmoiignore
├── .chezmoiscripts
│   └── macos
│       ├── run_onchange_after_disable-macos-animations.sh
│       ├── run_onchange_after_init-macos-machine.sh.tmpl
│       └── run_onchange_before_install-homebrew-bundle.sh.tmpl
├── .gitignore
├── Brewfile
├── README.md
├── dot_agents
│   └── skills
│       ├── go-modernize
│       ├── go-styleguide
│       └── meatspeak
├── dot_claude
│   ├── settings.json
│   └── symlink_skills.tmpl
├── dot_codex
│   └── private_config.toml
├── dot_config
│   ├── gh
│   │   ├── config.yml
│   │   └── private_hosts.yml
│   └── ghostty
│       └── config
├── dot_gitconfig
├── dot_gitconfig-pers
├── dot_gitconfig-werk
├── dot_shellcheckrc
├── dot_zsh_aliases
└── dot_zshrc

The list is short because I dislike customizing tools and stick to defaults where I can. The dotfiles proper are the zsh, git, shellcheck, ghostty, and GitHub CLI configs. I track Claude Code’s settings.json and Codex’s config.toml too, so the agents behave the same on every machine. The private_ prefix on gh’s hosts.yml and the Codex config keeps those two at 0600. I’ll talk about the skills under dot_agents at the end.

The three gitconfigs split my identities. All my projects live under two directories, ~/canvas/werk/ for work and ~/canvas/pers/ for everything personal, and both exist on every machine. The main gitconfig routes identity by where a repo lives:

[includeIf "gitdir:~/canvas/pers/"]
    path = ~/.gitconfig-pers

[includeIf "gitdir:~/canvas/werk/"]
    path = ~/.gitconfig-werk

Repos under ~/canvas/pers/ get my personal email and repos under ~/canvas/werk/ get the work one. That’s a plain git feature, not chezmoi templating, but chezmoi guarantees all three files exist on every machine.

That .chezmoi.toml.tmpl at the top is chezmoi’s own config template. It asks for the machine’s name once and remembers the answer in ~/.config/chezmoi/chezmoi.toml:

{{- $machineName := promptStringOnce . "machineName" "machineName" .chezmoi.hostname -}}

[data]
    machineName = {{ $machineName | quote }}

The machine setup script reads that value to set the hostname. It’s the only per-machine data in the whole repo. Everything else is identical everywhere. I keep it this way partly for simplicity and partly because I’m not a big fan of Go’s template syntax, so the less I have to muck around with it, the better.

.chezmoiignore lists README.md, the Brewfile, and Brewfile.lock.json, so all three stay in the source directory without ever being written to the home directory. A plain .gitignore keeps the lock file out of version control. I’ll cover the Brewfile and the scripts under .chezmoiscripts in the next section.

Bootstrapping a new Mac #

Homebrew goes on first, and then the whole setup is two commands:

brew install chezmoi
chezmoi init --apply \
    --promptString machineName=mini \
    https://github.com/rednafi/dotfiles.git

chezmoi init clones the repo into ~/.local/share/chezmoi, and --apply writes every tracked file into place right away. The --promptString flag pre-answers the config template’s question. Without it, chezmoi asks interactively. Scripts run as part of the same apply.

Anything under .chezmoiscripts/ gets executed during apply, and the file names control the timing:

A before script runs before chezmoi writes any files.
An after script runs once they’re all in place.
The run_onchange_ prefix makes a script fire on the first apply and after that only when its contents change.

On a fresh machine that works out to: install the Homebrew packages, lay down the dotfiles, then configure macOS itself. The onchange part enables a trick that comes straight from the chezmoi docs. Here’s the Homebrew script, trimmed:

#!/usr/bin/env bash
# Brewfile checksum: {{ include "Brewfile" | sha256sum }}

# ... elided

brewfile={{ joinPath .chezmoi.sourceDir "Brewfile" | quote }}

"$brew_bin" bundle check --no-upgrade --file "$brewfile" >/dev/null 2>&1 \
    || "$brew_bin" bundle install --no-upgrade --file "$brewfile"

The elided lines locate the Homebrew binary and store its path in $brew_bin. The template inlines a hash of the Brewfile into a comment. Adding a package to the Brewfile changes the hash, which changes the rendered script, which makes chezmoi run it again on the next apply. So brew bundle fires exactly when the package list changes and stays quiet otherwise. The --no-upgrade flag keeps it from touching packages that are already installed. Upgrades stay manual since I want to see what’s about to change first.

The Brewfile is about sixty lines long. An excerpt:

brew "chezmoi"
brew "fzf"
brew "gh"
brew "micro"
brew "ripgrep"
brew "uv"

cask "claude-code"
cask "codex"
cask "ghostty"
cask "raycast"

Two more scripts run after the files land. The first sets the hostname from machineName and writes every macOS default I’d otherwise set by clicking through System Settings on each new machine. The second turns off most of the UI animations. Both are long lists of plain defaults write calls, and the details are in the repo.

Every script starts with a Darwin check and exits early anywhere else, so nothing fires if I ever apply this on a Linux box. I used to keep all of this in a setup script that I’d forget to run. Now it’s part of apply and I can’t forget.

Day to day #

The whole routine is about five commands.

Edits usually start at the source. chezmoi edit opens the source copy behind a home file, and --apply writes it through when I close the editor:

chezmoi edit --apply ~/.zshrc

Sometimes the edit happens in the other direction. An installer appends to ~/.zshrc, or I tweak the live file directly out of habit. Now the home directory is ahead of the source, and chezmoi diff will show that an apply would undo my change. When the change should stick, I import the live file back into the source:

chezmoi add ~/.zshrc

When several home files have moved ahead of their sources like this, chezmoi re-add re-imports them all in one go.

Once the source state looks right, sharing it is plain git from inside the source repo:

chezmoi cd
git add -A
git commit -m "Update dotfiles"
git push
exit

On the other machines, catching up is one command:

chezmoi update --verbose

That pulls the repo and applies it in one shot. When I want to inspect what’s coming, I split it up and read the diff first:

chezmoi git pull -- --autostash --rebase
chezmoi diff
chezmoi apply --verbose

Packages can fall out of sync with the Brewfile too. brew bundle check reports anything the Brewfile expects but the machine lacks, brew outdated --greedy shows what’s stale, and brew bundle cleanup lists what’s installed but untracked:

brew bundle check --no-upgrade --file "$(chezmoi source-path)/Brewfile"
brew outdated --greedy
brew bundle cleanup --file "$(chezmoi source-path)/Brewfile"

Tracking agent skills #

The newest additions to the repo are skills for LLM agents. A skill is a folder with a SKILL.md and whatever reference files it needs. The SKILL.md carries name and description frontmatter followed by instructions. The layout comes straight from the Agent Skills spec, an open standard that started at Anthropic and has been adopted by a growing list of agent products.

Because the format is standard, one copy should work everywhere. I use both Claude Code and Codex, and the skills live in ~/.agents/skills, which Codex picks up by default. In chezmoi terms that’s a regular directory at dot_agents/skills/, tracked like any other config.

Claude Code hasn’t caught up with that convention yet. It looks for personal skills in ~/.claude/skills and knows nothing about ~/.agents. The fix is a one-line file in the source repo at dot_claude/symlink_skills.tmpl:

{{ .chezmoi.homeDir }}/.agents/skills

Three name parts work together here:

The dot_claude/ directory and the file name map the target to ~/.claude/skills, the same way dot_zshrc maps to ~/.zshrc.
The symlink_ prefix tells chezmoi to create that target as a symlink instead of a regular file, pointing wherever the file’s content says.
The .tmpl suffix makes chezmoi render the content first, so {{ .chezmoi.homeDir }} expands to the right home directory on whichever machine is applying.

After an apply:

ls -ld ~/.claude/skills

lrwxr-xr-x 1 rednafi staff 29 Jun 11 17:37 /Users/rednafi/.claude/skills -> /Users/rednafi/.agents/skills

There’s a mild irony in leaving stow to escape symlinks and then having chezmoi manage the one symlink I still need. But that’s on Anthropic being a baby and not following the convention other agents already follow. Now both agents read the same skill files, git holds a single copy, and a new machine picks all of it up from the same chezmoi init as everything else.

Everything here lives in my dotfiles repo. Steal whatever looks useful.

Putting this blog on ATProto with standard.site

Redowan Delowar — Sun, 07 Jun 2026 00:00:00 +0000

I put this blog on standard.site. Every post now also lives as a record on ATProto (the protocol behind Bluesky), and new ones publish themselves whenever I push to main.

What it is #

standard.site is a set of shared ATProto lexicons. The two that matter here are site.standard.publication and site.standard.document. The publication record describes the blog: name, URL, icon. Each post becomes a document record that lives in my own data repository on a PDS and points back at the publication. To prove the records are actually mine, there’s a /.well-known/site.standard.publication file on my domain and a link-rel tag in every post’s HTML pointing at the matching record. The two ends point at each other, with no central registry in between.

sequenceDiagram participant R as Reader participant S as rednafi.com participant P as PDS R->>S: GET /zephyr/carry-the-pager/ S-->>R: HTML + a site.standard.document link tag R->>P: resolve that document record P-->>R: site = publication URI, path = /zephyr/carry-the-pager/ R->>S: GET /.well-known/site.standard.publication S-->>R: the same publication URI Note over R: URIs match, so it's provably rednafi.com's

Why bother #

Mostly the previews. Share one of my posts on Bluesky and the link turns into a card with the title, description, and image instead of a bare URL. That works because the post is a real record the network can read. Bluesky shows richer previews for standard.site links.

It goes past Bluesky, though. The records sit in my own PDS, so any reader can pick them up on its own. docs.surf already lists my posts, and a search on pckt turns them up too:

It’s cheap POSSE on top of that: rednafi.com stays the canonical copy while a copy syndicates out into the ATmosphere.

Setting it up with Sequoia #

I didn’t hand-roll any of the ATProto plumbing. Sequoia is a CLI by Steve Simkins that does the whole thing for static sites. It doesn’t much care what built yours (Hugo, Astro, Eleventy) as long as it’s Markdown. If you want to put your own blog on standard.site, it goes roughly like this.

First, you need an ATProto identity, and a Bluesky account is the easy way to get one. The records live in your own PDS. Ownership is checked against a domain, so set your site’s domain as your handle (mine is rednafi.com) and mint an app password for the CLI.

Then run sequoia init in the repo. It authenticates against your PDS, creates a site.standard.publication record describing the blog (name, URL, icon), and scaffolds a sequoia.json. That config is small: it points at your content directory and maps the frontmatter fields it reads, like the publish date and the slug.

{
  "siteUrl": "https://rednafi.com",
  "contentDir": "content",
  "publicationUri": "at://did:plc:fgtm2c26vfcj74rfmeggbyqj/site.standard.publication/3mnl6f7ob462z",
  "frontmatter": { "publishDate": "date", "slugField": "slug" }
}

That publicationUri is the at:// address of the publication record init just made. The same URI also lands in static/.well-known/site.standard.publication, so the domain and the record name each other and the ownership check holds.

Each post’s HTML also needs a <link rel="site.standard.document"> pointing at that post’s record. sequoia inject can patch the tags into your built HTML; I emit them from my Hugo head partial instead.

With that wired up, sequoia publish walks the content, creates a site.standard.document record per post, and writes the resulting atUri back into each post’s frontmatter. State lives in .sequoia-state.json, so reruns only touch what actually changed.

Making it hands-free #

I didn’t want to run sequoia publish by hand, so it happens in CI. The job does a few things before the Hugo build:

checks that every post has the same frontmatter keys in the same order
sets slug and atprotoPath from the filename
stops the build if a post has extra fields, missing fields, or a wrong path
copies the posts to a temporary Sequoia directory and adds the shared cover there
asks Sequoia to publish only posts whose content changed
writes Sequoia’s returned atUri values and .sequoia-state.json back
commits only those generated metadata changes with [skip ci]
builds and deploys the Hugo site

My routine didn’t change: write Markdown and push to main. CI fills in the atUri, commits it back, and lets the deploy continue. This post turned into a site.standard.document the moment the deploy ran.

Seeing it work #

The previews are what I actually wanted. The card is just the record rendered, so I can put a live, clickable one right here instead of a screenshot:

If you won't carry the pager, maybe don't push to mainline

Drive-by AI changes break the shared model a team builds around its code, and the ICs end up cleaning up the mess. Why pushing to mainline should come with the pager.

May 30, 2026·8m read

Redowan's Reflections rednafi.com · by @rednafi.com View publication

That same post also exists as a record on pdsls. It has the title, description, path, tags, and the full body.

If you want to copy the setup, it’s all in the repo: config, script, and the ci workflow.

Hierarchical rate limiting with Redis sorted sets

Redowan Delowar — Sun, 12 Jan 2025 00:00:00 +0000

Recently at work, we ran into this problem:

We needed to send Slack notifications for specific events but had to enforce rate limits to avoid overwhelming the channel. Here’s how the limits worked:

Global limit: Max 100 requests every 30 minutes.
Category limit: Each event type (e.g., errors, warnings) capped at 10 requests per 30 minutes.

Now, imagine this:

There are 20 event types.
Each type hits its 10-notification limit in 30 minutes.
That’s 200 requests total, but the global limit only allows 100. So, 100 requests must be dropped - even if some event types still have room under their individual caps.

This created a hierarchy of limits:

Category limits keep any event type from exceeding 10 requests.
The global limit ensures the combined total stays under 100.

Every 30 minutes, the system resets. Here are two issues that could arise:

If some event types are busier, the global limit could block quieter ones.
Even with room under the global limit, some event types might still hit their category caps.

In our case, the event types are limited, and the category limits are both uniform and significantly smaller than the global limit, so this isn’t a concern.

Redis sorted sets #

The notification sender service runs on multiple instances, each processing events and sending notifications independently. Without a shared system to enforce rate limits, these instances would maintain separate counters for global and category-specific limits. This would create inconsistencies because no instance would have a complete view of the overall activity, leading to conflicts and potential exceedance of limits.

Redis provides a centralized state that all instances can access, ensuring they share the same counters for rate limits. This removes inconsistencies and makes rate limiting reliable, even when the notification sender scales to multiple instances.

Sorted sets in Redis track notifications within a rolling time window by using timestamps as scores, which keeps entries ordered by time. The implementation:

Maintains a global sorted set to enforce the overall limit (e.g., 100 notifications per 30 minutes).
Uses category-specific sorted sets to enforce category limits for each event type (e.g., 10 notifications per 30 minutes for errors, warnings, etc.).

The limits are enforced with two Redis commands:

ZREMRANGEBYSCORE removes entries with timestamps outside the rolling time window, keeping only recent notifications.
ZCARD counts the remaining entries in a set to check whether the global or category-specific limits have been reached.

Lua script #

Instead of embedding the rate-limiting logic directly into the notification sender, we chose to implement it as a Lua script in Redis. While we could write the logic in the code and run it in a Redis pipeline, we opted not to, for the following reasons:

A dedicated script keeps the rate-limiting logic separate and independently auditable.
It saves a few TCP calls, as the entire logic runs within Redis itself.
And most importantly, I wanted to write some Lua.

The script is as follows:

-- rate_limiter.lua
local function check_rate_limit(
  global_key, category_key, global_limit, category_limit, window
)
  -- Get the current timestamp in seconds (including microseconds)
  local current_time_raw = redis.call('TIME') -- {secs, usecs}
  local current_time = current_time_raw[1] + current_time_raw[2] / 1e6

  -- Step 1: Remove expired entries
  redis.call('ZREMRANGEBYSCORE', global_key, 0, current_time - window)
  redis.call('ZREMRANGEBYSCORE', category_key, 0, current_time - window)

  -- Step 2: Check the global limit
  local global_count = redis.call('ZCARD', global_key)
  if global_count >= global_limit then
      return 0 -- Reject the request if the global limit is reached
  end

  -- Step 3: Check the category-specific limit
  local category_count = redis.call('ZCARD', category_key)
  if category_count >= category_limit then
      return 0 -- Reject the request if the category limit is reached
  end

  -- Step 4: Add the current notification to the sorted sets
  redis.call('ZADD', global_key, current_time, current_time)
  redis.call('ZADD', category_key, current_time, current_time)

  return 1 -- Allow the request
end

-- Parameters passed to the script:
-- KEYS[1]: The Redis key for the global sorted set
-- KEYS[2]: The Redis key for the category-specific sorted set
-- ARGV[1]: Global limit (e.g., 100)
-- ARGV[2]: Category limit (e.g., 10)
-- ARGV[3]: Time window in seconds (e.g., 1800 for 30 minutes)

local global_key = KEYS[1]
local category_key = KEYS[2]
local global_limit = tonumber(ARGV[1])
local category_limit = tonumber(ARGV[2])
local window = tonumber(ARGV[3])

-- Execute the rate-limiting function and return the result
return check_rate_limit(
    global_key, category_key, global_limit, category_limit, window
)

The script performs the following operations in order:

Remove expired entries:
- It uses ZREMRANGEBYSCORE to remove notifications older than the time window (current_time - window). This ensures that only active notifications are considered for the limits.
- This eliminates the need for additional bookkeeping to remove expired keys. ZREMRANGEBYSCORE is fast enough to handle the removal of a small number of keys during each invocation.
Check the global limit:
- ZCARD counts the number of active notifications in the global sorted set.
- If this count equals or exceeds the global limit (e.g., 100), the request is rejected (return 0).
Check the category-specific limit:
- ZCARD is used again to count the active notifications for the specific category.
- If this count equals or exceeds the category limit (e.g., 10), the request is rejected (return 0).
Add the notification:
- If both limits are within bounds, the script uses ZADD to insert the current notification into both the global and category-specific sorted sets, using a timestamp as the score for accurate tracking.

Using the script #

You can load the Lua script from disk, register it with Redis, and call it before invoking the notification service. If the script returns 0, drop the notification request. If it returns 1, send the notification. Here’s how to do it in Python:

from redis import Redis
from redis.commands.core import Script


def load_lua_script(redis_client: Redis, script_path: str) -> Script:
    with open(script_path, "r") as file:
        lua_script = file.read()
    return redis_client.register_script(lua_script)


def send_notification(
    script: Script,
    global_key: str,
    category_key: str,
    global_limit: int,
    category_limit: int,
    window: int,
    message: str,
) -> None:
    # Check the rate limiter
    result: int = script(
        keys=[global_key, category_key],
        args=[global_limit, category_limit, window],
    )

    if result == 1:
        # Allowed: send the notification
        print(f"Notification sent: {message}")
        # Add actual notification-sending logic here
    else:
        # Blocked: drop the notification
        print(f"Notification dropped (rate limit exceeded): {message}")


if __name__ == "__main__":
    # Connect to Redis
    redis_client = Redis(host="localhost", port=6379)

    # Load and register the Lua script
    script_path = "rate_limiter.lua"
    script = load_lua_script(redis_client, script_path)

    # Define rate limiting parameters
    global_key = "rate_limit:global"
    category_key = "rate_limit:category:errors"
    global_limit = 100  # Max 100 requests globally
    category_limit = 10  # Max 10 requests per category
    window = 1800  # 30-minute window

    # Send a single notification
    send_notification(
        script,
        global_key,
        category_key,
        global_limit,
        category_limit,
        window,
        "This is a single notification message",
    )

Registering the Lua script loads it from disk once and reuses it, which is faster than repeatedly loading and evaluating it for each invocation.

To test this, you’ll need a running Redis instance. You can run one with Docker:

docker run --name redis-server -d -p 6379:6379 redis

Now, running the script will print:

Notification sent: This is a single notification message

Since this sends a notification only once, the rate limiting isn’t apparent yet, but it’s working under the hood and will kick in if any limit is exceeded. To see it in action, you can attempt to send multiple notifications in a tight loop.

Testing the rate limiter #

You can call the send_notification function multiple times to test the rate limiter. Below is an example that simulates several notification requests in a short loop, giving you a sense of how many will be allowed versus blocked:

from redis import Redis
import time


def main() -> None:
    # Connect to Redis
    redis_client = Redis(host="localhost", port=6379)

    # Load the Lua script
    with open("rate_limiter.lua", "r") as file:
        lua_script = file.read()

    # Register the Lua script
    script = redis_client.register_script(lua_script)

    # Example keys and arguments
    global_key = "rate_limit:global"
    category_key = "rate_limit:category:errors"
    global_limit = 10
    category_limit = 3
    window = 60  # 1 minute in seconds for this test

    # Run the script in a loop
    for i in range(10):
        time.sleep(0.1)
        result = script(
            keys=[global_key, category_key],
            args=[global_limit, category_limit, window],
        )
        message = "Some notification message"
        if result == 1:
            # Allowed: send the notification
            print(f"{i}. Notification sent: {message}")
            # Add actual notification-sending logic here
        else:
            # Blocked: drop the notification
            print(f"{i}. Notification dropped (rate limited): {message}")


if __name__ == "__main__":
    main()

This code demonstrates how to test the rate limiter by simulating multiple notification requests. The Lua script is loaded, registered with Redis, and executed in a loop to evaluate whether each request is allowed or blocked based on the defined rate limits.

Running this will produce output similar to:

0. Notification sent: Some notification message
1. Notification sent: Some notification message
2. Notification sent: Some notification message
3. Notification dropped (rate limit exceeded): Some notification message
4. Notification dropped (rate limit exceeded): Some notification message
5. Notification dropped (rate limit exceeded): Some notification message
6. Notification dropped (rate limit exceeded): Some notification message
7. Notification dropped (rate limit exceeded): Some notification message
8. Notification dropped (rate limit exceeded): Some notification message
9. Notification dropped (rate limit exceeded): Some notification message

Here, for demonstration, we set the global rate limit to 10 and the category limit to 3 with a 60-second rolling window. After three successful category notifications (and a total of three global notifications), the rate limiter rejects additional requests in the same window, illustrating how both the global and category limits work together.

Dynamic shell variables

Redowan Delowar — Sat, 11 Jan 2025 00:00:00 +0000

I came across a weird shell syntax today - dynamic shell variables. It lets you dynamically construct and access variable names in Bash scripts, which I haven’t encountered in any of the mainstream languages I juggle for work.

In an actual programming language, you’d usually use a hashmap to achieve the same effect, but directly templating variable names is a quirky shell feature that sometimes comes in handy.

A primer #

Dynamic shell variables allow shell scripts to define and access variables based on runtime conditions. Variable indirection (${!var} syntax) lets you reference the value of a variable through another variable. This can be useful for managing environment-specific configurations and function dispatch mechanisms.

Here’s an example:

#!/usr/bin/env bash
# script.sh

config_path="/etc/config"
var="config_path"

echo "The value of \$config_path is: ${!var}"

The value of $config_path is: /etc/config

Here, ${!var} resolves to the value of the variable config_path because var contains its name. This allows you to dynamically decide which variable to reference at runtime.

Context-aware environment management #

A more practical use of dynamic shell variables is managing environment-specific configurations. This is particularly handy in scenarios where you have multiple environments like staging and prod, each with its own unique configuration settings.

#!/usr/bin/env bash
# script.sh

# Define environment-specific configurations dynamically
declare staging_URL="https://staging.example.com"
declare staging_PORT=8081

declare prod_URL="https://example.com"
declare prod_PORT=80

# Set the current environment
env=$1

# Validate input
if [[ "$env" != "staging" && "$env" != "prod" ]]; then
  echo "Invalid environment. Please specify 'staging' or 'prod'."
  exit 1
fi

# Dynamically access the environment-specific variables
URL="${env}_URL"
PORT="${env}_PORT"

echo "URL: ${!URL}"
echo "Port: ${!PORT}"

Run the script with an environment as the argument:

./script.sh staging

Output for env="staging":

URL: https://staging.example.com
Port: 8081

By passing the environment as an argument, you can switch between environments without duplicating configuration logic.

One gotcha to be aware of is that appending text directly to the ${!VAR} syntax (e.g., ${!env}_URL) doesn’t produce the intended results. Instead of resolving staging_URL, this line will print only _URL:

echo "${!env}_URL"

Output:

_URL

This happens because ${!VAR} only resolves the value of VAR and doesn’t support direct concatenation. To avoid this, construct the full variable name (URL="${env}_URL") before using ${!VAR} for indirect expansion. This ensures the correct variable is accessed.

Function dispatch #

Another neat use case for dynamic variables is function dispatch - calling the appropriate function based on runtime conditions. This technique can be used to simplify scripts that need to handle multiple services or operations.

#!/usr/bin/env bash
# script.sh

# Define functions for operations on different services

web_start() {
    echo "Starting web service..."
}

web_stop() {
    echo "Stopping web service..."
}

db_status() {
    echo "Checking database status..."
}

# Dynamically bind operation to function
declare web_start_function="web_start"
declare web_stop_function="web_stop"
declare db_status_function="db_status"

# Input variables for service and operation
service=$1
operation=$2

# Build dynamic function name
func="${service}_${operation}_function"

# Dispatch function dynamically
if [[ $(type -t ${!func}) == "function" ]]; then
    ${!func}  # Call the dynamically resolved function
else
    echo "Unknown operation: $service $operation"
fi

Run the script with service and operation as arguments:

./script.sh web start

This returns:

Starting web service...

Similarly, running ./script.sh db status prints:

Checking database status...

Temporary file handling #

Dynamic variables can also help manage temporary files or logs in scripts that process multiple datasets. By dynamically generating variable names, you can track temporary file paths for each dataset without conflicts.

#!/usr/bin/env bash
# script.sh

# Process multiple datasets with temporary files
for dataset in data1 data2 data3; do
    # Dynamically declare a temporary file variable
    temp_file_var="${dataset}_temp_file"
    declare $temp_file_var="/tmp/${dataset}_processing.tmp"

    # Simulate processing and logging
    echo "Processing $dataset..." > ${!temp_file_var}
    cat ${!temp_file_var}

    # Clean up (or add a trap to make this more robust)
    rm -f ${!temp_file_var}
done

Running this prints the following:

Processing data1...
Processing data2...
Processing data3...

Here, each dataset gets a unique temporary file, managed dynamically by the script. It eliminates the need for manually creating and tracking file names.

This works, but like everything else in shell scripts, it can quickly turn into a hairball if we’re not careful. While the syntax is nifty, I find it a bit hard to read at times!

Link blog in a static site

Redowan Delowar — Mon, 06 Jan 2025 00:00:00 +0000

One of my 2025 resolutions is doing things that don’t scale and doing them faster without overthinking. The idea is to focus on doing more while worrying less about scalability and sustainability in the things I do outside of work. With that in mind, I’ve been thinking for a while about tracking some of my out-of-band activities on this blog. The goal is to:

List the things I do, books and articles I read, and talks I grok.
Add some commentary or quote something I liked from the content, verbatim, for posterity.
Not spam people who just want to read the regular articles.
Not turn into a content junkie, churning out slop I wouldn’t want to read myself.

This isn’t about getting more eyeballs on what I publish. It’s about tracking what I do so I can look back at the end of the year and enjoy a nice little lull of accomplishment. Plus, having a place to post stuff regularly nudges me to read more, explore more, and do more of the things I actually want to do.

Social media is usually where people do this, but digging up old posts and reviewing them later is a pain. Plus, platforms like Twitter tank your posts’ visibility if they have links to other sites. A simple link blog solves all that for me.

Having a dynamic backend would make setting up a link blog trivial. But I’m too lazy to maintain one. Right now, this site is just a static Hugo build with some custom CSS flair. Adding a link blog without a dedicated backend is the tricky part.

If you don’t mind maintaining a dynamic site, Simon Willison has a great piece on running a link blog.

Here’s what has worked for me #

I went with a dead simple approach that has worked for me for the past 5 months:

Detail view: Have a single markdown page dedicated to each year and add the links there. See 2025 feed detail view.
List view: Cluster the detail pages in a list view named feed.

Detail view #

Each year gets its own detail page where I add links in reverse chronological order. The content has the following structure in markdown:

---
title: "2024"
layout: post
---

### December 26

#### [Reflecting on life – Armin Ronacher][32]

...

[32]: https://lucumr.pocoo.org/2024/12/26/reflecting-on-life/

---

### December 24

#### [How I write HTTP services in Go after 13 years – Mat Ryer][31]

...

[31]: https://grafana.com/blog/2024/02/09/how-i-write-http-services/

---

Hugo renders the detail view like this:

List view #

Hugo automatically creates a list view from the yearly pages.

To avoid spamming people, the entries in the link blog are filtered out of the main RSS feed. Also, the client-side search functionality allows me to look for a particular link without having to maintain any extra infra.

This solution is simple and has been working well for me. I started it in August last year, and the 2024 feed page already has enough entries to make me feel good when I reviewed them at the end of the year.

Running only a single instance of a process

Redowan Delowar — Tue, 31 Dec 2024 00:00:00 +0000

I’ve been having a ton of fun fiddling with Tailscale over the past few days. While setting it up on a server, I came across this ufw firewall script that configures the firewall on Linux to ensure direct communication across different nodes in my tailnet. It has the following block of code that I found interesting (added comments for clarity):

#!/usr/bin/env bash

# Define PID file path using script's name for uniqueness
PIDFILE="/tmp/$(basename "${BASH_SOURCE[0]%.*}.pid")"

# Open file descriptor 200 for the PID file
exec 200>"${PIDFILE}"

# Try to acquire a non-blocking lock; exit if the script is already running
flock -n 200 \
    || {
        echo "${BASH_SOURCE[0]} already running. Aborting..."; exit 1;
    }

# Store the current process ID (PID) in the lock file for reference
PID=$$
echo "${PID}" 1>&200

# Do work (in the original script, real work happens here)
sleep 999

Here, flock is a Linux command that ensures only one instance of the script runs at a time by locking a specified file (e.g., PIDFILE) through a file descriptor (e.g., 200). If another process already holds the lock, the script either waits or exits immediately. Above, it bails with an error message and exit code 1.

If you try running two instances of this script, the second one will exit with this message:

<script-name> script is already running. Aborting...

On most Linux distros, flock comes along with the coreutils. If not, it’s easy to install with your preferred package manager.

A more portable version #

On macOS, the file locking mechanism is different, and flock doesn’t work there. To make your script portable, you can use mkdir in the following manner to achieve a similar result:

#!/usr/bin/env bash

LOCKDIR="/tmp/$(basename "${BASH_SOURCE[0]%.*}.lock")"

# Try to create the lock directory
mkdir "${LOCKDIR}" 2>/dev/null || {
  echo "Another instance is running. Aborting..."
  exit 1
}

# Set up cleanup for the lock directory
trap "rmdir \"${LOCKDIR}\"" EXIT

# Main script logic
echo "Acquired lock, doing important stuff..."
# ... your script logic ...
sleep 999

This works because mkdir is atomic. It creates the lock directory (LOCKDIR) in /tmp or fails if the directory already exists. This acts as a marker for the running instance. If successful, the script sets up a trap to remove the directory on exit and continues to the main logic. If mkdir fails, it means another instance of the process is running, and the script exits with a message.

This is almost as effective as the flock version. Since I rarely write scripts for non-Linux environments, either option is fine!

With Python #

Oftentimes, I opt for Python when I need to write larger scripts. The same can be achieved in Python like this:

import fcntl
import os
import sys
import time

# Use the script name to generate a unique lock file
LOCKFILE = f"/tmp/{os.path.basename(__file__)}.lock"


def work() -> None:
    time.sleep(999)


if __name__ == "__main__":
    try:
        # Open a file and acquire an exclusive lock
        with open(LOCKFILE, "w") as lockfile:
            fcntl.flock(lockfile, fcntl.LOCK_EX | fcntl.LOCK_NB)
            print("Acquired lock, running script...")

            # Main script logic here
            work()

    except BlockingIOError:
        print("Another instance is running. Exiting.")
        sys.exit(1)

The script uses fcntl.flock to prevent multiple instances from running. It creates a lock file (LOCKFILE) in the /tmp directory, named after the scripts filename. When the script starts, it opens the file in write mode and tries to lock it with fcntl.flock using an exclusive lock (LOCK_EX). The LOCK_NB flag makes the operation non-blocking. If another process holds the lock, the script exits with a message.

This approach works on both Linux and macOS, as both support fcntl for file-based locks. The lock is automatically released when the file is closed, either at the end of the script or the with block.

With Go #

I was curious about doing it in Go. It’s quite similar to Python:

package main

import (
    "fmt"
    "os"
    "path/filepath"
    "syscall"
    "time"
)

// Use the script name (basename) to generate a unique lock file
var lockfile = fmt.Sprintf("/tmp/%s.lock", filepath.Base(os.Args[0]))

func work() {
    time.Sleep(999 * time.Second)
}

func main() {
    // Open the lock file
    file, err := os.OpenFile(lockfile, os.O_CREATE|os.O_RDWR, 0644)
    if err != nil {
        fmt.Println("Failed to open lock file:", err)
        os.Exit(1)
    }
    defer file.Close()

    // Try to acquire an exclusive lock
    err = syscall.Flock(int(file.Fd()), syscall.LOCK_EX|syscall.LOCK_NB)
    if err != nil {
        fmt.Println("Another instance is running. Exiting.")
        os.Exit(1)
    }

    // Release the lock on exit
    defer syscall.Flock(int(file.Fd()), syscall.LOCK_UN)

    fmt.Println("Acquired lock, running script...")

    // Main script logic
    work()
}

Like the Python example, this uses syscall.Flock to prevent multiple script instances. It creates a lock file based on the script’s name using filepath.Base(os.Args[0]) and stores it in /tmp. The script tries to acquire an exclusive, non-blocking lock (LOCK_EX | LOCK_NB). If unavailable, it exits with a message. The lock is automatically released when the file is closed in the defer block.

Underneath, Go makes sure that syscall.Flock works on both macOS and Linux.

SSH saga

Redowan Delowar — Tue, 17 Dec 2024 00:00:00 +0000

Setting up SSH access to a new VM usually follows the same routine: generate a key pair, copy it to the VM, tweak some configs, confirm the host’s identity, and maybe set up an agent to avoid typing passphrases all day. Tools like cloud-init and Ansible handle most of the setup for me now, so I rarely think about it. But I realized I don’t fully understand how all the parts work together.

This post attempts to give an overview of what happens when you type ssh user@host. It covers key pairs, authorized_keys, sshd_config, ~/.ssh/config, known_hosts, and how they all fit together.

A new VM in the void #

You’ve provisioned a new VM and need key-based SSH access. This involves generating a key pair on your local machine, installing the public key on the remote VM, and ensuring the SSH daemon (sshd) on the VM trusts it. Done right, ssh user@host drops you into a shell without a password prompt.

First, generate a key pair on your local machine:

ssh-keygen -t ed25519 -C "your_email@example.com"

This creates two files: a private key (~/.ssh/id_ed25519) and a public key (~/.ssh/id_ed25519.pub). The private key stays local. The public key is shared.

Your local fortress #

Your local private key proves your identity and must be protected (~/.ssh/id_ed25519). The public key (~/.ssh/id_ed25519.pub) gets copied to the VM.

To view the public key locally:

cat ~/.ssh/id_ed25519.pub
# Output: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIG... user@local

Copy this public key to the VM.

The remote gatekeeper #

On the VM, sshd listens for connections and authenticates users. Its configuration file, /etc/ssh/sshd_config, defines policies: whether password logins are allowed, which keys are trusted, and which crypto settings to use. A hardened snippet might look like this:

# /etc/ssh/sshd_config (on the VM)
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
AuthorizedKeysFile     .ssh/authorized_keys

With PasswordAuthentication no, only keys can unlock access.

Authorized keys and the handshake #

The ~/.ssh/authorized_keys file on the VM decides who gets access. Add your public key there to tell sshd that anyone holding the matching private key (you) can connect.

On the VM, under the user’s home directory:

mkdir -p ~/.ssh
chmod 700 ~/.ssh
echo "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIG... user@local" \
    >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

Now when you run ssh user@host, the server matches your key to one in authorized_keys and lets you in.

The client and its configs #

Your local SSH client can be configured via ~/.ssh/config to simplify hostnames, ports, and key paths:

# ~/.ssh/config (on your local machine)
Host myvm
    HostName 203.0.113.10
    User ubuntu
    IdentityFile ~/.ssh/id_ed25519
    Port 22

Now you can connect with:

ssh myvm

Known hosts and server identity #

When you connect to the VM for the first time, SSH prompts you to confirm its identity. Accepting it adds the server’s host key to ~/.ssh/known_hosts. SSH will detect any identity changes during subsequent connections:

203.0.113.10 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI...

This prevents man-in-the-middle attacks.

Agent and forwarding #

If your private key has a passphrase, typing it constantly is annoying. ssh-agent caches the key in memory, so you only unlock it once:

eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519
# Enter passphrase once

For hopping through multiple servers (local → bastion → internal server), enable agent forwarding:

# ~/.ssh/config (local)
Host myvm
    HostName 203.0.113.10
    User ubuntu
    IdentityFile ~/.ssh/id_ed25519
    ForwardAgent yes

Now, you can connect through intermediary hosts without copying private keys around.

Configuring and hardening the daemon #

On the VM, refine /etc/ssh/sshd_config to enforce stricter security:

# /etc/ssh/sshd_config (on the VM)
PasswordAuthentication no
PermitRootLogin prohibit-password
AllowUsers ubuntu
PubkeyAuthentication yes
KexAlgorithms curve25519-sha256@libssh.org
Ciphers aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
MACs hmac-sha2-512-etm@openssh.com

These settings ensure that only trusted keys with modern crypto algorithms can connect.

Bringing it all together #

Here’s a quick summary of setting up SSH connection to a new machine:

Generate a key pair on your local machine.
Copy the public key to the authorized_keys file on the VM.
Configure sshd_config on the VM to allow key-based authentication.
Set up ~/.ssh/config on your local machine to simplify SSH commands (e.g., ssh myvm instead of ssh <vm-ip>).
Confirm the server’s identity and save it to known_hosts on your local machine.
Use ssh-agent on your local machine to cache your private key and avoid typing the passphrase repeatedly.
Enable agent forwarding in your SSH config to connect through intermediary servers without copying keys.
Harden sshd_config on the VM to enforce modern crypto algorithms and stricter security policies.

┌──────────────────────┐
│      LOCAL           │
│  ~/.ssh/config       │
│  ~/.ssh/id_*         │
│  ~/.ssh/known_hosts  │
│  ssh-agent           │
└───▲───────┬──────────┘
    │       │
    │       │ SSH Connection (Port 22)
    │       │
    │       │
    │       ▼
┌────────────────────────┐
│     REMOTE             │
│ /etc/ssh/sshd_config   │
│ ~/.ssh/authorized_keys │
│ sshd daemon            │
└────────────────────────┘

Docker mount revisited

Redowan Delowar — Tue, 22 Oct 2024 00:00:00 +0000

I always get tripped up by Docker’s different mount types and their syntax, whether I’m stringing together some CLI commands or writing a docker-compose file. Docker’s docs cover these, but for me, the confusion often comes from how “bind” is used in various contexts and how “volume” and “bind” sometimes get mixed up in the documentation.

Here’s my attempt to disentangle some of my most-used mount commands.

Volume mounts #

Volume mounts let you store data outside the container in a location managed by Docker. The data persists even after the container stops. On non-Linux systems, volume mounts are faster than bind mounts because data doesn’t need to cross the virtualization boundary.

The `-v` option #

The -v flag is the older and more common way to define volume mounts in the Docker CLI. For example:

docker run -v myvolume:/usr/share/nginx/html:ro nginx

Here’s what each part means:

myvolume: The name of the Docker-managed volume on the host.
/usr/share/nginx/html: The mount point inside the container.
:ro: Mounts the volume as read-only inside the container. The host can still write to the volume, but the container cannot.

The general syntax is:

-v [SOURCE]:[TARGET]:[OPTIONS]

It can be tricky to remember which part is the host and which is the container, especially since with volumes, the SOURCE is a volume name, not a host path.

The `--mount` option #

To make things clearer, Docker introduced the --mount option, which uses key-value pairs. The same volume mount using --mount looks like this:

docker run \
    --mount \
    type=volume,source=myvolume,target=/usr/share/nginx/html,readonly nginx

Or using shorthands:

docker run \
    --mount type=volume,src=myvolume,dst=/usr/share/nginx/html,ro nginx

I find this syntax more explicit and less error-prone, even if it’s a bit more verbose.

In `docker-compose.yml` #

In docker-compose, volumes can be defined using both the old and new syntax. Here’s how they compare:

Old style

services:
  app:
    image: nginx
    volumes:
      - myvolume:/usr/share/nginx/html:ro
volumes:
  myvolume:

New style

services:
  app:
    image: nginx
    volumes:
      - type: volume
        source: myvolume
        target: /usr/share/nginx/html
        read_only: true
volumes:
  myvolume:

I prefer the new style because it reduces ambiguity and makes the configuration clearer.

Bind mounts #

Bind mounts let you directly mount a file or directory from the host into the container. This is especially useful in development when you want the container to have access to your code or data.

The key difference between volume mounts and bind mounts is that volumes are fully managed by Docker and stored in a special location, while bind mounts rely on specific host paths. Volumes are more portable and isolated from the host, whereas bind mounts give you direct access to host files but can introduce permission issues and depend on the exact file structure of the host.

The `-v` option #

Using the -v syntax for bind mounts:

docker run -v /path/on/host:/usr/share/nginx/html:ro nginx

Here:

/path/on/host: The path on the host machine. This must be an absolute path.
/usr/share/nginx/html: The mount point inside the container.
:ro: Mounts the directory as read-only inside the container.

The `--mount` option #

Using --mount for a bind mount:

docker run \
    --mount \
    type=bind,src=/path/on/host,dst=/usr/share/nginx/html,ro nginx

This syntax makes it clear that you’re using a bind mount and specifies exactly which paths are involved.

In `docker-compose.yml` #

In docker-compose, bind mounts can be specified like this:

Old style

services:
  app:
    image: nginx
    volumes:
      - ./path/on/host:/usr/share/nginx/html:ro

New style

services:
  app:
    image: nginx
    volumes:
      - type: bind
        source: ./path/on/host
        target: /usr/share/nginx/html
        read_only: true

Note: In docker-compose, if you specify a source that doesn’t start with / (an absolute path) or ./ (a relative path), Docker might think you’re referring to a volume. To ensure it’s interpreted as a bind mount, start the path with ./ or /.

Tmpfs mounts #

Tmpfs mounts store data in the host’s memory, not on disk. This makes them ideal for temporary storage that doesn’t need to persist after the container stops. They’re great for things like caches or scratch space.

The `--tmpfs` option #

Docker provides a --tmpfs option to create a tmpfs mount more concisely:

docker run --tmpfs /app/tmp:rw,size=64m nginx

/app/tmp: The target directory inside the container.
rw: This option allows read and write access to the tmpfs mount.
size=64m: Sets the size of the tmpfs mount to 64 MB.

The `--mount` option #

Alternatively, using the more flexible --mount option:

docker run \
    --mount type=tmpfs,target=/app/tmp,tmpfs-size=64m,tmpfs-mode=1777 nginx

Here’s what each part means:

type=tmpfs: Specifies that this is a tmpfs mount, using the host’s memory.
target=/app/tmp: The directory inside the container where the tmpfs mount is mounted.
tmpfs-size=64m: Limits the size of the tmpfs mount to 64 MB.
tmpfs-mode=1777: Sets permissions for the tmpfs mount (1777 grants read, write, and execute permissions to everyone).

In `docker-compose.yml` #

In docker-compose, tmpfs mounts can be defined using both the old and new syntax.

Old style

services:
  app:
    image: nginx
    tmpfs:
      - /app/tmp:size=64m

New style

services:
  app:
    image: nginx
    volumes:
      - type: tmpfs
        target: /app/tmp
        tmpfs:
          size: 64m
          mode: 1777

Build cache mounts #

Build cache mounts help speed up Docker image builds by caching intermediate files like package downloads or compiled artifacts. They’re used during the build process and aren’t part of the final container image.

In a Dockerfile, you might use a build cache like this:

RUN --mount=type=cache,target=/var/cache/apt \
    apt-get update && apt-get install -y curl

Here’s what each option does:

--mount=type=cache: Defines a cache mount that stores the files from the apt-get commands to speed up future builds by reusing the downloaded packages.
target=/var/cache/apt: Specifies the location inside the container where the cache will be stored during the build process.

In my Python projects, I cache my dependencies and install them with uv like this:

RUN --mount=type=cache,target=/root/.cache/uv \
    --mount=type=bind,source=uv.lock,target=uv.lock \
    --mount=type=bind,source=pyproject.toml,target=pyproject.toml \
    uv sync --no-install-project --locked --no-dev

Above, the cache mount at /root/.cache/uv saves dependencies so they don’t need to be re-downloaded in future builds if nothing changes. The bind mounts provide access to uv.lock and pyproject.toml from the host, allowing the container to read these config files during the build. Any changes to the files are picked up, while cached dependencies are reused unless the configurations have been updated.

Discovering direnv

Redowan Delowar — Wed, 02 Oct 2024 00:00:00 +0000

I’m not a big fan of shims - code that messes with commands in the shell or prompt. That’s why, aside from occasional dabbling, I tend to eschew tools like asdf or pyenv and just use apt or brew for installs, depending on the OS.

Then recently, I saw Hynek extolling direnv:

If you’re old-school like me, my .envrc looks like this:
uv sync --frozen
source .venv/bin/activate
The sync ensures there’s always a .venv, so no memory-baking required.

And Brandur doing the same:

This is embarrassing, but after using direnv for 10+ years, I only discovered the source_env directive yesterday.

Game changer. I used it to improve our project’s dev configuration ergonomics so new environment variables are easily distributed via Git.

So I got curious and wanted to try the tool to see if it fits into my workflow, or if I’ll quickly abandon it when something goes wrong.

When I first visited their landing page, I was a bit confused by the tagline:

direnv – unclutter your .profile

But I don’t have anything custom in my .profile, or more specifically, my .zprofile. Here’s what’s in it currently:

cat ~/.zprofile

eval "$(/opt/homebrew/bin/brew shellenv)"

# Added by OrbStack: command-line tools and integration
source ~/.orbstack/shell/init.zsh 2>/dev/null || :

Then I realized that .profile is used here as a general term for various configuration files like .*profile, .*rc, and .*env. I have quite a bit set up in both my ~/.zshrc and ~/.zshenv - a mix of global and project-specific commands and environment variables.

To explain: .*profile files (like .profile or .bash_profile) are used by login shells, which are started when you log into a system, such as through SSH or a terminal login. In contrast, files like .bashrc or .zshrc are for interactive shells, meaning they run when you open a new terminal window or tab. For Zsh, .zshenv is sourced by all types of shells - both login and interactive - making it useful for global environment settings.

What problem it solves #

Direnv solves the hassle of managing environment variables across different projects by automatically loading them when you enter a directory and unloading them when you leave. It keeps your global environment clean and avoids cluttering up your shell configuration files.

It checks for an .envrc (or .env) file in the current or parent directories before each prompt. If found and authorized, it loads the file into a bash sub-shell and applies the environment variables to the current shell.

It supports hooks for common shells like Bash, Zsh, Tcsh, and Fish, allowing you to manage project-specific environment variables without cluttering your ~/.profile. Since it’s a fast, single static executable, direnv runs seamlessly and is language-agnostic, meaning you can easily use it alongside tools like rbenv, pyenv, and phpenv.

You might argue that source .env works just fine, but it’s an extra step to remember. Also, being able to communicate the project-specific environment commands and variables, and having them sourced automatically, is a nice bonus.

Why .envrc file and not just a plain .env file #

This was the first question that came to my mind: why not just use a .env file? Why introduce another configuration file? Grokking the docs clarified things.

The .envrc file is treated like a shell script, where you can also list arbitrary shell commands that you want to be executed when you enter a project directory. You can’t do that with a plain .env file. However, direnv does support .env files too.

It’s such a simple idea that opens up many possibilities.

How I use it #

Here are a few things I’m using it for:

Automatically loading environment variables from a .env file.
Loading different sets of values for the same environment keys, e.g., local vs. staging values.
Activating the virtual environment when I enter the directory of a Python project.

Let’s say you want to load your environment variables automatically when you cd into a directory and have them removed from the shell environment when you leave it. Suppose the project directory looks like this:

svc/
├── .env
├── .env.staging
└── .envrc

The .env file contains environment variables for local development:

FOO="foo"
BAR="bar"

And the .env.staging file contains the variables for staging:

FOO="foo-staging"
BAR="bar-staging"

The .envrc file can have just one command to load the default .env file:

dotenv

Now, from the svc directory, you’ll need to allow direnv to load the environment variables into the current shell:

direnv allow

This prints:

direnv: loading ~/canvas/rednafi.com/svc/.envrc
direnv: export +BAR +FOO

You can now print the values of the environment variables like this:

echo "${FOO-default}"; echo "${BAR-default}"

This returns:

foo
bar

If you want to load different variables depending on the environment, you can add the following shell script to the .envrc file:

case "${ENVIRONMENT}" in
  "staging")
    if [[ -f ".env.staging" ]]; then
      dotenv .env.staging
    fi
    ;;
  *)
    if [[ -f ".env" ]]; then
      dotenv
    fi
    ;;
esac

The script loads the .env.staging file if the value of $ENVIRONMENT is staging; otherwise, it loads the default .env file. From the svc root, run:

direnv allow

This will still load the variables from .env. To load variables from .env.staging, run:

export ENVIRONMENT=staging && direnv allow

This time, printing the variables returns the staging values:

foo-staging
bar-staging

Oh, and when you leave the directory, the environment variables will be automatically unloaded from your working shell.

cd ..

direnv: unloading

You can do a lot more with the idea, but going overboard with environment variables can be risky. You don’t want to accidentally load something into the environment you didn’t intend to. Keeping it simple with sane defaults is the way to go.

Like Hynek, I’ve adopted uv in my Python workflow, and now my default .envrc has these two commands:

uv sync --frozen
source .venv/bin/activate

The first command updates the project’s environment without changing the uv.lock file, and the second ensures I never need to remember to activate the virtual environment before running commands. Now, when I cd into a Python project and run:

echo $VIRTUAL_ENV

It shows that the local .venv is active:

/Users/rednafi/canvas/rednafi.com/.venv

No more worrying about mucking up my global Python installation while running some commands.

Another neat directive is source_up, which lets you inherit environment variables from the parent directory. Normally, when you move into a child directory, direnv unloads the parent directory’s environment variables. But with the source_up directive in your .envrc, it’ll keep those variables around in the child directory.

Then there’s the source_env directive, which lets you pull one .envrc file into another. So, if you’ve got some common, non-secret variables in an .envrc.local file, you can easily reuse them in your .envrc.

Here’s an example .envrc.local file:

export API_URL="http://localhost:5222"
export DATABASE_URL="postgres://localhost:5432/project-db"

You can import .env.local into the .envrc file like this:

source_env .envrc.local

# Other commands and variables go here

I haven’t used source_env much yet, but I love the possibilities it unlocks.

The biggest reason I’ve adopted it everywhere is that it lets me share my shell environment variables and the magic commands without having anything stashed away in my ~/.zshrc or ~/.zshenv, so there’s no need for out-of-band communication.

Notes on building event-driven systems

Redowan Delowar — Sat, 21 Sep 2024 00:00:00 +0000

I spent the evening watching this incredibly grokkable talk on event-driven services by James Eastham at NDC London 2024. Below is a cleaned-up version of my notes.

I highly recommend watching the full talk if you’re interested before reading this distillation.

The curse of tightly coupled microservices #

Microservices often start with HTTP-based request-response communication, which seems straightforward but quickly becomes a pain as systems grow. Coupling - where one service depends on another - creates a few issues. Take the order processing service in a fictional Plant-Based Pizza company. It has to talk to the pickup service, delivery service, kitchen, and loyalty point service. They’re all tied together, so if one fails, the whole system could go down.

The system relies on all services being up at the same time, which causes issues when any service crashes. Even something like loyalty points can take the whole thing offline, making you wonder if the order processing service really needs to care about that.

On top of that, there’s semantic coupling - things like data formats. “How do you handle null values in strings? What casing is your JSON using - camelCase?” These details might seem minor, but in tightly coupled systems, they pile up, making the integrations fragile and complicated.

What event-driven architecture solves #

Event-driven architecture offers a way to decouple services. Instead of one service needing to communicate directly with another, services react to events, giving more flexibility and scalability. “Event-driven architecture is about reversing dependencies - reversing the lines of integration.” Now, the order processing service doesn’t need to know where the downstream services are. It simply publishes an event, and the downstream services react to it.

This shift is powerful because it frees services from having to know about each other. In this model, the kitchen doesn’t wait for the order processing service to send a direct HTTP request. Instead, it listens for an event that triggers its response without direct integration. “You’ve removed that runtime coupling because producers and consumers no longer need to know each other exist.” By decoupling systems, event-driven architecture improves fault tolerance, scalability, and flexibility.

The nature of an event #

At the core of event-driven systems is the concept of an event. An event is “an immutable fact. It’s something that’s happened in the past. It cannot be changed.” When an event is published, it’s a record of something that has already occurred, like “order confirmed” or “pizza boxed.” Events are simple, factual, and unchangeable.

The analogy of a light switch brings this to life: “You hit the light switch, and that raises a light switched-on event. You can’t un-switch on a light.” To turn the light off, you generate a new event - “light switched-off” - but you don’t undo the original. This principle of immutability ensures that events in the system are reliable and unambiguous, forming the foundation for how systems react.

Event-driven vs. event-based systems #

It’s easy to confuse event-driven systems with event-based systems, but the distinction is crucial. Event-driven systems are driven by business-specific events - things that reflect real-world actions and decisions, not just technical events like a button click. “An event-driven system uses events like these: ‘order confirmed,’ ‘pizza boxed,’ ‘staff member clocked in.’” These are business-level events that reflect the narrative of the company, not just low-level system changes.

In contrast, event-based systems simply react to any change, such as a file being added to S3 or a button being clicked in a UI. “We’ve been building event-based systems for years… that doesn’t make it event-driven.” The difference is significant because event-driven systems align technical architecture with business needs, creating a more meaningful, coherent system where the events reflect the organization’s core processes.

Fat vs. sparse events #

One of the critical design decisions in event-driven systems is choosing between fat events (which carry a lot of data) and sparse events (which carry minimal data). Fat events, also known as Event-Carried State Transfer (ECST), include all the information a consumer might need. For instance, “the kitchen can consume this event - it’s got the list of items on the order, so now it knows what it needs to cook.” This reduces the need for callbacks or additional requests for data back to the original system that publishes the event, making the system more robust in terms of runtime interaction.

However, fat events come with risks. “The downside of that is that you get more coupling at the schema level.” Because fat events contain so much information, it becomes harder to change the event format without impacting multiple consumers. As more services depend on that data, the risk of breaking something grows.

In contrast, sparse events are lightweight but require callbacks to get additional information. Initially, this might seem more efficient, but as more services join the system, the number of callbacks increases exponentially. “Now you’ve got this potentially infinite number of downstream services that are all making calls back to get more information.” The result is a more tightly coupled system, albeit in a different form.

So which one of these is the right one to choose? The answer is: it depends. Sparse events reduce the need for frequent changes, but fat events reduce the need for constant back-and-forth communication. Often, a combination of both is necessary, depending on the use case.

The core of event-driven architecture lies in the publish-subscribe pattern, facilitated by an event broker. “At its core, an event-driven architecture is made up of three parts: You have a producer, a consumer, and some kind of event broker in the middle.” The producer generates the event, the broker routes it, and the consumer processes it. The beauty of this system is that producers and consumers don’t need to know about each other’s existence.

“The first thing you’ll notice is that the producer and the consumer here have no idea each other exists - the communication is managed by the broker.” This decoupling makes the system more flexible and scalable. A consumer can be added or removed without impacting the producer. The broker ensures that events are delivered, allowing the system to continue functioning smoothly even as it evolves.

However, one responsibility remains: “The schema of your event - the format of that event - is the biggest part of the coupling that you will see in event-driven architecture.” While runtime coupling is removed, semantic coupling still exists. Producers must ensure that the event schema doesn’t change in ways that break existing consumers.

Handling constraints and governance #

In event-driven systems, the responsibility for handling constraints shifts from the producer to the consumer. Producers generate events as quickly as they can, without worrying about the load on consumers. “As a producer, it’s not your responsibility to care about how your events are used… that’s the subscriber’s responsibility.” Consumers must handle their own ingestion rates and ensure they don’t get overloaded.

Governance plays a critical role in managing these systems, particularly as they evolve. When changes are made to event schemas, it’s essential to communicate those changes to all consumers. “Governance is really important with event-driven architecture because you’ve got these systems that just don’t care about each other.” One effective method for managing this is through Request for Comments (RFCs), which allow for collaborative discussion before any changes are implemented.

“Rather than just publishing an event and hoping for the best, introducing governance ensures that events remain consistent and understandable across teams.” This helps prevent breaking changes that could take down systems you didn’t even know were relying on your events.

Metadata-data pattern for evolvability #

To enhance the evolvability of an event-driven system, East recommends using the metadata-data pattern. This pattern separates the event’s core data from its metadata, allowing for greater flexibility. “Splitting your event down into a metadata section and the data section helps you to stay evolvable.” The data contains the specifics of the event, while the metadata includes information like “event type,” “event ID,” and “version.”

This separation allows consumers to understand and process events more easily while providing room for schema changes. For example, “event versioning allows you to introduce breaking changes in a controlled manner.” By publishing multiple versions of an event, you can ensure backward compatibility while encouraging consumers to upgrade to the latest schema.

Eventual consistency in event-driven systems #

One of the trade-offs in event-driven architecture is that systems must embrace eventual consistency. In a request-response system, actions happen immediately and are reflected in real-time. But in an event-driven system, updates propagate over time. “Eventually, over time, these systems will converge on the same view of the world.” This is a shift in mindset for many developers used to strong consistency.

To illustrate this, consider a card payment: “When you make a card transaction, all you’re doing is making a theoretical guarantee that, at some point in the future, that money is going to move from your bank account to theirs.” While the system is eventually consistent, the end result will be correct, just not immediately. Event-driven architecture functions similarly - updates happen asynchronously, and systems eventually reach a consistent state.

Handling HTTP communication in an event-driven world #

Not every system can fully adopt event-driven architecture, and many still rely on HTTP-based communication. To integrate these systems into an event-driven world, you need a middle layer. For example, if your loyalty point service is being replaced by a third-party SaaS product that only supports an HTTP API, you’d still have a service managing that integration. This service listens for events and translates them into HTTP requests for systems that aren’t event-driven.

To handle differences in response times and reliability between HTTP-based and event-driven systems, introducing a queue or intermediary storage is crucial. “Introducing this queue means you can keep this amount of durability… you can process to the third-party API as and when you need to.” This queue adds resilience, allowing your system to continue functioning smoothly, even when interacting with external services that don’t follow event-driven principles.

Asynchronous commands #

Commands in an event-driven system don’t always need to be synchronous. Instead of waiting for an immediate response, systems can issue commands asynchronously, allowing for greater flexibility and non-blocking workflows. “You want to send an email, but you might not necessarily want it to be completely request-response.”

An asynchronous command might still send a request to a service, but the response isn’t required to continue processing. This allows systems like the email notification service to handle requests at its own pace, rather than blocking the core order processing service. “Your email service can still expose an endpoint, but as opposed to that being an HTTP-based endpoint, that could just be a message channel.” This approach decouples the services further and ensures more efficient use of resources.

CQRS for separating reads and writes #

Command Query Responsibility Segregation (CQRS) is a powerful pattern that pairs well with event-driven architecture. CQRS separates the system into two parts: one for handling commands (writes) and another for handling queries (reads). “In CQRS, you split your system into two completely independent services - one for processing commands, one for handling queries.” This allows each part of the system to be optimized for its specific workload.

For example, the command service focuses on writing data to the database and publishing events, while the query service listens for those events and updates a read-optimized view of the world. This separation enables more efficient scaling, as the query service can be tuned for fast reads, potentially storing data in caches like Redis or even keeping it in memory.

I’m a bit skeptical about CQRS since I’ve worked on a system with a terrible implementation that went horribly wrong. But I intend to keep an open mind.

Handling failure with the outbox pattern #

In event-driven systems, failure is inevitable, so you need strategies to handle situations where events fail to publish. The outbox pattern is one such approach. “At the point you write the data to the main database… you also write the data to a secondary table.” This outbox table ensures that if the event fails to publish initially, it can be retried later.

This creates consistency across the system by acting as a buffer between the database and the event bus. Alternatively, systems can use change data capture to respond directly to changes in the database. “As a record is written to the database, you can stream that - you can react to that - and you can publish events off the back of that.” Both methods ensure reliability, preventing events from being lost due to temporary failures.

The outbox pattern sounds great in theory, but in practice, if you have a large system with many services publishing to the broker, managing an extra process for each service to read from the outbox table and publish to the event bus becomes a hassle. Instead, on the publisher side, retrying with a circuit breaker has worked better for me. Also, fun fact: I was asked about the outbox pattern in 4 of the last 5 places I interviewed for a backend role.

Bash namerefs for dynamic variable referencing

Redowan Delowar — Fri, 20 Sep 2024 00:00:00 +0000

While going through a script at work today, I came across Bash’s nameref feature. It uses declare -n ref="$1" to set up a variable that allows you to reference another variable by name - kind of like pass-by-reference in C. I’m pretty sure I’ve seen it before, but I probably just skimmed over it.

As I dug into the man pages, I realized there’s a gap in my understanding of how variable references actually work in Bash - probably because I never gave it proper attention and just got by cobbling together scripts.

Namerefs #

By default, Bash variables are global unless declared as local within a function. However, when you pass variables as arguments to a function, they are accessed via positional parameters like $1, $2, etc., and any changes to these parameters inside the function do not affect the original variables outside the function.

Namerefs allow you to essentially define a pointer to another variable. By creating a nameref, you can indirectly reference and manipulate the target variable without knowing its name beforehand. This is incredibly useful for writing generic functions that can operate on different variables based on input parameters.

Basic usage #

Here’s an example:

#!/usr/bin/env bash

# Declare a variable
original_var="Hello, World!"

# Function that creates a nameref to a variable
create_ref() {
    local ref_name=$1
    declare -n ref="$ref_name"
    ref="Hello from nameref!"
}

# Call the function with the name of the variable
create_ref original_var

# Print the updated variable
echo "$original_var"

Running this will print:

Hello from nameref!

By running the create_ref function, we can dynamically update the value of $original_var, which exists outside of it. Notice that the function doesn’t even need to know about $original_var; it works on any variable name provided, making it generic.

In this script:

We declare a variable original_var with the value "Hello, World!".
The create_ref function takes the name of a variable as an argument.
Inside the function, declare -n ref="$ref_name" creates a nameref ref that points to the variable named by $ref_name.
By setting ref="Hello from nameref!", we indirectly update original_var.
Finally, we print original_var to see the updated value.

Without the nameref, you could achieve the same thing with this eval (read: evil) trick:

#!/usr/bin/env bash

# Declare a variable
original_var="Hello, World!"

# Function that updates a variable dynamically using eval
create_ref() {
    local var_name=$1
    local new_value="Hello from eval!"
    eval "$var_name=\"$new_value\""  # eval 😈
}

# Call the function with the name of the variable
create_ref original_var

# Print the updated variable
echo "$original_var"

This achieves the same result. The eval "$var_name=\"$new_value\"" dynamically updates the $original_var variable through $var_name. However, eval can be risky for security, and the nameref approach looks much cleaner syntactically.

Managing multiple arrays #

Namerefs shine when you need to manage multiple arrays dynamically. Consider a scenario where you have several datasets stored in different arrays, and you want to process them using a single function.

#!/usr/bin/env bash

# Declare multiple arrays
declare -a dataset1=(1 2 3 4 5)
declare -a dataset2=(10 20 30 40 50)
declare -a dataset3=(100 200 300 400 500)

# Function to calculate the sum of an array
sum_array() {
    local array_name=$1
    declare -n arr="$array_name"
    local sum=0
    for num in "${arr[@]}"; do
        sum=$((sum + num))
    done
    echo "Sum of $array_name: $sum"
}

# Process each dataset
sum_array dataset1
sum_array dataset2
sum_array dataset3

This returns:

Sum of dataset1: 15
Sum of dataset2: 150
Sum of dataset3: 1500

Here:

We declare three arrays: dataset1, dataset2, and dataset3.
The sum_array function takes the name of an array as an argument.
Using declare -n arr="$array_name", we create a nameref arr that points to the specified array.
We then iterate over the elements of arr to calculate the sum.
Finally, we call sum_array for each dataset, and the function correctly processes each array based on the reference.

Without the nameref, you could again use the eval trick to achieve the same thing, but this time it looks even uglier:

#!/usr/bin/env bash

# Declare multiple arrays
declare -a dataset1=(1 2 3 4 5)
declare -a dataset2=(10 20 30 40 50)
declare -a dataset3=(100 200 300 400 500)

# Function to calculate the sum of an array without namerefs
sum_array() {
    local array_name=$1
    local sum=0
    local index=0
    local array_length
    eval "array_length=\${#$array_name[@]}"
    for (( index=0; index<array_length; index++ )); do
        eval "element=\${$array_name[$index]}"
        sum=$((sum + element))
    done
    echo "Sum of $array_name: $sum"
}

# Process each dataset
sum_array dataset1
sum_array dataset2
sum_array dataset3

This approach is more complex, less secure, and harder to read in general. But the above eval example was a bit contrived to make it look bad. You can achieve the same thing without eval or nameref in this particular case like this:

#!/usr/bin/env bash

# Declare multiple arrays
dataset1=(1 2 3 4 5)
dataset2=(10 20 30 40 50)
dataset3=(100 200 300 400 500)

# Function to calculate the sum of an array
sum_array() {
    local sum=0
    for element in "$@"; do
        sum=$((sum + element))
    done
    echo "Sum: $sum"
}

# Process each dataset
sum_array "${dataset1[@]}"
sum_array "${dataset2[@]}"
sum_array "${dataset3[@]}"

Here, instead of passing the name of the dataset arrays as strings, we pass the elements of the array to the function and add them. But I digress!

Associative arrays and nested references #

Namerefs also work with associative arrays and can be used for more complex data structures.

#!/usr/bin/env bash

# Declare an associative array
declare -A user_info=(
    [name]="Alice"
    [age]=30
    [email]="alice@example.com"
)

# Function to update user information
update_info() {
    local info_name=$1
    local key=$2
    local new_value=$3
    declare -n info="$info_name"
    info[$key]=$new_value
}

# Update the user's email
update_info user_info email "alice@newdomain.com"

# Print updated information
for key in "${!user_info[@]}"; do
    echo "$key: ${user_info[$key]}"
done

It prints:

name: Alice
age: 30
email: alice@newdomain.com

And voilà! We have a function that can dynamically update the values in an associative array. This technique is useful for changing environments or contexts (staging/production) in shell scripts.

In this example:

We declare an associative array user_info containing user details.
The update_info function takes the name of the associative array, the key to update, and the new value.
Using declare -n info="$info_name", we create a nameref info pointing to user_info.
We update the specified key in the array.
Finally, we echo the updated user information.

Doing this with eval isn’t pretty. I’ll leave that as an exercise for you if you like to torment yourself.

Implementing generic setter and getter functions #

Building on the earlier examples, you can use namerefs to create generic setter and getter functions, making it easier to manage configuration variables or environment settings in scripts.

Here’s an example:

#!/usr/bin/env bash

# Generic setter function
set_var() {
    local var_name="$1"
    local value="$2"
    declare -n ref="$var_name"
    ref="$value"
}

# Generic getter function
get_var() {
    local var_name="$1"
    declare -n ref="$var_name"
    echo "$ref"
}

# Usage example

env="staging"  # Can be passed as an argument to the script

# Define default variables
db_host="localhost"
db_port=5432
db_user="admin"
db_pass="secret"

# Set different values based on the environment
if [[ "$env" == "production" ]]; then
  set_var "db_host" "prod.db.example.com"
  set_var "db_user" "prod_admin"
elif [[ "$env" == "staging" ]]; then
  set_var "db_host" "staging.db.example.com"
  set_var "db_user" "staging_admin"
fi

# Retrieve and display values
echo "Using Database: $(get_var "db_host")"
echo "Database User: $(get_var "db_user")"

To keep things simple, the env variable isn’t a CLI argument. Based on whether env is set to staging or production, the script will print the relevant database values.

For staging, you’ll see:

Using Database: staging.db.example.com
Database User: staging_admin

For production:

Using Database: prod.db.example.com
Database User: prod_admin

Oh, one extra thing: nameref was introduced in Bash 4.3, so you might run into problems if you’re using an ancient version like the one shipped with macOS.

Behind the blog

Redowan Delowar — Sat, 14 Sep 2024 00:00:00 +0000

When I started writing here about five years ago, I made a promise to myself that I wouldn’t give in to the trend of starting a blog, adding one overly enthusiastic entry about the stack behind it, and then vanishing into the ether.

I was somewhat successful at that and wanted to write something I can link to when people are curious about the machinery that drives this site. The good thing is that the tech stack is simple and has remained stable over the years since I’ve only made changes when absolutely necessary.

Markdown #

I write plain Markdown files in my editor of choice, which has been VSCode since its launch. Once I’m finished, make format runs the local Markdown cleanup, including prettier, to fix line length and formatting.

Hugo #

Hugo is the static site generator that turns the Markdown files into HTML. I chose it because I needed something that can build the site quickly, even with lots of content. It lets me hot reload the server and check my changes as I write. Plus, I don’t get to write Go at work, so messing with Hugo templates or its source code gives me a reason to play around with Go.

I initially tried some JS-based SSGs but dropped them pretty quickly because I couldn’t keep up with the constant tooling changes in the JavaScript universe. I use a handrolled theme and have tweaked the CSS over time.

GitHub Issues #

I use GitHub Issues to brainstorm ideas and keep track of my writing. I usually gather ideas throughout the week, log them in Issues, and then write something over the weekend. This workflow is heavily inspired by Simon Willison’s blog on his workflow.

GitHub Actions and GitHub Pages #

Once I push content to the main branch, GitHub Actions automatically runs the checks, builds the site, and deploys it to GitHub Pages. There’s nothing to maintain, and I don’t have to worry about scaling, even if one of my posts hits the front page of Hacker News. Aside from the domain, this site costs me nothing to run, and I plan to keep it that way.

Cloudflare Cache and R2 #

I’m a huge fan of Cloudflare and often try to shoehorn their offerings into my projects. Since my domain is registered with them, setting up their proxy with my domain’s DNS and turning on caching took just a few minutes. Their caching layer absorbs most of the traffic, and less than 10% of the requests hit the origin server. Plus, having the proxy layer gives me access to more accurate analytics.

Static assets like images, CSS, JS, and other files are stored on Cloudflare R2. I used to host my images with GitHub Issues and serve CSS and JS from the origin, but I recently switched everything to R2. Now I can manage it all from one place without worrying about costs. Their free plan is super generous - there’s no egress bandwidth fee, and because of caching, I barely use any of the quota. It’s fantastic!

Oxipng #

Oxipng is used to losslessly compress PNG screenshots before uploading them to the Cloudflare R2 bucket with the wrangler CLI. Images are immutable: each uploaded object gets a content hash in the filename and long-lived cache headers, so Cloudflare can serve repeat reads from cache without revalidating R2.

make upload-post-image \
    post=content/go/request_coalescing.md \
    file=/tmp/singleflight-flow.png \
    name=singleflight-flow

This command derives the R2 directory from the post, normalizes names to kebab-case, runs oxipng -o max --strip safe for PNGs, uploads with Cache-Control: public, max-age=31536000, immutable, and prints the canonical URL. Most article images are PNG screenshots, so the default path is screenshot to PNG, lossless max compression, upload, paste the printed URL into Markdown.

Standard.site records use the same R2-hosted site cover. CI downloads and verifies it before publishing, then commits any generated metadata changes.

Google Analytics #

I’m still using Google Analytics, even though I’m not a huge fan. Cloudflare already gives me better traffic insights, but the free version doesn’t show how many hits each page gets. At some point, I might just pay for Cloudflare’s upgraded plan so I can get rid of the bulky, intrusive analytics scripts for good.

The source code and content for this site are all publicly available on GitHub.

Shell redirection syntax soup

Redowan Delowar — Thu, 12 Sep 2024 00:00:00 +0000

I always struggle with the syntax for redirecting multiple streams to another command or a file. LLMs do help, but beyond the most obvious cases, it takes a few prompts to get the syntax right. When I know exactly what I’m after, scanning a quick post is much faster than wrestling with a non-deterministic kraken. So, here’s a list of the redirection and piping syntax I use the most, with real examples.

Redirecting stdout and stderr #

Redirect stdout to a file #

Standard way:
```
command > file
```
This replaces the content of file with the stdout of command. For example:
```
echo "Hello, world!" > hello.txt
```
Print and redirect to file:
```
command | tee file
```
Example:
```
echo "Hello, world!" | tee hello.txt
```
This prints “Hello, world!” to the terminal and also writes it to hello.txt.

Redirect stderr to a file #

Standard way:
```
command 2> file
```
Sends all errors (stderr) to file. For example:
```
ls non_existing_file 2> error.log
```

Print and redirect stderr to file:

command 2> >(tee file)

Example:

ls non_existing_file 2> >(tee error.log)

Redirect both stdout and stderr to a file #

Common approach:
```
command > file 2>&1
```
Combines stdout and stderr into one stream and saves them to file. For example:
```
ls non_existing_file existing_file > output.log 2>&1
```

Print and redirect both to file:

command 2>&1 | tee file

Example:

ls non_existing_file existing_file 2>&1 | tee output.log

Convenient shorthand:

command &> file

Example:

ls non_existing_file existing_file &> output.log

Append instead of overwriting #

Append stdout to a file:

command >> file

Example:

echo "Appending line" >> hello.txt

Print and append stdout to file:

command | tee -a file

Example:

echo "Appending line" | tee -a hello.txt

Append both stdout and stderr (explicit):

command >> file 2>&1

Example:

ls non_existing_file existing_file >> output.log 2>&1

Print and append both stdout and stderr to file:

command 2>&1 | tee -a file

Example:

ls non_existing_file existing_file 2>&1 | tee -a output.log

Convenient shorthand for appending both:

command &>> file

Example:

ls non_existing_file existing_file &>> output.log

Piping output #

Pipe stdout to another command #

Basic usage:
```
command1 | command2
```
This sends the stdout of command1 to the input of command2. For example:
```
echo "Hello, world!" | grep "Hello"
```

Print and redirect piped stdout to file:

command1 | tee file | command2

Example:

echo "Hello, world!" | tee output.txt | grep "Hello"

Pipe both stdout and stderr #

Common way:
```
command1 2>&1 | command2
```
Combines stdout and stderr, then pipes the combined stream to command2. For example:
```
ls non_existing_file existing_file 2>&1 | grep "No"
```

Print and redirect both stdout and stderr to file:

command1 2>&1 | tee file | command2

Example:

ls non_existing_file existing_file 2>&1 | tee output.txt | grep "No"

Shorthand for piping both stdout and stderr (`|&`) #

Shorthand syntax:
```
command1 |& command2
```
This is equivalent to command1 2>&1 | command2, combining stdout and stderr. For example:
```
ls non_existing_file existing_file |& grep "No"
```

Print and redirect both stdout and stderr using |&:

command1 |& tee file | command2

Example:

ls non_existing_file existing_file |& tee output.txt | grep "No"

Redirecting file descriptors #

Custom file descriptors #

Create a new file descriptor (e.g., 3) and redirect stdout to it:
```
exec 3> outputfile
command >&3
```
This sends the stdout of command to file descriptor 3, which points to outputfile. For example:
```
exec 3> custom_output.txt
echo "Using FD 3" >&3
```
Print and redirect stdout to custom file descriptor:
```
exec 3> custom_output.txt
echo "Using FD 3" | tee /dev/tty > /dev/fd/3
```
This prints “Using FD 3” to the terminal and simultaneously writes it to custom_output.txt.

Redirect stderr to a file descriptor #

Common case:

command 2>&3

Redirects stderr to file descriptor 3. For example:

exec 3> error_output.txt
ls non_existing_file 2>&3

Print and redirect stderr to custom file descriptor:

command 2> >(tee >(cat > /dev/fd/3))

Example:

ls non_existing_file 2> >(tee >(cat > /dev/fd/3))

Redirect both stdout and stderr to a file descriptor #

Common way:
```
command > /dev/fd/3 2>&1
```
Combines stdout and stderr, and redirects them to file descriptor 3.

Note: There’s no shorthand equivalent for redirecting both stdout and stderr to a file descriptor. You need to use the full syntax. For example:
```
exec 3> combined_output.txt
ls non_existing_file existing_file > /dev/fd/3 2>&1
```

Discarding output #

Send stdout and stderr to /dev/null #

Common:

command > /dev/null 2>&1

Silences all output (stdout and stderr). For example:

ls non_existing_file > /dev/null 2>&1

Print and discard stdout and stderr (not sure why you’d ever need this):
```
command | tee /dev/null
```
Example:
```
ls non_existing_file | tee /dev/null
```

Convenient shorthand:

command &>/dev/null

Example:

ls non_existing_file &>/dev/null

At a glance #

Redirect stdout: command > file
Redirect stderr: command 2> file
Redirect both stdout and stderr:
- Standard: command > file 2>&1
- Shorthand: command &> file
Append stdout: command >> file
Append both stdout and stderr:
- Standard: command >> file 2>&1
- Shorthand: command &>> file
Pipe stdout: command1 | command2
Pipe both stdout and stderr:
- Standard: command1 2>&1 | command2
- Shorthand: command1 |& command2
Custom file descriptors:
- Create and redirect stdout: exec 3> file; command >&3
- Redirect stderr: command 2>&3
- Redirect both stdout and stderr: command > /dev/fd/3 2>&1 (no shorthand available)
Discard stdout and stderr:
- Standard: command > /dev/null 2>&1
- Shorthand: command &>/dev/null

HTTP requests via /dev/tcp

Redowan Delowar — Thu, 08 Aug 2024 00:00:00 +0000

I learned this neat Bash trick today where you can make a raw HTTP request using the /dev/tcp file descriptor without using tools like curl or wget. This came in handy while writing a health check script that needed to make a TCP request to a service.

The following script opens a TCP connection and makes a simple GET request to example.com:

#!/bin/bash

# Open TCP connection to example.com:80 and assign file descriptor 3
# exec keeps /dev/fd/3 open; 3<> enables bidirectional read-write
exec 3<>/dev/tcp/example.com/80

# Send the HTTP GET request to the server (>& redirects to /dev/fd/3)
echo -e \
    "GET / HTTP/1.1\r\nHost: example.com\r\nConnection: close\r\n\r\n" >&3

# Read and print the server's response
# <& redirects the output of /dev/fd/3 to cat
cat <&3

# Close the file descriptor, terminating the TCP connection
exec 3>&-

Running this will print the response from the site to your console.

The snippet first opens a TCP connection to example.com on port 80 and assigns file descriptor 3 to manage this connection. The exec ensures that the file descriptor 3 remains open for the duration of the script, allowing multiple read and write operations without needing to reopen the connection each time. Using a file descriptor makes the code cleaner. Without it, we’d need to redirect input and output directly to /dev/tcp/example.com/80 for each read and write operation, making the script more cumbersome and harder to read.

Then we send an HTTP GET request to the server by echoing the request to file descriptor 3. The server’s response is read and printed using cat <&3, which reads from the file descriptor and prints the output to the console. Finally, the script closes the connection by terminating file descriptor 3 with exec 3>&-.

This is a Bash-specific trick and won’t work in other shells like Zsh or Fish. It also allows you to open UDP connections in the same manner. The Bash manpage explains the usage like this:

/dev/tcp/host/port
    If host is a valid hostname or Internet address, and port
    is an integer port number or service name, bash attempts
    to open the corresponding TCP socket.

/dev/udp/host/port
    If host is a valid hostname or Internet address, and port
    is an integer port number or service name, bash attempts
    to open the corresponding UDP socket.

I used this to write the following health check script. I didn’t want to install curl in a sidecar container that just runs a single health check process, keeping things simpler.

#!/bin/bash

# Enable bash strict mode
set -euo pipefail

# Constants
readonly HOST="example.com"
readonly PORT=80
readonly HEALTH_PATH="/"

# Open a TCP connection to the specified host and port
exec 3<>"/dev/tcp/${HOST}/${PORT}"

# Send the HTTP GET request to the server
echo -e \
    "GET ${HEALTH_PATH} HTTP/1.1\r\n"\
"Host: ${HOST}\r\nConnection: close\r\n\r\n" >&3

# Read the HTTP status from the server's response
read -r HTTP_RESPONSE <&3
HTTP_STATUS=$(
    echo "${HTTP_RESPONSE}" \
    | grep -o "HTTP/1.1 [0-9]*" \
    | cut -d ' ' -f 2
)

if [[ "${HTTP_STATUS}" == "200" ]]; then
    echo "Service is healthy."
    exit 0
else
    echo "Service is not healthy. HTTP status: ${HTTP_STATUS}"
    exit 1
fi

# Close the file descriptor, terminating the TCP connection
exec 3>&-

The script makes a GET request to the service and checks that the HTTP status from the raw response is 200. If not, it exits with a non-zero status.

Note that the script will fail if your service returns a 301 redirect code. Plus, you need to make raw textual HTTP requests, which can become cumbersome if you need to do anything beyond a simple GET call. At that point, you’re better off using curl.

The *nix install command

Redowan Delowar — Sun, 28 Jul 2024 00:00:00 +0000

TIL about the install command on *nix systems. A quick GitHub search for the term brought up a ton of matches. I’m surprised I just found out about it now.

Often, in shell scripts I need to:

Create a directory hierarchy
Copy a config or binary file to the new directory
Set permissions on the file

It usually looks like this:

# Create directory hierarchy (-p creates parent directories)
mkdir -p ~/.config/app

# Copy current config to the newly created directory
cp conf ~/.config/app/conf

# Set the file permission
chmod 755 ~/.config/app/conf

Turns out, the install command in GNU coreutils can do all that in one line:

install -D -m 755 conf ~/.config/app/conf

You can check the file status with:

stat ~/.config/app/conf

On my machine, this prints:

 File: /Users/rednafi/.config/app
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: 1,16    Inode: 16439606    Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (  501/ rednafi)   Gid: (   20/   staff)
Access: 2024-07-28 20:51:42.793765043 +0200
Modify: 2024-07-28 20:51:42.793765043 +0200
Change: 2024-07-28 20:51:42.793907876 +0200
Birth: 2024-07-28 20:51:42.793765043 +0200

The -D flag directs install to create the destination directories if they don’t exist, and the -m flag sets file permissions. The result is the same as the three lines of commands before.

It’s common for Makefiles in C/C++ projects to install binaries like this:

install -D -m 744 app_bin /usr/local/bin/app_bin

It copies app_bin to /usr/local/bin, creates the parent directory hierarchy if necessary, and sets permissions on the binary so only the current user has read, write, and execute permissions, while others have read-only access.

You can also set directory permissions:

install -d -m 600 foo/bar/bazz

This creates the directory hierarchy first and then sets the permission. Here’s how they look:

tree foo

Output:

foo
└── bar
    └── bazz

3 directories, 0 files

Then you can copy a file to the destination and set file permissions with another install command if needed.

You can also set user or group ownership while copying a file:

install -D -m 644 -o root -g root seed.db /var/lib/app/seed.db

This command copies seed.db to the destination, creates the directory if needed, and gives access to the root user and group with the -o and -g flags, respectively.

There are a few other options you can read about in the man pages, but I haven’t needed anything beyond the above.

Misc on Redowan's Reflections

Migrating from GNU stow to chezmoi

The machines #

When I outgrew stow #

How chezmoi works #

What I track #

Bootstrapping a new Mac #

Day to day #

Tracking agent skills #

Putting this blog on ATProto with standard.site

What it is #

Why bother #

Setting it up with Sequoia #

Making it hands-free #

Seeing it work #

Hierarchical rate limiting with Redis sorted sets

Redis sorted sets #

Lua script #

Using the script #

Testing the rate limiter #

Dynamic shell variables

A primer #

Context-aware environment management #

Function dispatch #

Temporary file handling #

Link blog in a static site

Here’s what has worked for me #

Detail view #

List view #

Running only a single instance of a process

A more portable version #

With Python #

With Go #

SSH saga

A new VM in the void #

Your local fortress #

The remote gatekeeper #

Authorized keys and the handshake #

The client and its configs #

Known hosts and server identity #

Agent and forwarding #

Configuring and hardening the daemon #

Bringing it all together #

Docker mount revisited

Volume mounts #

The -v option #

The --mount option #

In docker-compose.yml #

Bind mounts #

The -v option #

The --mount option #

In docker-compose.yml #

Tmpfs mounts #

The --tmpfs option #

The --mount option #

In docker-compose.yml #

Build cache mounts #

Discovering direnv

What problem it solves #

Why .envrc file and not just a plain .env file #

How I use it #

Notes on building event-driven systems

The curse of tightly coupled microservices #

What event-driven architecture solves #

The nature of an event #

Event-driven vs. event-based systems #

Fat vs. sparse events #

Publish-subscribe pattern and the role of the broker #

Handling constraints and governance #

Metadata-data pattern for evolvability #

Eventual consistency in event-driven systems #

Handling HTTP communication in an event-driven world #

Asynchronous commands #

CQRS for separating reads and writes #

Handling failure with the outbox pattern #

Bash namerefs for dynamic variable referencing

Namerefs #

Basic usage #

Managing multiple arrays #

Associative arrays and nested references #

The `-v` option #

The `--mount` option #

In `docker-compose.yml` #

The `-v` option #

The `--mount` option #

In `docker-compose.yml` #

The `--tmpfs` option #

The `--mount` option #

In `docker-compose.yml` #

Shorthand for piping both stdout and stderr (`|&`) #