# SSH into the remote machine and run commands to deploy the service
ssh $SSH_USER@$SSH_HOST <<EOF
    # Go to the work directory
    cd $WORK_DIR

    # Make a git pull
    git pull

    # Export environment variables required for the service to run
    export AUTH_TOKEN=$APP_AUTH_TOKEN

    # Start the service
    docker compose up -d --build
EOF

The fully working version can be found in the serve-init repo with here-doc.

Here, environment variables like SSH_USER, SSH_HOST, and APP_AUTH_TOKEN are defined in the surrounding local scope of the CI. The variables then get propagated to the remote machine when we run the commands via here-doc.

However, I couldn’t figure out why the Docker containers weren’t able to access the value of the AUTH_TOKEN variable. The other variables were getting through just fine.

It turns out, export AUTH_TOKEN=$AUTH_TOKEN within the here-doc block, doesn’t export the variable in the remote shell. So this doesn’t do what I thought it would:

cat <<EOF
    export FOO=bar
    echo $FOO
EOF

I was expecting it to print:

export FOO=bar
echo bar

But instead, it just prints:

export FOO=bar
echo

So export FOO=bar in the here-doc block doesn’t set the variable in the remote shell. One solution is to set it before the block like this:

export FOO=bar
cat <<EOF
    echo $FOO
EOF

This prints:

echo bar

So, in the CI pipeline, we could do the following to propagate the environment variable from local to the remote machine:

export FOO=bar
ssh $SSH_USER@$SSH_HOST <<EOF
    echo $FOO
EOF

This will print the value of the environment variable on the remote machine correctly. However, this doesn’t set the value in the remote shell’s environment. If you SSH into the remote machine and try to print the variable’s value, you’ll see nothing gets printed. The previous command only passes the value to the remote machine temporarily and doesn’t set it permanently in the remote shell.

To fix it, you could pipe the value into a file and load it in the remote shell like this:

ssh $SSH_USER@$SSH_HOST <<EOF
    echo "export FOO=$FOO" > /tmp/.env
    source /tmp/.env
    echo \$FOO
EOF

Here, echo \$FOO instead of echo $FOO ensures that the shell expansion is done on the remote machine, not on the local. This allows us to know that the environment variable has been set in the remote shell correctly.

Maybe the behavior makes sense, but it still broke my mental model.

So I decided to get rid of here-doc in the pipeline altogether and went with this:

SCRIPT="
    # Go to the work directory
    cd $WORK_DIR

    # Make a git pull
    git pull

    # Export environment variables required for the service to run
    export AUTH_TOKEN=$APP_AUTH_TOKEN

    # Start the service
    docker compose up -d --build
    "

# Run the script on the remote machine
ssh $SSH_USER@$SSH_HOST "$SCRIPT"

It works without here-doc!

One thing to keep in mind with the second approach is that if you need to run any expanded commands, you’ll need to defer it with a backslash so that it’s run on the remote machine, not on the local:

SCRIPT="
    # ...

    # Without backslash, shell runs this locally
    docker rmi -f \$(docker compose images -q) || true
    "
# Run the script on the remote machine
ssh $SSH_USER@$SSH_HOST "$SCRIPT"

Without the backslash, the $(...) will be expanded on the local machine, which is not desirable here. The backslash defers it so that it runs on the remote instead.

While there’s been no shortage of text on why and how to craft atomic commits, I often find those discussions focus too much on VCS hygiene, and the main benefit gets lost in the minutiae. When working in a team setup, I’ve discovered that individual commits matter much less than the final change list.

Also, I find some of the prescriptive suggestions for easier review, like keeping the PR under ~150 lines, ensuring that the tests pass in each commit, and tidying the commits to be strictly independent, quite cumbersome. Stacked PRs sometimes help to make large changes a bit more tractable, but that comes with a whole set of review-conflict-feedback challenges. So this piece will mainly focus on making large PRs a wee bit easier to work with.

Here’s a quick rundown of the things I find useful to make reviewing the grunt work of pull requests a bit more tractable. I don’t always strictly follow them while doing personal or OSS work, but these steps have been helpful while working on a large shared repo at work.

• Avoiding the temptation to lump tangentially related changes into a PR to speed things up.

• Having a ton of fragmented commits makes filtering useless when navigating the PR diff in a platform like GitHub. I really like to filter diffs on GitHub, but it wouldn’t be useful if the commits are all over the place.

  

• To make diff filtering better, I often rebase my feature branch after a messy development workflow and divide the changes into a few commits clustered around the core implementation, tests, documentation, dependency upgrades, and occasional refactoring.

• Rebasing all the changes into a single commit is okay if the change is small, but for bigger changes, this does more harm than good.

• I’ve rarely spent the time to ensure that the individual commits are perfect - in the sense that they’re complete with passing tests or documentation. As long as the complete change list makes sense as a whole, it’s good enough. YMMV. The main goal is to make sure the diff makes sense to the person reviewing the work.

• Annotated comments from the author on the PR are great. I wish they’d take up less space and there was a way to collapse them individually.

  

• Each PR must be connected to either an Issue or a Jira ticket, depending on how the team works.

• Adding context, screenshots, gifs, and videos to the PR description makes things so much easier for me when I do the review. Being able to see that the changes work as intended without running the code has its benefits.

  

• Keeping the PR in draft state until it’s ready to be reviewed. I’m not a fan of getting a notification to review some work only to find that it’s not ready yet.

• Usually doesn’t care for keeping atomic commits.
• Creates a lot of short commits with messages like “fix” or “wip”.
• Likes to clean up the untidy commits before sending the branch for peer review.
• Prefers a linear history over a forked one so that git log --oneline --graph tells a nice story.

Git rebase allows me to squash my disordered commits into a neat little one, which bundles all the changes with passing tests and documentation. Sure, a similar result can be emulated using git merge --squash feat_branch or GitHub’s squash-merge feature, but to me, rebasing feels cleaner. Plus, over time, I’ve subconsciously picked up the tricks to work my way around rebase-related gotchas.

Julia Evans explores the pros and cons of rebasing in detail. Also, squashing commits is just one of the many things that you can do with the rebase command. Here, I just wanted to document my daily rebasing workflow where I mostly rename, squash, or fixup commits.

A few assumptions

Broadly speaking, there are two common types of rebasing: rebasing a feature branch onto the main branch and interactive rebasing on the feature branch itself. The workflow assumes a usual web service development cadence where:

• You’ll be working on a feature branch that’s forked off of a main branch.
• The main branch is protected, and you can’t directly push your changes to it.
• Once you’re done with your feature work, you’ll need to create a pull request against the main branch.
• After your PR is reviewed and merged onto the main branch, CI automatically deploys it to some staging environment.

I’m aware this approach doesn’t work for some niches in software development, but it’s the one I’m most familiar with, so I’ll go with it.

Rebasing a feature branch onto the main branch

Let’s say I want to start working on a new feature. Here’s how I usually go about it:

1. Pull in the latest main with git pull.

2. Fork off a new branch via git switch -c feat_branch.

3. Do the work in feat_branch, and before sending the PR, do interactive rebasing if necessary, and then rebase the feat_branch onto the latest changes of main with:

   git pull --rebase origin main
   

4. Push the changes to the remote repository with git push origin HEAD and send a PR against main for review.

   Here, ...origin HEAD instructs git to push the current branch that HEAD is pointing to.

The 3rd step is where I often do interactive rebasing before sending the PR to make my work presentable. The next section will explain that in detail.

Occasionally, the 4th step doesn’t go as expected, and merge conflicts occur when I run git rebase main from feat_branch. In those cases, I use my editor (VSCode) to fix the conflict, add the changes with git add ., and run git rebase --continue. This completes the rebase operation, and we’re ready to push it to the remote.

Rebasing interactively on the feature branch

This is an extension of the 3rd step of the previous section. Sometimes, while working on a feature, I quickly make many messy commits and push them to the remote branch. This happens quite frequently when I’m prototyping on a feature or updating something regarding GitHub Actions. In these cases, I tend to make quick changes, commit with a message like “fix” or “ci” and push to remote to see if the CI is passing. However, once I’m done, the commit log on that branch looks like this:

git log @ ^main --oneline --graph

This command instructs git to show only the commits that exist on feat_branch but not on main. I learned recently that in git’s context, @ indicates the current branch. Neat, this means I won’t need to remember the branch name or do a git branch and then copy the name of the current branch. Running the command returns:

* 148934c (HEAD -> feat_branch) ci
* e0f6152 ci
* 8f4dc4c ci
* bf33bf7 ci
* 2e3dce6 ci

I’m not too proud of the state of this feat_branch and prefer to tidy things up before making a PR against main. One common thing I do is squash all these commits into one and then add a proper commit message. Interactive rebasing allows me to do that. Let’s say you want to interactively rebase the 5 commits listed above and squash them. To do so, you can run the following command from the feat_branch:

git rebase -i HEAD~5

This will open a file named git-rebase-todo in your default git editor (set via git config) that looks like this:

pick 763e178 ci # empty
pick 4b10faf ci # empty
pick 7f7ce20 ci # empty
pick 88fc529 ci # empty
pick 8bc19b6 ci # empty

# Rebase a2e45d3..8bc19b6 onto a2e45d3 (5 commands)
#
# Commands:
# p, pick <commit> = use commit
# r, reword <commit> = use commit, but edit the commit message
# e, edit <commit> = use commit, but stop for amending
# s, squash <commit> = use commit, but meld into previous commit
# f, fixup [-C | -c] <commit> = like "squash" but keep only the previous
#                    commit's log message, unless -C is used, in which case
#                    keep only this commit's message; -c is same as -C but
#                    opens the editor
# x, exec <command> = run command (the rest of the line) using shell
# b, break = stop here (continue rebase later with 'git rebase --continue')
# d, drop <commit> = remove commit
# l, label <label> = label current HEAD with a name
# t, reset <label> = reset HEAD to a label
# m, merge [-C <commit> | -c <commit>] <label> [# <oneline>]
#         create a merge commit using the original merge commit's
#         message (or the oneline, if no original merge commit was
#         specified); use -c <commit> to reword the commit message
# u, update-ref <ref> = track a placeholder for the <ref> to be updated
#                       to this position in the new commits. The <ref> is
#                       updated at the end of the rebase
#
# These lines can be re-ordered; they are executed from top to bottom.
#
# If you remove a line here THAT COMMIT WILL BE LOST.
#
# However, if you remove everything, the rebase will be aborted.
#

Notice that the file has quite a bit of instructions that are commented out. You can perform actions like pick, reword, edit, fixup, etc. I usually use squash and edit the git-rebase-todo file like this:

pick 763e178 ci # empty
s 4b10faf ci # empty  # <- s=squash melds into prev commit
s 7f7ce20 ci # empty
s 88fc529 ci # empty
s 8bc19b6 ci # empty

# ... rest of the file remains untouched

Now, if you close the previous file, git will automatically open another file like the following:

# This is a combination of 5 commits.
# This is the 1st commit message:

ci

# This is the commit message #2:

ci

# This is the commit message #3:

ci

# This is the commit message #4:

ci

# This is the commit message #5:

ci

After the first comment, you can put in the message for all the combined commits:

# This is a combination of 5 commits.

Add pip caching to the CI      # <- message for the combined commits

# ... you can remove rest of the content

If you close this file, you’ll see a message on your console indicating that the rebase has been successful:

[detached HEAD 28f5084] Add pip caching to the CI
 Date: Wed Jun 19 22:42:07 2024 +0200
Successfully rebased and updated refs/heads/feat_branch.

Now running git log will show that the messy commit has been squashed into one.

git log @ ^main --oneline --graph

This displays:

* 28f5084 (HEAD -> feat_branch) Add pip caching to the CI

This is just one of the many things you can do during interactive rebasing. While I do this most commonly, sometimes I also drop unnecessary commits to tidy up things and group multiple commits instead of just squashing everything into one commit. All of these actions can be done in a similar manner to squashing commits as mentioned above.

Sometimes, I don’t know how many commits I’ll need to interactively rebase. In those cases, I can get the number of all the new commits on a feature branch by counting the entries in git log as follows:

git log @ ^main --oneline | wc -l

Then you can use the number from the output of the previous command to rebase n number of commits:

git rebase -i HEAD~n

Another thing you can do is split a single commit into multiple commits. This is quite a bit more involved and I rarely do it during interactive rebasing.

One last thing I learned recently is that you can run your tests or any arbitrary command during interactive rebasing. To do so, start your rebase session with --exec cmd as follows:

git rebase -i --exec "echo hello" HEAD~5

In the git-rebase-todo file this time, you’ll see that the command is run after each commit as follows:

pick dffb3c1 ci # empty
exec echo hello
pick 4d2fa08 ci # empty
exec echo hello
pick 2b35e4f ci # empty
exec echo hello
pick 6de7a52 ci # empty
exec echo hello

# ...

You can edit this file to run the exec command after any commit you want to. The commands will run once you save and close this file. This is a neat way to run your test suite and make sure they pass in the intermediate commits.

Fin!

Further reading

• Hackernews discussion on rebasing

Recently, I was doing some lightweight frontend work and needed to make some AJAX calls from one domain to another. Usually, browser’s CORS (Cross-Origin Resource Sharing) policy will get in your way if you try this. While you’re reading this piece, open the dev console and paste the following fetch snippet:

fetch("https://mozilla.org")
  .then((response) => response.text())
  .then((data) => {
    // Do something with the received data
    console.log(data);
  })
  .catch((error) => {
    // Handle any errors that occurred during the request
    console.error("Error:", error);
  });

This snippet will attempt to make a GET request from https://rednafi.com to https://mozilla.org. However, the client’s CORS policy won’t allow you to make an AJAX request like this and load external resources into the current site. On your console, you’ll see an error message like this:

Access to fetch at 'https://mozilla.org/' from origin 'https://rednafi.com'
has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is
present on the requested resource.

If an opaque response serves your needs, set the request's mode to
'no-cors' to fetch the resource with CORS disabled.

This is a good security measure. Without CORS, a malicious script could make a request to a server in another domain and access the resources that the user of the page is not intended to have access to. So much has been said and written about CORS that I won’t even attempt to explain it here. Here’s another high-level introduction to CORS.

CORS proxy

While CORS is generally a good thing, it can be quite annoying when you’re trying to build something that needs access to external resources. In those cases, you’ll have to mess around with the origin server and add a few headers that the browser can understand before it allows you to load those external resources. But sometimes you don’t have access to the origin server or simply don’t want to deal with modifying the server’s response headers every time you need to access external resources. That’s where CORS proxies can come in handy.

A CORS proxy server acts as a bridge between your client and the target server. It receives your request and forwards it to the target server with a modified origin header so that the target server thinks the request is coming from the same origin as itself.

This way, you can bypass the same-origin policy of browsers and access resources from different domains. I usually use free proxies like cors.sh to bypass CORS restrictions. You can drop this snippet to your browser’s console and this time it’ll allow you to load the contents of https://mozilla.org from https://rednafi.com:

// Notice how we're prepending CORS URL before the target URL
fetch("https://proxy.cors.sh/https://mozilla.org")
  .then((response) => response.text())
  .then((data) => {
    // Do something with the received data
    console.log(data);
  })
  .catch((error) => {
    // Handle any errors that occurred during the request
    console.error("Error:", error);
  });

The target server’s response looks somewhat like this:

<!doctype html>
<html class="windows no-js" lang="en" dir="ltr" data-country-code="US"
data-latest-firefox="113.0.1" data-esr-versions="102.11.0"
data-gtm-container-id="GTM-MW3R8V" data-gtm-page-id="Homepage"
data-stub-attribution-rate="1.0" data-convert-project-id="10039-1003343">
<head>
...

If you want to learn more about how CORS proxies work, here’s a fantastic article on CORS proxies that explains the inner machinery in more detail.

Free proxy servers can be pernicious

Using a free CORS proxy server can be dangerous as it might spill the beans on your requests and data to some random service you don’t know or trust. Since it plays as a middleman between your app and the resource you’re after, it could potentially snoop on, mess with, or keep tabs on your requests and data. Plus, some of those free CORS proxy servers might have restrictions on the size, type, or number of requests they can handle, or they might not even support HTTPS or other security bells and whistles.

Build your own CORS proxy with Cloudflare Workers

With all the intros out of the way, here’s how CloudFlare Workers afforded me to prop up a CORS proxy in less than half an hour. If you’re impatient and just want to take a look at the service in its full glory then head over to the cors-proxy repo. GitHub Actions deploys the service automatically to CloudFlare Workers every time a change is pushed to the main branch.

Installing the prerequisites

Assuming you have node installed on your system, you can fetch the wrangler CLI with the following command:

npm install -g wrangler

This will allow us to develop and test the service locally.

Bootstrapping the service

Create a new directory where you want to develop your service and bring it under source control. Now, run:

npm create cloudflare@latest

The CLI will guide you through the entire bootstrapping process interactively. You’ll have to create a Cloudflare account (if you don’t have one already) and log into the dashboard. Then it’ll prompt you to deploy your first hello-world API endpoint that you can immediately start to play with without doing anything else. Being able to see the serverless function in action within like 5 minutes gave me a huge dopamine boost that AWS Lambda never could. You can see the interactive bootstrapping section here:

using create-cloudflare version 2.0.7

╭ Create an application with Cloudflare Step 1 of 3
│
├ Where do you want to create your application?
│ dir cors-proxy
│
├ What type of application do you want to create?
│ type "Hello World" script
│
├ Do you want to use TypeScript?
│ typescript no
│
├ Copying files from "simple" template
│
╰ Application created

╭ Installing dependencies Step 2 of 3
│
├ Installing dependencies
│ installed via `npm install`
│
╰ Dependencies Installed

╭ Deploy with Cloudflare Step 3 of 3
│
├ Do you want to deploy your application?
│ yes deploying via `npm run deploy`
│
├ Logging into Cloudflare This will open a browser window
│ allowed via `wrangler login`
│
├ Deploying your application
│ deployed via `npm run deploy`
│
├  SUCCESS  View your deployed application at
│  https://cors-proxy.rednafi.workers.dev (this may take a few mins)
│
│ Run the development server npm run dev
│ Deploy your application npm run deploy
│ Read the documentation https://developers.cloudflare.com/workers
│ Stuck? Join us at https://discord.gg/cloudflaredev
│
╰ See you again soon!

Running the interactive session will create the following directory structure:

├── src
│   └── worker.js
├── package-lock.json
├── package.json
└── wrangler.toml

Developing the CORS proxy

We’ll write our proxy server in src/worker.js file. Copy the following JS snippet and paste it to the file:

export default {
  async fetch(request, env, ctx) {
    // Extract method, url and headers from the incoming request object.
    const { method, url, headers } = request;

    // Extract destination url from the query string.
    const destUrl = new URL(url).searchParams.get("url");

    // If the destination url is not present, return 400.
    if (!destUrl) {
      return new Response("Missing destination URL.", { status: 400 });
    }

    // If the request method is OPTIONS, return CORS headers.
    if (
      method === "OPTIONS" &&
      headers.has("Origin") &&
      headers.has("Access-Control-Request-Method")
    ) {
      const responseHeaders = {
        "Access-Control-Allow-Origin": headers.get("Origin"),
        "Access-Control-Allow-Methods": "*", // Allow all methods
        "Access-Control-Allow-Headers": headers.get(
          "Access-Control-Request-Headers"
        ),
        "Access-Control-Max-Age": "86400",
      };
      return new Response(null, { headers: responseHeaders });
    }

    const proxyRequest = new Request(destUrl, {
      method,
      headers: {
        ...headers,
        Origin: "",
      },
    });

    try {
      const response = await fetch(proxyRequest);
      const responseHeaders = new Headers(response.headers);
      responseHeaders.set("Access-Control-Allow-Origin", "*");
      responseHeaders.set("Access-Control-Allow-Credentials", "true");
      responseHeaders.set("Access-Control-Allow-Methods", "*");

      return new Response(response.body, {
        status: response.status,
        statusText: response.statusText,
        headers: responseHeaders,
      });
    } catch (error) {
      return new Response("Error occurred while fetching the resource.", {
        status: 500,
      });
    }
  },
};

The first section of the code deals with extracting relevant information from the incoming request. It destructures the method, url, and headers properties from the request object, which represents the client’s request.

Next, it extracts the destination URL from the query string. It extracts the URL parameter using the searchParams.get() method. If the destination URL isn’t provided, the function returns a Response object with an error message and a status code of 400 (Bad Request).

The code then checks if the request method is OPTIONS. The OPTIONS method is used in CORS preflight requests to determine if the actual request is safe to send. If the request is an OPTIONS request and contains specific headers indicating a CORS preflight request (Origin and Access-Control-Request-Method), the function generates a response with appropriate CORS headers. The response headers include Access-Control-Allow-Origin to reflect the client’s origin, Access-Control-Allow-Methods set to *, allowing any HTTP method, Access-Control-Allow-Headers based on the requested headers, and Access-Control-Max-Age set to 86400 seconds (one day) to cache the preflight response.

If the request is not an OPTIONS request or doesn’t meet the CORS preflight conditions, the code continues execution. It creates a new Request object named proxyRequest uses the extracted destination URL and sets the method and headers of the original request. The Origin header is removed to prevent CORS restrictions when forwarding the request.

The subsequent code performs the actual request forwarding. It uses fetch to send the proxyRequest to the destination URL. If the fetch is successful, the code proceeds to process the response. It creates a new Headers object from the response’s headers and modifies them to include the necessary CORS headers.

Finally, the function constructs a Response object using the response body, status, statusText, and modified headers. If an error occurs during the fetch operation, the code catches the error and returns a Response object with an error message and a status code of 500 (Internal Server Error).

Once you’ve pasted the snippet, you can redeploy the service from your local machine with:

wrangler deploy

This will deploy the service immediately:

 ⛅️ wrangler 3.0.0
------------------
Total Upload: 1.52 KiB / gzip: 0.57 KiB
Uploaded cors-proxy (0.51 sec)
Published cors-proxy (0.38 sec)
  https://<your-deployed-service>
Current Deployment ID: f300ac99-c15e-4e30-a910-a56d81c10b95

I’ve removed my root domain from the above output since I’m using the free version of Workers and don’t want people to exhaust my free request quota. Haha, security by obscurity! But once you’ve deployed your proxy server, you can go to the following URL from your browser:

https://<your-deployed-service>/?url=https://mozilla.com

This will send you to the Mozilla website through the deployed function. Now you can use it just like the free CORS proxy. Try it out by dropping the following snippet to your browser console. This is exactly the same as the previous fetch snippet but the only difference is this time, we’re using our own proxy server that we control:

// Notice how we're prepending CORS URL before the target URL
fetch("https://<your-deployed-service>?url=https://mozilla.org")
  .then((response) => response.text())
  .then((data) => {
    // Do something with the received data
    console.log(data);
  })
  .catch((error) => {
    // Handle any errors that occurred during the request
    console.error("Error:", error);
  });

Don’t forget to replace the <your-deployed-service> URL with your own service. This will result in a successful request. You can also interactively send requests to the destination URLs via the Cloudflare Workers dashboard. Go to your Cloudflare dashboard, head over to the Workers section, and select your deployed serverless function:

Deploying the service with GitHub Actions

For one-off services, wrangler deploy in the local machine works perfectly but I usually don’t consider a project fully done until I’ve automated away the whole process. So, I wrote a quick GitHub Actions workflow to run the linters and deploy the service automatically when a new commit is pushed to the main branch. Here’s how it looks:

# .github/workers/ci.yml

name: Deploy

on:
  push:
    branches:
      - main

  # Allow running this workflow manually.
  workflow_dispatch:


jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: "lts/*"
          cache: npm
          cache-dependency-path: cors-proxy/package-lock.json
      - name: Install dependencies
        working-directory: ./cors-proxy
        run: |
          npm install
      - name: Run linter
        working-directory: ./cors-proxy
        run: |
          npx prettier --check .

  deploy:
    runs-on: ubuntu-latest
    needs: build
    steps:
      - uses: actions/checkout@v3
      - name: Publish
        uses: cloudflare/wrangler-action@2.0.0
        with:
          apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
          workingDirectory: "cors-proxy"

For this to work, you’ll need to create a Cloudflare API token and add it to the GitHub Secrets of your proxy server’s repository. Here’s the complete CI workflow file.

• Opening a new GitHub issue for the feature in the corresponding repository.
• Adding a rough description of the feature to the issue.
• Creating a feature branch off of main/master/trunk. If the feature is trivial or just a doc update, this step can be skipped.
• Referring to the issue in every commit message as you start working on the feature:
  • Appending #refs <issue-number> to every commit message. This will attach the commit to the concerning issue on the GitHub UI.
  • Appending #closes <issue-number> to the final commit message when the feature is complete.
  • If you need to refer to an issue after it’s closed, you can still do that by appending #refs <issue-number> to the commit message. So a commit message should look similar to Feature foo, refs #120 or Update foo, closes #115. The comma (,) before refs/closes is essential here. I like to enforce it.

This pattern also works for bugfixes without any changes. Here’s an example issue that shows the workflow in action. Plus, I follow a similar pattern to write the blogs on this site as well. This is what a feature issue might look like on GitHub:

While I’m quite happy with how the process is working for me, often time, I get careless and push commits without a reference to any issue. This pollutes the Git history and breaks my streak of maintaining good hygiene. So, I was looking for a way to make sure that the CI fails and reprimands me whenever I’m not following the process correctly. It’s just one less thing to worry about.

I’ve decided to use GitHub Actions to audit the conformity of the commit messages. The CI pipeline is orchestrated as follows:

• After every push and pull-request, the audit-commits job in an audit.yml workflow file will verify the conformity of the commit messages. This job runs a regex pattern against every commit message and fails with exit code 1 if the message doesn’t respect the expected format.
• If the audit-commits job passes successfully, only then the primary jobs in the ci.yml workflow will execute. The entire pipeline will fail and the primary CI workflow won’t be triggered at all if the audit-commit job fails at any point.

On GitHub, you’re expected to place your workflow files in the .github/workflows directory. If you inspect this blog’s workflows folder, you’ll see this pattern in action. Here, the directory has three workflow files:

.github/workflows
├── audit.yml
├── automerge.yml
└── ci.yml

The automerge.yml file automatically merges a pull-request when the primary CI jobs pass. I wrote about it in more detail in another post about automerging Dependabot PRs. We’ll ignore the automerge.yml file for now. Here, the audit file runs after every push and pull-request and verifies the structure of the commit message. I picked a generic name like audit.yml instead of a more specific one like audit-commit.yml because in the future if I want to add another check, I can easily extend this file without renaming it. Here’s the unabridged content of the audit.yml file:

# .github/workflows/audit.yml
# Auditing commit structure.
name: Audit

on:
  workflow_call:


jobs:
  audit-commits:
    runs-on: ubuntu-latest
    if: ${{ github.actor != 'dependabot[bot]' }}
    steps:
      - name: "Check commit message format"
        shell: bash
        run: |
          set -euo pipefail

          # Get the commit payload from GH Actions push event.
          # See GitHub's webhook-events docs for details
          commits='${{ toJSON(github.event.commits) }}'

          # Exit with 0 if no new commit is found.
          if [[ $commits =~ "null" ]]; then
              echo "No commit found. Exiting..."
          exit 0
          fi

          # Get the unique messages from the commits event.
          parsed=$(echo -n "$commits" | jq -r ".[].message" | sort -u)
          mtch='(, refs|, closes) #[0-9]+'

          echo "$parsed" | while IFS= read -r raw_line; do
              line=$(echo "$raw_line" | tr -d "\r\n")

              # Ignore empty lines.
              if [[ -z "$line" ]]; then
                  continue

              # Check for 'refs #N' or 'closes #N'. Else error.
              elif [[ "$line" =~ $mtch ]]; then
                  echo "Commit message: $line ✅"
              else
                  echo "Commit message: $line ❌"
                  echo -n "Commit message must contain "
                  echo -n "'refs #issue_number' or 'closes #issue_number'."
                  exit 1
              fi
          done

I’ve defined this workflow as a reusable one. A reusable workflow can be called like a function with parameters from another workflow. The workflow_call node the audit.yml file makes it a reusable one and you can define additional parameters in this section if you need to do so. However, in this particular case, I don’t need to pass any parameters while calling the audit.yml workflow from the ci.yml workflow. You can find more details on how to define reusable workflows in the docs.

In the jobs section of the audit.yml file, we define a single audit-commits job that runs a bash script against every incoming commit message and verifies its structure. The commit messages can be accessed from the '${{ toJSON(github.event.commits) }}' context variable. Then the script loops over every commit message and verifies the structure. It’ll terminate the job with exit code 1 if the incoming message doesn’t match the expected structure. Otherwise, the script will gracefully terminate the job with exit code 0.

In the main ci.yml file the audit.yml workflow is called like this:

...

jobs:
  audit:
    uses: rednafi/reflections/.github/workflows/audit.yml@master
...

The ci.yml file roughly looks like this:

name: CI

on:
  push:
  pull_request:

  # Everyday at 0:37 UTC.
  schedule:
    - cron: "37 0 * * *"


# Cancel any running workflow if the CI gets triggered again.
concurrency:
  group: ${{ github.head_ref || github.run_id }}
  cancel-in-progress: true


jobs:
  audit:
    uses: rednafi/reflections/.github/workflows/audit.yml@master

  build:
    needs: ["audit"]
    runs-on: ubuntu-latest
    steps:
      ...

  test:
    needs: ["build"]
    runs-on: ubuntu-latest
    steps:
      ...

  deploy:
    needs: ["deploy"]
    runs-on: ubuntu-latest
    steps:
      ...

Here the needs: ["audit"] node in the build section ensures that the build will only trigger if the audit job passes successfully. Otherwise, none of the build, test, or deploy jobs will run and the CI will fail with a non-zero exit code. Here’s the fully working ci.yml file.

Notes

GitHub Actions terminology can be confusing.

• A workflow is a separate file that contains one or more jobs.
• A job is a set of steps in a workflow that executes on the same runner.
• A runner is a server that runs your workflows when they’re triggered. Each runner can run a single job at a time.
• A reusable workflow can be called from another workflow file.

The docs have more information on understanding GitHub Actions.

• You checkout from the main branch to work on a feature in a branch named alice.
• When you’re done, you merge alice into main.
• After that, if you try to pull the main branch from remote again and the content of the main branch changes by this time, you’ll encounter a merge error.

Reproduce the issue

Create a new branch named alice from main. Run:

git checkout -b alice

From alice branch, add a line to a newly created file foo.txt:

echo "from branch alice" >> foo.txt

Add, commit, and push the branch:

git commit -am "From branch alice" && git push

From the GitHub UI, send a pull request against the main branch and merge it:

In your local machine, switch to main and try to pull the latest content merged from the alice branch. You’ll encounter the following error:

hint: You have divergent branches and need to specify how to reconcile them.
hint: You can do so by running one of the following commands sometime before
hint: your next pull:
hint:
hint:   git config pull.rebase false  # merge (the default strategy)
hint:   git config pull.rebase true   # rebase
hint:   git config pull.ff only       # fast-forward only
hint:
hint: You can replace "git config" with "git config --global" to set a default
hint: preference for all repositories. You can also pass --rebase, --no-rebase,
hint: or --ff-only on the command line to override the configured default per
hint: invocation.
fatal: Need to specify how to reconcile divergent branches.

This means that the history of your local main branch and the remote main branch have diverged and they aren’t reconciliable.

Solution

From the main branch, you can run:

git pull --rebase

This will rebase your local main by adding your local commits on top of the remote commits.

Further reading

• When should I use git pull –rebase
• An example repo that reproduces the issue

• rubric: linter config initializer for Python
• exert: declaratively apply converter functions to class attributes
• hook-slinger: generic service to send, retry, and manage webhooks
• think-async: exploring cooperative concurrency primitives in Python
• epilog: container log aggregation with Elasticsearch, Kibana & Filebeat

While many of these prototypes become full-fledged projects, most end up being just one-time journies. One common theme among all of these endeavors is that I always include instructions in the readme.md on how to get the project up and running - no matter how small it is. Also, I tend to configure a rudimentary CI pipeline that runs the linters and tests. GitHub Actions and Dependabot make it simple to configure a basic CI workflow. Dependabot keeps the dependencies fresh and makes pull requests automatically when there’s a new version of a dependency used in a project.

Things can get quickly out of hand if you’ve got a large collection of repos where the automated CI runs periodically. Every now and then, I get a sizable volume of PRs in these fairly stale repos that I still want to keep updated. Merging these manually is a chore. Luckily, there are multiple ways to automatically merge PRs that GitHub offers. The workflow that is documented here is the one I happen to like the most. I also think that this process leads to the path of the least surprise. Instead of depending on a bunch of GitHub settings, we’ll write a GitHub Actions workflow to enable auto-merge to automate the process.

First, you’ll need to turn on the auto-merge option from the repository settings. To do so, go to the repo’s settings tab and turn on the Allow auto-merge option from the Pull Requests section:

Now, you probably don’t want to mindlessly merge every pull request Dependabot throws at you. You most likely want to make sure that a pull request triggers certain tests and it’ll be merged only if all of those checks pass. To do so, you can turn on branch protection. From the settings panel, select Branches on the left panel:

Once you’ve selected the tab, add a branch protection rule to the target branch against which Dependabot will send the pull requests:

In this case, I’m adding the protection layer to the main branch. I’ve turned on the Require status checks to pass before merging toggle and added the build step to the list of status checks that are required. Here, you can select any job from your CI files in the .github/workflows directory:

Once this is done, you can drop the following CI file in the .github/workflows directory of your repo. It’s the same automerge workflow file that’s currently living inside this site’s CI folder.

# .github/workflows/automerge.yml

name: Dependabot auto-merge

on: pull_request

permissions:
  contents: write
  pull-requests: write  # Needed if in a private repository

jobs:
  dependabot:
    runs-on: ubuntu-latest
    if: ${{ github.actor == 'dependabot[bot]' }}
    steps:
      - name: Enable auto-merge for Dependabot PRs
        run: gh pr merge --auto --merge "$PR_URL"
        env:
          PR_URL: ${{github.event.pull_request.html_url}}
          # GitHub provides this variable in the CI env. You don't
          # need to add anything to the secrets vault.
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

From now on, every time Dependabot sends a merge request, the checks will be triggered and if all the mandatory checks pass, the automerge.yml workflow will merge it into the target branch.

• If a new workflow is triggered while the previous one is running, the first one will get canceled.
• The CI is triggered every day at UTC 1.
• Tests and the lint-checkers are run on Ubuntu and MacOS against multiple Python versions.
• Pip dependencies are cached.
• Dependencies, including the Actions dependencies are automatically updated via Dependabot.

I use pip-tools for managing dependencies in applications and setuptools setup.py combo for managing dependencies in libraries. Here’s an annotated version of the template action syntax:

# .github/workflows/ci.yml

name: CI

on:
  # Triggers when something is pushed to the 'main' branch.
  push:
    branches:
      - master

  # Triggers when a pull request is sent against the 'main' branch.
  pull_request:
    branches:
      - master

  # Triggers everyday at 1 UTC.
  schedule:
    - cron: "0 1 * * *"

# Cancel any running workflow if the CI gets triggered again.
concurrency:
  group: ${{ github.head_ref || github.run_id }}
  cancel-in-progress: true

jobs:
  run-tests:
    # Tests are run on multiple Python versions.
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        # Multiple OSs.
        os: [ubuntu-latest, macos-latest]

        # Multiple Python versions.
        python-version: ["3.10", "3.11", "3.12"]
        include:
          - os: ubuntu-latest
            path: ~/.cache/pip # Cache location on Ubuntu
          - os: macos-latest
            path: ~/Library/Caches/pip # Cache location on MacOS

    steps:
      # Checkout to the codebase.
      - uses: actions/checkout@v3

      # Sets up Python.
      - uses: actions/setup-python@v3
        with:
          python-version: ${{ matrix.python-version }}

      # Cache pip dependencies via 'cache' actions.
      - uses: actions/cache@v2
        with:
          path: ${{ matrix.path }}
          key: "${{ runner.os }}-pip-${{ hashFiles('requirements*.txt') }}"
          restore-keys: |
            ${{ runner.os }}-pip-

      # Dev and app dependencies are kept in separate files.
      - name: Install the Dependencies
        run: |
          pip install --upgrade pip
          pip install -r requirements.txt
          pip install -r requirements-dev.txt

      # Run black, isort, flake8, etc.
      - name: Check Linter
        run: |
          echo "Checking black formatting..."
          python3 -m black --check .
          echo "Checking isort formatting..."
          python3 -m isort --check .
          echo "Checking flake8 formatting..."
          python3 -m flake8 .

      # Run the tests via pytest.
      - name: Run the tests
        run: |
          pytest -v -s

The dependabot config looks as follows:

# .github/dependabot.yml

version: 2
updates:
    - package-ecosystem: "pip" # See documentation for possible values.
    directory: "/" # Location of package manifests.
    schedule:
      interval: "daily"

  # Maintain dependencies for GitHub Actions.
    - package-ecosystem: "github-actions"
    directory: "/"
    schedule:
      interval: "daily"

Further reading

• An active version of the above workflow