JavaScript on Redowan's Reflections

Eschewing black box API calls

Redowan Delowar — Mon, 15 Jan 2024 00:00:00 +0000

I love dynamically typed languages as much as the next person. They let us make ergonomic API calls like this:

import httpx

# Sync call for simplicity
r = httpx.get("https://dummyjson.com/products/1").json()
print(r["id"], r["title"], r["description"])

or this:

fetch("https://dummyjson.com/products/1")
  .then((res) => res.json())
  .then((json) => console.log(json.id, json.type, json.description));

In both cases, running the snippets will return:

1 'iPhone 9' 'An apple mobile which is nothing like apple'

Unless you’ve worked with a statically typed language that enforces more constraints, it’s hard to appreciate how incredibly convenient it is to be able to call and use an API endpoint without having to deal with types or knowing anything about its payload structure. You can treat the API response as a black box and deal with everything in runtime.

For example, Go wouldn’t even allow you to do so in such a loosey-goosey way. To consume the API, you’d need to create a struct in the essence of the return payload and then unmarshal the payload into it.

Here’s the complete response payload that curl -s https://dummyjson.com/products/1 | jq returns:

{
  "id": 1,
  "title": "iPhone 9",
  "description": "An apple mobile which is nothing like apple",
  "price": 549,
  "discountPercentage": 12.96,
  "rating": 4.69,
  "stock": 94,
  "brand": "Apple",
  "category": "smartphones",
  "thumbnail": "https://cdn.dummyjson.com/product-images/1/thumbnail.jpg",
  "images": [
    "https://cdn.dummyjson.com/product-images/1/1.jpg",
    "https://cdn.dummyjson.com/product-images/1/2.jpg",
    "https://cdn.dummyjson.com/product-images/1/3.jpg",
    "https://cdn.dummyjson.com/product-images/1/4.jpg",
    "https://cdn.dummyjson.com/product-images/1/thumbnail.jpg"
  ]
}

This is how you’d call the API endpoint in Go. I’m using the json-to-go service to generate the Go struct instead of handwriting it:

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "net/http"
)

// Define the struct that reflects the response payload
type Product struct {
    ID                 int      `json:"id"`
    Title              string   `json:"title"`
    Description        string   `json:"description"`
    Price              int      `json:"price"`
    DiscountPercentage float64  `json:"discountPercentage"`
    Rating             float64  `json:"rating"`
    Stock              int      `json:"stock"`
    Brand              string   `json:"brand"`
    Category           string   `json:"category"`
    Thumbnail          string   `json:"thumbnail"`
    Images             []string `json:"images"`
}

func main() {
    // Ignore error handling for brevity
    var product Product

    response, _ := http.Get("https://dummyjson.com/products/1")
    defer response.Body.Close()

    body, _ := io.ReadAll(response.Body)

    _ = json.Unmarshal(body, &product) // project into struct

    // Do processing with product instance
    fmt.Println(product.ID, product.Title, product.Description)
}

This will give us the same output as the Python and JS code snippets:

1 iPhone 9 An apple mobile which is nothing like apple

Above, we had to create a new struct type to represent the response payload, instantiate it, and unmarshal the JSON payload into the struct before we were able to process it.

Notice that we’re only using 3 fields and ignoring the rest. In this case, you can get away with only including those 3 fields in the struct type, and Go will do the right thing:

// ... same as before

type Product struct {
    ID          int     `json:"id"`
    Title       string  `json:"title"`
    Description string  `json:"description"`
}

// ... same as before

While this is less work than having to emulate the whole structure of the JSON output in the struct definition, it’s still not winning any medals for brevity against the Python and JS snippets.

Dynamically processing a JSON payload is nice as long as you’re working on a throwaway script. Anything more, it becomes a headache since the readers won’t have any idea about what the API response looks like without looking at the documentation or traces.

Also, type safety is an issue. Since the imperative workflow doesn’t assume the structure of the response, you’ll be surprised with a runtime error if you make an incorrect assumption about the response structure. Sure, having to write a struct is a chore, but the free documentation and the type safety are things that you don’t get with the black box API calls.

Statically typed languages force you to maintain good hygiene while working with JSON payloads. Declaratively embedding the payload structure directly into the codebase is immensely beneficial; it reduces the out-of-band knowledge required to understand the code and adds type safety as a cherry on top. But how do you do that in a language like Python?

If you want to go with what’s in the standard library, you can handroll a dataclass like this and project the return payload onto it:

# ...

from dataclasses import dataclass
from typing import Self


@dataclass(slots=True)
class Product:
    id: int
    title: str
    description: str
    price: int
    discount_percentage: float
    rating: float
    stock: int
    brand: str
    category: str
    thumbnail: str
    images: list[str]

    @classmethod
    def from_dict(cls, d: dict) -> Self:
        # Reconcile snake_case and camelCase variables
        discount_percentage = d.pop("discountPercentage", None)
        return cls(discount_percentage=discount_percentage, **d)


# ...

Then just call Product.from_dict and pass the output of response.json() as before.

This way, the API response is documented in the code, and the reader won’t have to depend on out-of-band information while reading the code. However, you can see that hand rolling data classes can quickly become hairy when you have a large JSON payload and need to reconcile the discrepancies between snake case and camel case variables. We had to add a custom from_dict class method to convert the camel case variables to their snake case counterparts in Python.

Also, unlike Go, you can’t define a structure to represent only a portion of the whole payload in Python without adding extra code to ignore the rest of the fields that aren’t relevant to you.

Pydantic shines here. It not only allows you to define a class to represent a partial payload structure, but also applies runtime validation to guarantee operational type safety. As a bonus, you can use the json-to-pydantic tool to generate pydantic classes from JSON:

from pydantic import BaseModel, Field


class Product(BaseModel):
    id: int
    title: str
    description: str
    price: int
    discount_percentage: float = Field(alias="discountPercentage")
    rating: float
    stock: int
    brand: str
    category: str
    thumbnail: str
    images: list[str]

You can project your response onto the data class with Product(**response.json()) and get a rich object that also validates the incoming values. This will work the same way with partially defined classes:

# ...

from pydantic import BaseModel


class Product(BaseModel):
    id: int
    title: str
    description: str


# ...

Here’s a complete example:

import httpx
from pydantic import BaseModel
import asyncio


# Partially defined Pydantic model to represent the response
class Product(BaseModel):
    id: int
    title: str
    description: str


async def main() -> None:
    async with httpx.AsyncClient() as client:
        response = await client.get("https://dummyjson.com/products/1")
        response.raise_for_status()
        data = response.json()
        product = Product(**data)
        print(product.id, product.title, product.description)


if __name__ == "__main__":
    asyncio.run(main())

In the JS land, you can adopt TypeScript and zod to achieve a similar result:

// index.ts
import { z } from "zod";

const ProductSchema = z.object({
  id: z.number(),
  title: z.string(),
  description: z.string(),
  // Other fields can be added if needed
});

type Product = z.infer<typeof ProductSchema>;

fetch("https://dummyjson.com/products/1")
  .then((response) => response.json())
  .then((data) => ProductSchema.parse(data))
  .then((product: Product) => {
    console.log(product.id, product.title, product.description);
  });

I don’t mind the added verbosity if it leads to better readability and type safety.

Fin!

Bulk request Google search indexing with API

Redowan Delowar — Fri, 26 May 2023 00:00:00 +0000

Recently, I purchased a domain for this blog and migrated the content from rednafi.github.io to rednafi.com. This turned out to be a much bigger hassle than I originally thought it’d be, mostly because, despite setting redirection for almost all the URLs from the previous domain to the new one and submitting the new sitemap.xml to the Search Console, Google kept indexing the older domain. To make things worse, the search engine selected the previous domain as canonical, and no amount of manual requests were changing the status in the last 30 days. Strangely, I didn’t encounter this issue with Bing, as it reindexed the new site within a week after I submitted the sitemap file via their webmaster panel.

While researching this, one potential solution suggested that along with submitting the sitemap via Google Search Console, I’d have to make individual indexing requests for each URL to encourage faster indexing. The problem is, I’ve got quite a bit of content on this site, and it’ll take forever for me to click through all the links and request indexing that way. Naturally, I looked for a way to do this programmatically. Luckily, I found out that there’s an indexing API that allows you to make bulk indexing requests programmatically. This has one big advantage - Google responds to API requests faster than indexing requests with sitemap submission.

All you’ve to do is:

List out the URLs that need to be indexed.
Fulfill the prerequisites and download the private key JSON file required to make requests to the API. From the docs:

Every call to the Indexing API must be authenticated with an OAuth token that you get in exchange for your private key. Each token is good for a span of time. Google provides API client libraries to get OAuth tokens for a number of languages.

The private key file will look like this:
```
{
  "type": "service_account",
  "project_id": "...",
  "private_key_id": "...",
  "private_key": "...",
  "client_email": "...",
  "client_id": "...",
  "auth_uri": "...",
  "token_uri": "...",
  "auth_provider_x509_cert_url": "...",
  "client_x509_cert_url": "...",
  "universe_domain": "..."
}
```
Use an API client to make the requests.

In my case, this site’s sitemap.xml lists out all the URLs as follows:

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:xhtml="http://www.w3.org/1999/xhtml">
<url>
    <loc>https://rednafi.com/tags/github/</loc>
    <lastmod>2023-05-21T00:00:00+00:00</lastmod>
</url>
<url>
    <loc>https://rednafi.com/tags/javascript/</loc>
    <lastmod>2023-05-21T00:00:00+00:00</lastmod>
</url>
...

Here’s a NodeJS script that collects the URLs from sitemap.xml and makes requests to the indexing API:

// ES6 import
import { google } from "googleapis";
import { parseString } from "xml2js";
import fetch from "node-fetch";

import pkey from "./google-api-pkey.json" assert { type: "json" };

// Parse the sitemap.xml file and extract the URLs.
async function getUrls(url) {
  try {
    const response = await fetch(url);
    const xml = await response.text();
    let urls;
    parseString(xml, (err, result) => {
      if (err) {
        console.error("Error parsing XML:", err);
        return;
      }

      urls = result.urlset.url.map((url) => url.loc[0]);
    });
    return urls;
  } catch (error) {
    console.error("Error fetching sitemap:", error);
  }
}

// Initialize auth client
const jwtClient = new google.auth.JWT(
  pkey.client_email,
  null,
  pkey.private_key,
  ["https://www.googleapis.com/auth/indexing"],
  null
);

// Perfrom auth and make multiple API calls
jwtClient.authorize(async function (err, tokens) {
  if (err) {
    console.log(err);
    return;
  }

  const options = {
    url: "https://indexing.googleapis.com/v3/urlNotifications:publish",
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: `Bearer ${tokens.access_token}`,
    },
    json: {
      url: "",
      type: "URL_UPDATED", // Means we want to request indexing
    },
  };

  try {
    const urls = await getUrls("https://www.rednafi.com/sitemap.xml");

    // There's a bulk endpoint but looping through the list
    // and making multiple requests is just as easy
    for (const url of urls) {
      options.json.url = url;
      const response = await fetch(options.url, {
        method: options.method,
        headers: options.headers,
        body: JSON.stringify(options.json),
      });
      const body = await response.json();
      console.log(body);
    }
  } catch (error) {
    console.error("Error:", error);
  }
});

Before executing the script, npm install googleapis and xml2js. Now running the script will give you an output similar to this:

{
  urlNotificationMetadata: {
    url: 'https://rednafi.com/categories/',
    latestUpdate: {
      url: 'https://rednafi.com/categories/',
      type: 'URL_UPDATED',
      notifyTime: '2023-05-27T01:02:35.537421311Z'
    }
  }
}
{
  urlNotificationMetadata: {
    url: 'https://rednafi.com/search/',
    latestUpdate: {
      url: 'https://rednafi.com/search/',
      type: 'URL_UPDATED',
      notifyTime: '2023-05-27T01:02:35.789809492Z'
    }
  }
},
...

Here, the getUrls function is defined to fetch the sitemap content from a specified URL, parse the XML content and extract the URLs. It uses the fetch function to retrieve the file, then uses xml2js to parse the XML and extract the URLs from the result.

The script then initializes an authentication client using the imported private key and specifies the required API scope. The authorize function is called to authenticate the client and obtain access tokens. Inside the authorization callback, the script prepares the necessary options for making API requests to the Google indexing API. It then calls the getUrls function to fetch the URLs from the sitemap.xml file. For each URL, it updates the options with the URL and makes a POST request to the Indexing API to request indexing. The response from the API is then logged into the console.

One thing to keep in mind is that by default, the daily request quota per project is 200. But you can request more quota if you need it.

Building a CORS proxy with Cloudflare Workers

Redowan Delowar — Sun, 21 May 2023 00:00:00 +0000

Cloudflare absolutely nailed the serverless function DX with Cloudflare Workers. However, I feel like it’s yet to receive widespread popularity like AWS Lambda since as of now, the service only offers a single runtime - JavaScript. But if you can look past that big folly, it’s a delightful piece of tech to work with. I’ve been building small tools with it for a couple of years but never got around to writing about the immense productivity boost it usually gives me whenever I need to quickly build and deploy a self-contained service.

Recently, I was doing some lightweight frontend work and needed to make some AJAX calls from one domain to another. Usually, browser’s CORS (Cross-Origin Resource Sharing) policy will get in your way if you try this. While you’re reading this piece, open the dev console and paste the following fetch snippet:

fetch("https://mozilla.org")
  .then((response) => response.text())
  .then((data) => {
    // Do something with the received data
    console.log(data);
  })
  .catch((error) => {
    // Handle any errors that occurred during the request
    console.error("Error:", error);
  });

This snippet will attempt to make a GET request from https://rednafi.com to https://mozilla.org. However, the client’s CORS policy won’t allow you to make an AJAX request like this and load external resources into the current site. On your console, you’ll see an error message like this:

Access to fetch at 'https://mozilla.org/' from origin 'https://rednafi.com'
has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is
present on the requested resource.

If an opaque response serves your needs, set the request's mode to
'no-cors' to fetch the resource with CORS disabled.

This is a good security measure. Without CORS, a malicious script could make a request to a server in another domain and access the resources that the user of the page is not intended to have access to. So much has been said and written about CORS that I won’t even attempt to explain it here. Here’s another high-level introduction to CORS.

CORS proxy #

While CORS is generally a good thing, it can be quite annoying when you’re trying to build something that needs access to external resources. In those cases, you’ll have to mess around with the origin server and add a few headers that the browser can understand before it allows you to load those external resources. But sometimes you don’t have access to the origin server or simply don’t want to deal with modifying the server’s response headers every time you need to access external resources. That’s where CORS proxies can come in handy.

A CORS proxy server acts as a bridge between your client and the target server. It receives your request and forwards it to the target server with a modified origin header so that the target server thinks the request is coming from the same origin as itself.

This way, you can bypass the same-origin policy of browsers and access resources from different domains. I usually use free proxies like cors.sh to bypass CORS restrictions. You can drop this snippet to your browser’s console and this time it’ll allow you to load the contents of https://mozilla.org from https://rednafi.com:

// Notice how we're prepending CORS URL before the target URL
fetch("https://proxy.cors.sh/https://mozilla.org")
  .then((response) => response.text())
  .then((data) => {
    // Do something with the received data
    console.log(data);
  })
  .catch((error) => {
    // Handle any errors that occurred during the request
    console.error("Error:", error);
  });

The target server’s response looks somewhat like this:

<!doctype html>
<html class="windows no-js" lang="en" dir="ltr" data-country-code="US"
data-latest-firefox="113.0.1" data-esr-versions="102.11.0"
data-gtm-container-id="GTM-MW3R8V" data-gtm-page-id="Homepage"
data-stub-attribution-rate="1.0" data-convert-project-id="10039-1003343">
<head>
...

If you want to learn more about how CORS proxies work, here’s a fantastic article on CORS proxies that explains the inner machinery in more detail.

Free proxy servers can be pernicious #

Using a free CORS proxy server can be dangerous as it might spill the beans on your requests and data to some random service you don’t know or trust. Since it plays as a middleman between your app and the resource you’re after, it could potentially snoop on, mess with, or keep tabs on your requests and data. Plus, some of those free CORS proxy servers might have restrictions on the size, type, or number of requests they can handle, or they might not even support HTTPS or other security bells and whistles.

Build your own CORS proxy with Cloudflare Workers #

With all the intros out of the way, here’s how CloudFlare Workers afforded me to prop up a CORS proxy in less than half an hour. If you’re impatient and just want to take a look at the service in its full glory then head over to the cors-proxy repo. GitHub Actions deploys the service automatically to CloudFlare Workers every time a change is pushed to the main branch.

Installing the prerequisites #

Assuming you have node installed on your system, you can fetch the wrangler CLI with the following command:

npm install -g wrangler

This will allow us to develop and test the service locally.

Bootstrapping the service #

Create a new directory where you want to develop your service and bring it under source control. Now, run:

npm create cloudflare@latest

The CLI will guide you through the entire bootstrapping process interactively. You’ll have to create a Cloudflare account (if you don’t have one already) and log into the dashboard. Then it’ll prompt you to deploy your first hello-world API endpoint that you can immediately start to play with without doing anything else. Being able to see the serverless function in action within like 5 minutes gave me a huge dopamine boost that AWS Lambda never could. You can see the interactive bootstrapping section here:

Complete CLI output…

using create-cloudflare version 2.0.7

╭ Create an application with Cloudflare Step 1 of 3
│
├ Where do you want to create your application?
│ dir cors-proxy
│
├ What type of application do you want to create?
│ type "Hello World" script
│
├ Do you want to use TypeScript?
│ typescript no
│
├ Copying files from "simple" template
│
╰ Application created

╭ Installing dependencies Step 2 of 3
│
├ Installing dependencies
│ installed via `npm install`
│
╰ Dependencies Installed

╭ Deploy with Cloudflare Step 3 of 3
│
├ Do you want to deploy your application?
│ yes deploying via `npm run deploy`
│
├ Logging into Cloudflare This will open a browser window
│ allowed via `wrangler login`
│
├ Deploying your application
│ deployed via `npm run deploy`
│
├  SUCCESS  View your deployed application at
│  https://cors-proxy.rednafi.workers.dev (this may take a few mins)
│
│ Run the development server npm run dev
│ Deploy your application npm run deploy
│ Read the documentation https://developers.cloudflare.com/workers
│ Stuck? Join us at https://discord.gg/cloudflaredev
│
╰ See you again soon!

Running the interactive session will create the following directory structure:

├── src
│   └── worker.js
├── package-lock.json
├── package.json
└── wrangler.toml

Developing the CORS proxy #

We’ll write our proxy server in src/worker.js file. Copy the following JS snippet and paste it to the file:

export default {
  async fetch(request, env, ctx) {
    // Extract method, url and headers from the incoming request object.
    const { method, url, headers } = request;

    // Extract destination url from the query string.
    const destUrl = new URL(url).searchParams.get("url");

    // If the destination url is not present, return 400.
    if (!destUrl) {
      return new Response("Missing destination URL.", { status: 400 });
    }

    // If the request method is OPTIONS, return CORS headers.
    if (
      method === "OPTIONS" &&
      headers.has("Origin") &&
      headers.has("Access-Control-Request-Method")
    ) {
      const responseHeaders = {
        "Access-Control-Allow-Origin": headers.get("Origin"),
        "Access-Control-Allow-Methods": "*", // Allow all methods
        "Access-Control-Allow-Headers": headers.get(
          "Access-Control-Request-Headers"
        ),
        "Access-Control-Max-Age": "86400",
      };
      return new Response(null, { headers: responseHeaders });
    }

    const proxyRequest = new Request(destUrl, {
      method,
      headers: {
        ...headers,
        Origin: "",
      },
    });

    try {
      const response = await fetch(proxyRequest);
      const responseHeaders = new Headers(response.headers);
      responseHeaders.set("Access-Control-Allow-Origin", "*");
      responseHeaders.set("Access-Control-Allow-Credentials", "true");
      responseHeaders.set("Access-Control-Allow-Methods", "*");

      return new Response(response.body, {
        status: response.status,
        statusText: response.statusText,
        headers: responseHeaders,
      });
    } catch (error) {
      return new Response("Error occurred while fetching the resource.", {
        status: 500,
      });
    }
  },
};

The first section of the code deals with extracting relevant information from the incoming request. It destructures the method, url, and headers properties from the request object, which represents the client’s request.

Next, it extracts the destination URL from the query string. It extracts the URL parameter using the searchParams.get() method. If the destination URL isn’t provided, the function returns a Response object with an error message and a status code of 400 (Bad Request).

The code then checks if the request method is OPTIONS. The OPTIONS method is used in CORS preflight requests to determine if the actual request is safe to send. If the request is an OPTIONS request and contains specific headers indicating a CORS preflight request (Origin and Access-Control-Request-Method), the function generates a response with appropriate CORS headers. The response headers include Access-Control-Allow-Origin to reflect the client’s origin, Access-Control-Allow-Methods set to *, allowing any HTTP method, Access-Control-Allow-Headers based on the requested headers, and Access-Control-Max-Age set to 86400 seconds (one day) to cache the preflight response.

If the request is not an OPTIONS request or doesn’t meet the CORS preflight conditions, the code continues execution. It creates a new Request object named proxyRequest uses the extracted destination URL and sets the method and headers of the original request. The Origin header is removed to prevent CORS restrictions when forwarding the request.

The subsequent code performs the actual request forwarding. It uses fetch to send the proxyRequest to the destination URL. If the fetch is successful, the code proceeds to process the response. It creates a new Headers object from the response’s headers and modifies them to include the necessary CORS headers.

Finally, the function constructs a Response object using the response body, status, statusText, and modified headers. If an error occurs during the fetch operation, the code catches the error and returns a Response object with an error message and a status code of 500 (Internal Server Error).

Once you’ve pasted the snippet, you can redeploy the service from your local machine with:

wrangler deploy

This will deploy the service immediately:

 ⛅️ wrangler 3.0.0
------------------
Total Upload: 1.52 KiB / gzip: 0.57 KiB
Uploaded cors-proxy (0.51 sec)
Published cors-proxy (0.38 sec)
  https://<your-deployed-service>
Current Deployment ID: f300ac99-c15e-4e30-a910-a56d81c10b95

I’ve removed my root domain from the above output since I’m using the free version of Workers and don’t want people to exhaust my free request quota. Haha, security by obscurity! But once you’ve deployed your proxy server, you can go to the following URL from your browser:

https://<your-deployed-service>/?url=https://mozilla.com

This will send you to the Mozilla website through the deployed function. Now you can use it just like the free CORS proxy. Try it out by dropping the following snippet to your browser console. This is exactly the same as the previous fetch snippet but the only difference is this time, we’re using our own proxy server that we control:

// Notice how we're prepending CORS URL before the target URL
fetch("https://<your-deployed-service>?url=https://mozilla.org")
  .then((response) => response.text())
  .then((data) => {
    // Do something with the received data
    console.log(data);
  })
  .catch((error) => {
    // Handle any errors that occurred during the request
    console.error("Error:", error);
  });

Don’t forget to replace the <your-deployed-service> URL with your own service. This will result in a successful request. You can also interactively send requests to the destination URLs via the Cloudflare Workers dashboard. Go to your Cloudflare dashboard, head over to the Workers section, and select your deployed serverless function:

Deploying the service with GitHub Actions #

For one-off services, wrangler deploy in the local machine works perfectly but I usually don’t consider a project fully done until I’ve automated away the whole process. So, I wrote a quick GitHub Actions workflow to run the linters and deploy the service automatically when a new commit is pushed to the main branch. Here’s how it looks:

# .github/workers/ci.yml

name: Deploy

on:
  push:
    branches:
      - main

  # Allow running this workflow manually.
  workflow_dispatch:


jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: "lts/*"
          cache: npm
          cache-dependency-path: cors-proxy/package-lock.json
      - name: Install dependencies
        working-directory: ./cors-proxy
        run: |
          npm install
      - name: Run linter
        working-directory: ./cors-proxy
        run: |
          npx prettier --check .

  deploy:
    runs-on: ubuntu-latest
    needs: build
    steps:
      - uses: actions/checkout@v3
      - name: Publish
        uses: cloudflare/wrangler-action@2.0.0
        with:
          apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
          workingDirectory: "cors-proxy"

For this to work, you’ll need to create a Cloudflare API token and add it to the GitHub Secrets of your proxy server’s repository. Here’s the complete CI workflow file.

Fixed-time job scheduling with UNIX 'at' command

Redowan Delowar — Sun, 14 May 2023 00:00:00 +0000

This weekend, I was working on a fun project that required a fixed-time job scheduler to run a curl command at a future timestamp. I was aiming to find the simplest solution that could just get the job done. I’ve also been exploring Google Bard recently and wanted to see how it stacks up against other LLM tools like ChatGPT, BingChat, or Anthropic’s Claude in terms of resolving programming queries.

So, I asked Bard:

What’s the simplest solution I could get away with to run a shell command at a future datetime?

It introduced me to the UNIX at command that does exactly what I needed. Cron wouldn’t be a good fit for this particular use case, and I wasn’t aware of the existence of at before. So I started probing the model and wanted to document my findings for future reference. Also, the final hacky solution that allowed me to schedule jobs remotely can be found at the end of this post.

The insipid definition #

The command at in UNIX is used to schedule one-time jobs or commands to be executed at a specific time in the future. Internally, the system maintains a queue that adds a new entry when a job is scheduled, and once it gets executed, the job is removed from the queue.

NOTE: By default, the jobs will be scheduled using the targeted machine’s local timezone.

Prerequisites #

The command isn’t included in GNU coreutils, so you might have to install it separately on your machine.

Debian-ish #

On a Debian-flavored Linux machine, run:

apt install at

Then check the status of atd daemon. This daemon executes the scheduled jobs.

service atd status

* atd is running

If the service isn’t running, then you can start the daemon with this command:

service atd start

 * Starting deferred execution scheduler atd    [OK]

MacOS #

On MacOS, scheduled jobs are carried out by atrun and it’s disabled by default. I had to fiddle around quite a bit to make it work on my MacBook Pro running MacOS Ventura. First, you’ll need to launch the daemon with the following command:

sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.atrun.plist

This will start the atrun daemon. Or enable it for future bootups by modifying /System/Library/LaunchDaemons/com.apple.atrun.plist to have:

...
<key>Enabled</key>
...

On modern MacOS like Ventura, unfortunately, this requires disabling SIP. Next, you’ll need to provide full disk access to atrun. To do so:

Open Spotlight and type in Allow full disk access.
On the left panel, click on Allow applications to access all user files.

On the right panel, add /usr/libexec/atrun to the list of allowed apps. Press cmd + shift + g and type in the full path of atrun.

You can learn more about making atrun work on MacOS. Although I’m using MacOS for development, In my particular case, making at work on MacOS wasn’t the first priority because I deployed the final solution to an Ubuntu container.

A few examples #

The following sections demonstrates some examples of scheduling commands to be executed in a few different scenarios.

Schedule at a specific time #

To schedule a command to be executed at a specific time, use this command syntax:

at <time> <command>

For example, to schedule the command ls -lah >> foo.txt to be executed at 3:00 PM local time, you’d use the following command:

at 3pm
at> ls -lah >> foo.txt
at> <Ctrl-D>

Pressing <Ctrl-D> tells at that you have finished entering the command, and it should schedule the job to run at the specified time. You’ll see that at 3.00PM local time, a file named foo.txt containing the output of ls -l will be created.

Schedule after a certain period of time #

To schedule a command to run in a specific amount of time from now, use:

at now + <time> <command>

For example, to schedule ps aux >> foo.txt to run in 2 minutes from now, you’d use the following command:

at now + 2 minutes
at> ps aux >> foo.txt
at> <Ctrl-D>

This will schedule the command to run in two minutes in the current local time.

Schedule a script run #

You can also run a script containing multiple commands at a specific time. To do this, create a script file that houses the commands you want to run, and then use at to schedule the script to be executed at the desired time.

For example, suppose you have a script file called script.sh that contains a curl command which makes an API call and saves the output to a file. You can schedule it as such:

#!/usr/bin/env bash

curl -X GET https://httpbin.org/get >> foo.json

at -f script.sh now + 1 minute

The script will be executed in a minute from now. You can check the content of foo.json 1 minute later:

{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.85.0",
    "X-Amzn-Trace-Id": "Root=1-646162a8-71a232d563e0c16a4a497acf"
  },
  "origin": "74.140.2.169",
  "url": "https://httpbin.org/get"
}

Schedule in a non-interactive manner #

What if you don’t want to create a new script file and also don’t want to schedule a command interactively as shown before? You can echo the desired command and pipe it to at like this:

echo "dig +short rednafi.com >> foo.txt" | at now + 1 minute

We can also run multi-line commands in a single go by taking advantage of the heredoc format:

at now + 1 minute <<EOF
dig +short rednafi.com >> foo.txt
EOF

In either case, 1 minute later, you’ll see that a foo.txt file will be created in your local directory with the following content:

185.199.111.153
185.199.108.153
185.199.109.153
185.199.110.153

This command above uses at to schedule the execution of a dig command for the domain name rednafi.com. In this case, dig performs a DNS lookup, and the scheduled time is set to be 1 minute from now in the current local time. The output of the command is then appended to the file foo.txt. The <<EOF syntax is used for input redirection, which allows the command to be specified in a heredoc format without requiring you to enter the command in interactive mode as before.

Schedule with UNIX timestamp #

You can schedule jobs using a UNIX timestamp with the -t flag. The at command requires a timestamp in the format [[[mm]dd]HH]MM[[cc]yy][.ss]]. Here’s an example that uses the date command to generate the current datetime, adds a 30-second offset to it, formats it to the at’s expected format, and schedules a job.

On Linux, run:

at -t $(date -d "+30 seconds" +"%Y%m%d%H%M.%S") <<EOF
ping -c 5 rednafi.com >> foo.txt
EOF

On MacOS, run:

at -t $(date -v "+30S" +"%Y%m%d%H%M.%S") <<EOF
ping -c 5 rednafi.com >> foo.txt
EOF

View and manage scheduled jobs #

To view a list of scheduled jobs, use the following command:

atq

This will display a list of all the tasks that are currently scheduled.

36      Sun May 14 18:42:00 2023
37      Sun May 14 18:42:00 2023

To remove a scheduled task, use the following command:

atrm <job number>

The job number is the number assigned to the task when it was scheduled. You can find the job number by running the atq command. If you need to clear all the pending jobs, use this:

atrm $(atq | cut -f 1)

A hacky way to schedule jobs remotely #

This is a hacky and probably dangerous way to do remote job scheduling. However, the beauty of side projects is that nobody’s here to tell you what to do and it’s a fun way to play with hazmats.

I needed a way to quickly prop up a service that’d allow me to schedule webhook API calls at a fixed point in time in the future. So I exposed a simple NodeJS server that’d allow me to schedule an API call with at and execute the command at the desired datetime. Here’s the complete server:

// server.js
import express, { json } from "express";
import { exec } from "child_process";

const app = express();
const port = 3000;
const authToken = "some-token";

app.use(json());

app.post("/run-command", (req, res) => {
  const { command } = req.body;

  if (!command) {
    return res.status(400).json({ error: "Command not provided." });
  }

  const authHeader = req.headers.authorization;
  if (!authHeader || authHeader !== `Bearer ${authToken}`) {
    return res.status(401).json({ error: "Unauthorized." });
  }

  exec(command, (error, stdout, stderr) => {
    if (error) {
      return res
        .status(500)
        .json({ msg: "Command execution failed.", error: stderr });
    }
    res.json({ msg: "Command execution successful.", output: stdout });
  });
});

app.listen(port, () => {
  console.log(`Server listening on port ${port}`);
});

This endpoint takes in a shell command and just runs it on the server; bad idea, right? But the endpoint is secured behind a Bearer token and I’m the only one who’s going to use this. Security by obscurity!

Before running the server, you’ll need to install express and once you’ve done it, you can start the server with the following command:

node server.js

Now, from a different console panel, you can schedule a remote task as follows:

curl -X POST -H "Authorization: Bearer some-token" \
  -H "Content-Type: application/json" \
  --data '{"command":"echo ping -c 5 google.com | at now +1min"}' \
  http://localhost:3000/run-command

This will return:

{"msg":"Command execution successful.","output":""}

In my case, I needed to POST a payload at a certain time in the future:

curl -X POST -H "Authorization: Bearer some-token" \
     -H "Content-Type: application/json" \
     --data "{\"command\":\"echo \\\"curl -X POST \
     https://webhook.site/d667acd3-477f-453c-9375-0dcbb51703bd \
     -H 'Content-Type: application/json' --data \
     '{\\\"hello\\\": \\\"world\\\"}'\\\" | at now +1min\"}" \
     http://localhost:3000/run-command

Periodic readme updates with GitHub Actions

Redowan Delowar — Thu, 04 May 2023 00:00:00 +0000

I recently gave my personal blog a fresh new look and decided it was time to spruce up my GitHub profile’s landing page as well. GitHub has a special way of treating the README.md file of your repo, displaying its content as the landing page for your profile. My goal was to showcase a brief introduction about myself and my work, along with a list of the five most recent articles on my blog. Additionally, I wanted to ensure that the article list stayed up to date.

There are plenty of fancy GitHub Action workflows like the blog-post-workflow that allow you to add your site’s URL to the CI file and it’ll periodically fetch the most recent content from the source and update the readme file. However, I wanted to make a simpler version of it from scratch which can be extended for periodically updating any Markdown file in any repo, just not the profile readme. So, here’s the plan:

A custom GitHub Action workflow will periodically run a nodejs script.
The script will then:
- Grab the XML index of this blog that you’re reading.
- Parse the XML content and extract the URLs and publication dates of 5 most recent articles.
- Update the associated Markdown table with the extracted content on the profile’s README.md file.
Finally, the workflow will commit the changes and push them to the profile repo. You can see the final outcome in the profile repository.

Here’s the script that performs the above steps:

// importBlogs.js
/* Import the latest 5 blog posts from rss feed */

import fetch from "node-fetch";
import { Parser } from "xml2js";
import { promises } from "fs";

const rssUrl = "https://rednafi.com/index.xml";

const header = `<div align="center">
    Introducing myself...
<div>\n\n`;

const outputFile = "README.md";
const parser = new Parser();

// Define an async function to get and parse the rss data
async function getRssData() {
  try {
    const res = await fetch(rssUrl);
    const data = await res.text();
    return await parser.parseStringPromise(data);
  } catch (err) {
    console.error(err);
  }
}

// Define an async function to write the output file
async function writeOutputFile(output) {
  try {
    await promises.writeFile(outputFile, output);
    console.log(`Saved ${outputFile}`);
  } catch (err) {
    console.error(err);
  }
}

// Call the async functions
getRssData()
  .then((result) => {
    // Get the first five posts from the result object
    const posts = result.rss.channel[0].item.slice(0, 5);

    // Initialize an empty output string
    let output = "";

    // Add a title to the output string
    output += header;

    // Add a header row to the output string
    output += `#### Recent articles\n\n`;
    output += "| Title | Published On |\n";
    output += "| ----- | ------------ |\n";

    // Loop through posts and add a row for each to output
    for (let post of posts) {
      // Strip the time from the pubDate
      const date = post.pubDate[0].slice(0, 16);
      output += `| [${post.title}](${post.link}) | ${date} |\n`;
    }
    // Call the writeOutputFile function with the output string
    writeOutputFile(output);
  })
  .catch((err) => {
    // Handle the error
    console.error(err);
  });

The snippet above utilizes node-fetch to make HTTP calls,xml2js for XML parsing, and the built-in fs module’s promises for handling file system operations.

Next, it defines an async function getRssData responsible for fetching the XML data from the [https://rednafi.com/index.xml] URL. It extracts the blog URLs and publication dates, and returns the parsed data as a list of objects. Another async function, writeOutputFile, writes the parsed XML content as a Markdown table and saves it to the README.md file.

The script is executed by the following GitHub Action workflow every day at 0:00 UTC. Before the CI runs, make sure you create a new Action Secret that houses a personal access token with write access to the repo where the CI runs.

# Run a bash script to randomly generate empty commit to this repo.
name: CI

on:
  # Don't run on push; this CI pushes (would cause infinite loop)
  # push: [ main ]

  # Add a schedule to run the job every day at 0:00 UTC
  schedule:
    - cron: "0 0 * * *"

  # Allow running this workflow manually
  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repo
        uses: actions/checkout@v4
        with:
          # Required for pushing refs to destination repository
          fetch-depth: 0
          ref: ${{ github.head_ref }}
          token: ${{ secrets.ACCESS_TOKEN }}

      - uses: actions/setup-node@v3
        with:
          node-version: "lts/*"
          cache: npm
          cache-dependency-path: package-lock.json

      - name: Install dependencies
        run: |
          npm install

      - name: Run linter
        run: |
          npx prettier --write .

      - name: Run script
        run: |
          node scripts/importBlogs.js

      - name: Commit changes
        run: |
          git config --local user.name \
            "github-actions[bot]"
          git config --local user.email \
            "41898282+github-actions[bot]@users.noreply.github.com"
          git add .
          git diff-index --quiet HEAD \
            || git commit -m "Autocommit: updated at $(date -u)"

      - name: Push changes
        uses: ad-m/github-push-action@master
        with:
          force_with_lease: true

In the first four steps, the workflow checks out the codebase, sets up nodejs, installs the dependencies, and then runs prettier on the scripts. Next, it executes the importBlogs.js script. The script updates the README and the subsequent shell commands commit the changes to the repo. The following line ensures that we’re only trying to commit when there’s a change in the tracked files.

git diff-index --quiet HEAD \
  || git commit -m "Autocommit: updated at $(date -u)"

Then in the last step, we use an off-the-shelf workflow to push our changes to the repo. Check out the workflow directory of my profile’s repo to see the whole setup in action. I’m quite satisfied with the final output:

Building a web app to display CSV file stats with ChatGPT & Observable

Redowan Delowar — Mon, 10 Apr 2023 00:00:00 +0000

Whenever I plan to build something, I spend 90% of my time researching and figuring out the idiosyncrasies of the tools that I decide to use for the project. LLM tools like ChatGPT has helped me immensely in that regard. I’m taking on more tangential side projects because they’re no longer as time-consuming as they used to be and provide me with an immense amount of joy and learning opportunities. While LLM interfaces like ChatGPT may hallucinate, confabulate, and confidently give you misleading information, they also allow you to avoid starting from scratch when you decide to work on something. Personally, this benefits me enough to keep language models in my tool belt and use them to churn out more exploratory work at a much faster pace.

For some strange reason, I never took the time to explore ObservableHQ, despite knowing what it does and how it can help me quickly build nifty client-side tools without going through the hassle of containerizing and deploying them as dedicated applications. So, I asked ChatGPT to build me a tool that would allow me to:

Upload two CSV files
Calculate the row and column counts from the files
Show the number of rows and columns in a table and include the headers of the columns and their corresponding index numbers, so that you can compare them easily.

Here’s the initial prompt that I used:

Give me the JavaScript code for an Observable notebook that’ll allow me to upload a CSV file, calculate the row and column counts from it, and then display the stats with column headers and their corresponding index starting from 0. Display the info in an HTML table.

Then I refactored the JavaScript it returned so that it’ll allow me to upload two CSV files and compare their stats. I made ChatGPT do it for me with this follow-up prompt:

Can you change the code so that it allows uploading two CSV files and displays the stats of both of them in two HTML tables? Don’t blindly repeat the logic from the previous section twice.

Finally, I asyncified the code and changed some HTML parsing to make the table look a bit better. Here’s the complete 85-line code snippet:

{
  // create file input elements for the two files
  const fileInput1 = html`<input type="file" />`;
  const fileInput2 = html`<input type="file" />`;

  // create empty HTML tables for the two files
  const table1 = html` <table>
    <thead>
      <tr>
        <th></th>
      </tr>
    </thead>
    <tbody></tbody>
  </table>`;
  const table2 = table1.cloneNode(true);

  // function to handle file load event and display stats in table
  const handleFileLoad = (table) => async (event) => {
    const file = event.target.files[0];
    const reader = new FileReader();

    // read the file contents as text
    reader.readAsText(file);

    // create a promise to wait for the file to load and parse
    const fileLoaded = new Promise((resolve, reject) => {
      reader.onload = () => {
        const contents = reader.result;
        const lines = contents.trim().split("\n");
        const headers = lines[0].split(",");
        const numColumns = headers.length;
        const numRows = lines.length - 1;

        // create a row for the number of rows
        const numRowsRow = html`<tr>
          <td>Number of rows:</td>
          <td>${numRows}</td>
        </tr>`;

        // create a row for the number of columns
        const numColsRow = html`<tr>
          <td>Number of columns:</td>
          <td>${numColumns}</td>
        </tr>`;

        // create a row for the column names
        const headerRow = html`<tr>
          <td>Column names:</td>
          <td>${headers.map((h, i) => `${i}: ${h}`).join(", ")}</td>
        </tr>`;

        // add the rows to the table body
        const tableBody = html`<tbody>
          ${numRowsRow}${numColsRow}${headerRow}
        </tbody>`;

        table.replaceChild(tableBody, table.lastChild);

        // resolve the promise with the parsed data
        resolve({ numRows, numColumns, headers });
      };

      reader.onerror = () => {
        reject(reader.error);
      };
    });

    // Wait for promise before displaying results in table
    try {
      const { numRows, numColumns, headers } = await fileLoaded;
      console.log(
        `File loaded: ${file.name},
        Rows: ${numRows}, Columns: ${numColumns}, Headers: ${headers}`
      );
    } catch (err) {
      console.error(err);
    }
  };

  // add event listeners to the file input elements
  fileInput1.addEventListener("change", handleFileLoad(table1));
  fileInput2.addEventListener("change", handleFileLoad(table2));

  // display the file input and table elements in the notebook
  return html`${fileInput1} ${table1} ${fileInput2} ${table2}`;
}

The snippet above starts by creating two file input elements using HTML input tags. These are used to allow the user to select and upload CSV files. Two empty HTML tables are also created to hold the extracted statistics for each CSV file.

Next, it defines a function called handleFileLoad which takes a table element as its argument. This function is called when the user uploads a file, and it reads the contents of the file and extracts some basic statistics from it. These statistics are then used to populate the HTML table with the extracted information.

Inside the handleFileLoad function, the FileReader API is used to read the contents of the uploaded file. The file contents are then parsed as text and split into lines. The first line contains the column headers, which are extracted by splitting the line by commas. The number of columns is then determined by the number of headers, and the number of rows is determined by counting the number of lines in the file (excluding the header).

It then creates three rows for the extracted statistics: one row for the number of rows, one row for the number of columns, and one row for the column headers with their corresponding indexes starting from zero. The rows are then added to the HTML table.

Finally, the code adds event listeners to the file input elements to trigger the handleFileLoad function when the user uploads a file. The file input elements and HTML tables are then returned as an HTML fragment using the HTML template literal, and displayed in the notebook.

You can find the working application embedded in the following section. Try uploading two CSV files by clicking on the Choose File button and see how the app displays the stats in separate HTML tables.

Here’s a gif of it in action:

Click on the following thumbnail to take the notebook for a spin: