The current title of this post is probably incorrect and may even be misleading. I had a hard time coming up with a suitable name for it. But the idea goes like this: sometimes you might find yourself in a situation where you need to iterate through a generator more than once. Sure, you can use an iterable like a tuple or list to allow multiple iterations, but if the number of elements is large, that’ll cause an OOM error....
One of my favorite pastimes these days is to set BingChat to creative mode, ask it to teach me a trick about topic X, and then write a short blog post about it to reinforce my understanding. Some of the things it comes up with are absolutely delightful. In the spirit of that, I asked it to teach me a Shell trick that I can use to mimic maps or dictionaries in a shell environment....
I needed to compare two large directories with thousands of similarly named PDF files and find the differing filenames between them. In the first pass, this is what I did: Listed out the content of the first directory and saved it in a file: ls dir1 > dir1.txt Did the same for the second directory: ls dir2 > dir2.txt Compared the difference between the two outputs: diff dir1.txt dir2.txt This returned the name of the differing files likes this:...
Whenever I need to whip up a quick command line tool, my go-to is usually Python. Python’s CLI solutions tend to be more robust than their Shell counterparts. However, dealing with its portability can sometimes be a hassle, especially when all you want is to distribute a simple script. That’s why while toying around with argparse to create a dynamic menu, I decided to ask ChatGPT if there’s a way to achieve the same using native shell scripting....
When writing shell scripts, I’d often resort to using hardcoded ANSI escape codes1 to format text, such as: #!/usr/bin/env bash BOLD="\033[1m" UNBOLD="\033[22m" FG_RED="\033[31m" BG_YELLOW="\033[43m" BG_BLUE="\033[44m" RESET="\033[0m" # Print a message in bold red text on a yellow background. echo -e "${BOLD}${FG_RED}${BG_YELLOW}This is a warning message${RESET}" # Print a message in white text on a blue background. echo -e "${BG_BLUE}This is a debug message${RESET}" This shell snippet above shows how to add text formatting and color to shell script output via ANSI escape codes....
I was working on a project where I needed to poll multiple data sources and consume the incoming data points in a single thread. In this particular case, the two data streams were coming from two different Redis lists. The correct way to consume them would be to write two separate consumers and spin them up as different processes. However, in this scenario, I needed a simple way to poll and consume data from one data source, wait for a bit, then poll and consume from another data source, and keep doing this indefinitely....
Consider this iterable: it = (1, 2, 3, 0, 4, 5, 6, 7) Let’s say you want to build another iterable that includes only the numbers that appear starting from the element 0. Usually, I’d do this: # This returns (0, 4, 5, 6, 7). from_zero = tuple(elem for idx, elem in enumerate(it) if idx >= it.index(0)) While this is quite terse and does the job, it won’t work with a generator....
The colon : command is a shell utility that represents a truthy value. It can be thought of as an alias for the built-in true command. You can test it by opening a shell script and typing a colon on the command line, like this: : If you then inspect the exit code by typing $? on the command line, you’ll see a 0 there, which is exactly what you’d see if you had used the true command....
I’ve just migrated from Ubuntu to macOS for work and am still in the process of setting up the machine. I’ve been a lifelong Linux user and this is the first time I’ve picked up an OS that’s not just another flavor of Debian. Primarily, I work with Python, NodeJS, and a tiny bit of Go. Previously, any time I had to install these language runtimes, I’d execute a bespoke script that’d install:...
TIL that you can specify update_fields while saving a Django model to generate a leaner underlying SQL query. This yields better performance while updating multiple objects in a tight loop. To test that, I’m opening an IPython shell with python manage.py shell -i ipython command and creating a few user objects with the following lines: In [1]: from django.contrib.auth import User In [2]: for i in range(1000): ...: fname, lname = f'foo_{i}', f'bar_{i}' ....
TIL that returning a value from a function in bash doesn’t do what I thought it does. Whenever you call a function that’s returning some value, instead of giving you the value, Bash sets the return value of the callee as the status code of the calling command. Consider this example: #!/usr/bin/bash # script.sh return_42() { return 42 } # Call the function and set the return value to a variable....
While reading the second version of Brian Okken’s pytest book1, I came across this neat trick to compose multiple levels of fixtures. Suppose, you want to create a fixture that returns some canned data from a database. Now, let’s say that invoking the fixture multiple times is expensive, and to avoid that you want to run it only once per test session. However, you still want to clear all the database states after each test function runs....
Whenever your local branch diverges from the remote branch, you can’t directly pull from the remote branch and merge it into the local branch. This can happen when, for example: You checkout from the main branch to work on a feature in a branch named alice. When you’re done, you merge alice into main. After that, if you try to pull the main branch from remote again and the content of the main branch changes by this time, you’ll encounter a merge error....
I run git log --oneline to list out the commit logs all the time. It prints out a compact view of the git history. Running the command in this repo gives me this: d9fad76 Publish blog on safer operator.itemgetter, closes #130 0570997 Merge pull request #129 from rednafi/dependabot/... 6967f73 Bump actions/setup-python from 3 to 4 48c8634 Merge pull request #128 from rednafi/dependabot/pip/mypy-0.961 5b7a7b0 Bump mypy from 0.960 to 0.961 However, there are times when I need to list out the commit logs that only represent the changes made to a particular file....
While working on a project with EdgeDB1 and FastAPI2, I wanted to perform health checks against the FastAPI server in the GitHub CI. This would notify me about the working state of the application. The idea is to: Run the server in the background. Run the commands against the server that’ll denote that the app is in a working state. Perform cleanup. Exit with code 0 if the check is successful, else exit with code 1....
To avoid instantiating multiple DB connections in Python apps, a common approach is to initialize the connection objects in a module once and then import them everywhere. So, you’d do this: # src.py import boto3 # Pip install boto3 import redis # Pip install redis dynamo_client = boto3.client("dynamodb") redis_client = redis.Redis() However, this adds import time side effects to your module and can turn out to be expensive. In search of a better solution, my first instinct was to go for functools....
How’d you create a sub dictionary from a dictionary where the keys of the sub-dict are provided as a list? I was reading a tweet1 by Ned Bachelder on this today and that made me realize that I usually solve it with O(DK) complexity, where K is the length of the sub-dict keys and D is the length of the primary dict. Here’s how I usually do that without giving it any thoughts or whatsoever:...
I used to use Unittest’s self.assertTrue / self.assertFalse to check both literal booleans and truthy/falsy values in Unittest. Committed the same sin while writing tests in Django. I feel like assertTrue and assertFalse are misnomers. They don’t specifically check literal booleans, only truthy and falsy states respectively. Consider this example: # src.py import unittest class TestFoo(unittest.TestCase): def setUp(self): self.true_literal = True self.false_literal = False self.truthy = [True] self.falsy = [] def is_true(self): self....
To check whether an integer is a power of two, I’ve deployed hacks like this: def is_power_of_two(x: int) -> bool: return x > 0 and hex(x)[-1] in ("0", "2", "4", "8") While this works1, I’ve never liked explaining the pattern matching hack that’s going on here. Today, I came across this tweet2 by Raymond Hettinger where he proposed an elegant solution to the problem. Here’s how it goes: def is_power_of_two(x: int) -> bool: return x > 0 and x....
Django Rest Framework exposes a neat hook to customize the response payload of your API when errors occur. I was going through Microsoft’s REST API guideline1 and wanted to make the error response of my APIs more uniform and somewhat similar to this2. I’ll use a modified version of the quickstart example3 in the DRF docs to show how to achieve that. Also, we’ll need a POST API to demonstrate the changes better....