This post started as a pair of quick answers to questions on r/golang. The first was about whether a repository layer on top of sqlc is worth it. The second was about how to handle transactions when the interface hides storage details. Both turned into short shards on this site. This post ties them together and covers what to do when transactions need to span multiple repositories. It walks through three stages, each building on the last: ...
Previously, I showed how to put a small interface between your service logic and your storage layer so the service doesn’t know whether it’s talking to sqlc, raw SQL, or anything else. The interface looked like this: // book/book.go type Store interface { Get(ctx context.Context, id int64) (Book, error) Create(ctx context.Context, b Book) (int64, error) } A service depends on Store, a concrete postgres package satisfies it, and in tests you swap in an in-memory fake. The service never imports database/sql. ...
Today in r/golang, user Leading-West-4881 asked: Is a repository layer over sqlc over-engineering or necessary for scale? I’m building a notification engine in Go using sqlc for the DB layer. Do you just inject *db.Queries into your services, or do you find the abstraction of a repository layer worth the extra code? I attempted to answer it there and the gist is correct. But I wrote it in a hurry so the example and the explanation could be better. Capturing it properly here. ...
Recently at work, we ran into this problem: We needed to send Slack notifications for specific events but had to enforce rate limits to avoid overwhelming the channel. Here’s how the limits worked: Global limit: Max 100 requests every 30 minutes. Category limit: Each event type (e.g., errors, warnings) capped at 10 requests per 30 minutes. Now, imagine this: There are 20 event types. Each type hits its 10-notification limit in 30 minutes. That’s 200 requests total, but the global limit only allows 100. So, 100 requests must be dropped - even if some event types still have room under their individual caps. This created a hierarchy of limits: ...
I needed a way to sort a Django queryset based on a custom sequence of an attribute. Typically, Django allows sorting a queryset by any attribute on the model or related to it in either ascending or descending order. However, what if you need to sort the queryset following a custom sequence of attribute values? Suppose, you’re working with a model called Product where you want to sort the rows of the table based on a list of product ids that are already sorted in a particular order. Here’s how it might look: ...
I was working on a project where I needed to poll multiple data sources and consume the incoming data points in a single thread. In this particular case, the two data streams were coming from two different Redis lists. The correct way to consume them would be to write two separate consumers and spin them up as different processes. However, in this scenario, I needed a simple way to poll and consume data from one data source, wait for a bit, then poll and consume from another data source, and keep doing this indefinitely. That way I could get away with doing the whole workflow in a single thread without the overhead of managing multiple processes. ...
I was working with a table that had a similar (simplified) structure like this: | uuid | file_path | |----------------------------------|---------------------------| | b8658dfc3e80446c92f7303edf31dcbd | media/private/file_1.pdf | | 3d750874a9df47388569a23c559a4561 | media/private/file_2.csv | | d177b7f7d8b046768ab65857451a0354 | media/private/file_3.txt | | df45742175d7451dad59761f15653d9d | media/private/image_1.png | | a542966fc193470dab84351c15523042 | media/private/image_2.jpg | Let’s say the above table is represented by the following Django model: from django.db import models class FileCabinet(models.Model): uuid = models.UUIDField( primary_key=True, default=uuid.uuid4, editable=False ) file_path = models.FileField(upload_to="files/") I needed to extract the file names with their extensions from the file_path column and create new paths by adding the prefix dir/ before each file name. This would involve stripping everything before the file name from a file path and adding the prefix, resulting in a list of new file paths like this: ['dir/file_1.pdf', ..., 'dir/image_2.jpg']. ...
Django has a Model.objects.bulk_update method that allows you to update multiple objects in a single pass. While this method is a great way to speed up the update process, oftentimes it’s not fast enough. Recently, at my workplace, I found myself writing a script to update half a million user records and it was taking quite a bit of time to mutate them even after leveraging bulk update. So I wanted to see if I could use multiprocessing with .bulk_update to quicken the process even more. Turns out, yep I can! ...
While going through the documentation of Python’s sqlite3 module, I noticed that it’s quite API-driven, where different parts of the module are explained in a prescriptive manner. I, however, learn better from examples, recipes, and narratives. Although a few good recipes already exist in the docs, I thought I’d also enlist some of the examples I tried out while grokking them. Executing individual statements To execute individual statements, you’ll need to use the cursor_obj.execute(statement) primitive. ...
Python has a random.choice routine in the standard library that allows you to pick a random value from an iterable. It works like this: # src.py import random # The seed ensures that you'll get the same random choice # every time you run the script. random.seed(90) # This builds a list: ["choice_0", "choice_1", ..., "choice_9"] lst = [f"choice_{i}" for i in range(10)] print(random.choice(lst)) print(random.choice(lst)) This will print: choice_3 choice_1 I was looking for a way to quickly hydrate a table with random data in an SQLite database. To be able to do so, I needed to extract unpremeditated values from an array of predefined elements. The issue is, that SQLite doesn’t support array types or have a built-in function to pick random values from an array. However, recently I came across this trick from Ricardo Ander-Egg’s tweet, where he exploits SQLite’s JSON support to parse an array. This idea can be further extended to pluck random values from an array. ...
I’ve rarely been able to take advantage of Django’s bulk_create / bulk_update APIs in production applications; especially in the cases where I need to create or update multiple complex objects with a script. Often time, these complex objects trigger a chain of signals or need non-trivial setups before any operations can be performed on each of them. The issue is, bulk_create / bulk_update doesn’t trigger these signals or expose any hooks to run any setup code. The Django doc mentions these bulk_create caveats in detail. Here are a few of them: ...