Database

I was working on a project where I needed to poll multiple data sources and consume the incoming data points in a single thread. In this particular case, the two data streams were coming from two different Redis lists. The correct way to consume them would be to write two separate consumers and spin them up as different processes. However, in this scenario, I needed a simple way to poll and consume data from one data source, wait for a bit, then poll and consume from another data source, and keep doing this indefinitely. That way I could get away with doing the whole workflow in a single thread without the overhead of managing multiple processes. ...

Django has a Model.objects.bulk_update method that allows you to update multiple objects in a single pass. While this method is a great way to speed up the update process, oftentimes it’s not fast enough. Recently, at my workplace, I found myself writing a script to update half a million user records and it was taking quite a bit of time to mutate them even after leveraging bulk update. So I wanted to see if I could use multiprocessing with .bulk_update to quicken the process even more. Turns out, yep I can! ...

While going through the documentation of Python’s sqlite31 module, I noticed that it’s quite API-driven, where different parts of the module are explained in a prescriptive manner. I, however, learn better from examples, recipes, and narratives. Although a few good recipes already exist in the docs, I thought I’d also enlist some of the examples I tried out while grokking them. Executing individual statements To execute individual statements, you’ll need to use the cursor_obj.execute(statement) primitive. ...

Python has a random.choice routine in the standard library that allows you to pick a random value from an iterable. It works like this: # src.py import random # The seed ensures that you'll get the same random choice # every time you run the script. random.seed(90) # This builds a list: ["choice_0", "choice_1", ..., "choice_9"] lst = [f"choice_{i}" for i in range(10)] print(random.choice(lst)) print(random.choice(lst)) This will print: choice_3 choice_1 I was looking for a way to quickly hydrate a table with random data in an SQLite database. To be able to do so, I needed to extract unpremeditated values from an array of predefined elements. The issue is, that SQLite doesn’t support array types or have a built-in function to pick random values from an array. However, recently I came across this1 trick from Ricardo Ander-Egg’s tweet2, where he exploits SQLite’s JSON support to parse an array. This idea can be further extended to pluck random values from an array. ...