Forem: Tomislav Maricevic

The Illusion of Simplicity

Tomislav Maricevic — Tue, 25 Nov 2025 16:07:56 +0000

I really like Django. I would pick Django over any other option for setting up a website regardless of expected complexity. In my opinion, if you fully embrace Django, it will allow you to focus on the product and not fight an uphill battle against the computer.

I do a bit of freelancing on the side, and it saddens me that I rarely see Django projects in the wild. Wherever I join and I'm lucky enough that it's a Python gig, it's usually Flask or FastAPI.

When I ask why, it's usually something along the lines of: "Oh, we don't need Django, it's too complex. We just need a simple API".

Yet, they need database access, and ORMs are nice so they brought in SQLAlchemy. And they need user authentication, so they roll their own roles and permissions. And they need JWTs because the frontend is a React app with its own stack. And they need caching so they roll their own. And they need request validation and OpenAPI Javascript client generation so they bring in Pydantic. And of course they need horizontal scalability so they deploy everything on Kubernetes. And then: "We don't need Celery, it's too complex." so they add APScheduler. And then it turns out they do need simple workflows and CPU-heavy processing so they roll their own background task manager.

And here I am, looking at this amalgamation of bytes created in the name of simplicity, thinking: "What a poor reimplementation of Django."

Vim mandates >> AI mandates

Tomislav Maricevic — Tue, 16 Sep 2025 09:15:03 +0000

Intro

The internet is full of articles on CEOs declaring their companies "AI-first", in the name of increasing efficiency. After all, why have 50 engineers if you can have 5 managing a swarm of AI agents?

I am a heavy user of GenAI assistive tools and they truly do help me achieve results faster. But another thing that helps you achieve results faster is not having to fight an uphill battle against whichever text editor you're using. If you have to lift your hands from the keyboard, you're wasting time.

Yet I never heard any company issuing a "Vim keybindings mandate" and declaring themselves "modal-editing first", even though these tools are over 30 years old.

What is programming?

If we squint hard enough, defining programming is very simple -- we solve problems by translating abstract processes into textual artifacts we feed into our magic-rune-inscribed melted-sand tablets so they can understand and execute them.

This process is not linear: we do not first come up with a complete solution in our head, and then type it out, then run it, and call it a day. We iteratively think about the problem, type in some of the code, then think about it some more, then maybe look up something in the documentation, then type some more, etc.

Our brains are quite efficient at thinking, so most of the friction happens in other steps: typing text and looking things up. For now, the keyboard is still the best tool we have for this. From seasoned engineers to vibe coders, everyone has to move the text from their heads to the computer by typing out code (or prompts!).

If every time you notice a typo you have to hunt for the mouse, or smash those arrow keys for 10 seconds, you risk losing your train of thought and falling out of the flow state. The more you fight with inputting text, the less you focus on the problem itself.

Our duty as software engineering professionals is to reduce this friction as much as possible. We learn the ins and outs of programming languages so we can think in appropriate abstractions and avoid frequent lookups. We learn multiple languages so we can deliver efficient solutions without performing acrobatics in languages not appropriate for the task. We utilize GenAI tools to handle boilerplate and cruft. And we optimize our text editors so we can input text as fast as possible.

Why Vim?

Vim tries to reduce the friction in text editing as much as possible. Your hands stay on the keyboard on the home row all the time. The editor comes with a rich set of built-in shortcuts encompassing every text manipulation you can imagine. For specialized tasks, there are plugins that address them, and you can define your own custom shortcuts for your own workflow needs.

Building your own configuration is a crucial part of the process, where you get acquainted with the editor and the ecosystem. Realistically, the migration to Vim will be a J-curve: at first, you will be less productive, then after a week or two, you'll be where you were before, and if you pushed through, only after a couple of weeks of use you will start seeing gains in productivity.

The worst and the best part is that if you embrace Vim, it will ruin all other software for you. You will start looking at software through the lens of keyboard-only usability. If you have to reach for the mouse, or if it has its own silly shortcuts, you won't use it. And it really is a slippery slope: Vim is just a gateway drug, leading to tiling window managers and terminal multiplexers, and before you know it, you'll be using Nix and wondering how on Earth you managed to get anything done before.

Where does this leave us?

The "AI mandates" bullshit is purely performative, and the sad thing is everyone knows this -- the CEOs know it, the employees know it, the rest of us observing from the sidelines know it. LLMs truly are wonderful technology, and the productivity gains are real, but if the true goal is productivity, there are already many many ways it can be improved, and no one wrote memos about it. The truly efficient companies are staffed with conscientious engineers who do not have to be mandated to use the best tools available; they seek them out themselves.

Time is the only real currency in this world, and you're leaving money on the table if you're not using Vim. After all, Vim won't take your job. But someone using Vim will.

Configuring PostgreSQL server parameters on GitHub Actions

Tomislav Maricevic — Thu, 28 Aug 2025 09:01:51 +0000

If your project is not fully containerized, but you still want to use PostgreSQL in your GitHub Actions workflow, you can use the services feature of GitHub Actions to easily spin up a PostgreSQL container.

However, the services functionality restricts what you can configure declaratively in the workflow file -- namely you cannot configure the PostgreSQL server parameters that you would usually set up in the postgresql.conf file. Fortunately, most of these can be configured through ALTER SYSTEM commands.

For example, this is how to configure max_locks_per_transaction:

jobs:
  build:
    runs-on: ubuntu-latest

    services:
      postgres:
        image: postgres:17
        env:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: production
        ports:
          - 5432:5432
        options: >-
          --name pg_container
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

    steps:
      - name: Set up environment
        run: |
          sudo apt-get update
          sudo apt-get install -y \
            postgresql-client \
            wait-for-it

      - name: Configure PostgreSQL
        env:
          PGUSER: postgres
          PGPASSWORD: postgres
          PGHOST: 127.0.0.1
          PGPORT: 5432
          PGDATABASE: template1
        run: |
          psql -c "SHOW max_locks_per_transaction;"
          psql -c "ALTER SYSTEM set max_locks_per_transaction = 128;"
          docker restart pg_container
          wait-for-it localhost:5432 --timeout=30 --strict -- echo "PostgreSQL is up"
          psql -c "SHOW max_locks_per_transaction;"

# ... the rest of the workflow

Of course, if you need heavy customization, it makes more sense to skip services and run your own container through a docker run step or docker-compose, but for simple use cases, this is a quick way to get started.

On Usable Documentation

Tomislav Maricevic — Sun, 30 Mar 2025 05:53:21 +0000

Having no documentation is often less harmful than having inaccurate documentation.

Like code, documentation degrades over time. What was once accurate may now be obsolete. And practices we once ignored might now be part of our daily workflow. Unless maintaining documentation is an intentional process, it will rot, maybe beyond saving.

In this article I'll outline a few guidelines that worked well for me in the past for keeping the developer documentation usable. They're certainly not universal, but they are a good starting point for a small team of experienced developers. We’ll skip the philosophical debates and focus on real-world practices that work for small, fast-moving engineering teams - the aim is to get the most value with the least effort. Documentation doesn't pay the
bills.

1. Keep Documentation in the Codebase

If your documentation is for developers, the natural thing to do is to keep the documentation close to the code. Create a docs/ folder in the project root and add some markdown files to it. Everyone already has a documentation viewer -- their code editor. As a cherry on top, GitHub renders markdown files in the browser so you can visually browse the documentation. Add a README.md with a short description to every subfolder to make it easier to navigate on the web.

Greppability is a huge plus -- searching for a class name will render both code and documentation results.

I usually read the raw markdown files because most of our articles are text-only, but in case of multimedia, I can view it either using my editor's markdown preview, or navigate to GitHub.

2. Flat is generally better

Engineers tend to value neatness. It's very tempting to start developing a complex hierarchy of folders, enumerating all possible domains, adding placeholder folders and articles. The thing is, documentation is meant to be read. If the structure is too complex, the information becomes fragmented and hard to find, even though it's seemingly well-organized.

When organizing the documentation, ask yourself how would you find this if production was down and finding this information was the only thing that could help.

Another benefit of flatter organization is surfacing "unknown unknowns" — useful insights you wouldn’t have searched for but found by proximity to other information.

Not so good structure:

/docs
    /architecture
        /infrastructure
            servers.md
            linux.md
            dns.md
            cdn.md
        /backend
            django.md
            celery.md
        /frontend
            components.md

Better structure:

/docs
    README.md
    local_setup.md
    infrastructure.md
    deployment.md
    troubleshooting.md

3. Keep It Company-Specific

Your documentation should contain company-specific notes, and not write in general about engineering concepts. If we have a special way of handling CORS, write about our specifics, do not explain what CORS is. There are plenty of articles on that written by people who know more about it than you.

Not so good article:

CORS stands for Cross-Origin Resource Sharing.
It is a mechanism that allows ...

Better article:

CORS is handled by `django-cors-headers`.
See `settings.py` for list of allowed origins.

If it's really important to understand the concept well, link to an external source, but keep in mind that links rot as well, and you will need to update them periodically. Prefer well-established sources like MDN, Django documentation, etc.

4. Someone Has To Be the Documentation Police

Good documentation is a process, not a one-off task — and every process needs an owner.

Even in small teams, there are people with different priorities. If we're moving fast, it's OK to skip writing documentation for a while, but someone needs to keep the documentation debt on their mind, and schedule updates, deprecations and additions. This can be anyone who is willing. Tenured engineers are probably a better choice than engineering managers who are more removed from day to day operations.

5. Continual Improvement

Best test of your documentation is a new person joining the team. They come from a different way of doing things, and haven't yet got accustomed to "our way". Their first months in the company should be used to re-evaluate the usefulness and correctness of the documentation.

They might suggest new additions because we're taking some knowledge as a given. They will quickly catch things that are no longer true because they will be following the steps in the documentation to the letter. It would be a shame to not use this opportunity for improvement.

It's especially a good idea to make documentation review a part of the onboarding process: tell the new hires it's one of their first duties to review the documentation and suggest improvements. Make sure to follow-up on their suggestions, providing feedback on acceptance or rejection, and let them implement the changes.

6. Periodic Revision

Very related to previous point, but still different. You will probably not reorganize the documentation because one person found it unclear. But, if it's a pattern, then you will schedule time for this on the roadmap, and approach it very seriously.

6 or 12 month cadence seems reasonable. Dedicate some time to gather feedback. Take a high-level overview:

Which articles are rarely used?
What's outdated?
Are any links broken?
Are any articles too long or unfocused?
Are we missing any key topics?

Asking publicly in Slack what people think of the current state of the documentation is a good start.

7. Make Documentation a Habit

Unused documentation is a sign of bad documentation.

Propagate the documentation. Link to articles instead of providing the answer directly, this increases the chance of discovering unknown unknowns. If the answer is not in the documentation, then write it down. Encourage your team colleagues to do the same. Build the culture of writing documentation.

When done right, usable documentation is a force multiplier that makes onboarding faster, reduces support noise, and helps everyone move faster with confidence.

Handling CSRF Login Errors Gracefully in Django

Tomislav Maricevic — Sat, 22 Mar 2025 12:37:28 +0000

What's CSRF?

Cross site request forgery is a type of attack where a malicious website tricks a user into performing actions on another site where they're authenticated. This is usually done by embedding a form in the malicious site, and submitting it to the target site.

An example of this would be a card game website where, when you hit the "Play" button, it sends a POST request to another site with the payload to change your login email address to the attacker's. Since you're logged in to the target site, the request goes through and you lose access to your account.

How does it work in Django

By default, Django servers you a cookie with the CSRF token on the first request. This token (in a masked form) is embedded in every form that Django generates, and is unique to the user and the session.

The form token is checked on every unsafe request (POST, PUT, DELETE, PATCH). If the token is missing, invalid, or does not match the token in the cookie, the server responds with a 403 Forbidden response.

This way Django ensures that the request is coming from the site itself, and not from a malicious third party, since no other server can generate valid CSRF tokens.

The problem

The scenario is as follows:

You open a website in one tab
You open the same website in another tab
You log in in the second tab, and start using the website
You go back to the first tab, and try to do something that requires a POST request (like submitting a form)
You get a 403 Forbidden CSRF Error response

For security reasons, Django cycles CSRF tokens on every login. This means that the token embedded in the form in the first tab is now invalid since it was generated before your login in the second tab.

Django, being the best web framework out there, even warns you about this if you have DEBUG = True and you get a CSRF failure.

Since this can happen to regular users, it's not just a security problem, but also a UX problem. The users are most likely to encounter it on the login page because it is one of the few public forms every site has, and a successful login cycles the token.

Solution #1: Pure Django solution

Django allows setting a custom CSRF failure handler view via settings.CSRF_FAILURE_VIEW variable. For a seamless UX, in case this happens on the login view, you could redirect the user back to the referrer page. Since they're already logged in, they will be able to access it.

As a bonus, let's add nice template for the CSRF failure view that explains what happened and offers a button to go back to the previous page.

# settings.py
CSRF_FAILURE_VIEW = 'myapp.views.csrf_failure'

# views.py
from http import HTTPStatus

from django.shortcuts import redirect, render

def csrf_failure(request, reason=""):
    referer = request.META.get("HTTP_REFERER", "/")
    if resolve(request.path).url_name == "login":
        return redirect(referer)

    return render(request, "csrf_failure.html", context={"referer": referer}, status=HTTPStatus.FORBIDDEN)

Solution #2: Javascript

Another solution would be to use Javascript to periodically check if the CSRF cookie changed since the initial page load and warn the user if it did.

// csrf.js
const COOKIE_NAME = 'csrftoken';

function getCookie(name) {
    const value = `; ${document.cookie}`;
    const parts = value.split(`; ${name}=`);
    if (parts.length === 2) return parts.pop().split(';').shift();
}

function checkCSRFChange() {
    const currentToken = getCookie(COOKIE_NAME);
    if (currentToken && currentToken !== initialToken) {
        alert("Your session has changed or expired. Please reload the page to avoid losing changes.");
    }
}

const initialToken = getCookie(COOKIE_NAME);
setInterval(checkCSRFChange, 5000);

Better living through optimized Django

Tomislav Maricevic — Sat, 22 Mar 2025 12:32:08 +0000

Every engineer that loves Django and has a blog has at least one of these posts.

Django's ORM is excellent, but given enough time it's easy for approaches that weren't mistakes to grow into mistakes This is a great thing, because it usually means your company didn't go bankrupt, you're still here and can fix things, and the company is doing well because the scale increased (hopefully your compensation as well).

This is a recap of my recent experience into optimizing Celery tasks that started out as non-problematic, but with the passage of time became problematic, causing server and database stability issues.

`prefetch_related` + `iterator()` problem

Up until Django 4.1, calling iterator() on a queryset with prefetch_related() caused the prefetched data to be dropped, causing N+1 queries problem.

In Django 4.1 the iterator started accepting batch_size argument that allows us to get the best of both worlds -- avoid pulling the entire dataset into memory while avoiding the N+1 queries problem. Or at least turn the N+1 into N / batch_size + 1, which is considerably better.
But the batch_size's default value is None which reverts to old behavior, discarding the prefetched data silently.

The pseudocode from the problematic task looked something like this:

profiles = Profile.objects.filter(
    community=community
).select_related(
    a bunch of joins here
).prefetch_related(
    'user__groups', 
    a bunch of other m2ms
)

BATCH_SIZE = 1000
records = []
for profile in profiles.iterator():
    group_names = [g.name for g in profile.user.groups.all()]
    primary_group = determine_primary_group(group_names)
    denorm_profile = DenormProfileRecord(
        user_id=profile.user_id,
        primary_group=primary_group,
        groups=group_names,
    )
    records.append(denorm_profile)
    if len(records) == BATCH_SIZE:
        DenormProfileRecord.objects.bulk_create(records)
        records = []
DenormProfileRecord.objects.bulk_create(records)

This task took hours, and used up a lot of database resources. When it was originally written, there were not that many profiles and there were not many other tasks demanding database resources so it worked well. But, we were lucky: people registered in increasing numbers, other business processes took their own chunk of database's resources, and this really became a bottleneck.

My first optimization attempt was:

Increase the batch size to 10k, to reduce the number of bulk_create () calls
Use only() on the Profile queryset to avoid fetching unnecessary data
Since I believed only() reduced the memory requirements enough, I dropped the iterator() to enable prefetch_related() to do its thing

This optimization turned out to be ... less than optimal.

One week and one server and database outage later, I was forced to revisit my optimization.

My first failing was in not examining the entire context in which this code is run. It's part of a Celery task executed by a Celery worker with --concurrency=4, meaning that it's possible that we try to refresh 4 big communities at the same time.

Second, I failed to account for some communities having 100s of thousands of profiles. Removing the iterator() call means all of these profiles are loaded into memory at once.

Third, I underestimated the difference in memory consumption between Python models instances (which are still constructed when you use only()) and Python built-in types.

The second optimization attempt was:

Recognizing that we don't really need model instances from the prefetched relations, we just need certain values -- we can get much better performance by using PostgreSQL-specific ArrayAgg
only() only marginally reduced the memory footprint and since we don't actually need the Profile model instances, we can get a huge benefit from enumerating all required fields in a values_list() call and avoid constructing the model instances completely

The final version looked something like:

profiles = Profile.objects.filter(
    community=community
).annotate(
    group_names=ArrayAgg(
        'user__groups', 
        filter=Q(user__groups__isnull=False),
        distinct=True, 
        default=[]
    ),
    ...other values that can be aggreged as well,
).values_list(
    'user_id',
    'group_names',
    ...all other fields that were necessary for DenormProfileRecord
    named=True
)

BATCH_SIZE = 10000
records = []
for profile in profiles.iterator():
    primary_group = determine_primary_group(profile.group_names)
    denorm_profile = DenormProfileRecord(
        user_id=profile.user_id,
        primary_group=primary_group,
        groups=profile.group_names,
    )
    records.append(denorm_profile)
    if len(records) == BATCH_SIZE:
        DenormProfileRecord.objects.bulk_create(records)
        records = []
DenormProfileRecord.objects.bulk_create(records)

This way we could keep the iterator() call since there were no prefetch_related() calls. The values_list() optimization wasn't actually necessary because we only had a single row of Profile data in memory at the same time, but I kept it just in case.

This reduced the memory strain on the server from "fills up the RAM and swap and causes OOM killer to go on a rampage" to "unnoticeable". The runtime dropped from several hours to ~30s.

Optimizing Redis access

This one isn't really Django ORM related, but it was done in the same batch of optimizations so I'll touch on it.
We utilize Redis to keep a sorted set of video name prefixes allowing live autocomplete while typing in the search box on the site.

Populating this Redis sorted set is done a daily basis: it's completely dropped and recreated from scratch.

The function looks something like:

from redis.client import Redis

AUTOCOMPLETE_REDIS_KEY = 'autocomplete'
MAX_PREFIX_LENGHT = 8


def update_autocomplete_prefixes():
    redis = Redis.from_url(REDIS_CACHE_LOCATION)
    redis.delete(AUTOCOMPLETE_REDIS_KEY)

    for video_title in Video.objects.values_list('title', flat=True):
        title = strip(video_title.lower())
        for i range(1, MAX_PREFIX_LENGHT + 1):
            redis.zadd({title[0:i]: 0})

Upon investigating the used redis library, it turns out each zadd() call is a single network request. As the number of videos grew, the number of network requests grew as well, until this task took about 15 minutes to complete.

The optimization approach here was to collect all Redis updates in a single dictionary and push it in a single network call. This approach also allowed moving the delete call much closer to the single zadd() call, reducing the time where the autocomplete prefixes were only partially available.

One small database related improvement was pushing the strip() and lower() calls to the database utilizing the Trim() and Lower() database functions.

The rewritten task looks something like:

from redis.client import Redis

AUTOCOMPLETE_REDIS_KEY = 'autocomplete'
MAX_PREFIX_LENGHT = 8


def update_autocomplete_prefixes():
    prefixes = {}
    for video_title in Video.objects.annotate(values_list(Trim(Lower('title')), flat=True):
        prefixes |= {video_title[0:i]: 0 for i in range(1, MAX_PREFIX_LENGTH + 1)}

    redis = Redis.from_url(REDIS_CACHE_LOCATION)
    redis.delete(AUTOCOMPLETE_REDIS_KEY)
    redis.zadd(prefixes)

This reduced the runtime of the task from 15 minutes to ~5 seconds.

I also entertained the idea of using memoryviews to avoid constructing new string objects for each prefix, but there were risks associated with handling unicode characters (which were present in the video titles, and memoryviews operate on bytes), and not really being familiar with how the redis Python library handles the passed data (it could quite easily cast these memoryviews back to strings, annuling any gains).

Optimizing deletion of old records

For debugging purposes, we retain a copy of each email sent to our users. Since we like to keep things simple, this data is kept within a table in PostgreSQL, and the old records are purged from the table daily. Retention policy is 2 weeks, so every day there is a Celery task that identifies old records and deletes them:

def remove_old_emails():
    old_mailer_ids = list(
        Mailer.objects.filter(sent__lte=timezone.now() - relativedelta(weeks=2)
    )
    old_emails = Email.objects.filter(mailer_id__in=old_mailer_ids)
    old_email._raw_delete(old_emails.db)

This regularly took 20-30 minutes, even with the _raw_delete optimization. The table is not humongous, it definitely should not take that long.

For historical reasons (when the table was humongous and a join would kill the database) the table doesn't have any foreign key constraints, and all other tables are referenced through soft foreign keys (e.g. mailer_id is an integer column with an index, without a foreign key constraint). In the meantime we introduced a data retention policy to manage the table's size.

The problematic part quickly emerged upon inspecting the SQL query: the list of old mailer IDs has 100k members, and is growing daily since it's a list of all mailers ever sent. This makes Postgres' life hard, and degrades every query to a full table sequential scan.

The solution is clear: reduce the list of mailer IDs to something manageable. Since the task is run daily, it's safe to reduce the list to mailers that were sent between 2 weeks ago and 2 weeks and 1 day ago. We want to have some redundancy so we increased that range to 2 weeks an 3 days ago, in case something prevents the task from running for a day or two.

Postgres started using an index scan instead of a sequential scan, and things sped up drastically -- the runtime dropped from 20-30 minutes to 3-5 minutes.

Optimizing creation of new many-to-many records

Using Model.objects.create() in a for loop is a surefire way to degrade the performance of you database -- every create() call is a network request to the database with an INSERT command.

for_date = date(2024, 8, 24)
impression_data = Impression.objects.filter(
    created__date=for_date
).values(
    'content_type', 'object_id'
).annotate(total_impressions=Count('id'))
for content_impression in impression_data:
    DailyImpressionStats.objects.create(**content_impression)

One of basic ways to improve performance, if memory allows it, is to accumulate unsaved model instances in memory and then create them all at once using a bulk_create call:

for_date = date(2024, 8, 24)
impression_data = Impression.objects.filter(
    created__date=for_date
).values(
    'content_type', 'object_id'
).annotate(total_impressions=Count('id'))
daily_impressions_list = []
for content_impression in impression_data:
    daily_impressions_list.append(DailyImpressionStats(**content_impression))

DailyImpressionStats.objects.bulk_create(daily_impressions_list)

This is a fairly common optimization, but here's a follow-up problem: what if we also have to set a many-to-many relationship on the model we want to bulk create? At first, it seems like we cannot use the bulk_create() approach anymore:

for content_impression in impression_data:
    daily_impression = DailyImpressionStats(**content_impression)
    tags = get_tags_for_content(
        content_type=content_impression['content_type_id'], 
        object_id=content_impression['object_id'],
    )
    daily_impression.tags.set(tags)  # ERROR: daily_impression has to be saved before we can set the M2M relationship

Luckily, digging a bit deeper into how Django implements the many to many relationship offers an answer. When we define a many to many relationship between models, Django creates an intermediary table with an accompanying through model accessible via
Model.m2m_field.through. This allows us to also accumulate the through model instances in another list and bulk create them as well.

If the id field is declared as an AutoField, PostgreSQL, MariaDB and SQLite set the id field on model instances when using bulk_create(). Our many to many records reference these instances so we can first bulk create the model instances, and then bulk create the many to many instances:

for_date = date(2024, 8, 24)
impression_data = Impression.objects.filter(
    created__date=for_date
).values(
    'content_type', 'object_id'
).annotate(total_impressions=Count('id'))
daily_impressions_list = []
daily_impressions_tag_list = []
for content_impression in impression_data:
    daily_impression = DailyImpressionStats(**content_impression)
    daily_impressions_list.append(daily_impression)
    tags = get_tags_for_content(
        content_type=content_impression['content_type_id'],
        object_id=content_impression['object_id'],
    )
    daily_impressions_tag_list.extend(
        DailyImpressionStats.tags.through(
            daily_impression=daily_impression, tag=tag
        ) 
        for tag in tags
    )

DailyImpressionStats.objects.bulk_create(daily_impressions_list)
DailyImpressionStats.tags.through.objects.bulk_create(daily_impressions_tag_list)

Epilogue

It's easy to dismiss the problems with the original code snippets as "skill issues", and sometimes they really are. But we need to keep in mind that equally often code starts out as performant and ends up as a bottleneck. Conditions change, scale increases, tech stack evolves. If you tried to do some of the described optimizations in the initial code push, I would probably be the first one to invoke YAGNI and ask for a simplification. Business first, tech second -- every minute you spend on optimizing a query means the business might not make enough money and not live to see the day your optimization pays off.

It's not important to write the optimal code in the first go, it's important to be able to write it once it becomes problematic, and in the mean time, build things that matter.

On Python's @property decorator

Tomislav Maricevic — Mon, 30 Sep 2024 08:56:20 +0000

@property decorator is an excellent way to reduce the readability of Python code. It obfuscates a perfectly good
function call and tricks readers into thinking they're performing a regular attribute access or assignment.

Unless there's a really good and explicit reason to do this, don't.

List of good and explicit reasons:

Refactoring

That's pretty much it.

If you need to turn something that (rightfully so) started out as a simple attribute, but with time accrued some more
complex logic, @property is a good way to gracefully transition from attributes to function calls.

Version 1

We start out with a simple attribute. You can get it, you can set it. As a consenting adult, you're free to do with it
whatever you want.

class Client:
    def __init__(self, value):
      self.value = value

Version 2:

The project gains traction. You need to add two new features:

Emit an event whenever the Client.value attribute is accessed, so other parts of the code can listen to it and do their own thing
You want a central place to validate values being assigned, to avoid littering the rest of your codebase with error handling

Because we're a self-aware smol brain developer, we like plain old functions. We craft a plan
to change the class interface to use getter/setter functions instead of direct attribute access. But since
we're also responsible and respectful to our colleagues/clients, we don't just change the API abruptly. No, we will be
emitting a deprecation warning for some time, and only introduce breaking changes in the API after we've given everyone
ample time to migrate.

import warnings


class Client:
    def __init__(self, value):
        # We add a private attribute to hold the value
        self._value = None
        self.set_value(value)

    @property
    def value(self):
        # We can now emit a deprecation warning on 
        # each access, urging our users to migrate to the new API
        warnings.warn("A.value is deprecated, use A.get_value() instead!", DeprecationWarning)

        # ... and offload the act of retrieving the value 
        # to the newly-introduced function
        value = self.get_value()
        return value

    @property.setter
    def value(self, new_value):
        warnings.warn("A.value is deprecated, use A.set_value() instead!", DeprecationWarning)
        self.set_value(new_value)

    # We add getter/setter functions with the new logic
    def get_value(self):
        self._emit_event('value_access')
        return self._value

    def set_value(self, new_value):
        self._validate_value(new_value)
        self._value = new_value

Version 3:

Time has passed, and people have migrated to the new API. We're ready to make our lives easier, and simplify the codebase
by removing the dirty @property. Life is good again.

class Client:
    def __init__(self, value):
        self._value = None
        self.set_value(value)

    def get_value(self):
        self._emit_event('value_access')
        return self._value

    def set_value(self, new_value):
        self._validate_value(new_value)
        self._value = new_value

Going a bit deeper

@property is an example of a descriptor. Descriptors are a neat Python construct that "lets objects customize attribute
lookup, storage, and deletion". Some of the nicer things in life I
enjoy are made using descriptors, namely Django's ORM.

But just because you can doesn't mean you should. We always strive for the least complex option, and if you're certain
descriptors will make everyone's (not just yours!) lives easier, then go for it. Most of the time, though, plain
functions are the way to go.

Stop worrying and learn to love the function call.

Why I always assign intermediate values to local variables instead of passing them directly to function calls

Tomislav Maricevic — Thu, 26 Sep 2024 15:53:08 +0000

Instead of

def do_something(a, b, c):
    return res_fn(
        fn(a, b),
        fn(b),
        c
    )

I do:

def do_something(a, b, c):
    inter_1 = fn(a, b)
    inter_2 = fn(b)

    result = res_fn(inter_1, inter_2, c)
    return result

The first version is much shorter, and when formatted properly, equally readable.

But the reason I prefer the second approach is because all intermediate steps are saved to local variables.

Exception tracking tools like Sentry, and even Django's error page that pops up when DEBUG=True is set, capture the local context. On top of that, if you ever have to step through the function with a debugger, you can see the exact return value before stepping out from the function. This is the reason why I even save the final result in a local variable, just before returning it.

At the performance cost of couple of extra variable assignments, and couple of extra lines of code, this makes debugging much easier.