Forem: Kelvin Wangonya

Be careful with join type typos

Kelvin Wangonya — Thu, 07 Nov 2024 06:30:38 +0000

I noticed a typo in one of my sql queries today, but the funny thing is the query still worked fine. It looked something like this:

SELECT
    *
FROM
    table_1
LEFFT JOIN
    table_2
USING (a,b);

At first I thought BigQuery was pretty smart and just assumed I meant LEFT instead. Maybe it's a common typo and there's an alias for it. So I got curious and tried out this:

SELECT
    *
FROM
    table_1
LEFFFFTFT JOIN
    table_2
USING (a,b);

No error. That can't be right.

That's when it hit me. This query isn't doing a LEFT JOIN. It's using the incorrectly spelled name as an alias for table_1 and doing an INNER JOIN instead. Basically this:

SELECT
    *
FROM
    table_1 AS LEFFT
JOIN
    table_2
ON LEFFT.a = table_2.a AND LEFFT.b = table_2.b

So the query works fine but the results would be incorrect.

This is one merit of syntax highlighting I've never really thought about until now. The only reason I noticed this quickly was by seeing LEFFT wasn't highlighted as I expected. The results of the query looked fine at first glance so this would have definitely bit me somewhere down the line if it got to production.

Spotify Cleaner

Kelvin Wangonya — Wed, 25 Sep 2024 05:54:24 +0000

I've had my Spotify account for a long time, and over that time, my musical tastes have evolved. As a consequence, there are a lot of liked songs, playlists, and podcasts I no longer listen to but have in my library. There's nothing wrong with keeping them in the library—I don't have them downloaded, so they're not really taking up any space—but I don't like the clutter.

As far as I know, Spotify doesn't provide any way to batch delete things, so I created a spotify-cleaner for this.

I used it to just delete everything (all liked songs, albums, podcasts, and playlists), but it can also delete just a few selected items.

If this is something you feel might also be useful to you, check the repo for instructions on how to use it.

On embracing asynchronous communication

Kelvin Wangonya — Mon, 01 Apr 2024 02:54:56 +0000

I've been trying to improve my workflows this year. One thing I've known for a while, but never really got
serious about implementing, is defaulting to asynchronous communication. I first learned about this concept
a few years ago when reading Basecamp's Employee Handbook.

I work remotely for a company in the UK, and we use Slack for communication. The timezone difference is only
two/three hours, so the whole team is usually all online at the same time for most of the day. I, (and I'm sure most other
members of the team), just find it a lot easier to pull someone in for a 'quick' huddle each time there's something
to discuss. I always hesitate before doing this, but end up doing it anyway most of the time. The fact that it is
very easy to do doesn't help. I think it does more harm than good.

More than a few times I try to invite someone for a huddle, and they're not available. I then have to write down
whatever I wanted us to discuss - and it always ends up feeling more productive. I think better when I write. Or,
maybe writing actually forces me to think about the thing more - something I may have been avoiding and choosing
the easier option of 'discussing' it instead. I even come up with the solution sometimes in the middle of writing,
and find the call, or even the message itself, not to be necessary.

Another benefit of writing things down is that you can actually refer back to the messages months/years down the
line when you can't figure out why a certain decision was made. I know Slack huddles have threads attached to
them but we rarely ever use them. Handovers are a lot easier when you can actually search for something in threads
and follow discussions that were made in the past.

Writing is also great practice for communication skills as things usually need to be a lot more clear written down than spoken.

I know these kinds of impromptu calls are distracting, but I understand why people do them. It's just the
easiest thing to do in the moment. They're hardly ever necessary though.

In my experience, I have never been in a situation where I would be completely blocked unless I got into a call
to discuss something. I'm never in need of an immediate answer to be able to continue with my work. And even if
I am, there's no guarantee that calling someone is going to get me that answer. They may not be available, and then
I'll still have to write down whatever I wanted anyway.

It's usually not worth breaking someone's focus over whatever issue I feel needs immediate attention. It's not easy,
but it's worth embracing asynchronous communication more.

Write something down as clearly and precisely as you can and send it.
Don't expect an immediate reply. Find something else to do in the meantime.

I think that's a better way to work.

Pydantic validators don't raise validation errors immediately

Kelvin Wangonya — Wed, 06 Mar 2024 03:59:24 +0000

This one really had me confused. I was facing an error with Pydantic validators which I thought I had handled, but it just didn't work as expected.

The schema looked something like this:

class Schema(BaseModel):
    dates: List[str]
    start_date: Optional[str]
    end_date: Optional[str]
    ...

    @validator("dates")
    def update_date_format(cls, v):
        ...

    @validator("start_date")
    def set_start_date(cls, v, values):
        # some logic here that expects `dates` to exist
        v = values['dates'][0]
        return v

    @validator("end_date")
    def set_end_date(cls, v, values):
        # some logic here that expects `dates` to exist
        v = values['dates'][-1]
        return v

My assumption in the last two validators was that dates would always be available because it's a required field.
To my surprise, an IndexError was always raised on set_start_date.

So I thought, ok, I'll raise a ValueError myself on update_date_format since the required field check doesn't seem to be working.

class Schema(BaseModel):
    dates: List[str]
    start_date: Optional[str]
    end_date: Optional[str]
    ...

    @validator("dates")
    def update_date_format(cls, v):
        if not v:
            raise ValueError("dates is a required field")
        ...

    @validator("start_date")
    def set_start_date(cls, v, values):
        # some logic here that expects `dates` to exist
        v = values['dates'][0]
        return v

    @validator("end_date")
    def set_end_date(cls, v, values):
        # some logic here that expects `dates` to exist
        v = values['dates'][-1]
        return v

Again, an IndexError was raised on set_start_date. It's like my check on update_date_format was completely ignored.
I even put a breakpoint on the line to make sure it was being hit. It was.

And then it hit me. Validation errrors aren't raised immediately. They're collected and returned all at once in the response.

Despite seeing this countless times in 422 responses, I hadn't ever thought about it. It made perfect sense.
You don't want to return one error at a time as an API response. It's much better to return a response detailing everything that's wrong with the payload.

From the docs:

One exception will be raised regardless of the number of errors found, that ValidationError will contain information about all the errors and how they happened.

The important thing to remember is that this only happens for validation errors (i.e errors raised through ValueError or AssertionError).
This is why the IndexError was always raised immediately it happened.

How to stop validation on the first error

According to this answer:

If you have checks, the failure of which should interrupt the further validation, then put them in the pre=True root validator. Because field validation will not occur if pre=True root validators raise an error.

For example:
class PayloadValidator(BaseModel):
   emailId: List[str]
   role: str

   @root_validator(pre=True)
   def root_validate(cls, values):
       if not values['emailId']:
           raise ValueError("Email list is empty.")
       return values

   @validator("emailId")
   def valid_domains(cls, emailId):
       return emailId

Taking notes

Kelvin Wangonya — Sat, 24 Jun 2023 21:08:48 +0000

I need to get in the habit of taking notes while I work. Especially when I'm debugging. I've noticed a poor habit reoccur when I'm working on a particularly difficult problem.

It usually goes something like this:

Start working on problem. Do some research. Try out A.
A doesn't work. Do some more research. Try B.
B doesn't work.
Try C. Nothing.
At this point I'm already tired so I think to myself, "A seems like a pretty good idea. Why didn't it work? Let's try it again.".
Oh, that's why A didn't work. Of course.
Try some variation of B.
Try C again.
Think about A some more...

So much time is wasted, with very little progress made. Even worse, if this is at the end of the day and I have to leave it pending and come back to it the next day - or if I have to stash the changes and work on something else for a while then come back to it later, then the cycle begins all over again.

If notes are taken at each step of the way, it's clear what works and what doesn't - and why. This isn't just helpful in the moment. Chances are I'll come across a similar issue in the future. I can't always rely on memory on these things. Experience has shown it to be very unreliable.

As much as I plan to mostly use pen and paper for this, it should also give me ample opportunity to blog more. Many problems should be general enough for someone¹ else out there to benefit from.

Barely anyone reads my blog, I know, but it should at least act as a backup to my pen and paper. ↩

Handling missing dict keys

Kelvin Wangonya — Thu, 10 Feb 2022 15:25:42 +0000

Trying to access a non-existent key using this notation raises a KeyError: dict[key]. An easy workaround for this is to use get(key) instead, which returns None if key isn't found, or get(key, default) which returns default if key isn't found.

>>> d = {}
>>> d
{}
>>> d["x"]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'x'
>>> d.get("x")
>>> d
{}
>>> d.get("x","y")
'y'
>>> d
{}
>>>

But what if you want to not only return a default if the key is missing, but also assign that default to the specified key in the dict? There are a couple of ways to do that.

1. `setdefault`

setdefault(key[, default])

If key is in the dictionary, return its value. If not, insert key with a value of default and return default. default defaults to None.

docs

>>> d
{}
>>> d.setdefault("x","y")
'y'
>>> d
{'x': 'y'}
>>>

2. `defaultdict`

class collections.defaultdict(default_factory=None, /[, ...])

The first argument provides the initial value for the default_factory attribute; it defaults to None. All remaining arguments are treated the same as if they were passed to the dict constructor, including keyword arguments.

docs

>>> from collections import defaultdict
>>> i = defaultdict(int)
>>> i
defaultdict(<class 'int'>, {})
>>> i["x"]
0
>>> i
defaultdict(<class 'int'>, {'x': 0})
>>> i["y"]
0
>>> i
defaultdict(<class 'int'>, {'x': 0, 'y': 0})
>>>

Note that no KeyError is raised despite the key not existing at first. Instead, the key is created with the default value of the type passed into defaultdict, in this case 0 for int. If we used list instead, the default value would be [], an so on.

3. Implement `missing`

This is actually what defaultdict does behind the scenes. Use this when defaultdict doesn't fit your usecase.

>>> class M(dict):
...     def __missing__(self, key):
...             value = "my default value"
...             self[key] = value
...             return value
...
>>>
>>> m = M()
>>> m
{}
>>> m["x"]
'my default value'
>>> m
{'x': 'my default value'}
>>> m["y"]
'my default value'
>>> m
{'x': 'my default value', 'y': 'my default value'}
>>>

You can use your database as a simple calculator

Kelvin Wangonya — Sat, 23 Oct 2021 15:29:20 +0000

Saw this in the MySQL docs. I'm assuming this only works for relational databases.

I don't know if it's very useful but definitely interesting.

MariaDB[] > SELECT SIN(PI()/4), (4+1)*5;
+------------------+---------+
| SIN(PI()/4)      | (4+1)*5 |
+------------------+---------+
| 0.70710678118655 |      25 |
+------------------+---------+
1 row in set (0.02 sec)

postgres=# SELECT SIN(PI()/4), (4+1)*5;
        sin         | ?column?
--------------------+----------
 0.7071067811865475 |       25
(1 row)

Getting the running Go version

Kelvin Wangonya — Sat, 24 Jul 2021 03:01:19 +0000

This would mostly be useful for debugging.

package main

import (
    "log"
    "runtime"
)

func main() {
    log.Printf("Running Go: %s", runtime.Version())
}

Output: Running Go: go1.16.6

Writing DRYer tests using Pytest parametrize

Kelvin Wangonya — Thu, 08 Apr 2021 16:29:08 +0000

Tests tend to not always be so DRY, which isn't necessarily a bad thing.

This SO answer sums it up nicely:

Readability is more important for tests. If a test fails, you want the problem to be obvious. The developer shouldn't have to wade through a lot of heavily factored test code to determine exactly what failed. You don't want your test code to become so complex that you need to write unit-test-tests.

However, eliminating duplication is usually a good thing, as long as it doesn't obscure anything, and eliminating the duplication in your tests may lead to a better API. Just make sure you don't go past the point of diminishing returns.

Pytest gives some ways to reduce duplication with fixtures.

Say you had a couple of endpoints that return data to be used in a report.
The report data is supposed to be displayed in an excel sheet with different sheets.

Sometimes, only data for one sheet is required. Other times, data for all the sheets is fetched. So the endpoints end up being broken down like this:

/report/sheet-a
/report/sheet-b
/report/sheet-c
/report/sheet-d

Let's take a simple test case: checking that the endpoints return 200 when called.

import pytest

def test_report_sheet_a_returns_200():
    response = client.get('/report/sheet-a')
    assert response.status == 200

def test_report_sheet_b_returns_200():
    response = client.get('/report/sheet-b')
    assert response.status == 200

def test_report_sheet_c_returns_200():
    response = client.get('/report/sheet-c')
    assert response.status == 200

def test_report_sheet_d_returns_200():
    response = client.get('/report/sheet-d')
    assert response.status == 200

This might not look too bad, but if we wanted to test for another operation on the endpoints - checking if the endpoints require authentication for example - you get the feeling that this can be done a little bit better.

import pytest

# test GET requests

def test_report_sheet_a_returns_200():
    response = client.get('/report/sheet-a')
    assert response.status == 200

def test_report_sheet_b_returns_200():
    response = client.get('/report/sheet-b')
    assert response.status == 200

def test_report_sheet_c_returns_200():
    response = client.get('/report/sheet-c')
    assert response.status == 200

def test_report_sheet_d_returns_200():
    response = client.get('/report/sheet-d')
    assert response.status == 200

# test auth

def test_report_sheet_a_requires_auth():
    response = unauthorized_client.get('/report/sheet-a')
    assert response.status == 401

def test_report_sheet_b_requires_auth():
    response = unauthorized_client.get('/report/sheet-b')
    assert response.status == 401

def test_report_sheet_c_requires_auth():
    response = unauthorized_client.get('/report/sheet-c')
    assert response.status == 401

def test_report_sheet_d_requires_auth():
    response = unauthorized_client.get('/report/sheet-d')
    assert response.status == 401

Notice that the only thing changing in the tests is the endpoints. Everything else remains the same.

Given that all the sheets belong to one report, we can refactor the tests to reduce duplication without sacrificing readability.

Here's how the tests can be rewritten using Pytest parametrize:

import pytest

report_sheet_endpoints = (
  '/sheet-a',
  '/sheet-b',
  '/sheet-c',
  '/sheet-d',
)


@pytest.mark.parametrize('endpoint', report_sheet_endpoints)
def test_report_sheets_return_200(endpoint)
    response = client.get(f'/report{endpoint}')
    assert response.status == 200


@pytest.mark.parametrize('endpoint', report_sheet_endpoints)
def test_report_sheets_require_auth(endpoint)
    response = unauthorized_client.get(f'/report{endpoint}')
    assert response.status == 401

Now we have two tests instead of eight. But when you run the tests, 8 tests will run, not 2. Pytest takes each value in report_sheet_endpoints and feeds it into the test.
This reduces duplication while maintaining readability.

Running the tests gives this output:

test_reports.py::test_report_sheets_return_200[sheet-a] PASSED
test_reports.py::test_report_sheets_return_200[sheet-b] PASSED
test_reports.py::test_report_sheets_return_200[sheet-c] PASSED
test_reports.py::test_report_sheets_return_200[sheet-d] PASSED

test_reports.py::test_report_sheets_require_auth[sheet-a] PASSED
test_reports.py::test_report_sheets_require_auth[sheet-b] PASSED
test_reports.py::test_report_sheets_require_auth[sheet-c] PASSED
test_reports.py::test_report_sheets_require_auth[sheet-d] PASSED

Hackathon submission

Kelvin Wangonya — Sat, 09 Jan 2021 18:53:58 +0000

What I built

An auto repair shop management system.

Category Submission:

Built for Business

App Link

Link

Screenshots

Registration:

Dashboard:

Jobs:

Contacts:

Vehicles:

Staff:

Description

This is a system designed to help auto repair shops manage and keep track of jobs, clients, vehicles, etc. You can read a full description on the project's Readme.

Link to Source Code

Link

Permissive License

Apache License 2.0

Background

I have a friend who owns an auto repair shop. Process are being handled manually and as the shop grows this is getting out of hand. We had a conversation about coming up with a simple system to keep track of clients and jobs and this Hackathon gave me the perfect platform to get this done.

I'll continue improving the system to a point where it's fully usable by him i.e reduce any manual labor on their side as much as possible. Later on, this can be expanded to a few other local shops.

How I built it

I used Django + Postgres to build this. I followed the simple instructions on the sample django repo provided by Digital Ocean. I also added integration with Sendgrid for emails.

I had never used Digital Ocean for anything before but I didn't face any issues with the deployment.

I learned my way around the platform and I see myself using it for my other side projects.

Additional Resources/Info

To get started quickly, you can generate dummy data on the jobs page. This will create jobs, contacts, vehicles and staff to work with.
Apparently, Safari browser doesn't show a dropdown calendar on HTML date fields natively like other browsers. I'm yet to fix this so if you'll be testing the system on Safari, the date fields on the job creation form will appear like regular text fields.

Progress: Search

Kelvin Wangonya — Tue, 29 Dec 2020 05:13:50 +0000

Search

A simple implementation of Django's full text search.

Next steps

Various bug fixes and improvements before submission

Progress: Improved dashboard (with real data) and "no data" placeholders

Kelvin Wangonya — Sun, 27 Dec 2020 12:32:49 +0000

Dashboard

The dashboard now pulls data from the database based on the selected time period (week/month/year). I'm using Chartjs for the charts.

"No data" placeholders

Instead of showing a blank screen on login if there's no data.

Next steps

Enable search for vehicles, contacts and staff

Forem: Kelvin Wangonya

Be careful with join type typos

Spotify Cleaner

On embracing asynchronous communication

Pydantic validators don't raise validation errors immediately

How to stop validation on the first error

Taking notes

Handling missing dict keys

1. setdefault

setdefault(key[, default])

2. defaultdict

class collections.defaultdict(default_factory=None, /[, ...])

3. Implement __missing__

You can use your database as a simple calculator

Getting the running Go version

Writing DRYer tests using Pytest parametrize

Hackathon submission

What I built

Category Submission:

App Link

Screenshots

Description

Link to Source Code

Permissive License

Background

How I built it

Additional Resources/Info

Progress: Search

Search

Next steps

Progress: Improved dashboard (with real data) and "no data" placeholders

Dashboard

"No data" placeholders

Next steps

1. `setdefault`

2. `defaultdict`

3. Implement `missing`