Forem: Roman Sorin

Serializing OpenAI API Responses for Logging with Pydantic

Roman Sorin — Sat, 03 Feb 2024 03:16:36 +0000

TL;DR

from openai import OpenAI
from openai.types import CreateEmbeddingResponse

client: OpenAI = OpenAI(api_key="YOUR_OPENAI_KEY")

embeddings_response: CreateEmbeddingResponse = (
  client.embeddings.create(input=["Some embeddings to generate"])
)

# Using Pydantic's `model_dump_json` method,
# we can easily serialize any OpenAI responses
serialized_response: str = embeddings_response.model_dump_json()

# Now that our response is serialized,
# we can log and store it as needed
openai_response: OpenAiResponse = OpenAiResponse(
  embeddings_response=serialized_response,
  ...  # other model fields
)

Why you might consider serializing responses

When working with external APIs, I lean heavily on logging as much information as I can for requests and responses. Recently, I've been working on a side project that relies on OpenAI's embeddings and chat completions APIs, as well as Amazon Textract's OCR service. For these types of external APIs, logging entire responses can be incredibly useful for many reasons, whether it's trying to iron out pricing for a product or needing to track if you're receiving a non-deterministic output (despite configuring for reproducibility). Over time, the quality of outputs can drift over time, and you'll inevitably need to iterate on input prompts as your product evolves or parameters change, which is where detailed logging provides the opportunity to look at how cost and quality may be changing - almost like a passive A/B test.

For most small to medium-sized projects, I'll typically store these logs as a dedicated table or explicit metadata columns on an existing table using Postgres. You could break a response object and its items into dedicated columns, but APIs tend to change over time (sometimes without warning), and I don't want to spend time maintaining something that will be infrequently used for debugging, replaying requests, or analysis.

So, as part of this side project, I was attempting to store the response directly into an SQLAlchemy model like so:

from openai import OpenAI
from openai.types import CreateEmbeddingResponse

client: OpenAI = OpenAI(api_key="YOUR_OPENAI_KEY")

embeddings_response: CreateEmbeddingResponse = (
  client.embeddings.create(input=["Some embeddings to generate"])
)

# This will throw - we can't store the object directly!
openai_response: OpenAiResponse = OpenAiResponse(
  embeddings_response=embeddings_response,  # embeddings_response is a JSONB column
)

This, of course, would not work. Trying to insert this directly into the model would produce an error like TypeError: Object of type CreateEmbeddingResponse is not JSON serializable. While exploring solutions, I came across a gist that created an explicit serialize_completion function to accomplish this. While the approach is straightforward (and unblocked me temporarily while trying to solve an unrelated task!), it’s more work than necessary and would require more maintenance and a similar manual approach for other response types.

When in doubt, just read the source code

After creating another serializer for the embeddings response, I was frustrated. What if I miss a key or the API changes silently, meaning I may be losing information or have exceptions creeping up? There are many solutions, but I decided I'd jump into the definition for the CreateEmbeddingResponse type. Luckily, OpenAI represents all of their responses as Pydantic models. Pydantic is an extremely powerful data validation library that lets us solve this serialization issue natively, instead of having to come up with our own solution:

# Using Pydantic's `model_dump_json` method,
# we can easily serialize any OpenAI responses
serialized_response: str = embeddings_response.model_dump_json()

Reviewing the documentation for pydantic.BaseModel, we can find the model_dump_json() method that allows us to serialize it directly to a JSON-encoded string. A simple and obvious solution, but a source of frustration if you don’t know (or forget) where to look. Another personal reminder of why enforcing strong typing is so useful!

A closing note: I’m currently building an ETL product based around documents and document-heavy workflows. If you or your organization might be interested, please let me know!

This post was originally published at romansorin.com

Using Django's JSONField? You probably don't need it. Here's why

Roman Sorin — Wed, 06 Apr 2022 00:59:26 +0000

In my current role, we use Django with Django Rest Framework (DRF) to power our platform’s API, and leverage PostgreSQL to store both relational and nonrelational data. As part of the platform, we run asynchronous, on-demand batch jobs that can operate up to half a million rows at once.

Each of these jobs is represented by a row in a table and contains both top-level, single-responsibility columns as well as “deeper” structures that may contain more arbitrary data. As a working proof-of-concept, we avoided over-optimizing early and didn’t have a clear definition of what these job outputs might look like. We added JSON columns to our job model that would store different forms of data - metadata, debug output, and namely, job report output.

While we didn’t need to expose metadata or debug output to our API, returning the report data to describe the job results was the main requirement. This data varied little in size and complexity, only maintaining a couple of keys and a few levels of nesting.

This job model did not have any special qualities. It was defined and represented like any other type of model within our Django app, except for additional JSONB columns to represent arbitrary and unnormalized data.

Unexpectedly, we experienced major slowdowns both in API response time and internal queries made through an RDBMS. At a point, requesting a single one of these jobs would take upwards of 5 minutes. When JSONB columns were omitted from queries, the query had a subsecond execution time.

Interestingly, we found that converting all columns that did not need to use any of JSONB’s benefits – such as indexing and advanced querying – significantly improved our application’s performance. While I’m doubtful that this was the true culprit, it did get us to our desired performance in the shortest time possible.

With that personal anecdote, let’s explore the differences between JSONB and JSON, and how you can use JSON in your Django apps.

JSONB vs JSON, and how they’re handled

Both the JSON and JSONB data types are used for storing JSON data in PostgreSQL tables. These data types are almost identical, different only in how these values are stored in the database, and the efficiency with which operations [1] are performed.

Notably, JSON stores the exact input text, whereas JSONB instead stores a decomposed binary format. As a result, there’s a small amount of write/conversion overhead (converting from text to binary), making storing input on this field slightly slower overall. The binary format of JSONB allows for significantly faster processing and operations, as the field does not have to be reparsed each time.

In addition to the fields represented in a different format, JSONB allows for indexing of keys in the field, which allows for efficient arbitrary key lookups. This would take significantly longer if attempting to search within a JSON field, as you would have to both parse the field and perform a search within a text object.

In summary,

JSON:

Stores an exact copy of the input text, including “semantically-insignificant” whitespace, duplicate keys, and ordering.
The last value of a duplicate key is used.
Must reparse the field for each operation, causing these operations to take significantly longer

JSONB:

Stores a decomposed binary format, which does not preserve whitespace (outside of values), the order of keys, or duplicate keys. Only the last value of duplicate keys is kept.
No reparsing is needed and operations over the field are significantly quicker
Can take up more disk space than a regular JSON column
Supports indexing of keys, significantly improving the performance of lookups

When to use which

Deciding which format of JSON to use will largely depend on the needs of your application. Generally, the Postgres manual suggests that most applications should use the JSONB field, except in specialized circumstances or legacy applications where keys would not be able to be re-ordered on retrieval.

In normal usage, this makes sense as you are likely to be storing some form of structured yet arbitrary data that you may need to perform operations or lookups on. However, we found that a JSON field was a better option for our use case, as we stored large amounts of varying data to be only used for retrieval without manipulation.

Deciding between JSONB and JSON will depend on several factors, such as requirements for indexing, lookups, operations, and retrieval.

The short answer is,

Use JSON when:

You are only storing and retrieving data within this field
You do not need JSONB’s benefits and would prefer to save write time and disk space

Use JSONB when:

You need indexing of keys within the object
You will perform lookups over this field
You will perform operations over this field, such as sorting, ordering, or slicing, before working with the representation in your application [2].

Switching from JSONB to JSON

If you’re using Postgres with Django and importing the JSONField object, you’re using JSONB. This is the default in recent Django versions and is a sensible choice for most applications, but you may find that you’d prefer a standard JSON based on your data and workflow.

If you find that you don’t need JSONB’s features, you have two options:

You can use a text field, which will store your JSON in the exact input format. There are a few caveats to this approach:
1. You will need to serialize and deserialize the object when working with it, especially in an API context
2. A native text field does not provide JSON input validation. To ensure that you’re storing valid JSON objects, you’ll need to validate this before inserting it into your database.
3. You will not be able to retroactively cast the field to a jsonb if you later decide to use JSONB format.
If you want the convenience of a package, you can use packages like django-jsonfield or jsonfield, which will handle validation and serialization by default. If you opt-out of using JSONB, this is the preferred option.

If you’re using other databases like MySQL, you may opt for a package like django-mysql, which provides support for JSON fields and extends the functionality of MySQL in your Django application.

Using JSON with Django Rest Framework

To solve the performance issue, we opted to switch to using the JSONField field provided by django-jsonfield, allowing us to remove JSONB from the model entirely. With our new JSON fields, we suddenly found that the data provided by our API to our processing/job service was causing exceptions when we were attempting to extract and load information from the object. Where we originally were inferring a native JSON object, which was already deserialized by Django, we instead were receiving a JSON object in the form of a string. Our data providers didn’t expect this and quickly began failing when we attempted to use any metadata from our API.

After some experimentation, we found that responses needed to have these new JSON fields deserialized manually before being returned in the API. Django handled this originally, but switching to a non-default field required some adjustment.

Assuming that you’re using serializers within Django Rest Framework (DRF), you can update the serializer for the field to be a SerializerMethodField, which can be used to add any arbitrary data to your object’s serialized representation or change the representation of an existing field. Here’s how you can deserialize the JSONField within a serializer:

from rest_framework import serializers
from rest_framework.fields import SerializerMethodField

from models.cluster import Cluster

class ClusterSerializer(serializers.ModelSerializer):
  metadata = SerializerMethodField()

  class Meta:
    model = Cluster
    fields = ["metadata"]

  def get_metadata(self, obj):
    return obj.metadata

This forces the field to deserialize from a string to a Python dict. If you intend on using this field in more than one serializer, you may benefit from adding a property to the model to automatically deserialize this field instead of within your serializer:

from django.db import models

import jsonfield

class Cluster(models.Model):
  metadata = jsonfield.JSONField()

  @property
  def _metadata(self):
    return self.metadata

Further anecdotes

Admittedly, there were too many confounding factors relating to this issue to blame for the usage of the JSONB field. Because we had resolved the problem with this change, I stopped researching possible causes beyond what I had originally found and stopped looking further into performance benchmarks.

Jobs in this table and the corresponding column were being populated by data produced through a disconnected processor. The output had been sent through our API at the time (this has since changed as another performance consideration), and inserts were being handled by our main Django application.

Based on our usage, there may have been issues with indexing, or the query planner was somehow unable to work with the data in this column across all of our jobs. As we experiencing slowdowns both in our application and using an RDBMS, I’m unsure if this was amplified by passing the model to a DRF serializer, which has an overhead of its own; or if our managed database simply could not handle returning this data as part of our queries.

Interestingly, we had millions of rows in another table that had five times the number of JSONB columns with no obvious differences in performance. These JSONB columns were fully queryable and responded in a reasonable amount of time, concerning the size of the data being returned, despite no difference in the model’s definition of the JSON field. Many of these columns had significantly more data than the related column in any of the problematic jobs, and yet they were handled significantly better by the RDBMS. Our problematic table with only a few rows took up around 16MB of space, whereas switching to a normal JSON reduced table size to a few KB. The table with millions of rows and sets of JSONB columns held 20GB+ of data and responded within seconds to both complex and simple queries.

While this change likely isn’t the single reason behind performance improvement, it helped recognize that there is a need to make considerations about how you’re storing and handling related data. As our system matured and we had more examples of what distinct outputs of these jobs may look like, we eventually normalized this report column into its separate table, allowing us to improve performance even further by lazily loading report data while keeping our API less complex. After reviewing the output of several distinct jobs, we extracted upwards of 20 attributes into dedicated columns with explicit data types, which will help significantly with performance over time as the size of the data stored in this table continues to grow.

I don’t have the full answer to what happened here, as I didn’t get to spend more time investigating. If you have any thoughts on what may have occurred beyond what I’ve covered here or have experienced this yourself, I'd love to chat.

Footnotes

[1] Operations here refer to things like search, sorting, slicing, and ordering done directly through database queries.

[2] "Working with the representation in your application" refers to retrieving your object from the database, storing it in memory/as a local variable, and then working with the local object. At this point, it wouldn't matter what format the JSON is stored in, as you'll only get JSONB benefits when operations are done within the database context.

Deleting Git branches in bulk

Roman Sorin — Wed, 09 Mar 2022 00:02:01 +0000

Deleting several Git branches at once with commands like git branch -D can become a real headache. At any time, I have between 5-15 branches per project, all of which I may complete at random times based on priority and state. If you forget to delete your local branches after merging your work, you may end up having weeks of stale branches living within your project.

Thankfully, there’s an easier alternative to this, and we can run a single command using commands like a pipe (|) and grep to delete several branches at once.

The command

Deleting branches in bulk can be done in a single line:

git branch | grep -iE 'FIX|FEAT' | xargs git branch -D

Let’s take a look at what’s being done here:

First, we define our first command, git branch, on the left side of the first pipe. This lists all of the branches within your project.
Second, we define a grep command which will search the output of our branch command. Since we are using a pipe, the output of git branch is passed to grep. To match several branch patterns at once, we define the -E flag. This allows us to match several strings while also supporting extended regular expressions. We also may have case variations between “FIX”, and “fix”, meaning our grep should be case-insensitive. We do this with the -i flag.
Third, we define the normal branch deletion command on the right side of the last pipe, and also supply xargs as part of this command. xargs, or extended arguments, will take the left side of the last pipe (our grep command) and pass it as arguments to git branch -D, calling the branch deletion command for each matched branch.
As a result, all branches that start with “FEAT” or “FIX” (case insensitive) are automatically deleted.

Visualizing the deletion

To visualize this, let’s run each of these commands individually:

$ git branch

FEAT/RS-123
FIX/RS-456
IMPROV/RS-13
feat/RS-96
fix/RS-56
REF/RS-250

Running git branch returns all of the existing local branches. Now, let’s try to find all of the branches that we want to delete:

$ git branch | grep -iE 'FIX|FEAT'

* FEAT/RS-123
* FIX/RS-456
* feat/RS-96
* fix/RS-56

Our grep command highlighted the matching branches and each matched phrase. Using the pipe operator, we passed the output of git branch into the grep.

If we typed the deletions manually for each branch, instead of using the full command, it would look something like this:

$ git branch -D FEAT/RS-123
Deleted branch FEAT/RS-123 (was 1b39b22).

$ git branch -D FIX/RS-456
Deleted branch FIX/RS-456 (was ea9973f).

$ git branch -D feat/RS-96
Deleted branch feat/RS-96 (was 6c114fb).

$ git branch -D fix/RS-56
Deleted branch fix/RS-56 (was 5a82faa).

We can now see that instead of a single command that takes advantage of pipes, we had to type 6 different commands, each of which has a standard output of its own – making it tougher to keep track of what was deleted, and what still needs to be cleaned up.

Passing data using pipes

Many resources exist on pipes and their usages, but it may help to visualize usage in the context of deleting this branch deletion command. In our case, the condensed data flow looks something like this:

git branch -> grep -iE 'FIX|FEAT' -> xargs git branch -D

Each side of our pipe has a “standard input” (STDIN) and “standard output” (STDOUT). The STDOUT is the result of running a command and is passed in as STDIN to the next command. In our flow:

git branch (STDOUT) ->
(STDIN) grep -iE 'FIX|FEAT' (STDOUT) ->
(STDIN) xargs git branch -D (STDOUT)

While this is an overly simplistic explanation of what is happening behind the scenes, it hopefully provides an idea of how commands and their outputs can be spliced together using pipes.

Disabling the Tailwind input ring

Roman Sorin — Tue, 01 Feb 2022 13:30:57 +0000

If you’ve ever worked with Tailwind components or TailwindUI, you’ve probably come across the “focus ring”. On inputs (checkboxes, fields, etc.), this ring is shown when you’ve clicked or tabbed onto a component, indicating that you have selected it. Here’s an example:

Disabling the ring

I’ve seen myself and others try to disable this by simply calling the focus:ring-0 utility. This sort of works, but gives us this weird shadowing effect on the corners of our input:

By default, Tailwind will include this ring and an offset on certain input elements like checkboxes. If you’d rather not have this ring on all interactions, you can easily disable it:

<input type="checkbox" className="focus:ring-0 focus:ring-offset-0" />

With this, we’re able to get a nice looking checkbox, without the “ugly” looking ring:

Keeping things clean

To reduce redundancy across my project, I created a class that I can globally apply:

.without-ring {
  @apply focus:ring-0 focus:ring-offset-0;
}

Now, instead of re-defining these utilities on all of my inputs, I can use the without-ring class to achieve the desired output:

<input type="checkbox" className="without-ring" />

Keeping it accessible

Accessibility is important, but to keep it short, I won’t cover it here. Instead, this post by David Gilbertson provides a simple, framework-agnostic solution to showing the focus ring only when a user is tabbing and making it hidden otherwise.

Keeping it within Tailwind, we can use both our pre-defined without-ring class and native CSS selectors to show the ring for keyboard users, but hide it when using a mouse:

.without-ring:focus:not(:focus-visible) {
  @apply focus:ring-0 focus:ring-offset-0;
}

.without-ring:focus-visible {
  @apply ring-2 ring-offset-2;
}

// If we wanted to apply this globally, we could use native type selectors

[type="checkbox"]:focus:not(:focus-visible) {
  @apply focus:ring-0 focus:ring-offset-0;
}

[type="checkbox"]:focus-visible {
  @apply ring-2 ring-offset-2;
}

Cherry-picking done easy

Roman Sorin — Thu, 27 Jan 2022 19:05:06 +0000

Have you ever needed to cherry-pick several commits between two branches, but ran into conflicts every time? I made this mistake myself – but cherry-picking doesn’t have to be hard.

As you work on more features and fix more issues, some might not be released immediately. When moving code between environments, you may need to move specific changes over, even if they live alongside other things that you’ve written. This is where cherry-picking comes into play, and if you’ve ever worked in any type of git-flow, you’ve probably come across it.

Strategies to the rescue

Merge strategies allow you to define how to apply commits from one branch to another. By default, the cherry-pick strategy is recursive. Depending on what base branch you’re working off of, you’ll want to use the correct merge strategy for specifying what changes to use: this is where -X=theirs and -X=ours do us favors.

Performing the cherry-pick

The cherry-pick command is simple:

git cherry-pick --strategy=recursive -X=theirs hash0 hash1 hash2

You’re able to specify the strategy, the per-strategy option (theirs/ours), and the sequence of hashes (commits) to merge.

What is the difference between theirs and ours?

This option will specify which version of code to prefer in a merge. If you’re on the main branch and moving commits from develop, you’ll want to use theirs, as you are saying “let’s take develop’s (their) changes”.

What you’ll need:

A base and target branch (develop/main)
The commit hashes to cherry-pick (e.g. 7f7a499)
Choosing the right strategy based on what you want to achieve (ours/theirs)

For example, to merge a sequence of commits from a development branch into main:

Check out the main branch with git checkout main.
Using the cherry-pick command from above, we will merge several develop commits to our main branch, moving the develop changes onto main. You can specify the hashes individually, or you can cherry-pick a sequence of hashes by writing hash0...hash1.
If conflicts come up, you’ll be able to resolve them on a per-commit basis and commit the result to continue with your cherry-picking.

Although simple, learning this has turned large merge requests that affected over 40 files (including deletions, renames, and refactors) into a pain-free experience instead of a major ordeal. Something that would’ve taken me several hours with the potential of missing code only took an hour, and shortly thereafter, I was able to move on to working on my next task.

This isn’t all that cherry-pick has to offer, and there are flags for you to provide based on your use case. If you need something beyond this or if you want to read a little bit about how cherry-picks are done, the Git docs do a great job of explaining this concept.

The Impact of Layout on User Experience: A Study

Roman Sorin — Wed, 01 Dec 2021 22:14:49 +0000

A year ago, I wrote a research paper titled Identification and Measurement of Hierarchical Layout Patterns on User Experience. This paper documents a study that spanned over several months (September of 2019 until April of 2020), in which I performed a two-part analysis & experiment on the most popular site layouts and how user experience is affected. While this paper barely scratched the surface of what is possible, it documented a novel approach to studying user experience and how it may be possible to measure "success" through objective metrics — both quantitative and qualitative. Below is the abstract of that paper and a link to continue reading.

Abstract

User experience continues to grow in importance as web applications become increasingly widespread and utilized. Further, high-quality user experience often leads to increased user retention, product usage, and revenue for businesses and applications alike. Though user experience is a focus for some, there is often ambiguity in what an optimal or effective layout looks like, and how components within the hierarchy should be structured to accomplish a set goal. This study proposes a potential approach to objectively analyzing and identifying layouts from the highest indexing sites on a global scale, and methods of implementation to assess whether or not certain metrics representing user experience are impacted by layout. After analyzing 908 screenshots and four clusters from the globally highest ranking sites using a K-means algorithm and weighted subclustered images, three common layouts, or variants, were identified and created for use in a web application. Users were randomly assigned one of three layouts, and they were instructed to navigate through the layout to register or purchase the product while their session lengths and mouse movements were recorded. The results from this study found that while the differences in variances between the mean session lengths from 55 participants were statistically insignificant, interesting patterns were found from a manual qualitative analysis when comparing interface hierarchies to data. Subsequently, it was found that various avenues exist for improvement of the model, such as an objective, mathematical metric to understand how layout impacts user experience metrics.

Read the paper here.

More information

Title: Identification and Measurement of Hierarchical Layout Patterns on User Experience

Written: April 22, 2020

Keywords: user experience (UX), K-means clustering, Keras VGG16, AWS Alexa

How do you like to read articles/tutorials?

Roman Sorin — Mon, 22 Nov 2021 14:20:19 +0000

When reading documentation or articles that contain code, do your prefer to:

step through the file line-by-line
see the full code snippet pasted with inlined comments
see the full snippet, and a summary/walk-through after?

Here's an example of #2/3 taken from a recent post of mine:

import hashids from "hashids";

// Two arguments supplied: a salt and a minimum padding (length)
const postHash = new hashids("post", 8);

const post = { id: 4 };
post.id; // 4

const hashedPostId = postHash.encode(post.id);
hashedPostId; // 6akz1JVq
postHash.decode(hashedPostId); // [4]

"Here, we're importing the hashids package and creating an instance of the module, calling it postHash. ..."

vs. doing something like:

const postHash = new hashids("post", 8);

"First, we create an instance of the hashids module and supply a salt and padding. ..."

Hiding primary keys and database IDs using Hashids

Roman Sorin — Mon, 22 Nov 2021 13:07:51 +0000

In most CRUD operations and REST APIs, primary keys are used to reference models that you want to access or modify. A majority of APIs will take an ID as a parameter in a route:

GET /api/v1/posts/:id

// Return the Post resource with an ID of 457
GET /api/v1/posts/457

While it's the simplest and most effective way of specifying the model to use, we often don't want to show these IDs to the user. By displaying primary keys, you give users the ability to estimate the number of rows in your tables. If authorization isn't effective or routes aren't protected, users could input random numbers to access information that they otherwise shouldn't have.

Using obfuscated IDs can be useful in social media or feed contexts, where the content isn't used in the URL, but you want something less significant than a primary key. As an example, instead of showing the user a URL like this:

https://romansorin.com/posts/457

We may want to show them something like this instead:

https://romansorin.com/posts/akz1JV

In this case, you may want to use "hashes" as a way to obfuscate your ID. We'll use the lightweight Hashids package to make this happen.

Installation

Getting started with Hashids is simple. With your preferred package manager, add Hashids to your project:

# If you use yarn as a package manager
yarn add hashids

# Or if you use npm
npm install hashids

Usage

I've provided a Javascript example to begin working with Hashids, but Hashids has support for several languages!
Here's a brief use case, where you may want to hide the ID of a post:

import hashids from "hashids";

// Two arguments supplied: a salt and a minimum padding (length)
const postHash = new hashids("post", 8);

const post = { id: 4 };
post.id; // 4

const hashedPostId = postHash.encode(post.id);
hashedPostId; // 6akz1JVq
postHash.decode(hashedPostId); // [4]

Here, we're importing the hashids package and creating an instance of the module, calling it postHash.

I set up a dummy post object, but you can use any object or ID that you see fit. Due to package limitations, the argument supplied to the module must be an integer. Strings and objects cannot be obfuscated using Hashids.

Afterward, I supplied the ID into the encode function of the postHash object and then put this output back through the decode function to show how you can use encoding/decoding. Note that the return type of decode is an array, not an integer.

If that's all you're looking for, then that's it! You can also encode and decode multiple IDs at once:

const hashes = postHash.encode([1, 2, 3]);
postHash.decode(hashes); // [1, 2, 3]

Bonus: A utility class

If you want a common utility to work with, here is an abstraction on top of the Hashids package that will allow you to encode and decode IDs easily, without having to remember the package's methods.

This class is limited to encoding/decoding a single ID at a time but it helps me stay consistent within my projects. By using this utility, you could also set up a file/store of your hash objects, so you don't have to redefine it across your application:

// lib/Hash.ts
const Hashids = require("hashids/cjs");

export class Hash {
  private hashids;

  /**
   * Creates a Hash object.
   *
   * @param {string} salt The unique salt/alphabet to use for salting. Setting a salt allows output hashes to be more unique.
   * @param {number} padding The minimum output length of the hash (default is 6).
   */
  constructor(salt: string = "", padding: number = 6) {
    this.hashids = new Hashids(salt, padding);
  }

  /**
   * Encodes the provided ID argument and returns a string representing the hash.
   *
   * @param {number} id The numeric "id" to be encoded or turned into a hash.
   * @returns {string} Returns the encoded ID in the form of a hash, e.g. "o2fXhV"
   */
  public encodeId(id: number) {
    return this.hashids.encode(id);
  }

  /**
   * Decodes the provided hash argument and returns a number representing the ID.
   *
   * @param {string} id The numeric "id" to be encoded or turned into a hash.
   * @returns {number} Returns the numeric ID, e.g. "1"
   */
  public decodeId(hash: string) {
    const decoded = this.hashids.decode(hash);
    return decoded.length > 1 ? decoded : decoded[0];
  }

  /**
   * Sets the internal hashids object with the provided salt/padding arguments.
   *
   * @param {string} salt The unique salt/alphabet to use for salting. Setting a salt allows output hashes to be more unique.
   * @param {number} padding The minimum output length of the hash (default is 6).
   */
  public setHashids(salt: string = "", padding: number = 6) {
    this.hashids = new Hashids(salt, padding);
  }
}

Using this utility class is as simple as the native Hashids package. The implementation stays largely the same, but may be more readable and easy to remember:

const { Hash } = require("@lib/Hash");
const { Post } = require("@app/models/Post");

// Create a new Hash object with the salt "post"
const postHash = new Hash("post", 8);

// We may want to generate different sequences based on model, to get different values for the same ID
const userHash = new Hash("user", 8);

post = new Post();
post.id; // 4

const hashedPostId = postHash.encodeId(post.id);
hashedPostId; // 6akz1JVq
postHash.decodeId(hashedPostId); // 4

// Want to change the salt of the Hash object without creating a new object?
// Call "setHashids" through the utility function.
postHash.setHashids("comment", 8);
postHash.decode(hashedPostId); // Now, it returns undefined instead of 4

// With a different salt, we can use the old Post ID and get a different value:
const hashedUserId = userHash.encodeId(post.id);
hashedPostId; // dD0WnjRy

This example is a little bit more extensive, so let me walk you through it:

We created two hash objects to represent a Post and User model.
Like the previous example, I created a dummy Post object with an ID of 4.
I passed the ID into the encode function (of the custom utility) and then decoded it, which was the same as the previous example.
The utility allows you to set a new salt and padding within the same object instance, so I changed the salt to "comment". Now, when you try to decode the previous hash, you don't get the same ID.
Since the userHash object had a different salt, encoding the previous ID returns a completely different hash.

Unfortunately, a limitation of this utility is that you can't encode or decode multiple IDs at once, but this can be easily added in by extending the class functions. When developing a medium-scale app with the Hashids library, I found this utility to be super useful in keeping my code consistent across controllers.

Limitations

It's worth noting that Hashids should not be used as a legitimate hashing solution (such as passwords or other encrypted information). The Hashids package doesn't have support for strings anyway, but you shouldn't even consider this – instead, use an algorithm like bcrypt to encrypt your sensitive data.

Additionally, as the package creator describes, these aren't true "hashes". Cryptographic hashes cannot be decrypted, but the output looks similar which results in this obfuscation being considered a "hash".

More robust strategies

When I was looking into packages and solutions to masking IDs in an application of my own, my first thought was to look into what companies like Instagram and Twitter were doing. I noticed that despite the volume of data that is processed on these platforms, they didn't resort to using primary keys for their URLs. If you're interested in how they handled this ID generation (hint: it wasn't Hashids!), I would highly suggest reading the articles I linked above.

In the Medium post documenting Instagram's solution, the URL contains yet another example of hashes being used in the URL: first the slug of the article, and then a sequence of random characters after to maintain uniqueness.

https://instagram-engineering.com/sharding-ids-at-instagram-1cf5a71e5a5c

In a content-heavy context where the title of a post may be significant (blogs or forums), this approach keeps the URL significant but also minimizes the chance of collisions by keeping records unique.

Hashids is an effective approach for a small to medium-scale application that doesn't require more complicated strategies like combining different metadata (creation date, worker/sequence count, shard IDs). While it doesn't fit data or scale-intensive cases like Twitter, regular applications that process a reasonable amount of writes will do just fine with this approach. Collisions can be avoided by choosing unique salts for each model that you obfuscate and an appropriate minimum length (at least 8 characters).

Sticky Footers

Roman Sorin — Mon, 15 Nov 2021 13:26:48 +0000

In most of my apps, I almost always want the footer at the bottom of the page.

This is really easy to accomplish:

first, we make the actual page at least the height of the screen
then, we move the footer to the bottom of the page with auto margin if there isn't enough natural content

With CSS

With a simple page like this,

<body>
    <div id="app">
        ... other content ...
      <footer id="footer" />
    </div>
</body>

our CSS looks like this:

#app {
  display: flex;
  flex-direction: column;
  min-height: 100vh;
}

#footer {
  margin-top: auto;
}

With Tailwind

Tailwind makes it even easier.

<body>
  <div class="min-h-screen flex flex-col">
    ... other content ...
    <footer class="mt-auto" />
  </div>
</body>

Zillow's confusing keyword UX

Roman Sorin — Wed, 27 Oct 2021 02:20:14 +0000

Zillow has consistently been one of the most popular real estate marketplaces for several years, and for good reason — Zillow has dominated the space for buying and selling homes, as well as listing rentals and providing home loans for prospective buyers. While searching for apartments this summer (and as I'm planning to buy a home), I used Zillow almost exclusively to explore options and contact properties.

Almost half of all people searching for homes start by looking online before contacting a local realtor. Despite the amount of information that searchers have immediately at their disposal, finding a property is nothing like a product on Amazon. Homebuyers using Zillow and similar platforms spent, on average, ten weeks searching for homes; those who took traditional routes spent closer to four weeks.

This brings us to one of the key features in any search or commerce context: filtering. For being such a large name in the real estate space, it's hard to believe that the options for filtering are so primitive. Filtering is limited to specifying a price range, beds/baths, and other basic features, but almost nothing more. As you use the platform, you have to potentially spend hours trying to find properties that fit your criteria. While this increases engagement, you end up frustrating the user.

Intuitively, you would expect the "keywords" field to give you more control of what you're looking for. The ability to specify certain words or phrases is convenient, especially when you're looking for a certain style of home or other qualitative traits beyond the generics. With so much information attached to listings, it's surprising that there isn't a more developed way of finding this deeper engrained yet relevant information.

With this, let's explore the keywords field: what it gets right, how it can improve, and how other platforms can change the approach to home searching.

What they get right

From the input, it's clear that they expect you to input keywords as a comma-delimited list. After you've finished and clicked "Done", your list of results decreases. This is expected behavior but doesn't offer much feedback. Zillow isn't alone in this behavior – another well-known marketplace, Redfin, also has the same "keywords" field that functions similarly.

The keywords input is clear as it's comma-delimited and gives examples of what it's looking for.

Without trying several cases to see how the search performs, it's difficult to see how the keywords field works. After trying several sequences of keywords ("open", "open, modern", "open, modern, pool"), I found that it matches each keyword and works as a strict filter, rather than evaluating each keyword individually.

This works, and once you understand it, functions as expected. But it's kind of useless.

Complex queries aren't possible

To start, I made a baseline search around Dallas, TX:

Price Range: $400k-$1m
Home Type: Single-family, for sale
3+ beds, 2+ baths

Results: 5,964 listings

This yielded 5,964 listings which is a healthy number to filter down. At this point, any home matching these criteria will be returned and doesn't take other features into account.

As mentioned before, I'm using a combination of phrases such as "open, modern, pool" to evaluate how filtering works with this field and what it potentially misses.

Here are the results of using different combinations of these phrases to illustrate how big of an impact the keywords field has on search results:

Listings by keyword:

"Open": 2,749
"Pool": 1,273
"Open, Pool": 502

A single keyword significantly reduces the number of results and hopefully can filter out non-matching properties. However, this field is only useful if the information is available and it takes the correct information into account.

Wildly inaccurate

With a little bit more investigation on my baseline query, I decided I'd try to test the accuracy of the keywords field. As an example, I entered "3 bed" into the field on its own, which reduced the number of results to 162 listings. This seems somewhat promising, yet my first result was a four-bedroom house. Even if I'm allowing more than three bedrooms in my base query, properties matching the keyword should be returned first. What gives?

Just supplying "3 bed" as the phrase for the keywords returns 162 listings, opposed to the original 5,964.

I looked throughout the listing to find any mention of "3 bed" or the number 3 at all. All I could find was the "3 CAR GARAGE" tag, which is not even contextually relevant.

With "3 bed" in the keywords input, this listing is returned — but the description has no mention of "3 beds" (only "3 car garage"), and the house itself has 4 bedrooms.

I looked to see what the sorting options were for results to see if there was a "Relevance" option. Nothing would change the results appropriately to what I wanted, despite being a very basic filter. The nearest option is "Homes for You", which is supposedly determined based on the homes you've viewed and your prior searches, but I don't want a feed — I want better search results. Yes, there is a dedicated filter functionality for choosing the number of beds you want to have returned, but this result speaks volumes on the limitations of filtering.

Zillow offers different types of sorting options, but nothing to do with search relevancy beyond "Homes for You".

It's evident that when the filter does work, it seems to only search the description of the listing. This leaves out a lot of valuable information, especially if the description is incomplete, or if the relevant data is in some other property of the listing. Despite the number of attributes that can be filled out for a listing, these are very loosely compared against the keywords input.

For final verification of the attributes that are likely considered, I searched the "Parcel Number" that was found under a property listing in the keywords (Zillow gives an example of "MLS #" in their input, so I figured other IDs would be supported), and that listing could no longer be found. While I don't know how their data is modeled, I believe they could instead do a more exhaustive, comprehensive search over the listing with proper indexing.

A better way of home searching

From testing, it seems that keywords are all evaluated as a strict filter, meaning that they are matched directly against. I didn't see support for fuzzy text searching (i.e. misspelling a keyword may return zero results), and the "3 bed" experiment showed that there is no contextual relevancy.

On Zillow and other platforms, it would be nice if you could input a more generic search term or phrase that allowed for more specificity. There may not be a huge market need for it, but it could potentially be a major help to people who want to find homes that are aligned with their needs, without having to reach out to realtors and agents. I haven't come across a real estate listing site that operates as a search engine – while it's a lot to ask, it would offer homebuyers a faster path to finding candidate homes that they might want.

Aside from working with a realtor, it can be tough to find exactly what you want without spending a lot of time looking at options. Even if you're casually browsing, it can be frustrating to have a massive list of potentially irrelevant options shown to you, especially if you know what you're looking for.

I briefly explored this issue earlier in the year while developing an MVP for a small-time project of mine called Abode. The model for home searching has been consistent for years, but that doesn't necessarily mean it works well, especially as usage patterns evolve.

While drafting ideas on how to solve some of these problems, I came across an article written by Zillow which explored improving results by putting the listing's description through an embedding model paired with existing recommender systems. The article states that Zillow uses different features of the listing "to build recommendations on similar homes, personalized search rankings, and other personalization systems for customers", so it seems that much of this recommendation is done behind the scenes, without the user being aware of it. It's a bit ironic that their behind-the-scenes systems seem to be well-thought-out, but their search functionality is so primitive – despite being the point of entry for users looking to find a home.

It seems like Zillow is working on solutions to solve this through their existing filters, which looks to be based on localized demand (on-site parking, allowing pets in complexes, etc.) – but still haven't offered a solution to more "fuzzy" suggestion. Using embeddings and making predictions based on provided phrases seems like a great solution to improving search quality and allowing for a more attractive search process. While I'm doubtful Zillow will solve this problem themselves, it could be an attractive offering if a new platform presents itself.

But is it necessary?

Admittedly, it might not be solving all that large of a problem. After looking at various results on Google Trends, the most popular searches are things like "new construction houses for sale", "beach houses for sale", "manufacturer homes" — results which are easily returned through the dedicated filters. But, if a property listing is incomplete beyond the basic information and a description, information that is important to some people (such as the "view" or certain amenities) may be lost. People searching for apartments, for example, may input several criteria into the keywords field, but don't expect them all to be matched. Even if they do, too many keywords may return no results. Apps like ApartmentList have solved some of these issues, but it's less known than Zillow and other platforms.

Real estate listing sites and aggregators are difficult to do, in some respects. You can't always have perfect information, especially at scale — but there are ways of solving some of the limitations. The root of this problem seems to be the incompleteness of data. In the future, I would love to see a real estate platform use more advanced concepts, such as RNNs/CNNs, to automatically extract features from photos to pre-fill or supplement listings. Building extra layers such as checking image quality, description quality, and providing feedback or predictions on how well a listing may perform may ultimately ease frustrations for a homebuyer.

Design for your developers, not just your users

Roman Sorin — Sat, 16 Oct 2021 14:15:35 +0000

As a developer, I have worked on many feature definitions, tickets, and mockups. Working on both freelance projects and at companies, I have come across unclear requirements more times than I can count. Though it's impossible to be perfect, there are ways to improve developer productivity and establish a better foundation for how your apps are built.

It starts by designing for your developers.

Though users are typically the primary stakeholders, recognize that developers are the bridge between concept and execution. Developers are limited by similar factors; time and budget being two of the most common. However, some limitations are entirely within a designer's control.

For example, designs that are overly complex or ambitious can be a nightmare for developers. Even if the design looks eye-catching (think most Dribbble shots), it might simply be too difficult to implement or even use. Similarly, developers can perform higher quality work at a much faster rate if there doesn't need to be a constant feedback-support loop. A developer shouldn't have to constantly ask what should be displayed in some input case (think failure/success handling), or why two elements are inconsistent in design despite supposedly being the same.

Here are some basic but common examples of design states that are often forgotten:

Null/empty values
New values (reassigning from null)
In progress, loading, or pending states
Error, success, etc. Should these be messages? Other visual cues?
Consistency in implementation, such as text size and color, padding, grid spacing

While developers should be responsible for making reasonable assumptions and maintaining autonomy, their job is to produce a fluid, functional application from what's provided. If given something difficult to work with, there's no guarantee that the actual app will be any better.

Confusion breeds chaos and makes it tough for many devs – especially those who have little interest or background in design – to accomplish their work in the expected amount of time. For an application to be well-executed, all three points need to interact and consider each other's needs: designers, developers, and users.

As a designer, you’re probably dealing with different sets of requirements, communication across different departments, and responsible for designing flows. You may be constrained by deadlines or financial limitations, and so it’s tough. But keep developers in mind, and they’ll return the favor through a beautifully executed user experience.

In exploring this idea, I found that Invision's Inside Design published an article that covered a similar idea. It's more in-depth and put together than this article, and I highly suggest you read it if this idea interests you.

You can find more of my content here.

My experience of moving from Laravel to Django

Roman Sorin — Sat, 09 Oct 2021 18:10:49 +0000

Before my current role, I spent most of my professional development career writing web apps in Laravel. I learned Laravel out of necessity after learning PHP, but wanted to avoid working in WordPress – and I quickly grew fond of the framework. Even during Laravel 5, the framework just felt right.

It was highly opinionated (which was crucial as a new developer, trying to learn the best way to structure large scale, multi-faceted apps), had robust documentation that was extremely easy to follow, and had very clear design patterns that made it easy to interface with the ORM and concepts like Facades. When scaffolding out a new Laravel project, namespaces were clear and it was easy to see how different parts of the app interacted with each other. Laravel's use of Artisan was also a major developer experience boost – you could generate new controllers, models, notifications, and more – without having to rely on any other external tools or questioning if it's structured properly. It just worked.

Later on, I moved to using Django as my primary framework. I figured that the transition between Laravel to Django would be simple, but even some of the more basic concepts gave me trouble regarding terminology and patterns.

Why Django?

Whenever I was looking for new roles at the beginning of 2021, I ended up interviewing for a company (where I currently work) that was using Django as a large part of its stack. Consistently, I was seeing more opportunities for Django and other frameworks revolving around Python and felt that I may have been inhibiting my growth by staying on the Laravel track.

Even though I had used Flask before in several freelance projects, Django felt like a completely different beast. Flask touts itself as a "micro-framework" and compared to full-fledged frameworks like Django or Laravel, that's exactly what it is. It's very convenient for creating more directed projects quickly but falls short with tooling.

I had a preliminary interview and immediately began learning Django. Despite making it clear that I hadn't used Django before, I wanted to get as much knowledge of the framework as possible so that I wouldn't be subject to a major learning curve before beginning my new position.

I didn't want to put all of my eggs into one basket, so I kept the sample project I was working on small. It had to be just enough that I covered the major concepts of Django, but with enough complexity that it couldn't be classified as a "to-do list". At the time, I wanted a way of tracking inventory in my home office, so I built an incomplete inventory tracking system after following the Django docs. Despite having a lot more experience with the framework now, I still reference this project sometimes when I'm implementing new viewsets (more on this later).

Side note: If you really want to work at a company and don't have a solid grasp on the tech that they're using, pour your time into it (within reason). While it's perfectly okay to learn on the job (as I did, and still do), being able to hit the ground running and contribute is personally incredibly rewarding.

The parallels between Laravel and Django

While this won't be a detailed technical writeup of the two, there are some core concepts that exist in parallel between Laravel and Django. To start, Laravel and Django both follow the Model-view-controller (MVC) design pattern, which is a common way of developing both desktop and web applications.

Freedom to choose

Though they both follow MVC, Laravel feels more "strict" in enforcing certain patterns. If you've ever taken a look at the Laravel documentation or inside of a Laravel project, you'll see that there are a lot of different features of the framework that are already abstracted out for you. At a minimum, you run php artisan and generate the components that you need – at maximum, you have to do some configuration to make it fit eloquently into the rest of your app. I use Laravel's documentation as a gold standard in framework + technical documentation and wish more would follow suit.

Contrarily, I have always felt that Django gives you a bit more freedom to make your own decisions on how you want to structure your app, what namespaces (module directories) to use, etc. Django's documentation isn't fantastic compared to Laravel, but it's enough that you can reference and implement different concepts.

Building an API

Django also provides you with an "admin" view, which is essentially a CRUD panel for any models that you load. This is probably a better choice for someone looking for a framework with an admin dash prebuilt, but I use both Laravel and Django in the context of building an API only. With Django, you'd need to pull in Django REST framework (DRF) – Laravel offers it right out of the box.

Tinkering with data

Both Laravel and Django have a CLI tool that allows you to use parts of your app in isolation and interact with the database. This is really useful when you want to see what certain methods of the query builders do, or one-off functions that would take more writing in a controller/viewset. I personally feel that php artisan tinker is easier to use and integrated more closely with the framework. It's an incredibly tiny detail, but tinker utilizes autoloading and allows you to reference models by namespaces, without having to import them. This is different from the Django manage.py shell, where you have to import all of the modules and classes that you want to interact with.

Where my confusion lied

Following the MVC pattern, Laravel and Django both have models, views, and controllers. However, I felt that Django was a lot less clear in designating what maps to what, at least coming from a traditional background. Let me explain:

Laravel

Two majors concepts of building APIs in Laravel are models and controllers.

Model: A blueprint for objects, which is just a class that allows you to specify a different table name, primary key, attributes, etc. Unlike Django, defining fields is abstracted away to migration, and Laravel separates the Eloquent instance from the database representation.

<?php
// App\Models\Employee.php

namespace App\Models;

use Illuminate\Database\Eloquent\Model;

class Employee extends Model
{
    //
}

Controller: Typically, controllers return model instances with minimal processing. These are where you handle defining CRUD operations for your endpoints (web or API). Unlike Django's optional functional approach, these are always classes.

<?php
// App\Http\Controllers\API\EmployeeController.php

namespace App\Http\Controllers\API;

use App\Http\Controllers\Controller;
use App\Models\Employee;

class EmployeeController extends Controller
{
    public function show($id)
    {
        return response()->json(Employee::findOrFail($id));
    }
}

Views: Laravel views are Blade templates that are separated from your business logic, and only hold the presentation layer. These are used to display data provided from controllers – the use of "views" is going to be an important distinction in Django, where these are called "templates" in a Django context.

When defining an API route, you only need to reference the method of an API controller. You don't need to touch "views" at all, and in a lot of my projects, I never created a single view (I used a separate React/Vue app for this).

Django

On the other hand, Django uses the concepts of models, serializers, and views (which can be called viewsets). Even for API routes, all three of these are required.

Model: The model is similar to Laravel's definition, but must contain the fields and metadata at a minimum. If you want to define any custom attributes or model-specific business logic, this must be done in a serializer. On one hand, it's easier to follow and practices better separation of concerns – on the other hand, it's another file to create, maintain, and reference.

# app/models/employee.py
from django.db import models

class Employee(models.Model):
  name = models.CharField(max_length=255)
  age = models.PositiveSmallIntegerField(null=True)

Serializer: Serializers are going to be your definition for what a model will return. You have the ability to include/exclude fields from responses, define nested representations (show the entire User model, instead of just a user_id ), and create method fields that will return some value every time your object is being returned. Essentially, splitting a Laravel model returns a Django model + serializer. A serializer is technically an offering of DRF, but if you're building an API, you'll have this.

# app/serializers/employee.py
from rest_framework import serializers

from app.models import Employee

class EmployeeSerializer(serializers.ModelSerializer):
  class Meta:
    model = Employee
    fields = ['name', 'age']

Views/viewsets: Django views are Laravel's controllers. Here, you can define individual functions (similar to single-action controllers in Laravel), or entire viewsets which hold related methods (controllers). The use of "views" as terminology for a controller was one of the hardest things to wrap my head around. Below is a view built using DRF, which differs slightly from a generic Django view.

# app/views/employee.py
from rest_framework import generics

from app.models import Employee
from app.serializers import EmployeeSerializer

class EmployeeList(generics.ListCreateAPIView):
  queryset = Employee.objects.all()
  serializer_class = EmployeeSerializer

Templates: The template layer is Django's presentational "view". This is where data from your views will be rendered to the user and is analogous to Laravel's views.

The winner

There are other design patterns and features that I didn't dive into here, such as queues and signals/listeners. The usage of both is very different between Laravel and Django, where Django feels more primitive, but that flexibility is what makes it a great choice for many applications. Some may feel that Laravel is too opinionated or enforces too many rules, but that's what's great about it – it's hard to break out of the design pattern, which makes code more consistent and makes it easier to onboard new developers onto a project.

Personally, I reach for Laravel's ideas in every project, regardless of what language I'm using. The structure/flow is easy to follow and the documentation is fantastic. When I was using Laravel professionally, I enjoyed it a lot. Being able to use queues, caching, events, and notifications right out of the box made it easy to develop complex features without thinking about what other tools I'd have to pull in. There's a little bit more work involved with Django, but you also get to define your own starting point and make decisions how you see fit. In terms of performance, Django is probably a better choice, but everything comes down to what you need in an application.

tl;dr

Laravel adheres more closely to traditional MVC – models define business logic and attributes, controllers are used for endpoints and data manipulation, and views provide the presentation layer.

Django increases complexity a bit by using different terminology – a traditional "model" is essentially a model + a serializer, views/viewsets are traditional controllers, and templates are traditional "views" that provide presentation.