Forem: Ishan

The Cult of Not Done Manifesto

Ishan — Tue, 31 Oct 2023 00:00:00 +0000

To all those who ask why I haven’t been putting more posts out, this is for you

Dear Members of the Cult of Not Done,

I present to you a manifesto of not done. This was inspired by Kio Stark and Bre Pettis, and written over five days because it was, obviously, not done.

The Cult of Not Done Manifesto

There are four states of being. Not knowing, action, refinement, and completion.
Accept that everything is a draft. We have to make it better before we can be done.
We live in the editing stage. Nothing is ever done because it’s always being improved.
Pretending you know what you’re doing is almost the same as knowing what you are doing, but not good enough. So spend all your time making things perfect.
Embrace procrastination. If you wait more than a week to get an idea done, give it more time. Your subconscious needs to think more, and it will give you the answer soon.
The point of being done is to finish, and until we don’t, we can’t get other things done.
If you’re never done, you can throw it away.
Laugh at those who settle. It’s boring and keeps you from being perfection.
People without dirty hands are wrong. Doing something makes you right. The more time you spend trying different things and not being done, the more authority you have on it.
Failure and mistakes are natural and to be expected since you’re not done yet.
Destruction is a force of progress. Start over often with increased scope to make your creation perfect.
If you have an idea, never publish it on the internet, then it will be stolen from you.
In Progress is the engine of more.

Dear past me, use the flags

Ishan — Wed, 26 Jul 2023 00:00:00 +0000

Dear past me,

I know you’ve just launched your first desktop application. It’s a nightmare isn’t it? Well you’re in for a bigger one.

Your boss calls you, and they need a new feature immediately. You bang it out and send out the update.

Oh yeah, that’s right. You didn’t include an update mechanism. Please use one, you’ll save yourself a lot of pain.

You send out the update to users, but soon you find out everyone hasn’t updated yet. The boss insists that people on an older app version shouldn’t be able to use it.

But you didn’t include a forced version based lockout mechanism either. Well, you’ll remember that for next time!

Desperate times call for desperate measures, and you decide to break the server so it returns bad data to old app versions. Never mind that this permanently corrupts the local database and will require future tech support to help them clear their cache. But that’s future you’s problem.

The boss is happy, the app is chugging along. This cycle repeats a couple more times in various ways.

Save yourself the pain next time. Implement a feature flag / version system. Later, it’ll even help you roll out features to users incrementally.

Use the flags, Luke.

Sincerely, Present me

Programming with the grain

Ishan — Sun, 09 Jul 2023 00:00:00 +0000

When I was a kid, I observed that most sheets of paper have a sort of "direction" inside them. It's not possible to tell by looking, but there's a simple experiment to show:

Rip the same sheet of paper left and right, then up and down. One of the two will have a smooth, straight, nice-looking rip, while the other will have a rough, uneven edge.

Later, as a teenager, I learned that this is called the "grain" of the paper, and that this also applies to shaving and woodworking.

I have observed that computers and programs exhibit similiar characteristics which aren't obvious at first. Any given computer, platform, problem and program has a "grain" too. A hidden, but present underlying "optimal" (may I even call it "natural") structure of how data flows and the factorization of the various components that would constitute it. Others seem to have observed it too.

A tryst with locality

Memory can be a funny thing, both when it comes to the self, and to computers. Take the following two pieces of C++ code for example:

// First Example
for (int i = 0; i < size; i++)
{
    for (int j = 0; j < size; j++)
    {
        int element = array[i][j];
        // Do something with the thing item
    }
}

// Second Example
for (int j = 0; j < size; j++)
{
    for (int i = 0; i < size; i++)
    {
        int element = array[i][j];
        // Do something with the thing item
    }
}

Running 100 iterations on a 10x10 array filled with 0s, the first example takes 221ns and the second takes 223ns. That's not even 1%, and maybe it can be chalked up to statistial error. But what happens when we go bigger?

Array Size	Average 1	Average 2	Best 1	Best 2	Worst 1	Worst 2
10x10	221.52ns	223.42ns	205ns	211ns	480ns	402ns
100x100	24646ns	13296ns	8163ns	8266ns	143007ns	32424ns
1000x1000	806953ns	1.33ms	645242ns	951179ns	1.5ms	3.44ms
10000x10000	81.3ms	675.34ms	70.85ms	632.21ms	101.65ms	776.79ms

The first example performs dramatically better! But why? What is it doing different? Let's take a closer look:

Imagine a 3 x 3 array. It might look 2D to us:

1,2,3
4,5,6
7,8,9

But, 2D arrays are an abstraction. Memory in computers is single dimensional, so the computer stores this as:

1,2,3,4,5,6,7,8,9

This drastic performance difference is because of the concept of Locality of Reference and how C++ stores your data in memory. Since C++ uses row-major ordering, accessing elements in order of rows (as in the first example) goes with the grain.

Since the elements are stored are more likely to be located nearby in memory, the processor can also "cheat" and load a contiguous chunk of memory into the cache, which allows for much faster access. However, when iterating by column (or against the grain), the processor can't cache the columns from memory, because the columns don't exist next to each other.

There's more to the story than performance though.

The terrible tale of pthreads

When someone says "multithreading", they almost always mean threading with reference to the POSIX threads model. People often complain about how difficult and error prone the usage of threads are.

However, what most people won't tell you, is that POSIX threads are against the grain of computers. While threads can run concurrently, the underlying architecture does not directly resemble the interrupt-driven behavior of CPUs. Thread scheduling and synchronization are typically handled by the operating system, abstracting away the low-level hardware details.

The Communicating Sequential Processes (CSP) concurrency model, pioneered by Tony Hoare, focuses on communication and synchronization between concurrent processes rather than explicit thread management. In CSP, processes communicate by sending and receiving messages through channels, and synchronization is achieved through the coordination of message passing. This model aligns more closely with the interrupt-based architecture of CPUs. Go is based on this model.

When in Rome

When in Rome, one should always do as Romans do. When programming, this often extends to following the idioms of the platform and language you are working on, as well as the culture and coding guidelines of the team. That's why, when in Python land, you follow the Zen of Python, and in Go land, you don't use a full stack framework. This is because going against the grain of the platform makes it more difficult, and you'll definitely know when you're doing it.

As yacoset puts it so elegantly,

You cannot think "Fire rearward missile" and then translate it to Russian, you must think in Russian.

Consider the following program in Python that someone from a C background might write:

sum = 0
for i in range(1, 11):
    sum += i

print("Sum: %d" % sum)

This goes against the grain, and is non idiomatic Python.

sum = sum(range(1, 11))

print(f"Sum: {sum}")

The idiomatic Python version utilizes the built-in sum() function with a range to directly calculate the sum of numbers from 1 to 10. It then uses an f-string to format the output and achieves the same functionality in a more concise and Pythonic way.

How about this code in Go?

package main

import (
    "fmt"
    "io"
    "net/http"
    "sync"
)

func main() {
    urls := []string{"https://example.com", "https://google.com", "https://github.com"}

    var wg sync.WaitGroup
    var mu sync.Mutex

    for _, url := range urls {
        wg.Add(1)
        go func(u string) {
            defer wg.Done()

            resp, err := http.Get(u)
            if err != nil {
                mu.Lock()
                fmt.Printf("Error fetching %s: %s\n", u, err.Error())
                mu.Unlock()
                return
            }

            defer resp.Body.Close()

            body, err := io.ReadAll(resp.Body)
            if err != nil {
                mu.Lock()
                fmt.Printf("Error reading response body from %s: %s\n", u, err.Error())
                mu.Unlock()
                return
            }

            mu.Lock()
            fmt.Printf("URL: %s, Length: %d\n", u, len(body))
            mu.Unlock()
        }(url)
    }

    wg.Wait()
}

In this example, we manually manage the "threads" using the sync.WaitGroup. Each URL is processed by a separate thread, and synchronization is achieved using the sync.Mutex to protect shared access to the console output.

While this approach still accomplishes concurrent processing, it deviates from the idiomatic use of goroutines and channels in Go and involves lower-level thread management.

package main

import (
    "fmt"
    "io"
    "net/http"
)

func main() {
    urls := []string{"https://example.com", "https://google.com", "https://github.com"}

    // Create a channel to receive the results
    results := make(chan string)

    // Launch a goroutine for each URL
    for _, url := range urls {
        go fetchURL(url, results)
    }

    // Collect the results
    for i := 0; i < len(urls); i++ {
        result := <-results
        fmt.Println("Fetched:", result)
    }
}

func fetchURL(url string, results chan<- string) {
    resp, err := http.Get(url)
    if err != nil {
        results <- fmt.Sprintf("Error fetching %s: %s", url, err.Error())
        return
    }

    defer resp.Body.Close()

    body, err := io.ReadAll(resp.Body)
    if err != nil {
        results <- fmt.Sprintf("Error reading response body from %s: %s", url, err.Error())
        return
    }

    results <- fmt.Sprintf("URL: %s, Length: %d", url, len(body))
}

Now, we launch a goroutine for each URL, and each goroutine fetches the content of the URL. The results are then sent back through the results channel. Finally, we collect the results and print them.

This approach, (while still not optimal) aligns with the idiomatic way of utilizing goroutines and channels for concurrent tasks in Go and is much more readable and thus maintainable in the future.

Closing thoughts

Further quoting yacoset,

There's a thousand computer languages because there's a thousand classes of problems we can solve with software. In the 1980s, after the Macintosh debut, a hundred DOS products were ported to the new mouse-driven platform by clubbing the Alto-inspired UI over the head and brute-forcing the keyboard-driven paradigms of PCs into the Mac's visual atmosphere. Most of these were rejected by Apple or the market, and if they came back for a second try they came back because somebody flipped open the spiral-bound HIG and read it sincerely.

As programmers, I think we should take pride in our craft, and always try to program with the grain of the platform we are on. Many benefits await!

The secret life of .well-known

Ishan — Sun, 02 Jul 2023 00:00:00 +0000

My first encounter with the .well-known directory was when Let’s Encrypt first hit the scene and I was learning about VPS-es and how Linux on the server worked.

I noticed that it used certain URLs like .well-known/acme-challenge/3d2f4b8c9d0a4f8b9e6f7d8c9e0a4f8b. This led me down a nice rabbit hole about the ACME protocol, but that’s a story for another time.

Years later, when configuring servers to use paid HTTPs certificates, I noticed the use of the .well-known/pki-validation/ directory for verifying certificates. I filed this knowledge away in my brain as “the .well-known directory is for SSL related stuff” and continued on with my life.

Somewhere in the middle, I noticed that a lot of brute force attacks on servers I monitored would be directed to .well-known URLs. At that point, I had learned about DNS based verification, and quickly moved on from HTTP challenges, and blocked access to the directory in my web server configs. I made this a part of my base setup on every server, and didn’t really think too much about the directory any more.

That was, until very recently.

The Fiasco with NodeInfo

As a recent migrant to the Fediverse after the Reddit API changes shut down my 3rd party app of choice, I joined a fairly interesting Lemmy instance, and I quickly found the distributed nature fascinating. All I could think about was: “I want to see a giant map of all the instances and who is connected to who”, and immediately set out to create a web scraper that could gather this information for me.

Don’t worry, I made sure to respect robots.txt for every site the scraper visited.

Lemmy’s API documentation is fairly poor, and I flailed for quite a while before I discovered the endpoint (/api/v3/site) that would give me the list of federated servers.

That’s when I realized, that Lemmy could and did federate with other Fediverse software like Mastodon, Calckey, Pleroma, WriteFreely, PixelFed and more. And each one of these have different APIs and different endpoints that would give me the data I need.

I needed a way to detect which software each server was running. My first instinct was to use some sort of heuristic, but that’s unreliable, so I continued researching and discovered the NodeInfo protocol.

And wouldn’t you have it, the resource I need to request just happens to be_

/.well-known/nodeinfo

At this point, my curiousity was piqued. There was more to this .well-known directory than I had initially thought!

The RFC enters the fray

It turns out, that the .well-known directory is defined in RFC 8615. The intention seems to be to make it like a generic and extendable version of robots.txt

It is increasingly common for Web-based protocols to require the discovery of policy or other information about a host (“site-wide metadata”) before making a request. For example, the Robots Exclusion Protocol http://www.robotstxt.org/ specifies a way for automated processes to obtain permission to access resources; likewise, the Platform for Privacy Preferences [W3C.REC-P3P-20020416] tells user-agents how to discover privacy policy beforehand.

While there are several ways to access per-resource metadata (e.g., HTTP headers, WebDAV’s PROPFIND [RFC4918]), the perceived overhead (either in terms of client-perceived latency and/or deployment difficulties) associated with them often precludes their use in these scenarios.

When this happens, it is common to designate a “well-known location” for such data, so that it can be easily located. However, this approach has the drawback of risking collisions, both with other such designated “well-known locations” and with pre-existing resources.

To address this, this memo defines a path prefix in HTTP(S) URIs for these “well-known locations”, /.well-known/. Future specifications that need to define a resource for such site-wide metadata can register their use to avoid collisions and minimise impingement upon sites' URI space.

There’s a whole bunch of stuff that you can do with stuff in the .well-known folder, like:

and of course, let’s not forget about Webfinger.

“A webfinger? Is that what I use to poke someone on Facebook?” - Anonymous

UPDATE 2023-07-04 : I have published a follow-up - Addendum: Webfinger

UPDATE 2023-07-07 : This article was featured in tldr.tech, and as of time of writing I have had over 6500 hits on this post. I am overwhelmed and moved by this, and by all the positive comments I have received. I never imagined that my little blog would ever be read by so many people.

Using different Github accounts with different private keys on Linux

Ishan — Wed, 21 Jun 2023 00:00:00 +0000

Create a file ~/.ssh/config

If you have 2 keys, for example: id_rsa for your personal, and id_work for your work, set the config as:

Host github-work
    HostName github.com
    IdentityFile ~/.ssh/id_work
    IdentitiesOnly yes

Now, when cloning or adding remote, change the github.com in the clone url is changed to github-work

git clone git@github-work:username/whatever.git

Collapsible Comments on Dev.to

Ishan — Sun, 26 Aug 2018 04:37:57 +0000

Note from the Future
It's 2021, and dev.to has this feature now!

Hey everyone! First time posting here!

I love dev.to, but as an avid redditor, the lack of collapsible comments always kind of bothered me-- I mean, it's a feature that should totally exist. There's even a GitHub Issue for it.

So, I decided to finish a side-project for once. The result, a chrome extension that adds this very feature to dev.to.

Installation is a bit of a challenge as I'm pretty new to this making extensions thing, hopefully someone will be able to help me out with that.

Repo link: https://github.com/sad-pixel/dev-to-comment-collapser

Installation Instructions

Download the zip file or clone the repo
Go to chrome://extensions
Enable the developer mode toggle
Click "Load Unpacked"
Browse to the folder where the source files are located
Done

In case the buttons don't show up, refreshing the page should fix the issue. This is caused due to the way dev.to loads new pages.

Hopefully this will be of some help to someone!