Forem: Srikanth

Building a Password Manager in Go: Part 3

Srikanth — Mon, 07 Oct 2024 08:57:49 +0000

Welcome back to our journey of building a password manager in Go! In this third installment, we've made significant strides in functionality and usability. Let's dive deep into the exciting new features and improvements.

1. Implementing Password Storage

One of the most crucial additions to our password manager is the ability to store and retrieve passwords. We've implemented a Storage struct that handles these operations:

type Storage struct {
    path string
}

func GetStorage(path string) *Storage {
    s := Storage{path: path}
    return &s
}

func (s *Storage) Init() {
    if err := os.Mkdir(s.path, 0755); err != nil {
        panic(err)
    }
}

func (s *Storage) Add(password, identifier string) error {
    directory := filepath.Dir(identifier)
    dirPath := filepath.Join(s.path, directory)
    if _, err1 := os.Stat(dirPath); os.IsNotExist(err1) {
        log.Printf("%s does not exist, creating.", dirPath)
        if err3 := os.Mkdir(dirPath, 0755); err3 != nil {
            return err3
        } 
    }
    absPath := filepath.Join(s.path, identifier)
    if err2 := os.WriteFile(absPath, []byte(password), 0600); err2 != nil {
        log.Println("Error writing file:", err2)
        return err2
    }
    return nil
}

func (s *Storage) Show(identifier string) (string, error) {
    absPath := filepath.Join(s.path, identifier)
    password, err := os.ReadFile(absPath)
    if err != nil {
        return "", err
    }
    return string(password), nil
}

func (s *Storage) IsReady() bool {
    _, err := os.Stat(s.path)
    return !os.IsNotExist(err)
}

Let's break down these methods:

GetStorage: Creates a new Storage instance with a specified path.
Init: Initializes the storage directory.
Add: Stores a password, creating directories as needed.
Show: Retrieves a stored password.
IsReady: Checks if the storage has been initialized.

This implementation allows for a hierarchical storage structure, enabling users to organize passwords in categories (e.g., email/work@example.com).

2. Enhanced Command-Line Interface

We've significantly improved our CLI, adding new commands and better user guidance:

func main() {
    path := os.Getenv("HOME") + "/.dost"
    storage := internal.GetStorage(path)
    generateCmd := flag.NewFlagSet("generate", flag.ExitOnError)
    flag.Parse()

    if len(os.Args) < 2 {
        printHelp()
        os.Exit(1)
    }

    switch os.Args[1] {
    case "generate":
        if storage.IsReady() {
            password, passwordName := internal.Generate(generateCmd)
            if err := storage.Add(password, passwordName); err != nil {
                fmt.Printf("%v", err)
            }
        } else {
            fmt.Println("dost is not initialized. Run `dost init`")
            os.Exit(1)
        }
    case "init":
        storage.Init()
    case "show":
        password, err := storage.Show(os.Args[2])
        if err != nil {
            fmt.Printf("Something went wrong: %v", err)
        } else {
            fmt.Println(password)
        }
    default:
        printHelp()
    }
}

This new structure allows for three main commands:

init: Initializes the password store.
generate: Generates a new password and optionally stores it.
show: Retrieves a stored password.

3. User-Friendly Help Function

To make our tool more user-friendly, we've added a printHelp() function:

func printHelp() {
    fmt.Println("Invalid command to run dost password manager.")
    fmt.Println("Please choose one of the following options:")
    fmt.Println("dost init")
    fmt.Println("dost generate [-c] [-n] <password-name>")
    fmt.Println("dost show <password-name>")
}

This function is called when users provide invalid input, guiding them on how to use the password manager correctly.

4. Improved Password Generation

We've refined our password generation algorithm to ensure a good mix of character types:

func generatePassword(length int, noSymbols bool) (string, error) {
    allChars := uppercaseLetters + lowercaseLetters + digits
    var password []string

    password = append(password, selectRandomCharacter(uppercaseLetters))
    password = append(password, selectRandomCharacter(lowercaseLetters))
    password = append(password, selectRandomCharacter(digits))

    if !noSymbols {
        allChars += specialChars
        password = append(password, selectRandomCharacter(specialChars))
    }

    length -= len(password)

    for i := 0; i < length; i++ {
        randomIndex, err := rand.Int(rand.Reader, big.NewInt(int64(len(allChars))))
        if err != nil {
            return "", err
        }
        password = append(password, string(allChars[randomIndex.Int64()]))
    }

    password = shuffle(password)
    return strings.Join(password, ""), nil
}

This approach guarantees that each password includes at least one uppercase letter, one lowercase letter, one digit, and (unless specified otherwise) one special character. The shuffle function ensures that these mandatory characters are not always in the same position.

5. Comprehensive Testing

We've expanded our test suite to cover new functionalities:

func TestStorageInitReady(t *testing.T) {
    path := getRandomPath()
    defer cleanUpPath(path)

    storage := GetStorage(path)
    storage.Init()

    if !storage.IsReady() {
        t.Errorf("Expected directory: %s to be created on storage.Init()", path)
    }
}

func TestStorageAddShow(t *testing.T) {
    path := getRandomPath()
    defer cleanUpPath(path)

    storage := GetStorage(path)
    storage.Init()

    identifier := "email/sri@example.com"
    password := "someRandomPassword"

    addErr := storage.Add(password, identifier)
    if addErr != nil {
        t.Errorf("Did not expect an error when calling storage.Add: \n%v", addErr)
    }

    passwordFromFile, showErr := storage.Show(identifier)
    if showErr != nil {
        t.Errorf("Did not expect an error when calling storage.Show: \n%v", showErr)
    }

    if passwordFromFile != password {
        t.Errorf("Password that was added did not match the one from the one that got saved\npassword: %s, passwordFromFile: %s", 
                password, passwordFromFile)
    }
}

These tests ensure that our storage mechanisms work correctly, adding an extra layer of reliability to our password manager. We're using random paths for testing to avoid conflicts and cleaning up after each test.

6. Environment-Aware Storage Location

We've made our password manager more user-friendly by storing passwords in the user's home directory:

path := os.Getenv("HOME") + "/.dost"
storage := internal.GetStorage(path)

This approach ensures that the password store is easily accessible and follows common conventions for user-specific data storage.

What's Next?

While we've made significant progress, there's still room for improvement:

Implement encryption for stored passwords to enhance security.
Add a feature to update existing passwords.
Implement a search functionality for stored passwords.
Add support for importing and exporting passwords for backup purposes.

Conclusion

In this iteration, we've transformed our password manager from a simple password generator to a functional tool for storing and retrieving passwords. We've improved the user experience with better CLI interactions and added robust testing to ensure reliability.

The journey of building this password manager has been an excellent opportunity to explore various aspects of Go programming, from file I/O to testing and CLI development. As we continue to develop this tool, we're not just creating a password manager; we're also honing our Go skills and exploring best practices in software development.

Stay tuned for the next part of our series, where we'll tackle these challenges and continue to evolve our password manager!

Remember, the full source code is available on GitHub. Feel free to clone, fork, and contribute to the project. Your feedback and contributions are always welcome!

Happy coding, and stay secure! 🔐💻

svemaraju / dost

dost command line password manager written in Go

dost

dost is a CLI password manager written in Go.

Inspired by (Pass)[https://www.passwordstore.org/]

Features

Generate random passwords of configurable length
Copy generated passwords to clipboard automatically
Skip using symbols

Usage

> go build -o dost main.go

Generating password:

> ./dost generate email/vema@example.com
Generated Password: );XE,7-Dv?)Aa+&<{V-|pKuq5

Generating password with specified length (default is 25):

> ./dost generate email/vema@example.com 12
Generated Password: si<yJ=5/lEb3

Copy generated password to clipboard without printing:

> ./dost generate -c email/vema@example.com 
Copied to clipboard! ✅

Avoid symbols for generating passwords:

> ./dost generate -n email/vema@example.com 
Generated Password: E2UST}^{Ac[Fb&D|cD%;Eij>H

Under development

Insert a new password manually
Show an existing password
List all entries
Password storage
GPG Key based encryption

License

MIT

View on GitHub

Building a Password Manager in Go: Part 2

Srikanth — Tue, 24 Sep 2024 21:29:39 +0000

Welcome back to our journey of building a password manager in Go! In this second installment, we'll explore the progress we've made since our initial commit. We've added new features, improved the code structure, and implemented testing. Let's dive in!

Evolving the Project Structure

One of the first changes you'll notice is the improved project structure. We've separated our code into multiple files and packages, following Go's best practices:

dost/
.
├── LICENSE
├── README.md
├── go.mod
├── go.sum
├── internal
│   ├── internal_test.go
│   └── passgen.go
└── main.go

This structure allows for better organization and maintainability as our project grows.

Enhanced Command-Line Interface

We've significantly improved our CLI, making it more flexible and user-friendly. Here's a snippet from our main.go:

func main() {
    generateCmd := flag.NewFlagSet("generate", flag.ExitOnError)
    flag.Parse()

    switch os.Args[1] {
    case "generate":
        internal.Generate(generateCmd)
    }
}

This setup allows for subcommands, currently supporting the generate command. Users can now interact with our tool like this:

go run main.go generate email/reachme@example.com 15

Customizable Password Generation

We've added options to customize password generation. Users can now specify password length and choose to exclude special characters:

func Generate(generateFlags *flag.FlagSet) {
    generateFlags.BoolVar(&noSymbols, "n", false, "Skip symbols while generating password")
    generateFlags.BoolVar(&copyToClipBoard, "c", false, "Copy to clipboard.")
    generateFlags.Parse(os.Args[2:])
    passwordLength := 25
    // ... (code to parse custom length)

    password, err := generatePassword(passwordLength, noSymbols)
    // ... (code to handle password output)
}

This function allows users to use flags like -n to exclude symbols and -c to copy the password to clipboard instead of displaying it.

Improved Password Generation Algorithm

We've refined our password generation function to handle the new customization options:

func generatePassword(length int, noSymbols bool) (string, error) {
    const (
        uppercaseLetters = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
        lowercaseLetters = "abcdefghijklmnopqrstuvwxyz"
        digits           = "0123456789"
        specialChars     = "!@#$%^&*()-_=+[]{}|;:'\",.<>/?"
    )

    allChars := uppercaseLetters + lowercaseLetters + digits
    if !noSymbols {
        allChars += specialChars
    }

    var password string
    for i := 0; i < length; i++ {
    // Generate a random index to select a character from allChars
    randomIndex, err := rand.Int(rand.Reader, big.NewInt(int64(len(allChars))))
    if err != nil {
        return "", err
    }

    // Append the randomly selected character to the password
    password += string(allChars[randomIndex.Int64()])
    }

    return password, nil
}

This function now respects the noSymbols flag, allowing for more flexible password generation.

Implementing Tests

We've taken a significant step towards ensuring the reliability of our code by implementing tests. Here's a snippet from our test file:

func TestPasswordLength(t *testing.T) {
    password, err := generatePassword(10, true)
    if err != nil {
        t.Errorf("Expected no error, got %v", err)
    } else {
        if len(password) != 10 {
            t.Errorf("Expected 10 character password, got %d", len(password))
        }
    }
}

func TestSpecialCharacter10K(t *testing.T) {
    splCharMissing := 0
    for i := 1; i <= 10000; i++ {
        password, err := generatePassword(10, false)
        // ... (code to check for special characters)
    }
    if splCharMissing > 0 {
        t.Errorf("Special character was missing in %d / 10000 instances.", splCharMissing)
    }
}

These tests check for correct password length and the inclusion of special characters. Interestingly, our special character test revealed an area for improvement: in 10,000 generated passwords, 234 didn't contain special characters. This gives us a clear direction for our next refinement.

What's Next?

While we've made significant progress, there's still room for improvement:

Refine the password generation algorithm to ensure consistent inclusion of special characters.
Implement password storage functionality.
Add encryption for stored passwords.
Develop search and retrieval features.

Stay tuned for the next part of our series, where we'll tackle these challenges and continue to evolve our password manager!

Remember, the full source code is available on GitHub. Feel free to clone, fork, and contribute to the project. Your feedback and contributions are always welcome!

Happy coding, and stay secure! 🔐💻

svemaraju / dost

dost command line password manager written in Go

dost

dost is a CLI password manager written in Go.

Inspired by (Pass)[https://www.passwordstore.org/]

Features

Generate random passwords of configurable length
Copy generated passwords to clipboard automatically
Skip using symbols

Usage

> go build -o dost main.go

Generating password:

> ./dost generate email/vema@example.com
Generated Password: );XE,7-Dv?)Aa+&<{V-|pKuq5

Generating password with specified length (default is 25):

> ./dost generate email/vema@example.com 12
Generated Password: si<yJ=5/lEb3

Copy generated password to clipboard without printing:

> ./dost generate -c email/vema@example.com 
Copied to clipboard! ✅

Avoid symbols for generating passwords:

> ./dost generate -n email/vema@example.com 
Generated Password: E2UST}^{Ac[Fb&D|cD%;Eij>H

Under development

Insert a new password manually
Show an existing password
List all entries
Password storage
GPG Key based encryption

License

MIT

View on GitHub

Building a Password Manager in Go

Srikanth — Wed, 18 Sep 2024 10:46:16 +0000

As a software developer, I've always been fascinated by the intersection of security and usability. Recently, I decided to embark on an exciting project: creating a command-line password manager using Go. I want to share the beginning of this journey with you, starting with the very first commit.

The Genesis

On November 27, 2023, I made the initial commit for my project, which I've named "dost" (friend in Hindi, reflecting its role as a helpful companion for password management). This first step, while small, lays the foundation for what I hope will become a robust and user-friendly tool.

Inspiration and Vision

While embarking on this project, I drew inspiration from the popular command-line password manager pass. The simplicity and effectiveness of pass caught my attention, and I decided to use its API as a blueprint for building my own password manager in Go.

Diving into the source code of pass was an eye-opening experience. I was intrigued to discover that the entire functionality of this widely-used tool is encapsulated in one comprehensive Bash script. This elegant simplicity is something I admire and hope to emulate in my own project, albeit using Go's strengths.

By studying pass, I've gained valuable insights into the essential features of a command-line password manager and the user experience it should provide. As I continue to develop "dost", I'll be keeping these lessons in mind, aiming to create a tool that combines the simplicity of pass with the performance and cross-platform compatibility benefits of Go.

This exploration has not only provided a roadmap for features to implement but also reinforced my belief in the power of well-crafted, focused tools. I'm excited to see how this inspiration will shape the evolution of "dost" in the coming stages of development.

First Features

The initial commit focused on two core functionalities:

Password Generation: I implemented a basic password generator that allows users to specify their desired password length. This feature aims to create strong, randomized passwords tailored to various security requirements.
Clipboard Integration: To enhance user experience, I ensured that the generated password is automatically copied to the clipboard. This small but crucial feature saves time and reduces the risk of transcription errors.

Technical Insights

Let's dive into some of the technical aspects of this first iteration:

Go Version: The project is built using Go 1.21.0, leveraging the language's simplicity and efficiency.
External Dependencies: I'm using the github.com/atotto/clipboard package to handle clipboard operations across different operating systems seamlessly.
Random Generation: The password generation utilizes Go's crypto/rand package for secure random number generation, crucial for creating unpredictable and strong passwords.
Character Set: The password generator includes uppercase and lowercase letters, digits, and a variety of special characters to ensure complexity.

Code Snippets

Let's look at some key parts of the implementation:

Password Generation Function:



func generatePassword(length int) (string, error) {
    const (
        uppercaseLetters = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
        lowercaseLetters = "abcdefghijklmnopqrstuvwxyz"
        digits           = "0123456789"
        specialChars     = "!@#$%^&*()-_=+[]{}|;:'\",.<>/?"
    )

    allChars := uppercaseLetters + lowercaseLetters + digits + specialChars

    var password string
    for i := 0; i < length; i++ {
        randomIndex, err := rand.Int(rand.Reader, big.NewInt(int64(len(allChars))))
        if err != nil {
            return "", err
        }
        password += string(allChars[randomIndex.Int64()])
    }

    return password, nil
}

This function creates a password by randomly selecting characters from a predefined set, ensuring a mix of uppercase, lowercase, digits, and special characters.

Clipboard Integration:



func writeToClipboard(text string) error {
    return clipboard.WriteAll(text)
}

This simple function utilizes the clipboard package to write the generated password to the system clipboard.

Main Function:



func main() {
    passwordLength := flag.Int("length", 12, "length of your password")
    flag.Parse()

    password, err1 := generatePassword(*passwordLength)
    if err1 != nil {
        fmt.Println("Error generating password:", err1)
        return
    }

    fmt.Println("Generated Password:", password)

    err2 := writeToClipboard(password)
    if err2 != nil {
        fmt.Println("Error writing to clipboard:", err2)
        os.Exit(1)
    }

    fmt.Println("Copied to clipboard! ✅\n")
}

The main function ties everything together. It uses Go's flag package to allow users to specify the password length, generates the password, and copies it to the clipboard.

Command-Line Interface

As you can see in the main function, I've implemented a simple CLI using Go's flag package. Users can specify the desired password length using the -length flag, with a default of 12 characters if not specified.

Looking Ahead

This first commit is just the beginning. As I continue to develop this password manager, I plan to add features such as:

Secure storage of passwords
Encryption of stored data
Search and retrieval functionalities
Password strength analysis

I'm excited about the journey ahead and the challenges it will bring. Building a password manager is not just about coding; it's about understanding security principles, user needs, and creating a tool that people can trust with their sensitive information.

Stay tuned for more updates as this project evolves. I'll be sharing my progress, challenges, and learnings along the way. If you're interested in following along or contributing, feel free to check out the project on GitHub.

svemaraju / dost

dost command line password manager written in Go

dost

dost is a CLI password manager written in Go.

Inspired by (Pass)[https://www.passwordstore.org/]

Features

Generate random passwords of configurable length
Copy generated passwords to clipboard automatically
Skip using symbols

Usage

> go build -o dost main.go

Generating password:

> ./dost generate email/vema@example.com
Generated Password: );XE,7-Dv?)Aa+&<{V-|pKuq5

Generating password with specified length (default is 25):

> ./dost generate email/vema@example.com 12
Generated Password: si<yJ=5/lEb3

Copy generated password to clipboard without printing:

> ./dost generate -c email/vema@example.com 
Copied to clipboard! ✅

Avoid symbols for generating passwords:

> ./dost generate -n email/vema@example.com 
Generated Password: E2UST}^{Ac[Fb&D|cD%;Eij>H

Under development

Insert a new password manually
Show an existing password
List all entries
Password storage
GPG Key based encryption

License

MIT

View on GitHub

Discovering how python is saving my memory

Srikanth — Sat, 13 Jan 2024 01:21:00 +0000

When it comes to code conundrums, a quick visit to google or chatgpt usually satisfies my curiosity. However, I felt reluctant to read a bunch of blog posts and opted to write some code instead. This led me to discovering string interning in Python.

I have written a fair amount of code and by no means I am an expert. But over the years there has been a natural decline in discovering new things. That's the reason why I am so excited to share this article.

Unpacking the internal workings of CPython by observing a series of code snippets complimented by poring over specifics parts of python docs and aided some excellent articles. The very articles which I expertly avoided earlier, not because I chose not click but because I did not possess the knowledge to type the right search terms at the beginning.

A high level view of the following article:

Context and observing the behaviour: I explain my motivation to understand parts of Python behaviour and the onset of my quest.
Learning about the internals: I observed enough and was ready for some answers. I dig into python docs, continue observing and finally find out some truths.
Where does this come to play: I make my conclusions while carrying on looking at more interesting angles of string interning.

Context and observing the behaviour.

I was watching a video on programming basics as I was interested in how people explain these concepts to beginners. At one point there was a discussion on "pass by value" and "pass by reference". I have read about them in my undergraduate course. But I rarely think of them when I code in Python.

Instead of watching a video on the topic, I began to write some code to observe how arguments are passed to a function in Python.

I start with something familiar. Passing a list to a function and mutating the list will mutate the variable outside the function.

pets = ["dogs", "kittens"]
def add_pets(pets):
    pets.append("hamsters")


add_pets(pets)
print(pets)

# output
['dogs', 'kittens', 'hamsters']

To further confirm my understanding, I used built-in id function to verify that pets inside the function and pets outside the function point to the same object.

pets = ["dogs", "kittens"]
def add_pets(pets):
    pets.append("hamsters")
    print(f"ID of pets inside add_pets {id(pets)}")

add_pets(pets)
print(f"ID of pets outside add_pets {id(pets)}")

# output
ID of pets inside add_pets 4302154624
ID of pets outside add_pets 4302154624

Lists are mutable in Python, but strings are not. So I try the following code.

pet = "Panda"
def zoo(pet):
    pet += " Cow"
    print(f"ID of '{pet}': {id(pet)}")

zoo(pet)
print(pet)
print(f"ID of '{pet}': {id(pet)}")

# output
ID of 'Panda Cow': 4304152176
Panda
ID of 'Panda': 4304151280

Both the value and the memory location of the variable pet inside the function are different from those outside.

So does that mean, lists are passed by reference but strings are passed by value? I just wanted to try running one more piece of code before Googling that.

pet = "Panda"
def zoo(pet):
    print(f"ID of '{pet}' inside foo: {id(pet)}")

print(f"ID of '{pet}' outside foo: {id(value)}")
zoo(pet)

The result of the above code was a bit unexpected for me.

# ----output----
ID outside foo: 4343585264
ID inside foo: 4343585264

It seems that inside the function the variable that are passed refer to the same object location of the variables outside the function. When the variable gets reassigned, as in the case of immutable string, the variable starts pointing to a different memory location.

Learning about the internals.

I quickly found the FAQ page on docs.python.org. Anwering a question about call-by-refernce in Python, it reads as following --

"Remember that arguments are passed by assignment in Python. Since assignment just creates references to objects, there’s no alias between an argument name in the caller and callee, and so no call-by-reference per se."

"assignment just creates references to objects", that matches with what I have observed before. When I called function zoo by passing pet, the pet inside the function had the same reference as outside. The quote from the docs talks about assignment in general rather than being limited to passing variables to function. That made me write the following code.

color = "azure"
cloud = color

id(color) == id(cloud)
# ----output----
True

rainforest = "amazon"
cloud = rainforest

id(color) == id(rainforest)
# ----output----
True

colors = ["azure", "black"]
clouds = colors

id(colors) == id(clouds)
# ----output----
True

Since cloud was assigned to color or rainforest it referenced to either of their memory locations, as do colors and clouds. What if the variables are assigned directly to the values instead?

color = "azure"
cloud = "azure"

id(color) == id(cloud)
# ----output----
True

colors = ["azure", "black"]
clouds = ["azure", "black"]

id(colors) == id(clouds)
# ----output----
False

The strings color and cloud both reference the same memory location even though they have not been assigned to each other. However, the lists colors and clouds are pointing to different locations in the memory.

I printed the bytecode of the above and observed the parts relating to strings and list assignment.

  1           2 LOAD_CONST               0 ('azure')
              4 STORE_NAME               0 (color)

  2           6 LOAD_CONST               0 ('azure')
              8 STORE_NAME               1 (cloud)
...
  7         102 LOAD_CONST               0 ('azure')
            104 LOAD_CONST               3 ('black')
            106 BUILD_LIST               2
            108 STORE_NAME               4 (colors)

  8         110 LOAD_CONST               0 ('azure')
            112 LOAD_CONST               3 ('black')
            114 BUILD_LIST               2
            116 STORE_NAME               5 (clouds)

Whenever there is a string literal LOAD_CONST operation is being performed. What exactly is this LOAD_CONST?

Python docs have a minimal definition.

Pushes co_consts[consti] onto the stack.

I find out that co_consts is the list (in tuple form) of all constants in bytecode. Python code is not directly interpreted at the time of execution instead it is first compiled. One of the outputs of compiling is bytecode, which is the set of instructions executed by the python virtual machine.

Everything in python is an object and that applies to the code that is compiled. We can access the code object of a function like this:

def foo():
    a = "jello"

foo.__code__
#---- output ----
<code object foo at 0x100e1c920, file "<stdin>", line 1>

And the code object hosts the co_consts tuple.

foo.__code__.co_consts
#----output----
(None, 'jello')

A fun fact I discovered during this process, I can see None part of co_consts of my foo. But I don't use None anywhere. It turns out Python only ever has one instance of None at all times. All the None objects we use and pass around in our code point to a single instance of None.

Where does this come to play

Once I discovered string interning I quickly imagined how all the memory must have been getting saved, and my code is secretly being optimised without my knowledge. While it is definitely an optimization that speeds up a lot of computation, it's apparently not necessarily in action all the time.

What I found to be puzzling at first is the fact that strings with spaces are not interned.

color = "ms azure"
cloud = "ms azure" 
id(color) == id(cloud)
# ----output----
False

I found this excellent deep dive on Youtube, and its shown the CPython code checks for alphanumeric strings with underscores but no spaces when deciding to intern a string.

From CPython's codeobject.c

/* all_name_chars(s): true iff s matches [a-zA-Z0-9_]* */
static int
all_name_chars(PyObject *o)
{
    const unsigned char *s, *e;

    if (!PyUnicode_IS_ASCII(o))
        return 0;

    s = PyUnicode_1BYTE_DATA(o);
    e = s + PyUnicode_GET_LENGTH(o);
    for (; s != e; s++) {
        if (!Py_ISALNUM(*s) && *s != '_')
            return 0;
    }
    return 1;
}

When we think about it, strings are interned at compile time. So it doesn't come handy in many cases of programming which often deal with string input over HTTP, command line or files. To confirm I ran the following code.

hardcode_animal = "chimpanzee"
input_animal = input("enter animal name: ")

print("Are hardcode_animal and input_animal have same id:", id(hardcode_animal) == id(input_animal))
#---output--- 
enter animal name: chimpanzee
Are hardcode_animal and input_animal have same id: False

This brings us to the question, where is it actually used? In managing (hundreds of) thousands of lines of code that every python program needs. Even a simple print("hello world") requires the several functions and classes to be imported into the interpreter. Imagine a more complex program with a long list of third party dependencies. Names of functions, methods, classes and other variables are the perfect candidates for string interning.

By interning strings, Python is saving memory on any repeated string literals in the code. This further comes into play during dictionary key lookups, in case of string keys. Any string literal comparisons exploit performance upgrade as the interpreter can quickly check if two interned strings share the same memory address, avoiding character-by-character comparisons.

So that's my adventure into CPython's string interning. Understanding how strings are interned at compile time, and the nuances of when and where this optimization comes into play, sheds light on the intricacies of Python's memory management. While string interning may not be evident in everyday code, its impact is profound in managing large codebases and optimizing the performance of Python programs.

How Django keeps track of your model classes?

Srikanth ・ Dec 5 '23

#python #django #opensource #programming

System Design Case Study of an Online Booking System.

Srikanth ・ Aug 16 '23

#learning #architecture #systemdesign #beginners

Photo by John Fowler on Unsplash

Trim the fat! Strategies on writing better at work.

Srikanth — Fri, 22 Dec 2023 09:48:01 +0000

I've always relied on writing to explain my technical solutions. I used to think that a text rich with all the details is better. Is it though?

Here are some strategies that brought style to my substance and boosted my writing skills ~

Trim the fat: Cut irrelevant details that distract from the core idea. Every sentence should earn its place and serve the purpose of your message.
Jargon detox: Simplify technical terms. Your teammates are already battling code complexity; save them the mental gymnastics.
Structure makes sense: Organize your writing into clear sections like "Facts," "Assumptions," and "Questions." This lets everyone get on the same page quickly.
Reflect and confirm: Summarize key points and questions. This clarifies your understanding and avoids miscommunication.
Deep dive with Docs: Move discussions to longer-form platforms like Notion. This allows for structured explanations, collaborative edits, and richer context than chat or calls can offer.

These are some techniques I have started paying attention to. What writing tips have you picked up along the way?

Cover Photo by Daria Kraplak on Unsplash

How Django keeps track of your model classes?

Srikanth — Tue, 05 Dec 2023 21:14:51 +0000

I've run python manage.py makemigrations hundreds of times. Recently, curiosity struck: How does Django even know which models exist in my project?

The Problem: Finding Needles in a Haystack

Think about it … your Django project might have dozens of apps, each with multiple model classes scattered across files. When you run makemigrations, Django somehow finds every single model, compares it to the database schema, and generates the right migrations. How?

The answer reveals one of Django's cleverest design decisions.

The Entry Point: django.setup()

Every Django management command (makemigrations, migrate, runserver, etc.) starts by calling django.setup(). This is where Django's model registry gets populated:

# django/django/__init__.py
def setup(set_prefix=True):
    """
    Configure the settings (this happens as a side effect of accessing the
    first setting), configure logging and populate the app registry.
    """
    ...
    apps.populate(settings.INSTALLED_APPS)

The apps.populate() call kicks off the entire discovery process.

The Discovery Process: Import Every models.py

Django's Apps registry iterates through your INSTALLED_APPS and explicitly imports each app's models.py file:

# django/django/apps/registry.py
class Apps:
    """
    A registry that stores the configuration of installed applications.
    It also keeps track of models, e.g. to provide reverse relations.
    """

    def populate(self, installed_apps=None):
        """
        Load application configurations and models.
        Import each application module and then each model module.
        """
        ...
        # Phase 2: import models modules.
        for app_config in self.app_configs.values():
            app_config.import_models()

Each AppConfig then imports its models module:

# django/django/apps/config.py
class AppConfig:
    """Class representing a Django application and its configuration."""

    def import_models(self):
        self.models = self.apps.all_models[self.label]

        if module_has_submodule(self.module, MODELS_MODULE_NAME):
            models_module_name = "%s.%s" % (self.name, MODELS_MODULE_NAME)
            self.models_module = import_module(models_module_name)

So when you write:

from myapp.models import MyModel

Django isn't actually relying on you to import it. Django imports your models.py file automatically, which triggers the import of every model class defined in it.

The Magic: Metaclass Registration

But here's the clever part—importing a class doesn't automatically register it anywhere. This is where Python metaclasses come in.

Every Django model inherits from models.Model, which uses ModelBase as its metaclass. A metaclass is like a "class factory"—it controls how classes themselves are created.

When Python creates your model class (during import), it calls ModelBase.__new__():

# django/django/db/models/base.py
class ModelBase(type):
    """Metaclass for all models."""

    def __new__(cls, name, bases, attrs, **kwargs):
        super_new = super().__new__

        ...
        new_class._prepare()
        # This is where a model class gets registered
        new_class._meta.apps.register_model(new_class._meta.app_label, new_class)
        return new_class

The key insight: Registration happens at class definition time, not at runtime. The moment Python evaluates class MyModel(models.Model):, the metaclass automatically registers it in Django's app registry.

Why Metaclasses?

You might wonder: couldn't Django just scan for classes that inherit from models.Model after importing? Technically yes, but metaclasses offer several advantages:

Automatic and foolproof - You can't forget to register a model
Happens immediately - No separate registration step needed
Enables introspection - Django can modify the class during creation (adding reverse relations, etc.)

Putting It All Together

Now we can trace the complete flow when you run makemigrations:

Django calls django.setup()
setup() calls apps.populate(INSTALLED_APPS)
For each app, Django imports models.py
Importing causes Python to execute class definitions
Each class MyModel(models.Model): triggers ModelBase.__new__()
The metaclass registers the model in the app registry
makemigrations accesses registered models via apps.get_app_config(app_label).get_models()

Here's how makemigrations uses the registry:

# django/django/core/management/commands/makemigrations.py
class Command(BaseCommand):
    @no_translations
    def handle(self, *app_labels, **options):
        ...
        for app_label in consistency_check_labels:
            for model in apps.get_app_config(app_label).get_models():
                # Check if model needs migration
                ...

How Django Detects Model Changes

Now that we understand how Django finds your models, the next question is: how does it know what changed?

When you run makemigrations, Django needs to answer: "What's different between my current model definitions and the database schema?" This requires comparing two things:

Current state: The models in your Python code (what we just discussed)
Previous state: What Django thinks the database schema looks like

The State Framework: Django's Memory

Django maintains a representation of your database schema through migration files. Each migration file contains operations that describe changes, but more importantly, it contains a state - a snapshot of what all your models looked like after that migration was applied.

Here's a simplified example:

# myapp/migrations/0002_add_email_field.py
class Migration(migrations.Migration):
    dependencies = [
        ('myapp', '0001_initial'),
    ]

    operations = [
        migrations.AddField(
            model_name='user',
            name='email',
            field=models.EmailField(),
        ),
    ]

Django can "replay" all migrations to build a complete picture of what the schema should look like.

Building the Project State

When makemigrations runs, it builds two parallel representations:

# django/django/core/management/commands/makemigrations.py
class Command(BaseCommand):
    def handle(self, *app_labels, **options):
        # 1. Load the current state from migration files
        loader = MigrationLoader(None, ignore_no_migrations=True)

        # 2. Build state from current model definitions
        from_state = loader.project_state()
        to_state = ProjectState.from_apps(apps)

from_state (the "before"): Built by replaying all existing migration files in order. This represents what Django thinks your database schema currently looks like.

to_state (the "after"): Built directly from your current model classes using ProjectState.from_apps(apps). Remember that apps registry we discussed? This is where it gets used!

The Autodetector: Finding the Differences

Django's MigrationAutodetector compares these two states:

# django/django/db/migrations/autodetector.py
class MigrationAutodetector:
    def __init__(self, from_state, to_state, questioner=None):
        self.from_state = from_state
        self.to_state = to_state
        self.questioner = questioner or MigrationQuestioner()

    def changes(self, graph, trim_to_apps=None, convert_apps=None, migration_name=None):
        """
        Main entry point to produce a list of applicable changes.
        Returns a dict of {app_label: [Migration, ...]}
        """
        # Detect all the changes
        self._detect_changes()

        # Generate actual migration operations
        return self.arrange_for_graph(...)

The autodetector walks through a series of detection methods:

generate_created_models() - Finds models in to_state but not in from_state
generate_deleted_models() - Finds models in from_state but not in to_state
generate_added_fields() - Compares fields on each model
generate_removed_fields() - Looks for missing fields
generate_altered_fields() - Detects changes to field properties (max_length, null, etc.)
Many more - for indexes, constraints, model options, etc.

How Field Changes Are Detected

Here's a simplified look at how Django detects a changed field:

# django/django/db/migrations/autodetector.py
def generate_altered_fields(self):
    for app_label, model_name in sorted(self.kept_model_keys):
        old_model = self.from_state.models[app_label, model_name]
        new_model = self.to_state.models[app_label, model_name]

        for field_name in old_fields_keys & new_fields_keys:
            old_field = old_model.get_field(field_name)
            new_field = new_model.get_field(field_name)

            # This is the key comparison
            if old_field.deconstruct() != new_field.deconstruct():
                self.add_operation(
                    app_label,
                    operations.AlterField(
                        model_name=model_name,
                        name=field_name,
                        field=new_field,
                    ),
                )

The deconstruct() method is crucial here. Every Django field knows how to represent itself as a serializable tuple:

# Example: EmailField().deconstruct() returns:
# ('email', 'django.db.models.EmailField', [], {'max_length': 254})

# If you change max_length=100, the deconstruct output changes:
# ('email', 'django.db.models.EmailField', [], {'max_length': 100})

Django compares these deconstructed representations. If they differ, it knows the field changed.

The Complete makemigrations Flow

Let's trace the entire journey:

1. You run: python manage.py makemigrations

2. Django calls django.setup()
   └─> Imports all models (ModelBase metaclass registers them)

3. MigrationLoader reads existing migration files
   └─> Builds from_state (the "before" snapshot)

4. ProjectState.from_apps(apps) reads registered models
   └─> Builds to_state (the "after" snapshot)

5. MigrationAutodetector compares states
   ├─> generate_created_models()
   ├─> generate_added_fields()  
   ├─> generate_altered_fields() ← Uses field.deconstruct()
   └─> [many other detection methods]

6. Autodetector generates Operation objects
   └─> CreateModel, AddField, AlterField, etc.

7. Operations are packaged into Migration class
   └─> Written to migrations/000X_auto_YYYYMMDD_HHMM.py

8. Migration file saved to disk

Why This Design Matters

This architecture has interesting implications:

1. Migrations are the source of truth

Your database schema is defined by the sequence of migrations, not the current model code. This is why you can't just edit old migrations—they're historical records.

2. Autodetection isn't perfect

The autodetector can't detect everything. Field renames look like "delete old field, add new field". That's why Django asks you questions during makemigrations.

3. State is app-local, but dependencies are global

Each app tracks its own migrations, but models can reference each other across apps via ForeignKey. Django's dependency graph ensures migrations run in the right order.

A Practical Example

Let's say you change this:

class Article(models.Model):
    title = models.CharField(max_length=100)

To this:

class Article(models.Model):
    title = models.CharField(max_length=200)

Here's what happens:

from_state has a field with max_length=100 (from previous migration)
to_state has a field with max_length=200 (from your current code)
Autodetector calls old_field.deconstruct() != new_field.deconstruct()
Returns True because max_length differs
Generates AlterField operation
Creates migration file like:

operations = [
    migrations.AlterField(
        model_name='article',
        name='title',
        field=models.CharField(max_length=200),
    ),
]

The Takeaway

Django's model change detection is a two-phase process:

Phase 1 - Model Discovery (what we covered first):

django.setup() imports your models
ModelBase metaclass registers them automatically
Apps registry holds all model definitions

Phase 2 - Change Detection (what we just covered):

Build "before" state from migration files
Build "after" state from registered models
Compare using MigrationAutodetector
Generate operations for differences

Understanding this helps you debug:

"Why isn't Django detecting my change?" (Check if the field's deconstruct() actually changed)
"Why does Django think I'm deleting and adding a field?" (It can't detect renames automatically)
"Can I edit old migrations?" (No—they're the historical record used to build from_state)
"What if I squash migrations?" (You're combining the historical record, but the end state stays the same)

The elegance is in the separation: model discovery uses metaclasses and imports, while change detection uses state comparison. The only database access needed is reading the django_migrations table to see which migrations have already been applied—this tells Django which point in the migration history represents your current schema state.

Cover image by John Fowler

Chatting with your database: What did I learn by letting AI generate SQL?

Srikanth — Sun, 17 Sep 2023 11:10:32 +0000

This is one of the intriguing use cases of ChatGPT and other LLMs.

You pass the information about your database and a question to a large language model. In return you get an SQL query, which you can execute to get the answer.

As a cricket fan, I had an sqlite database with match statistics so I made AI answer some questions. Here's what I learned:

Prompt Engineering is a serious thing. When it comes to utilising LLMs, packaging the right amount of context into the prompts will influence your results. Before using LangChain I wrote custom logic to create context for db schema, and I found myself iterating over minute details to make the queries as accurate as possible.
LangChain is not just a wrapper. It is a very smart tool that glues together various moving parts. I found my results to be more consistent using it.
Denormalised schema will fetch more consistent results. As you can see in the video, one of the queries returned player uuid instead of name. This was because my data model was designed for a different use case where as it will make it easier for GPT-4 if I had the column as player_name instead of id.
There is a risk of data leak if you want to use this in production. Exposing such an interface publicly is risky. We can mitigate this risk my restricting the number of tables shared with AI model and pre-processing the SQL queries generated before executing them. Using a restricted read-only user will be another wise guard rail.

I will continue exploring this use case and share my learning here in future.

Thanks for reading.

// Detect dark theme var iframe = document.getElementById('tweet-1703365121735610859-340'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1703365121735610859&theme=dark" }

How do you stay creative as a dev?

Srikanth — Fri, 01 Sep 2023 10:42:22 +0000

I know this is quite subjective question and creativity is bit of an abstract thing. But I have been reading about this topic and also reflecting on my experience.

There are some specific times when I felt I was being very creative. When I am encountering an interesting bug or trying to code up a cool feature. My motivation is higher and I am able to stay more focussed in these scenarios.

I was wondering if I can do something to put myself in situations where I get this kind of creative boost.

What do you think? Is it something you wonder about?

The virtue of jumping to conclusions when you're learning

Srikanth — Wed, 30 Aug 2023 09:12:26 +0000

My driving instructor gave me a very useful insight on learning. To train your (driving) instincts, trust them and act on them. The feedback from your actions will help you develop your instincts further.

I have been thinking about revisiting my blog post on “Learning to work on existing codebases”, and I decided to apply the above strategy to it.

Learning to work on existing codebases.

Srikanth ・ Feb 4 '18

#python #django #php #learning

In the original blog post I wrote about “what” you need to successfully work with a legacy codebase. But I think this mental model is good way to answer the “how” part.

Jump to conclusions!

When you encounter a new codebase, forming initial assumptions can help you quickly establish a mental framework to make sense of the unfamiliar territory. This enables you to start exploring and asking more informed questions.

Developing assumptions is akin to forming hypotheses. You can use these hypotheses to guide your investigation. As you validate or invalidate these assumptions, your understanding becomes more refined.

By forming broad assumptions about the larger structure or purpose of a codebase, you can efficiently narrow down your exploration to areas that seem most relevant or impactful.

Jumping to conclusions doesn't mean settling on them forever. It's a stepping stone toward understanding. Continuously revising and updating your assumptions based on new information ensures that your understanding evolves.

Assumptions often lead to questions. As you interact with the codebase, you're likely to encounter discrepancies or areas that challenge your assumptions. These moments are valuable as they prompt you to dig deeper and seek clarification.

Assuming a certain context can free you to experiment creatively within that framework. This experimentation can help you identify new possibilities or angles of exploration that you might not have considered otherwise.

What is your strategy to learn complex things?

Cover image by Austin Neil on Unsplash

Vector database: What is chromadb doing under the hood?

Srikanth — Thu, 24 Aug 2023 09:02:34 +0000

I have been hearing about vector databases a lot recently. As I was exploring the python LangChain library, I stumbled upon chromadb. Available as python and javascript libraries, chromadb is an open source embedding (vector) database.

My curiosity for databases and their internals led me to look under the hood of chromadb and understand what it was doing.

But what is a vector in the first place? A vector is a high dimensional represenation of a datapoint in the vector space. Each vector can have from small number to a very large number of dimensions, depending on the complexity of data. Vectors make it easy to determine similarity between data.

Vector databases give us an easy way to store and retrieve vectors. But how do they work?

Vector databases work by creating an index of all the vectors in the database. This index is based on the vectors' characteristics and similarities. When a query is made to retrieve a vector, the database searches the index to find the most similar vectors and returns them as results. This allows for fast and efficient retrieval of vectors, even in large and complex databases.

This is what chromadb is doing as per my reading of the code

For the in-memory version, chromadb uses sqlite to store vectors. Sqlite is a file based relational database that does not have vector support out of the box.
When a document is being added to a collection, chromadb uses a default embedding function to create the vectors for it.
For each collection an index is created using the hnswlib python library (an implementaion of HNSW approximate nearest neighbor search algorithm)
When a text string is queried on the collection—chromadb creates vectors for the strings using the same embedding function as before. Then it searches through the index for k nearest neighbours, where k can be specified in the query.
The index contains the UUIDs for the documents and using them the actual matching text strings are returned.

It is interesting to see that chromadb has utilised existing technologies to create a vector database. I am interested to read more about the vector search algorithms.

Cover Photo by Rushikesh Gaikwad on Unsplash

System Design Case Study of an Online Booking System.

Srikanth — Wed, 16 Aug 2023 22:17:42 +0000

Have you ever experienced the rush of online shopping? I bet you felt an adrenaline rush as you clicked through the website, competing with thousands or even millions of people at the same time.

Top tech companies are behind some of these websites. They are capable of serving millions of users simultaneously. In this blog post I write about an online booking system that you may not have heard about. It is the system powering the booking process of the most visited Hindu Temple in the world, Tirumala.

Tirumala Tirupati Devasthanam is the independent trust which manages the aforementioned popular temple in the state of Andhra Pradesh, India.

My family visits this temple often. I saw the evolution of the digital systems empowering the temple through the years. I was planning to visit the temple last year, first time after the pandemic, and was stuck by the new online booking experience. It was near impossible to buy tickets. At first I attributed this to rapid digitalisation India has seen in the past few years. Although that does play a part, what I learned by analysing the website through a system design lens was fascinating.

My intention is to share my analysis and if possible start a discussion around promoting user centric design in Indian public digital services. I do not intend to disparage the current developers of the TTD booking system. I am only exercising my curiousity in this topic.

In this blog post I write:

An overview of the TTD online booking system: Visualising the user journey through the booking process.
How does it fare in practice: Brief look at how the system performs in real world and common user complaints.
Hard challenges: What are the common challenges faced in designing highly scalable and concurrent systems?
Analysing the weakness: I attempt to explain the critical pain points of the system borne out of its design.
Ways to improve: I conclude by suggesting ways to enhance the user experience. I offer solutions which are both techincal and product design focussed.

How does the booking system work?

The bookings become LIVE at an announced time.

Bookings for various services, including Special Entry Darshan, are open at specific dates and times. This information is communicated through the TTD website, social media, TV, and YouTube channels.
Special Entry Darshan offers a chance for devotees to have a viewing or be in the presence of a God at a temple. These bookings are typically accessible in the last week of the current month for the following month, although the exact dates can vary. Sometimes, the bookings for the following two months are made available simultaneously.
It's important to stay attentive to updates regarding opening times. It's uncertain whether the TTD app provides push notifications for such announcements.

You must be prepared to grab your slot.

People are required to be registered on the website with their email and phone number. One can login to the website using phone number and a one-time password sent via SMS.
Let's say the bookings are going live at 10AM on Friday. We need to be logged-in on the app or website as close to 10AM as possible. We would get auto logged out in a short interval so timing is important. 🕰

It's time for some speedy mouse clicks!

After you have logged in and you click on the Special Entry Darshan from the "Online Services" menu you will be shown a countdown timer. You are essentially in a virtual waiting room to be let in. It seems like the time can vary depending on the traffic.
Now we are finally in. We see a calendar interface with, hopefully, lots of open slots to book. Now we need to select a date and time slot to book tickets.
After we enter number of people who are going to attend, we will be redirected to a page that asks us fill in details of each attendee. There are 2 dropdowns and 3 free text fields for each attendee. We fill everything correctly and get directed to a payment page.
There are several options to pay, we select one and complete the payment. Finally we get redirected to a success page where there is a booking reference number and details about how to download the booking confirmation.

So how does the system perform?

We saw quite a few steps there and you can see that there is lot of work went into implementing the system.
I observe some pain points here. Firstly, the wait time to enter the booking system is bit opaque and doesn't make the experience smooth. (I know why this wait time is being used, we will talk about it in the next section.) Secondly there are lots of information to fill before finalising payment.
Both of the above slow down the user to complete the booking. I have personally experienced this and so have many others. By the time we are able to fill all the details and click continue, we are shown an error that "the slot was just booked by another pilgrim".

What are the challenges here?

We know that building software that attracts traffic in millions is hard. It requires real technical skills to ensure the system can handle the peak time traffic.
Let's see what are the technical challenges in some detail. The obvious one is to just being able to serve the peak time traffic. I am not aware of the internals but I can guess that there is a load balancer sitting in front of multiple servers or containers to manage load. Since the bookings go LIVE at a certain time, preemptively increasing the server capacity is good way to deal with this situation.
In one of the press releases, TTD stated that after the launch of the new booking system they saw 2,40,000 tickets booked within an hour. Around 900,000 concurrent sessions were open at the peak time. And, a total of 10 million hits were recorded by the system on that day.
Designing a booking system capable of serving ~1 million concurrent users, with 240,000 available tickets will lead to dealing issues like scalability, performance, and data integrity.
Moreover, preventing multiple users from booking the same ticket concurrently presents intricate challenges. These involve mitigating race conditions, maintaining atomicity throughout multi-step booking processes, and handling the intricacies of reservation periods.
Locking and concurrency control mechanism to avoid any data inconsistencies becomes crucial part of the architecture design. We would need a database with strong consistency and transaction support but also which is easily scalable. I can also imagine a scope for challenges like race conditions and redundant bookings.
Striking the right balance between ensuring all the technical intricacies and providing a seamless user experience will be a complex task.
This is also a product design challenge. Ultimately the goal of this system to facilitate problem-free booking. To overcome the technical challenges, there have been design decisions made here. The big one is the virtual queue. The intention is to hold people in queue so that the main system is not overwhelmed by load. But it brings a different set of challenges that we will see next.

What problems do I see in current design

I think the biggest problem is users reaching one step before payment and being told that the ticket is already booked by someone. Why does that happen? When we select the online booking service we are first added to a virtual queue. The waiting time in the queue is proportional to the traffic.
Let us consider that 900,000 users trying to grab 240,000 tickets at the same time. I don't have any internal information about the system design. But let's say 20,000 users are allowed in at a given time.
Since the number of tickets is much greater than number of users, that means these 20,000 users are supposed to have 100% chance to book their tickets.
In reality it is not the case. Users are trying to book slots -- at a specific date and time period. There is a real chance that X% of users may choose the same time slot, and if they are more than the available slots then they will encounter an error. Weekends and holidays are often booked first, which makes it more likely for an overlapping booking.
There is also a question of how to manage the virtual queue. Let us assume that N users gain access to the system every minute. A user, who is already in, needs to complete the process before any new users enter. The slower one takes, the more competition one faces.
I found a YouTube video which revealed a design flaw. Apparently the waiting time for virtual queue can be changed by refreshing the page. The system seems to be assigning wait times either randomly or based on variables which are highly volatile.

With the above context, entering 4 fields of data per person manually before being able to confirm the booking causes a significant obstacle for users trying to book higher number of tickets at a time. This probably why many people have had bad experience with booking.

How would I improve this

The goal of this exercise is to minimise the number of errors faced by users. Specifically the errors that stop them from completing their booking. I want to make the system more fair and transparent. Right now it favors someone who is very computer savvy or someone who knows about the loopholes in this system.

Small tweaks: A technical solution.

As described above, the process of entering pilgrim data manually slows the users down. Since this step comes after choosing the time slot, the chosen slots may be filled in the interim. If we tweak the system so that the data needs to be entered before choosing the time slot then it would give users more accurate picture of available places.
Similarly letting the users to save information ahead of time and selecting them during booking process can also improve the experience.
When users face the “the slot was just booked by another pilgrim” error, they can be given an option to choose an alternative time slot. They need not re-enter the queue for a chance to book a slot.
In order to make the system more transparent, a status message can be displayed to the users. This can indicate approximately how many users are online and how many tickets are available.
Lastly, I would monitor the number of errors caused by 2 or more users trying to book the same time slot. This will not only give us the current state of the system but will also become the metric to measure the success of any future changes.

Thinking from first principles: A system design solution.

What if we want to rethink how the system is supposed to work. An online booking system that is designed as first-come-first-serve is probably not a fully fair system. An online booking system that is designed as first-come-first-serve, but doesn’t guarantee tickets on first-come basis is definitely not a fair or transparent system.
For inspiration, I turned to sports. Wimbeldon, Lords Cricket Ground and Old Trafford Football stadium, all sell a portion of their tickets through a ballot process. Every year people are invited to apply for the ballot informing which days/matches they want to attend. Once the ballot closes, tickets are assigned randomly. Payment is processed only for those who are successfully assigned tickets.
We can adopt this system for TTD online booking. Although we will need to tweak a couple of things. Due to Reserve Bank of India’s directive on automatic payments, charging user’s cards when they are successfully assigned tickets might not be possible. Instead users may be sent emails/texts to inform about their tickets and asked to complete the payment within 72 hours. If a payment is not received, the tickets can be released back to the available pool.
Some constraints may also be placed on the number of bookings requested by users. Users who haven't recently visited can be given higher priority.
This approach not only gives everyone an equal chance but also reduces the burden on the system. A ballot can stay open for many days, hence there is no need for a traffic spike to occur.

Takeaways

The Tirumala Tirupati Devasthanam (TTD) online booking system for the most visited Hindu Temple, Tirumala, is a complex booking system designed to handle peak time traffic.
The system has challenges related to scalability, performance, and data integrity, and is designed to distribute traffic across servers, optimize database operations, minimize latency, and handle potential security threats.
The virtual queue is the big design decision that TTD made to handle peak time traffic. However, there are design flaws in the system that need to be addressed to make it more fair and transparent.
There can be several approaches to overcome the challenges faced by the system. I have only discussed a few. It is always helpful to reframe the problem. This allows us to come up with solutions that are simplest and most effective.

Thanks for reading. If you liked this blog post, you can follow me on LinkedIn for more.

Deploying software is like opening a new bakery 🧁

Srikanth — Sat, 05 Aug 2023 20:42:00 +0000

What the hell is a deployment? That's what my wife asked me when she came across this word on her beginner's IT course.

I was honestly stumped.

Software development is full of jargon. As I have spent years working in this field I have developed a habit of collecting the jargon and using it as shortcuts to communicate ideas.

But only when she asked me this question that I realised how confusing it can be for a newcomer to understand the dense language of software development.

Words like deployment, production, DevOps, CI/CD pipelines are powerful in communicating broad ideas, but are very hard to understand if you've never seen them in practice. For this reason, we must take a moment to bring back simplicity to how we explain and communicate.

This was my attempt to explain software deployment. I have obviously over-simplified some aspects, but the goal was to introduce the concept in an approachable way.

Imagine you're an aspiring baker with a passion for creating mouthwatering treats. You've spent weeks perfecting your recipes for delicious bakery items like cakes, cookies, and pastries. Now, it's time to open your bakery and share your creations with the world. Here's how the bakery setup process aligns with software deployment:

Bakery Items (Software): Just like you've prepared a variety of bakery items, in software development, you've written code to create different software applications or services that serve specific purposes. However is not always the case that software is the product, sometimes software is what enables you to sell your product.
Recipe Testing (Testing): Before your bakery's grand opening, you meticulously test each recipe to ensure the taste, texture, and presentation are just right. Similarly, during software development, developers rigorously test the software to identify and fix any bugs or issues. You might seek opinion of your family or friends to know if your pastries taste good, likewise a software is shared with a select group of people to understand if it meets their needs.
Bakery Equipment (Infrastructure): You invest in essential bakery equipment like ovens, mixers, and cutlery to help you prepare your delectable treats. Similarly, in software development, you set up a server or cloud infrastructure with the necessary hardware and software to run your applications. A server is just a computer in a data center that runs your software and makes it accessible over the internet.
Bakery Display (Packaging): In your bakery, you beautifully present your treats on shelves and displays, making them inviting for customers. Likewise, in software deployment, you package your application along with its required files and dependencies, ready to be showcased in the production environment. Software applications are usually developed on top other foundational software. Hence packaging them all to work together is essential, just like how you'd arrange a bakery with all its moving parts.
Grand Bakery Opening (Deployment): Finally, the big day arrives! You open the doors of your bakery to the public, inviting them to taste and enjoy your delightful creations. Similarly, when deploying software, you make the applications available to users, allowing them to access and benefit from their features. You would hear terms like "Going Live", "Launching" or "Going to production.". It is the digital equivalent of putting an "Open" sign on your bakery.
Customer Experience (Monitoring): As customers visit your bakery, you ensure they have a delightful experience by maintaining a clean and inviting atmosphere and offering excellent customer service. In the software realm, you monitor the application's performance, ensuring it runs smoothly and efficiently for users.
Feedback and Improvements (Updates): You listen to customer feedback to understand what they love about your treats and where improvements can be made. Similarly, in software development, developers gather user feedback to make updates and enhancements, improving the software's functionality and user experience.

Just as successfully “deploying” your bakery brings joy to customers who relish your delicious treats, smooth software deployment enables users to access and enjoy your applications, helping them in various ways.

And remember, just like a bakery requires ongoing attention and innovation to keep customers coming back for more, software applications also need regular monitoring and updates to maintain their reliability and user satisfaction.

Cover Photo by Michał Parzuchowski on Unsplash