Forem: Jessica Wang

Reverse Engineering DUPR's Pickleball Rating Algorithm

Jessica Wang — Wed, 31 Dec 2025 00:13:58 +0000

TL;DR: I built a machine learning tool that predicts DUPR rating changes with 86% accuracy using web scraping, Python, and Gradient Boosting. After scraping 6,844 player-match records from 35 players, I discovered DUPR uses a heavily modified ELO system with rating compression at extremes and opponent strength as the dominant factor.

GitHub: https://github.com/DaRubberDuckieee/pickleball-dupr-predictor
Live Demo: dupr-predictor.vercel.app

Intro

DUPR (Dynamic Universal Pickleball Rating) is the standard rating system for competitive pickleball, but it's not super clear how it gets calculated. For example, I could play somebody of a much lower DUPR than me, but my DUPR could still go down if I don't absolutely crush them. Or I could play somebody of a higher DUPR and lose, but my DUPR could still go up.

So I wanted to answer: Can I predict my rating change before playing a match?

Existing solutions didn't help.

There is some information on their website around match recency, volume, and reliability that is good conceptually but isn't specific.
DUPR does have an app with a "forecast" tool where I can input 4 players (to simulate doubles) but it will only tell you what it predicts the score should be if you were to play. It won't tell you how ratings will changed based off a hypothetical score you input.

Goals & Non-Goals

Goals:

Scrape enough real life match data to reverse engineer the algorithm
Build a predictive model based on that data with a "good enough" accuracy (good enough means I look at it and think it looks good enough)
Deploy a web interface for predictions

Non-Goals:

Perfect accuracy (probably always off by a teeny bit)
Singles ratings (focused on doubles only)
Real-time scraping (one-time data collection was sufficient)

High-Level Architecture

Scraper: Selenium + BeautifulSoup (Python)
Data processing: pandas + numpy
ML model: scikit-learn (Gradient Boosting Regressor)
Backend: Flask API
Frontend: Vanilla HTML/CSS/JS
Deployment: Vercel (frontend) + Render (API)

Here's how the data flow worked.

1) Selenium scrapes player match histories from pickleball.com. It will get back some HTML like this:

<div class="hidden md:block"> <table> <tr> <td>Dec 29, 2024</td> <td>Jessica Wang26 | F | WA, USA</td> <td>5.088 → 5.126 (+0.038)</td> <td>Partner: 5.353 → 5.383 (+0.030)</td> <td>Opponents: 5.148, 5.297</td> <td>Score: 15-1</td> </tr> </table> </div>

2) BeautifulSoup parses rating data from dynamically rendered pages and returns back data in this format:

{ 'date': '2024-12-29', 'team1_player1_rating_before': 5.088, 'team1_player1_rating_after': 5.126, 'team1_player1_rating_change': 0.038, 'team1_player2_rating_before': 5.353, 'team1_player2_rating_after': 5.383, 'team1_player2_rating_change': 0.030, 'team2_player1_rating_before': 5.148, 'team2_player2_rating_before': 5.297, 'team2_player1_rating_change': -0.093, # Inferred (they lost) 'team2_player2_rating_change': -0.175, 'game1_team1_score': 15, 'game1_team2_score': 1 }

3) pandas processes 6,844 player-match records into training data. Here's the code if you want to read it.

4) Gradient Boosting model trained on 14 engineered features, such as score margin, surprise level, and opponent's average rating.

# For Team 1 Player 1 (rating 5.088) features = { 'won': 1, 'rating_diff': 5.088 - 5.2225, # = -0.1345 'score_margin': 14, 'total_points': 14, 'partner_diff': 5.088 - 5.353, # = -0.265 'team_vs_opp': ((5.088 + 5.353)/2) - 5.2225, # = -0.002 'won_x_rating_diff': 1 * -0.1345, # = -0.1345 'won_x_score_margin': 1 * 14, # = 14 'rating_squared': 5.088 ** 2, # = 25.888 'surprise': 1 - (1 / (1 + 10**((5.2225 - 5.088)/4))), # = 0.52 'opp_spread': abs(5.148 - 5.297), # = 0.149 'player_rating': 5.088, 'partner_rating': 5.353, 'opp_avg': 5.2225 } Feature vector: [1, -0.1345, 14, 14, -0.265, -0.002, -0.1345, 14, 25.888, 0.52, 0.149, 5.088, 5.353, 5.2225] Target: 0.038

5) Flask API serves predictions, and static frontend sends match scenarios to API. So I could send this hypothetical score:

curl -X POST http://localhost:8080/predict \ -H "Content-Type: application/json" \ -d '{ "team1_player1": 5.0, "team1_player2": 5.2, "team2_player1": 4.8, "team2_player2": 5.1, "team1_score": 11, "team2_score": 9 }'

And the API would return this back:

{ "team1": { "player1": { "rating_before": 5.0, "rating_change": 0.022, "rating_after": 5.022 }, "player2": { "rating_before": 5.2, "rating_change": 0.016, "rating_after": 5.216 } }, "team2": { "player1": { "rating_before": 4.8, "rating_change": 0.042, "rating_after": 4.842 }, "player2": { "rating_before": 5.1, "rating_change": 0.049, "rating_after": 5.149 }

Model Types and Findings

Each graph shows how well a model predicts DUPR rating changes across all 13,832 training samples. The X-axis represents the actual rating change DUPR gave, while the Y-axis shows what each model predicted. The diagonal dashed line represents perfect prediction—if a model were 100% accurate, all dots would fall exactly on this line. You can visually see the improvement: poor models create horizontal clouds of dots, while good models create tight diagonal streams.

Before diving into the models, here's what the metrics mean:

👉 R² (R-squared): Measures how well the model explains the variance in rating changes. A score of 1.0 is perfect prediction, 0.0 means the model is no better than guessing the average. An R² of 0.86 means the model explains 86% of why ratings change the way they do.

👉 MAE (Mean Absolute Error): The average prediction error in rating points. An MAE of 0.11 means predictions are off by ±0.11 points on average.

Here are the 4 models we tested.

1) Linear Regression (Baseline)
What it is: Simple weighted sum of features. Assumes the relationship between inputs and rating change is a straight line.

Features used: won, rating_diff, score_margin, total_points

Performance:

R² = 0.12
MAE = 0.389

Why it failed: DUPR doesn't use linear math. A 4.0 player's rating change follows different rules than a 3.0 or 5.0 player. Linear models can't capture these rating-level specific behaviors.

2) Linear Regression with Engineered Features
What it is: Same linear model but with smarter features like ELO-style "surprise" (how unexpected the outcome was) and interaction terms.

Features added: surprise (won - expected_outcome), won_x_rating_diff, partner_diff, rating_squared

Performance:

R² = 0.54
MAE = 0.287

Breakthrough moment: Adding the surprise feature (ELO expected outcome) nearly 5x'd the R². This confirmed DUPR uses ELO-like logic. However, this still was not good enough, since linear models still fundamentally can't handle DUPR's complex rules.

3) Gradient Boosting - Aggressive

What it is: Gradient Boosting builds predictions by training a sequence of decision trees, where each tree learns to fix the mistakes of all previous trees. The first tree makes initial predictions based on the training data, but it won't be perfect—some predictions will be too high, others too low. The second tree doesn't try to predict the rating changes directly; instead, it predicts the errors (residuals) from the first tree. This correction is added to the first tree's prediction, creating a combined model that's more accurate. The process repeats: tree 3 predicts the remaining errors after trees 1 and 2, tree 4 fixes what tree 3 missed, and so on. Each tree is weighted by the learning rate (0.05 in our case), which controls how aggressively each new tree corrects previous mistakes. Lower learning rates make the model train slower but generalize better. After 100 trees, the final prediction is the sum of all individual tree predictions, creating a powerful non-linear model that can capture complex patterns like "4.5+ players follow different rules than 3.0 players" without being explicitly programmed with those rules.

Hyperparameters: 150 estimators, max_depth=5, learning_rate=0.1

Performance:

R² = 0.92
MAE = 0.089

Why we didn't deploy it: Too accurate on training data, which means it is likely overfitting. Would probably perform worse on unseen matches.

4) Gradient Boosting - Balanced ✅ (DEPLOYED)

Hyperparameters: 100 estimators, max_depth=3, learning_rate=0.05, min_samples_leaf=10

Performance:

R² = 0.86
MAE = 0.114

Why this won:

10× better R² than baseline (0.12 → 0.86)
Handles rating-level specific rules (3.0 players vs 5.0 players behave differently)
Doesn't overfit

Final Architecture

Deployed Model: Gradient Boosting (Balanced) - Model 3

Feature set (14 features):

Basic: won, score_margin, total_points
Ratings: player_rating, partner_rating, opp_avg
Derived: rating_diff, partner_diff, team_vs_opp, opp_spread
Interactions: won_x_rating_diff, won_x_score_margin
Non-linear: rating_squared
ELO-style: surprise (won - expected_outcome)

Key Findings About DUPR's Algorithm

After analyzing 6,844 player-match records with the trained model, here's what actually matters:

What Matters A LOT:

Opponent strength relative to yours (correlation: -0.329)
Playing up (underdog): Gain rating even when losing (+0.049 avg)
Playing down (favorite): Lose rating even when winning (-0.148 avg)
Rating differential is THE dominant factor
Your absolute rating level (correlation: -0.368)
Higher-rated players experience more deflation
4.5+ players: Wins are worth LESS than losses (-0.006 difference)
<3.0 players: Heavy inflation regardless of outcome
Expected outcome vs. actual outcome
DUPR uses ELO-style surprise calculation
Upset wins rewarded heavily (+0.400 for underdogs)
Expected wins barely rewarded (-0.148 for favorites)

What Doesn't Matter:

Score margin (correlation: +0.018) Winning 11-0 vs 11-9 changes your rating by only ~0.05 points Blowouts don't give bonus points
Win/loss alone (+0.040 difference) Winning helps, but barely WHO you beat matters 50× more than IF you beat them

The Zero-Change Mystery:
• 10.3% of all matches result in exactly 0.000 rating change
• Happens at all rating levels, both wins and losses
• Theory: DUPR applies threshold rule when match provides "no new information"
• More common in balanced matchups (±0.3 rating diff)

Strategic Implications:
• ✅ Play against higher-rated opponents (maximize rating gain)
• ✅ Win close matches as underdog (+0.400 avg boost)
• ❌ Don't worry about score margin (negligible impact)
• ❌ Avoid being heavily favored (>0.3 advantage = deflation)

Bottom line: DUPR is a heavily modified ELO system with rating compression at extremes and aggressive deflation for favorites. The model achieved R²=0.86, meaning it captures 86% of DUPR's logic—the missing 14% is likely hidden factors like match recency or tournament context.

Live demo: dupr-predictor.vercel.app

Resources

GitHub: https://github.com/DaRubberDuckieee/pickleball-dupr-predictor
Live Demo: dupr-predictor.vercel.app

What We Learned From Building Share-Brewfiles (Astro + React + Clack CLI)

Jessica Wang — Fri, 31 May 2024 06:42:54 +0000

INTRO

What did we build?

The Share Brewfiles project was built to make it easy for people to share out their full tech stack. We also added a social aspect to the site so that developers could have fun with it.

Here’s what we built:

A personalized Brewfile page that is easily shared and just requires one CLI command to be updated whenever you want.

Generates a developer personality profile based off the packages in your Brewfile, really fun to share with friends & social media.

A leaderboard that lets you see what’s popular overall and discover new tools.

Search that allows you to search by package or username so you can stalk other developers’ Brewfiles.

Run npx share-brewfiles in your CLI to try out the tool.

Also, this project is 100% open source. I want to give a huge thank you to Warp (the company I work at) for encouraging me to build this tool. We hope this project contributes to the developer community in a fun and educational way!

Our inspiration

Most developers’ careers can be tracked by the packages they’ve downloaded over time. At school, I downloaded frameworks like Python, Node.js, and Java. Upon graduating, Monokai Pro became my VSCode theme, Droid Sans Mono became my default NerdFont, and I used Starship to customize my terminal prompt. In recent years, I’ve experimented with newer tools like Warp, Raycast, and Github Copilot.

What tools we download tell the story of who we are as developers. We enjoy sharing our hand-selected tools in blogs or Youtube videos titled “Must-Have Tools And Apps” or “How To Customize Your Dev Machine In 2024”. But this content can’t be easily updated as you evolve, and usually only contain 5-10 tools when realistically most developers are probably using >50 packages.

Technical Overview + Tech Stack

Our tool can be split into 2 parts:

1: The command line experience
Stack: Clack CLI, Github auth, Node.js

Run npx share-brewfiles to generate your Brewfile package list for you if you don’t already have an existing one in that directory. It collects your Github auth and sanitizes your data before uploading it to the brewfiles.com API endpoint. The UI of the command line interface (white vertical lines, spinning and text animations) is dictated by Clack CLI.

2: The brewfiles.com website
Stack: Astro, React, Tailwind CSS, Firebase

Just a reminder that this project is completely open source. This code supports the entire web experience (which we described in the “What did we build?” section) which is too much to cover, but we’ll point out a few interesting chunks of our codebase in case you decide to poke around.

api/uploadBrewfile
This is the endpoint that the CLI logic will hit when uploading a Brewfile to the site. This logic will sanitize the data and check if this user already has an existing Brewfile (and update it if needed) before uploading it to our Firebase database. It will also asynchronously kick off the process of parsing through the packages to determine the developer’s personality type.
generatePersonality.ts
This file contains the logic through which we determine a developer’s personality type. You can see all the different possible personality types, as well as the criteria we set to be categorized for each personality. We realize these categories and criteria may be a bit arbitrary, but cut us some slack - we decided to build this feature 2 weeks before launching! If you want to see improvements, send in a PR.
labelledBrewfiles.ts
This file is a dictionary of packages categorized by certain characteristics, like whether it’s frontend/backend/devops/ai, whether it’s related to the CLI, and more. This dictionary is what we use to filter our leaderboard by “top dev apps” and “top CLI tools”, as well as what we use to determine a developer’s personality type. We used ChatGPT to help us label these packages.

What we learned

Astro vs React

When choosing a web framework, we settled on Astro. Admittedly, part of this decision was inherent bias. Astro was a technology that tech influencers were claiming to be “S-tier”, and we thought this newer tool would be fun to write about and attract attention from those who were curious. From a more logical standpoint, we chose Astro because SSR rendering meant better performance and better implications for SEO. Plus, it allowed for a high degree of flexibility by simplifying the management of dependencies like React/Vue/Svelte.

One downfall of this approach is that Astro is meant for content-heavy, mostly static websites, whereas the Brewfiles website has many dynamic elements. While content-rich sites have their place, our thoughts about Astro mostly came down to planning. Due to the nature of building the site as we were designing the UI/UX, we continued to make adjustments—each one moving more and more dynamic.

Thankfully, Astro can include React and other UI frameworks with a simple integration, but we lost most of the benefit of Astro as we continued to make the site more dynamic. So while Astro could handle it, a React meta framework would have likely been a cleaner experience. In the end, we were left with a few Astro route shells filled mostly with React components and manual vanilla JavaScript components.

For example, on our website, we have a screen that pops up an auth code that the user needs to copy when uploading their Brewfile to the site. Here’s what it looks like.

When the user presses “Copy Code”, the text on the button will then change to “Copied!”. Here is what the Astro code looks like (or see code here):

As you can see in the code, we need to interact with the DOM directory using JavaScript to add event listeners and perform actions based on those events. This approach is quite manual. It’s clear that Astro encourages developers to pre-render as much content as possible at build time. If we had written this code in React, we could simplify the logic using React’s state management and event handling capabilities. The code may look something like this:

As you can tell here, the use of useState and onClick() is much cleaner than using the <script> tag to embed Javascript code directly within the HTML markup. Because of the complexity caused by Astro, we ended up rewriting a few of our files using React instead.

Though we had to work around some of Astro’s capabilities as a framework, there were still some upsides, which we’ll talk about in the next section when it comes to rendering and performance.

Sever-Side(SSR) vs client side(CSR) rendering

Astro SSR adapters enable server-side rendering, meaning as routes are requested, we could tell Astro to generate a finalized HTML page on the server. Some pages, however, require processing a lot of data before rendering the page. With each new brewfile submission, more processing will be required, and yet we want to maintain quick page loading speed as our data entries grow.

We decided to server-side render each route and embed slower sections as client-side rendered components. As an example, the homepage includes a leaderboard list at the bottom. Generating the list requires fetching all users’ Brewfiles from the database and calculating a top 10 list.

We created a server API route to handle the calculations and then embedded a client-side React component to hit the endpoint and display the final list. This combination of SSR and CSR allowed for the best possible performance and user experience.

Data decisions

Basic data sanitation

We had to take into account the possibility that somebody could add random sentences into their Brewfile or generally corrupt the file (since it is just a plain .txt) before uploading it to our site. As a result, we checked for a format of followed by and sanitized the strings of extra quotes and spaces.

Client-side validation The reason that the CLI logic hits the API endpoint on the website and not Firebase directly is because we wanted to do some client-side validation before uploading the data to the database.

For example, we check if this user already has a Brewfile that’s been uploaded. If yes, then we will update the existing Brewfile with the incoming data so we can ensure that 1 user will only ever have 1 uploaded Brewfile on our site.

Database/Server-side validation

Firebase implements security rules to lock down read/write access to your documents, but validating the content of your data is less robust. We decided to approach validation in two ways to provide some basic safety for our data.

First, we provided some basic rules for reading and writing to our documents in Firestore rules. While Firebase recently started offering an early-access PostgreSQL database solution, we used its traditional document collections. The security rules ensure any document in the collection follows a basic structure with only a particular set of keys, but you can’t easily ensure the shape of your data.

Secondly, due to some of the limitations of verification in Firestore, we decided to provide more structured validation when fetching documents in our server-side API endpoints. The isValidBrewfile function offers four validating options, one for each piece of data associated with a brewfile. By default, all four options are enabled yet for any particular validation call we can turn off a setting to skip that validation when needed.

Since so much of the UX depends on valid data, this two-tiered validation system means we can trust the data when processing Brewfiles. It’s not iron-clad validation, but good enough for our purposes.

User experience decisions

Overall UI/UX

When thinking of the design, we considered a few things.

First, we are trying to bring some “pizzazz” into Brewfiles, which are normally just normal, boring, black-and-white .txt files. For that reason, the UI should feel cool, sleek, and modern.

Second, package lists are very text-heavy, meaning that there won’t be many opportunities to show off graphics/images on this website. That’s why we chose an in-your-face font and a bold-blue color scheme that would catch people’s eyes.

We didn’t have design resources when first starting this project, so one of our engineers decided to try mocking up some initial designs. Luckily, the UI you see today was created by Kyle, one of our very talented designers - but we thought it would be funny to show you some of the initial mockups:

Brewfile Cards

We had some interesting conversations around the design of our Brewfile card. Originally, we had coded our cards to look like this:

Where the card would show the first 7 packages in the developer’s Brewfile, and highlight packages when a certain keyword was being searched. However, we realized this wouldn’t work if the keyword matched with multiple packages within the Brewfile. For example, just the letter “a” would likely match with many packages. This is why product designers are important! Sadly, we had to rethink the UI for the card, which led to some backtracking in terms of our code and implementation. Here were the options we had come up with:

And you can check out the “All Brewfiles” page to see which option we decided on.

Product decisions + product philosophy

A large part of this project was deciding what features to build. For example, we could have doubled-down on features that analyzed the packages within Brewfiles and gave developers more extensive statistics about the tools they download. We could have also changed the UI to only show the top 10 uploaded packages and only updated it once per week, to encourage people to come back to see the new leaderboard.

The leading philosophy behind the features we decided to build was education and social fun. We wanted this website to be a place where developers could discover new tools, which is why we have a complete leaderboard of every single package that has been uploaded, as well as different filters to sort by different categories. We also wanted to create a sense of community, which is why we required developers to connect a Github account with their Brewfile and added a search functionality. And our “developer personality” summary added a little bit of fun to the whole experience, and encouraged developers to share their Brewfiles out with the world.

Discovering new tech/frameworks

Container queries have been stable in modern browsers since Firefox added them in February 2023 and they currently are supported in 90% of browsers used globally. In short, container queries provide control over styling decisions based on the size of a parent container (instead of @media queries which are based on the size of the viewport).

The cards on the /brewfiles route change sizes as the viewport changes, but consistently with the viewport. At first, the cards take up the full viewport width, continuing to increase until 768px, where the list moves to a two-column layout.

Once again, the cards expand until displaying in three columns at 1280px. A simple @media query would only adjust the typography and other UI elements in a straight line, getting larger with the viewport. However, with container queries, we could adjust all the elements based on the intrinsic size of the cards creating a truly responsive interface!

Conclusion

Future features

Before we end, here are some of the features/improvements we wanted to build, but didn’t get to before we launched.

“Annotating” packages
What if developers could “star” certain packages as ones they used daily or deserved special attention? Or flag other tools as deprecated?

Template Brewfiles
What if we had Brewfile templates for a specific project/project type? For example, if I am a frontend dev setting up a brand new MacOS and can run 1 command to download all packages from a specific template Brewfile.

Closing remarks

We had so much fun creating this tool. It has been such a rewarding experience taking such a boring, black-and-white Brewfile .txt and turning it into a cool, fun, and social experience for the developer community. If you’d like to try it out, here are 2 things you can do.

Run npx share-brewfiles in your CLI and follow the instructions to upload your Brewfile.
Go to your Brewfile page, generate your developer personality, and share it on social media!

And HUGE thank you again to Warp for encouraging me to build this project.

Thank you!

How To Change Your Zoom Background With Code

Jessica Wang — Tue, 09 Jan 2024 05:51:26 +0000

In this blog, I'm going to share a script you can use to change your Zoom background straight from your command line.

Context

So, I work remote at this company, and part of our work culture is that we set special Zoom backgrounds for birthdays, work anniversaries, and other special events. Here's an example from 2024:

Now, I might possibly be the only person in the world who is impatient enough to feel this, but I hate having to go into Zoom to manually upload and change my custom background.

You have to go into your settings for virtual backgrounds, THEN click on the option to upload an image, THEN fumble around in finder to grab the right image to upload to Zoom.

My Preferred Solution

Instead of doing it manually, I wrote a script in bash that would automate this manual work for you. Here is the script:

#!/bin/bash

ZOOMPATH={{zoom_path}}
UUID=$(find "{{zoom_path}}" -type f -not -name ".*" | head -n1 | awk -F: '{print $1}')
RECENTFILE=$(find "{{downloads_path}}" -type f -exec stat -f "%m %N" {} \; | sort -rn | head -n1 | awk -F/ '{print $NF}')
BGPATH="{{downloads_path}}/$RECENTFILE"

echo "Moving $BGPATH to $UUID"
cp -R "$BGPATH" "$UUID"

This script is basically taking your most recently downloaded file and replacing whatever default virtual background you have in Zoom with that downloaded file.

Your downloads_path is most likely something like /Users/<your_username>/Downloads or $HOME/Downloads, but I trust you'll be able to find that path yourself if you moved around your Downloads folder.

Your zoom_path can be found by running mdfind -name "VirtualBkgnd_Custom". If it helps, my path was $HOME/Library/Application Support/zoom.us/data/VirtualBkgnd_Custom. When you run the command, you'll get an output that looks something like this:

Making It Even Easier

After writing the script, I wanted to:

Save this script somewhere so I didn't have to rewrite it every time.
Make this widely accessible for the rest of my team so they could enjoy this automation as well.

To do this, I created a new workflow in my Warp terminal and pasted in my script and pressed autofill. For those of you who aren't familiar, workflows are kind of like aliases that you have in your shell config (.zshrc file, for example) in the sense that it gives you a way to quickly run complex & hard-to-remember commands. I like workflows slightly better because I can share the workflow with my team (if I prefer) and it also auto-parameterizes my command/script so I can vary things like paths, secrets, or more.

If you're curious about the terminal I'm referring to above, Warp is a Rust-based terminal with AI built in. I like it because it has things like autocompletions, history search, click-to-edit, and theming out-of-the-box. Feels super modern compared to the default Mac terminal or iTerm2. And if you do want to try it out, use my referral link & get a free theme. (I work there.)

Conclusion

And finally, if you want to see 1-minute demo of this Zoom-background-changing-script in action, you can check this Loom out. Thanks for reading!

Explaining the Fundamentals of Web Dev

Jessica Wang — Mon, 18 Dec 2023 23:06:07 +0000

Introduction

A couple years ago, I was working as a software engineer at Microsoft OneNote. I was placed on a frontend team and asked to start coding features in React, Typescript, and Redux. I knew nothing about these technologies, but ended up “learning-while-working”. It was relatively effective - I learned enough that after quitting Microsoft, I was able to secure another job as a frontend developer at DoorDash, even making a higher salary.

But, there was a downside to it. I never really took the time to learn fundamentals. For example, I knew how to build buttons using HTML, but couldn’t conceptualize how that a browser used that to render a webpage. I knew how to call the useState and useReducer methods, but didn’t understand when to use them in a way that complied with React’s philosophy. As a result, it was harder for me to debug issues because I didn’t have enough knowledge to understand the fundamental problem. My code was sloppy and led to more bugs. And it took me longer to read documentation since I had gaps in my understanding.

This point of this blog is to explain the fundamentals of web dev from the basics, as if you were my Uber Driver. Let's get started!

What IS web development?

Web development is the process of building and maintaining websites or web applications. Think of “websites” like Wikipedia, company homepages, or personal blogs because they are mostly serving content in a static way. Think of “web applications” like TikTok, Google Docs, and Slack which perform many different actions based on how the user interacts with it.

Frontend vs Backend

A webpage can be broken down into 2 main components.

The client
For a website, the client is the web browser (like Chrome or Safari). It renders everything that the user sees and interacts with, like text on the screen or buttons.
The server
The server is a powerful computer (it could be the laptop sitting on your desk, or it could be a $1000 supercomputer sitting in Texas, depending) that stores data. The reason why it’s called a server is because it has been designated to be a centralized machine that manages a lot of resources at once (and has the computing power to do so).

The client is always requesting information from the server, and the server sends that information back to the client. For example, let’s say we have an online bookstore and I click on a button to get info on “Harry Potter”. The client (my browser) will need to ask the server (wherever it is) to get that information. The server could fetch the title, author, description, and availability of the book. It could even do some logic to check related books to recommend to the user. It will then send that information back to the client. Once that information is received, I’ll see it displayed all pretty on my screen.

Because of this model, we also have two types of roles when it comes to developer jobs (makes sense, right?)

Frontend engineers work on the client-side, and create the visible part of the website like layout, user interface, and interactivity. Their main technologies are HTML, CSS, and Javascript.
Backend engineers work on the sever-side, usually interacting with the server, databases and application logic. Their main technologies are Node.js, Python, Ruby, MySQL, and MongoDB.

Of course, you also got the cool kids (or just people who’ve been working long enough in the industry) who work on both frontend and backend. We call those folks full stack engineers.

Now let’s dive into some of the fundamental basics of frontend development.

Frontend

HTML, CSS, and DOM Explained

Let’s say you’re trying to build out a web page.

In a perfect world, I would be able to tell Chrome “Hey, I want you to put a big blue button in the middle of the page and text centered in the middle saying ‘Click Me’”. But Chrome wouldn’t understand what the f*ck I was saying.

Let’s say the webpage you’re trying to build is like building a house.

HTML elements are building blocks like rooms, doors, and windows that you’d need to arrange when you’re planning the architecture. Each component will have it’s own little symbol so you can grok it quickly in a drawing.

In real life, HTML elements are building blocks like links, images, paragraphs, and headers that you’d need to arrange to build a unique webpage. Each HTML element has its own denotation. A paragraph is represented by <p>, image is represented by <img>, and bullet point list is presented by <ul> and <li>.

HTML is like the blueprint of the house. It outlines where the the components like rooms, doors, and windows should be placed. For example, you want a bigger window on the east side of the building to let more sunlight through in the morning, and a door between the living room and bedroom for privacy.

In real life, HTML would look something like this:

<header>
<h1>Website Header</h1>
</header>

<nav>
<ul>
<li><a href="#home">Home</a></li>
<!-- ... -->
</ul>
</nav>

<main>
<article>
<h2>Article Title</h2>
<p>Content of the article.</p>
</article>

<section>
<h2>Section Title</h2>
<p>Content of the section.</p>
</section>
</main>

<aside>
<p>Additional information or related content.</p>
</aside>

<footer>
<p>© 2023 Your Website</p>
</footer>

Where it arranges our HTML elements (like <body>, <nav>, and <header>) into a specific configuration for the website we are trying to build. Similar to the blueprint of the house.

CSS is the interior design plan that compliments the HTML blueprint. It specifies how each room should be colored, what type of flooring should be used, and what styling should go on the doors and windows.

In real life, CSS could look like this, where there’s specifications for color, font, and padding to stylize each unique HTML element:

/* Apply a background color and text color to the body */
body {
  background-color: #f4f4f4;
  color: #333;
  font-family: 'Arial', sans-serif;
}

/* Style the header */
header {
  background-color: #3498db;
  color: #fff;
  padding: 1em;
  text-align: center;
}

/* Style the navigation menu */
nav {
  background-color: #2c3e50;
  padding: 0.5em;
}

Finally, DOM is like the actual constructed house that is based off the blueprint, with all the rooms, doors, and windows now tangibly built according to the HTML. In real life, here’s what the DOM to the HTML above would look like:

|── html
│   ├── head
│   │   └── (head contents)
│   └── body
│       ├── header
│       │   └── h1
│       │       └── Website Header
│       ├── nav
│       │   └── ul
│       │       └── li
│       │           └── a (href="#home")
│       │               └── Home
│       ├── main
│       │   ├── article
│       │   │   ├── h2
│       │   │   │   └── Article Title
│       │   │   └── p
│       │   │       └── Content of the article.
│       │   └── section
│       │       ├── h2
│       │       │   └── Section Title
│       │       └── p
│       │           └── Content of the section.
│       ├── aside
│       │   └── p
│       │       └── Additional information or related content.
│       └── footer
│           └── p
│               └── © 2023 Your Website

Just like a fully built house is a 3D version of a blueprint, the DOM brings the HTML hierarchy to life. You can more clearly see the concept of parent and children elements here.

(By the way, the reason why developers don’t write directly in the DOM is because it’s pretty inefficient. HTML is more human readable (though honestly after fighting with HTML I’m not so sure about this statement) and directly manipulating the DOM can lead to performance issues.)

To put it all together, when a webpage is rendered, it first parses the HTML (that you coded) and creates the DOM to represent the structure of the webpage. Then it parses the CSS associated with the HTML document to understand color, font-size, layout, and positioning.

Javascript Explained

With just HTML and CSS, your webpage may look good but it’s static and has no interactive bells & whistles. That’s where Javascript comes in.

Javascript brings life and interactivity into your webpage. Using the house-building metaphor from before, Javascript is like smart-home automation. For example, it can do things like controlling lights in different rooms, having a virtual doorbell ring, and setting up a your coffee maker so it starts brewing coffee at 8am every morning.

In real life, here are some of the things Javascript can do (no fun little drawings here, now you get some real life examples in gif form):

1: Event handling
Responding to clicks, mouse movements keyboard inputs. For example, having more information about a restaurant appear when you click “see more” on Yelp.

2: Form validation
Validate form inputs on the client side before submitting the data to the server. For example, Gmail including a password strength indicator to guide users in creating strong passwords with visual cues.

3: Animations and transitions
Fading, sliding, moving elements. For example:

4: Dynamic styling
Changing colors, sizes, or positions based on user actions/conditions. For example, the fact that a link is colored blue when you’ve never clicked on it before, but colored purple if you’ve clicked on it before.

5: Error handling
Javascript can catch and handle errors, providing a more graceful user experience by displaying informative error messages. For example, Bank of America might show you a friendly message saying “You were logged out due to inactivity. Please log in again or contact customer service at this number” instead of just crashing the whole webpage.

6: Geolocation
Enabling location-based services. For example, Google maps reading your current location to calculate travel times to a destination.

7: Dynamic updates via websockets
Websocket connections are real-time communications between the client and the server, leading to quick dynamic updates. For example, Slack uses WebSocket connections to enable instant messaging and notifications.

8: Integration with third-party apps
Javascript can interact with third-party APIs, allowing developers to integrate external services. For example, travel booking platforms accessing the Google Maps API to display locations of hotels and attractions near you.

Other Popular Frontend Frameworks

In no particular order, here are some other popular frontend frameworks that are worth knowing. If you want to dive deep into them, I would recommend researching it more yourself. But this list is mostly just so you’ll be familiar when you hear it at work/see it in a blog.

React: A Javascript library that allows developers to create reusable UI components and manage the state of an application. I strongly recommend you start taking a React tutorial and build a project using React if you’re looking for your first job in web dev.
Angular: A Javascript framework for building dynamic web appliactions that follows the MVC architecture.
Vue.js: A Javascript framework meant to be very adaptable and flexible.
Next.js: A React-based web framework that focuses on ease of development and optimal performance.
Ruby on Rails: A framework that emphasizes convention over configuration and provides a lot of defaults for building web applications.
Svelte: A Javascript framework that compiles components at build time rather than relying on a runtime framework in the browser.

Wrapping up frontend

Frontend engineers should have a basic understanding of how UI on a webpage is generated and displayed through HTML, CSS, DOM, and Javascript.

Commonly, they also learn libraries created on top of these fundamentals, like Typescript React, Redux, Next.js, and more.

On top of that, more senior frontend engineers are familiar with complex implementation of responsive web design, taking into account design for mobile devices, cost effectiveness when implement code/architecture for a website, accessibility, and SEO (search engine optimization).

Backend

What happens on the backend

The backend is responsible for:

Handling requests from the client (frontend).
Interacting with the database to get/store/modify data.
Executing any logic or algorithms needed.

Here are some real-life examples of how the backend works with frontend:

User authentication Imagine a user trying to login to an online banking application. The client collects the users’ username/password and sends them to the server. The server verifies the credentials agains user data stored in the database. If the credentials are valid, the backend generates a token and sends it to the client so it can give the user access to the online bank.

File upload Imagine a user uploads an image to a social media platform. The client allows the user to select an image, then sends it to the server. The server will receive the image, do some logic to validate the format and size, and store it on the server or the cloud. It could also update the database with information about the image.

E-commerce transaction Imagine a user makes a purchase on an e-commerce website. After the user proceeds to checkout, the client sends the order details to the server. The backend does business logic like check product availability, calculates total cost, and processes the payment. It updates the inventory, generates an order confirmation, and sends it back to the frontend to display to the user.

What might a backend engineer work on

1: Writing APIs to allow frontend to communicate with backend. We’ll talk more about APIs later in the section labelled “How frontend and backend communicate”.

2: Manage database to store and retrieve data efficiently, like defining database schemas and creating tables. For example, a backend engineer may create a new table using this SQL command:

CREATE TABLE users (
user_id SERIAL PRIMARY KEY,
username VARCHAR(50) NOT NULL,
email VARCHAR(100) NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Which creates a table of all the applications’ users, with their user_id, their username, email, and the timestamp when that user was created.

3: Add middleware like logging, authentication, and caching.

4: Implementing strategies to scale backend infrastructure to handle increased traffic, like load balancing and horizontal scaling.

5: Implementing security measures to protect sensitive data, like encryption algorithms.

Backend technologies

In no particular order, here are some popular backend frameworks that are worth knowing. If you want to dive deep into them, I would recommend researching it more yourself. But this list is mostly just so you’ll be familiar when you hear it at work or see it in a blog.

Node.js: Allows developers to use Javascript for both client and server-side scripting.
Python: A very versatile programming language used in web dev for its readability and efficiency.
Java: General-purpose language used mainly for building enterprise-level applications.
.NET (C#): a framework used by Microsoft used to create dynamic and scalable applications.

Putting it all together

How frontend and backend communicate

Imagine if the client could just message the backend server and be like “Yo, send me info about Harry Potter ASAP!” and backend server was like “Bet.” Unfortunately, not how computers work in real life. As always, there’s a set of rules defining how messages are formatted and transmitted.

HTTP Requests

HTTP requests is a message sent by the client to a server, and is the foundation of data communication. Let’s dive into the main moving parts that go into building a HTTP request.

1: Methods
These are actions you can request from the server. Some of the main ones include:

GET: Gets data from the server
(For example, let’s say we make an HTTP GET request to https://example.com. You can do this yourself by running curl https://www.example.com in your terminal.)
POST: Submits data to the server.
DELETE: Deletes a resource from the server.
PATCH: Modifies a resource on the server

Remember, this is an article about fundamental knowledge. I only put definitions here so you can get a sense of each method, but no need to memorize them now.

2: Request URL
This is the URL I’m sending this HTTP request to. For example, I could do GET https://api.twitter.com/1.1/statuses/user_timeline.json?screen_name=Elon Musk to retrieve tweets that Elon Musk posts.

3: Request Headers
These are just additional information about the request. For example, I could add something like

Authorization: MY_ACCESS_TOKEN to indicate I have access. or
Accept: application/json to indicate that I prefer to receive the response in JSON format.

4: Request Body
For some actions like POST or PUT where I’m modifying information on the server, the request body contains the data I need to make that update. For example, if I did PUT https://en.wikipedia.org/w/api.php?action=edit&title=Godzilla to update the Wikipedia page on Godzilla, I’d include information in the request body on with the update, like saying Godzilla is 393 feet tall.

APIs

Another fundamental aspect of communication between frontend and backend is API, which allow different software applications to communicate with each other. The most famous APIs are Google Maps API, Twitter API, Facebook Graph API, OpenWeatherMap API, Youtube API, Github API, and Stripe API. Two good examples are ones I used just above, where I referenced hitting request URLs like https://en.wikipedia.org/w/api.php?action=edit&title=Godzilla and https://api.twitter.com/1.1/statuses/user_timeline.json?screen_name=Elon Musk. These APIs allows you to get information about tweets or even lets you update Wikipedia pages.

Conclusion

This is not necessarily an article that teaches you how to get a job in web dev (it’s also not NOT an article that’ll do that), but primarily more for a reader that, as I said, wants to build more of a foundational, conceptual understanding of some of these concepts.

If you want to see the origin story of "Explaining ___ to my Uber Driver series", check out the intro paragraph to my blog "Explaining Kubernetes to by Uber Driver".

Where do I currently work? I'm enjoying life as a developer advocate at a startup called Warp. We're building a new command line experience that makes people better developers because the UI is fast and intuitive. Check it out here.

And finally, as always, please leave comments below. Any feedback on my drawing, factualness of whatever I've included in this blog, and be nice.

Explaining SSH to my Uber Driver

Jessica Wang — Mon, 27 Nov 2023 22:44:12 +0000

In this blog, I'd like to explain SSH to you as if you were my Uber driver. The point of this exercise is to pretend like you have no prior experience with SSH or even much knowledge of technology in general, and get you to a point where you:

Aren't afraid by the vague term "SSH" because you have a concrete image in your head.
Understand how industry-level developers use SSH to test code changes, make server configurations, and more.
Understand how SSH's encryption/authorization protocols make it better than other tools.

Let's get started!

Conceptualizing SSH

What is SSH? Instead of starting with the conventional tech jumbo you'd see at the top of a Cloudflare article or Wikipedia, let me use an analogy to help you conceptualize it.

Imagine you've just gotten off school and you're back home in your bedroom. You're bored, and decide to put on your VR headset to access the Metaverse - an immersive experience that blocks out the physical world and embeds you in a virtual environment. You log in, punching in your username and password to verify that it's you. Once that's done, you see your ditigal avatar pop up. It looks like a spitting image of you - tall, great hair, and a fantastic sense of style. In front of you, you see a forest - or rather, a digital representation of a forest. Still standing in your bedroom with your VR headset on, you physically move your head to look around, and your avatar does the same. You say "hello forest" in your bedroom, and your avatar also says "hello forest" in the VR world.

In real life, this is what the analogy translates to:

You == developer.
Your bedroom == your laptop, or whatever local machine you're typing on.
The Metaverse == a remote machine/server you are trying to access from your local machine.
The VR headset == the SSH tunnel that allows you to access the remote machine (Metaverse) from your local machine (bedroom).
Login screen == the SSH key pair that ensures only authorized parties can use the SSH tunnel. I'll explain this more thoroughly in a later section of this blog.

How does this analogy help you? Next time you hear somebody say "I SSH-ed into the server", you don't need to freak out about this vague concept of "SSH". Instead, reword it in your head to mean "I put on a VR headset to access the Metaverse", and this will ground it something more concrete.

In layman's terms, SSH is like a secure, secret tunnel that lets you connect to another computer over the internet. It helps you do things on that computer as if you were sitting right in front of it.

Of course, this is just a surface-level, conceptual understanding of SSH. I'm hoping right now, you're thinking of some questions to help you dig more into SSH. I can't read your mind, but here some questions I had, answered for myself, and I think would be useful to explain to you.

Why do I need to use SSH?

SSH is important because developers commonly need to access remote machines. For example, large applications that would consume too much memory and computing power on your own measly laptop is hosted on cloud platforms like AWS, Azure, or Google Cloud, which all need to be accessed remotely. Other times, development teams are distributed across different locations, so remote servers allows team members to collaborate and share code in one consistent place even if they aren't physically in the same location.

Let me walk you through 2 examples of when an industry-level developer would need to SSH into a remote machine.

1: Deploying Code

As a developer, you need to deploy a new version of your website to your staging server. This is how it might work, using SSH.

On your local machine, you push your latest code changes up using Git.

Then you would SSH into the remote server that has your staging environment.

(By the way, you might be thinking why I need to have my staging environment in a remote server and not on my local machine. Here are some reasons. One is that it's better to simulate a realistic setup, like checking for performance bottlenecks or scalability concerns that would only happen in a server environment and not your laptop. another is simply ensuring that all team members can test in a consistent environment, instead of them all having a staging environment on their own machines or (god forbid), using your local machine whenever they needed to test something.)

Finally, you've SSH'ed in the remote server that hosts the staging environment, you can pull your latest code changes from that remote server, run the build, and test your application (by navigating to http://staging.example.com, for example).

2: Updating a server configuration

Let's say the web application you're building has grown, and you need to adjust the server configuration to accomodate increased traffic. Here is an example of some commands you might run:

a. SSH into the server

ssh user@your-server-ip

b. Locate Apache configuration files

cd /etc/apache2

c. Backup the configuration files

cp apache2.conf apache2.conf.bak

d. Edit configuration files.

vi apache2.conf

e. Verify configuration

apache2ctl configtest

Please don't feel like you need to know what these commands do.

In all honesty, I asked ChatGPT to generate an example for me, and don't quite understand it myself. The takeaway here is understanding the types of commands a developer might run while SSH'ed into a server. A lot of it is simple commands to move directories (cd), manipulate files (cp), and edit files (vi).

Why SSH?

One question I had while learning about SSH were "what are the alternatives to SSH, and why did SSH win out"?

I guess now is a good time for a quick history lesson.

Before SSH was widely adopted, there were other procotols for communication between two machines. Some notable ones include Telnet, RSH, FTP, and Rlogin. But each of them suffered from lack of encryption and security features.

To paint a picture, here's how it might happen. I (the malicious hacker) gain access to the network where Telnet is passing through - the network through which you're trying to send information from your laptop to another machine through Telnet protocols. I use a tool like Wireshark to capture the traffic. Once I get the traffic on my machine in the form of downloaded "packets", I can just click through them and look for any login attempts. All the data is in plaintext (which means it's human-readable and not some garbled symbols), so if I find it, I can literally use the login information I see in front of me. As you can see, these older tools were not very secure!

In 1995, SSH was invented. It was better for 2 main reasons.

Encrypting data
Let's say that part of the data being transmitted from my machine to a remote machine is the string "username: jesswang password: ?IamC00l!". Without encryption (so with older protocols like Telnet or FTP), a hacker would be able to read that string as you just read it. With encryption, the data they get back might look like "H23kLs*^!d8%PwZx0KsL!9B#N%u@i8". In reality, the encrypted data would be a different sequence of bytes that adheres to encrypted algorithmic standards (that I don't know much about), but this is just to give you an idea.
Authentication
Older programs like Telnet relied solely on password-based authentication. When you connect to a remote system using Telnet, the Telnet server prompts you to enter a username and password to login. However, Telnet transmits your passwords in plaintext, making it easy for hackers to read. SSH primarily uses a different (and better) approach called public-key authentication. I'll explain how it works below so you can understand how your information is better protected from hackers.

First, your information is put in a briefcase, and you put your own lock on it.

You send the briefcase to your remote server. The remote server doesn't have the key to unlock your lock, so instead the remote server puts its own lock on the briefcase.

The remote server sends the briefcase back to you, and you unlock your lock.

You send the briefcase back to the remote server, and it unlocks its lock with its key. Then it opens the briefcase, and reads the note.

And that's the gist of public-key authentication! By the way, SSH also still does support password-based authentication, but even then the password is encrypted so if a hacker were to intercept it, they would not be able to read it.

How do I use SSH?

Enough of the conceptual learning. Here's the actual commands I would run to SSH into a server.

To SSH into a server, you would typically run something like: ssh username@server_ip_or_hostname

You'd replace username with your actual username on the server
You'd replace server_ip_or_hostname with the IP address or hostname of the server. So for example, ssh john@123.456.789.123.

After running the command, you may be prompted for a password for the username of the server. Or if your server uses the key-based authentication we covered in the previous section, you might not be prompted for a password.

Once you're in, you can start executing commands. These commands will be executed on the remote server you're connected to.

I understand it's hard to fully learn if you're not physically typing things yourself and seeing what happens. If you already have a remote server for work or personal that you have set up, feel free to give it a try. If you don't have a remote serve to connect to, that's fine. You can follow these steps:

Go to http://sdf.org/
Create an account. You'll probably receive an email to giving you a generated password for your account. You may have to wait 1-2 for that email (at least that was my experience).
You can SSH into the public UNIX system by running ssh <your_username>@sdf.org. So for example, I made my username poop123. So I ran ssh poop123@sdf.org.
When prompted, enter the password you got earlier.
You've successfully SSH-ed into a server! You can run commands like echo 'hello world' to print out "hello world" or ls to see a list of files in your current directory.

Note To Readers / Conclusion

Though it's relatively easy to SSH into another server (maybe your coworkers give you a command you just paste into your command line), I find it difficult to use the command if I don't understand, at least on a high-level, what is happening when I run it. Hopefully, others can relate with me and this blog will help scratch that itch.

If you're curious how I came up with the title, the story is in the intro paragraphs of the original blog of this series, "Explaining Kubernetes to my Uber Driver".

As always, I would love any feedback on the information, drawings, or storytelling in this blog. And let me know what other topics you'd like for me to write about.

Here are some resources I referenced when writing this blog:

ChatGPT
Beginners Guide to SSH Youtube Video
My 60 second TikTok video summary of the above video.
I used Charm VHS to generate the last gif you saw. Very cool tool that allows you create & easily edit CLI-related demos.
The terminal I used to film the demo, Warp (it's also the company I work at. I highly recommend you check us out).

Explaining Kubernetes To My Uber Driver

Jessica Wang — Sun, 12 Nov 2023 22:04:36 +0000

A week ago, I attended Kubecon 2023 in Chicago. I read a few blogs and attended some 101 tutorials at the conference, but still didn’t have a good understanding of the technology. The worst part was the last day of the conference - I ordered an Uber to bring me back to my hotel. My driver asked me “What’s the convention about?” to which I responded “It’s about Kubernetes", but after some explaining, it was clear I had no idea what I was talking about.

Imagine coming out of a 3 day conference, and not being able to describe the technology to your Uber driver. Facepalm. So, in an attempt to redeem myself, here’s my reimagined conversation with my Uber driver.

Start of Conversation

Me: Imagine you’re a chef at a busy restaurant kitchen. You’ve got a team of chefs working for you, and each one is preparing a different part of the meal - one group for appetizers, one for main courses, and another for desserts. Coordinating these chefs to make sure the meal is served to the customer on time is your job. You have a picture in your head?

Driver: Got it.

Me: In this scenario, the head chef is Kubernetes. Just like that head chef needs to manage all the different chefs in the kitchen, Kubernetes helps manage all the different pieces needed to run your software. The official definition of Kubernetes is a “container-orchestration tool”, but since the word “container” is pretty abstract here, you can substitute the word “container” for “chef”. So Kubernetes would be a "chef orchestration tool". That way, you can form a picture in your head of your kitchen every time you hear the word.

Driver: Okay, makes sense so far. But what are these containers? I can't keep imagining them as chefs forever.

Me: Yeah good point. So now that you have a picture in your head of your Kubernetes kitchen, let’s dive into how all the different kitchen roles map to Kubernetes concepts, going from smallest to largest.

Container
The smallest part of this puzzle is the container, and it’s basically any piece of software. For example, it could be a Node.js web server that hosts a web application, or a MongoDB database container to store data (this sentence is more for the engineers reading this blog, I wouldn’t say this to my Uber driver 😛). In the kitchen, imagine you’re serving soup and a salad for your appetizer. The soup would be your container. The salad would also be it’s own container.

I know this definition seems a bit arbitrary right now, but it’ll make more sense once I explain it in the context of the upcoming components.

Pod
In the kitchen, a pod would be the plate/tray that holds your soup and salad. In Kubernetes, a pod is something that can hold 1 or more containers. The reason for this is that containers within a pod can communicate with each other.

For the engineers: as an example, imagine I have a container for a web server and a container for a database in my pod. They can communicate with each other over localhost.

As for an appropriate analogy with the kitchen, I can’t really think of anything. Imagine some Sausage Party shenanigans where your anthropomorphized soup and salad start chatting to each other. But the soup and salad from the appetizer plate can’t talk to the steak and potatoes on the dinner plate because they are on different plates (aka, different pods don’t share the same network namespace and therefore can’t communicate with each other.)

Master Node
The head chef who manages and oversees the entire kitchen. This is that concept of “container orchestration” or “chef orchestration” we talked about earlier. Some real life examples of the orchestration work that this master node would do would be:

Scaling, aka adjusting up or down the number of running pods based on CPU utilization. In a busy kitchen, when there’s a surge in customer demand, chefs may need to scale up their operations by preparing more servings of a dish. By the way, one thing to note in this visualization - you’re probably imagining the kitchen hiring in new chefs, but I want you to imagine it more like the current chefs are getting cloned. When scaling happens, pods are essentially being copied.

Automated deployment, aka defining your application’s dependencies & runtime instructions in a YAML file so it can deploy based off this configuration. In a kitchen, this YAML file is analogous to a written recipe that tells the chef how to make the dish to ensure consistency and efficiency and preparing it.

Load balancing, aka distributing network traffic across different pods. In kitchen, load balancing involved assigning tasks to different chefs at a cooking station. Maybe Bob at the dessert station is overloaded with requests to scoop ice cream, so the master chef clones Bob and has Bob 2.0 take some of the ice cream orders off Bob 1.0’s hands.

Also important to note: each worker node has something called a "kubelet". In the kitchen scenario, "kubelet" would be analogous to the chef at each table. The chef has a bunch of jobs, like making sure the trays of food are being assembled correctly, helping prep ingredients, and throwing away the scraps. Similarly, "kubelet" does things like ensuring containers within pods are running, helps with initializing pods (e.g. have necessary dependencies installed), help with garbage collection, and more.

Some extra context for the engineers: Kubelet is an open-source, executable binary (aka file with machine code instructions that a CPU can directly execute), written in Go programming language.

Let's stop here for a second. If you understand everything I've said up to this point, you understand the basics of Kubernetes architecture! If you don't want to rely on the kitchen imagery forever, I've replaced all the kitchen drawings with only Kubernetes terminology in the diagram below.

Driver: Actually, all of this makes a lot of sense. So I get what Kubernetes is. But I still don't get why it's useful. Like what's the point of having it or learning about it?

Me: Yep, the last part of the puzzle is understanding how Kubernetes is used. How do humans interact with Kubernetes? How is Kubernetes relevant/useful in the world of technology? Let's lean back on our kitchen analogy for a second to explain some more concepts.

The owner of the restaurant/franchise is analogous to the software developers that are building out an app or service. At McDonalds, the franchise owner (Let's say their name is Francis Cockadoodledoo) wants get information on how much money each McDonalds location is making, and have the ability to fire/hire employees as needed. To do this, Francis Cockadoodledoo may pick up their phone and call the master chef to get information and give orders. In Kubernetes, software engineers can't really pick up a phone to interact with their Kubernetes cluster, but the “master node” has an API server that you can call, and this allows you access all tasks. For example, engineers can get information on all the pods, nodes, services, understand the health and metrics information, and have the ability to delete or create resources.
The customers eating at the restaurant is analogous to the users of an app or service. Similar to how the Mcdonalds kitchen produces Big Macs for me to eat, the Spotify Kubernetes cluster is providing me the service of listening to a large selection of music from a web browser.

I've factored this new information into the drawing I've included below. What you'll see is actually very similar to the diagrams you'll see when you google "Kubernetes architecture".

And of course, I realize there are some abstractions I left out of my explanation because I felt like they were not important in forming a basic mental model of Kubernetes. Feel free to dig in more. As I dig in more myself, I may add to this blog with some links to resources I find useful.

Conclusion

The reason why I picked this mode of storytelling (describing tech through the lens of a conversation with an Uber driver) is because I wanted to break down Kubernetes into something that was universally understandable and felt approachable.

Thank you for reading! If you have any constructive feedback for improving my writing (or my shitty drawings), please leave them in the comments.

Enjoy this picture of me and my coworker at #Kubecon 2023. We were there to promote the product I work on, BTW.

If you're interested, you should check it out. It's a modern take on the outdate terminal (command line) experience & makes you a better developer. Go to https://www.warp.dev/ to learn more.

Beginners Guide to ChatGPT API (Let's Build A Clone)

Jessica Wang — Fri, 20 Oct 2023 20:44:56 +0000

Building a personal ChatGPT CLI is super simple! Who knew you could have a whole conversation in the terminal!

To get started, you’ll need an OpenAI account and tokens to burn. That's it! We’ll cover:

Install and setup
Initialize the project
Build your ChatGPT clone

If you don't like reading, check out the Youtube video instead:

Install and Setup

First, let’s focus on setting everything up.

Note: I am going to assume you have Python and Home Brew installed, but if not, you can download them at https://python.org and https://brew.sh/.

Create a Python environment

Python environments can be confusing, so I’d strongly recommend you use Pyenv. It’s a tool for managing all your Python installations.

brew install pyenv

I’m doing this on a mac, but if you’re on a different OS like Windows or Linux, follow the instructions on Pyenv’s repo for your system before continuing.

curl https://pyenv.run | bash

It’s common in software development to create virtual environments to isolate the project and manage its dependencies. We’ll use Pyenv-evirutalenv to create this virtual environment in a moment.

For macOS:

brew install pyenv-virtualenv

For linux:

pip install virtualenv

For Windows:

virtualenv --python C:\Path\To\Python\python.exe venv

Install dependencies

Next, we’ll need several dependencies for the project. Run this install command to install each of these packages globally on your system:

pip install typer openai python-dotenv

Here’s a brief explanation for each package

Typer: a tool for writing command line applications in Python
OpenAI: handles all things OpenAI
Python-dotenv: a tool for reading and loading from .env files

Initialize the Project

Start by creating and navigating to an empty directory called “chatgptclone.”

mkdir chatgptclone
cd chatgptclone

Next, let’s create a virtual environment. Again, virtual environments isolate different Python installations and dependencies.

pyenv virtualenv 3.11 chatgptclone

This command generates a virtual environment called ChatGPTClone using Python 3.11. If you get an error, run pyenv install 3.11 to install Python 3.11 in Pyenv.

Now, let’s activate it!

pyenv activate chatgptclone

Build Your ChatGPT Clone

With everything installed and your Python environment activated, it's time to build. We’ll do this in three stages:

Stage 1: Create a basic version
Stage 2: Add memory to the program
Stage 3: Improve the CLI experience

Stage 1: Create a Basic Version

To start, we’ll need two files: main.py and .env.

1. Create a file in your directory called “main.py.”

At the top of the file, import all the dependencies we downloaded earlier along with the os package from Python. Your main.py file should look like this.

import os
import typer
import openai
from dotenv import load_dotenv

Note: os stands for “operating system” and is a built-in Python package that will help load the API key securely.

2. Create a .env file to hold the API key.

Copy the key that you get to your clipboard. In your .env file, type OPENAI_KEY="YOUR_API_KEY".

Note: Ensure you have credit balance (under Billing) to make API requests to OpenAI, since each request will require available credits.

3. Initialize the CLI application.

Next, set the API_KEY for openAI and create an application object called app.in the main.py file. Your code should look like this:

import os
import typer
import openai
from dotenv import load_dotenv

load_dotenv()

opanai.api_key = os.getenv("OPENAI_KEY")

app = type.Typer()

Awesome! We’re all set up, so let’s start making magic happen!

4. Create the chat function.

Create a function called interactive_chat and then welcome your users with the echo command from Typer recognizes commands by using decorators that go on top of a function. Then we just add the classic loop in Python at the bottom.

Here’s what we have so far:

import os
import typer
import openai
from dotenv import load_dotenv

load_dotenv()
openai_key = os.getenv("OPENAI_KEY")

app = typer.Typer()

@app.command()
def interactive_chat():
    """Interactive CLI tool to chat with ChatGPT."""
    typer.echo(
        "Starting interactive chat with ChatGPT. Type 'exit' to end the session."
    )

if __name__ == "__main__":
    app()

5. Add the chat interaction loop.

Since we’re building a chat application, we’ll need to add an infinite loop within the interactive_chat function that prompts the user for a chat input and calls the OpenAI ChatCompletion model.

def interactive_chat():
    """Interactive CLI tool to chat with ChatGPT."""
    typer.echo(
        "Starting interactive chat with ChatGPT. Type 'exit' to end the session."
    )

    while True:
        prompt = input("You: ")
        if prompt == "exit":
            typer.echo("ChatGPT: Goodbye!")
            break

            response = openai.ChatCompletion.create(
            model="gpt-3.5-turbo", messages=[{"role":"user", "content":prompt}]
        )

Note: if you get a billing error (like this), you may have to wait 5-10 minutes for the Billing information you just entered to be recognized by OpenAI.

On success, OpenAI will return a JSON object, which we’ll use to echo back the response message in the CLI application we’re building.

typer.echo(f'ChatGPT: {response["choices"][0]["message"]["content"]}')

Here’s the current interactive_chat function code:

def interactive_chat():
    """Interactive CLI tool to chat with ChatGPT."""
    typer.echo(
        "Starting interactive chat with ChatGPT. Type 'exit' to end the session."
    )

    while True:
        prompt = input("You: ")
        if prompt == "exit":
            typer.echo("ChatGPT: Goodbye!")
            break

            response = openai.ChatCompletion.create(
            model="gpt-3.5-turbo", messages=[{"role":"user", "content":prompt}]
        )

        typer.echo(f'ChatGPT: {response["choices"][0]["message"]["content"]}')

Run python main.py in your terminal and the chatbot will answer your questions!

Stage 2: Add memory to the program

While working, the chatbot currently has no memory of previous messages.

Currently, when calling the create function, we’re passing it an array of messages, which gives ChatGPT context. To give it all our context (and not only the current prompt) let’s make four changes:

Define an empty list called messages.
Append new prompts to the messages list.
Append the response from OpenAI to the messages list.
Send the entire messages list to the create function.

Here’s the current interactive_chat function code:

def interactive_chat():
    """Interactive CLI tool to chat with ChatGPT."""
    typer.echo(
        "Starting interactive chat with ChatGPT. Type 'exit' to end the session."
    )

    messages = []

    while True:
        prompt = input("You: ")
        messages.append({"role":"user", "content":prompt})

        if prompt == "exit":
            typer.echo("ChatGPT: Goodbye!")
            break

            response = openai.ChatCompletion.create(
            model="gpt-3.5-turbo", messages=messages
        )

        typer.echo(f'ChatGPT: {response["choices"][0]["message"]["content"]}')

            messages.append(response["choices"][0]["message"])

Run python main.py in your terminal and not your chatbot has a memory! Awesome!

Stage 3: Improve the CLI experience.

So, ChatGPT works exactly as we expect now. But… let’s make it even better.

Right now all of the OpenAI functionality is hard coded in the API call, so you’d have to go into the code to tweak the algorithm.

Add Customization parameters.

We can add a lot of customization as parameters in the CLI, like:

temperature (how creative ChatGPT is)
max_tokens (how large the responses will be)
model (which OpenAI model it should use)

You can set user parameters with the typer.Option. Let’s add all three of these parameters in the function definition along with help messages.

Here’s the current interactive_chat function definition code:

def interactive_chat(
    temperature: float = typer.Option(0.7, help="Control Randomness. Defaults to 0.7"),
    max_tokens: int = typer.Option(
        150, help="Control length of response. Defaults to 150"
    ),
    model: str = typer.Option(
        "gpt-3.5-turbo", help="Control the model to use. Defaults to gpt-3.5-turbo"
    ),
):

Running python main.py --help will return options for each parameter. Very professional!

To use these parameters, change the API call like this:

response = openai.ChatCompletion.create(
            model=model,
            messages=messages,
            max_tokens=max_tokens,
            temperature=temperature,
      )

You can now change any of these parameters when running the application.

python main.py --max-tokens 10

You should receive a response with only 10 tokens, meaning the max-tokens parameter worked!

Add your initial question on call.

Let’s make one final change. Let's add a text parameter that can let us ask our first message immediately with the --text or -t flag.

Here’s the updated interactive_chat function definition code:

def interactive_chat(
        text: Optional[str] = typer.Option(None, "--text", "-t", help="Start with text"),
    temperature: float = typer.Option(0.7, help="Control Randomness. Defaults to 0.7"),
    max_tokens: int = typer.Option(
        150, help="Control length of response. Defaults to 150"
    ),
    model: str = typer.Option(
        "gpt-3.5-turbo", help="Control the model to use. Defaults to gpt-3.5-turbo"
    ),
):

Now in the prompt logic, check if there is text that exists. That's it!

Final Code!

Just like that, we made a ChatGPT clone!

import os
import typer
import openai
from dotenv import load_dotenv
from typing import Optional

load_dotenv()

openai.api_key = os.getenv("OPENAI_KEY")

app = typer.Typer()

@app.command()
def interactive_chat(
    text: Optional[str] = typer.Option(None, "--text", "-t", help="Start with text"),
    temperature: float = typer.Option(0.7, help="Control Randomness. Defaults to 0.7"),
    max_tokens: int = typer.Option(
        150, help="Control length of response. Defaults to 150"
    ),
    model: str = typer.Option(
        "gpt-3.5-turbo", help="Control the model to use. Defaults to gpt-3.5-turbo"
    ),
):
    """Interactive CLI tool to chat with ChatGPT."""
    typer.echo(
        "Starting interactive chat with ChatGPT. Type 'exit' to end the session."
    )

    messages = []

    while True:
        if text:
            prompt = text
            text = None
        else:
            prompt = typer.prompt("You")

        messages.append({"role": "user", "content": prompt})
        if prompt == "exit":
            typer.echo("ChatGPT: Goodbye!")
            break

        response = openai.ChatCompletion.create(
            model=model,
            messages=messages,
            max_tokens=max_tokens,
            temperature=temperature,
        )

        typer.echo(f'ChatGPT: {response["choices"][0]["message"]["content"]}')
        messages.append(response["choices"][0]["message"])

if __name__ == "__main__":
    app()

Check out my source code if you have any questions.

Demystifying the Terminal: How It Works Behind the Scenes

Jessica Wang — Wed, 06 Sep 2023 21:03:41 +0000

If you've ever opened up a terminal and entered a command, you've witnessed the magic of computing in action. But have you ever wondered what happens under the hood to fulfill that seemingly simple command? The truth is, the inner workings of a terminal are surprisingly complex. In this article, we'll scratch the surface of how a terminal operates from a technical standpoint.

^ Youtube video if you prefer that. The visuals are a lot better here :-)

The Terminal's Historical Roots

Before we delve into the intricacies of modern terminals, let's take a trip back in time, over 60 years ago. The modern terminal we use today draws inspiration from teletype computers, which were designed to facilitate text-based communication through keyboards and printed output.

In the 1960s, many places with access to computing power relied on mainframe computers connected to teletype interfaces. Here's a simplified breakdown of how this technology functioned:

Every keypress transmitted a character over a wire.
The mainframe computer's kernel received and decoded this input.
The decoded text was sent to a component called the TTY driver, responsible for relaying user input to programs and collecting output.
Finally, the output was sent back to the user for display.

What made this system remarkable was the introduction of "line discipline," which buffered characters in memory until the Enter key was pressed. Over time, various optimizations, such as error correction and shortcuts, made modern terminals resemble their teletype predecessors.

Evolution of Terminals

In the 1970s, technology took a giant leap with the introduction of video terminals like the VT05 and VT100. These terminals supported ANSI escape codes, colors, sounds, and marked a revolution in computing. However, the core principle remained the same—input and output.

As technology progressed, computers ceased being mere tools to communicate with larger machines; they became ubiquitous in operating systems.

The Modern Terminal: An Application

Fast forward to today, and the terminals we use have evolved significantly. Computers no longer rely solely on text-based interactions. Instead, they handle a multitude of intricate processes simultaneously.

Modern terminals are now software applications, overseen by your chosen operating system. When you open a terminal, it listens for user input and instructs the operating system on what to display in the terminal window. Essentially, it emulates the hardware of the terminal.

Terminal Emulation

When you click on the terminal icon, behind the scenes, the terminal establishes a communication channel with your shell—the software responsible for executing commands. In the 1960s, this connection was a physical wire. Today, it's known as a pseudo-teletype or PTY, consisting of pairs of file descriptors that bridge communication between the terminal and running processes.

One end of the PTY is the "leader," used by the terminal to process user input and display output. The other end is the "follower," utilized by the shell and other tasks initiated by the terminal.

Spawning the Shell

After creating the necessary files, the terminal's critical task is spawning the shell—the command interpreter and the brains of the operation. Each time you enter a command, the shell runs it as a child process, with the terminal acting as the parent.

The terminal configures the child process to read and write from the PTY follower, establishing a seamless connection between user input and the shell.

Shell Initialization

Once the shell is born, it executes scripts to allow users to customize their experience. Depending on whether you're using a GUI-less server or a personal computer, the behavior may differ. Typically, the shell prints information like the username, time, hostname, and more, preparing itself to accept user input.

Running a Command

When you type a command, the characters are translated into Unicode characters, which are then processed by the shell's line editor. Special characters trigger specific functions, while non-special characters are passed to the line discipline.

Modern shells often disable the line discipline's canonical features, operating in raw mode. This allows the shell to handle all characters before submitting the command, enabling features like auto-completion, suggestions, and syntax highlighting.

Hitting Enter

Pressing Enter is when the real action happens. The shell parses the command by tokenizing and analyzing it. It checks various sources for validity, including aliases, functions, environment variables, built-in commands, and executable files in the PATH.

Once it identifies a valid source, such as an executable file, it locates it on the filesystem and runs it as a separate child process, passing any additional arguments.

Returning Output

When you run a command, like "ls," it spawns a child process that inherits the shell's file descriptors. All output from the command is directed to the PTY follower. The terminal emulator reads and displays the raw bytes streamed from the child process.

For the colorful text you see in modern terminals, escape sequences play a crucial role. These special codes embedded in text output instruct the terminal on how to render text, including colors, cursor movements, and more.

Closing the Terminal

When you close your terminal window, several steps occur behind the scenes:

The shell performs cleanup, such as saving command history and running logout scripts.
Child processes spawned by the shell are terminated.
Resources like open file descriptors are released.
The shell process exits and signals the terminal emulator to close.
The operating system cleans up any remaining resources.
The terminal window is fully closed, and your session is logged out and cleaned up.

All of this happens in a matter of seconds, showcasing the complexity hidden behind what seems like a simple command line.

Conclusion

The terminal, despite its apparent simplicity, remains the most straightforward way to communicate with a computer. Its evolution from teletypes to modern terminal emulators reflects the enduring power and adaptability of this technology.

If you want to try out a modern terminal that will boost your productivity, check out Warp.

Build Your Own CLI App in 5 Minutes (ClackJS + Node + NPM Tutorial)

Jessica Wang — Sat, 05 Aug 2023 21:57:48 +0000

Introduction

Today, I’m going to show you how to build your own command line application and deploy it as a NPM package. I will be using npm, Node.js, and Clack to build this.

You can even try it out for yourself right now by running:

npx cli-trivia-game

Here's a sneak peek of the trivia game (full .gif of play-through + answers at the bottom):

Getting Started

1) Navigate to your command line and run:

npm init -y

This is going to initialize a new project, and the -y flag automatically answers “yes” to all the prompts. This generates a package.json file that has all the default settings without having to manually provide details.

2) Next, you will run:

npm -install @clack/prompts

Once you do that, you’ll have a framework that lets you create things like spinners, multi-selection, and self-validated cancel logic all with just a few lines of code.

3) Let’s leverage Clack’s built-in setup logic. To create an intro, I’m going to call intro here and pass in this string that let’s the player know they’re about to play in a trivia quiz that tests their CLI knowledge.

4) Since this is a trivia quiz, let’s ask some questions. First, I’ll create a class called Question that requires a question string, an array of potential answers, and the index of the correct answer.

5) Then, I’m going to create 10 question objects using this class template.

6) Now that we’ve got our questions, how do we display them on the command line? We’re going to use Clack’s select component to do this. All I have to do is pass in the question, initial value, and options, and voila.

7) To make it feel a little more sophisticated, I’m going to add a spinner after the user answers just so they have time to process the UI and makes the program feel intelligent.

8) And then to tie up, I’m going to declare a global variable called totalCorrect which will increment when the player answers a question correctly.

9) At the end, I’ll add some logic to display different outros based on how many questions the player got correct.

And the best part - we’re going to publish this as an npm package so that every developer in the world can download on their machines if they choose to. It’s actually pretty easy.

1) Go to package.json and add “bin”:”./index.js”

2) Run npm login

3) Run npm publish

4) Then run npx cli-trivia-game

3 Min Youtube version (if you prefer that) 👇

Done! But before you start this project for yourself, take into consideration that Clack is really lacking in the documentation. Anything fairly complex, you won’t benefit from the pre-made components of clack/prompts. Here's a list of other CLI frameworks you can use instead.

Check out the full code for the project here:
https://github.com/theRubberDuckiee/cli-trivia-game

Conclusion

Just like that, you have made and published your first CLI application! Congrats :)

And, if you'd like to know the answers to all 10 questions in my trivia game, here you go:

The Battle For Perfomance In The Terminal

Jessica Wang — Tue, 09 May 2023 18:09:19 +0000

Grids - two-dimensional arrays of characters - are the universal building blocks of terminals. The basic operations we expect from our terminals - entering a command, receiving output, scrolling through a file - are, at their core, operations on grids. This piece is an attempt at explaining the terminal from the bottom up, starting from the grid. The goal is to spell out some of the not-so-obvious performance calculus behind terminal grids: what are the operations being optimized and at what cost?

If a terminal session is really some conversation history between the terminal and the commands it has executed or is currently executing, then it’s the grid that provides some agreed-upon format or ledger to converse. It’s not just that grids are a convenient choice for terminal builders; the grid is actually what’s required by the program the terminal is layered over, the shell. The shell moves a cursor and outputs characters within a defined two-dimensional space, and the terminal is responsible for storing and rendering the contents of that space to the end-user. This makes the concept of the grid a – if not the – fundamental engineering specification of terminals.

(If you’re curious to learn more about the dependencies between terminals and shells, I highly recommend this blog post.)

In Warp, the grid is especially important because it’s everywhere. Warp separates commands into blocks, and each block consists of three grids: one for the prompt, one for the command input, and one for the command output. Even opening full-screen programs like vim and emacs are themselves instances of this same grid, just with expanded dimensions.

Here, a command in Warp is the combination of three grids receiving and sending characters to the shell. (Technical note: Warp also offers a default prompt instead of a customized shell prompt).

Because the grid is the means of storage and communication, it is at this layer that terminals make meaningful trade-offs around performance. For example, imagine you’re running a server that endlessly prints out log lines. When the log lines fill up the available grid space on the screen, how many new rows should Warp allocate for the grid? 1? 100? 1000? Allocating 1000 could buy us time before the command runs out of space again, saving us the performance penalty of repeated memory allocations, but of course the command might not use the full 1000. Should we risk a slightly greater memory footprint for an uncertain promise of faster execution?

Even more interesting are the trade-offs the grid must make between the most basic terminal operations. Some examples of these operations are reading in data from the shell, scrolling through content, and resizing the screen or splitting panes. Warp’s implementation of the grid is itself an opinionated stance on which of these operations a terminal should be most efficient at, with the reality being that optimizing for one almost always means not optimizing for another. Should we be excellent at one thing if it means being just okay at something else? The grid is the layer at which this type of performance calculus is baked in and shipped to our users.

Getting technical

Warp’s grid is a circular buffer that stores rows sequentially in a vector. Say, for example, you create a grid storing a subset of Shakespeare plays that are sorted alphabetically. How this renders on screen and how this is stored internally might look something like this:

On screen, the contents are rendered alphabetically.

In storage (the underlying vector), the contents remain sequential but may be rotated. Notice the top row in the grid is stored at vector index 2, the second row is stored at vector index 3, and so on.

The conversion between the screen index and the vector index is a really, really important one. When the grid renders to the screen, we’ll want to locate the contents of screen index 0 and then 1 and then 2 and so on. To optimize for this, the grid stores two integers that allow us to efficiently translate between screen space and vector space.

bottom_row: the vector index of the last row in the grid, aka the furthest the grid extends. In the example, bottom_row = 1 (Tempest is the last row in the grid).
length: the total number of rows that we’ve received from the shell, which may be different from the actual capacity of the vector at any given moment. In the example, length = 5.

Using these indices, we can convert from screen space to vector space like this:

Nevertheless, the whole process of index conversions presupposes that the grid needs to be rotated in the first place. Why is that the case? Why not skip the silly conversions and enforce equivalence between the screen indices and the vector indices, meaning that “As You Like It,” the first row on screen, is also stored as the first row in the vector? Not surprisingly, this is because the grid is heavily optimizing for something else entirely: scrolling!

Scrolling the grid

In Warp, scrolling in a command like vim, emacs, git log, git diff, or many others is optimized to avoid the ultimate price in the grid world – having to move physical memory. Inspired from the previous section, imagine an example of scrolling the contents of the Shakespeare file upwards to reveal a new row at the bottom of the screen:

What would this look like at the grid layer? To start off, the underlying vector is unrotated. It looks like this, with bottom_row = 8 and the conceptual “top row” = 0.

The first step to scrolling up is to extend bottom_row by one while keeping length the same. When we wrap bottom_row, it becomes 0 and top row becomes 1.

From there, we overwrite the contents of the bottom row with whatever the shell feeds to Warp as the next line in the file – in this case, “Tempest”. After that, the scroll operation is finished and the resulting grid looks like this:

What’s noteworthy is that Warp completes this scroll operation without moving any data in the vector, besides overwriting the exiting row with the entering row. The scroll operation succeeded solely by changing the metadata of the grid, specifically the index of the bottom_row. The result is an extremely efficient scroll operation at the cost of leaving the grid in a rotated state.

One of the penalties of the now-rotated grid is the added complexity of index conversion that we discussed above. Broadly speaking, this penalty is relatively minor and is more than compensated for by the efficiency we gain during scrolling. Where this becomes a little bit more harrowing as a performance calculation is when we actually receive output from the shell and find ourselves needing an extra row. At that time, having a grid that’s rotated leaves us no choice but to move memory.

Extending the grid

A lot of terminal commands print content to the screen rapidly and require the size of the grid to grow dynamically.

Here, the terminal reads content from the shell (via the receiving end of a pipe known as the PTY, short for pseudo-TTY) and writes it to a grid. This is a fundamental operation for the terminal: how efficiently can we read streamed input from an open file descriptor and write it character-by-character to the specified grid location?

Specifically, the problem becomes interesting in the common case where the shell issues a newline – this is the shell warning the terminal that the command requires more than the current size of the grid, implying the vertical grid dimension needs to be expanded by one row. From the terminal’s perspective, let’s consider this request under the worst-case grid configuration: the underlying vector is already at capacity and the grid is rotated (again, index on screen != index in vector).

Capacity: if the vector has no more free memory, then we cannot avoid a vector resize cost. Vectors are heap-allocated data types, and resizing requires (1) finding a heap chunk that’s big enough for the new capacity and (2) copying the old contents into the new memory space.
Rotation: if the grid is currently rotated, then we must “re-zero” the vector before increasing its size. This implies costly memory management: we reverse the rotation until the conceptual “top row” is back to vector index 0. Only once the vector is unrotated can we tag new rows onto the end.

Here, we cannot extend the vector until the existing contents are unrotated. Otherwise, we’d conceptually be tagging on new rows to somewhere that’s not the end of the logical grid - that would be a mess! This “re-zero” is a costly copying operation.

Because the penalty under the worst-case grid configuration is very high, Warp does its best to make this circumstance as infrequent as possible. As an example, we only allocate new rows in chunks of 1000, opting to make the resize slightly more heavyweight with the benefit of amortizing the cost to every 1000 newlines.

Resizing a grid

A final, essential piece to this weaving narrative about grid optimizations and their externalities is the process of resizing a grid. Importantly, grids in Warp are always stored at the dimensions of the pane in which they render, which are not necessarily equivalent to the dimensions at which we received them from the shell. Said differently, we always rewrite the grid so any horizontal limit is imposed at the time of resizing and not at the time of rendering. This makes more sense with a visual:

If the shell originally outputs a grid that is 10 rows by 10 columns and the user proceeds to resize the screen (or split a pane) that makes the grid 5 columns, then internally the vector is re-written to account for this new horizontal constraint: the grid becomes 20 rows by 5 columns both on screen and in storage. Today, every resize in Warp requires re-writing every grid currently displayed on the screen from the top down, row by row.

We permit this considerable performance penalty – for now – because, although resizing is an essential operation, it’s hard designing optimizations that won’t slow down everything else: doing index conversions, scrolling, adding a new line to the grid, etc. From a holistic standpoint about overall performance priorities in the app, it’s also not clear that resizing should take precedence over any of those things.

Conclusion

All told, an intelligent, intentional design of the terminal grid should recognize that oftentimes the most important operations are in competition with one another. Every block in Warp is a careful calculus about where to excel and how to minimize the resulting cost: that the rotated vector streamlines scrolling but costs us the amortized “re-zero” every 1000 lines, that allocating new rows in chunks means we’re willing to risk some bloat for fewer memory allocations, that storing the grid in its renderable dimensions is optimal but leaves us vulnerable at resize, to name but a few.

If you have opinions on what our performance priorities should be – or any insights about the trade-offs discussed here – we’d love to hear from you! Better yet, if you find this type of problem interesting, come work with us!

How I Write Easy CLI Scripts to Automate My Life.

Jessica Wang — Wed, 12 Apr 2023 07:00:52 +0000

I take hundreds of screenshots a day, and I always forget to delete them. They end up taking up a lot of memory on my machine, and sometimes I can't just bulk highlight & delete in fear of deleting important pictures & videos that I actually need.

Here's how I wrote a script to automate this for me.

If you don't like reading, here's a video tutorial you can watch:

Before Getting Started

I want to preface this by saying this tutorial heavily hinges on me using Warp AI, which is basically a ChatGPT integration built into the terminal that quickly generates answers/code snippets to any question.

If you want to follow along, you'll need to be using the Warp terminal app which you can download for free here.

In the next section, I'll be putting the prompt (what I asked Warp AI) and what Warp AI's output was. I basically just followed those instructions to write my script in under 5 minutes.

A Summary Of My Process

So my process for writing this script was just:

Asking Warp AI how to do something.
Copying the output it gave me (code blocks + commands to run).
Checking to see if it worked.
Asking Warp AI about any errors/getting it to tweak its output.

In the section below, I'll include the entire transcript of my interaction with Warp AI as I was building my script. This probably took me less than 5 minutes. Here's a summary if you don't want to read the whole transcript.

1) Ask Warp AI how to write a script that deletes all screenshots from my Documents directory.

2) Copying the output, but realizing it looks for everything named "screenshots.png" whereas my screenshots were named "Screen Shot XYZ.png"

3) Asking Warp AI to tweak its output to look for the phrase Screen Shot and also be case insensitive.

4) Getting base functionality working, and then improving on it by getting the script to print out the names of files it was deleting.

5) Asking Warp AI to explain some of the outputs so I can understand what my script is doing & learn in the process.

Warp AI Transcripts (Ok To Skip to Conclusion)

Prompt: Write me a script that deletes all screenshots from my Documents directory

Warp AI: Sure! Here's a script that will delete all screenshots from your Documents directory:

#!/bin/bash

cd ~/Documents
rm -f *screenshot*.png

Save this script as a file with a .sh extension (e.g. delete_screenshots.sh) and make it executable with the following command:

chmod +x delete_screenshots.sh

Then, you can run the script in the terminal with the following command:

./delete_screenshots.sh

This will change to your Documents directory and delete all files with "screenshot" in the filename and a ".png" extension. Be careful when running this script, as it will permanently delete these files without prompting for confirmation.

Prompt: do I have to run chmod +x command every time I edit the .sh file?

Warp AI: No, you don't have to run the chmod +x command every time you edit the .sh file. The chmod +x command only needs to be run once to make the script executable.

However, if you make changes to the script and save the file, you will need to run the script again to see the effects of the changes. If you have already made the script executable with chmod +x, you can simply run the script again with ./script_name.sh to execute the updated version of the script.

If you make changes to the script and want to see the effects of the changes without having to run the chmod +x command again, you can use the following command to reload the script in the current shell session:

source script_name.sh

This will reload the script and apply any changes you have made to it, without requiring you to make the script executable again.

Prompt: Can you edit the script above to delete anything with file name that includes "Screen Shot" and is case insensitive?

Warp AI: Yes, here's an updated script that will delete all files with "Screen Shot" in the filename (case insensitive) from your Documents directory:

#!/bin/bash

cd ~/Documents
shopt -s nocaseglob
rm -f *Screen\\ Shot*

The shopt -s nocaseglob command enables case-insensitive globbing, which means that the *Screen\\ Shot* pattern will match filenames with "Screen Shot" in any combination of upper and lower case letters.

Save this script as a file with a .sh extension (e.g. delete_screen_shots.sh) and make it executable with the following command:

chmod +x delete_screen_shots.sh

Then, you can run the script in the terminal with the following command:

./delete_screen_shots.sh

This will change to your Documents directory and delete all files with "Screen Shot" in the filename (case insensitive). Be careful when running this script, as it will permanently delete these files without prompting for confirmation.

Prompt: can you edit this script above to output the names of all the screenshots it deleted, each on their own line?

Warp AI: Yes, here's an updated script that will delete all files with "Screen Shot" in the filename (case insensitive) from your Documents directory and output the names of the deleted files:

#!/bin/bash

cd ~/Documents
shopt -s nocaseglob
deleted_files=(*Screen\\ Shot*)
rm -f "${deleted_files[@]}"
printf "Deleted files:\\n%s\\n" "${deleted_files[@]}"

The deleted_files=(*Screen\\ Shot*) line creates an array of all files with "Screen Shot" in the filename (case insensitive), and the rm -f "${deleted_files[@]}" line deletes all of these files. Finally, the printf command outputs the names of the deleted files.

Save this script as a file with a .sh extension (e.g. delete_screen_shots.sh) and make it executable with the following command:

chmod +x delete_screen_shots.sh

Then, you can run the script in the terminal with the following command:

./delete_screen_shots.sh

This will change to your Documents directory, delete all files with "Screen Shot" in the filename (case insensitive), and output the names of the deleted files. Be careful when running this script, as it will permanently delete these files without prompting for confirmation.

Prompt: What does the \n%s\n do?

Warp AI: The \\n characters in the printf command are escape sequences that represent a newline character, which causes the output to be formatted with each deleted filename on a separate line.

The %s format specifier in the printf command is a placeholder for a string value, which is replaced by the values of the deleted_files array. The @ symbol after the array name tells printf to expand the array into separate arguments, with each element of the array being treated as a separate string value.

So, the printf command with the %s specifier and the @ symbol expands the deleted_files array into separate string arguments, which are then printed on separate lines with a newline character before and after each string.

Conclusion

And in the end, I now have a working script that I run pretty easily from the command line.

It's crazy that I can so easily write bash scripts without knowing anything about CLI scripting.

Warp AI is OP.

Become a VIM PRO in 2023 without having to memorize ANYTHING (ChatGPT + Warp + MacOS)

Jessica Wang — Sun, 02 Apr 2023 05:39:17 +0000

I think we can all agree that Vim is pretty intimidating and not super intuitive - especially because you need to memorize so many keyboard shortcuts just to move around.

In this blog, I’m going to show you how I became a Vim PRO without having to memorize anything. Let’s get started!

So why do people use vim?

Here are a few reasons:

1) Vim is the default editor on any operating system in the world.

2) Vim is super fast and responsive.

3) Lastly, I think there are some developers that just enjoy the challenge of mastering Vim.

How I mastered Vim without memorizing anything

I’m using my Warp terminal on my MacOS. You’re going to need the Warp terminal to follow along. To install it, run

brew install warp

in your CLI or download here.

For the purposes of this demo, I’ve created this .txt file that you can think of as a little game loosely inspired by British TV-show Taskmaster.

Each line has a task that I need to follow, and we’re going to complete all these tasks using Vim keyboard shortcuts in the MOST EFFICIENT WAY POSSIBLE.

To open up Warp AI, press

^ + Space

and then as you try to complete these tasks, ask Warp AI the bullet point questions to learn the corresponding Vim keyboard shortcuts.

1) Remove the period from the end of this line.
I could technically just press right to get to the end of the line, but that’s not efficient!

“How do I navigate to the end of the line in vim”
“How do I edit in vim”

2) Go to the last line and fill in the blank, then come back up to this line.
I could technically just press down to get to the end of the file, but that’s not efficient!

“How do I navigate 3 lines down”
“How do I navigate by word”
“How do I delete a word”
“How do I navigate 3 lines up”

3) Delete this entire line.
I could navigate to the end of the line and just press backspace, but that’s not efficient!

“How do I delete an entire line”

4) Find and replace all instances of the word “line” with “COOKIE”
I could do this manually, but that’s not efficient!

“How do I find and replace all instances of a word”

5) Please like and _____ to this channel for more tech and terminal tips!

Conclusion

And that’s it! Hopefully this helped.