Forem: cieplik

What’s so special about PS1? Fun with customizing Bash command prompts

cieplik — Thu, 17 Mar 2022 05:13:44 +0000

Long ago, when I was still a console newbie, I copied my friend’s bash configuration file. It had all the necessary stuff already included - aliases, colors, and most importantly: a nice prompt setup. I used it on all machines I had access to due to all the extra context it provided. For example, it would turn red when I was on a production machine, show me a current git branch from the repository I was working on, and indicate whether I had any changes to commit by showing a star.

Today there are a seemingly endless set of tools for configuring the command prompt to your liking, but back then things like Starship were completely non-existent. Most developers I know have customized their prompts in one way or another. This fact, combined with the feedback we’ve received from the Warp community, was good motivation to dig deeper into prompt customization in Warp.

In this post I’m going to tell you how we implemented support for PS1 in Warp, and why adding it was technically challenging. Along the way, we’ll also tackle the DCS (device control string) and some fun shell tricks!

Hey, Cieplik, what actually is PS1?

Glad you asked! PS1 is one of the few variables used by the shell to generate the prompt. As explained in the bash manual, PS1 represents the primary prompt string (hence the “PS”) - which is what you see most of the time before typing a new command in your terminal. On top of that there are actually few more - from PS0 to PS4, each of them executed and printed in different contexts of command execution. For example, you’ll see PS2 whenever the command has multiple lines as a secondary prompt string. And then zsh also provides RPS1, which displays the Prompt String on the right-hand side… there's a lot to work with.

Each of the Prompt String variables can easily be customized with the backslash-escaped special characters, but they can also contain shell function calls or even emojis, since most modern terminals support unicode.

With Warp in the works we made a decision to provide useful defaults for users to help them hit the ground running when using a new app. This included the prompt. We explicitly decided to not support PS1 in Warp, as it could clash with the bootstrapping script and PRECMD/PREEXEC hooks we use to create blocks of inputs and outputs. By implementing our own default prompt we had much more control over the user experience, and that was critical at the early stages. Eventually, however, it became obvious that our users really need their custom setups and lack of certain information on the prompt may actually slow them down. It came time for us to honor the user’s PS1 settings. Keep in mind that this is not our last word on this topic

What’s under the Prompt String’s hood?

In Warp, because we buffer the user’s input, we can’t allow the shell to directly render the prompt. Instead we use the metadata from the shell to do our own rendering. Our app receives it from the shell via the PRECMD hook, to do things like retrieve the current git branch, which is used in Warp’s default prompt. We use a special escape sequence and a JSON string packed with data to pass all the necessary information.

This is why we decided that the best way to support PS1 in Warp is to pass it as part of our PRECMD hook. The first challenge, however, comes from printing the variable into this JSON string.

Your PS1 is usually a set of variables, like information about the colors you want to see, but sometimes it can be a function call too. We have to translate it into a rendered Prompt String - a set of escape sequences and characters that tells the terminal the exact string to print and how to print it.

Here’s the example of an oh-my-zsh PS1 setting

> echo $PS1
%F{magenta}%n%f at %F{yellow}%m%f in %B%F{green}%~%f%b$(git_prompt_info)$(ruby_prompt_info)
$(virtualenv_info) $(prompt_char)

In zsh rendering a prompt is actually quite simple. You just print it. (and the shell expands all parameters for you via prompt parameter expansion.) We insert the output into our JSON and display it on our terminal.

Note that the middle prompt is the output of the print command, prepended with "rendered prompt" for clarity.

Bash does have a similar way of expanding prompt parameters as zsh does, so we could easily run echo ${PS1@P} in our script to get the rendered prompt and call it a day! Except…

That functionality has only been introduced in bash 4.4+. But Macs do not ship with that by default (this is related to a licensing change: since version 4.0, bash switched to a GPL license, which is not supported by Apple). The question, then, is how do we render the PS1 in older versions of bash without using any special tools or libraries?

The way we ended up solving this issue is by invoking a subshell, executing a very simple command in it, capturing the entire subshell output (including the prompt), and then manipulating it to only capture the prompt itself.

This is the final code we’re currently using in Warp to get the value of the rendered bash prompt:

$(echo -e "\n"  | PS1="$WARP_PS1" bash --norc -i 2>&1 | head -2 | tail -1)

Let me break that down, pipe by pipe:

echo -e "\n" prints an empty line in our subshell;
PS1="WARP_PS1" bash --norc -i 2>&1 we pass a previously captured $WARP_PS1 value as the PS1 in the subshell, and specify that no configuration files should be loaded (norc argument) to improve the performance of this operation; the -i flag denotes an interactive shell, while 2>&1 redirects the stderr to stdout, which allows us to capture the rendered prompt;
Head & tail operations simply help us manipulate the output to extract the prompt only.

That’s it. We've got our rendered prompt, we can start showing it in Warp!

Escaping the Prompt

Some things and places are really hard to escape, like the Russian city of Omsk, where even stray dogs cannot leave. Escaping in the terminal realm is really close to that experience, and Prompt String seems to make it even harder.

When juggling shell data and passing it to Warp via our PRECMD hook, we escape escape sequences that may break our JSON string with our magic sed invocation:

sed -E 's/(["\])/\\1/g; s/''\t''/\t/g; s/''\r''/\r/g; /\n/'

This is actually a series of replacements:

All double-quotes and backslashes are replaced with their escaped versions (\” and \);
\b (backspace), \t (tab), \f (form feed), \r (carriage return) are all replaced with their escaped versions;
We add an escaped \n (new line) on every line explicitly, since sed analyzes the input line-by-line and thus is not aware of the actual new lines.

This worked great, yet with the Prompt String you actually get many more things to escape: it’s not uncommon for the rendered prompt to include some extra bells and whistles (I mean the actual bell character - \a) and lots of other non-printable characters and escape sequences (most significant being \x1b aka \e or \033, which is literally ESC). Note that those sequences also had to be double escaped, to not only create a valid JSON string, but also not break our Rust implementation when unescaped! At least we were able to use the same tool for both supported shells this time!

In the first iteration, the final sed call looked a lot like:

sed -E 's/'$'\e''/\\e/g; s/'$'\a''/\\a/g;  s/(["\\])/\\\1/g; s/'$'\b''/\\b/g; s/'$'\t''/\\t/g; s/'$'\f''/\\f/g; s/'$'\r''/\\r/g; $!s/$/\\n/'

Back to the whiteboard

At this point, we started testing internally. It was then that we found a mysterious behavior.

A common tool used to customize the prompt in zsh - oh-my-zsh - is used by many team members. The prompt rendering worked for every theme...except the default oh-my-zsh theme. When the theme was used, the entire bootstrapping script would fail, leaving Warp in an unusable state.

Robby Russell’s oh-my-zsh prompt theme

It turned out the little arrow (➜) to the left was the root cause!

With shells and terminal emulators it always makes sense to analyze the actual bytes that are being processed. Let's check what’s under the hood of this little arrow:

This particular emoji consists of 4 bytes: e2, 9e, 9c and 0a. A careful reader may notice that both 9e and 9c bytes come from the extended ASCII table, but do they carry any special meaning? When in doubt, it’s always good to go back to the source - in this case, check the shell parser documentation. For completeness, let's quickly unpack what VT100 (and other VTs) is, and what this mysterious DCS is that are both mentioned in the linked docs:

VT100 was a Video Terminal introduced in the 70s. It was one of the first machines that allowed for cursor control with ANSI-escape codes, and added a bunch of other control options. Later on its spec became a de facto standard, and modern terminal emulator programs try to follow it (including Warp).

Control characters and control sequences in the terminal world are special ANSI-escape characters that control the terminal’s behavior. It can be anything from the cursor position, mouse control, even colors. DCS (device control string) is a special control sequence that is followed by a data string. You can find out more here.

unhook

When a device control string is terminated by ST, CAN, SUB or ESC, this action calls the previously selected handler function with an “end of data” parameter. This allows the handler to finish neatly.

As it turned out - the 9c byte carried a special meaning in a terminal world - it’s an ST (string terminator) escape sequence, which also happens to denote the unhook operation. As a result, the emoji in our JSON string would prematurely end the PRECMD hook, making it impossible for Warp to fully start.

From there the solution was simple - rather than trying to escape emojis and unicode characters that may have similar issues in the future, we simply encoded the entire Prompt String as a hexadecimal string. This completely eliminated the need for using sed and simplified our shell script.

Below you'll find a snippet with our current precmd function (example in bash):

   warp_escape_ps1 () {
           tr '\n\n' ' ' <<< "$*" | xxd -p | tr -d '\n'
        }

                # Format a string value according to JSON syntax - Adapted from https://github.com/lework/script.
        warp_escape_json () {
            # Explanation of the sed replacements (each command is separated by a `;`):
            # s/(["\\])/\\\1/g - Replace all double-quote (") and backslash (\) characters with the escaped versions (\" and \\)
            # s/\b/\\b/g - Replace all backspace characters with \b
            # s/\t/\\t/g - Replace all tab characters with \t
            # s/\f/\\f/g - Replace all form feed characters with \f
            # s/\r/\\r/g - Replace all carriage return characters with \r
            # $!s/$/\\n/ - On every line except the last, insert the \n escape at the end of the line
            #              Note: sed acts line-by-line, so it doesn't see the literal newline characters to replace
            #
            # tr -d '\n' - Remove the literal newlines from the final output
            #
            # Additional note: In a shell script between single quotes ('), no escape sequences are interpreted.
            # To work around that and insert the literal values into the regular expressions, we stop the single-quote,
            # then add the literal using ANSI-C syntax ($'\t'), then start a new single-quote. That is the meaning
            # behind the various `'$'\b''` blocks in the command. All of these separate strings are then concatenated
            # together to form the full argument to send to sed.
            sed -E 's/(["\\])/\\\1/g; s/'$'\b''/\\b/g; s/'$'\t''/\\t/g; s/'$'\f''/\\f/g; s/'$'\r''/\\r/g; $!s/$/\\n/' <<<"$*" | tr -d '\n'
        }

        warp_precmd () {
            # $? is relative to the process so we MUST check this first
            # or else the exit code will correspond to the commands
            # executed within this block instead of the actual last
            # command that was run.
            local exit_code=$?
            # Clear the prompt again before the command is rendered as it could
            # have been reset by the user's bashrc or by setting the variable
            # on the command line.
            if [[ -n $PS1 ]]; then
              WARP_PS1="$PS1"
            fi
            unset PS1
            unset PROMPT
            # Escaped PS1 variable
            local escaped_ps1
            if [[ $WARP_FEATURE_FLAG_HONOR_PS1 == "1" ]]; then
              # Tricking the shell into rendering the prompt
              # Note that in more modern versions of bash we could use ${PS1@P} to achieve the same,
              # but macOs comes by default with a much older version of bash, and we want to be compatible.
              deref_ps1=$(echo -e "\n"  | PS1="$WARP_PS1" bash --norc -i 2>&1 | head -2 | tail -1)
              escaped_ps1=$(warp_escape_ps1 "$(echo "$deref_ps1")")
            fi
            # Flush history
            history -a
            # Reset the custom kill-whole-line binding as the user's bashrc (which is sourced after bashrc_warp)
            # could have added another bind. This won't have any user-impact because these shortcuts are only run
            # in the context of the bash editor, which isn't displayed in Warp.
            bind -r '"\C-p"'
            bind "\C-p":kill-whole-line
            local escaped_pwd
            escaped_pwd=$(warp_escape_json "$PWD")
            local escaped_virtual_env=""
            if [ ! -z "$VIRTUAL_ENV" ]; then
                escaped_virtual_env=$(warp_escape_json "$VIRTUAL_ENV")
            fi
            local escaped_conda_env=""
            if [ ! -z "$CONDA_DEFAULT_ENV" ]; then
                escaped_conda_env=$(warp_escape_json "$CONDA_DEFAULT_ENV")
            fi
            local git_branch
            git_branch=$(git rev-parse --abbrev-ref HEAD 2>/dev/null || echo "")
            local escaped_git_branch
            escaped_git_branch=$(warp_escape_json "$git_branch")
            # At this point, escaped prompt looks something like
            # \\u{001B}\\u{005B}\\u{0030}\\u{0031}\\u{003B} ...
            # We need to maintain the double quoting of \\u in the message that
            # is sent otherwise the receiving side will interpret the value
            # as JS string literals of the form \uHEX, and will include
            # ctrl characters (like ESC) in the json, which will cause a JSON
            # parse error.
            # Note WARP_SESSION_ID doesn't need to be escaped since it's a number
            local escaped_json="{\"hook\": \"Precmd\", \"value\": {\"pwd\": \"$escaped_pwd\", \"ps1\": \"$escaped_ps1\", \"git_branch\": \"$escaped_git_branch\", \"virtual_env\": \"$escaped_virtual_env\", \"conda_env\": \"$escaped_conda_env\", \"exit_code\": $exit_code, \"session_id\": $WARP_SESSION_ID}}"
            warp_send_message "$escaped_json"
        }

Future of the prompt

Warp now has the option to honor the user's PS1 setting. It works with most configurations, though you’ll find the full compatibility table in our docs.

It is not, however, the last time you'll hear from us about working on the prompt. Our key product principles include fixing the UI and providing a great out-of-the box experience. One of the ideas we're currently exploring is Context Chips - our low-calorie approach to snacking on bite-size information.

We’re working on plenty of interesting problems just like this one at Warp; and we’re hiring! If you want to be a part of our team and work together on improving the day-to-day life of developers from around the world - check out our careers page.

Who’s behind Warp? - Meet the team

cieplik — Thu, 17 Mar 2022 05:10:57 +0000

My idea for a perfect opening line of this blog post was to use a quote about the importance of building a great team. I scrolled through pages of quotes from personas such as Henry Ford, or Michael Jordan, eventually settling on… nothing. We all know that teamwork is critical and a huge part of engineering work, no need to back this up by famous people’s quotes! It becomes even more important in small companies and startups, such as Warp—where everyone works together to deliver the best terminal experience to other developers. In the last post we uncovered a little bit about how we think about building the product; in today’s, I’ll introduce you to people behind Warp and the story of how we built the new terminal!

Why did I join Warp?

Let me start by telling you my part of this story.

As a person who rides a 700lbs motorcycle that doesn’t have a reverse (try parking that thing!) and a digital clock is its most advanced piece of technology; who writes her code in Vim and still mixes all the ingredients for the cake manually—I clearly like to... do things in a more traditional way. No wonder the terminal has been my main tool ever since I started working as an engineer.

I was initially very skeptical about Warp—after all bash scripts, grep,sedandawk` can already get you a pretty long way when working with the terminal, right? I was still curious about what the company has to offer, and that’s why I joined one of the first user research sessions. I even borrowed a MacOS laptop specifically for this!

This was the first time I met the entire Warp team and got to try the application. Learning about the roadmap (especially the team features that will improve efficiency and safety of devops work!) and seeing how user-focused the team is, was what got me excited about the product and the company. 2 months ago I quit my job as an Infrastructure Software Engineer in Dropbox, and joined Warp—the company that builds a next-gen Rust-based terminal.

Why did Zach decide to build a terminal?

Zach, whom you might’ve seen on Twitter or a recent webinar (talking about Warp, obviously), is the CEO of Warp. He previously worked as a Principal Engineer at Google, CTO at Time and co-founded the venture-backed startup SelfMade, yet his interests circle around improving the developer’s experience.

Zach Lloyd, Warp's CEO

There’re plenty of tools around that improve the development process in one way or another—focusing on a broad group of engineers, or on a very specific niche. However, only 2 are truly used by almost every programmer there is—they are IDEs and code editors or a terminal. The former already went a long way—we’ve had Eclipse and JetBrains, or IntelliJ; there was notepad and Atom and Sublime; and nowadays VSCode seems to be the majority’s favorite IDE and code editing tool.

On the other hand, the terminal stayed the same for almost 40 years (and still counting)! It’s slowly evolving with extra functionality coming from new shells (like fish) or prompt-tools (like starship.rs). Yet, the main experience has stayed the same, and modern terminals still emulate the physical ones, implementing the subset of functionality from VT100 (hence the correct term is actually “terminal emulator”). Lack of innovation in this area is what made Zach take a closer look at building a terminal. And that’s how Warp was born.

People behind Warp

Soon, Shikhiu, Michelle, and Aloke joined Zach in the effort of working on Warp. The initial prototype was an Electron app, though the poor performance pushed the team to pivot and rewrite the application in Rust. Besides making Warp an OS-native, and blazingly fast application which utilizes GPU rendering, Rust is a language with a strong community support and tools that make it easy to ramp up new developers (Rustlings, we’re looking at you!). As a result the team (with support from Nathan Sobo—the Atom editor creator) created a Rust UI framework which we plan to open source in the near future.

Michelle used to work as an engineering intern in Facebook, Slack, and Robinhood. She also pursued an entrepreneurial path co-founding the first health tech incubator in Yale and creating her own health tech app during her college years. Viral growth and user feedback loops are clearly her thing, however, not the only thing. Michelle has a black belt in taekwondo; loves watching movies and analyzing them for hours afterwards; and recently she started to learn how to drive (on the streets of New York City!).

The only person who did not pick up Rust (just yet), is Shikhiu. His excuse is being the Founding Designer. As a master of Figma, Shikhiu creates mockups of new Warp features in minutes, helping the team visualize the work and iterate faster. He recently told me he’s into plants, which would explain why the “leafy” terminal theme (soon to ship in Warp) is so beautiful. He used to work at Adobe and later on at Google where he led a team of 30+ designers on design for the Docs suite, yet (luckily for us) he chose to join us to reimagine the terminal product experience!

Work in progress of themes in Warp - with "leafy" being one of the concepts!

Working as a TL for Google Docs, Aloke had already collaborated with Shikhiu during his time in Google. He also came across multiple TODOs left in the Google Docs codebase by the mysterious zachlloyd, but hadn't had a chance to work with Zach prior to joining Warp. Before his interview with Warp, Aloke dropped a very sharp knife on his foot, but decided to tough it up and pass the interview with a bleeding foot. This definitely showed how dedicated he is, though we are all glad that he saw a doctor and his foot is now fully healed… Besides cooking as a hobby, Aloke lifts (not just our spirits) and is very much into public transportation. He gets triggered by lack of periods at the end of the comment, since that makes it an incomplete sentence—I like to think it was the reason he started working on completions in Warp.

Currently, Warp consists of 7 engineers with Zheng, Chuck, Kevin and me joining over the course of the last 6 months.

Zheng, Chuck, Kevin & me all wearing our Warp's T-Shirts

Chuck plays a critical role in the team since he’s the only person with prior Rust experience. Besides being a mentor, who leaves helpful and constructive feedback in our PRs, Chuck collects Pez candy dispensers and can solve a Rubik’s cube in under 1 minute. One of the things that drew him to Warp is being able to improve developers’ experience first hand—he used to work in the devtools team in LinkedIn and maintains Volta tool manager. Working on the terminal was a logical next step.

After a few years of working at Dropbox, Zheng decided to dive into the startup world. First, she worked for Gem, and later on met Zach and the team. She deeply cares about the work culture, so her joining Warp says a lot about the company. She likes to travel, pet cute animals and drink good wine; and also she likes to dig really deep into why and how our users use the terminal and other tools day to day. That’s why her current main focus is driving the tmux-like experience in Warp.

Kevin is our most recent hire. He joined as a full-time engineer only a month ago, but has been working part-time at Warp for the past year. He spent his college years traveling to different countries while continuing the education remotely and recently graduated. As part of his international adventures, he interned in a Japanese company in Tokyo for 2 summers, and thanks to that polished not only his engineering skills, but also learned to speak the language! During his internships he tried himself in different fields, including data science and ML, and figured that he’s really interested in developing tools for other engineers, wanted to work on something that will have a direct impact on users and was pretty much a blank canvas. Warp matched his interests perfectly.

Over the summer, we also worked with Saumya & Matt—summer interns who delivered functionality such as notification on long running commands and custom theming. Big thanks to both of you!

Working at Warp

The team works remotely and is spread across different time zones (though we recently opened an optional office space in NYC!). The company was founded during the peak of COVID-19 pandemic, so it couldn’t have been different anyways.

We meet daily to discuss progress and show demos of new features, and organize more Zoom meetings whenever necessary. As a group of very opinionated people (who in the tech industry isn’t?), we sometimes tend to stay online longer, discussing new ideas, making sure that everybody has a space to express their point of view. Somehow we still manage to make the majority of the decisions unanimously - unless it’s about the fate of the ctrl-r keybinding… In those moments of doubt, Blue’s (Zach’s dog) appearance in the background cheers everyone up, especially when he tries to eat our CEO’s dinner. Altersnatively, Backup (my dog) sometimes comes to the standup and adds his voice to the pool of our opinions.

When it comes to figuring out the road map for Warp, we try not to rely on our personal views only. Hence, the closed beta program and user focus groups - both helping us learn what’s the right set of features to build next. Zach wrote about our 8 product principles in a separate post, and you can read more about the team’s culture and approach on his blog: thezbook.com.

And most importantly—if you feel like our culture suits you and you’re passionate about bringing the developers’ experience to the next level—we are hiring and would love to work with you!