Forem: Zaerald

Using Vim As Your Shell Command-Line Scratch

Zaerald — Wed, 20 Apr 2022 15:02:38 +0000

Using (n)vim as your command-line scratch helps you to efficiently fix, create, and run ad-hoc commands.

If you are not familiar with vim, I suggest you start playing with the vimtutor first. I’ll try my best to explain each keys that I have pressed. In this blog, I’m using Neovim, but you can still follow along even if you’re using Vim. I have aliased my nvim to vim with alias vim=nvim. Please follow along with the demonstrations, so you can get used to this workflow.

Fixing Ad-Hoc Commands

Have you experienced running a command and realizing that you made a mistake after executing it? You missed some syntax or forgot adding a sudo, and much more?

Commands that requires privilege

You are copying a file to another directory, or installing a package, but you forgot that it requires a sudo privilege.

I - insert mode before the first character
Escape or Ctrl + C - to go to the normal mode
ZZ or :wq- to save and exit

Note that in vim, keys are case-sensitive, this means that the lowercase i is different to the capital I.

As you have noticed, we were able to edit our previously executed command with fix command (fc), it opened the vim editor for us with the command as our content. If we exit from the editor, it’ll run the command that we have edited.

For this case, it is better to use sudo-bang-bang (sudo !!) if you need to add sudo before the previous command.

!! - will use your previously executed command

If we view the man page of fc. It checks for the FCEDIT and EDITOR environment variables. If both of them are unassigned, it’ll use vi.

-e ename
...
If ename is not given, the value of the FCEDIT variable is used,
and the value of EDITOR if FCEDIT is not set.
If neither variable is set, vi is used.
When editing is complete, the edited commands are echoed and executed.

I have exported my EDITOR variable in my config file to use neovim.

export EDITOR=nvim

You can assign nano, or code here to use VS Code as your fc editor, but I do not recommend it. You’ll see why it’s better to use vim later on.

Commands with typo

Another use case is if you made a typo in the middle of the command, or missed any quotation marks.

Arrow Keys Spam

When we try to fix our command, we will spam the thing out of the left arrow key to navigate to the middle of the command to fix it. This workflow is so slow, and even annoying to deal with.

fc to the rescue!

Invoke the fc command to fix the previous command that you have executed. This will open a vim editor which we can easily perform search and replace commands. This would be more useful if we have really long commands and arguments.

:s/ooo/o - perform a substitute ooo to o

:s/hir/here - perform a substitute hir to here

For more information about the vim substitute command, please check the help file with :h :s.

Commands with shell expansion

Let’s say you want to say hello to the current USER.

Hmmmmm 🤔, our USER is not printing, but it has value as we have checked it. We have found out that we were using single quotes ('), and this means that shell expansion never happened, what we need is to surround them with double quotes (").

We can still use :s/'/" here, but we can also leverage the power of vim with plugins!

I have navigated to the text to replace the surrounding quotes, then used tpope/vim-surround plugin.

cs'" - it means change surrounding single quote (') to double quote (").
                c      s                         '                   "

If you want to delete a quote, you can use ds', or ds" for double quotes.

Commands that you want to edit without executing

We have learned that we can use fc to edit or fix our previous command. What if you noticed a typo, and you do not want to execute the command? In that case, what we can do is press ctrl + a to move the cursor before the first character in your command line and type pound (#), this results into making the command as a comment.

If the command with pound (#) in front is executed, it won’t execute the command itself, but instead it’ll result into being a shell comment. The good thing with this is you can use fc to open it in your editor and edit it. You can do this because the commented-out command was already added to your command history, and fc will edit the previously executed command. Just remove the # on your edit.

Commands with long parameters

If we want to run a postgres container with a command from their documentation, we will run the following command:

docker run -d --name some-postgres -e POSTGRES_PASSWORD=mysecretpassword -e PGDATA=/var/lib/postgresql/data/pgdata -v /custom/mount:/var/lib/postgresql/data postgres

After we have pasted this in our terminal, it’s hard for us to update the parameter variables. We will use so many arrow key presses just to navigate to the environment variable values and edit it. We can comment this for now and run the fc command. By doing this, we can leverage the vim editor to manipulate the text.

x - delete the # symbol

:s/some-/my- - substitute the container name

/secretciwmyverysecret
  - '/secret' search and navigate to the first 'secret' word match
  - 'ciw' delete the inner word, and go into insert mode
  - 'myverysecret' literal word to replace the password value

Add new lines before the -e and -v

:s/ -[ev]/\r\0/g
  - ':s/' substitute command
  - ' -[ev]' search for text that has ' -e' or ' -v'
  - '/\r' add a carriage return
  - '\0' substitute with the first captured group
  - '/g' substitute globally, not just the first match

With this substitute, we were able to format our command, but there is a problem, in shell with multiple lines, we need to suffix every line with a backslash (\). We can achieve that with performing a normal command.

:%norm A \
  - ':%norm' perform a normal command for the whole buffer
  - 'A' go to insert mode to the end of the line
  - ' \' add literal space and backslash

The normal command helps you to perform it on multiple lines as if you’re actually typing it directly on the buffer, to learn more about it, check the help file for normal using :h norm (norm command is just a shorter version of normal command).

We were now able to update the values of a command with multiple long parameters, and also properly formatted it. If you noticed that my line numbers are not linearly increasing, I’m using a relative numbers, you can learn more about it with :h relativenumber. I highly recommend setting set rnu config together with set nu so you can easily jump to arbitrary lines relative to your current line.

Commands that you are currently fixing and want to abort it

If you have edited your previous command and you do not want to execute it. If you use ZQ or :wq! to exit vim without saving, it’ll still execute your command.

What happened? It turns out that when the editor exits with 0, it’ll execute your command, and using :wq! gracefully exits vim.

echo $?
0

How to solve this? You need to use :cq to exit vim with error code. You can add a key bind for this, or you can just delete the content of your vim buffer with dd and exit.

Vim Buffers and Shell

Fixing your previous command is cool, but what if you want to perform a command operation to the output of your previous command, and update its content? You can use Vim to achieve that.

If we perform an operation and pipe it to our Vim buffer, it’ll use the stdin as your buffer content.

echo hello | nvim -
# look ma, no dash
echo hello | nvim

There are times that you need to use dash (-) to pipe the standard output to standard input of the next command. In nvim, the dash is not required.

Sort and remove duplicate names

We have a file with a list of names, but there are duplicates. We want to remove the duplicates, and sort them.

cat names.txt
John Doe
John Doe
David William
Joseph Thomas
David William
Joseph Thomas
Joseph Thomas

We can pipe the output of the cat names.txt, or just directly open it with vim.

vim names.txt

Then we can now perform a command operation to update the content.

:%!sort -u
David William
John Doe
Joseph Thomas

Anything after the ! character is the shell command.

Of course, we do not need to open it in vim to remove the duplicates, it is to simply demonstrate that you can perform commands on the current buffer and replace it with the output of the shell command.

Sort and remove duplicate names of some part of the buffer

We can also specify some area that we want to perform the operation using the visual mode of vim.

cat names2.txt
John Doe
John Doe
David William
Joseph Thomas
David William
Joseph Thomas
Joseph Thomas
# please do not modify the following lines
Christopher Daniel
Mark Anthony
Christopher Daniel
Paul Steven
Mark Anthony
Paul Steven

First, we need to select the range with the visual line (V), and run the :!sort -u command.

Note that the '<,'> characters are automatically added by vim, this represents the selected line range.

Formatting and filtering JSON content

cat characters.json
[ {
"id": 1,
          "name": "Tomoko Kuroki",
    "age": 15,
"height": "147 cm"
  }, {
    "id": 2,
          "name": "Marin Kitagawa",
    "age": 15,
"height": "164 cm" },
  {
          "id": 3,
"name": "Kaguya Shinomiya", "age": 17,
                "height": null
  } ]

This is a JSON file with a content that is not properly formatted. We also want to remove some noise, we only want the id, and the name properties of the characters. We can do this by using the jq command. We can also use any vim plugins like prettier for different file types.

We have formatted it with :%!jq .

:%!jq .
[
  {
    "id": 1,
    "name": "Tomoko Kuroki",
    "age": 15,
    "height": "147 cm"
  },
  {
    "id": 2,
    "name": "Marin Kitagawa",
    "age": 15,
    "height": "164 cm"
  },
  {
    "id": 3,
    "name": "Kaguya Shinomiya",
    "age": 17,
    "height": null
  }
]

Filtering the properties that we only want with :%!jq '[.[] | {id, name}]'. This is really helpful to analyze some JSON data, and want to play with the properties.

:%!jq '[.[] | {id, name}]'
[
  {
    "id": 1,
    "name": "Tomoko Kuroki"
  },
  {
    "id": 2,
    "name": "Marin Kitagawa"
  },
  {
    "id": 3,
    "name": "Kaguya Shinomiya"
  }
]

Pressing u to undo the changes to the buffer. You can learn more about jq from their manual.

We were able to easily parse the JSON without leaving our editor!

Searching file contents

You can also pipe the output of your grep to vim, so you can easily navigate to the matched files. If the cursor is under a file path, and pressing gf will open another buffer with the file and the cursor is on the line number that was created in our grep result.

grep -Hnri 'search keyword here' . | nvim
H - print file name
n - add line number
r - recursive search for each directory
i - ignore case

Now that we have the matched searches in our buffer with their associated file names and line numbers, we can use gf here to navigate to the file under the cursor. This can help you preview and analyze the files, you can get back to the previous buffer with ctrl + 6. See :h buffers for more information.

Writing to Shell

Can we do it the other way around? Using the content of your buffer to execute them in your shell?

One common misconception when you run vim <file>, is that your’e editing the actual file, but in fact, you are actually opening a buffer of that file, and not editing the file itself.

This is different to the other editors where they will immediately create the file for you.

When you save it with :w, you are just overwriting the file with your current buffer, and if the file does not exist, it’ll create it. This is equivalent to :w % or :w yourfile.txt

With this in mind, what if you write the buffer to your shell? Well...you guessed it right (pun not intended 😉), if you write the buffer to your shell, you can execute commands from your Vim buffer!

I am using zsh here, but any shell should work.

If we write the buffer to a shell, it’ll invoke the command to a sub-shell.

:w !zsh
:w !$SHELL

cat file-that-contains-shell-script.txt

echo 'i have echo'
echo $(pwd)
touch newfile.txt

We can also pick which line to execute.

:.w !zsh

Notice the dot(.) before w, this means “write the current line of the cursor to the shell”.

cat sample-line.txt
echo hello1
echo hello2
echo hello3
echo hello4
echo hello5

Shell Level

If we print out the SHLVL environment variable, we are currently at level 1. Invoking this command inside vim will result into the level 2. This means that it is executing it to the sub-shell.

Bindings Configuration

Typing :w !zsh or :w !$SHELL again and again is too much, I recommend adding your own bindings similar to what I have

" write to shell
let mapleader=" "
nnoremap <silent> <leader>xs :.w !$SHELL<CR>
xnoremap <silent> <leader>xs :.w !$SHELL<CR>
nnoremap <silent> <leader>xS :w !$SHELL<CR>

You can put this into your .vimrc file or the init.vim for neovim. Then you can just press the leader key which is set to space and xs.

File Analysis

From our previously found files in the “Searching file contents” section, we decided that we want to delete those files. We need to filter out the file paths first, and there are multiple ways to achieve this. We can use cut here, but for now, let’s use awk with a colon separator.

:%!awk -F: '{print $1}'
./subfolder0/somefile.txt
./subfolder0/anotherfile.txt
./subfolder1/alpha.txt
./subfolder3/beta.txt
./subfolder4/magic.txt
./sample-line.txt
./sample-line.txt
./sample-line.txt
./sample-line.txt
./sample-line.txt

There are duplicate files, let’s perform a unique sort and prepare to delete with rm.

:%!sort -u
./sample-line.txt
./subfolder0/anotherfile.txt
./subfolder0/somefile.txt
./subfolder1/alpha.txt
./subfolder3/beta.txt
./subfolder4/magic.txt

Now that we have added the rm command on each line, and writing them to your shell will remove the files.

Shell Intellisense

We can improve our workflow by adding intellisense using the Language Server Protocol (LSP). LSP is a protocol that can help provide some auto completion, quick documentation, go to definition, and many other features. You can use any LSP provider, at the time of this writing, I’m using coc.nvim for my LSP.

Setting up your own LSP and their configuration is out of scope for this blog, you can check their documentation for it.

In order for our LSP to work, there are multiple ways to enable it, but the easiest one I can think of is to set the file type of the buffer that we are currently working on. You can check the current file type with set filetype?, and assign a file type using set filetype=sh. Now that we have an sh filetype, our LSP now knows that we are editing a shell buffer, which will provide us the language features of a shell.

Command Auto-Completion and Documentation

Path Auto-Completion

This is just the tip of LSP, and you can check the language server of what features it can offer for you to help in your shell scratch workflow.

Other Use Cases

In this section, I will share some of my workflow that I use day to day.

Workflows in Git

Long Branch Names

Whenever we create new features, or fixing a bug. It’s better to create new branch from the main or any base branch. We usually create branches using the ticket name feat/ZERO-1234, but browsing these branches does not provide any context as to what is the purpose of that branch, unless you memorized the ticket number and their description.

We can add more details to the branch name, but the problem with this branch names is that the checkouts, rebase, and renames is a bit of pain to type, unless you have auto complete. You can pipe the branch output to vim, and perform the checkouts.

feat/ZERO-123-create-some-cool-feature-here-that-no-one-usees
fix/ZERO-456-the-bug-that-will-consume-your-days

The aliases that you have configured in your Git will work here like git config --global alias.co checkout.

Deleting multiple branches

Deleting multiple branches with grep is a bit of a hassle, unless you run the grep first to see the list of the branches. What we can do is pipe the output to vim, and perform any searches.

By doing this, you have a faster feedback of what branches would be deleted, you can even easily remove any branch that was part of the initial grep.

Reading Command Outputs

APIs mostly use JSON as their payload. We can easily create them using jo. We can read the command output and put it to your current buffer. For example, we want to create a JSON object with a lower case uuid value for its id property, and a simple name.

:read !jo id=$(uuidgen | tr A-Z a-z) name='zaerald'

Anything after the ! will run a shell command.

Conclusion

You can now efficiently edit previously executed commands using fc. Using vim with plugins and LSP can help improve your workflow. Sky is the limit! Using vim as your command-line scratch is much fun to use. Just play with it, and don’t forget to have fun!

You can check my dotfiles in github.com/zaerald/dotfiles. Please note that my dotfiles continues to evolve, and there’s a higher chance that it is different now compared at the time of this writing.

This blog post is originally posted here.

How to improve Java Unit Testing Skills to a more advanced level?

Zaerald — Mon, 31 Aug 2020 17:05:45 +0000

What are the efficient ways to improve and learn more about Java unit testing? Is there a repository where I can practice Unit Testing with ways how to solve it, so I can compare it with mine? How did you practice doing it? As sometimes when I'm doing stuff I get stuck and not able to test the other parts.

How to Hack Learning Curves for Slow Learners Like Me

Zaerald — Mon, 26 Aug 2019 17:38:27 +0000

Before I would discuss the techniques I use, I'll start giving a backstory of when I was before I discovered these techniques, when I was studying something I couldn't get it and I always get stuck! It is so frustrating that would make me feel so dumb that I'll start to believe that it is not for me, I started to think that maybe programming is not for me. It always reminded me that I'm no genius because whenever I browse the internet and look at how others do things and look easy for them, that's the time when I wonder that is there a way to learn how to learn?

I tried to search for things on how to learn and what I had discovered changed my life and I will share it with you. You should keep in mind that learning is emotional, so you have to believe that you have the ability to learn, and you do, as I had said earlier it is bad to think that the one thing you're learning is not for you when we are a kid, and we couldn't walk or run properly, we didn't say that "running is not for me", we always believed, failed at some times and we did. Brains are like muscles that when stretched they'll break and repair themselves and be stronger, the brain also keeps changing and creates stronger neuron connections based on what we do with it, we just need to utilize it and exercise it.

Techniques

Here is the list of techniques I used to learn efficiently. Please be mindful that not all of these techniques could apply to you, some may work and some may not, you could try and combine these techniques when you're trying to learn.

First-principles

Whenever you encounter a new topic, break it down into its first principles--deconstruct it as each topic is just a composition of smaller topics or concepts that you can boil down to something that you do know. After breaking things down look for the essentials, see if some of it could be postponed for you to focus your energy towards those mostly or commonly used. I create mind maps so the things I break down are linked to their parents, so I can look up to their relations and associations which helps me to understand the overall idea of the topic.

I use coggle for my digital mind maps, but I mostly use pen and paper to create my mind maps. You can learn more about mind mapping by watching the talk by Hazel Wagner, Ph.D..

Transfer Learning

Now that you managed to break down the topic into smaller ones, you can then find patterns that smaller sets that can be used to other skills or can be used to a different topic. For example, you're trying to learn a programming language like Java and as you have broken its concept and as you see the language has control structures that have if-else statements, and you'll soon realize that when you learned it you can use it and transfer the learning to many programming languages, with that in mind, it is very helpful for you to study and master it.

With this, you'll learn faster because you know that it is a part of it and helps you not to be overwhelmed by it. Be comfortable being uncomfortable, when you're trying to learn your brain receives a pain signal to your neurons, and you would try to avoid learning, this is sometimes the reason why we procrastinate as it makes it hard to learn. Just let it sink in and sooner or later the pain will ease.

Brain Modes

Have you ever experienced it when you're taking a shower or just walking after you worked on some topic and left it for a while and knowing the answer? Or when you're working on a bug, and you decided to sleep and when you're sleeping you'll know the fix or even dream about the fix!?

A study conducted by Prof. Barbara Oakley mentions that our brains have two modes: They are focused mode and diffused mode. This is different research on the left brain or right brain. The focus mode is the mode wherein you're working on learning the topic or when you're in the zone, the other mode is very different because this mode is making your brain at rest and let your mind wander.

While taking that hot shower, you would find out that the topics now make more sense, or you'll know the resolution to the problem. That's because the diffuse mode is at work, these modes cannot work in parallel and diffuse mode can only work effectively after you have focused on the topic first, so the brain can manage those things you try to put into your brain.

Pair programming also works effectively because of this, the driver is in the focused mode while the observer is in diffused mode, this is sometimes the observer can see the mistakes of the driver because the driver is in the focused mode and not seeing the bigger picture, he is so focused that sometimes he couldn't see the other solutions. This means that when you're trying to solve an unsolvable problem, try to solve it away from your workstation and just let your mind wander.

Learn more about focused and diffused mode.

Feynman technique

This technique is developed by Richard Feynman which is named to himself, the technique is just to study the topic and try to teach it and when you can't teach some part of it, you'll have to go back to study the topic again. I use this when I just learned a new topic I try to share the learnings with others which means that I have to teach it to them too.

This may also refer to Einstein's quotation "If you can't explain it simply, you don't understand it well enough". The statement could also be reversed and when you don't understand it enough you couldn't explain it simply. Try to explain it to someone or your rubber duck when you're studying, with this you're helping yourself to learn, and also you're helping others to learn too, I call this process a peer to peer (p2p) learning as there's free-flowing information through your network.

The reason why this works is that whenever you're thinking, your brain short-circuits in order for you to think fast, this was also referred to as System 1 on Thinking, Fast and Slow by Daniel Kahneman that is why when you try to use a language to convey it, your brain needs to create a structure of the way how you think and your brain can only create the structure when you get a good deeper understanding of the topic.

Habits

Learning is a skill and when learning it won't happen overnight you need to develop it, you have to make it a habit to learn. To create a habit, you need to know its steps. A habit consists of 3 steps, a cue, routine, and reward. Cue is a trigger that tells your brain to go to the habit. Routine is the action itself and the Reward makes the particular routine worth remembering because your brain releases dopamine which makes you feel motivated and do it again. Try it when you have finished learning something to find your reward, as for me my rewards are playing games or just mindlessly surfing the internet.

You could also create your if-then statements to create your cue, add the if statement to your existing routine. For example, if I opened my laptop then I'll open my digital notes and if I opened my notes then I'll read what I had studied yesterday to review it. My other cue is if I finished drinking my coffee then I'll start learning (this was maybe the reason why I like coffee, it helps me to learn 😁). Try to develop a habit to learn 25 minutes a day, it is better to learn 25 minutes a day for a week compared to 175 minutes in just a day. Because your brain has to create the building blocks, imagine it as if you're creating the foundation of the house, it needs time to strengthen its foundation for you to build on top of it and when you try to cram it in a day, you wouldn't have a stronger foundation.

ELI5

Another thing I use is to create analogies, I mostly use Explain Like I'm 5 to try to explain it simply and create analogies that a 5-year-old can understand, that's also the same as whenever I try to learn something and I get stuck, I'll try to look up the term I wanted to understand and add the "ELI5" to it and see how others explain it to a 5-year-old. Recently I'm trying to learn about Kubernetes (K8s) and looked up "Kubernetes eli5" on Google and has a result of ELI5: What is Kubernetes..., this helped me to understand its concepts more. But there's a danger when creating analogies, as there are some topics that are not a complete analogy so just be aware of that.

Spaced Repetition

After you have studied the topics, you have to repeat it, as the neurons in your brain only remember the things that are repeated over and over to them, the reason why your brain forgets things because your brain tries to remove the things that you do not need, so you have to tell your brain that you need to learn the topic by repeating it over time.

Spaced repetition is a learning technique that helps you review the topics at gradually increasing intervals, what does it mean? It means that when you try to learn a topic try to recall it again after 3 days and when you still know it tries it again after 6 days and if you noticed that you forget some of it you have to review it again and review it in 3 days and so on. I use Anki for this to help me manage the spaced repetitions of the topic. Also, one thing to improve retention is to write it, use pen and paper! Even if we are in the digital age. This will also help you to have an overview of the written notes or mind maps.

Recognition vs Recall

There are also times that you think that you already know about the topic and mastered it. This is the feeling when you're watching some tutorial and feels like you're learning it but when you try it, you're lost, this is the recognition pitfall, as you're just recognizing it based on what you see you're not actually recalling it, recalling is when you have the capacity to self-correct and by recalling your brain tries to create a connection and improves it, the more you recall it the more the connection strengthens and improves, which makes it easier for you to pick it up when you needed that fragment of information, make sure to try it and do it on your own for you to recall it when you needed it.

Exercise!!!

I know this seems weird right? I use this technique to improve my capability to learn, as the exercise increases levels of the important neurotransmitters (serotonin, norepinephrine, and dopamine) that is responsible in how you process your thoughts and emotions that affect how you think, you'll notice that when you're stressed or in anger it is harder to focus and think properly. With exercise, it'll help you to feel energized and motivated this will help you to learn faster, and when you noticed that you can't focus, try to walk around your block, do some jumping jacks or even sprint just to move your muscles that could help produce energy levels that travel to your bloodstream and into your brain.

On Dr. Terrence Sejnowski interview on how he learns, he uses exercise to learn more which also helps him take advantage of the diffuse mode thinking.

Closing notes

Lastly, to learn how to learn you have to apply some techniques if not all, try to use one or a couple of the tips/techniques first and integrate another one, try each of everything and see if it works for you, as we do have different solutions. Or you could discover or even develop your own! Enjoy the process, because if you don't enjoy the process, you'll get disappointed, sometimes you have to just take a break if you get stuck. As stated I am no genius and by no means learning fast, and I still have many knowledge gaps, I'm still learning, keep learning, and don't stop learning. GLHF!

I recommend you to take the course of Learn how to learn by Dr. Barbara Oakley.