Forem: Sebastian Witowski

Add Comments To Your Commands

Sebastian Witowski — Thu, 20 Oct 2022 00:00:00 +0000

A quick tip: when you write an important command in shell, put a comment next to it so you can easily find it later or remember what it does:

pytest -m slow --durations=10 # Run slow tests, return the slowest

Why?

Source: https://www.commitstrip.com/en/2017/02/28/definitely-not-lazy/

I rely heavily on searching commands in history. Something like the guy above, but I usually press ctrl+r instead of ↑.

The problem with this approach is that sometimes it's not easy to figure out how to find a specific command. You have to know at least part of the command, which might be difficult if you copied it from somewhere and immediately forgot about it. Or maybe the part that you remember is too generic. If I want to rerun pytest with a specific set of parameters, I know it starts with pytest, but that doesn't really narrow it down.

So I started annotating commands I might want to reuse in the future. Let's say I want to run pytest on all the tests marked as "slow" and return the slowest ones. I can put a comment right next to the command:

pytest -m slow --durations=10 # Run slow tests, return the slowest

This will make it much easier to find this command in the future. It works especially well in combination with a fuzzy finder like the fzf:

A real-world example

I often use this trick in IPython when I want to find some example data in the system. In one of the projects I worked on, we had a database full of objects with numeric IDs. If I wanted to grab an object, I could either:

Search in the UI for the ID of an object I want to retrieve from the DB. This takes time. I don't want to fire up the UI and do all the filtering to get the ID. So usually, I use the second approach.
Grab a random object from the DB and hope it has the type I'm looking for. If not, grab a different one.

Sure, the second approach might take much longer than the first approach. But just like the guy pressing the arrow up 50 times, I kept thinking that "the next object has to be the right one."

I solved that problem by adding comments - each time I grabbed a specific object from the DB, I wrote a comment on what's that:

read_object("/Trade/98182379") # Trade X from "Portfolio A"
read_object("/Trade/18293712") # Trade Y from "Portfolio B"

That way, the next time I wanted to grab an object from Portfolio A, I just needed to hit ctrl+r and type "Portfolio A".

25 IPython Tips for Your Next Advent of Code

Sebastian Witowski — Wed, 27 Jan 2021 00:00:00 +0000

I’ve decided to skip last year’s Advent of Code edition. Mostly because I didn’t have time, but I also knew that I probably wouldn’t finish it. I’ve never finished any edition. I’m not very good at code katas, and I usually try to brute force them. With AoC, that works for the first ten days, but then the challenges start to get more and more complicated, and adding the @jit decorator to speed up my ugly Python code can only get me so far.

But one thing that helped me a lot with the previous editions was to use IPython. Solving those problems incrementally is what actually makes it fun. You start by hard-coding the simple example that comes with each task. Then you try to find a solution for this small-scale problem. You try different things, you wrangle with the input data, and after each step, you see the output, so you know if you are getting closer to solving it or not. Once you manage to solve the simple case, you load the actual input data, and you run it just to find out that there were a few corner cases that you missed. It wouldn’t be fun if I had to use a compiled language and write a full program to see the first results.

This year, instead of doing the “Advent of Code,” I’ve decided to do an “Advent of IPython” on Twitter - for 25 days, I’ve shared tips that can help you when you’re solving problems like AoC using IPython. Here is a recap of what you can do.

1. Display the documentation

In [1]: import re

In [2]: re.findall?
Signature: re.findall(pattern, string, flags=0)
Docstring:
Return a list of all non-overlapping matches in the string.

If one or more capturing groups are present in the pattern, return
a list of groups; this will be a list of tuples if the pattern
has more than one group.

Empty matches are included in the result.
File: ~/.pyenv/versions/3.9.0/lib/python3.9/re.py
Type: function

That’s one of my favorite features. You can display the documentation of any function, module, and variable by adding the “?” at the beginning or at the end of it. It’s called “dynamic object introspection,” and I love it because I don’t have to leave the terminal to get the documentation. You can use the built-in help() function to get this information with the standard Python REPL, but I find the “?” much more readable. It highlights the most important information like the signature and the docstring, and it comes with colors (even though you can’t see them here because my syntax highlighting library doesn’t support IPython)

2. Display the source code

In [1]: import pandas

In [2]: pandas.DataFrame??

Init signature:
pandas.DataFrame(
    data=None,
    index: Optional[Collection] = None,
    columns: Optional[Collection] = None,
    dtype: Union[ForwardRef('ExtensionDtype'), str, numpy.dtype, Type[Union[str, float, int, complex, bool]], NoneType] = None,
    copy: bool = False,
)
Source:
class DataFrame(NDFrame):
    """
    Two-dimensional, size-mutable, potentially heterogeneous tabular data.

    Data structure also contains labeled axes (rows and columns).
    Arithmetic operations align on both row and column labels. Can be
    thought of as a dict-like container for Series objects. The primary
    pandas data structure.

    Parameters
    ----------

... and so on

And if you want to see the full source code of a function (or class/module), use two question marks instead (function_name?? or ??function_name).

3. %edit magic function

Sorry for the embedded tweets, but dev.to doesn't support longer GIFs (at least I didn't find a way to embed them including hosting them from my website, giphy, or imgur).

Sebastian Witowski

@sebawitowski

3. If you want to write a longer function, use "%edit" command.
IPython will open a temporary file in your favorite editor (you need to set it with the EDITOR environment variable), and when you save and close it, it will execute that code.

09:59 AM - 03 Dec 2020

I use it with vim, and it works great when I want to write a bit longer function (with vim I have a lightweight linter, and moving around the code is faster). It’s a nice middle ground when you are too lazy to switch to your code editor to write the whole code, but at the same time, the function that you are writing is a bit too big to write it comfortably in IPython.

4. Reopen last file with “%edit -p”

Sebastian Witowski

@sebawitowski

4. Speaking of the %edit command, you can run "%edit -p" to reopen the same file that you edited last time. This is great to incrementally build some bigger code block or to quickly fix a mistake you made last time.

08:43 AM - 04 Dec 2020

5. Wildcard search

In [1]: import os

In [2]: os.*dir*?
os. __dir__
os.chdir
os.curdir
os.fchdir
os.listdir
os.makedirs
os.mkdir
os.pardir
os.removedirs
os.rmdir
os.scandir
os.supports_dir_fd

In [3]: os.chdir("/some/other/dir")

If you forget the name of some function, you can combine the dynamic object introspection (the “?”) and a wildcard (the “*”) to perform a wildcard search. For example, I know that the os module has a function to change the current directory, but I don’t remember its name. I can list all the functions from the os module, but I’m sure that a function like this must contain “dir” in its name. So I can limit the search and list all the functions from the os module that contain “dir” in their names.

6. post-mortem debugging

In [1]: from solver import solve

In [2]: solve()
IndexError: list index out of range

In [3]: %debug
> /Users/switowski/workspace/iac/solver.py(11)count_trees()
      9 x = (x + dx) % mod
     10 y += dy
--------> 11 if values[y][x] == "#":
     12 count += 1
     13 return count

ipdb>

Displaying the documentation is one of my favorite features, but post-mortem debugging is my favorite feature. After you get an exception, you can run %debug, and it will start a debugging session for that exception. That’s right! You don’t need to put any breakpoints or run IPython with any special parameters. You just start coding, and if when an exception happens, you run this command to start debugging.

7. Start the debugger automatically

In [1]: %pdb
Automatic pdb calling has been turned ON

In [2]: from solver import solve

In [3]: solve()
IndexError: list index out of range

> /Users/switowski/workspace/iac/solver.py(11)count_trees()
      9 x = (x + dx) % mod
     10 y += dy
--------> 11 if values[y][x] == "#":
     12 count += 1
     13 return count

ipdb> y
1
ipdb> x
3
ipdb>

And if you want to start a debugger on every exception automatically, you can run %pdb to enable the automatic debugger. Run %pdb again to disable it.

8. Run shell commands

In [1]: !pwd
/Users/switowski/workspace/iac

In [2]: ls -al
total 8
drwxr-xr-x 5 switowski staff 480 Dec 21 17:26 ./
drwxr-xr-x 55 switowski staff 1760 Dec 22 14:47 ../
drwxr-xr-x 9 switowski staff 384 Dec 21 17:27 .git/
drwxr-xr-x 4 switowski staff 160 Jan 25 11:39 __pycache__ /
-rw-r--r-- 1 switowski staff 344 Dec 21 17:26 solver.py

# Node REPL inside IPython? Sure!
In [3]: !node
Welcome to Node.js v12.8.0.
Type ".help" for more information.
> var x = "Hello world"
undefined
> x
'Hello world'
>

You can run shell commands without leaving IPython - you just need to prefix it with the exclamation mark. And the most common shell commands like ls, pwd, cd will work even without it (of course, unless you have a Python function with the same name).

I use it mostly to move between folders or to move files around. But you can do all sorts of crazy things - including starting a REPL for a different programming language inside IPython.

9. Move around the filesystem with %cd

In [1]: !pwd
/Users/switowski/workspace/iac/input_files/wrong/folder

In [2]: %cd ../..
/Users/switowski/workspace/iac/input_files

In [3]: %cd right_folder/
/Users/switowski/workspace/iac/input_files/right_folder

Alternatively, you can also move around the filesystem using the %cd magic command (press Tab to get the autocompletion for the list of available folders). It comes with some additional features - you can bookmark a folder or move a few folders back in the history (run %cd? to see the list of options).

10. %autoreload

Sebastian Witowski

@sebawitowski

10. Use "%autoreload" to automatically reload any imported function/module before running them.
It's great when you want to edit code in one window and run it in IPython without reimporting it every time (more info here: switowski.com/blog/ipython-a…).

07:23 AM - 10 Dec 2020

Use %autoreload to automatically reload all the imported functions before running them. By default, when you import a function in Python, Python “saves its source code in memory” (ok, that’s not what actually happens, but for illustration purposes, let’s stick with that oversimplification). When you change the source code of that function, Python won’t notice the change, and it will keep using the outdated version.

If you are building a function or a module and you want to keep testing the latest version without restarting the IPython (or using the importlib.reload()), you can use the %autoreload magic command. It will always reload the source code before running your functions. If you want to learn more - I wrote a longer article about it.

11. Change the verbosity of exceptions

By default, the amount of information in IPython’s exceptions is just right - at least for me. But if you prefer to change that, you can use the %xmode magic command. It will switch between 4 levels of traceback’s verbosity. Check it out - it’s the same exception, but the traceback gets more and more detailed:

Minimal

In [1]: %xmode
Exception reporting mode: Minimal

In [2]: solve()
IndexError: list index out of range

Plain

In [3]: %xmode
Exception reporting mode: Plain

In [4]: solve()
Traceback (most recent call last):
  File "<ipython-input-6-6f300b4f5987>", line 1, in <module>
    solve()
  File "/Users/switowski/workspace/iac/solver.py", line 27, in solve
    sol_part1 = part1(vals)
  File "/Users/switowski/workspace/iac/solver.py", line 16, in part1
    return count_trees(vals, 3, 1)
  File "/Users/switowski/workspace/iac/solver.py", line 11, in count_trees
    if vals[y][x] == "#":
IndexError: list index out of range

Context (that’s the default setting)

In [5]: %xmode
Exception reporting mode: Context

In [6]: solve()
--------------------------------------------------------------------------------
IndexError Traceback (most recent call last)
<ipython-input-8-6f300b4f5987> in <module>
---------> 1 solve()

~/workspace/iac/solver.py in solve()
     25 def solve():
     26 vals = getInput()
--------> 27 sol_part1 = part1(vals)
     28 print(f"Part 1: {sol_part1}")
     29 print(f"Part 2: {part2(vals, sol_part1)}")

~/workspace/iac/solver.py in part1(vals)
     14
     15 def part1(vals: list) -> int:
--------> 16 return count_trees(vals, 3, 1)
     17
     18 def part2(vals: list, sol_part1: int) -> int:

~/workspace/iac/solver.py in count_trees(vals, dx, dy)
      9 x = (x + dx) % mod
     10 y += dy
--------> 11 if vals[y][x] == "#":
     12 cnt += 1
     13 return cnt

IndexError: list index out of range

Verbose (like “Context” but also shows the values of local and global variables)

In [7]: %xmode
Exception reporting mode: Verbose

In [8]: solve()
--------------------------------------------------------------------------------
IndexError Traceback (most recent call last)
<ipython-input-10-6f300b4f5987> in <module>
---------> 1 solve()
        global solve = <function solve at 0x109312b80>

~/workspace/iac/solver.py in solve()
     25 def solve():
     26 values = read_input()
--------> 27 part1 = solve1(values)
        part1 = undefined
        global solve1 = <function solve1 at 0x109f363a0>
        values = [['..##.......', ..., '.#..#...#.#']]
     28 print(f"Part 1: {part1}")
     29 print(f"Part 2: {solve2(values, part1)}")

~/workspace/iac/solver.py in solve1(values=[['..##.......', ..., '.#..#...#.#']])
     14
     15 def solve1(values: list) -> int:
--------> 16 return count_trees(values, 3, 1)
        global count_trees = <function count_trees at 0x109f364c0>
        values = [['..##.......', ..., '.#..#...#.#']]
     17
     18 def solve2(values: list, sol_part1: int) -> int:

... and so on

IndexError: list index out of range

12. Rerun commands from the previous sessions

In [1]: a = 10

In [2]: b = a + 20

In [3]: b
Out[3]: 30

# Restart IPython

In [1]: %rerun ~1/
=== Executing: ===
a = 10
b = a + 20
b
=== Output: ===
Out[1]: 30

In [2]: b
Out[2]: 30

You can use the %rerun ~1/ to rerun all the commands from the previous session. That’s a great way to get you back to the same place where you left IPython. But it has one huge downside - if you had any exception (and I’m pretty sure you did), the execution will stop there. So you have to remove the lines with exceptions manually. If you are using Jupyter Notebooks, there is a workaround that allows you to tag a notebook cell as “raising an exception.” If you rerun it, IPython will ignore this exception. It’s not a perfect solution, and an option to ignore exceptions during the %rerun command would be much better.

13. Execute some code at startup

If you want to execute some code each time you start IPython, just create a new file inside the “startup” folder (~/.ipython/profile_default/startup/) and add your code there. IPython will automatically execute any files it finds in this folder. It’s great if you want to import some modules that you use all the time, but if you put too much code there, the startup time of IPython will be slower.

14. Use different profiles

Sebastian Witowski

@sebawitowski

14. You can create a new profile with different settings. For example, you can create a profile just for debugging with verbose exceptions and plenty of debugging/profiling libraries automatically imported.

08:21 AM - 14 Dec 2020

Maybe you have a set of modules that you want to import and settings to set in a specific situation. For example, when debugging/profiling, you want to set the exceptions to the verbose mode and import some profiling libraries. Don’t put that into the default profile because you don’t debug or profile your code all the time. Create a new profile and put your debugging settings inside. Profiles are like different user accounts for IPython - each of them has its own configuration file and startup folder.

15. Output from the previous commands

In [1]: sum(range(1000000))
Out[1]: 499999500000

In [2]: the_sum = _

In [3]: the_sum
Out[3]: 499999500000

In [4]: _1
Out[4]: 499999500000

If you forgot to assign an expression to a variable, use var = _. _ stores the output of the last command (this also works in the standard Python REPL). The results of all the previous commands are stored in variables _1 (output from the first command), _2 (output from the second command), etc.

16. Edit any function or module

You can use %edit to edit any Python function. And I really mean ANY function - functions from your code, from packages installed with pip, or even the built-in ones. You don’t even need to know in which file that function is located. Just specify the name (you have to import it first), and IPython will find it for you.

In the above example, I’m breaking the built-in randint() function by always returning 42.

17. Share your code

In [1]: welcome = "Welcome to my gist"

In [2]: welcome
Out[2]: 'Welcome to my gist'

In [3]: a = 42

In [4]: b = 41

In [5]: a - b
Out[5]: 1

In [6]: %pastebin 1-5
Out[6]: 'http://dpaste.com/8QA86F776'

If you want to share your code with someone, use the %pastebin command and specify which lines you want to share. IPython will create a pastebin (something similar to GitHub gist), paste selected lines, and return a link that you can send to someone. Just keep in mind that this snippet will expire in 7 days.

18. Use IPython as your debugger

Maybe some of the tips that I’ve shared convinced you that IPython is actually pretty cool. If that’s the case, you can use it not only as a REPL (the interactive Python shell) but also as a debugger. IPython comes with “ipdb” - it’s like the built-in Python debugger “pdb”, but with some IPython’s features on top of it (syntax highlighting, autocompletion, etc.)

You can use ipdb with your breakpoint statements by setting the PYTHONBREAKPOINT environment variable - it controls what happens when you call breakpoint() in your code. This trick requires using Python 3.7 or higher (that’s when the breakpoint() statement was introduced).

19. Execute code written in another language

In [1]: %%ruby
   ...: 1.upto 16 do |i|
   ...: out = ""
   ...: out += "Fizz" if i % 3 == 0
   ...: out += "Buzz" if i % 5 == 0
   ...: puts out.empty? ? i : out
   ...: end
   ...:
   ...:
1
2
Fizz
4
Buzz
Fizz
7
8
Fizz
Buzz
11
Fizz
13
14
FizzBuzz
16

Let’s say you want to execute some code written in another language without leaving IPython. You might be surprised to see that IPython supports Ruby, Bash, or JavaScript out of the box. And even more languages can be supported when you install additional kernels!

Just type %%ruby, write some Ruby code, and press Enter twice, and IPython will run it with no problem. It also works with Python2 (%%python2).

20. Store variables between sessions

In [1]: a = 100

In [2]: %store a
Stored 'a' (int)

# Restart IPython
In [1]: %store -r a

In [2]: a
Out[2]: 100

IPython uses SQLite for some lightweight storage between sessions. That’s where it saves the history of your previous sessions. But you can use it to store your own data. For example, with the %store magic command, you can save variables in IPython’s database and restore them in another session using %store -r. You can also set the c.StoreMagics.autorestore = True in the configuration file to automatically restore all the variables from the database when you start IPython.

21. Save session to a file

In [1]: a = 100

In [2]: b = 200

In [3]: c = a + b

In [4]: c
Out[4]: 300

In [5]: %save filename.py 1-4
The following commands were written to file `filename.py`:
a = 100
b = 200
c = a + b
c

You can save your IPython session to a file with the %save command. That’s quite useful when you have some working code and you want to continue editing it with your text editor. Instead of manually copying and pasting lines to your code editor, you can dump the whole IPython session and then remove unwanted lines.

22. Clean up “>” symbols and fix indentation

# Clipboard content:
# >def greet(name):
# > print(f"Hello {name}")

# Just pasting the code won't work
In [1]: >def greet(name):
   ...: > print(f"Hello {name}")
  File "<ipython-input-1-a7538fc939af>", line 1
    >def greet(name):
    ^
SyntaxError: invalid syntax

# But using %paste works
In [2]: %paste
>def greet(name):
> print(f"Hello {name}")

## -- End pasted text --

In [3]: greet("Sebastian")
Hello Sebastian

If you need to clean up incorrect indentation or “>” symbols (for example, when you copy the code from a git diff, docstring, or an email), instead of doing it manually, copy the code and run %paste. IPython will paste the code from your clipboard, fix the indentation, and remove the “>” symbols (although it sometimes doesn’t work properly).

23. List all the variables

In [1]: a = 100

In [2]: name = "Sebastian"

In [3]: squares = [x*x for x in range(100)]

In [4]: squares_sum = sum(squares)

In [5]: def say_hello():
   ...: print("Hello!")
   ...:

In [6]: %whos
Variable Type Data/Info
----------------------------------------
a int 100
name str Sebastian
say_hello function <function say_hello at 0x111b60a60>
squares list n=100
squares_sum int 328350

You can get a list of all the variables from the current session (nicely formatted, with information about their type and the data they store) with the %whos command.

24. Use asynchronous functions

In [1]: import asyncio

In [2]: async def worker():
   ...: print("Hi")
   ...: await asyncio.sleep(2)
   ...: print("Bye")
   ...:

# The following code would fail in the standard Python REPL
# because we can't call await outside of an async function
In [3]: await asyncio.gather(worker(), worker(), worker())
Hi
Hi
Hi
Bye
Bye
Bye

You can speed up your code with asynchronous functions. But the thing about asynchronous code is that you need to start an event loop to call them. However, IPython comes with its own event loop! And with that, you can await asynchronous functions just like you would call a standard, synchronous one.

25. IPython scripts

$ ls
file1.py file2.py file3.py file4.py wishes.ipy

$ cat wishes.ipy
files = !ls
# Run all the files with .py suffix
for file in files:
    if file.endswith(".py"):
        %run $file

$ ipython wishes.ipy
Have a
Very Merry
Christmas!
🎄🎄🎄🎄🎄🎄

You can execute files containing IPython-specific code (shell commands prefixed with ! or magic methods prefixed with %). Just save the file with the “.ipy” extension and then pass it to the ipython command.

Conclusions

If you have been reading my blog for a bit, you probably already realize that IPython is one of my favorite Python tools. It’s an excellent choice for solving code challenges like the Advent of Code, and it has a lot of cool tricks that can help you. Leave a comment if you know some other cool tricks that you want to share!

Photo by Valeria Vinnik from: Pexels

Sorting Lists in Python

Sebastian Witowski — Thu, 22 Oct 2020 00:00:00 +0000

There are at least two common ways to sort lists in Python:

With sorted function that returns a new list
With list.sort method that modifies list in place

Which one is faster? Let's find out!

sorted() vs list.sort()

I will start with a list of 1 000 000 randomly shuffled integers. Later on, I will also check if the order matters.

# sorting.py
from random import sample

# List of 1 000 000 integers randomly shuffled
MILLION_RANDOM_NUMBERS = sample(range(1_000_000), 1_000_000)

def test_sort():
    random_list = MILLION_RANDOM_NUMBERS[:]
    return random_list.sort()

def test_sorted():
    random_list = MILLION_RANDOM_NUMBERS[:]
    return sorted(random_list)

$ python -m timeit -s "from sorting import test_sort" "test_sort()"
1 loop, best of 5: 352 msec per loop

$ python -m timeit -s "from sorting import test_sorted" "test_sorted()"
1 loop, best of 5: 385 msec per loop

sorted is less than 10% slower (385/352≈1.094). Since we only run one loop, the exact numbers are not very reliable. I have rerun the same tests a couple more times, and the results were slightly different each time. sort took around 345-355 msec and sorted took around 379-394 msec (but it was always slower than sort). This difference comes mostly from the fact that sorted creates a new list.

Why do I make a copy of the original list in each test? Well, in the original version of this article, I forgot to do this, and I ended up with completely wrong benchmarks. sort was running on a sorted list (because the first iteration of timeit sorted it), and sorted was running on a random list. You can see the whole explanation in the original post.

Initial order matters

What happens when our initial list is already sorted?

MILLION_NUMBERS = list(range(1_000_000))

$ python -m timeit -s "from sorting import test_sort" "test_sort()"
20 loops, best of 5: 12.1 msec per loop

$ python -m timeit -s "from sorting import test_sorted" "test_sorted()"
20 loops, best of 5: 16.6 msec per loop

Now, sorting takes much less time and the difference between sort and sorted grows to 37% (16.6/12.1≈1.372). Why is sorted 37% slower this time? Well, creating a new list takes the same amount of time as before. And since the time spent on sorting has shrunk, the impact of creating that new list got bigger.

And if we try to sort a list of 1 000 000 numbers ordered in descending order:

DESCENDING_MILLION_NUMBERS = list(range(1_000_000, 0, -1))

$ python -m timeit -s "from sorting import test_sort" "test_sort()"
20 loops, best of 5: 11.7 msec per loop

$ python -m timeit -s "from sorting import test_sorted" "test_sorted()"
20 loops, best of 5: 18.1 msec per loop

The results are almost identical as before. The sorting algorithm is clever enough to optimize the sorting process for a descending list.

For our last test, let’s try to sort 1 000 000 numbers where 100 000 elements are shuffled, and the rest are ordered:

# 10% of numbers are random
MILLION_SLIGHTLY_RANDOM_NUMBERS = [*range(900_000), *sample(range(1_000_000), 100_000)]

$ python -m timeit -s "from sorting import test_sort" "test_sort()"
5 loops, best of 5: 61.2 msec per loop

$ python -m timeit -s "from sorting import test_sorted" "test_sorted()"
5 loops, best of 5: 71 msec per loop

Both functions get slower as the input list becomes more scrambled.

Using list.sort() is my preferred way of sorting lists - it saves some time (and memory) by not creating a new list. But that's a double-edged sword! Sometimes you might accidentally overwrite the initial list without realizing it (as I did with my initial benchmarks 😅). So, if you want to preserve the initial list's order, you have to use sorted instead. And sorted can be used with any iterable, while sort only works with lists. If you want to sort a set, then sorted is your only solution.

Conclusions

sort is slightly faster than sorted, because it doesn't create a new list. But you might still stick with sorted if:

You don't want to modify the original list. sort performs sorting in-place, so you can't use it here.
You need to sort something else than a list. sort is only defined on lists, so if you want to sort a set or any other collection of items, you have to use sorted instead.

If you want to learn more, the Sorting HOW TO guide from Python documentation contains a lot of useful information.

For Loop vs. List Comprehension

Sebastian Witowski — Thu, 15 Oct 2020 00:00:00 +0000

Many simple "for loops" in Python can be replaced with list comprehensions. You can often hear that list comprehension is "more Pythonic" (almost as if there was a scale for comparing how Pythonic something is, compared to something else 😉). In this article, I will compare their performance and discuss when a list comprehension is a good idea, and when it's not.

Filter a list with a "for loop"

Let's use a simple scenario for a loop operation - we have a list of numbers, and we want to remove the odd ones. One important thing to keep in mind is that we can't remove items from a list as we iterate over it. Instead, we have to create a new one containing only the even numbers:

# filter_list.py

MILLION_NUMBERS = list(range(1_000_000))

def for_loop():
    output = []
    for element in MILLION_NUMBERS:
        if not element % 2:
            output.append(element)
    return output

if not element % 2 is equivalent to if element % 2 == 0, but it's slightly faster. I will write a separate article about comparing boolean values soon.

Let's measure the execution time of this function. I'm using Python 3.8 for benchmarks (you can read about the whole setup in the Introduction article on my blog):

$ python -m timeit -s "from filter_list import for_loop" "for_loop()"
5 loops, best of 5: 65.4 msec per loop

It takes 65 milliseconds to filter a list of one million elements. How fast will a list comprehension deal with the same task?

Filter a list with list comprehension

# filter_list.py

MILLION_NUMBERS = list(range(1_000_000))

def list_comprehension():
    return [number for number in MILLION_NUMBERS if not number % 2]

$ python -m timeit -s "from filter_list import list_comprehension" "list_comprehension()"
5 loops, best of 5: 44.5 msec per loop

"For loop" is around 50% slower than a list comprehension (65.4/44.5≈1.47). And we just reduced five lines of code to one line! Cleaner and faster code? Great!

Can we make it better?

Filter a list with the "filter" function

Python has a built-in filter function for filtering collections of elements. This sounds like a perfect use case for our problem, so let's see how fast it will be.

# filter_list.py

MILLION_NUMBERS = list(range(1_000_000))

def filter_function():
    return filter(lambda x: not x % 2, MILLION_NUMBERS)

$ python -m timeit -s "from filter_list import filter_function" "filter_function()"
1000000 loops, best of 5: 284 nsec per loop

284 nanoseconds?! That's suspiciously fast! It turns out that the filter function returns an iterator. It doesn't immediately go over one million elements, but it will return the next value when we ask for it. To get all the results at once, we can convert this iterator to a list.

# filter_list.py

MILLION_NUMBERS = list(range(1_000_000))

def filter_return_list():
    return list(filter(lambda x: not x % 2, MILLION_NUMBERS))

$ python -m timeit -s "from filter_list import filter_return_list" "filter_return_list()"
2 loops, best of 5: 104 msec per loop

Now, its performance is not so great anymore. It's 133% slower than the list comprehension (104/44.5≈2.337) and 60% slower than the "for loop" (104/65.4≈1.590).

While, in this case, it's not the best solution, an iterator is an excellent alternative to a list comprehension when we don't need to have all the results at once. If it turns out that we only need to get a few elements from the filtered list, an iterator will be a few orders of magnitude faster than other "non-lazy" solutions.

More than one operation in the loop

List comprehensions are often faster and easier to read, but they have one significant limitation. What happens if you want to execute more than one simple instruction? List comprehension can't accept multiple statements (without sacrificing readability). But in many cases, you can wrap those multiple statements in a function.

Let's use a slightly modified version of the famous "Fizz Buzz" program as an example. We want to iterate over a list of elements and for each of them return:

"fizzbuzz" if the number can be divided by 3 and 5
"fizz" if the number can be divided by 3
"buzz" if the number can be divided by 5
the number itself, if it can't be divided by 3 or 5

Here is a simple solution:

# filter_list.py

def fizz_buzz():
    output = []
    for number in MILLION_NUMBERS:
        if number % 3 == 0 and number % 5 == 0:
            output.append('fizzbuzz')
        elif number % 3 == 0:
            output.append('fizz')
        elif number % 5 == 0:
            output.append('buzz')
        else:
            output.append(number)
    return output

Here is the list comprehension equivalent of the fizz_buzz():

['fizzbuzz' if x % 3 == 0 and x % 5 == 0 else 'fizz' if x % 3 == 0 else 'buzz' if x % 5 == 0 else x for x in MILLION_NUMBERS]

It's not easy to read - at least for me. It gets better if we split it into multiple lines:

[
    "fizzbuzz" if x % 3 == 0 and x % 5 == 0
    else "fizz" if x % 3 == 0
    else "buzz" if x % 5 == 0
    else x
    for x in MILLION_NUMBERS
]

But if I see a list comprehension that spans multiple lines, I try to refactor it. We can extract the "if" statements into a separate function:

# filter_list.py

def transform(number):
    if number % 3 == 0 and number % 5 == 0:
        return 'fizzbuzz'
    elif number % 3 == 0:
        return 'fizz'
    elif number % 5 == 0:
        return 'buzz'
    return number

def fizz_buzz2():
    output = []
    for number in MILLION_NUMBERS:
        output.append(transform(number))
    return output

Now it's trivial to turn it into a list comprehension. And we get the additional benefit of a nice separation of logic into a function that does the "fizz buzz" check and a function that actually iterates over a list of numbers and applies the "fizz buzz" transformation.

Here is the improved list comprehension:

def fizz_buzz2_comprehension():
    return [transform(number) for number in MILLION_NUMBERS]

Let's compare all three versions:

$ python -m timeit -s "from filter_list import fizz_buzz" "fizz_buzz()"
2 loops, best of 5: 191 msec per loop

$ python -m timeit -s "from filter_list import fizz_buzz2" "fizz_buzz2()"
1 loop, best of 5: 285 msec per loop

$ python -m timeit -s "from filter_list import fizz_buzz2_comprehension" "fizz_buzz2_comprehension()"
1 loop, best of 5: 224 msec per loop

Extracting a separate function adds some overhead. List comprehension with a separate transform() function is around 17% slower than the initial "for loop"-based version (224/191≈1.173). But it's much more readable, so I prefer it over the other solutions.

And, if you are curious, the one-line list comprehension mentioned before is the fastest solution:

def fizz_buzz_comprehension():
    return [
        "fizzbuzz" if x % 3 == 0 and x % 5 == 0
        else "fizz" if x % 3 == 0
        else "buzz" if x % 5 == 0
        else x
        for x in MILLION_NUMBERS
    ]

$ python -m timeit -s "from filter_list import fizz_buzz_comprehension" "fizz_buzz_comprehension()"
2 loops, best of 5: 147 msec per loop

Fastest, but also harder to read. If you run this code through a code formatter like black (which is a common practice in many projects), it will further obfuscate this function:

[
    "fizzbuzz"
    if x % 3 == 0 and x % 5 == 0
    else "fizz"
    if x % 3 == 0
    else "buzz"
    if x % 5 == 0
    else x
    for x in MILLION_NUMBERS
]

There is nothing wrong with black here - we are simply putting too much logic inside the list comprehension. If I had to say what the above code does, it would take me much longer to figure it out than if I had two separate functions. Saving a few hundred milliseconds of execution time and adding a few seconds of reading time doesn't sound like a good trade-off 😉.

Clever one-liners can impress some recruiters during code interviews. But in real life, separating logic into different functions makes it much easier to read and document your code. And, statistically, we read more code than we write.

Conclusions

List comprehensions are often not only more readable but also faster than using "for loops." They can simplify your code, but if you put too much logic inside, they will instead become harder to read and understand.

Even though list comprehensions are popular in Python, they have a specific use case: when you want to perform some operations on a list and return another list. And they have limitations - you can't break out of a list comprehension or put comments inside. In many cases, "for loops" will be your only choice.

I only scratched the surface of how useful list comprehension (or any other type of "comprehension" in Python) can be. If you want to learn more, Trey Hunner has many excellent articles and talks on this subject (for example, this one for beginners).

Should We Still Use OrderedDict in Python?

Sebastian Witowski — Thu, 08 Oct 2020 00:00:00 +0000

If you worked with Python 2 or an early version of Python 3, you probably remember that, in the past, dictionaries were not ordered. If you wanted to have a dictionary that preserved the insertion order, the go-to solution was to use OrderedDict from the collections module.

In Python 3.6, dictionaries were redesigned to improve their performance (their memory usage was decreased by around 20-25%). This change had an interesting side-effect - dictionaries became ordered (although this order was not officially guaranteed). "Not officially guaranteed" means that it was just an implementation detail that could be removed in the future Python releases.

But starting from Python 3.7, the insertion-order preservation has been guaranteed in the language specification. If you started your journey with Python 3.7 or a newer version, you probably don't know the world where you need a separate data structure to preserve the insertion order in a dictionary.

So if there is no need to use the OrderedDict, why is it still included in the collections module? Maybe it's more efficient? Let's find out!

OrderedDict vs dict

For my benchmarks, I will perform some typical dictionary operations:

Create a dictionary of 100 elements
Add a new item
Check if an item exists in a dictionary
Grab an existing and nonexistent item with the get method

To simplify the code, I wrap steps 2-4 in a function that accepts a dictionary (or OrderedDictionary) as an argument.

# dictionaries.py

from collections import OrderedDict

def perform_operations(dictionary):
    dictionary[200] = 'goodbye'
    is_50_included = 50 in dictionary
    item_20 = dictionary.get(20)
    nonexistent_item = dictionary.get('a')

def ordereddict():
    dictionary = OrderedDict.fromkeys(range(100), 'hello world')
    perform_operations(dictionary)

def standard_dict():
    dictionary = dict.fromkeys(range(100), 'hello world')
    perform_operations(dictionary)

Let's compare both functions. I run my benchmarks under Python 3.8:

$ python -m timeit -s "from dictionaries import ordereddict" "ordereddict()"
50000 loops, best of 5: 8.6 usec per loop

$ python -m timeit -s "from dictionaries import standard_dict" "standard_dict()"
50000 loops, best of 5: 4.7 usec per loop

OrderedDict is over 80% slower than the standard Python dictionary (8.6/4.7≈1.83).

What happens if the dictionary size grows to 10 000 elements?

# dictionaries2.py

from collections import OrderedDict

def perform_operations(dictionary):
    dictionary[20000] = 'goodbye'
    is_5000_included = 5000 in dictionary
    item_2000 = dictionary.get(2000)
    nonexistent_item = dictionary.get('a')

def ordereddict():
    dictionary = OrderedDict.fromkeys(range(10000), 'hello world')
    perform_operations(dictionary)

def standard_dict():
    dictionary = dict.fromkeys(range(10000), 'hello world')
    perform_operations(dictionary)

$ python -m timeit -s "from dictionaries import ordereddict" "ordereddict()"
200 loops, best of 5: 1.07 msec per loop

$ python -m timeit -s "from dictionaries import standard_dict" "standard_dict()"
500 loops, best of 5: 547 usec per loop

After increasing the dictionary size by 100x times, the difference between both functions stays the same. OrderedDict still takes almost twice as long to perform the same operations as a standard Python dictionary.

There is no point in testing even bigger dictionaries. If you need a really big dictionary, you should use more efficient data structures from the Numpy or Pandas libraries.

When to use OrderedDict?

If the OrderedDict is slower, why would you want to use it? I can think of at least two reasons:

You are still using a Python version that doesn't guarantee the order in dictionaries (pre 3.6). In this case, you don't have a choice.
You want to use additional features that OrderedDict offers. For example, it can be reversed. If you try to run reversed() function on a standard dictionary, you will get an error, but OrderedDict will nicely return a reversed version of itself.

How to stay up to date on Python changes?

If you are using one of the latest versions of Python, dictionaries are ordered by default. But it's easy to miss changes like this, especially if you upgrade Python version by a few releases at once, and you don't read the release notes carefully. I usually read some blog posts when there is a new version of Python coming out (there are plenty of blog posts around that time), so I catch the essential updates.

The best source of information is the official documentation. Unlike a lot of documentation that I have seen in my life, the "What's New in Python 3" page is written in a very approachable language. It's easy to read and grasp the most significant changes. If you haven't done it yet, go check it out. I reread it a few days ago, and I was surprised how many features I forgot about!

Easy Speedup Wins With Numba

Sebastian Witowski — Thu, 01 Oct 2020 00:00:00 +0000

If you have functions that do a lot of mathematical operations, use NumPy or rely heavily on loops, then there is a way to speed them up significantly with one line of code. Ok, two lines if you count the import.

Numba and the @jit decorator

Meet Numba and its @jit decorator. It changes how your code is compiled, often improving its performance. You don't have to install any special tools (just the numba pip package), you don't have to tweak any parameters. All you have to do is:

Add the @jit decorator to a function
Check if it's faster

Let's see an example of code before and after applying Numba's optimization.

# numba_testing.py

import math

def compute():
    # Bunch of dummy math operations
    result = 0
    for number in range(1_000_000):
        double = number * 2
        result += math.sqrt(double) + double
    return result

The only purpose of this code is to do some calculations and to "be slow." Let's see how slow (benchmarks are done with Python 3.8 - I describe the whole setup in the Introduction article on my blog):

$ python -m timeit -s "from numba_testing import compute" "compute()"
1 loop, best of 5: 217 msec per loop

Now, we add @jit to our code. The body of the function stays the same, and the only difference is the decorator. Don't forget to install Numba package with pip (pip install numba).

# numba_testing.py

import math

from numba import jit

@jit
def compute_jit():
    # Bunch of dummy math operations
    result = 0
    for number in range(1_000_000):
        double = number * 2
        result += math.sqrt(double) + double
    return result

Let's measure the execution time once more:

$ python -m timeit -s "from numba_testing import compute_jit" "compute_jit()"
200 loops, best of 5: 1.76 msec per loop

Using @jit decorator gave us a 120x speedup (217 / 1.76 = 123.295)! That's a huge improvement for such a simple change!

Other features of Numba

@jit is the most common decorator from the Numba library, but there are others that you can use:

@njit - alias for @jit(nopython=True). In nopython mode, Numba tries to run your code without using the Python interpreter at all. It can lead to even bigger speed improvements, but it's also possible that the compilation will fail in this mode.
@vectorize and @guvectorize - produces ufunc and generalized ufunc used in NumPy.
@jitclass - can be used to decorate the whole class.
@cfunc - declares a function to be used as a native callback (from C or C++ code).

There are also advanced features that let you, for example, run your code on GPU with @cuda.jit. This doesn't work out of the box, but it might be worth the effort for some very computational-heavy operations.

Numba has plenty of configuration options that will further improve your code's execution time if you know what you are doing. You can:

Disable GIL (Global Interpreter Lock) with nogil
Cache results with cache
Automatically parallelize functions with parallel.

Check out the documentation to see what you can do. And to see more real-life examples (like computing the Black-Scholes model or the Lennard-Jones potential), visit the Numba Examples page.

Conclusions

Numba is a great library that can significantly speed up your programs with minimal effort. Given that it takes less than a minute to install and decorate some slow functions, it's one of the first solutions that you can check when you want to quickly improve your code (without rewriting it).

It works best if your code:

Uses NumPy a lot
Performs plenty of mathematical operations
Performs operations is a loop

How did I find Numba?

I first learned about Numba when I was doing code challenges from the Advent of Code a few years ago. I wrote a pretty terrible algorithm, left it running, and went for lunch. When I came back after one hour, my program wasn't even 10% done. I stopped it, added the @jit decorator to the main function, rerun it, and I had the results in under one minute! Fantastic improvement with almost no work!

This story doesn't mean that it's ok to write sloppy code, and then use hacks to speed it up 😉. But sometimes you just need to make some one-off calculations. You don't want to spend too much time writing the perfect algorithm. Or maybe you can't think of a better algorithm, and the one you have is too slow. Using tools like Numba can be one of the fastest and easiest to apply improvements!

Find Item in a List

Sebastian Witowski — Thu, 24 Sep 2020 00:00:00 +0000

Find a number

If you want to find the first number that matches some criteria, what do you do? The easiest way is to write a loop that checks numbers one by one and returns when it finds the correct one.

Let's say we want to get the first number divided by 42 and 43 (that's 1806). If we don't have a predefined set of elements (in this case, we want to check all the numbers starting from 1), we might use a "while loop".

# find_item.py

def while_loop():
    item = 1
    # You don't need to use parentheses, but they improve readability
    while True:
        if (item % 42 == 0) and (item % 43 == 0):
            return item
        item += 1

It's pretty straightforward:

Start from number 1
Check if that number can be divided by 42 and 43.
- If yes, return it (this stops the loop)
Otherwise, check the next number

Find a number in a list

If we have a list of items that we want to check, we will use a "for loop" instead. I know that the number I'm looking for is smaller than 10 000, so let's use that as the upper limit:

# find_item.py

def for_loop():
    for item in range(1, 10000):
        if (item % 42 == 0) and (item % 43 == 0):
            return item

Let's compare both solutions (benchmarks are done with Python 3.8):

$ python -m timeit -s "from find_item import while_loop" "while_loop()"
2000 loops, best of 5: 134 usec per loop

$ python -m timeit -s "from find_item import for_loop" "for_loop()"
2000 loops, best of 5: 103 usec per loop

"While loop" is around 30% slower than the "for loop" (134/103≈1.301).

Loops are optimized to iterate over a collection of elements. Trying to manually do the iteration (for example, by referencing elements in a list through an index variable) will be a slower and often over-engineered solution.

Python 2 flashback

In Python 2, functions like range, filter, or zip were eager , so they would always create the whole collection when initialized. All those elements would be loaded to the memory, increasing the execution time of your code and its memory usage. To avoid this behavior, you had to use their lazy equivalents like xrange, ifilter, or izip.

Out of curiosity, let's see how slow is the for_loop() function if we run it with Python 2.7.18 (the latest and last version of Python 2):

$ pyenv shell 2.7.18
$ python -m timeit -s "from find_item import for_loop" "for_loop()"
10000 loops, best of 3: 151 usec per loop

That's almost 50% slower than running the same function in Python 3 (151/103≈1.4660). Updating Python version is one of the easiest performance wins you can get!

If you are wondering what's pyenv and how to use it to quickly switch Python versions, check out this section of my PyCon 2020 workshop on Python tools.

Let's go back to our "while loop" vs. "for loop" comparison. Does it matter if the element we are looking for is at the beginning or at the end of the list?

def while_loop2():
    item = 1
    while True:
        if (item % 98 == 0) and (item % 99 == 0):
            return item
        item += 1

def for_loop2():
    for item in range(1, 10000):
        if (item % 98 == 0) and (item % 99 == 0):
            return item

This time, we are looking for number 9702, which is at the very end of our list. Let's measure the performance:

$ python -m timeit -s "from find_item import while_loop2" "while_loop2()"
500 loops, best of 5: 710 usec per loop

$ python -m timeit -s "from find_item import for_loop2" "for_loop2()"
500 loops, best of 5: 578 usec per loop

There is almost no difference. "While loop" is around 22% slower this time (710/578≈1.223). I performed a few more tests (up to a number close to 100 000 000), and the difference was always similar (in the range of 20-30% slower).

Find a number in an infinite list

So far, the collection of items we wanted to iterate over was limited to the first 10 000 numbers. But what if we don't know the upper limit? In this case, we can use the count function from the itertools library.

from itertools import count

def count_numbers():
    for item in count(1):
        if (item % 42 == 0) and (item % 43 == 0):
            return item

count(start=0, step=1) will start counting numbers from the start parameter, adding the step in each iteration. In my case, I need to change the start parameter to 1, so it works the same as the previous examples.

count works almost the same as the "while loop" that we made at the beginning. How about the speed?

$ python -m timeit -s "from find_item import count_numbers" "count_numbers()"
2000 loops, best of 5: 109 usec per loop

It's almost the same as the "for loop" version. So count is a good replacement if you need an infinite counter.

What about a list comprehension?

A typical solution for iterating over a list of items is to use a list comprehension. But we want to exit the iteration as soon as we find our number, and that's not easy to do with a list comprehension. It's a great tool to go over the whole collection, but not in this case.

Let's see how bad it is:

def list_comprehension():
    return [item for item in range(1, 10000) if (item % 42 == 0) and (item % 43 == 0)][0]

$ python -m timeit -s "from find_item import list_comprehension" "list_comprehension()"
500 loops, best of 5: 625 usec per loop

That's really bad - it's a few times slower than other solutions! It takes the same amount of time, no matter if we search for the first or last element. And we can't use count here.

But using a list comprehension points us in the right direction - we need something that returns the first element it finds and then stops iterating. And that thing is a generator! We can use a generator expression to grab the first element matching our criteria.

Find item with a generator expression

def generator():
    return next(item for item in count(1) if (item % 42 == 0) and (item % 43 == 0))

The whole code looks very similar to a list comprehension, but we can actually use count. Generator expression will execute only enough code to return the next element. Each time you call next(), it will resume work in the same place where it stopped the last time, grab the next item, return it, and stop again.

$ python -m timeit -s "from find_item import generator" "generator()"
2000 loops, best of 5: 110 usec per loop

It takes almost the same amount of time as the best solution we have found so far. And I find this syntax much easier to read - as long as we don't put too many ifs there!

Generators have the additional benefit of being able to "suspend" and "resume" counting. We can call next() multiple times, and each time we get the next element matching our criteria. If we want to get the first three numbers that can be divided by 42 and 43 - here is how easily we can do this with a generator expression:

def generator_3_items():
    gen = (item for item in count(1) if (item % 42 == 0) and (item % 43 == 0))
    return [next(gen), next(gen), next(gen)]

Compare it with the "for loop" version:

def for_loop_3_items():
    items = []
    for item in count(1):
        if (item % 42 == 0) and (item % 43 == 0):
            items.append(item)
            if len(items) == 3:
                return items

Let's benchmark both versions:

$ python -m timeit -s "from find_item import for_loop_3_items" "for_loop_3_items()"
1000 loops, best of 5: 342 usec per loop

$ python -m timeit -s "from find_item import generator_3_items" "generator_3_items()"
1000 loops, best of 5: 349 usec per loop

Performance-wise, both functions are almost identical. So when would you use one over the other? "For loop" lets you write more complex code. You can't put nested "if" statements or multiline code with side effects inside a generator expression. But if you only do simple filtering, generators can be much easier to read.

Conclusions

Generator expression combined with next() is a great way to grab one or more elements based on specific criteria. It's memory-efficient, fast, and easy to read - as long as you keep it simple. When the number of "if statements" in the generator expression grows, it becomes much harder to read (and write).

With complex filtering criteria or many ifs, "for loop" is a more suitable choice that doesn't sacrifice the performance.

Python: Ask for Forgiveness or Look Before You Leap?

Sebastian Witowski — Thu, 20 Aug 2020 00:00:00 +0000

“Ask for forgiveness” and “look before you leap” (sometimes also called “ask for permission”) are two opposite approaches to writing code. If you “look before you leap”, you first check if everything is set correctly, then you perform an action. For example, you want to read text from a file. What could go wrong with that? Well, the file might not be in the location where you expect it to be. So, you first check if the file exists:

import os
if os.path.exists("path/to/file.txt"):
    ...

# Or from Python 3.4
from pathlib import Path
if Path("/path/to/file").exists():
    ...

Even if the file exists, maybe you don't have permission to open it? So let's check if you can read it:

import os
if os.access("path/to/file.txt", os.R_OK):
    ...

But what if the file is corrupted? Or if you don't have enough memory to read it? This list could go on. Finally, when you think that you checked every possible corner-case, you can open and read it:

with open("path/to/file.txt") as input_file:
    return input_file.read()

Depending on what you want to do, there might be quite a lot of checks to perform. And even when you think you covered everything, there is no guarantee that some unexpected problems won't prevent you from reading this file. So, instead of doing all the checks, you can "ask for forgiveness."

With "ask for forgiveness," you don't check anything. You perform whatever action you want, but you wrap it in a try/catch block. If an exception happens, you handle it. You don't have to think about all the things that can go wrong, your code is much simpler (no more nested ifs), and you will usually catch more errors that way. That's why the Python community, in general, prefers this approach, often called "EAFP" - "Easier to ask for forgiveness than permission."

Here is a simple example of reading a file with the "ask for forgiveness" approach:

try:
    with open("path/to/file.txt", "r") as input_file:
        return input_file.read()
except IOError:
    # Handle the error or just ignore it

Here we are catching the IOError. If you are not sure what kind of exception can be raised, you could catch all of them with the BaseException class, but in general, it's a bad practice. It will catch every possible exception (including, for example, KeyboardInterrupt when you want to stop the process), so try to be more specific.

"Ask for forgiveness" is cleaner. But which one is faster?

"Ask For Forgiveness" vs "Look Before You Leap" - speed

Time for a simple test. Let's say that I have a class, and I want to read an attribute from this class. But I'm using inheritance, so I'm not sure if the attribute is defined or not. I need to protect myself, by either checking if it exists ("look before you leap") or catching the AttributeError ("ask for forgiveness"):

# permission_vs_forgiveness.py

class BaseClass:
    hello = "world"

class Foo(BaseClass):
    pass

FOO = Foo()

# Look before you leap
def test_lbyl():
    if hasattr(FOO, "hello"):
        FOO.hello

# Ask for forgiveness
def test_aff():
    try:
        FOO.hello
    except AttributeError:
        pass

Let's measure the speed of both functions.

For benchmarking, I'm using the standard timeit module and Python 3.8. I describe my setup and some assumptions in the Introduction to the Writing Faster Python.

$ python -m timeit -s "from permission_vs_forgiveness import test_lbyl" "test_lbyl()"
2000000 loops, best of 5: 155 nsec per loop

$ python -m timeit -s "from permission_vs_forgiveness import test_aff" "test_aff()"
2000000 loops, best of 5: 118 nsec per loop

"Look before you leap" is around 30% slower (155/118≈1.314).

What happens if we increase the number of checks? Let's say that this time we want to check for three attributes, not just one:

# permission_vs_forgiveness.py

class BaseClass:
    hello = "world"
    bar = "world"
    baz = "world"

class Foo(BaseClass):
    pass

FOO = Foo()

# Look before you leap
def test_lbyl2():
    if hasattr(FOO, "hello") and hasattr(FOO, "bar") and hasattr(FOO, "baz"):
        FOO.hello
        FOO.bar
        FOO.baz

# Ask for forgiveness
def test_aff2():
    try:
        FOO.hello
        FOO.bar
        FOO.baz
    except AttributeError:
        pass

$ python -m timeit -s "from permission_vs_forgiveness import test_lbyl2" "test_lbyl2()"
500000 loops, best of 5: 326 nsec per loop

$ python -m timeit -s "from permission_vs_forgiveness import test_aff2" "test_aff2()"
2000000 loops, best of 5: 176 nsec per loop

"Look before you leap" is now around 85% slower (326/176≈1.852). So the "ask for forgiveness" is not only much easier to read and robust but, in many cases, also faster. Yes, you read it right, "in many cases," not "in every case!"

The main difference between "EAFP" and "LBYL"

What happens if the attribute is actually not defined? Take a look at this example:

# permission_vs_forgiveness.py

class BaseClass:
    pass  # "hello" attribute is now removed

class Foo(BaseClass):
    pass

FOO = Foo()

# Look before you leap
def test_lbyl3():
    if hasattr(FOO, "hello"):
        FOO.hello

# Ask for forgiveness
def test_aff3():
    try:
        FOO.hello
    except AttributeError:
        pass

$ python -m timeit -s "from permission_vs_forgiveness import test_lbyl3" "test_lbyl3()"
2000000 loops, best of 5: 135 nsec per loop

$ python -m timeit -s "from permission_vs_forgiveness import test_aff3" "test_aff3()"
500000 loops, best of 5: 562 nsec per loop

The tables have turned. "Ask for forgiveness" is now over four times as slow as "Look before you leap" (562/135≈4.163). That's because this time, our code throws an exception. And handling exceptions is expensive.

If you expect your code to fail often, then "Look before you leap" might be much faster.

Verdict

"Ask for forgiveness" results in much cleaner code, makes it easier to catch errors, and in most cases, it's much faster. No wonder that EAFP("Easier to ask for forgiveness than permission") is such a ubiquitous pattern in Python. Even in the example from the beginning of this article (checking if a file exists with os.path.exists) - if you look at the source code of the exists method, you will see that it's simply using a try/except. "Look before you leap" often results in a longer code that is less readable (with nested if statements) and slower. And following this pattern, you will probably sometimes miss a corner-case or two.

As a rule of thumb - ask yourself: "Is it more common that this code will throw an exception or not?" Use "look before you leap" if you expect some problems (and if you can predict what those problems will be). Otherwise, use the "ask for forgiveness" for cleaner and faster code.

Let’s Write an IPython Extension the Hard Way

Sebastian Witowski — Thu, 16 Jul 2020 00:00:00 +0000

I love IPython. I love using it, I love writing about it, I love taking pictures with its core contributors (Hi Paul!). If I ever get invited to the “Talk Python To Me” podcast (not that I have anything interesting to talk about), and Michael Kennedy is going to ask me what my favorite Python package is, you know what I’m going to say? Yep, IPython.

So, when a friend of mine asked me what kind of lightning talk I want to prepare for one of the upcoming micro-conferences, my first thought was: “Let’s try to make something cool with IPython”. Maybe I can improve the %rerun magic function?

What’s wrong with the `%rerun` function?

You can use %rerun to run commands from this or previous sessions of IPython. The most common use case for me is to get my IPython session to the same state when I left it last time. It usually happens when I close it because I think I’m done, and then I realize that I actually want to check one more thing with all the previous data that I had. Well, in standard Python REPL, to rerun all the previous commands, I would have to press “arrow up” + “enter” a lot. With IPython, I can just run this one command: %rerun ~1/, and it will execute all the commands from the previous session:

In [1]: a = 5

In [2]: b = 10

# Close IPython and open a new session

In [1]: %rerun ~1/
=== Executing: ===
a = 5
b = 10
=== Output: ===

In [2]: a
Out[2]: 5

In [3]: b
Out[3]: 10

But, there is one problem. If one of the past commands throws an exception, IPython will stop. It executes all the commands up to the exception, fails on the exception, and that’s it:

In [1]: a = 5

In [2]: b = 1/0
-----------------
ZeroDivisionError
----> 1 b = 1/0

ZeroDivisionError: division by zero

In [3]: c = 10

# Close IPython and open a new session

In [1]: %rerun ~1/
=== Executing: ===
a = 5
b = 1/0
c = 10
=== Output: ===
-----------------
ZeroDivisionError
      1 a = 5
----> 2 b = 1/0
      3 c = 10

ZeroDivisionError: division by zero

In [2]: a
Out[2]: 5

In [3]: c
--------------
NameError
----> 1 c

NameError: name 'c' is not defined

So if I want to execute the code after the exception, I have to figure out which command fails and rerun all commands after that line. For example, I can call: %rerun ~1/5-66, and this will run all commands from 5th until 66th command.

This has some limitations. First of all, you need to say which commands you want to run explicitly. And for that, you need to know the total number of commands for a given session. It’s no longer as convenient as just writing %rerun ~1/ and letting IPython figure out the rest.

Second of all - I don’t know about you, but I have plenty of exceptions in my IPython sessions. After all, that’s why IPython is for - experimenting with the code to find out what works and then copying that. So for each of the failed commands, I would have to exclude this line from the %rerun. This can quickly turn into a monstrosity like this: %rerun ~1/1-2 ~1/4-6 ~1/8-12 ~1/15-23 ...

It would be much better to temporarily ignore all the exceptions.

“Let’s write an extension that does that!” - I thought. “It probably won’t take longer than 5 minutes to explain everything, so it’s a perfect material for a lightning talk”.

It took longer than 5 minutes. Much longer. But it was an exciting journey!

Is there a plugin for that?

The first step was to check if someone already asked about this feature in the past. “Ignore exceptions” setting would be useful, so clearly someone must have thought about that before me.

And someone did - already in 2012: Add an ‘ignore exceptions’ mode to run commands in notebook. Starting from version 5.1, you can tag a cell in IPython notebooks with “raises-exception” to indicate that an exception is expected. IPython will ignore it and continue the execution. Unfortunately, it works only for the notebooks (so Jupyter Notebook), not for IPython REPL.

“Fine, I will do it myself.”

Let’s look inside the %rerun function and see how it works:

# IPython/core/magics/history.py

def rerun(self, parameter_s=''):
    opts, args = self.parse_options(parameter_s, 'l:g:', mode='string')
    if "l" in opts: # Last n lines
        n = int(opts['l'])
        hist = self.shell.history_manager.get_tail(n)
    elif "g" in opts: # Search
        p = "*"+opts['g']+"*"
        hist = list(self.shell.history_manager.search(p))
        for l in reversed(hist):
            if "rerun" not in l[2]:
                hist = [l] # The last match which isn't a %rerun
                break
        else:
            hist = [] # No matches except %rerun
    elif args: # Specify history ranges
        hist = self.shell.history_manager.get_range_by_str(args)
    else: # Last line
        hist = self.shell.history_manager.get_tail(1)
    hist = [x[2] for x in hist]
    if not hist:
        print("No lines in history match specification")
        return
    histlines = "\n".join(hist)
    print("=== Executing: ===")
    print(histlines)
    print("=== Output: ===")
    self.shell.run_cell("\n".join(hist), store_history=False)

Basically, it reuses the %history magic command to grab a list of commands from history and executes them.

I could monkey-patch this function and check if a given command is successful before adding it to the hist list. But monkey-patching external libraries is a bad idea. I would have to fork IPython and take care of upgrading it, or I would be stuck on the current version. So I needed a better way.

My first idea was to use one of IPython’s events. You can register a callback function when a specific event happens, for example, before or after you run a cell. When this event happens, IPython calls your callback function. That looked like a perfect solution! I could check if a given cell throws an exception and, if it does, skip it.

Writing a callback for IPython events

To register my custom callback, I needed to create an IPython extension (I wrote an article explaining how extensions work and how to create one):

# ~/.ipython/extensions/ignore_exceptions.py

def clean_input(info):
    try:
        exec(info.raw_cell)
    except Exception:
        info.raw_cell = ''

def load_ipython_extension(ipython):
    ipython.events.register('pre_run_cell', clean_input)

The load_ipython_extension registers clean_input callback that will be run before executing every cell. And clean_input does the following steps:

Takes an argument (info) containing various information about the cell we are about to run. This parameter is mandatory, and IPython will complain if we forget it.
Extracts the content of the cell (raw_cell).
Tries to run it.
And if it raises an exception, replaces that cell with an empty string.

We save that file in the ~/.ipython/extensions/ignore_exceptions.py directory and enable our extension with:

%load_ext ignore_exceptions

If we try it:

In [1]: %load_ext ignore_exceptions

In [2]: 1/0
ZeroDivisionError
----> 1 1/0

ZeroDivisionError: division by zero

…it doesn’t really work. There are multiple problems with this code:

First of all - we are using exec, which is not the best practice. But since the code we want to run is the same code we wrote before, I’m fine with that.
The info parameter can contain multiple commands. If we call %rerun 1-10, info will contain the first ten commands from the current session. If the exception happens, let’s say in the 5th command, our current implementation will ignore all the commands. But we want to ignore only that 5th one and still execute the other nine commands. So that’s not going to work.
And even if we didn’t have all those problems, overwriting the info.raw_cell doesn’t actually change the code that will be executed.

We need something better.

Input transformation

If we want to modify the input cells before they get executed, there is a feature exactly for that called “input transformation”. You write a function that takes the input, modifies it, and returns it. And you can register your input transformation as an extension, exactly as we did before.

There are two types of input transformers:

input_transformers_cleanup - they run first. At this point, the cell is not guaranteed to contain valid Python code (e.g., IPython-specific syntax, like the magic functions).
input_transformers_post - they run last, and here the cell contains only the Python code. In most cases, you want to use this one.

Input transformation solves all our problems (except for using the exec, but I will stick with it for simplicity). First of all, it’s an “input” transformation, so we can use it to change the input cell’s content. And since each line of the input is a separate element in a list, we can just execute them one by one and discard those that raise an exception. So this is what we are going to do:

def clean_input(lines):
    new_lines = []
    for line in lines:
        try:
            exec(line)
            new_lines.append(line)
        except Exception:
            pass
    return new_lines

def load_ipython_extension(ipython):
    ipython.input_transformers_post.append(clean_input)

Let’s load it and run it:

In [1]: %load_ext ignore_exceptions

In [2]: 1/0

In [3]: 1+2
Out[3]: 3
In [4]: my_number = 10

In [5]: my_number + 5
# Nothing is returned!

It’s working, but partially. If we assign a variable (In [4]) and then reference it (In [5]), you will notice that nothing is returned! That’s because my_number variable was saved in the local namespace of IPython, and the exec function call is running without access to those variables. Our extension tries to reference variable my_number, gets a NameError, and since our code is supposed to ignore errors, it ignores that line. Thus - nothing is printed!

We need to tell exec about all the variables that have been defined so far. If you check the signature of the exec() function, you will see that apart from the code to execute, we can also pass global and local variables. Bingo! Let’s grab the user_ns from the ipython object and pass it to the exec function. user_ns contains all the variables that were defined so far, but also all the imported modules, classes, etc.

We need to change this line:

exec(line)

into this:

exec(line, None, ipython.user_ns)

But the clean_input function doesn’t have access to the ipython object! And we can’t pass it as a parameter:

def clean_input(lines, ipython):

because input transformer functions need to have a specific signature and they can accept only one parameter - the list of lines. We could solve this problem using a class, but I prefer to use a technique called currying:

def curry_clean(ipython):
    def clean_input(lines):
        new_lines = []
        for line in lines:
            try:
                exec(line, None, ipython.user_ns)
                new_lines.append(line)
            except Exception:
                pass
        return new_lines
    return clean_input

def load_ipython_extension(ipython):
    ipython.input_transformers_post.append(curry_clean(ipython))

Let’s try again:

In [1]: %load_ext ignore_exceptions

In [2]: 1/0

In [4]: my_number = 10

In [5]: my_number + 5
Out[5]: 10

It’s working! Kind of…. We can declare a variable and reference it later, so that’s good. If the code throws an exception, it will be suppressed - that’s also what we want. But if we call a print() function, we get the output three times!

In [6]: print('hello world')
hello world
hello world
hello world

That’s because our input transformation function is actually called twice! Once in the should_run_async() function and then in the run_cell() function. That’s the behavior of version 7.15 of IPython - the latest one when I write this. And there is nothing you can do about it. If you ever decide to write some input transformations, keep in mind that they can run multiple times.

And when we actually execute the code, that’s the third print(). This “triple printing” can be mildly annoying, but maybe it’s not that bad? All we want to do is to rerun previous commands in one go, so we can live with duplicated output that happens only once.

But if printing happens multiple times, then every other command happens multiple times. And if we run a command that is not idempotent:

In [7]: my_number = 10

In [8]: my_number += 1

In [9]: my_number
Out[9]: 13

We get a bug. 10 incremented by 1 should give us 11, but instead, we get 13. Modifications that we do in try/except block get propagated back to our IPython session because we are accessing variables from that namespace. Remember the ipython.user_ns? It works both ways - for accessing and assigning variables.

If we don’t want to modify the original namespace, let’s make a copy and execute our code there! Except that we would need to do a deep copy. Why a deep copy? Because if we have mutable variables (like a list) and do a shallow copy, modifying the mutable variable in the copied namespace will still affect the original namespace. We need to cut the connections between both namespaces completely, so we use the deep copy.

Deep copy basically pickles and un-pickles every object. Do you know what can’t be pickled? Modules. And do you know what IPython’s (or Python’s) namespace contains, except for functions and variables? Modules! So deep copy is going to crash.

Unless we filter the namespace and only pickle things that can be pickled! In the end, I only need to copy variables. I won’t be able to modify modules anyway, so it’s fine to leave them in the namespace as they are. Let’s add a new function for this:

from copy import deepcopy

def copy_namespace(local_ns):
    copy_ns = {}
    for key, value in local_ns.items():
        try:
            copy_ns[key] = deepcopy(value)
        except TypeError:
            # TypeError is raised when an object can't be pickled
            copy_ns[key] = value
    return copy_ns

And here are two changes that we need to make in the curry_clean function:

def curry_clean(ipython):
    def clean_input(lines):
        new_lines = []
+ local_ns = copy_namespace(ipython.user_ns)
        for line in lines:
            try:
- exec(line, None, ipython.user_ns)
+ exec(line, None, local_ns)
                new_lines.append(line)
            except Exception:
                pass
        return new_lines
    return clean_input

We are almost there, but there are two problems left:

We still get a lot of duplicated output for print and similar commands.
Our function doesn’t work for multiline strings (so all for-loops, function definitions, and similar multiline statements will crash it).

Let’s solve them now. First, the duplicated output. We can suppress it by redirecting the standard output to /dev/null:

import os
import sys

sys.stdout = open(os.devnull, 'w')
print("I'm invisible")

If we run the above code, the output of a print statement won’t be displayed. Actually, no other output will be displayed afterward, so we need to set the stdout back to the original value after the try/except check.

If we want to execute some code, run a function, and then execute more code, the best tool to use is a context manager. Let’s write one:

import os
import sys
from contextlib import contextmanager

@contextmanager
def no_output():
    original_stdout = sys.stdout
    sys.stdout = open(os.devnull, 'w')
    yield
    sys.stdout = original_stdout

I’m using the contextmanager decorator to turn my function into a context manager (it saves me from writing boilerplate code). Here is what my function does:

Store the original stdout in a variable.
Replace the stdout with /dev/null (whatever you write to dev/null is gone).
Yield - so run the code inside the context manager.
And, finally, restore stdout to the initial value.

Now, we wrap our exec function with this context manager:

for line in lines:
+ with no_output():
        try:
            exec(line, None, local_ns)
            new_lines.append(line)
        except Exception:
            pass
return new_lines

Uff, almost there! The last part is to enable running multiline code. For that, we just need to check if the line starts with a whitespace character. If it does, combine it with the previous line.

Let’s write a helper to transform multiline strings into one-line strings:

def combine_multiline(lines):
    new_lines = []
    for line in lines:
        if line.startswith(' ') and new_lines:
            new_lines[-1] += line
        else:
            new_lines.append(line)
    return new_lines

The second part of the if line.startswith('') and new_lines is just a sanity check to make sure that new_lines already contains items. Normally, we wouldn’t bother with checking for this. If the first line starts with spaces, there is something wrong with it, and it will fail during the execution anyway.

Let’s call our function in the correct place:

def clean_input(lines):
    new_lines = []
+ lines = combine_multiline(lines)

And finally, our hackish input transformation seems to be working!

Seems to, because there is an interesting bug. If we try to rerun a command that reloads our extension (%reload ignore_exceptions), IPython will get into an infinite loop. It will try to add clean_input callback again. This will run the callback itself. The callback will try to execute %reload ignore_exceptions that will register clean_input that will call the callback, and so on. Until it crashes.

We can solve this problem by adding the “unload” function that lets you disable an extension. In our case, we want to remove clean_input from the list of input_transformers_post:

def unload_ipython_extension(ipython):
    ipython.input_transformers_post = [
        f for f in ipython.input_transformers_post if f. __name__!= 'clean_input'
    ]

The above code goes through the list of all input transformers and removes the one called “clean_input”. We can’t replace the if f. __name__!= 'clean_input' with if f == curry_clean(ipython) because each time we call curry_clean(ipython) it creates a different object. So the curry_clean function added to the input transformers list has a different id than curry_clean called in the comparison. We could use a singleton design pattern if we want, but we might just compare functions’ names, which will work just fine.

With unload in place, our extension is finally ready. You can activate it, run some code, and notice that no exceptions are raised. You can also deactivate it with:

%unload_ext ignore_exceptions

Meet %rerunplus: %rerun that ignores exceptions

But how can we use it in combination with %rerun?

The simplest way is to create a new magic function that wraps a call to %rerun in load/unload functions. Let’s make a new file inside ~/.ipython/profile_default/startup/ directory and define our magic function there. During startup, IPython automatically executes files defined in this directory. It’s a good place to put magic functions that we want to enable in each IPython session.

from IPython.core.magic import register_line_magic

@register_line_magic
def rerunplus(line):
    get_ipython().run_line_magic('load_ext', 'ignore_exceptions')
    get_ipython().run_line_magic('rerun', line)
    get_ipython().run_line_magic('unload_ext', 'ignore_exceptions')

def load_ipython_extension(ipython):
    ipython.register_magics(rerunplus)

Your code editor will complain that the get_ipython() is not defined, but actually, that’s a global function in IPython, so it’s fine. If you restart IPython, you will now have access to a %rerunplus magic function. Let’s give it a try.

First, run some code with exceptions:

a = 1
b = 2
1/0
c = 3
myfile = open('beep')
d = 4

Next, close and reopen IPython. If you try to rerun all the commands using the standard %rerun, it will fail at 1/0 exception. a and b variables will be set, but c and d won’t. But if we use our new, cool %rerunplus, it will ignore all the exceptions and set all variables correctly:

In [1]: %rerunplus ~1/
=== Executing: ===
a = 1
b = 2
1/0
c = 3
myfile = open('beep')
d = 4
=== Output: ===

In [2]: a
Out[2]: 1

In [3]: b
Out[3]: 2

In [4]: c
Out[4]: 3

In [5]: d
Out[5]: 4

Here is the complete code from this article.

Conclusions

We have built an improved version of %rerun that ignores exceptions. Is it efficient? Nope. It’s a hackish solution that you should use only with your own code, especially since we use exec. exec should never be used to run some code that you have no control over! But those 60 lines of code are good enough to fix one of the biggest shortcomings of the original %rerun command.

P.S.

What happens if we try to %rerun code that contains %rerun instruction? We get a RecursionError. This is a bug that both standard %rerun and my %rerunplus share. So if you got curious about digging into IPython, you can take my extension and try to fix it on your own.

My Favorite CLI Tools

Sebastian Witowski — Fri, 19 Jun 2020 00:00:00 +0000

Previously, I wrote about my favorite Mac apps. But I spend half of my time in the terminal, and I have a handful of CLI tools that makes my life easier. Here are some of them:

Tools that I use every day

fish shell

Shell - the most important tool that you use every time you open the terminal. I’ve used Bash and Z shell in the past, and currently, I’m using fish. It’s a great shell with plenty of features out of the box, like the autosuggestions, syntax highlighting, or switching between folders with ⌥+→ and ⌥+←.

On the one hand, this makes it perfect for beginners, because you don’t have to set up anything. On the other hand, because it’s using a different syntax than other shells, you usually can’t just paste scripts from the internet. You either have to change the incompatible commands to fish scripts or start a Bash session to run the bash scripts. I understand the idea behind this change (Bash is not the easiest language to use), but it doesn’t benefit me in any way. I write bash/fish scripts too seldom to memorize the syntax, so I always have to relearn it from scratch. And there are fewer resources for fish scripts than for bash scripts. I usually end up reading the documentation, instead of copy-pasting ready-made scripts from StackOverflow.

Do I recommend fish? Yes! Switching shells is easy, so give it a try. Especially if you don’t like to tinker with your shell and want to have something that works great with minimal configuration.

Fish plugins

You can add more features to fish with plugins. The easiest way to install them is to use a plugin manager like Fisher, Oh My Fish, or fundle.

Right now, I’m using Fisher with just three plugins:

franciscolourenco/done - sends a notification when a long-running script is done. I don’t have a terminal open all the time. I’m using a Guake style terminal that drops down from the top of the screen when I need it and hides when I don’t. With this plugin, when I run scripts that take longer than a few seconds, I get a macOS notification when they finish.
evanlucas/fish-kubectl-completions - provides autocompletion for kubectl (Kubernetes command line tool).
fzf - integrates the fzf tool (see below) with fish.

I had more plugins in the past (rbenv, pyenv, nodenv, fzf, z), but I switched to different tools to avoid slowing down my shell (a mistake that I did in the past with Z shell).

If you want to see more resources for fish, check out the awesome-fish repository. Compared with Z shell and Bash, fish has fewer plugins, so it’s not the best option if you want to tweak it a lot. For me - that’s a plus. It stops me from enabling too many plugins and then complaining that it’s slow 😉.

Starship

If I had to choose one favorite tool from this whole list - it would be Starship. Starship is a prompt that works for any shell. You install it, add one line of config to your .bashrc/.zshrc/config.fish, and it takes care of the rest.

It shows:

git status of the current directory and different symbols, depending on if you have new files, pending changes, stashes, etc.
Python version if you are in a Python project folder (the same applies to Go/Node/Rust/Elm and many other programming languages)
How long it took the previous command to execute (if it was longer than a few milliseconds)
Error indicator if the last command failed

Starship prompt in action

And a bazillion other information. But, in a smart way! If you are not in a git repository, it hides the git info. If you are not in a Python project - there is no Python version (because there is no point in displaying it). It never overwhelms you with too much information, and the prompt stays beautiful, useful, and minimalistic.

Did I mention that it’s fast? It’s written in Rust, and even with so many features, it’s still faster than all my previous prompts! I’m very picky about my prompt, so I was usually hacking my own version. I was taking functions from existing prompts and gluing it together to make sure I only have things that I need and it stays fast. That’s why I was skeptical about Starship. “There is no way that an external tool can be faster than my meticulously crafted prompt!” Well, I was wrong. Give it a try, and I’m sure you are going to love it! Huge kudos to the creators of Starship!

z

“z” lets you quickly jump around your filesystem. It memorizes the folders that you visit, and after a short learning time, you can move between them using z path_of_the_folder_name.

For example, if I often go to folder ~/work/src/projects, I can just run z pro and immediately jump there. z’s algorithm is based on frecency - a combination of frequency and recency that works very well. If it memorizes a folder that you don’t want to use, you can always remove it manually.

It speeds up moving between commonly visited folders on my computer and saves me a lot of keystrokes (and path memorization).

fzf

fzf stands for “fuzzy finder”. It’s a general-purpose tool that lets you find files, commands in the history, processes, git commits, and more using a fuzzy search. You type some letters, and it tries to match those letters anywhere in the list of results. The more letters you type, the more accurate the results are. You probably know this type of search from your code editor - when you use the command to open a file, and you type just part of the file name instead of the full path - that’s a fuzzy search.

I use it through the fish fzf plugin, so I can search through command history or quickly open a file. It’s another small tool that saves me time every day.

fd

find (left) versus fd (right)

Like the find command but much simpler to use, faster, and comes with good default settings.

You want to find a file called “invoice,” but you are not sure what extension it has? Or maybe it was a directory that was holding all your invoices, not a single file? You can either roll up your sleeves and start writing those regex patterns for the find command or just run fd invoice. For me, the choice is easy 😉.

By default, fd ignores files and directories that are hidden or listed in the .gitignore. Most of the time - that’s what you want, but for those rare cases when I need to disable this feature, I have an alias: fda='fd -IH'.

The output is nicely colorized and, according to the benchmarks (or the GIF above), it’s even faster than find.

ripgrep

Poor grep (on the left) was running for ages through all the node_modules before finding something useful

In a similar manner to fd mentioned above, ripgrep is an alternative to the grep command - much faster one, with sane defaults and colorized output.

It skips files ignored by .gitignore and hidden ones, so you will probably need this alias: rga='rg -uuu'. It disables all smart filtering and makes ripgrep behave as standard grep.

htop and glances

The most common tool to show information about processes running on Linux or Mac is called top. It’s the best friend of every system administrator. And, even if you are mostly doing web development like me, it’s useful to see what’s going on with your computer. You know, just to see if it was Docker or Chrome that ate all your RAM this time.

htop is an excellent alternative for top

top is quite basic, so most people switch to htop. htop is top on steroids - colorful, with plenty of options, and overall more comfortable to use.

glances gives you a quick overview of your system

glances is a complementary tool to htop. Apart from listing all the processes with their CPU and memory usage, it also displays additional information about your system.

You can see:

network or disks usage
used and total space on your filesystem
data from different sensors (like the battery)
and a list of processes that recently consumed an excessive amount of resources

I still use htop for faster filtering and killing processes, but I use glances to quickly glance at what’s going on with my computer. It comes with API, Web UI, and various export formats, so you can take system monitoring to the next level. I highly recommend it!

virtualenv and virtualfish

Virtualenv is a tool for creating virtual environments in Python (I like it more than the built-in venv module).

VirtualFish is virtual environment manager for the fish shell (if you are not using fish, check out virtualenvwrapper). It provides a bunch of additional commands to create, list, or delete virtual environments quickly.

pyenv, nodenv, and rbenv

pyenv makes it easy to switch Python versions

Pyenv, nodenv, and rubyenv are tools for managing different versions of Python, Node, and Ruby on my computer.

Let’s say you want to have two versions of Python on your computer. Maybe you are working on two different projects, or you still need to support Python 2. Managing different Python versions is hard. You need to make sure that different projects install packages with the correct version. If you are not careful, it’s easy to mess up this fragile setup and overwrite binaries used by other packages.

Version manager helps a lot and turns this nightmare into a pretty manageable task. Good version manager can swap the Python version globally or “per folder”. And every version is isolated from others.

I’ve recently found a tool called asdf that can replace pyenv, nodenv, rbenv, and other *envs with one tool to rule them all. It provides version management for pretty much any programming language, and I will definitely give it a try next time I need to set up a version manager for a programming language.

pipx

Virtualenv solves many problems with package management in Python, but there is one more use case to cover. If I want to install a Python package globally (because it’s a standalone tool, like glances mentioned before), I have a problem. Installing packages outside of a virtual environment is a bad idea and can lead to problems in the future. On the other hand, if I decide to use a virtual environment, then I need to activate that virtual environment each time I want to run my tool. Not the most convenient solution either.

It turns out that pipx tool can solve this problem. It installs Python packages into separate environments (so there is no risk that their dependencies will clash). But, at the same time, CLI commands provided by those tools are available globally. So I don’t have to activate anything - pipx will do this for me!

ctop and lazydocker

ctop in action (source: https://github.com/bcicen/ctop)

Both of those tools are useful when you are working with Docker. ctop is a top-like interface for Docker containers. It gives you:

A list of running and stopped containers
Statistics like memory usage, CPU, and an additional detailed window for each container (with open ports and other information)
A quick menu to stop, kill, or show logs of a given container

It’s so much nicer than trying to figure out all this information from docker ps.

lazydocker is my favorite Docker tool (source: https://github.com/jesseduffield/lazydocker)

And if you think that ctop was cool, wait until you try lazydocker! It’s a full-fledged terminal UI for managing Docker with even more features. My favorite tool when it comes to Docker!

Tools that I don’t use every day

Apart from the tools that I use almost every day, there are some that I collected over the years and found them particularly useful for specific tasks. There is something to record GIFs from the terminal (that you can pause and copy text from!), list directory structure, connect to databases, etc.

Homebrew

Homebrew needs no introduction if you are using a Mac. It’s a de facto package manager for macOS. It even has a GUI version called Cakebrew.

asciinema

asciinema is a tool that you can use to record your terminal sessions. But, unlike recording GIFs, it will let your viewers select and copy the code from those recordings!

It’s a great help for recording coding tutorials - not many things are as frustrating as typing long commands because the instructor didn’t provide you with code snippets.

colordiff and diff-so-fancy

colordiff brings some colors to your diffs

I rarely do diffs (compare differences between two files) in the terminal anymore, but if you need to do one, use colordiff instead of the unusable diff command. colordiff colorizes the output, so it’s much easier to see the changes instead of trying to follow all the “<” and “>” signs.

diff-so-fancy - even better alternative to colordiff

For running git diff and git show commands, there is an even better tool called diff-so-fancy. It further improves how the diff looks like by:

highlighting changed words (instead of the whole lines)
simplifying the headers for changed files
stripping the + and - symbols (you already have colors for this)
clearly indicating new and deleted empty lines

tree (`brew install tree`)

If you want to present the content of a given directory, tree is a go-to tool to do that. It displays all the subdirectories and files in a nice, tree-like structure:

$ tree .
.
├── recovery.md
├── README.md
├── archive
├── automator
│   ├── Open\ Iterm2.workflow
│   │   └── Contents
│   │   ├── Info.plist
│   │   ├── QuickLook
│   │   │   └── Thumbnail.png
│   │   └── document.wflow
│   └── Start\ Screen\ Saver.workflow
├── brew-cask.sh

bat

Like cat (command most commonly used to display the content of a file in a terminal) but better.

Adds syntax highlighting, git gutter marks (when applicable), automatic paging (if the file is large), and in general, makes the output much more enjoyable to read.

httpie

httpie in action (source: https://httpie.org/)

If you need to send some HTTP requests and you find curl unintuitive to use, try httpie.

It’s an excellent alternative. It’s easier to use with sensible defaults and simple syntax, returns a colorized output, and even supports installing additional plugins (for different types of authentication).

tldr

Simplified man pages. “man pages” contain manuals for Linux software that explain how to use a given command. Try running man cat or man grep to see an example. They are very detailed and sometimes can be difficult to grasp. So tldr is a community effort to extract the essence of each man page into a brief description with some examples.

tldr works for the most popular software. As I said, it’s a community effort, and there is a slim chance that someone will document an obscure package for you. But when it works, the information it provides usually contains what you are looking for.

For example, if you want to create a gzipped archive of a few files, man tar will overwhelm you with the possible options. tldr tar will instead list some common examples - the second one being exactly the thing that you want to do:

man pages are great, but sometimes using tldr will be much faster to find a specific information

exa

exa can be a replacement for the ls command.

It’s colorful, displays additional information like the git status, automatically converts file size to human-readable units, and all that while staying equally fast to ls.

Even though I like it and recommend it, for some reason, I still stick with ls instead. Muscle memory, I guess?

litecli and pgcli

My go-to CLI solutions for SQLite and PostgreSQL. With the auto-completion and syntax highlighting, they are much better to use than the default sqlite3 and psql tools.

mas

mas is a CLI tool to install software from the App Store. I used it once in my life - when I was setting up my Macbook. And I will use it to set up my next Macbook too.

mas lets you automate the installation of software in macOS. It saves you from a lot of clicking. And, since you are reading an article about CLI tools, I assume that - just like me - you don’t like clicking.

I keep a list of apps installed from the App Store in my “disaster recovery” scripts. If something bad happens, I hopefully should be able to reinstall everything with minimal hassle.

ncdu

Disk usage analyzer for the terminal. Fast and easy to use. My default tool when I need to free some space (“Ohh, I’m sure that 256GB of disk space will be plenty!”).

That’s all folks

It was a long list, but hopefully, you discovered something new today.

Some of the tools like the fd, ripgrep, or httpie are improved versions of things that you probably already know. Except that the new versions are easier to use, they provide better output, and sometimes are actually faster. So don’t cling to old tools only because everyone else is using them.

A common argument for sticking with the “standard Linux tools” that I hear is:

But what if you need to log in to a Linux server and do some work there? You won’t have access to your fancy tools. It’s better to learn how to use tools that come built-in with most Linux distributions.

When was the last time you had to log in to a Linux server? One where you can’t install software, but you had to debug some issues manually? I don’t even remember. Not many people do that anymore. Maybe it’s time to rethink how you do the deployment and move away from manual work into something more scalable?

Don’t let your tool-belt get rusty and add some new CLI tools there!

Many of the tools that I mentioned are related to Python programming. If you want to learn more and see how I use them, I’ve made a free video for PyCon 2020 conference called "Modern Python Developer's Toolkit".

It's a two-hour-long tutorial on how to set up a Python development environment, which tools to use, and finally - how to make a TODO application from scratch (with tests and documentation). You can find it on YouTube.

18 Plugins for Writing Python in VS Code

Sebastian Witowski — Mon, 27 Apr 2020 00:00:00 +0000

VS Code is a great text editor. But when you install it, its functionality is limited. You can edit JavaScript and TypeScript, but for other programming languages, it will be just a text editor. You will need to add some plugins to turn it into a proper IDE.

Luckily, when you open a file in a new language, VS Code will suggest an extension that can help you. With the Python extension, you can already do a lot - you get syntax highlighting, code completion, and many other features that turn a text editor into a code editor.

But there are many other plugins that I discovered when working with Python. Some add entirely new functionality, and others offer just a small improvement here and there. I’ve decided to write them down. I hope some of you will find them useful!

Python and other language-specific plugins

First and foremost - the Python plugin for VS Code. Out of the box, there is no support for Python in VS Code, but when you open a Python file, VS Code will immediately suggest this plugin. It adds all the necessary features:

Syntax highlighting for Python files
Intellisense (code-completion suggestions)
Ability to start a debugger
Support for collecting and running tests (with different testing frameworks like pytest or unittest)
Different linters
And plenty of other small features that turn VS Code into a proper Python editor

And it’s the same with different languages. Each time you open a file that VS Code doesn’t support, you get a suggestion of a plugin for that language. It’s a great approach! On the one hand, you don’t have to figure out which extensions you need to install, but on the other hand, you don’t slow down your IDE with plugins that you will never use.

Django and other framework-specific plugins

If you are working with frameworks, there is usually a plugin that will make your life easier, like Django or flask-snippets. They bring some additional improvements for a given framework like:

Better syntax highlighting for framework-specific files (e.g., template files in Django that combine HTML with Django tags)
Additional snippets - especially useful for the templating systems. Being able to insert loops and if-s with a two letter shortcut without opening and closing all those {% tags is a blessing!
Improved support for different functions. For example, Django plugin adds the ability to “Go to definition” from the templates.

IntelliCode

Source: https://docs.microsoft.com/en-us/visualstudio/intellicode/intellicode-visual-studio-code

Intellicode makes the autocompletion a bit smarter. It tries to predict which term you are most likely to use in a given situation and puts that term at the top of the list (marked with a ☆ symbol).

It works surprisingly well!

Emmet

Source: https://code.visualstudio.com/docs/editor/emmet

Technically, Emmet is not an extension because it’s already integrated with VS Code by default (due to its huge popularity). But it still deserves mention, in case there is someone who never heard about it.

Emmet is going to be your best friend if you are writing a lot of HTML and CSS. It lets you expand simple abbreviations into full HTML, it adds CSS prefixes (together with vendor prefixes), and a whole bunch of other useful functions (rename a tag, balance in/out, go to matching pair, etc.)

I absolutely love it when I need to write HTML. I started using it to quickly add a class to a tag (div.header or a.btn.btn-primary) and then I learned new features. With Emmet you can write:

ul>li.list-item*3

and if you press Enter, it will turn into:

<ul>
  <li class="list-item"></li>
  <li class="list-item"></li>
  <li class="list-item"></li>
</ul>

Autodocstring

This plugin speeds up writing Python documentation by generating some of the boilerplate for you.

Write a function signature, type """ to start the docstring, press Enter, and this plugin does the rest. It will take care of copying the arguments from the function signature to the docs. And if you add types to your arguments, it will recognize them and put them in the correct place in the documentation.

Bookmarks

This extension lets you bookmark locations in your code, easily list all your bookmarks in a sidebar, and move between them with keyboard shortcuts.

It’s incredibly useful then I’m digging into a new codebase (so I can jump around and not get lost). I also find it helpful when I’m trying to debug some complicated issues - VS Code has a functionality to “Go to Previous/Next location”, but without bookmarks, it’s easy to get lost.

Dash

With Dash extension, you can access offline documentation for basically any programming language or framework.

It requires installing one of the additional tool to provide the documentation:

Once you download the documentation, you can access it offline.

I’m not using it very often, but it’s a great tool if you need to work without access to the Internet.

Error Lens

Sometimes the errors marks in VS Code are hard to spot (especially the “info” hints). If you don’t wrap lines, it’s even worse - the error can be in the part of the code not visible on the screen.

That’s why I’m using Error Lens. It lets me modify how the errors should be displayed. It can display the error message next to the line where it occurs and a Sublime-like error icons in the gutter (next to the line number).

File Utils

This small plugin adds a few file-related commands to the Command Palette (normally you can perform them by right-clicking in the sidebar):

Rename
Move
Duplicate
Copy path or name of the file

It also adds a “Move/Duplicate File” option to the context menu.

GitLens

Massive plugin - adds a lot of git integration to VS Code:

Can show blame annotations per line, per file, in the status bar, or on hover.
Provides you with context links to show changes, show diff, copy commit ID.
Brings a sidebar with probably every possible information about the git repository, file and line history, compare and search menus, etc.

It’s much more powerful than the default “source control” panel of VS Code. I don’t think I’m using even 20% of its features.

indent-rainbow

Very helpful plugin for working with languages like Python, where indentation matters. Every level of indentation gets a slightly different color, so it’s easier to see at a glance where a given code block ends.

jumpy (or MetaGo)

jumpy is a very peculiar plugin that takes some time to get used to. Basically, it’s supposed to help you move around your code faster.

If you press a keyboard shortcut, jumpy will display a 2-letter code next to every word on the screen. If you type those two letters, your cursor will jump to that location. Similar to what you can do with vim in “normal” mode (with less typing).

Paste and Indent

If you find that VS Code is not doing a good job when you paste code, try this extension. It will let you assign a “Paste and Indent” action to any key shortcut. This command will do its best to indent the code correctly after you paste it (to match the surrounding code). I’m using the “Command+Shift+V” shortcut for it.

Project Manager

VS Code supports the concept of workspaces - you can group some files and folders together and easily switch between them. But you still need to save the workspace configuration, and sometimes it can get lost - I either accidentally remove it or forget where I saved it.

Project Manager takes this hassle away. You can save projects and then open them, no matter where they are located (and you don’t have to worry about storing the workspace preference files). Also, it adds a sidebar to browse all your projects.

Quick and Simple Text Selection

I like to use shortcuts that let me select all the text in brackets, tags, etc. By default, VS Code has command to “Expand/Shrink selection” that works ok-ish, but I found the Quick and Simple Text Selection plugin to be a much better way.

It adds a few new shortcuts to select text in:

single/double quotes
parentheses
square/angular/curly brackets
tag

I tried to map them to some intuitive shortcuts and they work like a charm:

Command + ‘ (⌘ + ‘) - select text in single quotes
Command + “ (⌘ + ⇧ + ‘)- select text in double quotes
Command + ( (⌘ + ⇧ + 9)- select text in parentheses
Command + < (⌘ + ⇧ + ,)- select text in tag
Command + , (⌘ + ,)- select text in angular brackets

Settings Sync

It’s not really related to Python, but it’s a very important plugin, so I wanted to mention it.

Settings Sync lets you save the VS Code settings to a private GitHub gist, so you can easily restore them if you switch to a different computer (or if you lose/destroy your current one).

In one of the upcoming versions of VS Code, settings synchronization will become built-in.

TODO Highlight

Highlights all TODO/FIXME/NOTE in the code, so you can easily spot them. You can easily customize it by adding new words and changing the highlight style.

Spell Right

It’s strange, but VS Code doesn’t have a built-in spell checker. So you have to install one as an extension.

My Favorite Macbook Tools

Sebastian Witowski — Thu, 09 Jan 2020 00:00:00 +0000

I could spend days just tweaking things on my computer. Actually, I do. Whenever I see something annoying, I want to drop everything and try to fix it right away. It can be anything. From a minor: “Hey, I just run the same command twice, I should create an alias!” to installing random tools (“Hmm, I’m wondering if there is a way to get notifications when a long-running job in a terminal finishes, so I can do other stuff in the meantime?”). I also love to read what tools other people are using. This time, I’ve decided to share mine.

Since this is a very long post, I’ve to split it into two parts: the Mac apps and the CLI stuff (coming soon!).

Alfred

Alfred is a launcher - this app that opens when you press ⌘+Spacebar and lets you quickly open other apps. On steroids. And I absolutely love it! I knew about its existence even before owning a Mac, and I envied Mac users to have such a great launcher. I used Launchy on Windows and GNOME Do on Linux, and they are both great (just having a launcher will make you feel much more productive).

So when I got my Mac, one of the first things I did was to replace Spotlight with Alfred. Even without the PowerPack, it’s much more powerful than Spotlight. But the PowerPack (a paid extension) is where it really packs a punch. It enables a few more features like a text expansion (here called “snippets”) or the clipboard history. But the best features that come with the Powerpack are the “Alfred Workflows” - user-defined scripts that let you automate many tasks. I have a bunch of workflows to quickly create reminders, to-do tasks, temporary emails, convert currency, or search for emoji and Unicode characters.

There is already version 4 of Alfred, but I still haven’t upgraded from version 3.

My Alfred workflows

Here are some of the workflows that I’m using with Alfred:

Alfred Workflow Todoist - lets me quickly add tasks to Todoist
Convert - convert from one unit to another. I use it mostly for currencies.
EggTimer - a bit outdated workflow, but the only one that lets me easily create timers (I want to take a short break every 60 minutes of work)
Emoji search - much faster emoji search than the default emoji icon panel on Mac.
Reminders for Alfred 3 - typing “r do stuff in 20 minutes” will create a reminder “do stuff” 20 minutes from now. It lets me quickly set reminders without getting distracted from the current task.
TemporaryEmail - create a disposable temporary email when I need to register in some spammy service.
Toggl - interact with Toggl time tracker
Unicode Symbol Search - quickly find and copy any Unicode symbol.

Magnet

Magnet is a window manager. It’s unbelievable that Mac still doesn’t have a built-in window manager that supports keyboard shortcuts. Luckily, Magnet fills this gap (for a price of a few $). By default, it uses ⌃+⌥ as a modifier, which conflicts with a lot of my VS Code settings (I’m using ⌃+⌥ as a modifier for all my custom shortcuts). So I changed it to ⌘+⌥+⌃. It sounds like a lot of keys to press simultaneously, but I mapped all 3 of them to 1 key using Karabiner-Elements (mentioned below).

If you prefer a free alternative, there is also Rectangle - a successor of the discontinued Spectacle app. I started using Magnet because Spectacle didn’t have some shortcuts (like “Split into 1/3 of the screen”). But Rectangle has even more options than Magnet, so it looks like an excellent replacement.

If you miss a tiling window manager, there is also Amethyst. I tried it, but it didn’t feel as good as i3, so I didn’t stick with it. It was missing the basic functionality, like moving into a specific direction with a modifier + arrow key.

Todoist

My to-do list manager. I tried using different apps (gosh, I think I’ve tested all of them), and I was always coming back to Todoist. I like it for its clean interface, the “Next 7 days” view, and the ability to use special shortcuts when creating tasks. For example, if I add a task: “Write a blog post tod p1 #blog” it will create a “Write a blog post” task, set due date to today, add “Priority 1” flag, and assign it to a project called “Blog.” Neat!

Dozer

This is how my menu bar looks like most of the time:

This is how it really looks like:

Dozer is a small application that keeps my menu bar manageable. It lets me hide those items that I’m not using very often. And with a keyboard shortcut, I can quickly show/hide them. If you need more features, there is also a paid app called Bartender, but Dozer works like a charm for me.

BetterTouchTool

BetterTouchTool (or BTT for short) lets you customize any input device for your Mac. You can create custom Trackpad or Magic Mouse gestures, custom keyboard shortcuts, and macros (sequences of keys). It comes with a clipboard manager and window manager. It could replace a bunch of other tools that I’m using.

But none of those features are the reason why I decided to pay $10 for this app. I bought it so I can customize my Macbook’s touch bar. In its original form, the touch bar was unusable for me. Buttons that change depending on which application you are using is one of the worst ideas ever. I don’t want to rediscover what’s on the touch bar each time I change the currently used app. And half of the buttons are useless anyway. I don’t want a slider for the volume or brightness! 99% of the time, I only need to increase or decrease it by a notch. I should be able to do this with one press of a button. So, as most of the people that I know, I didn’t even use the touch bar. And one day, I stumbled upon an article on HackerNews that was explaining how to make the touch bar actually useful. After reading this article, I immediately bought the BTT.

This is how my touch bar looks like today

Now on my touch bar, there are (from left to right):

Esc button (that I don’t really use, but you can’t remove it).
A special button that will switch the touch bar back to the default, crappy Mac version (again, it’s there by default, and I don’t use it).
Buttons that open (or switch to) the Calendar, Mail, Todoist, and Toggl apps. I use them from time to time (mostly when I’m working with Macbook on my laps, otherwise it’s faster for me to switch with the keyboard).
A script that shows the currently playing song from Spotify or Youtube. It’s only visible when one of those apps is open. If I press the name of the song, it will skip to the next one. Best button! Ever!
Play/Pause button for Spotify/Youtube.
Brightness down and up buttons.
Volume down, up, and mute buttons.

There is a ton of other widgets that you can use - weather, temperature, time and date, custom shell, or Apple Scripts. The BTT author has added a lot of new ones since I set up my touch bar, so I will have to revise it at some point.

Franz

One app to aggregate all my messaging apps. It’s a great, free tool that supports most of the messaging apps. I’m using it with Slack, Skype, WhatsApp, Telegram, Twitter, Hangouts, and Facebook Messenger. The only downside is - each service that you add spins a new electron app. The memory consumption can go through the roof sometimes. But so far, I haven’t found any reliable alternative.

Flux

Flux adapts the color of the screen to the time of the day. Early in the day or late in the night (for most of us, programmers, it’s usually the same thing), it will make the colors of your computer warmer (yellow). It’s one of those applications that you install once, and you forget about it.

Karabiner-Elements

Karabiner-Elements is a free app to customize your keyboard (it has a lot of duplication with BetterTouchTool). You can swap keys, change the behavior of function keys, or define some complex macros. I use it for three things:

I have swapped the Caps Lock and Esc keys (and I can’t believe how I could live before doing it). Luckily, I discover this trick before I switched to Macbook with its useless Esc key on the touch bar.
Since I’m using Kinesis Advantage 2 keyboard, I have remapped the Home key to ⌃(Ctrl) and End to a combination of ⌘+⌥+⌃ (which is a modifier that I use for window management with Magnet). This combination (often with the addition of Shift) is called the “Hyper” key, and it gives you another modifier key to use with your shortcuts. Just in case having three modifier keys on Mac is not enough.
If you are using a “normal” keyboard, I saw people mapping “pressing Ctrl” to act like the Esc key and “holding Ctrl” to act like the Ctrl key. Which is pretty smart, since we never just press Ctrl - it’s always used in combination with other keys. This will free up the “Esc” (or “Caps Lock” if you followed my previous advice and swapped them) for your Hyper key or any kind of crazy combination that you can think of.
A careful reader might notice that I no longer have a way to enable the Caps Lock. The “Caps Lock” key is now my Esc, and Macbook doesn’t let you redefine the “Esc” key on the touch bar. So I have 2 Esc keys and no Caps Lock. Which is bad - how am I supposed to argue with people on the Internet without the Caps Lock? Luckily, there was a solution for that - I mapped pressing both Shift keys simultaneously to Caps Lock. It sounds crazy, but it’s actually quite intuitive and easy to remember. I have no idea why it’s not a default behavior. To make it work, I had to modify the Karabiner-Elements config file directly. You can check my dotfiles repository to see how to do this. Just remove the stuff that you don’t need and put this karabiner.json in your ~/.config/karabiner/karabiner.json file.

Fish

When I first started using Linux, I used Bash. Then I switched to Z shell. With oh-my-zsh. Then I realized that it’s becoming a bit sluggish, mostly during the startup time. I decided that I don’t have time to sit down and stare at my screen for almost 2 seconds waiting for it to load. So I switched to prezto when it was a cool, new thing (and not an abandoned package as it is today). In 2017 I switched to a Macbook. Which was a great opportunity to try something new! Some of my colleagues at work were using fish at that time, and it looked interesting. It had some neat features out of the box, like the autosuggestions, syntax highlighting, or switching between directories using ⌥+→ and ⌥+←. So I tried it. And I liked it, so I’m still using it today. I use fisher package manager with a few plugins (there is also oh-my-fish, but fisher was supposed to be faster, so I stick to it). I try to avoid adding too many plugins to make the terminal startup faster (I’m starting a bunch of terminal sessions on an average day, so those seconds add up).

Captin

It shows you a notification when you turn the Caps Lock on or off. Since I’m using a slightly unusual way to operate the Caps Lock, I like to have this little visual (and audio) cue when I turn it off.

Anki

Anki is a spaced repetition app. For most of my life I thought that I was stupid because I had terrible problems with memorizing things. It turns out that I was using the wrong technique. Spaced repetition is a simple mechanism that makes sure that you don’t repeat things that you already remember, but only those things that you have problems remembering. I’m using Anki purely on my phone, but adding new words is much easier to do on a computer.

Also, Anki is not only useful for learning new languages (although I’m using it right now for learning Japanese)! I’ve meet someone who started working on a very complicated software project a few months ago. There were so many parts of the system that it would take him (and every new person on the team) a long time to learn how they all work. So he started creating flashcards with Anki describing various parts of the system and how to use them. Soon, other people from his team started using Anki as well, saying that it’s much better than reading pages and pages of documentation!

Grammarly

Grammarly is an app that checks the grammar and typos in your text. As a non-native speaker, it helps me a lot to find some things that I wouldn’t probably notice. I used the free version for a few months, and a few days ago, I finally bought the premium plan.

Skitch

Taking screenshots on Mac is easy. There are keyboard shortcuts that let you quickly take a screenshot of the whole desktop, just a part of it, or even choose some options, like a delayed screenshot, if you want a screenshot of a hover effect on a website. Then you can open the screenshot, put some arrows or text, and send it to someone. So that worked for me for a long time. Until I found Skitch. With Skitch, when you take a screenshot, it automatically opens a simple app with some default tools for annotating them. It has huge pink arrows, large text, etc. When you finish, you can just drag and drop the screenshot from the app to a chat window. No more opening the Finder to find the screenshot, no more adjusting the text size and color in Preview app or clumsy arrows that you make from 3 straight lines glued together. With Skitch I can take a screenshot, annotate the problem and send it to someone on a Slack in less than 10 seconds.

The only downside is that Sketch’s keyboard shortcuts are non-customizable and the conflict with Mac’s ones. I ended up changing Mac’s shortcuts.

KeyCastr

If you ever wondered what app people use in their videos to show which keys they are pressing, then KeyCastr is one of those apps. I have used it literally once, but if I ever need to record some screencast, I will definitely use it again.

LICEcap

They say that a picture is worth a thousand words. Well, a GIF is probably worth over 9000 words. Especially if you are trying to show some buggy behavior. LICEcap is a free software to quickly record GIFs from your screen. I love it!

SimpleMind Lite

Tool for mind mapping.

The Unarchiver

I mostly use its CLI command unar to extract any kind of archive. It’s so good not have to remember the tar or unzip flags anymore!

Visual Studio Code

My code editor. I used Sublime for many years, but when I was switching to a Macbook, I decided to finally move to VS Code (after two unsuccessful attempts to do it in the past).

Workflowy

I use it to organize my long- and short-term goals that I later convert into actionable steps in Todoist.

Mundane but mandatory stuff

There are also a bunch of tools that probably everyone knows, but to keep this list complete, here they are:

Evernote - a note-taking app that doesn’t need an introduction. It doesn’t get much love nowadays, but I never actually moved away from it. The free plan is still perfectly fine for me.
git - THE version control tool.
vim - I use vim mostly for quick edits (small files or notes that don’t require a full-fledged IDE).
VLC - video player. I used it on all my Mac, Windows, and Linux computers.
GIMP - I should probably learn how to use one of those hip, beautiful (and expensive) Mac apps for editing images. But I’m too lazy, and I know GIMP for years, so it’s fine for now.
Inkscape - see above.
Dropbox - my preferred file synchronization tool.
iterm2 - de facto terminal app for Mac. It works great, stores its configuration nicely in my Dropbox backup folder, and I’m not planning to change it to anything else any time soon.
VirtualBox - even though I have not used it in ages, it’s my go-to virtualization tool. Back in the days, when I was still using Windows at home, VirtualBox was a great tool to easily spin up Linux containers, so I could actually do some programming (this was before Windows 10 with all the programming goodies).
dotfiles. I’m a big fan of dotfiles, and while there are some great tools to manage them (like yadm or dotbot), I’ve always used some bash scripts to manage them. I try to keep them up to date, so in case of a laptop failure, I can switch to a new one with all my settings and software. You can find my dotfiles here.
If you are using Mac or Linux and you don’t have dotfiles, but you would like to preserve the configuration of your software, there is also mackup. It will back up the configuration of most of your apps, including git, bash, etc. (it supports over 450 applications) into a folder in Dropbox, iCloud, or a similar service. Plus, it will replace the configuration files with symlinks to the backup location so that they will be automatically backed up. And with just one command, you can recover it on a different machine.
Docker for Mac - Mac client for Docker. Before I had a Mac, I was jealous of the beautiful Docker interface with the Kitematic app that Mac users had. Downloading images, listing running containers, or executing commands inside them through a nice GUI sounded like a sweet option. By the time I switched to a Mac, I already knew how to use Docker enough that I was doing all those things comfortably in the terminal. So I actually never used the GUI (but if you are a “GUI person”, I’m sure you are going to love it!).
To make it even worse, Docker for Mac has a memory leeks problem. It’s something that has been reported already in 2018 and closed because “It’s a problem with Mac not with Docker.” Even though people are still regularly commenting that they have this issue. Even with stopped containers, after running for a while, Docker starts consuming quite a lot of memory (and its energy consumption is high, so I usually disable it when I’m on a battery).

Tools that I don’t use, but I can highly recommend

Finally, there are some great tools that I currently don’t use. Some of them are the tools that I used in the past. And for some of them, I still haven’t found a good use case:

tmux - I’ve tried to use this terminal multiplexer a few times. But I never stuck to it. It’s easy to define shortcuts in iterm2 to split terminals and move withing them, so I don’t really need a separate tool for this. I also don’t need to keep my terminal sessions running for ages. If a terminal dies, it’s fine - I just start a new one. The obvious reason to use tmux is when you regularly connect to multiple servers. Which I no longer do. With Docker and the whole CI/CD movement, I realized that I less and less have to SSH anywhere. I work on my local machine, push my code to GitHub\GitLab, and the machines take over from there.
BitBar - a great little app that lets you put the output from any script or program in your menu bar. CPU usage graphs? Checked. Little menulet to interact with MySQL? Docker status? Bitcoin price? Ten different battery level indicators or your Apple keyboard battery level indicator? All checked! The only reason why I don’t use it is - I don’t need any of those scripts for now. I only needed to show the RAM and CPU graphs, but unfortunately, there were no RAM graph plugins when I checked, so I installed MenuMeters instead.
Amphetamine - prevents your Macbook from going to sleep. Good tool when you are giving presentations.
Dash - a great tool to access the documentation. Supports probably every programming language ever invented, works offline, and you can even search directly on StackOverflow. But for some reason, I never actually get used to it.
Flycut a clipboard manager that I used for some time before switching to Alfred. It’s free.
Sublime Text - I’ve used it as my default programming editor for years. Then I switched to VS Code. I still keep Sublime Text to quickly preview and edit large files (although I use vim more and more for this).
Slow Quit Apps - it’s frustrating when I accidentally press ⌘+Q instead of ⌘+W, and I close the current app. Slow Quit App prevents this by adding a time threshold. By default you need to keep pressing ⌘+Q for 1 second to actually close an app. I would probably use it if I knew about it before, but since I didn’t, I decided to remap ⌘+Q to some useless command. So now it’s “inverting colors.”
CheatSheet - I was really excited when I found this small helper. When you install it, each time you hold the ⌘ key, it will display an overlay with shortcuts for the current application. But then I realized that I don’t really use it, so I uninstalled it.

Conclusions

I learned about most of those tools from my colleagues, browsing through dotfiles of other people, and reading posts on HackerNews. Without other people sharing their tools, I would probably still be using Nano. So I hope that someone will find this list useful and create an even better one on top of it!

Preparing those lists was also an excellent opportunity to clean up my Mac a bit - I removed a bunch of unused applications, scripts, and aliases.

Stay tuned for the second part, where I will talk about my favorite CLI tools!

Photo by Lachlan Donald on Unsplash

Forem: Sebastian Witowski

Add Comments To Your Commands

Why?

A real-world example

25 IPython Tips for Your Next Advent of Code

1. Display the documentation

2. Display the source code

3. %edit magic function

4. Reopen last file with “%edit -p”

5. Wildcard search

6. post-mortem debugging

7. Start the debugger automatically

8. Run shell commands

9. Move around the filesystem with %cd

10. %autoreload

11. Change the verbosity of exceptions

12. Rerun commands from the previous sessions

13. Execute some code at startup

14. Use different profiles

15. Output from the previous commands

16. Edit any function or module

17. Share your code

18. Use IPython as your debugger

19. Execute code written in another language

20. Store variables between sessions

21. Save session to a file

22. Clean up “>” symbols and fix indentation

23. List all the variables

24. Use asynchronous functions

25. IPython scripts

Conclusions

Sorting Lists in Python

sorted() vs list.sort()

Initial order matters

Conclusions

For Loop vs. List Comprehension

Filter a list with a "for loop"

Filter a list with list comprehension

Filter a list with the "filter" function

More than one operation in the loop

Conclusions

Should We Still Use OrderedDict in Python?

OrderedDict vs dict

When to use OrderedDict?

How to stay up to date on Python changes?

Easy Speedup Wins With Numba

Numba and the @jit decorator

Other features of Numba

Conclusions

How did I find Numba?

Find Item in a List

Find a number

Find a number in a list

Python 2 flashback

Find a number in an infinite list

What about a list comprehension?

Find item with a generator expression

Conclusions

Python: Ask for Forgiveness or Look Before You Leap?

"Ask For Forgiveness" vs "Look Before You Leap" - speed

The main difference between "EAFP" and "LBYL"

Verdict

Let’s Write an IPython Extension the Hard Way

What’s wrong with the %rerun function?

Is there a plugin for that?

“Fine, I will do it myself.”

Writing a callback for IPython events

Input transformation

Meet %rerunplus: %rerun that ignores exceptions

Conclusions

My Favorite CLI Tools

Tools that I use every day

Fish plugins

htop and glances

virtualenv and virtualfish

pyenv, nodenv, and rbenv

ctop and lazydocker

Tools that I don’t use every day

colordiff and diff-so-fancy

tree (brew install tree)

What’s wrong with the `%rerun` function?

tree (`brew install tree`)