Forem: Stav Shamir

Fix history: Edit the last bash command

Stav Shamir — Mon, 06 Jan 2020 13:24:05 +0000

How many times have you typed a long command and quickly pressed enter, only to find out that you misspelled something or missed an argument?

As a CLI user, you probably look to minimize the number of actions it takes to get a task done - so instead of rewriting the entire command (or arrowing up and changing the string), you can easily replace a string in your command with builtin bash tools.

Event Designators: Quick Substitutions

An event designator is a reference to a command line entry in the history list.

The quick substitution is performed by using the following syntax: ^str1^str2^, which means that the last command will be run again with str1 replaced with str2. Let's see an example - say I want to copy a file named file1.py to the current directory:

$ cp ~/foo/bar/fle1.py .

Oops, I misspelled file1.py and wrote fle1.py. I can easily rerun the command with the correct file name using quick substitution:

$ ^fle^file^

Pressing enter prints the corrected command to the standard output and runs it as intended! You should note that the only the first matching string is replaced. There are ways to perform global replace, but the syntax is quite arcane and easy to forget. However, there is an alternative, which brings us to the next builtin.

By the way, the quote above is taken from the man pages. You should look it up (it is generally a good idea to read man once in while).

fc: memorable substitution and much more

Substitution

Another builtin way to perform quick substitution is with the fc command (the name can be thought of as a mnemonic for 'fix command').

To perform the same substitution from the previous example, run:

$ fc -s fle=file

This can be easily aliased to become even more comfortable:

$ alias r="fc -s"
$ r fle=file

I find this much more memorable than the ^str1^str2^ syntax.
Do note that this command performs a global substitution, so you have to be more careful about it.

Editing previous command

Actually fc can do much more. Sometimes a simple substitution is not enough to fix the command. For example, when running a long curl command with payload, and you need to change some of the payload's values.

Arrowing up and editing in the command line feels really inefficient, especially if you are a vim user. Yeah, I could copy the command to vim, make the changes, copy it again to the command line and run it. But that's an awful lot of steps for such a simple task. Well, this paragraph does not reside under the fc title for nothing! Try running the following:

$ fc

Yes, it does exactly what we needed it to (or maybe it opened nano, but we can fix that if needed)! Running fc with no flags or arguments, opens up the previous command to edit in a predefined editor. Edit, save and exit and the fixed command will be run!

Changing the editor

For one time (or always by aliasing and saving in .bashrc):

$ fc -e ename

Where ename is the name of the desired editor.
Another option is to define the environment variable FCEDIT with the desired editor. For vim that would be:

$ export FCEDIT=vim

Aborting the fix and avoiding unwanted re-reruns

You will notice that even if you don't save the changes and quit the editor (:q! in vim), the unedited command will be executed, which is probably not what you want.
To avoid that, the editor must return a non-zero exit code. You can do this in vim by exiting with :cq (or :cq! if you made changes).

Conclusion

These are just very few ways to fix previously executed commands in bash. There are of course others (such as the more known event designator !).

What are your favorites? Please write them in the comments!

The other benefit of Python type annotations

Stav Shamir — Mon, 18 Nov 2019 13:39:54 +0000

First published in my blog

Coming from statically typed languages, C and Java, I felt a little insecure writing Python code. Suddenly, silly type mismatch errors which I was used to catch during compilation were only caught (if at all, in the best case scenario) at runtime. This became especially annoying while learning new APIs or diving into a new large codebase, and made me completely reliant on documentation. While reading the docs is important on its own, I truly missed the comfortable and time-saving code completion on typing ‘.’ using IDEs such as IntelliJ.

Luckily, since Python 3.5, type-annotations have been officially added to Python (PEP 484), and now we can write something like this:

def hello(name: str) -> None:
    print(f'hello {name}')

Type annotations are optional

While having a different syntax than in C and Java, I find it rather intuitive (at least the simple cases). Importantly, the type annotations are completely optional and are ignored at runtime — meaning:

You can’t break the code by adding them (almost — I will elaborate in a future article)
They provide no performance gain
You may add them only where you see fit — I made myself a balanced rule of always annotating function signatures and module-scope variables, but I usually don’t bother to do so for local variables

Some benefits of type annotations

With type annotations out there for quite some time, several articles and guides were written about their benefits, of which the most common are:

We can employ static code analysis to catch type errors prior to runtime— PyCharm does this automatically, highlighting type errors (albeit they are easy to miss in my opinion). Another option, which is definitely recommended, is a useful tool named mypy, which can be run on specific type-annotated files to find type-related errors (I always run it after running my unit tests).
Cleaner code/the code is self-documenting — as Uncle Bob writes in his (very recommended) book “Clean Code”: “don’t use a comment when you can use a function or a variable”, we can now say “don’t use comments to specify a type, when you can use type annotation”. Comments tend to wear out and rot, while code is alive and must stay fresh. Change the types of the variables without changing the comment specifying the type — nothing happens. Change it without changing the type annotation? Your static analysis tool, whichever it may be, will shout at you. > “don’t use a comment when you can use a function or a variable”

The other benefit

One important benefit which, as I see it, should tilt your favor towards adopting this new feature if you have resisted it so far, is code completion. Early error-catching and clean code are super important, but I have seen people writing that it is more “pythonic” to write parameter assertions (which include type checking) in the beginning of a function, or that docstrings should be enough. While we can argue about that, the advantages of code completion (of course while using a good IDE, such as PyCharm) are undisputed.

First, let’s see an example of such code completion. Let’s say we are writing a library modeling music objects. We have an Album class and a Track class, and in the Album class we have a method returning all tracks shorter than a specified duration:

class Album:
    def __init__(self, title, artist, release_year, tracks):
        self.title = title
        self.artist = artist
        self.release_year = release_year
        self.tracks = tracks

    def all_tracks_shorter_than(self, minutes=0, seconds=0):
        duration = datetime.timedelta(minutes=minutes, seconds=seconds)
        return [t for t in self.tracks if t.duration < duration]


class Track:
    def __init__(self, title, duration):
        self.title = title
        self.duration = duration

Now let’s say I am a user of this library, and I want to print the names of all tracks shorter than 10 minutes in Wish You Were Here (which I instantiated earlier):

Wait a minute, was that variable called ‘name’ or ‘title’? I don’t know that! And the code completion is of no help at all — I must go to the declaration of the Track class or read its documentation — that takes time! This is the best case scenario — what if the programmer does not acknowledge she does not remember the variable name and guesses ‘name’ instead of ‘title’? It will cost even more time because this error will be found only in runtime (hopefully shes use uses TDD and catch it quickly with a pre-written test).

Now what happens if we use type annotations? Let’s add only the minimal type annotation directly needed at the moment:

def all_tracks_shorter_than(self, minutes=0, seconds=0) -> List[Track]:
    duration = datetime.timedelta(minutes=minutes, seconds=seconds)
    return [t for t in self.tracks if t.duration < duration]

The return value of this method is a list of Tracks, so we add ‘-> List[Track]’ to its signature (more about the syntax of type annotations). Let’s see what happens now on typing ‘.’:

Great! The IDE now knows what type this variable is, and can recommend the correct attributes — error averted! And the users of this package will have a friendlier time coding with it. Here is how the entire Album class looks like annotated (with my style of annotating):

class Album:
    def __init__(self, title: str, artist: str, release_year: int, tracks: List[Track]) -> None:
        self.title = title
        self.artist = artist
        self.release_year = release_year
        self.tracks = tracks

    def all_tracks_shorter_than(self, minutes: int = 0, seconds: int = 0) -> List[Track]:
        duration = datetime.timedelta(minutes=minutes, seconds=seconds)
        return [t for t in self.tracks if t.duration < duration]

Don’t use comments to specify a type, when you can use type annotation

Wrapping up, we’ve seen that type annotations are a great addition to Python — they improve the integrity of the code, they are self-documenting code, and they may improve the productivity of the clients of your code.

Dockerizing a Flask-MySQL app with docker-compose

Stav Shamir — Mon, 18 Nov 2019 11:41:21 +0000

Also published in my blog.

You have heard so much about docker and its advantages, and are now eager to use it with your new web app — but where do you start?
In this tutorial we will go through an example of taking an existing simple web app based on Flask and MySQL and making it run with Docker and docker-compose. Code is available here:

stavshamir / docker-tutorial

Code for a creating a docker app with Flask and MySQL tutorial

It is considered a best practice for a container to have only one responsibility and one process, so for our app we will need at least two containers — one for running the app itself, and one for running the database. How do we coordinate these containers? This is where docker-compose comes in. From the official docs:

Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration.

Let’s start!

If you don’t have them installed yet, install Docker and docker-compose.
We begin with the following project layout:

dockerize/
├── app
│   └── app.py
└── db
    └── init.sql

app.py — contains the Flask app which connects to the database and exposes one REST API endpoint
init.sql — an SQL script to initialize the database before the first time the app runs

Creating a Docker image for our app

We want to create a Docker image for our app, so we need to create a Dockerfile in the app directory. A Dockerfile contains a set of instructions describing our desired image and allow its automatic build.

# Use an official Python runtime as an image
FROM python:3.6

# The EXPOSE instruction indicates the ports on which a container 
# will listen for connections
# Since Flask apps listen to port 5000  by default, we expose it
EXPOSE 5000

# Sets the working directory for following COPY and CMD instructions
# Notice we haven’t created a directory by this name - this instruction 
# creates a directory with this name if it doesn’t exist
WORKDIR /app

# Install any needed packages specified in requirements.txt
COPY requirements.txt /app
RUN pip install -r requirements.txt

# Run app.py when the container launches
COPY app.py /app
CMD python app.py

What this does is simply as described in the file — base the image on a Python 3.6 image, expose port 5000 (for Flask), create a working directory to which requirements.txt and app.py will be copied, install the needed packages and run the app.

We need our dependencies (Flask and mysql-connector) to be installed and delivered with the image, so we need to create the aforementioned requirements.txt file:

Flask
mysql-connector

Now we can create a docker image for our app, but we still can’t use it, since it depends on MySQL, which, as good practice commens, will reside in a different container. We will use docker-compose to facilitate the orchestration of the two independant containers into one working app.

Creating a docker-compose.yml

So let’s create a new file, docker-compose.yml, in our project’s root directory:

version: "2"
services:
  app:
    build: ./app
    links:
      - db
    ports:
      - "5000:5000"

We are using two services, one is a container which exposes the REST API (app), and one contains the database (db).

build: specifies the directory which contains the Dockerfile containing the instructions for building this service
links: links this service to another container. This will also allow us to use the name of the service instead of having to find the ip of the database container, and express a dependency which will determine the order of start up of the container
ports: mapping of <host>:<container> ports.

  db:
    image: mysql:5.7
    ports:
      - "32000:3306"
    environment:
      MYSQL_ROOT_PASSWORD: root
    volumes:
      - ./db:/docker-entrypoint-initdb.d/:ro

image: Like the FROM instruction from the Dockerfile. Instead of writing a new Dockerfile, we are using an existing image from a repository. It’s important to specify the version — if your installed mysql client is not of the same version problems may occur.
environment: add environment variables. The specified variable is required for this image, and as its name suggests, configures the password for the root user of MySQL in this container. More variables are specified here.
ports: Since I already have a running mysql instance on my host using this port, I am mapping it to a different one. Notice that the mapping is only from host to container, so our app service container will still use port 3306 to connect to the database.
volumes: since we want the container to be initialized with our schema, we wire the directory containing our init.sql script to the entry point for this container, which by the image’s specification runs all .sql scripts in the given directory.

We are now ready to start the dockerized app! But before we do that, let’s peek at the code connecting to the database (app.py):

config = {
        'user': 'root',
        'password': 'root',
        'host': 'db',
        'port': '3306',
        'database': 'knights'
    }
connection = mysql.connector.connect(**config)

We are connecting as root with the password configured in the docker-compose file. Notice that we explicitly define the host (which is localhost by default) since the SQL service is actually in a different container than the one running this code. We can (and should) use the name ‘db’ since this is the name of the service we defined and linked to earlier, and the port is 3306 and not 32000 since this code is not running on the host.

Running the app

In order to run the our dockerized app, we will execute the following command from the terminal:

$ docker-compose up

You can see the image being built, the packages installed according to the requirements.txt, etc. If everything went right, you will see the following line:

app_1  |  * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)

We can find out that everything is running as expected by typing this url in a browser or using curl, and receiving the following response:

{"favorite_colors": [{"Lancelot": "blue"}, {"Galahad": "yellow"}]}

You can access the database directly using the mysql client and following command (make sure your client is the same version of MySQL specified in the docker-compose.yml):

$ mysql --host=127.0.0.1 --port=32000 -u root -p

It is a must to specify the host IP, since when the default ‘localhost’ is used MySQL ignores the port parameter.

Conclusion

We have learned how to dockerize a simple Flask-MySQL using docker-compose. Now this app can be used without tiresome preconfiguration on every host with Docker and docker-compose, and is encapsulated from the host environment!

Forem: Stav Shamir

Fix history: Edit the last bash command

Event Designators: Quick Substitutions

fc: memorable substitution and much more

Substitution

Editing previous command

Changing the editor

Aborting the fix and avoiding unwanted re-reruns

Conclusion

The other benefit of Python type annotations

Type annotations are optional

Some benefits of type annotations

The other benefit

Don’t use comments to specify a type, when you can use type annotation

Further reading:

Dockerizing a Flask-MySQL app with docker-compose

stavshamir / docker-tutorial

Code for a creating a docker app with Flask and MySQL tutorial

Let’s start!

Creating a Docker image for our app

Creating a docker-compose.yml

Running the app

Conclusion