Forem: drcloudycoder

Develop python CLI with subcommands using Click

drcloudycoder — Fri, 27 Nov 2020 22:30:35 +0000

While there are already great posts about developing CLI applications using Python's Click library showing examples of commands being setup from single python file, however you may instead want to organize commands into their own individual python files.

I had similar requirement while building a CLI application and spent quite some time in getting the structure I wanted to wire together and resolve minor errors. This tutorial is an attempt to show developing a successfully working example of CLI application with separate python files organized for each command.

For this tutorial, we will develop a simple CLI application that has multiple commands build and deploy like this:

$ python3 main.py
Usage: main.py [OPTIONS] COMMAND [ARGS]...

  CLI tool to manage full development cycle of projects

Options:
  --help  Show this message and exit.

Commands:
  build
  deploy

Create a new project directory with following structure and blank python files as below:
```
$ tree
.
├── commands
│   ├── __init__.py
│   ├── build.py
│   ├── deploy.py
└── main.py

1 directory, 4 files
```
You can create a separate python file inside commands dir for each command you want to expose to your main cli.
Install click python library using this:
```
pip3 install click
```

Let's start with a starter code of main.py:

import click

@click.group(help="CLI tool to manage full development cycle of projects")
def cli():
    pass

if __name__ == '__main__':
    cli()

📌 Note that we have annotated starter function cli() with @click.group indicating that this will be a group of commands.

Verify this part working successfully by running:

$ python3 main.py
Usage: main.py [OPTIONS] COMMAND [ARGS]...

  CLI tool to manage full development cycle of projects

Options:
  --help  Show this message and exit.

Next, let's write code for build sub-command in commands/build.py:
```
import click

@click.command()
@click.option('--docker', is_flag=True, help='Indicates the project should be built into docker image')
def build(docker):
    if docker:
        print(f'Building this repo into a docker image...')
    else:
        print(f'Building this repo using default method...')
```
📌 Note these points before moving forward:
- We have defined a function build(docker) that is annotated with @click.command() to indicate that this is a command.
- We also define a boolean type of option --docker for build command whose value gets passed to build() function. You can define additional options here for build command and write your own logic for this command here.
At this point, we have created a group (of commands) in main.py called "cli" and a separate command "build", but have not tied them together yet.

Let's add this build command to our main group:

6.1. Import commands/build library in main.py:
```
  from commands import build
```
6.2. Add build command to the main group in main.py:
```
  cli.add_command(build.build)
```
📌 Note:
- We performed add_command() on cli object instead of click object
- We added build.build to cli instead of build

Verify that you can now run build as a subcommand of your main program:

$ python3 main.py build --help
Usage: main.py build [OPTIONS]

Options:
  --docker  Indicates the project should be built into docker image
  --help    Show this message and exit.

📌 If you have a working structure at this point, then you can easily follow similar process for adding new sub-commands into separate python files under commands/ dir or any other dir.

For example, let's add a deploy subcommand functionality to our main CLI program. This subcommand should expose 2 options (--env and --cloud) and prompt the user to enter input while displaying allowed values and a default value.

Define functionality and available options for deploy subcommand:

import click

@click.command()
@click.option('--env', '-e', default="dev", type=click.Choice(['dev', 'stg', 'prd'], case_sensitive=False), prompt='Enter env name to deploy', help='Env to deploy')
@click.option('--cloud', '-c', default="aws", type=click.Choice(['aws', 'gcp', 'azure'], case_sensitive=False), prompt='Enter cloud to deploy to', help='Cloud to deploy to')
def deploy(env, cloud):
    print(f'Deploying current application artifact to {env} environment in {cloud} cloud...')

Import this new deploy subcommand to main CLI program:
```
from commands import deploy
```
Add deploy subcommand to main CLI program:
```
cli.add_command(deploy.deploy)
```

Verify that you can now run deploy subcommand:

$ python3 main.py deploy
Enter env name to deploy (dev, stg, prd) [dev]: 
Enter cloud to deploy to (aws, gcp, azure) [aws]: 
Deploying current application artifact to dev environment in aws cloud...

🏁 🏆 I hope you found this tutorial useful and it helped you implement a working CLI structure with commands organized into separate python files/dir. Feel free to leave questions or feedback in comments. 🔚

🛎 P.S.: Here are the final version of above files for reference:

main.py

import click

from commands import build
from commands import deploy


@click.group(help="CLI tool to manage full development cycle of projects")
def cli():
    pass


cli.add_command(build.build)
cli.add_command(deploy.deploy)

if __name__ == '__main__':
    cli()

commands/build.py

import click

@click.command()
@click.option('--docker', is_flag=True, help='Indicates the project should be built into docker image')
def build(docker):
    if docker:
        print(f'Building this repo into a docker image...')
    else:
        print(f'Building this repo using default method...')

commands/deploy.py

import click

@click.command()
@click.option('--env', '-e', default="dev", type=click.Choice(['dev', 'stg', 'prd'], case_sensitive=False), prompt='Enter env name to deploy', help='Env to deploy')
@click.option('--cloud', '-c', default="aws", type=click.Choice(['aws', 'gcp', 'azure'], case_sensitive=False), prompt='Enter cloud to deploy to', help='Cloud to deploy to')
def deploy(env, cloud):
    print(f'Deploying current application artifact to {env} environment in {cloud} cloud...')

Kubernetes Cronjob Best Practices

drcloudycoder — Wed, 25 Nov 2020 21:56:30 +0000

This post is to document the best practices I have realized over last few months of using kubernetes cronjob for scheduled workloads.

Like most developers out there, I started writing my first kubernetes cronjob yaml files following the official guide example that does not specify a lot of useful parameters. However, it took me quite some time and unexpected execution behavior of kubernetes cronjobs to realize the importance of consciously specifying all parameters and not rely blindly on default values.

Here are such useful parameters that I think we should always define for a kubernetes cronjob specification (the emphasis is on always defining these parameters and not necessarily on all their values I am showing below) -

concurrencyPolicy
- Set concurrencyPolicy: Forbid if you want to restrict another cron execution when a previous one is still running.
suspend
- Set suspend: true if you want to temporarily disable cronjob executions for some time to troubleshoot any issue without having to delete the cronjob itself.
successfulJobsHistoryLimit
- Set successfulJobsHistoryLimit: 2 to retain last 2 successful cronjob execution history.
failedJobsHistoryLimit
- Set failedJobsHistoryLimit: 3 to retain last 3 failed cronjob execution history. Default value is 1 which I think may not be enough to troubleshoot failures.\

Although following parameters belong to kubernetes Job which is the unit of Cronjob, but I think are equally important when configuring a cronjob -

parallelism
- Even though this defaults to 1, set it explicitly in your code to help yourself understand your code weeks/months after.
- Set parallelism: 1 to restrict only 1 job to run in parallel at any point of time.
completions
- Set completions: 1 when you want the job to be marked as successful.
- This makes more sense when, say, you have 3 pods running in parallel and want to mark the job as successful when all 3 pods successfully complete execution.
restartPolicy
- Set restartPolicy: OnFailure at the pod specification level to let kubernetes restart the pod container when your workload process exits with failure.
- Use backoffLimit parameter along with this to protect from back to back restarting behavior during consistent failures.
backoffLimit
- Jobs can fail for variety of reasons like a process failing within pod or a failure at kubernetes controller layer, that can cause automatic retries of job executions.
- Set backoffLimit: 3 to ensure that jobs are not retried unlimited times but only 3 times after which the job (i.e. cronjob execution in our article) will fail and halt.
activeDeadlineSeconds
- If there is a possibility of your workload process potentially consuming longer than expected (like when interacting with external dependent service), then you must set a time limit up to which kubernetes will allow the cron execution and kill it if exceeded.
- Set activeDeadlineSeconds: 60 (example value) to let kubernetes kill the job execution if it takes longer than 60 seconds.

I hope this helps someone and feel free to share your related experiences.