Forem: Suresh Kumar

How to manage k8s yaml manifests for multiple environments with kustomize?

Suresh Kumar — Thu, 26 May 2022 14:13:24 +0000

What is Kustomize?

Kustomize is a CLI configuration manager for Kubernetes objects that leverage layering to preserve the base settings of the application. This is achieved by overlaying the declarative YAML artifacts to override default settings without actually making any changes to the original manifest. Kustomize is also integrated with kubectl.

Kustomize is aware of kubernetes resources and their fields and is not just a simple text templating solution like other tools.

With Kustomize you can reuse one of the base files across all environments (development, staging, production, etc.) and overlay specifications for each of those environments.

Kustomize can also be used with helm and CD solutions like argo CD.

To install kustomize checkout --> https://kubectl.docs.kubernetes.io/installation/kustomize/

how kustomize works?

kustomization.yaml

Each directory contains a kustomization.yaml file, which is essentially a list of resources or manifests that describes how to generate or transform Kubernetes objects.

With Kustomize, you can configure raw, template-free YAML files, which allows you to modify settings/annotations between deployment and production easily.

Kustomize provides 2 methods to apply patch,

patchesStrategicMerge
patchesJson6902

patchesStrategicMerge is the most common and easy to use merge strategy. To know more about patching checkout --> https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/#customizing

base folder

The base folder holds common resources, such as the deployment.yaml, service.yaml, and configmap.yaml. It contains the initial manifest and includes a namespace and label for the resources.

overlays folder

The overlays folder has environment-specific overlays, which use patches to allow YAML files to be defined and overlaid on top of the base for any changes.

Example structure,
To create a base configmap resource and change configmap variable for staging and production. To get an full fledged example checkout --> https://github.com/sureshdsk/kustomize-k8s-example

kustomize-k8s
├── base
│   ├── configmap.yaml
│   ├── kustomization.yaml
└── overlays
    ├── production
    │   ├── configmap-patch.yaml
    │   ├── kustomization.yaml
    └── staging
        ├── configmap-patch.yaml
        ├── kustomization.yaml

base/configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: django-configmap
data:
  DJANGO_AUTH_PUBLIC_URI: "http://dj.192.168.0.139.sslip.io"
  DEBUG: "True"

base/kustomization.yaml

# common labels to be added on all manifests
commonLabels:
  app: demo

# resources that needs to be kustomized
resources:
  - configmap.yaml

Now, lets change the DJANGO_AUTH_PUBLIC_URI value for staging environment.

overlays/staging/configmap-patch.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: django-configmap
data:
  DJANGO_AUTH_PUBLIC_URI: "http://staging.192.168.0.139.sslip.io"

overlays/staging/kustomization.yaml

# prefix to be added to name of the resource
namePrefix: staging-
commonLabels:
  env: staging
# directory contains base yaml
bases:
  - ../../base
# patch strategy
patchesStrategicMerge:
  - configmap-patch.yaml

Clone the repo

git clone git@github.com:sureshdsk/kustomize-k8s-example.git
cd kustomize-k8s-example

Preview and apply manifests

We can preview the kustomize output using kustomize build command.

# preview output
kustomize build overlays/staging

# apply output to kubernetes
kustomize build overlays/staging | kubectl apply -f -

We can also use kustomize under kubectl kustomize as kubectl plugin.

# preview output
kubectl kustomize overlays/staging

# apply output to kubernetes
kubectl apply -k overlays/staging

Resources

How to create your first Helm plugin?

Suresh Kumar — Tue, 03 May 2022 07:12:02 +0000

Helm

Helm helps you manage Kubernetes applications using helm charts. Helm Charts help you define, install, and upgrade Kubernetes applications.

Example:

# add chart repository
helm repo add bitnami https://charts.bitnami.com/bitnami

# update charts
helm repo update

# install mysql using helm chart by bitnami
helm install bitnami/mysql --generate-name

Helm plugins

Helm plugins provide a way to extend Helm features without modifying the helm core codebase.

Helm plugins have the following features:

They can be added and removed from a Helm installation without impacting the core Helm tool.
They can be written in any programming language.
They integrate with Helm, and will show up in helm help and other places.

Helm plugins are stored in $HELM_PLUGINS directory, you can change this using helm env command.

Install a helm plugin

A Helm plugin can be installed from a git repo, a tar release or from a local directory.

helm plugin install <path|url>

Helm plugin anatomy

A helm plugin constists of a plugin.yaml file, which has the plugin definition and its associated commands.

─ fullenv
   ├── plugin.yaml
   └── run.sh

plugin definition

plugin.yaml defines the name, usage, description and command to run when invoked.

The Command can point to your own script of any programming language.
The ignoreFlags switch tells Helm to not pass flags to the plugin.

So if a plugin is called with helm myplugin --foo and ignoreFlags: true, then --foo is silently discarded.

plugin.yaml

name: fullenv
usage: "show env vars"
description: "show all env vars"
command: "$HELM_PLUGIN_DIR/run.sh"
ignoreFlags: true

run.sh - just prints helm environment variables.

#!/bin/sh
echo $HELM_PLUGIN_NAME
echo $HELM_PLUGIN_DIR
echo $HELM_PLUGINS
echo $HELM_REPOSITORY_CONFIG
echo $HELM_REPOSITORY_CACHE
echo $HELM_BIN

Platform specific commands

Helm plugin commands can be configured to target a os platform and architecture.

name: fullenv
usage: "show env vars"
description: "show all env vars"
command: "$HELM_PLUGIN_DIR/run.sh"
platformCommand:
  - os: linux
    arch: i386
    command: "$HELM_PLUGIN_DIR/run.sh"
  - os: linux
    arch: amd64
    command: "$HELM_PLUGIN_DIR/run.sh"

Helm follows the below rules to choose a command.

If platformCommand is present, it will be searched first.
If both os and arch match the current platform, search will stop and the command will be used.
If os matches and there is no more specific arch match, the command will be used.
If no platformCommand match is found, the default command will be used.
If no matches are found in platformCommand and no command is present, Helm will exit with an error.

Install the plugin

cd into the plugin directory and then run the command below

# install
helm plugin install .

# check if the installation is successful
helm plugin list

Test the plugin

helm fullenv

If you like this article, subscribe to the newsletter and Connect with me on twitter to get updates on my future articles. ✅

How to install Kyverno in a k8s cluster?

Suresh Kumar — Tue, 26 Apr 2022 12:37:52 +0000

In the k8s policy management series, We're going to look at how to install Kyverno dynamic admission controller and its CRDs can installed in k8s cluster using helm or yaml manifests.

Install using helm

helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update

# Install kyverno controller
helm install kyverno kyverno/kyverno --namespace kyverno --create-namespace

# Install kyverno policies
helm install kyverno-policies kyverno/kyverno-policies --namespace kyverno

Install using yaml

kubectl create -f https://raw.githubusercontent.com/kyverno/kyverno/main/config/install.yaml

To check the Kyverno controller status, run the command:

kubectl get pods -n <namespace>

If the Kyverno controller is not running, you can check its status and logs for errors:

kubectl describe pod <kyverno-pod-name> -n <namespace>
kubectl logs -l app.kubernetes.io/name=kyverno -n <namespace>

How JWT (JSON Web Token) authentication works?

Suresh Kumar — Mon, 25 Apr 2022 05:03:03 +0000

What is JWT (JSON Web Token)?

JSON Web Token (JWT) is an open standard (RFC 7519) for securely transmitting information between parties as JSON object.

It is compact, readable and digitally signed using a private key/ or a public key pair by the Identity Provider(IdP). So the integrity and authenticity of the token can be verified by other parties involved.

The purpose of using JWT is not to hide data but to ensure the authenticity of the data. JWT is signed and encoded, not encrypted.

JWT is a token based stateless authentication mechanism. Since it is a client-side based stateless session, server doesn't have to completely rely on a datastore(database) to save session information.

Structure of JWT

A JSON Web Token consists of 3 parts separated by a period.



header.payload.signature

Header

JWT header consists of token type and algorithm used for signing and encoding. Algorithms can be HMAC, SHA256, RSA, HS256 or RS256.



{ 
  "typ": "JWT",    
  "alg": "HS256"
}

Payload

Payload consists of the session data called as claims. Below are some of the the standard claims that we can use,

Issuer(iss)
Subject (sub)
Audience (aud)
Expiration time (exp)
Issued at (iat)



{
"sub": "user10001",
"iat": 1569302116
}

Custom claims can also be included in the claim set. When using custom claim sets,

Do not put large data in claim sets. Claim sets meant to be compact.
Do not put sensitive informations since, JWT can be decoded easily.



{
"sub": "user10001",
"iat": 1569302116,
"role": "admin",
"user_id": "user10001"
}

Signature

Signature is most important part of a JSON Web Token(JWT). Signature is calculated by encoding the header and payload using Base64url Encoding and concatenating them with a period separator. Which is then given to the cryptographic algorithm.



// signature algorithm
data = base64urlEncode( header ) + "." + base64urlEncode( payload )
signature = HMAC-SHA256( data, secret_salt )

So when the header or payload changes, signature has to calculated again. Only the Identity Provider(IdP) has the private key to calculate the signature which prevents the tampering of token.

How it works?

Basically the identity provider(IdP) generates a JWT certifying user identity and Resource server decodes and verifies the authenticity of the token using secret salt / public key.

User sign-in using username and password or google/facebook.
Authentication server verifies the credentials and issues a jwt signed using either a secret salt or a private key.
User's Client uses the JWT to access protected resources by passing the JWT in HTTP Authorization header.
Resource server then verifies the authenticity of the token using the secret salt/ public key.

Security

Just like any other authentication mechanism, JWT also has its own pros and cons.

Must use HTTPS to secure the Authorization headers.
Validate algorithm name explicitly. Do not completely rely on the algorithm mentioned in the header of JWT. There are a few known attacks based on the header like algo none attack, header stripping.
Revoking the session of a user from backend server is difficult. Since a JWT is set to automatically expire, If an attacker gets the token before it expires It leads to various exploits. Building a token revocation list on your server to invalidate tokens could be best way to mitigate.
If JWT is persisted on cookies, we need to create HttpOnly cookie. This will restrict third party javascripts from reading jwt token from cookie.
XSS - backend servers must always sanitize user generated data.
CSRF - If JWT in persisted on cookies, CSRF attacks are possible. We can mitigate CSRF by using origin of request and special request headers.

If you like this article, subscribe to the newsletter and Connect with me on twitter to get updates on my future articles. ✅

How to use Kyverno CLI to validate k8s manifests?

Suresh Kumar — Sun, 24 Apr 2022 06:11:55 +0000

In the previous article, we have seen what is Kyverno, its features, its use-cases and hot it works. In this article we will install kyverno cli in our local machine and explore its usecases.

Install Kyverno CLI

The Kyverno CLI is designed to validate and test policy behavior to resources prior to adding them to a cluster.
Used in CI/CD pipelines to validate manifests before they are deployed.
Can be integrated into precommit hooks

Install Kyverno CLI via Krew

Krew is the plugin manager for kubectl command-line tool. If do not have krew installed already, please follow the instructions --> https://krew.sigs.k8s.io/docs/user-guide/setup/install/

# Install Kyverno CLI using kubectl krew plugin manager
kubectl krew install kyverno

# test the Kyverno CLI
kubectl kyverno version

Install Kyverno CLI via Brew (MacOS)

# Install Kyverno CLI using brew
brew install kyverno

# test the Kyverno CLI
kyverno version

Kyverno CLI Commands

Apply

Performs a dry run on one or more policies for the given manifest(s)
Executes mutate policies and shows mutated resource as an output

kyverno apply /path/to/policy.yaml --resource /path/to/resource.yaml

Test

tests policy from a git repo or local directory
recursively looks for YAML files in a directory and executes tests
kyverno test definition consists of test name, policies, resources and expected results.

An example test would look like

name: disallow_latest_tag
policies:
  - policy.yaml
resources:
  - resource.yaml
results:
  - policy: disallow-latest-tag
    rule: require-image-tag
    resource: myapp-pod
    kind: Pod
    result: pass
  - policy: disallow-latest-tag
    rule: validate-image-tag
    resource: myapp-pod
    kind: Pod
    result: pass

To Run the test,

kyverno test /path/to/yamls

Validate

check if a policy is syntactically valid.
can validate multiple policy resource description files or a folder containing policy resource description files.

kyverno validate /path/to/policy1.yaml /path/to/policy2.yaml /path/to/folderFullOfPolicies

Jp

Kyverno CLI also provides a utility called jp to work with JMESPath and expressions.

$ echo '{"foo": "BAR"}' | kyverno jp 'to_lower(foo)'
"bar"

$ cat pod.json
{
  "apiVersion": "v1",
  "kind": "Pod",
  "metadata": {
    "name": "mypod",
    "namespace": "foo"
  },
  "spec": {
    "containers": [
      {
        "name": "busybox",
        "image": "busybox"
      }
    ]
  }
}

$ kyverno jp -f pod.json 'spec.containers[0].name' -u
busybox

Kyverno precommit hooks

Kyverno can be integrated into precommit hooks to test and validate policies. To setup precommit hook, checkout -> https://github.com/kyverno/pre-commit-hook

.pre-commit-config.yaml

repos:
  - repo: https://github.com/kyverno/pre-commit-hook
    rev: v1.0.0
    hooks:
      - id: kyverno-test
        args: ["kyverno-policies"]
      - id: kyverno-validate
        args: ["kyverno-policies"]

If you like this article, subscribe to the newsletter and Connect with me on twitter to get updates on my future articles. ✅

Kyverno - Kubernetes native policy management for k8s admins

Suresh Kumar — Fri, 22 Apr 2022 05:25:27 +0000

Kubernetes Policy management

As k8s cluster administrator, you would want to enforce certain protocols to be followed on the cluster like below

enforce standard best practices
improve security
eliminate misconfigurations
reduce configuration complexity
automate policy management
synchronize configurations across namespaces
block non-conformant resources using admission controls, or report policy violations
test policies and validate resources using the Kyverno CLI, in your CI/CD pipeline, before applying to your cluster
manage policies as code using familiar tools like git and kustomize

Kyverno

Kyverno is a Kubernetes-native policy engine that helps you define policies using k8s manifests.
Using Kyverno, platform team can define policies and ensure the configurations are compliant with their security and best practices standards.
Kyverno policies can be defined with declaratively like standard k8s manifests.
Kyverno can be easily integrated into your existing CI/CD Pipelines.
Under the hood kyverno makes use of k8s admission webhooks to manage and validate policies.

Kyverno Features

easy to write and manage policies with yaml manifests.
support all k8s resource types including Custom Resources.
Validate, Mutate, or Generate any resource using policy
Image Verify - verify container images for software supply chain security

Common use cases

Pod and workload security configurations
Best practice configurations
Fine-grained RBAC
Auto-Labeling resources
Image Signing and Supply Chain Security

Kyverno Architecture

Kyverno runs as a dynamic admission controller in a Kubernetes cluster. Kyverno receives validating and mutating admission webhook HTTP callbacks from the kube-apiserver and applies matching policies to return results that enforce admission policies or reject requests.

Kyverno policies can match resources using the resource kind, name, and label selectors.

Mutating policies can be written as overlays (similar to Kustomize) or as a RFC 6902 JSON Patch. Validating policies also use an overlay style syntax, with support for pattern matching and conditional (if-then-else) processing.

An admission controller is a piece of code that intercepts requests to the Kubernetes API server before the persistence(storing in ETCD) of the object, but after the request is authenticated and authorized(RBAC). It can be regarded as checkpoints for Kubernetes API requests and can completely deny/accept the requests or change the request object altogether.

Kubernetes also has some Dynamic Admission Controllers like ValidatingAdmissionWebhooks and MutatingAdmissionWebhooks, which can validate and mutate the requests sent by the user.

The following diagram below shows how the admission controller works after authorization/authentication and before persisting it to etcd.

References

In the next article, we will learn to install kyverno and write policy for kubernetes manifests.

If you like this article, subscribe to the newsletter and Connect with me on twitter to get updates on my future articles. ✅

5 different ways to use an else block in python

Suresh Kumar — Thu, 21 Apr 2022 07:05:13 +0000

There are several ways to use an else block in python. Lets look at each method and its usecase.

1. if else

This is commonly used if else block. if block is executed if the condition is true otherwise else block will be executed.


x = True

if x:
    print 'x is true'
else:
    print 'x is not true'

2. if else shorthand

This if else shorthand method is a ternary operator equivalent in pythom if else statement. If you loot at the code, boolean value True will assigned to the variable is_pass if the expression mark >= 50 is true, otherwise False will be assigned.


mark = 40
is_pass = True if mark >= 50 else False
print "Pass? " + str(is_pass)

3. for-else loop

We can use an else block on a for loop too. The else block is executed only when the for loop completes its iteration without breaking out of the loop.

for loop below, will print from 0 to 10 and then 'For loop completed the execution' as it doesn't break out of the for loop.


for i in range(10):
    print i
else:
    print 'For loop completed the execution'

for loop below, will print from 0 to 5 and then breaks out of the for loop, so else block will not be executed.

for i in range(10):
    print i
    if i == 5:
        break
else:
    print 'For loop completed the execution'

4. while-else loop

We can also use an else block with while loop, The else block is executed only when the while loop completes its execution without breaking out of the loop.

a = 0
loop = 0
while a <= 10:
    print a
    loop += 1
    a += 1
else:
    print "While loop execution completed"

a = 50
loop = 0
while a > 10:
    print a
    if loop == 5:
        break
    a += 1
    loop += 1
else:
    print "While loop execution completed"

5. else on try-except

We can use an else block on a try except block too. This is type of not required in most cases. The else block is only executed if the try block doesn't throw any exeception.

In this code, else block will be executed if the file open operation doesn't throw i/o exception.

file_name = "result.txt"
try:
    f = open(file_name, 'r')
except IOError:
    print 'cannot open', file_name
else:
    # Executes only if file opened properly
    print file_name, 'has', len(f.readlines()), 'lines'
    f.close()

follow me on twitter to get more content and connect with me.

Python decorator to measure execution time

Suresh Kumar — Thu, 21 Apr 2022 07:02:29 +0000

In the previous articles on the Python decorator series, we have learnt decorators, how they work and to implement a simple function based decorator and a class based decorator and decorator that supports parameters. In this article, we will create reusable decorator utility to measure execution time of a function and instance method in a class.

Measure execution time of a function

from functools import wraps
import time


def timeit(func):
    @wraps(func)
    def timeit_wrapper(*args, **kwargs):
        start_time = time.perf_counter()
        result = func(*args, **kwargs)
        end_time = time.perf_counter()
        total_time = end_time - start_time
        print(f'Function {func.__name__}{args} {kwargs} Took {total_time:.4f} seconds')
        return result
    return timeit_wrapper


@timeit
def calculate_something(num):
    """
    Simple function that returns sum of all numbers up to the square of num.
    """
    total = sum((x for x in range(0, num**2)))
    return total

if __name__ == '__main__':
    calculate_something(10)
    calculate_something(100)
    calculate_something(1000)
    calculate_something(5000)
    calculate_something(10000)

How it works

We decorate the function with timeit decorator
decorator makes note of start time
then executes the function
decorator marks end time
calculates time difference and prints the time taken for the function

Output

Function calculate_something(10,) {} Took 0.0000 seconds
Function calculate_something(100,) {} Took 0.0008 seconds
Function calculate_something(1000,) {} Took 0.0760 seconds
Function calculate_something(5000,) {} Took 2.4503 seconds
Function calculate_something(10000,) {} Took 7.9202 seconds

Measure execution time of a method inside a class

from functools import wraps
import time


def timeit(func):
    @wraps(func)
    def timeit_wrapper(*args, **kwargs):
        start_time = time.perf_counter()
        result = func(*args, **kwargs)
        end_time = time.perf_counter()
        total_time = end_time - start_time
        # first item in the args, ie `args[0]` is `self`
        print(f'Function {func.__name__}{args} {kwargs} Took {total_time:.4f} seconds')
        return result
    return timeit_wrapper


class Calculator:
    @timeit
    def calculate_something(self, num):
        """
        an example function that returns sum of all numbers up to the square of num
        """
        total = sum((x for x in range(0, num**2)))
        return total

    def __repr__(self):
        return f'calc_object:{id(self)}'


if __name__ == '__main__':
    calc = Calculator()
    calc.calculate_something(10)
    calc.calculate_something(100)
    calc.calculate_something(1000)
    calc.calculate_something(5000)
    calc.calculate_something(10000)

How it works

we decorate a method inside a class with timeit
timeit takes all arguments, note that args[0] is self so you can call any other method inside the class using self
decorator makes note of start time
then executes the function
decorator marks end time
calculates time difference and prints the time taken for the function

Output

Function calculate_something(calc_object:140246512997904, 10) {} Took 0.0000 seconds
Function calculate_something(calc_object:140246512997904, 100) {} Took 0.0007 seconds
Function calculate_something(calc_object:140246512997904, 1000) {} Took 0.1820 seconds
Function calculate_something(calc_object:140246512997904, 5000) {} Took 1.9241 seconds
Function calculate_something(calc_object:140246512997904, 10000) {} Took 7.4005 seconds

In the next article, we will implement various kinds decorator recipes. Stay tuned for upcoming articles. Subscribe to the newsletter and Connect with me on twitter to get my future articles.

Memoization decorator in python

Suresh Kumar — Wed, 20 Apr 2022 04:03:33 +0000

In the previous articles on the Python decorator series, we have learnt decorators, how they work and to implement a simple function based decorator and a class based decorator and decorator that supports parameters. In this article, we will create a simple memoization decorator function that caches result.

Memoization is an optimisation technique used to speed up programs by storing the results of expensive function calls and returning the cached result when the same inputs occur again.

Factorial of a number

Let's create a function that calculates factorial of a number in a recursive way.


def factorial(n):
    if n == 1:
        return n
    else:
        return n * factorial(n - 1)


if __name__ == '__main__':
    print(factorial(20))
    print(factorial(10))
    print(factorial(15))
    print(factorial(5))

Output

2432902008176640000
3628800
1307674368000
120

Since factorial of a given number is always same(idempotent), We can optimize this function by caching results for the given value. Let's create the decorator function,

from functools import wraps


def memoize(func):
    cache = {}

    @wraps(func)
    def wrapper(*args, **kwargs):
        if args not in cache:
            print(f"Calling for {func.__name__}({args})")
            cache[args] = func(*args, **kwargs)
        else:
            print(f"Using cached result for {func.__name__}({args})")
        return cache[args]

    return wrapper


@memoize
def factorial(n):
    if n == 1:
        return n
    else:
        return n * factorial(n - 1)


if __name__ == '__main__':
    print(factorial(20))
    print(factorial(10))
    print(factorial(15))
    print(factorial(5))

Output

Calling for factorial((20,))
Calling for factorial((19,))
Calling for factorial((18,))
Calling for factorial((17,))
Calling for factorial((16,))
Calling for factorial((15,))
Calling for factorial((14,))
Calling for factorial((13,))
Calling for factorial((12,))
Calling for factorial((11,))
Calling for factorial((10,))
Calling for factorial((9,))
Calling for factorial((8,))
Calling for factorial((7,))
Calling for factorial((6,))
Calling for factorial((5,))
Calling for factorial((4,))
Calling for factorial((3,))
Calling for factorial((2,))
Calling for factorial((1,))
2432902008176640000
Using cached result for factorial((10,))
3628800
Using cached result for factorial((15,))
1307674368000
Using cached result for factorial((5,))
120

From the output, we can see that for computing the factorial of 10, 15 and 5, we used cached result.

In the next article, we will implement various kinds decorator recipes. Stay tuned for upcoming articles. Subscribe to the newsletter and Connect with me on twitter to get my future articles.

Python decorators with parameters

Suresh Kumar — Tue, 19 Apr 2022 12:14:35 +0000

In the previous articles on the Python decorator series, we have learnt decorators, how they work and to implement a simple function based decorator and a class based decorator. In this article we will learn to create decorators that supports parameters.

Function based decorator with parameters

from functools import wraps


def hello_decorator(num):
    """Simple decorator function that supports parameters"""

    def inner_func(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            """Simple decorator wrapper function"""
            result = func(*args, **kwargs)
            result = result + num
            return result
        return wrapper
    return inner_func


@hello_decorator(100)
def add(a, b):
    """Simple function that returns sum of two numbers"""
    return a + b


@hello_decorator(200)
def multiply(a, b):
    """Simple function that returns multiplication of two numbers"""
    return a * b


if __name__ == '__main__':
    output1 = add(2, 2)
    print('Result:: ', output1)
    print("=" * 25)

    output2 = multiply(4, 2)
    print('Result:: ', output2)

As you notice, the structure is little different from our previous examples,

decorator takes a parameter @hello_decorator(100) - this is how we can pass arguments to our decorator
hello_decorator function returns an inner function.
inner_func takes the function to be decorated as an argument and returns the wrapper function.
wrapper function executes the add function and manipulates the output based on argument result = result + num and returns the final result

Class based decorator with parameters

from functools import wraps


class HelloDecorator:
    """Simple class decorator"""

    def __init__(self, num):
        self.num = num

    def __call__(self, func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            """Simple class call method"""
            result = func(*args, **kwargs)
            result = result + self.num
            return result
        return wrapper


@HelloDecorator(100)
def add(a, b):
    """Simple function that returns sum of two numbers"""
    return a + b


@HelloDecorator(200)
def multiply(a, b):
    """Simple function that returns multiplication of two numbers"""
    return a * b


if __name__ == '__main__':
    output1 = add(2, 2)
    print('Result:: ', output1)

    output2 = multiply(4, 2)
    print('Result:: ', output2)

This class based decorator with parameters is pretty much similar to our simple function based decorator. The best thing with this method is, we do not need the extra boilerplate code to fix the doc strings.

So now to implement decorator with arguments, I would prefer class based approach. since it is very intuitive and doesn't requires an additional inner function and additional boilerplate code fixes for docs.

So far, we've covered the basic decorator implementation with examples. In the next article, we will implement various kinds decorator recipes. Stay tuned for upcoming articles. Subscribe to the newsletter and Connect with me on twitter to get my future articles.

Class based decorators in python

Suresh Kumar — Tue, 19 Apr 2022 05:03:02 +0000

In the previous articles of Python decorator series we learnt to create a simple function based decorator, how it works under the hood and how to fix the doc strings.

In this article, we will learn to create a class based decorator and use it to decorate functions.


class HelloDecorator:
    """Simple class decorator"""

    def __init__(self, func):
        self.func = func

    def __call__(self, *args, **kwargs):
        """Simple class call method"""
        print(f'Calling {self.func.__name__}')
        result = self.func(*args, **kwargs)
        return result


@HelloDecorator
def add(a, b):
    """Simple function that returns sum of two numbers"""
    return a + b


if __name__ == '__main__':
    output1 = add(2, 2)
    print('Result:: ', output1)

Calling add
Result::  4

OK, We got the decorator working. Lets look at the help and name and doc attributes,

help(add) # --> prints class definition
print(add.__doc__) # --> prints Simple class decorator
print(add.__name__) # --> Raises AttributeError

Let's try to fix the doc strings,
use funct_tools.partial to define get magic method and then funct_tools.update_wrapper to let python know the wrapped function.

from functools import update_wrapper, partial


class HelloDecorator:
    """Simple class decorator"""

    def __init__(self, func):
        self.func = func
        # fixes __name__ and __doc__ attributes
        update_wrapper(self, func)

    def __get__(self, obj):
        """Fixes help description"""
        return partial(self, obj)

    def __call__(self, *args, **kwargs):
        """Simple class call method"""
        print(f'Calling {self.func.__name__}')
        result = self.func(*args, **kwargs)
        return result

now lets check the docs,

help(add) # --> prints
"""
add(a, b)
    Simple function that returns sum of two numbers
"""
print(add.__doc__) # --> prints "add"
print(add.__name__) # --> prints "Simple function that returns sum of two numbers"

Conclusion, while some may class based decorators prefer, I personally do not prefer class based decorator because of the extra boilerplate code to fix the doc strings.

In the next article, we will implement various kinds decorator recipes. Stay tuned for upcoming articles. Connect with me on twitter to get my future articles.

Python Decorators 201

Suresh Kumar — Mon, 18 Apr 2022 12:16:07 +0000

In the previous article Python decorators 101 we learnt, how to create a simple function based decorator and how it works under the hood. In this article, we will improve the traceability and the readability of the decorator function.

Let's add some docstrings to our code. A docstring is a string literal that is written as the first statement in a module, function, class, or method definition. It becomes the doc special attribute of that object. docstring acts a a built-in documentation.


def hello_decorator(func):
    """Simple decorator function"""

    def wrapper(*args, **kwargs):
        """Simple decorator wrapper function"""
        result = func(*args, **kwargs)
        return result

    return wrapper


@hello_decorator
def add(a, b):
    """Simple function that returns sum of two numbers"""
    return a + b


@hello_decorator
def multiply(a, b):
    """Simple function that returns multiplication of two numbers"""
    return a * b


if __name__ == '__main__':

    help(add)

    print(add.__name__)
    print(add.__doc__)

Output

Help on function wrapper in module __main__:

wrapper(*args, **kwargs)
    Simple decorator wrapper function

wrapper
Simple decorator wrapper function

As you notice that, help and doc string of add function returns the doc string from the decorator wrapper function. This is not good for readability and IDEs will show wrong definition and signature of the decorated function. we don't want this to happen. Lets see how we can fix this,

Fix decorated function docs using functools

from functools import wraps


def hello_decorator(func):
    """Simple decorator function"""

    @wraps(func)
    def wrapper(*args, **kwargs):
        """Simple decorator wrapper function"""
        result = func(*args, **kwargs)
        return result

    return wrapper


@hello_decorator
def add(a, b):
    """Simple function that returns sum of two numbers"""
    return a + b


@hello_decorator
def multiply(a, b):
    """Simple function that returns multiplication of two numbers"""
    return a * b


if __name__ == '__main__':

    help(add)

    print(add.__name__)
    print(add.__doc__)

    output1 = add(2, 2)
    print('Result:: ', output1)

    print("=" * 25)
    help(multiply)
    print(multiply.__name__)
    print(multiply.__doc__)
    output2 = multiply(4, 2)
    print('Result:: ', output2)

@wraps function tracks both original function and decorator function and fixes the doc string properly.

In the next article, we will implement various kinds decorator recipes. Stay tuned for upcoming articles. Connect with me on twitter to get my future articles.