Forem: Josue Balandrano Coronel

Developer Secret News

Josue Balandrano Coronel — Thu, 08 Jul 2021 16:59:28 +0000

These are secret news, keep them in the down low, and don’t share them with anyone.

We write unnecessary code.
What we think the code will do is wrong.
Not everything is testable.
Not everyone will understand the code you write.
Code is not always extendible nor manageable.
The Software you build for a company will never be for the greater good.
You are being exploited, and your job exists because others get exploited.
You will never know everything there is about Software.
Technology moves in the direction of money, not progress.
Software will never keep up with real life.
AI will neither save us nor destroy us.
Things change around you; you don’t change things directly.
You don’t need everything in your tech stack.
Your architecture is unnecessarily complicated.
Your documentation is not good.
Every computer system is compromised.

Don't look at these for too long!

Python Fundamentals: Everything you need to know about dataclasses

Josue Balandrano Coronel — Sun, 03 Jan 2021 20:30:21 +0000

Python data classes makes it super easy to write better classes by automatically implementing handy dunder methods like __init__, __str__ (string representation) or __eq__ (equals == operator). Data classes also make it easier to create frozen (immutable) instances, serialize instances and enforce type hints usage.

What's in this article:

How to create a data class
Field definition
- Specify a default value
- Include or exclude fields in automatically implemented dunder methods
- Add field-specific metadata
Customize object initialization
Data classes that we can compare and order
Frozen or immutable instances
Replacing or updating an object instance
Adding class attributes
Inheritance in data classes
Object hash
A use case for data classes
- Benefits of using data classes
- Disadvantages of using data classes

The main parts of a data class are:

@dataclass decorator which returns the same defined class but modified
field function which allow for per-field customizations.

Note: throughout this article we will be using different variations of this Response class. This class is meant to be a simplified representation of an HTTP response.

How to create a data class

To create a data class all we need to do is use the @dataclass decorator on a custom class like this:

fom dataclasses import dataclass

@dataclass
class Response:
    status: int
    body: str

The previous example creates a Response class with a status and body attributes. The @dataclass decorator by default gives us these benefits:

Automatic creation of the following dunder methods:
1. __init__
2. __repr__
3. __eq__
4. __str__
Enforcing type hints usage. If a field in a data class is defined without a type hint a NameError exception is raised.
@dataclass does not create a new class, it returns the same defined class. This allows for anything you could do in a regular class to be valid within a data class.

We can appreciate data classes' benefits by taking a look at the previously defined Response class.

Instance initialization:

>>> resp = Response(status=200, body="OK")

Correct representation of a class:

>>> import logging 
>>> logging.basicConfig(level=logging.INFO)
>>> resp = Response(status=200, body="OK")
>>> logging.info(resp)
... INFO:root:Response(status=200, body='OK')

Instance equality

>>> resp_ok = Response(status=200, body="OK")
>>> resp_500 = Response(status=500, body="Error")
>>> resp_200 = Response(status=200, body="OK")
>>> resp_ok == resp_500
... False
>>> resp_ok == resp_200
... True

Note: we can customize the implementation of each dunder method. We'll see how later in this article.

Field definition

There are two ways of defining a field in a data class.

Using type hints and an optional default value

from dataclasses import dstaclass

@dataclass
class Response:
    body: str
    status: int = 200

The previous class can be instantiated by passing only the message value or both status and message

>>> import logging
>>> logging.basicConfig(level=logging.INFO)
>>> # Create 200 response
>>> resp_ok = Response(body="OK")
>>> logging.info(resp_ok)
... INFO:root:Response(body='OK', status=200)
>>> # Create 500 response
>>> resp_error = Response(status=500, body="error")
>>> logging.info(resp_error)
... INFO:root:Response(body='error', status=500)

Using the field method. This is recommended when there's a need for more fine grained configuration on a field.

By using the field method we can:

Specify a default value

When using the field method we can specify a default value by passing a default parameter:

from dataclasses import dataclass

@dataclass
class Response:
    body: str
    status: int = field(default=200)

In Python it is not recommended to use mutable values as argument defaults. This means it's not a good idea to define a data class like this (the following example is not valid)

from dataclasses import dataclass

@dataclass
class Response:
    status: int
    body: str
    headers: dict = {}

If we could use the previous code every instance of response would share the same headers object and that's not good.

Fortunately data classes help us prevent this by raising an error when something like the example above is used. And if we need to add an immutable object as a default value we can use default_factory.

The default_factory value should be a function with no arguments. Commonly used functions include dict or list :

from dataclasses import dataclass, field

@dataclass
class Response:
    status: int
    body: str
    headers: dict = field(default_factory=dict)

We can then use this class like so:

>>> import logging
>>> logging.basicConfig(level=logging.INFO)
>>> # Create 200 response
>>> resp = Response(status=200, body="OK")
>>> logging.info(resp)
... INFO:root:Response(status=200, body='OK', headers={})

Note: for mutable default values use default_factory

Include or exclude fields in automatically implemented dunder methods

By default every defined fields are used in __init__, __str__, __repr__, and __eq__. The field method allows to specify which fields are used when implementing the following dunder methods:

__init__

from dataclasses import dataclass

@dataclass
class Response:
    body: str
    headers: dict = field(init=False, default_factory=dict)
    status: int = 200

This data class will implement an __init___ method like this one:

def __init__(self, body:str, status: int=200):
    self.body = body
    self.status = status
    self.headers = dict()

This version of the Response class will not allow for a headers value on initialization. Here's how we could use it:

>>> import logging
>>> logging.basicConfig(level=logging.INFO)
>>> # Create 200 response
>>> 
>>> resp = Response(body="Success")
>>> logging.info(resp)
... INFO:root:Response(body='Success', headers={}, status=200)
>>>
>>> # passing a headers param on initialization will raise an srgument error.
>>> resp = Response(body="Success", headers={})
... TypeError: __init__() got an unexpected keyword argument 'headers'
>>>
>>> # 'headers' is an instance attribute and can be used after initialization.
>>> resp.headers = {"Content-Type": "application/json"}
>>> logging.info(resp)
... INFO:root:Response(body='Success', headers={'Content-Type': 'application/json'}, status=200)

Note: Fields that are not used in __init__ method can also be populated after init using __post_init__.

__repr__ and __str__

from dataclasses import dataclass

@dataclass
class Response:
    body: str
    headers: dict = field(repr=False, init=False, default_factory=dict)
    status: int = 200

Now, the Response class will not print the value of headers when an instance is printed.

>>> resp = Response(body="Success")
>>> logging.info(resp)
... INFO:root:Response(body='Success', status=200)

__eq__

from dataclasses import dataclass, field

@dataclass
class Response:
    body: str
    headers: dict = field(compare=False, init=False, repr=False, default_factory=dict)
    status: int = 200

This version of the Response class will not take the headers value into consideration when comparing if an instance is equal to another.

>>> resp_json = Response(body="Success")
>>> resp_json.headers = {"Content-Type": "application/json"}
>>> resp_xml = Response(body="Success")
>>> resp_xml.headers = {"Content-Type": "application/xml"}
>>> resp_json == resp_xml
... True

Both objects are equal because only the status and body values are considered when checking for equality and not the headers value.

Note: when setting compare to False on a field it will not be used to automatically implement any comparable methods (__lt__, __gt__, etc...). More on comparisons later.

Add field specific metadata

We can add metadata to a field. The metadata is a mapping and it's meant to be used by 3rd party libraries. The data classes implementation does not use field metadata at all.

Note: If you decide to use field-specific metadata, be mindful, other 3rd party libraries could overwrite any value. It's recommended to use a specific key to avoid collisions.

from dataclasses import dataclass, field
from typing import Any

@dataclass
class Response:
    body: Any = field(metadata={"force_str": True})
    headers: dict = field(init=False, repr=False, default_factory=dict)
    status: int = 200

This Response class assigns a mapping with the key force_str as metadata. The metadata mapping can be used as configuration to force using the string representation of whatever is passed as body.

To access a field's metadata the fields method can be used.

>>> from dataclasses import fields
>>> resp = Response(body="Success")
>>> fields(resp)
...(Field(name='body',type=typing.Any,default=<dataclasses._MISSING_TYPE object at 0x7f955a0e97f0>,default_factory=<dataclasses._MISSING_TYPE object at 0x7f955a0e97f0>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({'force_str': True}),_field_type=_FIELD),
 Field(name='headers',type=<class 'dict'>,default=<dataclasses._MISSING_TYPE object at 0x7f955a0e97f0>,default_factory=<class 'dict'>,init=False,repr=False,hash=None,compare=True,metadata=mappingproxy({}),_field_type=_FIELD),
 Field(name='status',type=<class 'int'>,default=200,default_factory=<dataclasses._MISSING_TYPE object at 0x7f955a0e97f0>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),_field_type=_FIELD))

The fields method returns a tuple of Fields objects. It can be used on an instance or a class.

To retrieve the body field we can use a comprehension and next

>>> body_field = next(
    (field
     for field in fields(resp)
     if field.name == "body"),
    None
)
>>> logging.info(body_field)
... INFO:root:Field(name='body',type=typing.Any,default=<dataclasses._MISSING_TYPE object at 0x7f955a0e97f0>,default_factory=<dataclasses._MISSING_TYPE object at 0x7f955a0e97f0>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({'force_str': True}),_field_type=_FIELD)
>>> logging.info(body_field.metadata)
... INFO:root:{'force_str': True}

Customize object initialization

The @dataclass decorator automatically implements an __init__ method. By using __post_init__ we can add custom logic on initialization without having to re-implement __init__.

from dataclasses import dataclass, field 
from typing import Any  
from sys import getsizeof

@dataclass
class Response:
    body: str  
    headers: dict = field(init=False, compare=False, default_factory=dict)
    status: int = 200

    def __post_init__(self):
        """Add a Content-Length header on init"""
        self.headers["Content-Length"] = getsizeof(self.body)

When the previous class is instantiated the content length is automatically calculated.

>>> import logging
>>> logging.basicConfig(level=logging.INFO)
>>> # Create 200 response
>>> resp = Response("Success")
>>> logging.info(resp)
... INFO:root:Response(body='Success', headers={'Content-Length': 56}, status=200)

We can also access field specific metadata in __post_init__

from dataclasses import dataclass, is_dstaclass, asdict

@dataclass
class Response:
    body: Any = field(metadata={"force_str": True})
    headers: dict = field(init=False, compare=False, default_factory=dict)
    status: int = 200

    def stringify_body(self):
        """Returns a string representation of the value in body"""
        body = self.body
        if is_dataclass(body):
            body = asdict(body)
        if isinstance(body, dict):
            return json.dumps(body)
        if not isinstance(body, str):
            return str(body)
        return body

    def __post_init__(self):
        """Custom int logic.

        - Check if body is configured to force value as string  
        - Calculate body's length and add corresponding header. 
        """
        body_field = self.__dataclass_fields__["body"]
        if body_field.metadata["force_str"]:
            self.body = self.stringify_body()
        self.headers["Content-Length"] = getsizeof(self.body)

And we can use this class like this:

>>> import logging
>>> logging.basicConfig(level=logging.INFO)
>>> # Create 200 response
>>> resp = Response(body={"message": "Success"})
>>> logging.info(resp)
... INFO:root:Response(body='{"message": "Success"}', headers={'Content-Length': 71}, status=200)

The body value is automatically serialized into a string and stored in the calss on initialization.

The previous example is mainly to show custom initialization logic. In reality you might not want to store the string representation of a response body, instead it's better to make the class serializable.

Note: you can use asdict to transform a data class into a dictionary, this is useful for string serialization.

We can also specify fields which will not be attributes of an instance but will be passed onto the __post_init__ hook by using dataclasses.InitVar

from dataclasses import dataclass, is_dstaclass, asdict, InitVar

@dataclass
class Response:
    body: Any
    headers: dict = field(init=False, compare=False, default_factory=dict)
    status: int = 200
    force_body_str: InitVar[bool] = True

    def stringify_body(self):
        """Returns a string representation of the value in body"""
        body = self.body
        if is_dataclass(body):
            body = asdict(body)
        if isinstance(body, dict):
            return json.dumps(body)
        if not isinstance(body, str):
            return str(body)
        return body

    def __post_init__(self, force_body_str):
        """Custom int logic.

        - Check if body is configured to force value as string  
        - Calculate body's length and add corresponding header. 
        """
        if force_body_str:
            self.body = self.stringify_body()
        self.headers["Content-Length"] = getsizeof(self.body)

We can easily configure if the value of body will be stored as string or not:

>>> import logging
>>> logging.basicConfig(level=logging.INFO)
>>> # Create 200 response where 'body' will be stored as a dict.
>>> resp = Response(body={"message": "Success"}, force_body_str=False)
>>> logging.info(resp)
... INFO:root:Response(body={'message': 'Success'}, headers={'Content-Length': 232}, status=200)
>>> # Create 200 response where 'body' will be stored as a string.
>>> resp_str = Response(body={"message": "Success"})
>>> logging.info(resp)
... INFO:root:Response(body='{"message": "Success"}', headers={'Content-Length': 71}, status=200)

Data classes that we can compare and order

By default a data class implements __eq__. We can pass an order boolean argument to the @dataclass decorator to also implement __lt__ (less than), __le__ (less or equal), __gt__ (greater than) and __ge__ (greater or equal).

The way these rich comparison methods are implemented take every defined field and compare them in the order they are defined until there's a value that's not equal.

from dataclasses import dataclass

@dataclass(order=True)
class Response:
    body: str
    status: int = 200

The previous data class can now be compared using >=, <=, > and < operands. The best use case for this is when sorting:

>>> resp_ok = Response(body="Success")
>>> resp_error = Response(body="Error", status=500)
>>> sorted([resp_ok, resp_error])
... [Response(body='Error', status=500), Response(body='Success', status=200)]

In this example resp_error goes before resp_ok because the unicode value of E is less than the unicode value of S.

Note: implementing rich comparison methods allow us to easily sort objects.

The implemented comparison methods will check the value of body, if both are equal it will continue to status. If the class had more fields the rest of the fields would be checked in order until a non-equal value is found.

The previous example is valid but it does not make much sense to sort Response objects based on the body and status values. It makes more sense to sort them on the length of the body. We can specify which fields to use in comparison by using the field method:

from dataclasses import dataclass, field 
from sys import getsizeof

@dataclass(order=True)
class Response:
    body: str = field(compare=False)
    status: int = field(compare=False, default=200)
    _content_length: int = field(compare=True, init=False)

    def __post_init__(self):
        """Calculate and store content length on init"""
        self._content_length = getsizeof(self.body)

In the previous example we specified which fields are used when implementing comparison methods by passing a boolean compare parameter to the field method.

This class will now be sorted by the size of the value of body. We can also judge if an instance is larger than another judging by the size of the value of body.

>>> resp_ok = Response(body="Success")
>>> resp_error = Response(body="Error", status=500)
>>> sorted([resp_ok, re sp_error])
...[Response(body='Error', status=500, _content_length=54), Response(body='Success', status=200, _content_length=56)]
>>> # resp_error is smaller than resp_ok because
>>> # "Error" is smaller than "Success"

One downside of this implementation is that two given instances will be equal as long as the size of the body attribute is the same.

>>> resp_ok = Response(body="Success")
>>> resp_error = Response(body="Failure")
>>> resp_ok == resp_error
... True
>>> # both instances are equal because Success and Failure have the same amounts of chars
>>> # and getsizeof() returns the same size for both strings.

For equality it would be better to also check if the value of the body attribute is the same:

from dataclasses import dataclass, field 
from sys import getsizeof

@dataclass(order=True)
class Response:
    _content_length: int = field(compare=True, init=False)
    body: str = field(compare=True)
    status: int = field(compare=False, default=200)

    def __post_init__(self):
        """Calculate and store content length on init"""
        self._content_length = getsizeof(self.body)

By moving the _content_length field definition above body the length of the content will be used first for any comparisons. We also set the body field as a compare field. When checking for equality if the content length is the same the actual value of body will be checked, making for a better way to check for equality.

>>> resp_ok = Response(body="Success")
>>> resp_error = Response(body="Failure")
>>> resp_ok == resp_error
... False

This also works for sorting since response instances with the same content length will be sorted by the weight of the characters. Sorting will always yield the same order.

>>> sorted([resp_ok, resp_error])
... [Response(_content_length=56, body='Failure', status=200), Response(_content_length=56, body='Success', status=200)]

Note: the order in which fields are defined matter for comparisons.

Frozen or immutable instances

We can create frozen instances by passing frozen=True to the @dataclass decorator.

from dataclasses import dataclass, field

@dataclass(frozen=True)
class Response:
    body: str
    status: int = 200

This is helpful when you want to make sure read-only data is not mistakenly modified by your code or 3rd party libraries. If you try to modify a value a FrozenInstanceError exception will be raised:

>>> resp_ok = Response(body="Success")
>>> resp_ok.body = "Done!"
... dataclasses.FrozenInstanceError cannot assign to field 'body'

In Python we cannot really have immutable objects. If you make an effort you can still modify a frozen data class instance:

>>> import logging
>>> logging.basicConfig(level=logging.INFO)
>>> # Check values of 'resp_ok'
>>> logging.info(resp_ok)
... INFO:root:Response(body='Success', status=200)
>>>
>>> object.__setattr__(resp_ok, "body", "Done!")
>>> # We have modified a "frozen" instance
>>> logging.info(resp_ok)
... INFO:root:Response(body='Done!', status=200)

This is unlikely to happen but it's worth knowing.

Note: use a frozen data class when using read-only data to avoid unwanted side-effects.

Note: You cannot implement __post_init__ hook in a frozen data class.

Replacing or updating an object instance

The data classes module also offers a replace method which created a new instance using the same class. Any updates are passed as parameters:

from dataclasses import dataclass, replace

@dataclass(frozen=True)
class Response:
    body: str
    status: int = 200

>>> import logging
>>> logging.basicConfig(level=logging.INFO)
>>> # Create 200 response
>>> resp_ok = Response(body="Success")
>>> logging.info(resp_ok)
... INFO:root:Response(body='Success', status=200)
>>> # Replace instance
>>> resp_ok = replace(resp_ok, body="OK")
>>> logging.info(resp_ok)
... INFO:root:Response(body='OK', status=200)

The value of body is updated and the value of status is copied over. Any reference to resp_ok is now pointing to the new, updated object.

Note: using replace ensures _init_ and __post_init__ are run with the updated values.

Adding class attributes

In Python a class can have a class attribute, the difference from instance attributes are mainly these two:

Class attribute are defined outside __init__
Every instance of the class will share the same value of a class attribute.

We can define class attributes in a data class by using the pseudo-field typing.ClassVar

from dataclasses import dataclass
from typing import ClassVar, Any
from sys import getsizeof 
from collections.abc import Callable

@dataclass
class Response:
    body: str 
    _content_length: int = field(default=0, init=False)
    status: int = 200
    getsize_fun: ClassVar[Callable[[Any], int]] = getsizeof

    def __post_init__(self):
        """Calculate content length by using getsize_fun"""
        self._content_length = self.getsize_fun(self.body)

In this version of Response we can specify a function used to calculate the content's size. By default sys.getsizeof is used.

from functools import reduce

def calc_str_unicode_weight(self, string: str):
    """Calculates strn weight by adding each character's unicode value"""
    return reduce(lambda weight, char: weight+ord(char), string, 0)

@dataclass
class ResponseUnicode(Response):
    getsize_fun: ClassVar[Callable[[Any], int]] = calc_str_unicode_weight

>>> import logging
>>> logging.basicConfig(level=logging.INFO)
>>> # Create 200 response, using getsizeof to calculate content length
>>> resp_ok = Response(body="Success")
>>> logging.info(resp_ok)
... INFO:root:Response(body='Success', _content_length=56, status=200)
>>> # Override function to use when calculating content length
>>> resp_ok_unicode = ResponseUnicode(body="Success")
>>> logging.info(resp_ok_unicode)
... INFO:root:ResponseUnicode(body='Success', _content_length=729, status=200)

To overwrite the functino used to calculate the content lenght we subclass Response and pass the function we want as getsize_fun

Note: fields that use ClassVar are not used in data class mechanism like __init__, equality or comparison dunder methods.

Inheritance in data classes

When using inheritance with data classes fields are merged, meaning child classes can overwrite field definitions. Everything else works the same since the @dataclass decorator returns an old regular Python class.

from dataclasses import dataclasses

@dataclass
class Response:
    body: str
    status: int
    headers: dict

@dataclass
class JSONResponse(Response):
    status: int = 200
    headers: dict = field(default_factory=dict, init=False)

    def __post_init__(self):
        """automatically add Content-Type header"""
        self.headers["Content-Type"] = "application/json"

In the previous example the parent class Response defined the basic fields and the children class JSONResponse overwrites the headers field and sets a default value for the status field.

>>> import logging
>>> logging.basicConfig(level=logging.INFO)
>>> # Create 200 response
>>> resp_ok = JSONResponse(body=json.dumps({"message": "OK"}))
>>> logging.info(resp_ok)
... INFO:root:JSONResponse(body='{"message": "OK"}', status=200, headers={'Content-Type': 'application/json'})

Object hash

The @dataclass decorator will automatically implement __hash__ method if the parameters frozen and eq are True. frozen is False by default and eq is True by default.

from dataclasses import dataclass

@dataclass(frozen=True)
class Response:
    body: str
    status: int = 200

We can now use any instance of this class as a key in a dict or in a set. For I stance, we can create a mapping of responses to users

>>> import logging
>>> logging.basicConfig(level=logging.INFO)
>>> # Create 200 response
>>> resp_ok = Response(body="Success")
>>> # Create 500 response
>>> resp_error = Response(body="Error", status=500)
>>> # Create a mapping of response -> usernames
>>> responses_to_users = {
...     resp_ok: ["j_mccain", "a_perez"],
...     resp_error: ["d_dane", "b_rodriguez"]
... }
>>> logging.info(responses_to_users[resp_ok])
... INFO:root:['j_mccain', 'a_perez']

We can force a __hash__ function implementation even if we don't set frozen and eq to True by passing force_hash=True to the @dataclass decorator. This should only be used if you are 100% sure you need the functionality.

A use case for data classes

Throughout this article we've made different updates to a Response class which represents a simplified HTTP response object. Let's put everything together.

For simplicity we're gonna write every class and function we're going to use in the same file, in really these should be spread out into sensible modules.

Note: This is still not a 100% real HTTP response class, but it has enough information to see how we can use data classes

import logging
from sys import getsizeof
from typing import ClassVar, Any, Optional, cast
from collections.abc import Callable
from dataclasses import dataclass, field, InitVar, asdict

logging.basicConfig(level=logging.DEBUG)

class APIException(Exception):
    """Custom API exception."""

    def __init__(self, message: str, **kwargs):
        self.message = message;
        self.data: Dict[str: Any] = kwargs

class PositiveNumberValidator:
    """Descriptor to make sure a value is a positive number"""

    def __set_name__(self, owner, name):
        self.name = f"_{name }"

    def __get__(self, obj, objtype=None):
        return getattr(obj, self.name)

    def __set__(self, obj, value):
        self.validate(value)
        setattr(obj, self.name, value)

    def validate(self, value):
        if not isinstance(value, int):
            raise AttributeError(f"value of '{self.name}' must be a number")
        if value < 0:
            raise AttributeError(f"value of '{self.name}'' must be a positive number")

@dataclass(eq=False)
class Pager:
    """Pager class.

    This class is to hold any pager related data sent in the response.
    The prev and next paramteres are meant to be links sent in the 'Link' header.
    """

    page: int = cast(int,PositiveNumberValidator())
    prev: str = ""
    next: str = ""


@dataclass(order=True)
class HTTPResponse:
    """Parent HTTPResponse

    This class can:
      1. Ordered and compared by its content size and body using regular operators
      2. Pass a 'content_type' string to be used as header value.
      3. Update headers directly as a regular dictionary.
      4. Customize the function to calculate content-length.
    """
    _content_length: int = field(init=False)

    body: Any
    pager: Optional[Pager] = field(default=None, compare=False)
    headers: dict = field(default_factory=dict, init=False, compare=False)
    status: int = field(default=200, compare=False)

    content_type: InitVar[str] = "text/html"

    getsize_fun: ClassVar[Callable[[Any], int]] = getsizeof

    def __post_init__(self, content_type):
        """automatically calculate header values."""
        self._content_length = self.getsize_fun(self.body)
        self.headers["Content-Length"] = self._content_length
        self.headers["Content-Type"] = content_type

@dataclass(frozen=True)
class JSONBody:
    """Class to hold data sent in a JSON Response.

    This class is immutable to avoid any unwanted modification of data.
    """
    message: str
    data: dict

    @classmethod
    def from_exc(cls: type, exc: APIException):
        """Initializes a JSONBody object from an exception."""
        return cls(message=str(exc), data=getattr(exc, "data", {}))

@dataclass
class JSONResponse(HTTPResponse):
    """Class to represent a JSON Response. Child of HTTPResponse."""
    body: JSONBody
    content_type: InitVar[str] = "application/json"

Here, we created a parent class HTTPResponse which will hold the basic data needed to send an HTTP response. Then, we created a JSONResponse class which inherits from HTTPResponse and overwrites body and content_type attributes. Overwriting these attributes allow us to specify a different default content type and a different type for the body. There is also a Pager class which is used to hold any data related to pagination that's sent in the response. The Pager class uses a descriptor to validate that page is always a positive number. And we also have a JSONBody which can be initialized by passing a message and a dictionary for data or can be initialized by passing an APIException instance. APIException is a custom exception we created to store an exception message and also some data related to said exception.

Here's some basic examples how we can use these classes:

We can create a JSONResponse only by passing a JSONBody instance:

>>> import logging
>>> logging.basicConfig(level=logging.INFO)
>>> body = JSONBody(message="Success", data={"values": ["value1", "value2"]})
>>> resp = JSONResponse(body)
>>> logging.info("resp: %s", resp)
... INFO:root:resp: JSONResponse(_content_length=48, body=JSONBody(message='Success', data={'values': ['value1', 'value2']}), pager=None, headers={'Content-Length': 48, 'Content-Type': 'application/json'}, status=200)

This is already a powerful class and the code needed to implement it is quite short

We can also pass a Pager instance to make it a more robust response:

>>> pager = Pager(1, prev="?prev=0", next="?next=2")
>>> resp = JSONResponse(body, pager=pager)
>>> logging.info("resp: %s", resp)
... INFO:root:resp: JSONResponse(_content_length=48, body=JSONBody(message='Success', data={'values': ['value1', 'value2']}), pager=Pager(prev='?prev=0', next='?next=2'), headers={'Content-Length': 48, 'Content-Type': 'application/json'}, status=200)

We can easily conver data classes to dictionaries or tuples even when using nested data classes:

>>> from dataclasses import asdict, astuple
>>> logging.info("serialized resp: %s", asdict(resp))
... INFO:root:serialized resp: {'_content_length': 48, 'body': {'message': 'Success', 'data': {'values': ['value1', 'value2']}}, 'pager': {'prev': '?prev=0', 'next': '?next=2'}, 'headers': {'Content-Length': 48, 'Content-Type': 'application/json'}, 'status': 200}
>>> logging.info("resp as tuple: %s", astuple(resp))
... INFO:root:resp astuple: (48, ('Success', {'values': ['value1', 'value2']}), ('?prev=0', '?next=2'), {'Content-Length': 48, 'Content-Type': 'application/json'}, 200)

With a more real example we can see the strengths and weaknesses of data classes.

Benefits of using data classes

We can create powerful classes with less code.
Type hints are enforced for every class and instance attribute.
We can customize how special dunder methods are implemented.
We can use data classes in the same way we use regular classes. In the previous example we used descriptors and class methods without an issue.
Inheritance can be used to make it easier to use data classes.
It's esier to serialize instsances to dictionaries or tuples.
We can mix regular classes and data classes.

Disadvantages of using data classes

When creating data classes that can be compared and ordered the order in which you define the fields matters. Read-ability can take a hit because of this. It is recommended to try and separate fields by type. In the HTTPResponse class we have first private attributes, then instance attributes, init only parameters and class attributes.
Field definition order also matters when using default values. Since __init__ 's arguments are implemented using the same order the fields are defined, we have to first define attributes without default values and then attributes with default values.
When using frozen=True we cannot update values in __post_init__
We have to manually optimize attribute access if needed. Meaning, adding __slots__. Real Python has a great example of this.

I hope this article sheds some light on how and when to use data classes. If you like it, please follow this blog and make sure to follow me on twitter.

Introduction to Python logging

Josue Balandrano Coronel — Tue, 01 Dec 2020 22:56:27 +0000

Logging is one of the best way to keep track of what is going on inside your code while it is running. An error message can tell you details about the state of the application when an error happens. But proper logging will make it easier to see the state of the application right before the error happened or the path the data took before the error happened.

What's in this article?

Basic overview of Python's logging system.
Explanation of how each logging module interact with each other.
Examples on how to use different types of logging configurations.
Things to look out for
Recommendations for better logging

If you want you can jump to a basic configuration example or a full fledged example used in an application.

The basics

At its simplest form a log message, in python is called a LogRecord, has a string (the message) and a level. The level can be:

CRITICAL: Any error that makes the application stop running.
ERROR: General error messages.
WARNING: General warning messages.
INFO: General informational messages.
DEBUG: Any messages needed for debugging.
NOTSET: Default level used in handlers to process any of the above types of messages.

Python logging library has different modules. Not every module is necessary to start logging messages.

Loggers: the object that implements the main logging API. We use a logger to create logging events by using the corresponding level method, e.g. my_logger.debug("my message"). By default there's a root logger which can be configured using basicConfig and we can also create custom loggers.
Formatters: these objects are in charge of creating the correct representation of the logging event. Most of the time we ant to create a human-readable representation but in some cases we can output a specific format like a json object.
Filters: We can use filters to avoid printing logging events based on more complicated criteria than logging levels. We can also use filters to modify logging events.
Handlers: everything comes together in the handlers. A handler defines which formatters and filter a logger is going to use. A handler is also in charge of sending the loging output to the corresponding place, this could be stdout, email or almost anythihg else we can think of.

Loggers can have filters and handlers. Handlers can have filters and formatters. We will dive into each one of these parts but it's a good idea to keep in mind how they relate with each other. Here's a visual representation:

Python logger structure

Following Python's philosophies the logging library can be easily used, for example:

import logging

logging.warning("Warning log message.")

And that's it. The example above will print a log message with WARNING level.
If you run the code above you will see this output:

WARNING:root:Warning log message.

The first part of the output (WARNING:root:), comes from
the default logging configuration. The default configuration
will only print WARNING and above levels and will prepend the log level and the name of the logger -- since we haven't created any loggers the root logger is used. More on this later --.

The logger in the example above is the root logger. This is a logger which is the parent of every logger that's created. Meaning, if you configure the root logger then you are basically configuring every logger you create.

What is `basicConfig`?

To quickly configure any loggers used in your application you can use basicConfig. This method will by default configure the root logger with a StreamHandler to print log messages to stdout and a default formatter (like the one in the example above).

We can configure the logging format using basicConfig. For instance, if we would like to print only the log level, line number, log message and log DEBUG events and up we can do this:

import logging
logging.basicConfig(
    format="[%(levelname)s]:%(lineno)s - %(message)s",
    level=logging.DEBUG
)

⚡ Note:

By using logging.basicConfig we are configuring the root logger

For each log event there is an instance of LogRecord. We can set the format for our log messages using the LogRecord class' attributes and %-style formatting -- %-style formatting is still
used to maintain backwards compatibility --. Since Python 3.2 we can also use $ and {} style to format messages,
but we have to specify the style we're using, by default %-style is used. The style is configured by using the style parameter in basicConfig (logging.basicConfig(style='{')). With the updated configuration we can now see every log level message, for example:

[CRITICAL]:9 - Critical log message.
[ERROR]:10 - Error log message.
[WARNING]:11 - Warning log message.
[INFO]:12 - Info log message.
[DEBUG]:13 - Debug log message.

🐍 Best Practice
Define a custom log format for easier log parsing.

⚡ Note:
Use the LogRecord class' attributes to configure a custom format.

basicConfig will always be called by the root logger if no handlers are defined, unless the parameter force is set to True (logging.basicConfig(force=True)). By default basicConfig will configure the root logger to output logs to stdout, with the default format of {level}:{logger_name}:{message}.

Custom Loggers

Python's logging library allow us to create custom loggers which will always be children of the root logger.

We can create a new logger by using logging.getLogger("mylogger").
This method only accepts one parameter, the name of the logger. getLogger is a get_or_create method. We can use getLogger with the same name value and we'll be working with the same logger configuration regardless if we're doing this in a different class or module.

The logging library uses dot notation to
create hierarchies of loggers. Meaning, if we create three loggers with the names app, app.models and app.api the parent logger will be app and, app.db and app.api will be the children. Here's a visual representation:

 + app # main logger
 |
 + app.api # api logger, child of "app" logger
   |
   - app.api.routes # routes logger, child of "app" and "app.api" loggers
   |
   - app.api.models # models logger, sibling of "app.api.routes" logger
 |
 - app.utils # utils logger, sibling of "app.api" logger

We can get a module's dot notation name from the global variable __name__. Using __name__ to create our custom loggers simplifies configuration and avoids collisions:

# project/api/utils.py

import logging

# Use __name__ to create module level loggers
LOG = logging.getLogger(__name__)

def check_if_true(var):
    """Check if variable is `True`.

    :param bool var: Variable to check.
    :return bool: True if `var` is truthy.
    """
    LOG.info("This logger's name is 'project.api.utils'.")
    return bool(var)

⚡ Note:
If you are defining loggers at the module level (like the example above)
is better to stick to global variable naming. Meaning, use LOG or LOGGER
instead of the lowercase version log or logger.

🐍 Best Practice

Create loggers with custom names using __name__ to avoid collision and for
granular configuration.

When using a logger the message's level is checked against the logger's level attribute if it's the same or above then the log message is passed to the logger that's being used and every parent unless one of the logger in the hierarchy sets propagate to False -- by default propagate is set to True --.

How to configure loggers, formatters, filters and handlers

We've talked about using basicConfig to configure a logger but there are other ways to configure loggers. The recommended way of creating a logging configuration is using a dictConfig. The examples in this article will be using dictConfig.

As we've learned before python's logging
library already comes with some useful default values, which makes defining our own formatters, filters and handlers optional. As a matter of fact when using a dictionary to configure logging the only required key is version, and currently the only valid value is 1.

Whenever we use a logger (LOG.debug("Debug log message.")) the first thing that happens is that a LogRecord object is created with our log message and other attributes. This LogRecord instance is then passed to any filters attached to the logger instance we used. If the filter does not reject the LogRecord
instance then the LogRecord is passed to the configured handlers. If any of the configured handlers are enable for the level in the passed LogRecord
then the handlers apply any configured filters to the LogRecord. Finally, if the LogRecord is not rejected by any of the filter the LogRecord is emitted. A more detailed diagram can be seen in python's documentation.

To simplify Python's logging flow we can focus on what happens in a single logger:

Python logging flow simplified

Here's a few important things to note:

The diagram above is from the point of view of the logger used.
Filters, handlers and formatters are defined once and can be used multiple times.
Only the filters and formatters assigned to the parent's handler are applied to the LogRecord (this is the loop that says "Parent Loggers*")

Every log event created will be processed except in the following cases:

The logger or the handler configured in the logger are not enabled for the log level used.
A filter configured in the logger or the handler rejects the log event.
A child logger has propagate=False causing events not to be passed to any of the parent loggers.
Your are using a different logger or the logger is not a parent of the one being used.

Formatters

A formatter object transforms a
LogRecord instance into a human readable string or a string that will be consumed by an external service. We can use any LogRecord attribute
or anything sent in the logging call as the extra parameter.

⚡ Note:
Formatters can only be set to handlers

For example, we can create a formatter to show all the details of where and when a log message happened:

LOGGING_CONFIG = {
    "version": 1,
    "formatters": {
        "detailed": {
            "format": "[APP] %(levelname)s %(asctime)s %(module)s "
                      "%(name)s.%(funcName)s:%(lineno)s: %(message)s"
        }
    },
    "handlers": {
        "console": {
            "class": "logging.StreamHandler",
            "formatter": "detailed",
            "level": "INFO"
        }
    }
}

When using a dictionary to define a formatter we have to use the "formatters" key. Any key inside the "formatters" object will become a formatter. Every key/value inside the formatter object will be sent as parameters when intializing the formatter instance.

Whenever this formatter is used it will print the level, date and time, module name, function name, line number and any string sent as parameter. We can add other variables not present in LogRecord's attributes by using
the extra attribute:

LOGGING_CONFIG = {
    "version": 1,
    "formatters": {
        "short": {
            "format": "[APP] %(levelname)s %(asctime)s %(module)s "
                      "%(name)s.%(funcName)s:%(lineno)s: "
                      "[%(session_key)s:%(user_id)s] %(message)s" # add extra data
        }
    },
    #[...]
}

And to send that extra data we can do it like this:

# app.api.UserManager.list_users

LOG.info(
    "Listing users",
    extra={"session_key": session_key, "user_id": user_id}
)

The output would be:

[APP] INFO 2020-11-07 20:47:00,123 user_manager app.api.user_name.list_users:15 [123abcd:api_usr] Listing users

The downside of referencing variables sent via the extra parameter in a format is that if the variable is not passed the log event is not going to be logged because the string cannot be created.

⚡ Best Practice

Make sure to send every additional variable that the configured
formatter references if using the extra parameter

Filters

Filters are very interesting. They can be used both at the logger level or at the handler level, they can be used to stop certain log events from being logged and they can also be used to inject additional context into a LogRecord instance which will be, eventually, logged.

The Filter class in Python's logging library filters LogRecords by logger name. The filter will allow any LogRecord coming from the logger name configured in the filter and any of its children.

For instance, if we have these loggers configured:

 + app.models
 | |
 | - app.models.users
 |
 + app.views
   |
   - app.views.users
   |
   - app.views.products

And we define the filter config like this:

LOGGING_CONFIG = {
    "version": 1,
    "filter": {
        "views": {
            "name": "app.views"
        }
    },
    "handlers": {
        "console": {
            "class": "logging.StreamHandler",
            "level": "INFO",
            "filter": "views"
        }
    },
    "loggers": {
        "app": { "handlers": ["console"] }
    }
}

We use the "filter" key in a dictionary configuration to define any filters. Each key inside the "filter" object will become a filter we can later reference. Every key and value inside the filter object we create will be sent as parameters when intializing the filter.

The previous configuration will only allow LogRecord coming from the app.views, app.views.users and app.views.products loggers.

When you set a filter to a specific logger the filter will only be used when calling that logger directly and not when a descendant of said logger is used. For example, if we had set the filter in the previous example to the app.view logger instead of the app logger handler. The filter will not reject any LogRecord coming from app.models loggers simply because when using the logger app.models, or any of it's childrens, the filter will not be called.

Let's see how can we create custom filters to prevent LogRecords to be processed based on more complicated conditions and/or to add more data to a LogRecord.

Since Python 3.2 you can use any callable that accepts a record parameter

def custom_filter(record):
    """Filter out records that passes a specific key argument"""

    # We can access the log event's argument via record.args
    return record.args.get("instrumentation") == "console"

The way to configure these custom classes/callables using a dictionary or a file is by using the special keyword (). Whenever Python's logging config sees () it will create an instance of the class (dot notation has to be used).

⚡ Note:
When using a dictionary to configure logging you can use the ()
keyword to configure custom handlers or filters

Another interesting thing about filters is that they see virtually every LogRecord that might be logged. This makes filters a great place to further customize LogRecords. This is called "adding context".

Custom filter example

The following custom filter will apply a mask to every password passed to a LogRecord if the pwd or password key is used.

# module: app.logging.filters

def pwd_mask_filter(record)
    """A factory function that will return a filter callable to use."""

    def filter(record):
        # a Logger cord instance holds all it's arguments in record.args
        def mask_pwd(record):
            return '*' * 20

        if record.args.has_key("pwd"):
            record.args["pwd"] = mask_pwd()
        elif record.args.has_key("password"):
            record.args["pwd"] = mask_pwd()

    return filter

LOGGING_CONFIG = {
    "version": 1,
    "formatters": {
        "detailed": {
            "format": "[APP] %(levelname)s %(asctime)s %(module)s "
                      "%(name)s.%(funcName)s:%(lineno)s: %(message)s"
        }
    },
    "filters": {
        "mask_pwd": {
            "()": "app.logging.filters.mask_pwd"
        }
    },
    "handlers": {
        "console": {
            "class": "logging.StreamHandler",
            "formatter": "detailed",
            "level": "INFO",
            "filters: ["mask_pwd"]
        }
    }
}

When using () in a dictionary configuration the referenced module will be imported and instantiated. In this case the factory function mask_pwd will be called and the actual function that handles filtering will be returned.

Handlers

Handlers are objects that implement how formatters and filters are used. if we remember the flow diagram we can see
that whenever we use a logger the handlers of all the parent loggers are going to be called recursivley.
This is where the entire picture comes together and we can take a look at a complete loggin configuration
using everything else we've talked about.

Python offers a list of very useful handlers. Handler classes have a very simple API and in most
cases we will only use these classes to add a formatter, zero or more filters and setting the logging level.

Note:

Only one formatter can be set to a handler.

For instance, the following configuration makes sure that every log is printed to the standard output, uses a filter
to select a different handler depending if the application is running in debug mode or not and it uses a different
format for each different environment (based on the debug flag).

Full logging configuration example

LOGGING = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "long": {
            "format": "[APP] {levelname} {asctime} {module} "
                      "{name}.{funcName}:{lineno:d}: {message}",
            "style": "{"
        },
        "short": {
            "format": "[APP] {levelname} [{asctime}] {message}",
            "style": "{"
        }
    },
    "filters": {
        "debug_true": {
            "()": "filters.require_debug_true_filter"
        },
        "debug_false": {
            "()": "filters.require_debug_false_filter"
        }
    },
    "handlers": {
        "console": {
            "level": "INFO",
            "class": "logging.StreamHandler",
            "formatter": "short",
            "filters": ["debug_false"]
        },
        "console_debug": {
            "level": "DEBUG",
            "class": "logging.StreamHandler",
            "formatter": "long",
            "filters": ["debug_true"]
        }
    },
    "loggers": {
        "external_library": {
            "handlers": ["console"],
            "level": "INFO"
        },
        "app": {
            "handlers": ["console", "console_debug"],
            "level": "DEBUG"
        }
    }
}

When using a dictionary configuration disable_existing_loggers is set to True by default.
What this does is that it will grab all the loggers created before the configuration is applied and disable them so they cannot process any log events. This is specially important when creating loggers at the module level.

Say we have a app.models.client module in which we create a logger like this:

# app.models.client

import logging

LOG = logging.getLogger(__name__)

class Client:
    # ...

And then in our application's entry point we import our models and configure our logging:

# app.__main__

import logging
from app.settings.logging import LOGGING_CONFIG # A dictionary
from app.models import Client

logging.config.dictConfig(LOGGING_CONFIG)

In this case the logger created in app.models.client will not work unless disable_existing_loggers. Because when we import the Client class from app.models we are initializing the logger used in app.models.client. Then, after the logger is already initialized we apply our configuration, making any existing loggers disabled.

When definig a handler using a dictionary, any key that is not class, level, formatter or filters will be passed as a parameter to the handler's class specified for instantiation.

Filters cannot be used when using a file to configure logging. One more reason why is better to use a dictionary for configuration.

Things to lookout for when configuring your loggers

Filters can be set both to Loggers and Handlers. If a filter is set at the logger level it will only be used when that specific logger is used and not when any of its descendants are used.
File config is still support purely for backwards compatibility. Avoid using file config.
() can be used in config dictionaries to specify a custom class/callable to use as formatter, filter or handler. It's assumed a factory is referenced when using () to allow for complete initialization control.
Go over the list of cases when a log event is not processed if you don't see what you expect in your logs.
Make sure to set disable_existing_loggers to False when using dict config and creating loggers at the module level.

Recommendations for better logging

Take some time to plan how your logs should look like and where are they going to be stored.
Make sure your logs are easily parsable both visually and programmatically, like using grep.
Most of the time you don't need to set a filter at the logger level.
Filters can be used to add extra context to log events. Make sure any custom filters are fast. In case your filters do external calls consider using a queue handler
Log more than errors. Make sure whenever you see an error in the logs you can trace back the state of the data in the logs as well.
If you are developing a library, always set a NullHandler handler and let the user define the logging configuration.
Use the handlers already offered by Python's logging library unless is a very special case. Most of the time you can hand off the logs and then process them with a more specialized tool. For example, send logs to syslog using syslogHandler handler and then use rsyslog to maintain your logs.
Create a new logger using the module's name to avoid collisions, logging.getLogger(__name__)

⚡ Note:
If you want to use the test the configuration above or just play around with different configs,
checkout the complementary repo to this article

Special thanks to Žan Anderle for proof reading.

Follow me on twitter @rmcomplexity