Forem: Alexander Lavrov

How I taught and then wrote a Python tutorial

Alexander Lavrov — Sun, 05 Sep 2021 18:14:31 +0000

Last year I worked as a teacher in one of the provincial training centers (hereinafter referred to as the TC) specializing in programming training. I will not name this training center, I will also try to do without the names of companies, the names of authors, etc.

So, I worked as a teacher in Python and Java. This TC bought methodological materials for Java, and they launched Python when I came and offered it to them.

A manual for students (in fact, a textbook or a self-help guide) I wrote on Python, but the teaching of Java and the methodological materials that were used there had no small impact.

To say that they were terrible is to say nothing. The modus of the Java textbook, which is supplied by a very well-known company in Russia, was not to teach a person the basics of this language in general and the OOP paradigm in particular, but to make parents who came to open lessons see how their son or daughter copied a snake or chess from the textbook. Why do I say written off? Very simply, the fact is that the textbook provided entire sheets (A4) of code, some points of which were not explained. As a result, the teacher either has to control at what point in the code each student is currently located, explaining each line, or everything slides into cheating.

You will say: "Well, what's wrong, let the teacher work better, and chess and the snake are finally cool!"

Well, everything would be cool if the number of people in the group was not under 15, and this is already significant if you are going to follow everyone, explaining: "And yet, what are we writing this?"

In addition to the number of people in the group, there is another problem associated with this manual. The code is written... how should I tell you, just awful. A set of anti-moderns, archaic, since the textbook has not been updated for a long time, and our favourite, of course, is the style guide. Therefore, even if you control all your wards and can quickly and clearly explain to them what the code being written off means, the code itself is so terrible that it will teach them the wrong thing, to put it mildly.

Well, the finale that literally destroys this textbook - from the very beginning there is no at least some adequate introduction explaining what data types are, that they are object and primitive, what criterion checks the property that generates this dichotomy, etc. In the first chapter, you and your students are invited to make (write off) a program that makes a window and writes "Hello!" there, but it does not explain what this code sheet actually means, only links to further classes, for example, "main" is mentioned — the entry point, but the very concept of "entry point"is not even mentioned.

To sum up, this waste paper was a meme even among teachers and management. She did not teach the children absolutely nothing, once I came across a group that had been studying for a year on these materials, as a result, they could not even write a cycle, I note that they were all very smart and soon everything was not so bad. Most colleagues tried to deviate from the methodological materials so that the material was absorbed, and not just flew into the air, although there were less conscientious people who considered it normal that their student writes off without any explanations.

When it became clear that I would leave the UC, and the Python program should somehow be continued next year, I began to write my textbook. In short, I divided it into two parts, in the first I explained everything about data types, their essence, operations with them and language instructions. Between the topics, I did QnA so that the future teacher could understand how the student learned the topic. Well, in the end, I did a small task-a project. The first part explains the basics of the language in this way and chews them, which is about 12-13 lessons of 30-40 minutes each. In the second part, I have already written about OOP, described how the implementation of this paradigm in Python differs from most others, made many links to the style guide, etc. To summarize — I tried to be as different as possible from what was in the Java tutorial. I recently wrote to the current Python teacher, asked for feedback on the materials, and now I am glad that everything is fine, that the children really understand Python programming.

What conclusion I would like to draw from this story: my dear parents, if you decide to send your child to the UC, then carefully monitor what they are doing, that your child is not wasting time in vain, so as not to discourage him from programming in the future.

UPD: As you correctly noticed in the comments, I said almost nothing about the submission of the material. I will say right away that I believe that there should be more practice, as much as possible. At the end of each lesson in the first part, I did 4-5 small practical tasks on the topic of the chapter. Between the chapters there were QnA (control lessons), where there were also practical, but already evaluated tasks, as well as at the end of the first part there was a project with a topic to choose from. In the second part, I made an introduction to OOP through the creation of a mini-game console, the development of which was the entire second part and the entire introduction to the paradigm.

Sitri = Vault + Pydantic: continuation of the saga, local development.

Alexander Lavrov — Tue, 15 Dec 2020 21:55:24 +0000

Background

In the previous article I wrote about how to configure your application using Sitri, however, I missed the point with local development, since you will agree that it is not very convenient deploying a Vault locally, and storing a local config in a common Vault, especially if several people are working on a project, is doubly inconvenient.

In Sitri this problem is solved quite simply - using the local mode for your settings classes, that is, you do not even have to rewrite anything or duplicate code, and the structure json file for local mode will almost completely repeat the structure of secrets.

So, well, now, let's add literally a couple of lines of code to our project + I'll show you how you can work with all this if your project is run locally in docker-compose...

Preparing the code

To begin with, let's agree that local_mode is true when ENV = "local" :)

Next, I propose to slightly edit our provider_config.py and create a BaseConfig class there from which we will inherit in our Config settings classes. We will do this in order not to duplicate the code, that is, the settings classes themselves will contain only what is specific to them.

import hvac
from sitri.providers.contrib.system import SystemConfigProvider
from sitri.providers.contrib.vault import VaultKVConfigProvider
from sitri.settings.contrib.vault import VaultKVSettings

configurator = SystemConfigProvider(prefix="superapp")
ENV = configurator.get("env")

is_local_mode = ENV == "local"
local_mode_file_path = configurator.get("local_mode_file_path")


def vault_client_factory() -> hvac.Client:
    client = hvac.Client(url=configurator.get("vault_api"))

    client.auth_approle(
        role_id=configurator.get("role_id"),
        secret_id=configurator.get("secret_id"),
    )

    return client


provider = VaultKVConfigProvider(
    vault_connector=vault_client_factory,
    mount_point=f"{configurator.get('app_name')}/{ENV}",
)


class BaseConfig(VaultKVSettings.VaultKVSettingsConfig):
    provider = provider
    local_mode = is_local_mode
    local_provider_args = {"json_path": local_mode_file_path}

A bit about local_provider_args in this field we specify the arguments for creating an instance of JsonConfigProvider, they will be validated and this dictionary must match the schema, so don't worry - this is not some dirty trick. However, if you want to create an instance of the local provider yourself, then you just put it in the optional local_provider field.

Now, we can easily inherit the config classes from the base one. For example, a settings class for connecting to Kafka would look like this:

from typing import Any, Dict

from pydantic import Field
from sitri.settings.contrib.vault import VaultKVSettings

from superapp.config.provider_config import BaseConfig, configurator


class KafkaSettings(VaultKVSettings):
    mechanism: str = Field(..., vault_secret_key="auth_mechanism")
    brokers: str = Field(...)
    auth_data: Dict[str, Any] = Field(...)

    class Config(BaseConfig):
        default_secret_path = "kafka"
        default_mount_point = f"{configurator.get('app_name')}/common"

        local_mode_path_prefix = "kafka"

As you can see, the required changes are minimal. local_mode_path_prefix we specify that the structure of the general config is saved in our json file. Now, let's write this json-file for the local configuration:

{
    "db":
    {
        "host": "testhost",
        "password": "testpassword",
        "port": 1234,
        "user": "testuser"
    },
    "faust":
    {
        "agents":
        {
            "X":
            {
                "concurrency": 2,
                "partitions": 5
            }
        },
        "app_name": "superapp-workers",
        "default_concurrency": 5,
        "default_partitions_count": 10
    },
    "kafka":
    {
        "auth_data":
        {
            "password": "testpassword",
            "username": "testuser"
        },
        "brokers": "kafka://test",
        "mechanism": "SASL_PLAINTEXT"
    }
}

... Well, or just copy and paste it from the end of the last article. As you can see, everything is very simple here. For our further research, rename main.py in the root of the project to __main__.py so that you can run the package with a command from docker-compose.

Put the application into the container and enjoy the build

The first thing we should do is write a small Dockerfile:

FROM python:3.8.3-buster

ENV PYTHONUNBUFFERED=1 \
    POETRY_VIRTUALENVS_CREATE=false \
    POETRY_VIRTUALENVS_IN_PROJECT=false \
    POETRY_NO_INTERACTION=1


RUN pip install poetry

RUN mkdir /superapp/
WORKDIR /superapp/

COPY ./pyproject.toml ./poetry.lock /superapp/
RUN poetry install --no-ansi

WORKDIR /

Here we just install the dependencies and that's it, since it is for local development, we do not copy the project code.

Next, we need an env file with the variables required for local-mode:

SUPERAPP_ENV=local
SUPERAPP_LOCAL_MODE_FILE_PATH=/config.json
SUPERAPP_APP_NAME=superapp

As you can see, nothing superfluous, no configuration information is needed for Vault, since in local mode, the application will not even try to "knock" on Vault.

And the last thing we need to write is the docker-compose.yml file itself:

# docker-compose config for local development
version: '3'
services:
  superapp:
    command: python3 -m superapp
    restart: always
    build:
      context: ./
      dockerfile: Dockerfile
    volumes:
      - ./superapp:/superapp
      - ./config.json:/config.json
    env_file:
      - .env.local

Everything is simple here too. We put our json file in the root, as write above in the environment variable for the container.

Now, launch:

docker-compose up

Creating article_sitri_vault_pydantic_superapp_1 ... done
Attaching to article_sitri_vault_pydantic_superapp_1

superapp_1  | db=DBSettings(user='testuser', password='testpassword', host='testhost', port=1234) faust=FaustSettings(app_name='superapp-workers', default_partitions_count=10, default_concurrency=5, agents={'X': AgentConfig(partitions=5, concurrency=2)}) kafka=KafkaSettings(mechanism='SASL_PLAINTEXT', brokers='kafka://test', auth_data={'password': 'testpassword', 'username': 'testuser'})

superapp_1  | {'db': {'user': 'testuser', 'password': 'testpassword', 'host': 'testhost', 'port': 1234}, 'faust': {'app_name': 'superapp-workers', 'default_partitions_count': 10, 'default_concurrency': 5, 'agents': {'X': {'partitions': 5, 'concurrency': 2}}}, 'kafka': {'mechanism': 'SASL_PLAINTEXT', 'brokers': 'kafka://test', 'auth_data': {'password': 'testpassword', 'username': 'testuser'}}}

As you can see, everything started successfully and the information from our json file successfully passed all checks and became settings for the local version of the application, yuhhu!

Code of this "continuation" I put in a separate branch of the repository, so you can take a look at how it all looks after the changes: branch

Pushing Prometheus metrics with pushgateway

Alexander Lavrov — Fri, 11 Dec 2020 23:13:34 +0000

Preface

This post is generally about pushing metrics in pushgateway, however, I will warn you and admit right away that the text will contain an example - an anti-pattern of pushing metrics, since using pushgateway is recommended in the case when the service does not work constantly (or the service/task being started does not have any interface at all), and therefore it is better for prometheus not to constantly knock on closed doors and not do unnecessary work.

Introduction

So, pushgateway is a service where you can push metrics when the standard prometheus pull-model is not applicable (in the preface, I generally described how such a situation can arise). After the metrics have entered the pushgateway, prometheus already picks them up from there, and this implies several restrictions related to pushing metrics, for example, the absence of the up metric, since it is generated by prometheus itself for the polled app instance, and in this case, it is only the pushgateway.

p.s. Although, if we talk about the up metric, then it is not needed if you use pushgateway in a recomended way.

Bake Prometheus for poll pushgateway

Let's say we have a compose like this with prometheus and pushgateway:

# ....(тут какие-нибудь графаны и т.д.)   
prometheus:  
    restart: always  
    image: bitnami/prometheus:latest  
    links:  
        - pushgateway  
    volumes:  
        - ./.prom.yml:/opt/bitnami/prometheus/conf/prometheus.yml  

pushgateway:  
    restart: always  
    image: bitnami/pushgateway:latest  
    ports:  
        - 9091:9091

In this case, prom.yml should look something like this to collect data from pushgateway:

global: null
scrape_interval: 5s
scrape_timeout: 2s
evaluation_interval: 15s

scrape_configs:
  - job_name: pushgateway
    honor_labels: true
    static_configs:
      - targets:
          - 'pushgateway:9091'

Everything is quite clear here, they added only honor_lables, which, if briefly resolves conflicts of label names, that is, for example, if your service has a metric with the "X" label and pushgateway has a label "X", then with honor_lables = false you will have label "X" with pushgateway and "exported_X" from your service that pushed metrics to pushgateway, and if true, only your service labels will be displayed (again, if there is a conflict).

p.s. Pushgateway security - docs recommends by default, for example, to use basic_auth.

Push metrics

I could give a beautiful example that corresponds to normal practice, however, I thought and decided that they hadn't given me any disrespect for a long time, therefore there will be an example of pushing metrics due to the fact that the service_discovery setting is absent (in production, of course, it is impossible).

So, let's say we have workers Faust there are a lot of them and they are not in the cluster (there is no swarm, no k8s), there are also no consul and other ways in which prometheus is able, they just run quietly in docker-compose and multiply with the scale argument.

More horror and sedition, in addition to pushing metrics, can be done by raging ports, for example, like this:

ports:  
- "9100-9200:6066"

And write targets with the whole set of ports into the prometheus config.

Let's continue through the code. For metrics from workers, you can use an already ready library. All we will do in this case is write a small timer in order to send metrics using the standard push_to_gateway function.

async def push_metrics():  
    def auth_handler(url, method, timeout, headers, data):  
        return basic_auth_handler(url, method, timeout, headers, data, PUSHGATEWAY_USERNAME, PUSHGATEWAY_PASSWORD)  
    push_to_gateway(PUSHGATEWAY_URI, job=f"{WORKERS_APP_NAME}-{ENV}", registry=registry_metrics, handler=auth_handler)  

@app.timer(interval=PUSH_METRICS_INTERVAL)  
async def push_metrics_cron():  
    await push_metrics()

As you can see, everything is quite simple here - we specify the job name (with a pool of metrics - this is done in the prometheus config), substitute the handler for authentication and specify the registry from which the metrics will be pushed. Well, that's all, we launch and when we open the pushgateway of the web-panel, we see that the metrics have been loaded at an interval, then the previously configured prometheus will take them from there.

Afterword

I decided to write a note, since I encountered a similar one in my work, I will say right away that the method from the example will not go to production, however, as using pushgateway in the absence of a service discovery, for testing it can come off.

Sitri = Vault+Pydantic: configuration with clear structure and validation

Alexander Lavrov — Thu, 10 Dec 2020 18:04:11 +0000

Introduction

In this article, I will talk about the configuration for your services using the Vault (KV and so far only the first version, i.e. without versioning secrets) and Pydantic (Settings) under the patronage of Sitri.

So, let's say that we have a superapp service with configs set up in Vault and authentication using AppRole, we'll set it up like this (I'll leave the policies setting for access to secret engines and the secrets themselves behind the scenes, since this is quite simple and the article is not about this):

Key                        Value
---                        -----
bind_secret_id             true
local_secret_ids           false
policies                   [superapp_service]
secret_id_bound_cidrs      <nil>
secret_id_num_uses         0
secret_id_ttl              0s
token_bound_cidrs          []
token_explicit_max_ttl     0s
token_max_ttl              30m
token_no_default_policy    false
token_num_uses             50
token_period               0s
token_policies             [superapp_service]
token_ttl                  20m
token_type                 default

Note: naturally, if you have the opportunity and the application goes into production mode, then secret_id_ttl is better to do not infinite, setting 0 seconds.

Superapp requires configuration of database connection, the connection to kafka and faust configuration for cluster of workers.

At the end, we should have this structure of our test-project:

super_app
├── config
│ ├── __init__.py
│ ├── provider_config.py
│ ├── faust_settings.py
│ ├── app_settings.py
│ ├── kafka_settings.py
│ └── database_settings.py
├── __init__.py
└── main.py

Bake Sitri provider

Basic library documentation has a simple example, configuration via the vault provider, however, it does not cover all the features and can be useful if your application is configured easily enough.

So, first of all, let's configure the vault provider in a file provider_config.py:

import hvac  

from sitri.providers.contrib.vault import VaultKVConfigProvider  
from sitri.providers.contrib.system import SystemConfigProvider  

configurator = SystemConfigProvider(prefix="superapp")  
ENV = configurator.get("env")  


def vault_client_factory() -> hvac.Client:  
    client = hvac.Client(url=configurator.get("vault_api"))  

    client.auth_approle(  
        role_id=configurator.get("role_id"),  
  secret_id=configurator.get("secret_id"),  
  )  

    return client  


provider = VaultKVConfigProvider(  
    vault_connector=vault_client_factory, mount_point=f"{configurator.get('app_name')}/{ENV}"  
)

In this code, we get variables from the environment using the system provider to configure the connection to vault, i.e. the following variables must be exported initially:

export SUPERAPP_ENV=dev
export SUPERAPP_APP_NAME=superapp
export SUPERAPP_VAULT_API=https://your-vault-host.domain
export SUPERAPP_ROLE_ID=<YOUR_ROLE_ID>
export SUPERAPP_SECRET_ID=<YOUR_SECRET_ID>

The example assumes that the base mount_point to your secrets for a specific environment will contain the application name and the environment name, which is why we exported SUPERAPP_ENV. We will define the path to the secrets of individual parts of the application in the settings classes later, so we leave it empty in the secret_path vault provider argument.

Settings classes

DBSetting - database connection

To create the settings class, we must use VaultKVSettings as the base class.

File database_settings.py:

from pydantic import Field  

from sitri.settings.contrib.vault import VaultKVSettings  

from superapp.config.provider_config import provider  


class DBSettings(VaultKVSettings):  
    user: str = Field(..., vault_secret_key="username")  
    password: str = Field(...)  
    host: str = Field(...)  
    port: int = Field(...)  

    class Config:  
        provider = provider  
        default_secret_path = "db"

As you can see, config data for the database connection is quite simple. This class will by default look at the secret superapp/dev/db, as we specified in the Config class. At first glance these are simple pydantic.Field, but they all have an extra argument vault_secret_key - it is needed when the key in the secret does not match the name (alias) of the pydantic field in our class, if vault_secret_key is not specified, the provider will search for the key by the field alias.

For example, in our superapp, it is assumed that the superapp/dev/db secret has the "password" and "username" keys, but we want the latter to be placed in the "user" field for convenience and brevity.

Let's put the following data in the above secret:

{
  "host": "testhost",
  "password": "testpassword",
  "port": "1234",
  "username": "testuser"
}

Now, if we run this code, we will get our secret from vault with our DBSettings class:

from superapp.config.database_settings import DBSettings

db_settings = DBSettings()
pprint(db_settings.dict())
# -> 
# {
#     "host": "testhost",
#     "password": "testpassword",
#     "port": 1234,
#     "user": "testuser"
# }

KafkaSettings - connection to brokers

File kafka_settings.py:

from typing import Dict, Any  

from pydantic import Field  

from sitri.settings.contrib.vault import VaultKVSettings  

from superapp.config.provider_config import provider, configurator  


class KafkaSettings(VaultKVSettings):  
    auth_mechanism: str = Field(...)  
    brokers: str = Field(...)  
    auth_data: Dict[str, Any] = Field(...)  

    class Config:  
        provider = provider  
        default_secret_path = "kafka"  
        default_mount_point = f"{configurator.get('app_name')}/common"

In this case, let's imagine that there is one kafka instance for different environments of our service, so the secret is stored along the path superapp/common/kafka:

Note: You can also set secret_path or/and mount_point at the field level so that the provider requests specific values from different secrets (if required). Here is a quote with prioritization of the secret path and mount point from the documentation:

Secret path prioritization:

vault_secret_path (Field arg)

default_secret_path (Config class field)

secret_path (provider initialization optional arg)

Mount point prioritization:

vault_mount_point (Field arg)

default_mount_point (Config class field)

mount_point (provider initialization optional arg)

{
  "auth_data": "{\"password\": \"testpassword\", \"username\": \"testuser\"}",
  "auth_mechanism": "SASL_PLAINTEXT",
  "brokers": "kafka://test"
}

{
    "auth_data":
    {
        "password": "testpassword",
        "username": "testuser"
    },
    "brokers": "kafka://test",
    "auth_mechanism": "SASL_PLAINTEXT"
}

Note: VaultKVSettings can understand both json and Dict itself.

As a result, this class of settings will be able to collect data from the secret like this:

{
    "auth_data":
    {
        "password": "testpassword",
        "username": "testuser"
    },
    "brokers": "kafka://test",
    "auth_mechanism": "SASL_PLAINTEXT"
}

FaustSettings - global configure faust and individual agents

File faust_settings.py:

from typing import Dict  

from pydantic import Field, BaseModel  

from sitri.settings.contrib.vault import VaultKVSettings  

from superapp.config.provider_config import provider  


class AgentConfig(BaseModel):  
    partitions: int = Field(...)  
    concurrency: int = Field(...)  


class FaustSettings(VaultKVSettings):  
    app_name: str = Field(...)  
    default_partitions_count: int = Field(..., vault_secret_key="partitions_count")  
    default_concurrency: int = Field(..., vault_secret_key="agent_concurrency")  
    agents: Dict[str, AgentConfig] = Field(default=None, vault_secret_key="agents_specification")  

    class Config:  
        provider = provider  
        default_secret_path = "faust"

Secret superapp/dev/faust:

{
  "agent_concurrency": "5",
  "app_name": "superapp-workers",
  "partitions_count": "10"
}

If our secret is written as indicated above, then individual agents will take information about the number of partitions in their topics and concurrency from the default fields.

{
  "agents": None,
  "app_name": "superapp-workers",
  "default_concurrency": 5,
  "default_partitions_count": 10
}

However, with the help of the AgentConfig model, we can set individual values for a specific agent. For example, if we have agent X with 5 partitions and concurrency 2, then we can change our secret so that information about this agent is in the "agents" field.

Secret superapp/dev/faust:

{
  "agent_concurrency": "5",
  "agents_specification": {
    "X": {
      "concurrency": "2",
      "partitions": "5"
    }
  },
  "app_name": "superapp-workers",
  "partitions_count": "10"
}

If we initialize our settings now, we will receive, in addition to the default fields, also the configuration for a specific agent:

{
    "agents":
    {
        "X":
        {
            "concurrency": 2,
            "partitions": 5
        }
    },
    "app_name": "superapp-workers",
    "default_concurrency": 5,
    "default_partitions_count": 10
}

Combine the settings classes into a single configuration model

To make our configs even more convenient to use, let's combine all our settings classes into one model by applying the default_factory:

File app_settings.py:

from pydantic import BaseModel, Field  

from superapp.config.database_settings import DBSettings  
from superapp.config.faust_settings import FaustSettings  
from superapp.config.kafka_settings import KafkaSettings  


class AppSettings(BaseModel):  
    db: DBSettings = Field(default_factory=DBSettings)  
    faust: FaustSettings = Field(default_factory=FaustSettings)  
    kafka: KafkaSettings = Field(default_factory=KafkaSettings)

Now, let's go to the main.py file and test the collection of complete configuration data for our application:

from superapp.config import AppSettings  

config = AppSettings()  

print(config)  
print(config.dict())

We get the output of application configuration:

db=DBSettings(user='testuser', password='testpassword', host='testhost', port=1234) 
faust=FaustSettings(app_name='superapp-workers', default_partitions_count=10, default_concurrency=5, agents={'X': AgentConfig(partitions=5, concurrency=2)}) 
kafka=KafkaSettings(auth_mechanism='SASL_PLAINTEXT', brokers='kafka://test', auth_data={'password': 'testpassword', 'username': 'testuser'})

{
    "db":
    {
        "host": "testhost",
        "password": "testpassword",
        "port": 1234,
        "user": "testuser"
    },
    "faust":
    {
        "agents":
        {
            "X":
            {
                "concurrency": 2,
                "partitions": 5
            }
        },
        "app_name": "superapp-workers",
        "default_concurrency": 5,
        "default_partitions_count": 10
    },
    "kafka":
    {
        "auth_data":
        {
            "password": "testpassword",
            "username": "testuser"
        },
        "brokers": "kafka://test",
        "auth_mechanism": "SASL_PLAINTEXT"
    }
}

Happiness, joy, delight!

Afterword

As you can see, the configuring is quite simple with Sitri, after it we get a clear configuration scheme with the required data types for the values, even if they were stored in strings in the vault by default.

Write comments about lib, code or general impressions. I will be glad to any feedback!

P.S. I have uploaded the code from the article to github