Forem: Manuel Kanetscheider

A Python Implementation of the Unit of Work and Repository Design Pattern using SQLModel

Manuel Kanetscheider — Mon, 22 May 2023 22:54:54 +0000

In this blogpost I would like to introduce two design patterns:

Repository pattern
Unit of Work pattern

So let's get started!

Repository pattern

The Repository pattern is a Domain-Driven Design pattern intended to keep persistence concerns outside of the system's domain model.

This means that the repository pattern represents our data layer and is a strict separation between domain and data model. A concrete repository gives access to an entity, an entity consists of one or more related tables:

Source: Design the infrastructure persistence layer - Define one repository per aggregate

A repository encapsulates all CRUD operations, in addition the repository also acts as an interface between domain and data model. This means that the domain and data model are only loosely coupled via well-defined interfaces.

This approach offers several advantages:

❇️ Separation of domain and data model.
❇️ Enabling of DDD and TDD. For each repository you can easily create so called fake repositories, which in turn can be used to create unit tests.
❇️ Centralization of queries, all query functions are defined and maintained in one place.

However, this pattern offers not only advantages:

🔴 Introduces another layer and increases the complexity.
🔴 Leads, in dependence of in the implementation, to redudant code because eventually domain model and data model must be maintained twice and additionally a mapping logic between domain and data model and vice versa is required.

Unit of Work pattern

This is a pattern that is commonly used with the repository pattern. To understand this pattern, we must first define the term "unit of work":

A unit of work refers to a single transaction that involves multiple insert, update, or delete operations.

In other words, all database operations should be handled in a single database transaction. This means that the repositories do not commit their respective transactions themselves, the database session of the repositories is managed by the unit of work instance, therefore the unit of work is also responsible for committing or rolling back the transaction.

This is especially important when multiple repositories are involved: the unit of work instance ensures that either all or none of the database operations are committed, keeping our data consistent.

Implementation

A special challenge in the implementation is that the domain model must not be dependent on the data layer, otherwise the loose coupling is no longer given. For Java and C# there are various tutorials on how to implement this pattern, something that almost all these implementations have in common is that the domain and data model are developed separately, so there are separate domain model and separate data classes.

Thus, the repository pattern is also responsible for the mapping between domain and data model. A consequence is that there are inevitably certain redudances and an additional mapping logic. In Python you would have basically the same problem but that is exactly where SQLModel comes to the rescue.

fastapi / sqlmodel

SQL databases in Python, designed for simplicity, compatibility, and robustness.

SQLModel, SQL databases in Python, designed for simplicity, compatibility, and robustness.

Documentation: https://sqlmodel.tiangolo.com

Source Code: https://github.com/fastapi/sqlmodel

SQLModel is a library for interacting with SQL databases from Python code, with Python objects. It is designed to be intuitive, easy to use, highly compatible, and robust.

SQLModel is based on Python type annotations, and powered by Pydantic and SQLAlchemy.

The key features are:

Intuitive to write: Great editor support. Completion everywhere. Less time debugging. Designed to be easy to use and learn. Less time reading docs.
Easy to use: It has sensible defaults and does a lot of work underneath to simplify the code you write.
Compatible: It is designed to be compatible with FastAPI, Pydantic, and SQLAlchemy.
Extensible: You have all the power of SQLAlchemy and Pydantic underneath.
Short: Minimize code duplication. A single type annotation does a lot of work. No…

View on GitHub

So what can SQLModel do for us? SQLModel is from the same developer who developed FastAPI, so this library works very well with FastAPI, but can also be used for other projects.

In FastAPI domain models are developed with Pydantic, Pydantic is responsible for data validation and serialization or deseralization of the data. FastAPI has no built-in ORM framework, a popular ORM framework is SQLAlchemy. What SQLModel does now is the combination of Pydantic and SQLAlchemy, so the models become domain and data model at the same time.

Thanks to this particular feature, a major pain point can be eliminated, namely the need to maintain domain and data models, as well as the maintenance and development of a mapping logic between domain and data model.

Overview

First, let's get a brief overview of the software we are going to create. For the tables I will follow the tutorials of SQLModel a little bit. Let's take a look at the class diagram first:

This probably looks a bit overwhelming now, but stick with me, let's go through it step by step:

GenericRepository: This is the abstract base class of a repository, in which all general repository functions are defined.
GenericSqlRepository: Inherits from GenericRepository. This class wraps SQLModel and as the name suggests acts as a generic SQL repository. All concrete repositories will be inherited from this class.
HeroReposityBase and TeamRepositoryBase: These classes are abstract classes that inherit from GenericRepository, where specific functionalities of the respective entities (hero and team) are defined. There are two approaches to the repository pattern: the generic and the "one repository per entity" approach. I personally prefer the second one, because you have the possibility to define specific functionalities, but this is a matter of preference or depends on the requirements.
HeroRepository and TeamRepository: These repositories are concrete repositories, they inherit from GenericSqlRepository and from the respective entity base repositories (HeroReposityBase or TeamRepositoryBase), in which the entity specific functions are defined. The entity specific functions are implemented in these repositories.
UnitOfWorkBase: Abstract context manager which defines the basic functions of the Unit of Work interface.
UnitOfWork: Concrete Unit of Work, inherits from the base class and wraps and manages the database session.

Generally speaking, all of the above classes can be divided into two categories: abstract and concrete classes. For example, if you want to create fake classes for unit tests, you would have to create fake classes that inherit from all the "base" classes, i.e. from the following:

GenericRepository
HeroReposityBase
TeamRepositoryBase
UnitOfWorkBase

All other classes are concrete classes and cannot be used for unit tests, unless you create integration tests in which you use e.g. an in-memory SQLite database.

In this blogpost I will not cover the creation of unit tests, but as I said before only the abstract classes need to be derived. This concept is especially strong when working with a dependency injection system, so in unit tests the dependencies can be easily injected using DI, creating a loose coupling between domain and data layer.

Let's write some code!

Defining the models

As you can see here I have defined my own base model, the settings chosen here are similar to what you would use in a FastAPI project. In the config you basically only set that the models can be imported and exported in a JSON compliant style.

The concrete tables inherit from the base model. Contrary to the official documentation, I defined the columns via the property sa_column, passing an SQLALchemy column object as a parameter. This allows a more precise column definition, so the column name and the exact data type can be specified accurately. I personally prefer this approach, but of course you can also define the tables as described in the official SQLModel documentation.

ORM

Engine and session factory

This file contains the code needed to connect to a database, i.e. the creation of a SQLModel engine and session maker function. The latter returns a function which in turn returns a new database session whenever it is called.

Repositories

Generic Repository

As before, this is the base repository class which defines all basic functions (= CRUD).

SQL Repository

This repository is a generic SQL repository. The constructor takes as arguments a database session and a SQLModel type.

Here all methods are implemented and two new methods for the creation of the GET and LIST SQL statements. By introducing these two methods you could customize the SQL statements for GET and LIST in the derived repositories.

Base Entity Repositories

These repositories are the abstract base classes of the respective repositories. These classes are completely optional, but offer the advantage that entity-specific functions can be defined here, which should not be available for all repositories.

Entity repositories

These are now concrete repositories, here one could overload the functions of the general SQL repository, in addition the abstract methods of the respective base repositories are implemented here.

Service Layer

The Unit of Work functionalities are not directly part of the ORM framework, but is a service that connects everything together.

Unit of Work

The service unit of work is responsible for the handling of the session and thus guarantees that everything runs as a single unit of work.

This service is a context manager that calls the session_maker aka session factory function each time in the enter method and thus creates a new session each time. This session is passed on to the respective repositories, thereby ensuring that all operations run within one transaction.

The rollback functions are always called at the end of the context manager. If the commit function was called before, the rollback function has no effect. If the commit function was not called before or an error occurred, the entire transaction is rolled back.

Let's test it!

Now that we have written all the necessary code, it is time to make some database calls:

Even if it requires a lot of boiler code, especially for larger projects these design patterns really pay off, because they achieve a strict separation of domain and data layer, and the layers are only loosely coupled with each other.

As already mentioned, you could now derive the base classes and test the business logic without database operations. Alternatively, you can also run the unit tests with a SQLite in-memory database.

Conclusion

If you've made it so far, thanks for reading! I hope I could bring you the two design patterns a bit newer and encourage you to use these patterns for your next project. Even though there is a lot of code involved in setting up this structure, I think these patterns will pay off especially for larger projects.

All code is available at this repository, check it out!

manukanne / sqlmodel-repository-pattern

SQLModel Repository pattern

This repository includes a example implementation of the Unit of Work design and Repository design pattern.

For more details please check out my blog post!

---
title: Unit of Work + Repository pattern
---
classDiagram
     direction LR
    GenericRepository <|--GenericSqlRepository
    GenericRepository <|-- HeroReposityBase
    GenericRepository <|-- TeamRepositoryBase

    GenericSqlRepository <|-- TeamRepository
    TeamRepositoryBase <|-- TeamRepository
    GenericSqlRepository <|-- HeroRepository
    HeroReposityBase <|-- HeroRepository

    UnitOfWorkBase <|-- UnitOfWork
    HeroReposityBase *-- UnitOfWorkBase
    TeamRepositoryBase *-- UnitOfWorkBase

    HeroRepository *--UnitOfWork
    TeamRepository *--UnitOfWork

    class GenericRepository~SQLModel~{
        +get_by_id(int id) SQLModel
        +list(**filters) List~SQLModel~
        +add(T record) SQLModel
        +update(T recored) SQLModel
        +delete(id int)
    }

    class GenericSqlRepository~SQLModel~{
        -_construct_list_stmt(id)
        -_construct_list_stmt(**filters)
    }
    class HeroReposityBase{
        +get_by_name(str name) Hero
    }
    class HeroRepository{

    }
    class TeamRepositoryBase{
        +get_by_name(str name) Team
    }
    class TeamRepository{
    }
    class UnitOfWorkBase{
        teams: TeamRepositoryBase
        heroes: HeroReposityBase
        + __enter__()
        + __exit__()
        + commit()
        + rollback()
    }
    class UnitOfWork{
        teams: TeamRepository
        heroes: HeroRepository
    }

Getting Started

Tech-stack

Python 3.10
SQLModel
- Pydantic
- SQLAlchemy

Setup

1. Clone the project:

git

…

View on GitHub

Further references

Acknowledgments

Azure Functions for Python - Introduction to the new V2 programming model

Manuel Kanetscheider — Wed, 21 Dec 2022 22:34:29 +0000

Introduction

Recently, Microsoft introduced a decorator based approach for the development of Python based functions. This concept is very similar to the approach that FastAPI follows. So what has changed?

⚠️ The V2 programming model is currently in public preview. Some things may still change, so the features described here should not be used in production yet. Also, not all trigger types are fully supported at this time.

The V1 programming model

With the v1 programming model, each trigger of a function consists of two files:

function.json
- This is a configuration file that defines, among other things, the trigger type of the function and how the data is bound.
__init__.py
- Contains the actual code of the function and must match the function.json (such as names of input or output variables defined in function.json).

For example, an HTTP Azure Function looks like this:

function.json:

{
    "scriptFile": "__init__.py",
    "bindings": [
        {
            "authLevel": "function",
            "type": "httpTrigger",
            "direction": "in",
            "name": "req",
            "methods": [
                "get",
                "post"
            ]
        },
        {
            "type": "http",
            "direction": "out",
            "name": "$return"
        }
    ]
}

__init__.py:

import azure.functions


def main(req: azure.functions.HttpRequest) -> str:
    user = req.params.get('user')
    return f'Hello, {user}!'

The V2 programming model

As you could see in the V1 programming model, for each trigger two files are needed, one is a configuration of the trigger and the other is the actual code. This has changed with the V2 programming model, the configuration information can now be placed directly in the code via decoraters.

For example, an HTTP Azure Function looks like this:

import azure.functions

@app.function_name(name="HttpTrigger1")
@app.route(route="req")

def main(req: azure.functions.HttpRequest) -> str:
    user = req.params.get('user')
    return f'Hello, {user}!'

The triggers can now be configured via decorates. The types of the attributes, both input and output, can be specified via type annotations.

At the moment the following triggers and bingings are supported:

Type	Trigger	Input	Output
HTTP	✅
Timer	✅
Azure Queue Storage	✅		✅
Azure Service Bus topic	✅		✅
Azure Servuce Bus queue	✅		✅
Azure Comos DB	✅	✅	✅
Azure Blob Storage	✅	✅	✅
Azure Hub	✅		✅

ℹ️ As you can see here, not all triggers and binding are supported yet, but Microsoft will deliver them later. If your desired type is not listed yet, you have to stay with the programming model V1.
For an up-to-date overview of all supported types, please check out this link.

As you could already see the new programming model V2 leads to a simpler file structure and a more code based approach. Another cool feature introduced are the blueprints.

In case you are familiar with FastAPI, blueprints may remind you of FastAPI router functionality. Blueprint allows to split the Azure Function into several modules and also to create interfaces that can be reused in other projects.

import logging 

import azure.functions as func 

bp = func.Blueprint() 

@bp.route(route="default_template") 
def default_template(req: func.HttpRequest) -> func.HttpResponse: 
    logging.info('Python HTTP trigger function processed a request.') 

    name = req.params.get('name') 
    if not name: 
        try: 
            req_body = req.get_json() 
        except ValueError: 
            pass 
        else: 
            name = req_body.get('name') 

    if name: 
        return func.HttpResponse( 
            f"Hello, {name}. This HTTP-triggered function " 
            f"executed successfully.") 
    else: 
        return func.HttpResponse( 
            "This HTTP-triggered function executed successfully. " 
            "Pass a name in the query string or in the request body for a" 
            " personalized response.", 
            status_code=200 
        )

This blueprint then only needs to be imported and registered:

import azure.functions as func 
from http_blueprint import bp

app = func.FunctionApp() 

app.register_functions(bp)

A unique feature of Azure functions in combination with Python is the use of ASGI and WSGI compatible web frameworks, such as FastAPI or Flask. This is already possible with the programming model V1 and is also maintained here:

ASGI (FastAPI)

# function_app.py

import azure.functions as func 
from fastapi import FastAPI, Request, Response 

fast_app = FastAPI() 

@fast_app.get("/return_http_no_body") 
async def return_http_no_body(): 
    return Response(content='', media_type="text/plain") 

app = func.AsgiFunctionApp(app=fast_app, 
                           http_auth_level=func.AuthLevel.ANONYMOUS)

WSGI (Flask)

# function_app.py

import azure.functions as func 
from flask import Flask, request, Response, redirect, url_for 

flask_app = Flask(__name__) 
logger = logging.getLogger("my-function") 

@flask_app.get("/return_http") 
def return_http(): 
    return Response('<h1>Hello World™</h1>', mimetype='text/html') 

app = func.WsgiFunctionApp(app=flask_app.wsgi_app, 
                           http_auth_level=func.AuthLevel.ANONYMOUS)

Conclusion

These were my highlights of the programming model V2 for Python Azure Fuctions! Although this feature is still in preview, I find these changes incredibly exciting as they allow us to develop Azure Functions with a more code based approach, similar to FastAPI.

However, keep in mind that these are preview features at this time and may change. Also, not all triggers and binding types are
supported currently, but these will surely follow shortly!

Let me know your thoughts on this and thanks for reading!

Resources:

Data Structures: Arrays and Lists in Comparison

Manuel Kanetscheider — Tue, 20 Dec 2022 01:22:14 +0000

Usually, when you start your programming journey, the first data types you come up against are arrays and lists. Arrays are a fundamental data type, many more complex data types are built on top of arrays. Let's have a look at arrays:

var arr = new int[4];
arr[0] = 10;
arr[1] = 20;

We have just created an array with a length of 4. Arrays use the contiguous memory allocation technique for memory allocation. This means that all the fields must be continuous next to each other:

There were not four consecutive memory blocks free in the first row, so the required memory for the array was allocated in the second row.

So far so good. Now let's imagine we want to resize our array because we want to store e.g. 6 values. What we would have to do is to create a new array with a length of 6 and copy the values of the previous array into the new array:

var arr = new int []{ 10, 20, 30, 40 };
var biggerArr = new int[6];
arr.CopyTo(biggerArr, 0);

If we now think of lists, lists have a key characteristic: you don't have to specify a fixed length in advance, lists can dynamically allocate more memory. For lists, however, you can specify a capacity value; if no value is specified, a default value is used. This default value varies depending on the programming language. Lists also use arrays in the background, so the given capacity size represents the size of the initial array. For example:

var list = new List<int>(2);

for (var i = 10; i <= 90; i+=10)
{
   list.Add(i);
   Console.WriteLine($"Added {i} to list. Current list capacity is {list.Capacity}");
}

The output looks like this:

Added 10 to list. Current list capacity is 2
Added 20 to list. Current list capacity is 2
Added 30 to list. Current list capacity is 4
Added 40 to list. Current list capacity is 4
Added 50 to list. Current list capacity is 8
Added 60 to list. Current list capacity is 8
Added 70 to list. Current list capacity is 8
Added 80 to list. Current list capacity is 8
Added 90 to list. Current list capacity is 16

The initial capacity is 2. As soon as this was exceeded, the capacity doubled, i.e. it was 4. As soon as the capacity of 4 was exceeded, it doubled again, and so on.

When we resized our array in our previous example, we basically did something similar than the list. With one exception. If we remember, the occupied memory of an array must always be contiguous. But this is not true for lists, here the allocated memory is not contiguous and may looks like this:

The list consists of many small, not directly contiguous memory blocks. What is immediately noticeable is that this allows the list to make better use of the available memory. If, as in the chosen example, we need to allocate memory for an array with a length of 16, a memory block with an appropriate length must first be found.

But don't get me wrong, this doesn't mean that arrays are inefficient data structures, quite the opposite! Arrays are one of the most efficient data structures ever, especially when we access values via indexes.

I know this looks a little bit messy, but stay with me! The two diagrams show how our memory could be distributed. Our program or our process probably does not have one large continuous block, but many small blocks, in between other processes or programs also occupy memory. On the left side we see in blue our array and on the right side in purple our list.

If we now imagine we want to access any element of our array. Because arrays always allocate a continuous block of memory, we always know exactly where an array begins and ends. This makes it extremely easy to access elements using indexes.
This is different for lists. Elements of a list are also accessed via indexes, but since the elements are not on a continuous memory block, such accesses can be very costly. Of course, this depends primarily on the size of the list, but with large amounts of data, the array will always win.

Does this mean that arrays are better than lists? As always, it depends. If we want to increase our array size, this is an extremely expensive task. As shown before, we need to create a new array with the desired length and copy all the values of the original array.
Lists, on the other hand, simply allocate new memory dynamically. Even if elements are removed, the list can free memory dynamically. So for many add and delete operations, the list clearly has the upper hand. In addition, an array always occupies a fixed amount of memory, regardless of whether all fields are assigned values or not.

Summary:

	Array	List
Length	Arrays always have a fixed length. The length of an array cannot be changed, a new array must always be created with a copy of the previous values.	Lists can be resized dynamically.
Memory consumption	Arrays have a fixed length and the memory is contiguous. Generally speaking, arrays are more memory efficient than lists, unless the array is oversized.	Lists have a non-contiguous memory space, therefore the internal implementation of the list is more complex and needs additional structures that consume more memory.
Usage	Arrays are pointed if the number of elements is known. In addition, due to the contiguous memory, elements can be quickly accessed via indexes.	Because lists can resize themselves automatically, they are particularly suitable for frequent add and remove operations and can thus also achieve a better memory performance than arrays under certain circumstances.

Conclusion

Finally, I would like to emphasize that there is no clear winner here, it simply depends on the use-case. In most cases the list is the better choice, most of the time you don't know how big or small the amount of data is and lists can allocate the needed size dynamically.

I hope I could bring you a little bit closer to how arrays and lists work in the background. I tried to stay as general as possible without going into a specific programming language, in essence this should work similarly for all programming languages.

I hope you enjoyed my blogpost, thanks for reading! If you have any suggestions or comments, let me know!

Implement a Batch API using FastAPI

Manuel Kanetscheider — Sat, 19 Nov 2022 13:52:24 +0000

In this blogpost I would like to present you a possible implementation for a batch route using FastAPI. FastAPI is a great framework for creating REST API's, unfortunately there is currently no native support for batching.

Why batching?

Batching is the process of combining multiple API requests into a single request with a single response. Without batching, the whole thing looks like this:

Each request between client and server has a certain network latency, so the processing of several consecutive requests can take some time.

With a batch request, the roundtrip between client and server and thus the number of requests can be reduced, which in turn rapidly improves performance:

Technical consideration of the batch route

In the course of my research, I looked at the batch routes of other major API's, including:

Based on the API's listed above, there are two different ways to submit batch requests:

Submit batch request with MIME type multipart/mixed
- The advantage of this approach is that requests can be sent with different MIME types (e.g. application/json, application/http, application/octet-stream etc.).
- The body of the request is divided into parts that are separated from each other by a boundary string that is specified in the header of the request.
Submit batch request with MIME type application/json
- The batch request is submitted in JSON format. This limits the API to the MIME type application/json.

I personally prefer the JSON batch approach, assembling the batch request is slightly easier with this approach. Of course this approach is limited to the MIME type application/json only, but since most FastAPI routes implement exactly this MIME type this is acceptable for me. The choice therefore varies simply on the technical requirements.

Our batch route is therefore inspired by the Microsoft Graph API, the batch request will have the following structure:



POST /batch HTTP/1.1
Content-Type: application/json

{
  "requests": [
    {
      "id": "1",
      "url": "/products?skip=0&limit=1",
      "method": "GET"
    },
     {
      "id": "2",
      "url": "/products/1",
      "method": "GET"
    },
    {
        "id": "3",
        "url": "/products",
        "method": "POST",
        "headers": {
            "Content-Type": "application/json"
        },
        "body": {
            "title": "Test Product"
        }
    }
  ]
}

Property	Mandatory	Description
id	Yes	A correlation value to associate individual responses with requests. This value allows the server to process requests in the batch in the most efficient order.
url	Yes	The relative resource URL the individual request would typically be sent to.
method	Yes	The HTTP method (e.g. GET, POST etc.).
headers	No	A JSON object with the key/value pair for the headers.
body	No	JSON body.

The next step is to consider how the individual requests should be processed on the server side. Behind each FastAPI route there is a corresponding Python function that can be called directly. At first this sounds like a good idea, but then FastAPI is no longer able to determine the dependencies via depency injection, so we would have to do this manually.

Determining the dependencies manually could be difficult and probably often leads to unexpected behavior of the API route. So I did some research and came across this repository:

dtkav / flask-batch

Handle many API calls from a single HTTP request

Flask-Batch

Batch multiple requests at the http layer. Flask-Batch is inpsired by how google cloud storage does batching.

It adds a /batch route to your API which can execute batched HTTP requests against your API server side. The client wraps several requests in a single request using the multipart/mixed content type.

Installation

pip install Flask-Batch

# to include the dependencies for the batching client
pip install Flask-Batch[client]

Getting Started

Server

from flask import Flask
from flask_batch import add_batch_route

app = Flask(__name__)
add_batch_route(app)

# that's it!

Client

The client wraps a requests session.

from flask_batch.client import Batching
import json
alice_data = bob_data = {"example": "json"}

with Batching("http://localhost:5000/batch") as s:
    alice = s.patch("/people/alice/", json=alice_data)
    bob = s.patch("/people/bob/", json=bob_data)

alice         #

…

View on GitHub

Flask-Batch performs batching on the HTTP layer. This means that on the server side a HTTP client executes all batch requests. The network latency is much lower on the server side which implies that the requests can be processed faster on the server side. Additionally, since the requests are processed on the HTTP layer, FastAPI is also able to resolve the dependencies via dependicy injection. Therefore I also decided to do batching on HTTP level.

Let's get started

Defining the batch models

HTTPVerbs

This enum defines the allowed HTTP methods.

BatchRequest

This model represents a single API request. This model also ensures that the referenced endpoint meets our requirements, in this case that no absolute URL is given and that the route starts with a leading slash. Especially the check that it is not an absolute URL is extremely important to prevent potential improper usage of our batch route.

BatchResponse

This model represents the result of the corresponding request. Here also the ID of the request is returned so that the client can map request and response (theoretically the order of processing could vary). Besides the ID, the HTTP status code, the headers and the JSON body are also returned.

BatchIn

This is the container that includes all our batch requests. Here it is also checked if all ID's are unique and the allowed maximum amount of requests has not been exceeded.

BatchOut

This is the container that holds all the responses, this model is the one that is passed back to the caller.

Implement the batch route

The batch requests are processed using the Python package aiohttp. This allows the requests to be processed in an asynchronous manner. One blog post that helped me a lot with the implementation is the post Making 1 million requests with python-aiohttp which I would definitely like to recommend to you!

In the next step we implement the actual batch route, here we just have to plug everything together:

The repository, which contains all the code including a small test API, can be found here:

manukanne / fastapi-batch-requests-demo

FastAPI Batch Tutorial

This repository contains a possible implementation of a batch route using FastAPI For more details, please check out my blog post on dev.to

Description

Json based batching - The data is exchanged using the MIME type application/json, this approach is inspired by the Microsoft Graph Batch API.
Batching takes place on the HTTP layer. This approach was chosen to ensure consistent behaviour.

Why batching

Batching is the process of combining multiple API requests into a single request with a single response. Without batching, the whole thing looks like this:

sequenceDiagram
 autonumber
 Client ->> Server: API Request
 Server ->> Client: API Response
 Client ->> Server: API Request
 Server ->> Client: API Response
 Client ->> Server: API Request
 Server ->> Client: API Response

Each request between client and server has a certain network latency, so the processing of several consecutive requests can take some time.

With a…

View on GitHub

Conclusion

Bathing is an important component to combine many requests to a single API call and thus increase the performance rapidly.
I hope you enjoyed my blog, thanks for reading! If you have any recommendations on how to improve my implementation, please let me know in the comments section.

Infrastructure as Code - Deploy an Azure Function using Bicep

Manuel Kanetscheider — Mon, 25 Apr 2022 17:58:52 +0000

Introduction

In this guide, we will look at the topic Infrastructure as Code using the example of an Azure Function. But what is Infrastructure as Code? Based from the official Microsoft documentation, Infrastructure as Code can be described as follows:

Infrastructure as Code (IaC) is the management of infrastructure (networks, virtual machines, load balancers, and connection topology) in a descriptive model, using the same versioning as DevOps team uses for source code. Like the principle that the same source code generates the same binary, an IaC model generates the same environment every time it is applied. IaC is a key DevOps practice and is used in conjunction with continuous delivery.

A key property of IaC is the so-called Idempotence, which means that the deployment command always brings the target system into the desired state, regardless of the initial state. This can be achieved either by configuring the target system or by creating a fresh environment.

In simpler terms, IaC ensures that we always have the desired configuration of our target system. In classic on-prem scenarios, the target systems are unique, e.g. staging and production systems may differ. These differences can cause the application to work on the staging system but then fail on the production system (e.g. missing permissions, network access, etc.). IaC tackles this issues and helps to reduce such risks. Additionally, since the required infrastructure is now available as code, it can be easily versioned via Git.

For IaC various tools are around:

The built-in tools for Azure are ARM and Bicep Templates, the other tools listed are third party tools for which additional costs may apply.
In this guide we will cover only ARM and (mainly) Bicep. ARM and Bicep offer the advantage that they are developed directly by Microsoft and therefore always have access to the latest features. The only downside is that these technologies are not suitable for multicloud scenarios, as they are developed exclusively for the Microsoft Azure Cloud.
Also, another plus point for ARM and Bicep, both are free and fully supported by Microsoft 😉.

ARM vs. Bicep

ARM and Bicep both offer basically the same functionalities, but Bicep is a newer technology compared to ARM, so it can happen that some ARM features are not yet available in Bicep (for example like User-defined ARM Functions).

ARM templates are JSON files, all resources are represented as JSON. Bicep on the other hand is a declarative language, comparable to Terraform. Let's have a look:

Example ARM Template:



{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "location": {
      "type": "string",
      "defaultValue": "[resourceGroup().location]"
    },
    "storageAccountName": {
      "type": "string",
      "defaultValue": "[format('toylaunch{0}', uniqueString(resourceGroup().id))]"
    }
  },
  "resources": [
    {
      "type": "Microsoft.Storage/storageAccounts",
      "apiVersion": "2021-06-01",
      "name": "[parameters('storageAccountName')]",
      "location": "[parameters('location')]",
      "sku": {
        "name": "Standard_LRS"
      },
      "kind": "StorageV2",
      "properties": {
        "accessTier": "Hot"
      }
    }
  ]
}

Example Bicep Template:



param location string = resourceGroup().location
param storageAccountName string = 'toylaunch${uniqueString(resourceGroup().id)}'

resource storageAccount 'Microsoft.Storage/storageAccounts@2021-06-01' = {
  name: storageAccountName
  location: location
  sku: {
    name: 'Standard_LRS'
  }
  kind: 'StorageV2'
  properties: {
    accessTier: 'Hot'
  }
}

I think it's obvious that the Bicep template is significantly shorter and even if you've never worked with either technology, it's much easier to see what's going on inside the Bicep template than in the ARM template.

In my opinion, Bicep offers the following advantages over ARM templates:
👑 Less code
👑 Much easier to read
👑 Fully integrated into VS Code with awesome intellisense capabilities

Even if you need ARM templates, you can compile Bicep into ARM templates. This actually happens in the background when you deploy a Bicep template to Azure. During the deployment the Bicep template is compiled into an ARM template and then the ARM template is submitted to Azure.

It is also possible to decompile an ARM template and thus convert an ARM to a Bicep template.

Personally, I think if you don't have a strong reason to use ARM templates, you should definitely use Bicep, as the syntax is much simpler and more readable.

Let's get started

Perquisites

Azure Account with an active subscription. In case you do not have an Azure Account, go ahead and create one for free here
Azure CLI
Bicep Tools for Azure CLI and VS Code

ℹ️ It is absolutely possible to create the Bicep templates with the editor of your choice, however, I strongly recommend you to try the VS Code Extension for Bicep as it greatly simplifies the template development.

Azure Function Resources Overview

Before we actually getting started, let's take a look at what resources we typically need for an Azure Function:

Function App
This is the actual "Function App" service, where the code is running.

App Service Plan
The App Service Plan is comparable to a web server and describes how the Function App is hosted. The App Plan also specifies how the Function App scales, the resources available per instance, and which "advanced" features can be used, such as Azure Virtual Network connectivity. In addition, different prices apply depending on the choice of the hosting plan. The following plans are available:

Consumption Plan (= serverless)
Premium Plan
Dedicated hosting plan

For more information, please checkout the official documentation on Azure Functions hosting options.

Storage Account
When using the Consumption/Premium hosting plan, the function code and binding configuration files are stored in Azure Files in the main storage account. In addition, certain platform features may use the storage account for internal operations, such as Azure Durable Functions.

Fore more information, please checkout the official documentation on Azure Function Storage Considerations.

Application Insights (optional)
Application Insights is the preferred monitoring service of Azure Functions and offers built-in logging functionalities. Application Insights collects log, performance, and error data. By automatically detecting performance anomalies and featuring powerful analytics tools, you can more easily diagnose issues and better understand how your functions are used.

ℹ️ Application insights is an optional service that is billed separately. However, the cost are very low and it is highly recommended to use it with your Function App.

For more information, please checkout the official documentation on Monitoring Azure Functions and Azure Application Insights.

Key Vault (optional)
Azure Key Vault is a cloud service for securely storing and accessing secrets. A secret is anything to which access should be strictly controlled, such as API keys, passwords, certificates, or cryptographic keys.

Azure Key Vault Secrets can only be accessed via a secured and authenticated connection. In the context of the Azure Function, a system or user assigned identity must be created with which the Azure Function authenticates itself to the Key Vault service. The permissions must then be set up on the Key Vault in form of an access policy.

ℹ️ Azure Key Vault is an optional service and is primarily used for secure storage and retrieval of sensitive data. Storing sensitive data directly in the function is not recommended, as this data is stored in plain text, meaning that anyone who has access to the storage account can read this information. In addition to secrets, certificates and keys can also be stored in the Key Vault, and the Key Vault also offers various other functionality such as the encryption of data via public and private key. This service is also charged separately.

To learn more about Azure Key Vault or on how to Integrate Azure Key Vault into your Azure Function, please checkout the official documentation.

Templates

Now that we know what resources we need for our function, we can proceed with the development of the templates.

Storage Account

This template creates the Storage Account, for later use the connection string is constructed and exported. The connection string is needed later in order to connect the Function App to the storage account.

As briefly mentioned before, certain function features, such as Durable Functions, require certain Storage Services. If further storage services are required, they can be added to this template.

Application Insights

Creates the Application Insights Resource for monitoring the Function App. To connect the Application Insights instance to the Function App, the Instrumentation Key must be exported for later assignment.

App Service Plan

Deploys the App Service Plan of the Function, here the plan id is exported for later usage.

⚠️ The operating system varies depending on the language, please make sure to use the correct OS based on the chosen programming language, for more details please checkout the official documentation.

Function App (without settings)

Deploys the function app without settings, here also the previously exported plan id is passed into the template (the plan id specifies the selected hostion option of the Azure Function).
Also a Managed Identity is assigned, as discussed before this is needed for the authentication against the Key Vault. The identity information, such as principal id and tenant id, as well as the name of the Function App are exported for later use.

Key Vault

Creates the Key Vault. The previously configured managed identity is passed to the template and the necessary permissions for this managed identity are then set up using an access policy. In addition, any required secrets are created here and then exported for later referencing.

Function App Settings

Creates the Function App Settings. Here various settings of the Function App are set, such as the Storage Account, the Application Insights instance, Function App runtime and any other custom parameters including secrets.

Main

In order to create the function this template must be deployed. This template glues everything together and deploys all previously created templates in the correct order. The dependencies of the resources are also defined here, e.g. the function app should only be deployed if the deployment of the storage account, the application insights instance and the app service plan was successful.

Deployment

For creating the function, a resource group is required in advance. If the desired resource group is already present, this step can be skipped.



az group create -n <name> -l <location>

For the deployment various parameters have to be specified including:

Location
Storage Account name
Storage Account SKU
Application Insights name
App Service Plan name
Function App operating system
Function App name
Function App runtime

We have two options for specifying the parameters:
Inline parameters:



az deployment group create \
  --resource-group testgroup \
  --template-file <path-to-bicep> \
  --parameters exampleString='inline string' exampleArray='("value1", "value2")'

Parameter file:



az deployment group create \
  --name ExampleDeployment \
  --resource-group ExampleGroup \
  --template-file storage.bicep \
  --parameters @storage.parameters.json

Since we need have to specify a few parameters I suggest that we go with the parameter file variant. A parameter also offers the following advantages:

Is part of the repository and can therefore be versioned in Git
A separate configuration can be created for each environment (e.g. staging, production, etc.)
Facilitates deployment via CI/CD pipelines

The parameter looks like this, please just adjust the parameters as needed:

ℹ️ For Azure there are recommendations from Microsoft on how resources should be named, for more details please checkout Recommended abbreviations for Azure resource types. Of course, these are only recommendations, resources can be named in any way you like.

To deploy the Azure Function we can use the following command:



az deployment group create \
  --name <deployment-name> \
  --resource-group <resource-group-name> \
  --template-file bicep\main.bicep
  --parameters @<path-to-parameters>

⚠️ For some names there are certain limitations, e.g. special characters, character lengths, etc. For example, the name of a storage account must be between 3 and 24 characters, should not contain any special characters and the name must be globally unique. If an error occurs during deployment, please read the error message carefully and adjust the names if necessary.

That's it! Deployment takes a few seconds and then the Azure Function is ready to go.

Conclusion

Infrastructure as Code is an extremely powerful tool and an important building block that paves the way for further DevOps steps, such as the automated creation and configuration of environments via CI/CD mechanics.
On top of that, Microsoft offers with Bicep and VS Code two excellent tools with which the creation of templates is a breeze.

Even if we have only looked at the tip of the iceberg, I hope I could awaken your interest and that you liked my blog post.

Thanks for reading, if you have any questions feel free to leave a comment 😃

Here you can find the repository with the full source code:

manukanne / tutorial-az-func-iac

Deploy an Azure Function using Bicep

This project illustrates how an Azure Function can be deployed as IaC via Bicep.

Description

The project provides the following Bicep templates:

Function App (without settings)
App Service Plan
Storage Account
Application Insights
Key Vault
Function App Settings

The individual resources can be found under the bicep/templates folder, these are all linked and deployed from the main.bicep file.

For more detailed information, please checkout my blog post 😃

Getting Started

Perquisites

Azure Account with an active subscription. In case you do not have an Azure Account, go ahead and create one for free here
Azure CLI
Bicep Tools for Azure CLI and VS Code

How to deploy the templates

Create a parameter file

The example parameter file looks like this:

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#"
    "contentVersion": "1.0.0.0",
    "parameters": {
      "storageAccountName": {
        "value": "stvm<storage-account-name>"

…

View on GitHub

Serverless Thumbnail Generation with Azure Computer Vision

Manuel Kanetscheider — Sun, 17 Apr 2022 00:43:56 +0000

Introduction

In this guide, I would like to introduce a pre-build AI service called Azure Cognitive Service, in particular Azure Computer Vision with the Thumbnail Generation API.
As surely known a thumbnail is simply a smaller version of the original image, in web applications thumbnails are used in many places like for displaying product images etc. Because of the fact that thumbnails are smaller they need less bandwidth than the original images and speed up the loading of websites.

Azure Computer Vision is part of Azure Cognitive Services. But what are Azure Cognitive Services? At a glance, Azure Cognitive Services offers the following features:

Speech
- Speech to text
- Text to speech
- Speech translation
- Speaker recognition
Languages
- Entity recognition
- Sentiment analysis
- Question Answering
- Conversational language understanding
- Translator
Vision
- Computer Vision
- Custom Vision
- Face API
Decision
- Anomaly Detector
- Content Moderator
- Personalizer
OpenAI
- OpenAI Service

In a nutshell, Azure Cognitive Services offer various pre-build AI services, one of which is Azure Computer Vision. Azure Computer Vision offers much more functionality than just thumbnail generation, but we'll leave that out for now (this maybe a topic for another blog post 😀).

For more details on Azure Cognitive Services, please checkout the
official documentation

Why should we use Azure Computer Vision for Thumbnail Generation?

Creating thumbnails can sometimes be quite challenging and there are various tutorials for countless programming languages that cover this topic. Azure Computer Vision does the following things for us when creating the thumbnail:

Remove distracting elements from the image and identify the area of interest—the area of the image in which the main object(s) appears.
Crop the image based on the identified area of interest.
Change the aspect ratio to fit the target thumbnail dimensions.

Especially the Smart Cropping, i.e. the detection and removal of distracting elements, is an extremely useful feature. Imagine you implement your own thumbnail creation algorithm and the center is always cropped out, but the area of interest is outside the center. So the algorithm would crop something that is actually relevant and that's where Smart Cropping comes to the rescue!

Prerequisites

Azure Account with an active subscription. In case you do not have an Azure Account, go ahead and create one for free here.
Azure CLI, you can either install the Azure CLI locally or use the Azure Cloud Shell.

Let's get started

Computer Vision Deployment

Create a resource group:

 az group create -l <location> --name <name>

Create the Computer Vision resource:

az cognitiveservices account create \
    --name <computer-vision-resource-name> \
    --resource-group <resource-group> \
    --kind ComputerVision \
    --sku F0 \
    --location <location> \
    --yes

ℹ️ For the pricing tier, instead of F0 (= free tier) you can also specify S1 (= Standard S1).

⚠️ Only one free tier can be deployed at a time.

Grab the endpoint URL of your Computer Vision service via the Azure Portal:

Retrieve the access keys:

az cognitiveservices account keys list \
--name <computer-vision-resource-name> \
--resource-group <resource-group>

ℹ️ In case you want to regenerate or rotate your keys, you can use the following command:

az cognitiveservices account keys regenerate \
--key-name <Key1, Key2> \
--name <computer-vision-resource-name>
--resource-group <resource-group>

Generating of Thumbnails via the Azure Computer Vision API

Requirements and limitations of the Thumbnail API

Before we start calling the endpoint, here are some image requirements and limitations:

Image file size must be less than 4MB
Image dimensions should be greater than 50 x 50
Supported image formats
- JPEG
- PNG
- GIF
- BMP

How to call the API

Request Parameters:

Name	Type	Description
width	number	Width of the thumbnail. It must be between 1 and 1024. Recommended minimum of 50.
height	number	Height of the thumbnail. It must be between 1 and 1024. Recommended minimum of 50.
smartCropping (optional)	boolean	Boolean flag for enabling smart cropping.
model-version (optional)	string	Optional parameter to specify the version of the AI model. The default value is "latest".

Request Header:

Name	Type	Description
Content-Type	string	Media type of the body sent to the API.
Ocp-Apim-Subscription-Key	string	Subscription key which provides access to this API

Request Body:

The Thumbnail API supports the following content types:

application/json
application/octet-stream
multipart/form-data

application/json:

{"url":"http://example.com/images/test.jpg"}

application/octet-stream:

[Binary image data]

multipart/form-data:

[Binary image data]

ℹ️ The above section is an excerpt from the official documentation. For more details, please checkout the official documentation.

Example requests

application/json:

curl --location --request POST 'https://<computer-vision-resource-name>.cognitiveservices.azure.com/vision/v3.2/generateThumbnail?width=200&height=200&smartCropping=true&model-version=latest' \
--header 'Content-Type: application/json' \
--header 'Ocp-Apim-Subscription-Key: <api-key>' \
--data-raw '{
    "url":"http://example.com/images/test.jpg"
}'

application/octet-stream:

curl --location --request POST 'https://<computer-vision-resource-name>.cognitiveservices.azure.com/vision/v3.2/generateThumbnail?width=200&height=200&smartCropping=true&model-version=latest' \
--header 'Content-Type: application/octet-stream' \
--header 'Ocp-Apim-Subscription-Key: <api-key>' \
--data-binary '@<path-to-image>'

multipart/form-data:

curl --location --request POST 'https://<computer-vision-resource-name>.cognitiveservices.azure.com/vision/v3.2/generateThumbnail?width=200&height=200&smartCropping=true&model-version=latest' \
--header 'Content-Type: multipart/form-data' \
--header 'Ocp-Apim-Subscription-Key: <api-key>' \
--form 'img=@"<path-to-image>"'

For my example requests I used the following image:

ℹ️ The image is taken from Unsplash. Unsplash is really awesome and offers freely-usable images. You should definitely check it out!

This is the image with Smart Cropping:

And this is the image without Smart Cropping:

As you can clearly see in the picture without Smart Cropping parts of the keyboard are missing, as well the right arm was cropped away. Also in the left corner the plant is still visible. In the picture with Smart Cropping the keyboard, as well as both arms, are nicely visible and the plant in the left corner has been completely removed.

Pricing

As you could hopefully see, Azure Computer Vision thumbnail generation is extremely easy to use and offers useful features like smart cropping. But how much does the API cost?

Azure Computer Vision is a fully managed serverless service and pricing is "pay-as-you-go", in addition Microsoft offers a free tier for development purposes. The Computer Vision service for thumbnail generation is very affordable, 1000 transactions costs around $1.

⚠️ Disclaimer: This is an example configuration. Other prices may apply depending on configuration and region. For more precise estimation, I highly recommend using the Azure Pricing Calculator.

ℹ️ For enterprise scenarios with high volumes, Azure also offers a Committed Tier, where fixed contingents can be purchased at a reduced price. For more details, checkout the official pricing page for Azure Computer Vision.

Conclusion

I hope I could bring you a little bit closer to Azure Computer Vision for thumbnail generation. It really is an incredible service that greatly simplifies thumbnail creation and can make a developer's life immensely easier with features like Smart Cropping. Additionally, the service doesn't cost a fortune either 😉

Thanks for reading, let me know your thoughts on this feature!

Tutorial: Create an Azure Blob Storage with SFTP integration

Manuel Kanetscheider — Wed, 30 Mar 2022 01:16:03 +0000

Introduction

Azure has announced a SFTP feature for Storage Accounts. Currently, the feature is still in preview and should therefore not be used for production workloads until it reaches GA.

The new Azure Storage Account SFTP feature provides a full managed PaaS service. Before that, an SFTP service had to be provided in other ways such as:

Custom implementation: Development of a custom service that provides the SFTP functions, which then persists the files in a blob storage. For this use case Microsoft offers a template, consisting of an Azure Storage Account fileshare and a container instance. For more information, please checkout this link.
Virtual Machine: Deploy a VM with a SFTP service, either Windows or Linux. Especially for small use cases, this variant could be a overkill and depending on the configuration, this solution is also quite expensive. Furthermore, the administration overhead is significantly higher than with the integrated SFTP option.

Advantages of the PaaS variant:

✅ Easy to setup & use
✅ Fully managed serverless service
✅ Extremely cheap

I would like to emphasize the last point in particular. Azure Blob Storage is an extremely cost-effective storage option, here is an example calculation:

Storage Account Configuration:

Configuration Property Name	Value
Region	West Europe
Type	Block Blob Storage
Performance Tier	Standard
Storage Account Type	General Purpose V2
Access Tier	Hot
Redundancy	LRS

Storage Account Pricing:

Name	Description	Price
Capacity	100 GB	1.76 €
Write Operations	10 x 10.000	0.49 €
List and create Container Operation	10x 10.000	0.04 €
Data Retrieval	1000 GB	0.0 €
Data Write	1000 GB	0.0 €
Support	Included	0.0 €

Monthly cost: 2.78 €

As illustrated here, Azure Blob Storage is really affordable! In the example you get for not even 3€ a complete managed storage with many different features like SFTP support and many more!

⚠️ Disclaimer: This is an example configuration. Storage accounts are "pay as you go" and other prices may apply depending on configuration and region. For more precise estimation, I highly recommend using the Azure Pricing Calculator

Let's get started

Prerequisites

Azure Account with an active subscription. In case you do not have an Azure Account, go ahead and create one for free here
Azure CLI, you can either install the Azure CLI locally or use the Azure Cloud Shell.

ℹ️ If the Azure Cloud Shell is used, then the command az login is not needed. In the Cloud shell you are automatically logged in with the current user.

Enable the SFTP preview feature

In order to use this new feature, it has to be activated in advance for the respective subscription:

Navigate to Azure Portal
In the search bar, search for Subscriptions
Select your subscription (in my case, my subscription is named Pay-As-You-Go)
Inside your subscription, search for Preview Features
Inside the available features, search for "sftp" and register the SFTP support in Azure Blob Storage

Azure CLI:

Install the storage-preview extension:



az extension add -n storage-preview

Azure login:



az login

List available subscriptions and set active azure subscription:



az account list --all
az account set --subscription <subscription-id>

Enable SFTP feature:



az feature register --namespace Microsoft.Storage --name AllowSFTP

Verify feature registration:



az feature show --namespace Microsoft.Storage --name AllowSFTP

Create and configure the Azure Storage Account

Microsoft provides a demo template , in order to deploy the template click on the following button:

With this template we are ready to go and can test the SFTP feature. But of course we will also deploy everything from scratch:

Create the Azure Storage Account:



az storage account create -n <storage-account-name> \
-l <azure-location> \
-g <resource-group> \
--sku Standard_LRS \
--https-only=true \
--hierarchical-namespace=true \
--kind StorageV2

Enable SFTP:



az storage account update -g <resource-group> \
-n <storage-account> \
--enable-sftp=true

Create an Azure Storage Container:



az storage container create -n <container-name> \
--account-name <storage-account-name>

Create local user for SFTP access:



az storage account local-user create \
--account-name <storage-account> \
-g <resource-group> \
-n <local-user-name> \
--home-directory <home-dir> \
--permission-scope permissions=<permissions> service=blob resource-name=<container-name> \ 
--ssh-authorized-key key="<ssh-key>" \
--has-ssh-key true \
--has-ssh-password true

Authentication:
For authentication, there are two options available:

Password
SSH Key

In order to create or regenerate a password, use this command:



az storage account local-user regenerate-password \
--account-name <storage-account> \
-g <resource-group> \
-n <local-user-name>

There are many ways to generate SSH key pairs. Just use your preferred tool, alternatively the Azure CLI can be used:



az sshkey create --name "<ssh-key-name>" \
--resource-group "<resource-group>"

This command creates a new SSH key Azure resource:

For more details, checkout the official documentation.

⚠️ While you can enable both forms of authentication, SFTP clients can connect by using only one of them. Multifactor authentication, whereby both a valid password and a valid public and private key pair are required for successful authentication is not supported.

⚠️ You can't retrieve the generated password later, so make sure to copy the password, and then store it in a place where you can find it. If you lose this password, you'll have to generate a new one.

Home directory:
Home directory is only the initial directory that the connecting local user is placed in. Local users can navigate to any other path in the container they are connected to if they have the appropriate container permissions.
Format: container/folder/...

Available container permissions:

Permission	Permission Code	Description
Read	r	Read all file contents
Write	w	Upload file Create directory Upload directories
List	l	List contents within container List contents within directories
Delete	d	Delete files/directories
Create	c	Upload file if file doesn't exist Create directory if it doesn't exist Create directories

You can also specify multiple permission scopes:



--permission-scope permissions=rw service=blob resource-name=container1 \
--permission-scope permissions=rwd service=file resource-name=share2

Connect via SFTP to the Azure Storage Account

I will briefly go over the creation of the SFTP user again here. Lets get started! In my case, I created the following SFTP user:



az storage account local-user create --account-name <storage-account> \
-g <resource-group> \
-n mka \
--home-directory mystoragecontainer \
--permission-scope permissions=rwcdl service=blob resource-name=mystoragecontainer \
--has-ssh-password true

ℹ️ For the sake of simplicity, I only use password authentication. As mentioned above you can also choose SSH key authentication or even both (please note that you can't use both at the same time, you can authenticate either via password or SSH key).

In order to obtain the password of the newly created user, please use this command:



az storage account local-user regenerate-password \
--account-name <storage-account> \
-g <resource-group> \
-n mka

⚠️ Do not forget to copy the password after creating it, it cannot be requested again. If you lose the password, you must create a new one.

To connect to the storage account we need the blob primary endpoint. This can be obtained with the following command:



az storage account show -n <storage-account> -g <resource-group>

The previous command gives the general info about the storage account. Search for the section "primaryEndoints" and copy the "blob" URL without the prefix https or http(for example .blob.core.windows.net):



"primaryEndpoints": {
    "blob": "https://<storage-account-name>.blob.core.windows.net/",
    "dfs": "https://<storage-account-name>.dfs.core.windows.net/",
    "file": "https://<storage-account-name>.file.core.windows.net/",
    "internetEndpoints": null,
    "microsoftEndpoints": null,
    "queue": "https://<storage-account-name>.queue.core.windows.net/",
    "table": "https://<storage-account-name>.table.core.windows.net/",
    "web": "https://<storage-account-name>.z6.web.core.windows.net/"
  }

Now you can connect to the storage account with your **preferred SFTP client*. In my case I use PowerShell with OpenSSH and password authentication:



sftp <storage-account>.mka@<storage-account>.blob.core.windows.net

ℹ️ The username of the SSH user is in following format: storage-account.sftp-user-name

ℹ️ You might be prompted to trust a host key. During the public preview, valid host keys are published here.

After that you will be asked to authenticate yourself, just enter the password here.

After successfully authenticating, a file can be uploaded with this command:



put <path-to-file>

Verify the file upload either directly within the SFTP client or via the following Azure CLI command:

Retrieve Azure Storage Account Keys (for authentication):



az storage account keys list -n <storage-account> -g <resource-group>

The result of this should look something like this:



[
  {
    "creationTime": "2022-03-27T17:51:51.501304+00:00",
    "keyName": "key1",
    "permissions": "FULL",
    "value": "<key-value-1>"
  },
  {
    "creationTime": "2022-03-27T17:51:51.501304+00:00",
    "keyName": "key2",
    "permissions": "FULL",
    "value": "<key-value-2>"
  }
]

Simply copy one of the keys.

List files:



az storage fs file list -f <container-name> \
--recursive true \
--account-name <storage-account> \
--account-key <insert-previous-copied-key-here>

The output should look something like this:



[

  {

    "contentLength": 53759,

    "etag": "0x8DA11E2F03CABAB",

    "group": "$superuser",

    "isDirectory": false,

    "lastModified": "2022-03-30T20:19:21",

    "name": "<my-file>",

    "owner": "$superuser",

    "permissions": "rw-r-----"

  }

]

Conclusion

As mentioned before, Azure Storage Accounts are a very cost effective option to store files that now offer SFTP integration.
Since this variant is a so-called serverless, there are many exciting use-cases! Especially many legacy systems that work with files often offer an SFTP upload. These programs can now upload the files directly to the cloud and once the files are in the cloud they can be easily processed further, e.g. with an Azure function that processes the uploaded files with an "on-file-created" trigger.

If you made it this far, thanks for reading! Let me know your thoughts on this new feature.

Resources

Azure Functions and FastAPI

Manuel Kanetscheider — Thu, 24 Mar 2022 22:59:58 +0000

Introduction

Azure Functions are a great cloud service that enables the implementation of event-driven architectures like:

Building a web API
Process file uploads
Scheduled tasks
Processing events or queues
and much more!

In this blogpost I want to focus on the first point, building a web API with Azure Functions. Azure Functions offer several hosting options, but probably the most common hosting plan chosen is the consumption plan, better known as serverless plan.

While it is absolutely possible to create a web API with the built-in Azure function framework features I like to show how to use Azure Functions in combination with FastAPI.
For example, a vanilla Azure Function with Python might look like this:



def main(req: func.HttpRequest) -> func.HttpResponse:
    headers = {"my-http-header": "some-value"}

    name = req.params.get('name')
    if not name:
        try:
            req_body = req.get_json()
        except ValueError:
            pass
        else:
            name = req_body.get('name')

    if name:
        return func.HttpResponse(f"Hello {name}!", headers=headers)
    else:
        return func.HttpResponse(
             "Please pass a name on the query string or in the request body",
             headers=headers, status_code=400
        )

To learn more about Azure Functions check out the following links:

But why should we use FastAPI? FastAPI offers many useful features like:

Fast: Very high performance, on par with NodeJS and Go (thanks to Starlette and Pydantic). One of the fastest Python frameworks available.
Fast to code: Increase the speed to develop features by about 200% to 300%.
Fewer bugs: Reduce about 40% of human (developer) induced errors.
Intuitive: Great editor support. Completion everywhere. Less time debugging.
Easy: Designed to be easy to use and learn. Less time reading docs.
Short: Minimize code duplication. Multiple features from each parameter declaration. Fewer bugs.
Robust: Get production-ready code. With automatic interactive documentation.
Standards-based: Based on (and fully compatible with) the open standards for APIs: OpenAPI (previously known as Swagger) and JSON Schema.

In addition to the features already mentioned, FastAPI also offers the following advantages over vanilla Azure Functions:

Model binding for requests and response with additional model validation features provided by Pydantic.
Dependency management and the creation of reusable components (like common query parameters).
Open API definition with built-in /docs route

For more details please visit the official FastAPI documentation.

Let's get started

Prerequisites

Install the Azure Function Core Tools
Python 3.9.0
Azure Subscription (optional)

Setup the local development environment

Create the Function project:



func init <your_function_project_name> --worker-runtime python

Navigate into your newly created function project and create a HTTP function:



func new --template "Http Trigger" --name main

In order to start the Azure Function, please use this command:



func start --python

After creating the project, open the project in the code editor of your choice and edit the following files as follows:



{
  "scriptFile": "__init__.py",
  "bindings": [
    {
      "authLevel": "function",
      "type": "httpTrigger",
      "direction": "in",
      "name": "req",
      "methods": [
        "get",
        "post",
        "patch",
        "delete"
      ],
      "route": "/{*route}"
    },
    {
      "type": "http",
      "direction": "out",
      "name": "$return"
    }
  ]
}

function.json

This endpoint becomes the main entry point into the application. For this route to match all patterns the "route" property must be modified as shown above. This way all incoming requests will be handled by this route. The processing of the web requests is managed by FastAPI.

Additionally the allowed HTTP methods like GET, POST etc. have to be specified. If a HTTP method is used that was not specified, the Azure function throws a "method not allowed" exception.
Furthermore, the host.json file must also be updated as follows:



{
  "version": "2.0",
  "logging": {
    "applicationInsights": {
      "samplingSettings": {
        "isEnabled": true,
        "excludedTypes": "Request"
      }
    }
  },
  "extensionBundle": {
    "id": "Microsoft.Azure.Functions.ExtensionBundle",
    "version": "[2.*, 3.0.0)"
  },
  "extensions": 
  {
    "http": 
    {
        "routePrefix": ""
    }
  }
}

host.json

For more details, please checkout the official documentation.

Let's write some code

The snippet from the official documentation looks like this:



app=fastapi.FastAPI()

@app.get("/hello/{name}")
async def get_name(
  name: str,):
  return {
      "name": name,}

def main(req: func.HttpRequest, context: func.Context) -> func.HttpResponse:
    return AsgiMiddleware(app).handle(req, context)

The code shown above glues the Azure Function and FastAPI together. After this snippet all features of the standard FastAPI framework can be used.

For example my implementation looks like this:



from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
import azure.functions as func

from routers import products
from utilities.exceptions import ApiException


description = """
This is a sample API based on Azure Functions and FastAPI.

This API is used to illustrate how a potential API with Azure Functions and FastAPI could look like, it is a demo API only.
I hope you like it and help you to build awesome projects based on these great frameworks!

## Products
* Add products
* Retrieve products
* Retrieve a specific product by ID
* Update existing products
* Delete products by ID
"""

app = FastAPI(
    title="Azure Function Demo FastAPI",
    description=description,
    version="0.1",
    contact={
        "name": "Manuel Kanetscheider",
        "url": "https://dev.to/manukanne",
        "email": "me@manuelkanetscheider.net"
    },
    license_info= {
        "name": "MIT License",
        "url": "https://github.com/manukanne/tutorial-az-func-fastapi/blob/main/LICENSE"
    }
)
app.include_router(products.router)
# Add additional api routers here


@app.exception_handler(ApiException)
async def generic_api_exception_handler(request: Request, ex: ApiException):
    """
    Generic API exception handler. 
    Ensures that all thrown excpetions of the custom type API Excpetion are returned 
    in a unified exception JSON format (code and description).    
    Args:
        request (Request): HTTP Request
        ex (ApiException): Thrown exception

    Returns:
        JSONResponse: Returns the exception in JSON format
    """
    return JSONResponse(
        status_code=ex.status_code,
        content={
            "code": ex.code,
            "description": ex.description
        }
    )


def main(req: func.HttpRequest, context: func.Context) -> func.HttpResponse:
    """
    Azure function entry point.
    All web requests are handled by FastAPI.
    Args:
        req (func.HttpRequest): Request
        context (func.Context): Azure Function Context

    Returns:
        func.HttpResponse: HTTP Response
    """
    return func.AsgiMiddleware(app).handle(req, context)

And the implementation of the example product routes looks like this:



import logging
from fastapi import APIRouter, Depends
from typing import Optional, List

from orm import DatabaseManagerBase
from dependencies import get_db
from utilities.exceptions import EntityNotFoundException, ApiException
import schemas

router = APIRouter(
    prefix="/products",
    tags=["products"]
)


@router.post("/", response_model=schemas.Product, summary="Creates a product")
async def add_product(product_create: schemas.ProductCreate, db: DatabaseManagerBase = Depends(get_db)):
    """
    Create a product:

    - **title**: Title of the product
    - **description**: Description of the product
    - **purch_price**: The purch price of the product
    - **sales_price**: The sales price of the product
    """
    logging.debug("Products: Add product")
    product = db.add_product(product_create)
    return product


@router.get(
    "/",
    response_model=Optional[List[schemas.Product]],
    summary="Retrieves all prodcuts",
    description="Retrieves all available products from the API")
async def read_products(db: DatabaseManagerBase = Depends(get_db)):
    logging.debug("Product: Fetch products")
    products = db.get_products()
    return products


@router.get(
    "/{product_id}",
    response_model=Optional[schemas.Product],
    summary="Retrieve a product by ID",
    description="Retrieves a specific product by ID, if no product matches the filter criteria a 404 error is returned")
async def read_product(product_id: int, db: DatabaseManagerBase = Depends(get_db)):
    logging.debug("Prouct: Fetch product by id")
    product = db.get_product(product_id)
    if not product:
        raise EntityNotFoundException(code="Unable to retrieve product",
                                      description=f"Product with the id {product_id} does not exist")
    return product


@router.patch("/{product_id}", response_model=schemas.Product, summary="Patches a product")
async def update_product(product_id: int, product_update: schemas.ProductPartialUpdate, db: DatabaseManagerBase = Depends(get_db)):
    """ 
    Patches a product, this endpoint allows to update single or multiple values of a product

    - **title**: Title of the product
    - **description**: Description of the product
    - **purch_price**: The purch price of the product
    - **sales_price**: The sales price of the product
    """
    logging.debug("Product: Update product")

    if len(product_update.dict(exclude_unset=True).keys()) == 0:
        raise ApiException(status_code=400, code="Invalid request",
                           description="Please specify at least one property!")

    product = db.update_product(product_id, product_update)
    if not product:
        raise EntityNotFoundException(
            code="Unable to update product", description=f"Product with the id {product_id} does not exist")
    return product


@router.delete("/{product_id}", summary="Deletes a product", description="Deletes a product permanently by ID")
async def delete_product(product_id: int, db: DatabaseManagerBase = Depends(get_db)):
    logging.debug("Product: Delete product")
    db.delete_product(product_id)

I won't go into the details of the technical implementation of my FastAPI routes here, that's a topic for maybe another blog post. Nevertheless, you can download all the documented source code in the linked GitHub repo.

Testing the API

You can test the API I developed either via the provided Postman Collection or via the built-in documentation of FastAPI (the docs are provided via the route /docs).

Deploy to Azure

This step requires an Azure Account. In case you do not have an Azure Account you can go ahead and create an account for free here.

The repository contains an ARM template. An ARM template contains all the necessary information to deploy resources in Azure, in this case it contains the information to deploy an Azure Function including:

Storage Account
Serverless Hosting Plan
Function App

For more information about ARM templates, please checkout the official documentation.

Also worth mentioning is the new technology "Bicep". It is similar to ARM templates but offers a declarative approach. Bicep is comparable to Terraform, but unlike Terraform, Bicep can only be used for Azure. For more information about Bicep, please checkout the official documentation.

The provided ARM template was originally developed by Ben Keen, for more information about this ARM template, please checkout his blog post.

Deploy the ARM template and the function code using the Azure CLI:



az deployment group create --resource-group <resource-group> --template-file .\az-func-template.json --parameters appName='<your_app_name>' storageAcctName='<your_storage_account_name>' hostingPlanName='<your_hosting_plan_name>'

func azure functionapp publish <your_function_app_name>

Before executing the commands, please make sure that you have called "az login"

Conclusion

Azure Functions enable you to create scalable and cost-effective web APIs. With FastAPI, the functionality of Azure functions can be extended tremendously, making it easy to create complex APIs.

Thank you for reading, I hope you enjoyed it!

Link to the full source code on GitHub:

manukanne / tutorial-az-func-fastapi

Tutorial: Azure Function with FastAPI

This project is a demonstration of how Azure Functions can be used in combination with FastAPI.

Description

Demo API with the following endpoints:

Product
- Create
- Read all products
- Read a specific product
- Patch product data
- Delete a product

It should be noted that this is only a demo API. This API does not use a real database and therefore only uses a very simplified database manager with an "InMemory" database.

Getting started

Prerequisites

Install the Azure Function Core Tools
Python 3.9.0
Azure Subscription (optional)

After that, create a python virtual environment, activate it and install the requirements.

Start the function

func start --python

In order to start the Azure Function via the Azure Function Core Tools (CLI) a activated virtual environment is required.

After starting Azure Functions you can access the documentation via this link:

http://localhost:7071/docs

Deploy to Azure:

This step requires an Azure Account…

View on GitHub

AtlasHackathon - A Quiz API using MongoDB and FastAPI

Manuel Kanetscheider — Wed, 12 Jan 2022 20:06:47 +0000

I created this application to take part in MongoDB Atlas Hackathon.

Project Idea

I wanted to create a quiz API. The API allows to create quizzes and subsequently also to submit a quiz and calculate the achieved points. Quizzes can also be tagged with categories (e.g. fun, programming, general knowledge etc.).
The API also includes authentication using OAUTH2, the user data as well as all other data is stored in the MongoDB cloud atlas database.

Core functions at a glance:

Authentication/ Authorization using OAuth2
CRUD Operations for quiz categories
CRUD Operations for quizzes
- Including Quiz submition and calulating the total and reached points
Also, the project is fully covered by unit tests

Overview of My Submission

This project was built on top of the following tech-stack:

You can test the API with the included Postman collection:

Before using the collection, please create a Postman environment:

api_url: URL to the API
username: email of your user
password: plain text password of your user

After this step the Access Token can be obtained:

Alternatively, the API can also be tested with the built-in Swagger UI:

The docs are available at this link: {host}:{port}/docs

After that: Have fun :)

For more details, please read the README.md

Submission Category: Choose Your Own Adventure

Link to Code

manukanne / cloudatlas-quiz-api

MongoDB Quiz API

This project was created as part of the MongoDB Hackathon, it is a quiz API that provides the following functions:

Authentication/ Authorization using OAuth2
CRUD Operations for quiz categories
CRUD Operations for quizzes
- Including Quiz submition and calulating the total and reached points
Also, the project is fully covered by unit tests

Getting Started

This project is built on top of the Python FastApi framework using the awesome cloud based MongoDB Atlas database.

Dependencies

Tech-stack

Installing

1. Clone the git repo:

git clone https://github.com/manukanne/cloudatlas-quiz-api.git

2. Download and install Python

To download python, please vistit this link.

For this project the following Python version was used: 3.9.*

3. Create a python virtual envrionment:

For more information on how to create please visit this link.

4. Install the required packages

pip install requirement.txt

For more information, please vist this link…

View on GitHub

Additional Resources / Info

Architecture Patterns with Python