Forem: Be Hai Nguyen

rlox: A Rust Implementation of “Crafting Interpreters” – Scanner

Be Hai Nguyen — Sat, 14 Jun 2025 02:13:25 +0000

I am attempting a Rust implementation of Robert Nystrom's Lox language discussed in Crafting Interpreters. This post describes my Rust code equivalence for the Scanning chapter.

🦀 Index of the Complete Series.

This is the long list of existing Rust Lox Implementations. I downloaded and ran the first two, but I did not have a look at the code. I would like to take on this project as a challenge. If I complete it, I want it to reflect my own independent effort.

🚀 Please note, code for this post can be downloaded from GitHub with:

git clone -b v0.1.0 https://github.com/behai-nguyen/rlox.git

● To run interactively, first change to the rlox/ directory, then run the following command:

$ cargo run

Enter something like var str2 = "秋の終わり";, and press Enter — you will see the tokens printed out. Please refer to the screenshot below for an illustration.

At the moment, inputs are processed independently, meaning each new input does not retain any connection to previous inputs.

To exit, simply press Enter without entering anything.

● To Run with a Lox script file, first change to the rlox/ directory, then run the following command:

$ cargo run ./tests/data/scanning/numbers.lox

If there are no errors, you will see the tokens printed out.

● To run existing tests, first change to the rlox/ directory, then run the following command:

$ cargo test

❶ Repository Layout

.
├── Cargo.toml
├── README.md
├── src
│   ├── lib.rs
│   ├── lox_error.rs
│   ├── main.rs
│   ├── scanner_index.rs
│   ├── scanner.rs
│   ├── token.rs
│   └── token_type.rs
└── tests
    ├── data
    │   └── scanning
    │       ├── identifiers.lox
    │       ├── keywords.lox
    │       ├── numbers.lox
    │       ├── punctuators.lox
    │       ├── README.md
    │       ├── sample.lox
    │       ├── strings.lox
    │       ├── utf8_text.lox
    │       └── whitespace.lox
    ├── test_common.rs
    └── test_scanner.rs

❷ Let's briefly describe the project.

● Identifier names follow Rust convention. In the Scanning chapter, method names such as scanTokens(), peekNext() are scan_tokens() and peek_next() in Rust respectively.

● Identifier names which are keywords in Rust will simply have an underscore () suffix appended. For example, match() becomes match(), and type becomes type_.

● The src/scanner_index.rs module is not in the original Java version. It implements the Java variables start, current, line and some additional fields to support UTF-8 text scanning and slicing; please refer to this post for a full discussion on supporting UTF-8 text slicing.

● In the src/token.rs module, I am not sure if we need the literal field in the Token struct in the future. I leave it for the time being.

● 💥 In the src/scanner.rs module, the method scan_tokens() returns an array (vector) of Token; and the run() function in the src/main.rs module consumes this array and drops it. This array is local. In the Java implementation, it is a global class variable. This implementation might change in the future.

● The src/lox_error.rs module is also not in the original Java version. It implements a Rust specific error struct.

● Under tests/data/scanning/ directory, except for utf8_text.lox which is mine; the README.md lists the original addresses of all other test data files.

● The tests/test_scanner.rs module implements test for each of the test data files in the tests/data/scanning/ directory.

❸ The above points are specific to this implementation, otherwise the code adhere to Crafting Interpreters, chapter Scanning.

Thank you for reading. I hope you find this post helpful. Stay safe, as always.

✿✿✿

Feature image sources:

🦀 Index of the Complete Series.

Raspberry Pi 4B: Natively Build a 64 Bit Fully Preemptible Kernel (Real-Time) with Desktop

Be Hai Nguyen — Sun, 03 Nov 2024 12:05:48 +0000

In this article, I present a step-by-step procedure to natively patch and build a 64-bit Fully Preemptible Kernel (Real-Time) for my Raspberry Pi 4B (Pi 4B). We start with a pre-made operating system image that includes a desktop environment. After building and installing the real-time patch kernel, querying the kernel information with the command uname -a should report something similar to:

Linux picnc 6.6.59-rt45-v8-behai-rt-build+ #1 SMP PREEMPT_RT Sat Nov  2 10:20:46 AEDT 2024 aarch64 GNU/Linux

This indicates that the token PREEMPT_RT is present in the output. Natively in this instance means that we patch and build the kernel using the very Pi 4B on which the newly built kernel is installed.

Let's first describe the steps and then provide some background information on why I looked into this.

❶ We will now discuss the step-by-step procedure.

⓵ I use a newly installed Ubuntu 24.04.1 LTS to write the target operating system (OS) image onto a microSD card. We need to install Raspberry Pi Imager:

$ sudo apt install rpi-imager

There should not be any problem. Raspberry Pi Imager v1.8.5 should appear in the Ubuntu application list. 💥 I first tried to install it on Ubuntu 22.10 Kinetic, but it reported a lot of errors, which I tried to fix without success.

⓶ Use Raspberry Pi Imager to write a pre-made OS image onto a microSD card. This application should be self-explanatory. For Operating System, I select Raspberry Pi OS (other), then Raspberry Pi OS Full (64-bit). Please see the screenshots below:

The NEXT button lets us configure the new OS. Please see the illustration below:

In the SERVICES tab, we can enable SSH. The writing of the OS image should not take too long. After finishing writing, we should be able to boot without any issues. My hostname is picnc.local, and my username is behai. Both of the following commands should work:

$ ping picnc.local
$ ssh behai@picnc.local

We can query the kernel information with:

$ uname -a

In my case, the output is:

Linux picnc 6.6.51+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.51-1+rpt3 (2024-10-08) aarch64 GNU/Linux

Note the PREEMPT token. We are going to change that to PREEMPT_RT.

We can also use the following commands to query the kernel information:

$ uname -r
$ uname -srm
$ hostnamectl
$ cat /proc/version

👉 The following steps are extracted from the Raspberry Pi page The Linux kernel. This official page covers multiple OSes. I have arranged the steps to suit the OS that I am using.

⓷ Download the latest kernel source code. Please refer to this official section. Change to the $HOME directory:

$ cd $HOME

And run the command:

$ git clone --depth=1 https://github.com/raspberrypi/linux

This downloads the kernel source code to $HOME/linux. Please note that the git utility is included with the OS image. It can be installed with:

$ sudo apt install git

⓸ Patch the real-time kernel. Please refer to this official section.

● Determine the current kernel version:

$ uname -r

The output will be something like:

6.6.51+rpt-rpi-v8

● Determine the kernel source version. Change to the kernel source directory:

$ cd linux

Run the following command to get the kernel source version:

$ head Makefile -n 4

The output will be something like:

# SPDX-License-Identifier: GPL-2.0
# SPDX-License-Identifier: GPL-2.0
VERSION = 6
PATCHLEVEL = 6
SUBLEVEL = 59

In this instance, the kernel source version is 6.6.59. At the time, there was no version 6.6.59 of the real-time kernel patch at https://www.kernel.org/pub/linux/kernel/projects/rt/6.6/, so I used patch version 6.6.58 instead. Please note that on the internet, there are numerous mentions of this issue. Sometimes patches are behind, and using a not-so-older version of the patch should be okay.

● Apply the target real-time kernel patch. 🙏 Please recall that the present working directory is $HOME/linux. The following commands download, uncompress, and patch the kernel source code with the target real-time kernel patch:

$ wget https://www.kernel.org/pub/linux/kernel/projects/rt/6.6/patch-6.6.58-rt45.patch.gz
$ gunzip patch-6.6.58-rt45.patch.gz
$ cat patch-6.6.58-rt45.patch | patch -p1

⓹ Configure the kernel. Please refer to this official section.

● Install build tools/dependencies:

$ cd $HOME
$ sudo apt install libncurses5-dev flex build-essential bison libssl-dev bc make
$ cd linux

I don't think it's necessary to navigate to the $HOME directory to perform the installation, but I prefer to do so.

● Prepare the default kernel configuration: $HOME/linux/.config. Please refer to this official section. 🙏 The present working directory is $HOME/linux. Run the following commands:

$ KERNEL=kernel8
$ make bcm2711_defconfig

The last output lines of the second command are:

...
#
# configuration written to .config
#

● Check real-time configuration items in /home/behai/linux/.config. Open /home/behai/linux/.config and look for the following real-time entries:

...
CONFIG_PREEMPT_BUILD=y
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
CONFIG_PREEMPT=y
# CONFIG_PREEMPT_RT is not set
CONFIG_PREEMPT_COUNT=y
CONFIG_PREEMPTION=y
# CONFIG_PREEMPT_DYNAMIC is not set
...

Based on my previous failed attempts, I want to ascertain that PREEMPT_RT is not turned on. And indeed, it is not.

⓺ Customise the kernel version using LOCALVERSION. Please refer to this official section.

In /home/behai/linux/.config, locate the LOCALVERSION entry, which should be:

CONFIG_LOCALVERSION="-v8"

And update it to:

CONFIG_LOCALVERSION="-v8-behai-rt-build"

In the next section on menuconfig, we will verify that this manual update takes effect.

⓻ Using the UI tool menuconfig to configure the kernel to Fully Preemptible Kernel (Real-Time). Please refer to this official section. 🙏 The present working directory is $HOME/linux.

-- Please note that this UI tool also updates the $HOME/linux/.config file.

Run the command:

$ make menuconfig

If everything goes well, and it should, you will get the UI shown below:

Use the up and down arrow keys to move between the vertical entries, use the left and right arrow keys to move between the horizontal entries at the bottom of the screen. Press the Enter key to select the highlighted item. Activate the < Load > menu item. The following pop-up appears:

Select < Ok > to confirm using the $HOME/linux/.config configuration file, which has been loaded by default.

Activate the first menu item General setup --->. On the next screen, select Preemption Model (Preemptible Kernel (Low-Latency Desktop)) ---> as shown below:

In the pop-up dialog, select ( ) Fully Preemptible Kernel (Real-Time) as per the screenshot below:

The main screen should show Preemption Model (Fully Preemptible Kernel (Real-Time)) ---> as per the following screenshot:

Recall that in a previous section, we customised the kernel version LOCALVERSION. The third entry in the previous screen confirms this manual customisation as shown below:

Save the changes and exit menuconfig. 💥 As a precaution, rerun menuconfig to confirm that your changes were saved.

👉 As a final verification, ensure that CONFIG_PREEMPT_RT is updated to y. Open /home/behai/linux/.config and look for the following real-time entries:

...
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set
CONFIG_PREEMPT_RT=y
CONFIG_PREEMPT_COUNT=y
CONFIG_PREEMPTION=y
...

⓼ We now discuss the actual kernel build process.

● Install kernel headers. Please refer to this official section. Run the following commands:

$ cd $HOME
$ sudo apt install linux-headers-rpi-v8
$ cd linux

● Build the kernel. Please refer to this official section. 🙏 The present working directory is $HOME/linux.

My Pi 4B has 8GB of RAM and 4 cores. The build process takes around 2 hours. To determine the number of cores, run the command:

$ nproc

We use the number of cores in the build command:

$ make -j4 Image.gz modules dtbs

It takes a few hours. We can do something else in the meantime, but keep an eye on it.

⓽ Install the newly built kernel. Please refer to this official section. 🙏 The present working directory is still $HOME/linux.

● First, install the kernel modules onto the boot media:

$ sudo make -j4 modules_install

● Then, install the kernel and Device Tree blobs into the boot partition, backing up the original kernel with the commands listed below. Please recall that the temporary environment variable $KERNEL is defined in this step. If you were interrupted and had to turn off the computer or terminated the SSH session, you will need to set it again.

$ sudo cp /boot/firmware/$KERNEL.img /boot/firmware/$KERNEL-backup.img
$ sudo cp arch/arm64/boot/Image.gz /boot/firmware/$KERNEL.img
$ sudo cp arch/arm64/boot/dts/broadcom/*.dtb /boot/firmware/
$ sudo cp arch/arm64/boot/dts/overlays/*.dtb* /boot/firmware/overlays/
$ sudo cp arch/arm64/boot/dts/overlays/README /boot/firmware/overlays/

⓾ Finally, reboot to ensure that the OS works 😂 and the kernel is indeed a Fully Preemptible Kernel (Real-Time) kernel.

To reboot:

$ sudo reboot

Ping should respond successfully:

$ ping picnc.local

SSH should work:

$ ssh behai@picnc.local

The response I received was:

Linux picnc 6.6.59-rt45-v8-behai-rt-build+ #1 SMP PREEMPT_RT Sat Nov  2 10:20:46 AEDT 2024 aarch64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sat Nov  2 12:03:31 2024

Please note the tokens 6.6.59-rt45-v8-behai-rt-build+ and PREEMPT_RT in the first line. We have successfully built and installed a Fully Preemptible Kernel (Real-Time) kernel. ✔️ I also ran the commands to query the kernel information as discussed in a prior section.

💥 As a final note on this OS, the open-source Remote Desktop Protocol server xrdp does not seem to work properly. After writing the initial OS image, I installed this xrdp server, but Windows Remote Desktop Connection just shows a blank black screen. A few days prior, I tried this same image, but the release date was 04/July/2024, as seen in the screenshot below:

And Remote Desktop Connection worked perfectly. Raspberry Pi updated this image while I was still trying to resolve this Fully Preemptible Kernel (Real-Time) issue.

❷ As mentioned at the beginning, in this section I'm discussing why I have looked into this issue. I am interested in learning CNC, and my goal is to build my own CNC machine. I know nothing about CNC and electronics. I have been doing a little bit of basic physical computing with the Pi 4B using Python. In the process, I have learned some basic electronics as well.

My initial research on CNC pointed to the Mach3 CNC machine controller, which runs on Windows. Further research led me to LinuxCNC, which I found to be a more attractive option, especially since it can run on a Raspberry Pi computer. It became apparent that we need a real-time kernel.

⓵ My initial source of information is a 6-year-old YouTube series, LinuxCNC for the Hobbyist, by Mr. Joe Hildreth. The 17th video, Compiling a Realtime Kernel for LinuxCNC, discusses building a real-time kernel. I found the accompanying written tutorial, Compiling a Realtime Linux kernel with the preemp-rt kernel patch, much easier to follow.

⓶ My second source of information is a 2-year-old LinuxCNC forum post titled Installing LinuxCNC 2.9 on Raspberry Pi 4B with Preempt-RT kernel.

⓷ I will not go into the details of my failed attempts to build a real-time kernel. However, between the instructions, I had about 6 tries over the span of 12 days 😂, and despite successfully building and installing, the kernel only showed PREEMPT, not PREEMPT_RT.

On my last failed attempt, I went ahead and installed LinuxCNC anyway. A variation of it was installed successfully, and I was able to run it via Remote Desktop Connection as shown below:

⓸ This nearly 3-year-old Stack Overflow post, Raspberry pi4: kernel 64bit with RT extension, addresses the very same problem I had, and the author managed to solve it. This post eventually led me to the previously mentioned Raspberry Pi official page, The Linux kernel, and this successful build and installation.

Compiling the Linux kernel is not my cup of tea. However, I wouldn't say my failed attempts were a waste of time. I have learned many things during these exercises, and some of the steps are similar to the final successful attempt.

⓹ This has been a new and interesting experience for me. I have learned quite a few things during this process. I am not sure if I can achieve my goal, as I have done some research into hardware, and there are many things I have no knowledge of. But I keep my fingers crossed...

Thank you for reading. I hope you find the information in this post useful. Stay safe, as always.

✿✿✿

Feature image source:

https://www.instructables.com/Easy-Raspberry-Pi-Based-ScreensaverSlideshow-for-E/

Python & MariaDB: Which Driver? An Example of Executing a Stored Procedure That Returns Multiple Result Sets

Be Hai Nguyen — Sun, 28 Jul 2024 11:25:43 +0000

In this discussion, I explain why I prefer the Python MySQL driver, mysql-connector-python, for the MariaDB database over the mariadb driver. The latter appears to be recommended by the official MariaDB documentation and is also mentioned on the SQLAlchemy page for MySQL and MariaDB.

❶ This new post could be considered as related to some previous posts listed below:

Synology DS218: MariaDB 10 enabling remote connection: This Synology DS218 Linux box, which serves as the MariaDB 10 database server, is used in this new post.
Python: executing MySQL stored procedures which return multiple result sets: This new post is an extension of the previous one. We will be using the same database content, the same stored procedure, and the same Python test code.
Python: executing PostgreSQL stored functions which return multiple result sets.
Rust & MySQL: executing MySQL stored procedures which return multiple result sets using crate sqlx.

❷ Around two years ago, when I bought my Synology DS218 box, I became aware of the MariaDB database. My research suggests that it is superior to and also compatible with the MySQL database. As demonstrated in a previously mentioned post, Synology DS218: MariaDB 10 enabling remote connection, I can use all MySQL client tools with the MariaDB database. Please refer to the following post, MariaDB vs MySQL: A Detailed Comparison, for further information.

❸ I have not been using the MariaDB database in my development, as I have both the MySQL and PostgreSQL databases available.

I have been exploring the MariaDB database recently. I tried out the recommended mariadb driver. Instead of following the official example, I experimented with my own stored procedure which returns multiple result sets, replicating the example in a previously mentioned post.

To recap:

⓵ The source database is the MySQL test data released by Oracle Corporation, downloadable from https://github.com/datacharmer/test_db. Using MySQL tools, I backed up a MySQL database and restored the backup content to the MariaDB database server on the Synology DS218 Linux box.

⓶ The stored procedure is reprinted below:

delimiter //

drop procedure if exists DemoStoredProc1; //

create procedure DemoStoredProc1( pm_dept_no varchar(4) )
reads sql data
begin
  select * from departments where dept_no = pm_dept_no;
  select * from dept_manager where dept_no = pm_dept_no;
end; //

👉 I verified that this stored procedure works as expected with the MariaDB database. If there is a match, the first record set will have only a single record, the second record set will have one or more records: they are distinct record sets. Please note, we could have more than two, but for the purpose of this post, two should suffice.

⓷ Install the mariadb driver into the active virtual environment.

⓸ The Python code is copied from the previous post as mentioned several times above. The only modification is: replacing the connection string as appropriate for the mariadb driver.

Content of mariadb-example.py:

from sqlalchemy import create_engine
from contextlib import closing

engine = create_engine( 'mariadb+mariadbconnector://behai:O,U#n*m:5QB3_zbQ@192.168.0.14:3306/employees', echo = False )
# engine = create_engine( 'mysql+mysqlconnector://behai:O,U#n*m:5QB3_zbQ@192.168.0.14:3306/employees', echo = False )

connection = engine.raw_connection()

try:
    with closing( connection.cursor() ) as cursor:
        cursor.callproc( 'DemoStoredProc1', [ 'd001' ] )

        data = []
        for sr in cursor.stored_results():
            #-- 
            columns = [ column[0] for column in sr.description ]
            ds = sr.fetchall()

            dataset = []
            dataset.append( columns )
            for row in ds:
                dataset.append( list(row) )

            data.append( dataset )
            #--
            sr.close()

        cursor.close()

        import pprint
        print( '\n' )
        pprint.pprint( data )

except Exception as e:
    print( f'Exception. Type {type(e)}: {str(e)}' )
finally:
    if 'connection' in locals():
        connection.close()

Please note, 192.168.0.14 is the IP address of my Synology DS218 Linux box.

It produces the following runtime error:

Exception. Type <class 'AttributeError'>: 'Cursor' object has no attribute 'stored_results'

Please also see the screenshot below:

This is not a database server error, but rather a driver error. However, we know the MariaDB server executes the above stored procedure correctly, so there is no problem on the server-side. The following page is the official MySQL documentation on the MySQLCursor.stored_results() method. I have not been able to locate an equivalent for the MariaDB database.

Having already installed the mysql-connector-python driver, switching the connection string to mysql+mysqlconnector:

# engine = create_engine( 'mariadb+mariadbconnector://behai:O,U#n*m:5QB3_zbQ@192.168.0.14:3306/employees', echo = False )
engine = create_engine( 'mysql+mysqlconnector://behai:O,U#n*m:5QB3_zbQ@192.168.0.14:3306/employees', echo = False )

produces the expected results. Please refer to the two screenshots below:

I am not sure if I missed something about the mariadb driver. I have looked through its documentation, and I have yet to find an equivalent of MySQLCursor.stored_results().

❹ Testing with the bh-database wrapper classes for SQLAlchemy.

For the two examples provided, if you want to access the MariaDB employees database, simply modify the connection string to match the one used in the above example:

SQLALCHEMY_DATABASE_URI = mysql+mysqlconnector://behai:O,U#n*m:5QB3_zbQ@192.168.0.14:3306/employees

All functions from the two examples will continue to operate correctly when using the MariaDB database.

❺ Given all the observations discussed above, the mysql-connector-python driver appears to be suitable for the MariaDB database. I plan to undertake some development work with MariaDB in the future. I will share any interesting findings I encounter with the MariaDB database.

Thank you for reading. I hope you find the information in this post useful. Stay safe, as always.

✿✿✿

Feature image source:

Python FastAPI: Implementing Non-Blocking Logging with Built-In QueueHandler and QueueListener Classes

Be Hai Nguyen — Tue, 02 Jul 2024 10:57:17 +0000

Continuing with our Python FastAPI learning series, this post explores the implementation of non-blocking logging using Python’s built-in QueueHandler and QueueListener classes.

🐍 Index of the Complete Series.

Starting from this post, the code will require Python 3.12.4. Please refer to the following discussion on how to upgrade to Python 3.12.4.

🚀 Please note, complete code for this post can be downloaded from GitHub with:

git clone -b v0.5.0 https://github.com/behai-nguyen/fastapi_learning.git

❶ Definition and Complete Working Example of Non-Blocking Logging
❷ Structure of Logging
❸ Project Layout
❹ Implementation of Non-Blocking Logging
- ⓵ YAML Logger Configuration File
- ⓶ New Python Module: common/queue_logging.py
- ⓷ Updates to the main.py Module
- ⓸ Incorporating Logging into Existing Modules
❺ Essential Official Documentation
❻ Concluding Remarks

❶ In essence, non-blocking logging means that the actual logging task does not hold up the thread performing the logging. This thread does not have to wait for the logging to complete and can move to the next instruction immediately.

Non-blocking logging is achieved via three principal built-in classes: queue.Queue, QueueHandler, and QueueListener. An instance of queue.Queue is accessible by both a non-blocking QueueHandler instance and a QueueListener instance. The QueueListener instance passes the logging messages to its own blocking handler(s), such as a RotatingFileHandler.

According to the official documentation for QueueHandler and QueueListener, they have their own separate thread for logging. This frees the main thread from waiting for the logging to finish, thereby preventing it from being blocked.

The complete working example below, adapted from a Stack Overflow answer and a Medium post, illustrates how the aforementioned classes fit together to implement non-blocking logging. Logging messages are written to the queue.log file in the same directory as the Python script file:

from contextlib import asynccontextmanager

from fastapi import FastAPI, Request
import logging

import queue
from logging.handlers import QueueHandler, QueueListener, RotatingFileHandler

logger = logging.getLogger()
logger.setLevel(logging.DEBUG)

log_queue = queue.Queue()
# Non-blocking handler.
queue_handler = QueueHandler(log_queue)  

queue_handler.setFormatter(logging.Formatter("%(asctime)s - %(levelname)s - %(message)s"))

# Attached to the root logger.
logger.addHandler(queue_handler)           

# The blocking handler.
rot_handler = RotatingFileHandler('queue.log')

# Sitting comfortably in its own thread, isolated from async code.
queue_listener = QueueListener(log_queue, rot_handler)

# Start listening.
queue_listener.start()

@asynccontextmanager
async def lifespan(app: FastAPI):
    yield

    logger = logging.getLogger()
    logger.info("Application is shutting down.")

    # Should stop listening.
    queue_listener.stop()

app = FastAPI(lifespan=lifespan)

@app.get("/")
async def root(request: Request):
    logger.debug(f"I am {request.url}")

    return {"message": "Hello World"}

I am not entirely sure if the above logging approach can also be classified as asynchronous logging, because only one thread can control the Python interpreter at a time, as discussed in this article.

The final implementation for this post is slightly more complex than the example above, but the underlying technical principles remain the same.

❷ Let’s discuss how the logging messages for each request should be structured.

When a request is being served, the application automatically logs the following message: * Request Started [< Session Id not available >][UUID session Id]
The request's endpoint handler can optionally log its own messages.
After a request has been served, the application automatically logs the following message: * Request Finished [< Session Id not available >][UUID session Id]

We will refer to the two messages * Request Started ... and * Request Finished ... as a marker pair throughout the rest of this post.

As you may recall from the third post, we implemented the UUID session Id. An authenticated session is uniquely identified by a UUID session Id. Thus, a UUID session Id will be logged with the marker pair if one is available, otherwise the text < Session Id not available > will be logged. Please see the following illustrative examples.

⓵ Logging where a UUID session Id is not yet available, i.e., the request is from an un authenticated session:

INFO:     ... * Request Started < Session Id not available >
INFO:     ... Path: http://192.168.0.16:5000/auth/login?state=0
DEBUG:    ... Attempt to deliver the login page.
DEBUG:    ... Delivering the login page.
INFO:     ... * Request Finished < Session Id not available >
INFO:     ... 192.168.0.2:61019 - "GET /auth/login?state=0 HTTP/1.1" 200

⓶ Logging with a UUID session Id, i.e., the request is from an authenticated session:

INFO:     ... * Request Started f9b96bcdab8b5153c44ca02e0a489c7d
INFO:     ... Path: http://192.168.0.16:5000/admin/me
DEBUG:    ... Returning a valid logged-in user.
INFO:     ... * Request Finished f9b96bcdab8b5153c44ca02e0a489c7d
INFO:     ... 192.168.0.2:61016 - "GET /admin/me HTTP/1.1" 200

💥 Please note the last line in each of the above two examples. It originates from the httptools_impl.py module, specifically the method send on line 466. It is outside of the marker pair, and I did not attempt to have it logged within the marker pair. I believe that, together with the thread Id (not shown) and the path, we can visually trace the originating request.

❸ The changes to the code are quite minimal. Essentially, we’re adding a new YAML logging configuration file, a new Python module, and incorporating logging into other modules. The updated structure of the project is outlined below.

-- Please note, those marked with ★ are updated, and those marked with ☆ are new.

/home/behai/fastapi_learning/
.
├── logger_config.yaml ☆ 
├── main.py ★
├── pyproject.toml ★ 
├── pytest.ini
├── README.md ★
├── src
│ └── fastapi_learning
│     ├── common
│     │ ├── consts.py
│     │ └── queue_logging.py ☆
│     ├── controllers
│     │ ├── admin.py ★
│     │ ├── auth.py ★
│     │ └── __init__.py
│     ├── __init__.py
│     ├── models
│     │ └── employees.py
│     ├── static
│     │ └── styles.css
│     └── templates
│         ├── admin
│         │ └── me.html
│         ├── auth
│         │ ├── home.html
│         │ └── login.html
│         └── base.html
└── tests
    ├── conftest.py
    ├── __init__.py
    ├── integration
    │ ├── test_admin_itgt.py
    │ ├── test_api_itgt.py
    │ └── test_auth_itgt.py
    └── README.md

❹ In this section, we discuss the implementation of non-blocking logging.

I have used RotatingFileHandler for logging via an external YAML configuration file before. I appreciate this approach because we can adjust the logging by tweaking the configuration file without having to refactor the code. I would like to adopt the same approach for this implementation. We will first examine the YAML configuration file and then the associated Python code.

⓵ The full content of the YAML configuration file, logger_config.yaml, is listed below:

version: 1
disable_existing_loggers: False
formatters:
  default:
    (): uvicorn.logging.DefaultFormatter
    format: '{levelprefix} [{asctime}] {thread} {filename} {funcName} {lineno} {message}'
    style: '{'
    datefmt: '%d-%m-%Y %H:%M:%S'
  colours_removed:
    (): uvicorn.logging.DefaultFormatter
    format: '{levelname} [{asctime}] {thread} {filename} {funcName} {lineno} {message}'
    style: '{'
handlers:
  console:
    formatter: default
    class: logging.StreamHandler
    stream: ext://sys.stdout
  rotating_file:
    formatter: colours_removed
    class: logging.handlers.RotatingFileHandler
    filename: ./logs/fastapi_learning.log
    maxBytes: 4096
    backupCount: 5  # Keep 5 old log files    
    encoding: utf-8
  queue_rotating_file:
    class: logging.handlers.QueueHandler
    # queue: fastapi_learning.common.queue_logging.queue_factory
    # listener: fastapi_learning.common.queue_logging.CustomListener
    handlers:
      - rotating_file
loggers:
  # uvicorn.error is also valid. It is the equivalence of root.
  # uvicorn.error: 
  #  level: INFO  
  #  handlers:
  #    - console
  #    - qhand
  #  propagate: no
  fastapi_learning.debug:
    level: DEBUG
    handlers:
      - console
      - queue_rotating_file
    propagate: no
root:
  level: INFO
  handlers:
    - console
    - queue_rotating_file

All aspects of the above configuration file are documented in the official Python logging documentation. We will discuss some implementation details below.

For completeness, we log to both the console (stdout) and log files. The default formatter is for the console, where we retain the default colours. The colours_removed formatter is for the log file. Without removing the text colours, log files will have control codes written in place of text colours. For example, ^[[32mINFO^[[0m:, where ^[ denotes the ESC control character, whose ASCII code is 27. Please refer to this screenshot for the exact printout. Please further note the following:
- Log attributes such as levelname, thread etc., are covered by the official documentation on LogRecord attributes.
- I cannot find official documentation for levelprefix. However, it is mentioned and used in discussions about Python logging across the internet. I have observed that levelprefix and levelname both print out the text logging level such as DEBUG, INFO etc.; levelprefix prints out with colours while levelname does not.
- For the rather unusual entry (): uvicorn.logging.DefaultFormatter, please refer to the official documentation on User-defined objects.

For the rotating_file handler, please note:
- 💥 For the filename: ./logs/fastapi_learning.log property, we configured the log files to be written to the ./logs sub-directory. This is not something supported out of the box. We will discuss this property in a later section.
- The values of maxBytes and backupCount are deliberately set low for debugging purposes.
For the queue_rotating_file handler, please note:
- The YAML configuration is a copy of the snippet from the Python documentation on Configuring QueueHandler and QueueListener. We left both the queue key and the listener out. We use the standard implementations as documented.
  
  💥 It is the application’s responsibility to start and stop any QueueListener instances in use. We will discuss this in a later section.
- The QueueHandler is a special case: it has its own handlers, in this case, it is the rotating_file.
  
  It is worth noting that the structure of the queue_rotating_file is very similar to the non-blocking logging example presented in an earlier section.

We configure only one logger: fastapi_learning.debug. Its log messages are consumed by both the console and the queue_rotating_file handlers.

⓶ We will now discuss the new Python module, common/queue_logging.py, which works in conjunction with the above YAML configuration file. This is a straightforward module, comprising less than 90 lines.

In the rotating_file handler section, we mentioned that the ./logs sub-directory is not supported out of the box. As it is configured, it is a sub-directory immediately under the directory where the main.py module resides.

We need to manage this sub-directory ourselves: we must ensure that this sub-directory exists before the YAML configuration file is loaded, otherwise Python will raise an exception.

💥 Therefore, passing the YAML configuration file in the command line, such as uvicorn main:app --log-config=logger_config.yaml, is not possible, or more accurately, I don’t know how to enable that. I tried and failed to run my code before the configuration file was loaded.

The next best option is to load the configuration file ourselves: we have full control. Please refer to the function prepare_logging_and_start_listeners, lines 57-65, in the new common/queue_logging.py module. The actual code consists of only 4 lines:
- Always create the ./logs sub-directory.
- We then load the configuration file and pass it to logging.config.dictConfig.
💥 Please note that, due to the above implementation, the loggers have not been configured yet when the application starts up. The startup messages below are not written to the current log file. The default existing loggers are still in use at this point.
```
INFO:     Started server process [29204]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:5000 (Press CTRL+C to quit)
```

In a previous section, we mentioned that it is the application’s responsibility to start and stop any QueueListener instances. The last part of the function prepare_logging_and_start_listeners, lines 67 to 69, implements the code to get the listeners to start listening.

We retrieve all listener instances and start each one. The private helper function __retrieve_queue_listeners should be self-explanatory.

We currently have only one listener instance, but in the future, we might configure more, such as for sending out emails. In such a case, we would need to update only the private function __retrieve_queue_listeners.

Before the application shuts down, it should call logging_stop_listeners to get the listeners to stop listening.

And finally, the RequestLoggingMiddleware class implements the request logging marker pair that was mentioned in an in an earlier section.

⓷ The changes in main.py are straightforward and should be self-explanatory.

For information on the new lifespan function, please refer to the official FastAPI documentation on Lifespan.

⓸ Having implemented all of the above, we are finally able to incorporate logging into the methods in the two modules, controllers/auth.py and controllers/admin.py.

❺ Some of the referenced official documentation has already been mentioned throughout the discussion. However, I believe it is rather essential, so I would like to reiterate it in this separate section. I have personally read through all the Python documentation on logging. They include:

The last one is particularly interesting. Python logging is indeed a powerful library.

❻ When I first started this logging process, I thought it was going to be simple. However, it took a bit longer than I anticipated. I encountered some problems, but I managed to find solutions. The code presented in this post has gone through several refactorings. This is the first time I have explored Python logging in detail. I learned a lot during the writing of this post. There is room for improvement, but overall, I think the implementation is acceptable.

Thank you for reading. I hope you find the information in this post useful. Stay safe, as always.

✿✿✿

Feature image source:

🐍 Index of the Complete Series.

Python FastAPI: Integrating OAuth2 Security with the Application's Own Authentication Process

Be Hai Nguyen — Tue, 14 May 2024 00:59:49 +0000

In the first post, we explore some aspects of OAuth2 authentication, focusing on the /token path as illustrated in an example from the Simple OAuth2 with Password and Bearer section of the Tutorial - User Guide Security. In this subsequent post, we implement our own custom preliminary login process, leveraging the /token path. This means that both the Swagger UI Authorize button and our application's login button utilise the same server code.

🐍 Index of the Complete Series.

🚀 Please note, complete code for this post can be downloaded from GitHub with:

git clone -b v0.2.0 https://github.com/behai-nguyen/fastapi_learning.git

During my research on “FastAPI application custom login process”, I've encountered implementations where there are two endpoints for handling authentication: the /token endpoint, as discussed in the Tutorial - User Guide Security section, and the application's own login endpoint.

💥 This approach doesn't seem right to me. In my view, the /token endpoint should serve as the application's login endpoint as well. In this post, we introduce a preliminary custom login process with the endpoint being the /token endpoint.

The code developed in this post maintains the one-module application structure from the original example. However, we've added a login HTML page and organised the project directory structure to prepare for further code changes as we progress. The updated project layout is listed below.

-- Please note, those marked with ★ are updated, and those marked with ☆ are new.

/home/behai/fastapi_learning/
├── main.py ★
├── pyproject.toml
├── README.md ★
└── src ☆
    └── fastapi_learning
        ├── static
        │ └── styles.css
        └── templates
            ├── auth
            │ └── login.html
            └── base.html

Changes to main.py include the following:

⓵ Lines 21 to 23: Added required imports to support HTML output.

⓶ Lines 25 to 45: Completely refactored the fake_users_db database to slowly match the test employees MySQL database by Oracle Corporation, as utilised in the SQLAlchemy database wrapper component FastAPI example, and Update the employees table, adding new fields email and password.

⓷ Lines 49 to 50: Added initialisation code to prepare support for HTML template processing.

⓸ Lines 66 to 72: Refactored models to align with the changes in fake_users_db.

⓹ Lines 95 to 100: Refactored the get_current_active_user(...) method to cease checking the disabled attribute of the User model, as this attribute has been removed from the model.

⓺ Lines 120 to 123: Implemented the new /login endpoint, which simply returns the login HTML page.

🚀 Note that the endpoint code for the /token path, specifically the method async def login(form_data: Annotated[OAuth2PasswordRequestForm, Depends()]):, remains unchanged.

💥 Regarding the HTML login page, please take note of the following points:

...
    <form method="POST" action="/token" id="loginForm">
        <h1 class="h3 mb-3 fw-normal">Please login</h1>

        {% if message is defined %}
            <h2>{{ message }}</h2>
        {% endif %}

        <div>
            <label for="username">Email address:</label>
            <input type="email" class="form-control" id="username" name="username" placeholder="name@example.com" required value="">
        </div>

        <div>
            <label for="password">Password:</label>
            <input type="password" class="form-control" id="password" name="password" placeholder="Password" required value="">
        </div>
        <button type="submit">Login</button>
    </form>
...

⓵ The action of the login form is directed to the /token path. In other words, when the login form is submitted, it sends a POST login request to the same endpoint used by the Authorize button on the Swagger UI page.

⓶ The names of the two login fields are username and password. This requirement is specified in the tutorial in the section titled Simple OAuth2 with Password and Bearer:

OAuth2 specifies that when using the "password flow" (that we are using) the client/user must send a username and password fields as form data.

🚀 Our application's login process now shares the same server code as the Swagger UI login process. We have two separate “clients”:

http://0.0.0.0:port/docs: The Swagger UI client page.
http://0.0.0.0:port/login: Our own custom login page.

On Ubuntu 22.10, run the application with the command:

(venv) ...:~/fastapi_learning$ venv/bin/uvicorn main:app --host 0.0.0.0 --port 5000

When accessing the Swagger UI page on Windows 10 at http://192.168.0.16:5000/docs, we encounter a familiar page:

The GET /login path should simply return the login HTML page, while the remaining paths should function as discussed in the first post.

When accessing the application's login page on Windows 10 at http://192.168.0.16:5000/login, we are presented with our custom login page:

Upon logging in using one of the following credentials: behai_nguyen@hotmail.com/password or pranav.furedi.10198@gmail.com/password, we should receive a successful JSON response, as depicted in the screenshot below:

When attempting to log in using an invalid credential, we should receive an HTTP 400 response, which indeed occurs, as seen in the screenshots below:

In the current implementation, the login process is incomplete, but it serves as an appropriate preliminary step nonetheless. We will conclude this post here as I don't want to make it too long... In the next post or so, we will implement stateful sessions and extend OAuth2PasswordBearer to maintain authenticated sessions. This means that after a successful login, users can access protected application routes until they choose to log out.

Thank you for reading. I hope you find the information in this post useful. Stay safe, as always.

✿✿✿

Feature image source:

🐍 Index of the Complete Series.

Python: A SQLAlchemy Wrapper Component That Works With Both Flask and FastAPI Frameworks

Be Hai Nguyen — Sat, 04 May 2024 03:46:58 +0000

In late 2022, I developed a database wrapper component for SQLAlchemy. Initially designed for use with the Flask framework, it was discovered that this component also seamlessly integrates with the FastAPI framework. In this post, I will describe this component and provide examples of how it is used within both frameworks.

This component is based on a concept I previously implemented using Delphi and later PHP. Essentially, it consists of base classes representing database tables, equipped with the ability to interact with databases and implement generic functionalities for CRUD operations.

While learning the Flask framework, I found the conventional approach of accessing the database layer through the Flask application instance uncomfortable. In my previous projects, the database layer has always remained independent of other layers. The business layer ensures data validity and then delegates CRUD operations to the database layer. The UI layer, whether a desktop application or web client, communicates solely with the business layer, never directly accessing the database layer.

In SQLAlchemy, models representing database tables typically subclass sqlalchemy.orm.DeclarativeBase (this class supersedes the sqlalchemy.orm.declarative_base function). Accordingly, the abstract base class in this database wrapper component is a sqlalchemy.orm.DeclarativeBase subclass, accompanied by another custom base class providing additional dunder methods.

Further subclasses of this abstract base class implement additional functionalities. Application models inherit from either the ReadOnlyTable or the WriteCapableTable base classes.

Application models are required to implement their own specific database reading methods. For example, selecting all customers with the surname Nguyễn.

The Database class is responsible for establishing connections to the target database. Once the database connection is established, application models can interact with the target database.

🚀 The full documentation can be found at https://bh-database.readthedocs.io/en/latest/.

Next, we will explore some examples. 💥 The first two are simple, single-module web server applications where the web layer directly accesses the database layer. Although not ideal, it simplifies usage illustration.

The latter two examples include complete business layers, where submitted data is validated before being passed to the database layer for CRUD operations.

❶ example.py: A Simple Single-Module Flask Application.

● Windows 10: F:\bh_database\examples\flaskr\example.py

● Ubuntu 22.10: /home/behai/bh_database/examples/flaskr/example.py

from sqlalchemy import (
    Column,
    Integer,
    Date,
    String,
)

import flask

from bh_database.core import Database
from bh_database.base_table import WriteCapableTable

from bh_apistatus.result_status import ResultStatus

SQLALCHEMY_DATABASE_SCHEMA = 'employees'
SQLALCHEMY_DATABASE_URI = 'mysql+mysqlconnector://root:pcb.2176310315865259@localhost:3306/employees'
# Enable this for PostgreSQL.
# SQLALCHEMY_DATABASE_URI = 'postgresql+psycopg2://postgres:pcb.2176310315865259@localhost/employees'

class Employees(WriteCapableTable):
    __tablename__ = 'employees'

    emp_no = Column(Integer, primary_key=True)
    birth_date = Column(Date, nullable=False)
    first_name = Column(String(14), nullable=False)
    last_name = Column(String(16), nullable=False)
    gender = Column(String(1), nullable=False)
    hire_date = Column(Date, nullable=False)

    def select_by_partial_last_name_and_first_name(self, 
            last_name: str, first_name: str) -> ResultStatus:

        return self.run_stored_proc('get_employees', [last_name, first_name], True)

def create_app(config=None):
    """Construct the core application."""

    app = flask.Flask(__name__, instance_relative_config=False)

    init_extensions(app)

    init_app_database(app)

    return app

def init_extensions(app):
    app.url_map.strict_slashes = False

def init_app_database(app):    
    Database.disconnect()
    Database.connect(SQLALCHEMY_DATABASE_URI, SQLALCHEMY_DATABASE_SCHEMA)

app = create_app()

@app.get('/employees/search/<last_name>/<first_name>')
def search_employees(last_name: str, first_name: str) -> dict:
    """ last_name and first_name are partial using %.

    An example of a valid route: http://localhost:5000/employees/search/%nas%/%An
    """

    return Employees() \
        .select_by_partial_last_name_and_first_name(last_name, first_name) \
        .as_dict()

if __name__ == '__main__':  
   app.run()

To execute the example.py application:

▶️<code>Windows 10:</code> (venv) F:\bh_database\examples\flaskr>venv\Scripts\flask.exe --app example run --host 0.0.0.0 --port 5000
▶️<code>Ubuntu 22.10:</code> (venv) behai@hp-pavilion-15:~/bh_database/examples/flaskr$ venv/bin/flask --app example run --host 0.0.0.0 --port 5000

Accessing the example.py application running locally from Windows 10:

http://localhost:5000/employees/search/%nas%/%An

Accessing the example.py application running on Ubuntu 22.10 from Windows 10:

http://192.168.0.16:5000/employees/search/%nas%/%An

❷ example.py: A Simple Single-Module FastAPI Application.

● Windows 10: F:\bh_database\examples\fastapir\example.py

● Ubuntu 22.10: /home/behai/bh_database/examples/fastapir/example.py

from sqlalchemy import (
    Column,
    Integer,
    Date,
    String,
)

from fastapi import FastAPI
from fastapi.responses import JSONResponse

from bh_database.core import Database
from bh_database.base_table import WriteCapableTable

from bh_apistatus.result_status import ResultStatus

SQLALCHEMY_DATABASE_SCHEMA = 'employees'
SQLALCHEMY_DATABASE_URI = 'mysql+mysqlconnector://root:pcb.2176310315865259@localhost:3306/employees'
# Enable this for PostgreSQL.
# SQLALCHEMY_DATABASE_URI = 'postgresql+psycopg2://postgres:pcb.2176310315865259@localhost/employees'

class Employees(WriteCapableTable):
    __tablename__ = 'employees'

    emp_no = Column(Integer, primary_key=True)
    birth_date = Column(Date, nullable=False)
    first_name = Column(String(14), nullable=False)
    last_name = Column(String(16), nullable=False)
    gender = Column(String(1), nullable=False)
    hire_date = Column(Date, nullable=False)

    def select_by_partial_last_name_and_first_name(self, 
            last_name: str, first_name: str) -> ResultStatus:

        return self.run_stored_proc('get_employees', [last_name, first_name], True)

app = FastAPI()

Database.disconnect()
Database.connect(SQLALCHEMY_DATABASE_URI, SQLALCHEMY_DATABASE_SCHEMA)

@app.get("/employees/search/{last_name}/{first_name}", response_class=JSONResponse)
async def search_employees(last_name: str, first_name: str):
    """ last_name and first_name are partial using %.

    An example of a valid route: http://localhost:5000/employees/search/%nas%/%An
    """

    return Employees() \
        .select_by_partial_last_name_and_first_name(last_name, first_name) \
        .as_dict()

To execute the example.py application:

▶️<code>Windows 10:</code> (venv) F:\bh_database\examples\fastapir>venv\Scripts\uvicorn.exe example:app --host 0.0.0.0 --port 5000
▶️<code>Ubuntu 22.10:</code> (venv) behai@hp-pavilion-15:~/bh_database/examples/fastapir$ venv/bin/uvicorn example:app --host 0.0.0.0 --port 5000

Accessing the example.py application running locally from Windows 10:

http://localhost:5000/employees/search/%nas%/%An

Accessing the example.py application running on Ubuntu 22.10 from Windows 10:

http://192.168.0.16:5000/employees/search/%nas%/%An

❸ A more comprehensive Flask application: a fully documented web server example with CRUD operations.

Please refer to https://github.com/behai-nguyen/bh_database/tree/main/examples/flaskr for the full source code, instructions on setting up the environment, installing packages, running tests, and finally running the application.

The layout of the example project is as follows:

/home/behai/bh_database/examples/flaskr
├── app.py
├── .env
├── pyproject.toml
├── pytest.ini
├── README.md
├── src
│ └── flaskr
│     ├── business
│     │ ├── app_business.py
│     │ ├── base_business.py
│     │ ├── base_validation.py
│     │ ├── employees_mgr.py
│     │ └── employees_validation.py
│     ├── config.py
│     ├── controllers
│     │ └── employees_admin.py
│     ├── __init__.py
│     ├── models
│     │ └── employees.py
│     ├── static
│     │ └── styles.css
│     └── templates
│         ├── admin
│         │ ├── emp_edit.html
│         │ ├── emp_search.html
│         │ └── emp_search_result.html
│         └── base.html
└── tests
    ├── business
    │ └── test_employees_mgr.py
    ├── conftest.py
    ├── __init__.py
    ├── integration
    │ └── test_employees_itgt.py
    └── unit
        └── test_employees.py

❹ A more comprehensive FastAPI application: a fully documented web server example with CRUD operations.

Please refer to https://github.com/behai-nguyen/bh_database/tree/main/examples/fastapir for the full source code, instructions on setting up the environment, installing packages, running tests, and finally running the application.

The layout of the example project is as follows:

/home/behai/bh_database/examples/fastapir
├── .env
├── main.py
├── pyproject.toml
├── pytest.ini
├── README.md
├── src
│ └── fastapir
│     ├── business
│     │ ├── app_business.py
│     │ ├── base_business.py
│     │ ├── base_validation.py
│     │ ├── employees_mgr.py
│     │ └── employees_validation.py
│     ├── config.py
│     ├── controllers
│     │ ├── employees_admin.py
│     │ └── __init__.py
│     ├── __init__.py
│     ├── models
│     │ └── employees.py
│     ├── static
│     │ └── styles.css
│     └── templates
│         ├── admin
│         │ ├── emp_edit.html
│         │ ├── emp_search.html
│         │ └── emp_search_result.html
│         └── base.html
└── tests
    ├── business
    │ └── test_employees_mgr.py
    ├── conftest.py
    ├── __init__.py
    ├── integration
    │ └── test_employees_itgt.py
    └── unit
        └── test_employees.py

💥 Except for the framework-specific layer code, the remaining code in these two examples is very similar.

Let's briefly discuss their similarities:

/models and /business code are identical. They could be shared across both examples, but I prefer to keep each example self-contained.
/tests/unit and /tests/business code are identical.

And there are differences in the following areas:

/controllers: This is the web layer, which is framework-specific, so understandably they are different.
/tests/integration: The sole difference is framework-specific: how the HTTP response value is extracted:
- Flask: response.get_data(as_text=True)
- FastAPI: response.text
/tests/conftest.py: This file is framework-dependent. Both modules return the same fixtures, but the code has nothing in common.
/templates/base.html: There is one difference:
- Flask: `<link rel="stylesheet" href="{{ url_for('static', filename='styles.css') }}">`
- FastAPI: `<link rel="stylesheet" href="{{ url_for('static', path='/styles.css') }}">`
That is, Flask uses filename, while FastAPI uses path.

The /controllers layer is thin in the sense that the code is fairly short; it simply takes the client-submitted data and passes it to the business layer to handle the work. The business layer then forwards the validated data to the database layer, and so on. The differences between the two implementations are minor.

It has been an interesting exercise developing this wrapper component. The fact that it seamlessly integrates with the FastAPI framework is just a bonus for me; I didn't plan for it since I hadn't learned FastAPI at the time. I hope you find this post useful. Thank you for reading, and stay safe as always.

✿✿✿

Feature image source:

Rust: Actix-web -- Async Functions as Middlewares

Be Hai Nguyen — Wed, 20 Mar 2024 12:26:10 +0000

In the tenth post of our actix-web learning application, we added an ad hoc middleware. In this post, with the assistance of the actix-web-lab crate, we will refactor this ad hoc middleware into a standalone async function to enhance the overall code readability.

🦀 Index of the Complete Series.

🚀 Please note, complete code for this post can be downloaded from GitHub with:

git clone -b v0.13.0 https://github.com/behai-nguyen/rust_web_01.git

The actix-web learning application mentioned above has been discussed in the twelve previous posts. The index of the complete series can be found here.

The code we're developing in this post is a continuation of the code from the twelfth post. 🚀 To get the code of this twelfth post, please use the following command:

git clone -b v0.12.0 https://github.com/behai-nguyen/rust_web_01.git

-- Note the tag v0.12.0.

While this post continues from previous posts in this series, it can be read in conjunction with only the tenth post, focusing particularly on the section titled Code Updated in the src/lib.rs Module.

❶ For this post, no new modules are introduced. Instead, we will update existing modules and files. The layout chart below displays the updated files and modules, with those marked with ★ indicating the ones that have been updated.

.
├── Cargo.toml ★
├── ...
├── README.md ★
└── src
  ├── lib.rs ★
  └── ...

❷ An update to the Cargo.toml file:

...
[dependencies]
...
actix-web-lab = "0.20.2"

We added the new crate actix-web-lab. This crate is:

In-progress extractors and middleware for Actix Web.

This crate provides mechanisms for implementing middlewares as standalone async functions, rather than using actix-web's wrap_fn.

According to the documentation, the actix-web-lab crate is essentially experimental. Functionalities implemented in this crate might eventually be integrated into the actix-web crate. In such a case, we would need to update our code.

❸ Refactor an existing ad hoc middleware out of wrap_fn.

As mentioned at the beginning, this post should be read in conjunction with the tenth post, where we introduced this ad hoc middleware. The description of this simple middleware functionality is found in the section Code Updated in the src/lib.rs Module of the tenth post.

Below, we reprint the code of this ad hoc middleware:

            //
            // This ad hoc middleware looks for the updated access token String attachment in 
            // the request extension, if there is one, extracts it and sends it to the client 
            // via both the ``authorization`` header and cookie.
            //
            .wrap_fn(|req, srv| {
                let mut updated_access_token: Option<String> = None;

                // Get set in src/auth_middleware.rs's 
                // fn update_and_set_updated_token(request: &ServiceRequest, token_status: TokenStatus).
                if let Some(token) = req.extensions_mut().get::<String>() {
                    updated_access_token = Some(token.to_string());
                }

                srv.call(req).map(move |mut res| {

                    if updated_access_token.is_some() {
                        let token = updated_access_token.unwrap();
                        res.as_mut().unwrap().headers_mut().append(
                            header::AUTHORIZATION, 
                            header::HeaderValue::from_str(token.as_str()).expect(TOKEN_STR_JWT_MSG)
                        );

                        let _ = res.as_mut().unwrap().response_mut().add_cookie(
                            &build_authorization_cookie(&token));
                    };

                    res
                })
            })

It's not particularly lengthy, but its inclusion in the application instance construction process makes it difficult to read. While closures can call functions, refactoring this implementation into a standalone function isn't feasible. This is because the function would require access to the parameter srv, which in this case refers to the AppRouting struct. Please refer to the screenshot below for clarification:

The AppRouting struct is located in the private module actix-web/src/app_service.rs, which means we don't have direct access to it. I attempted to refactor it into a standalone function but encountered difficulties. Someone else had also attempted it before me and faced similar issues.

Please refer to the GitHub issue titled wrap_fn &AppRouting should use Arc<AppRouting> #2681 for more details. This reply suggests using the actix-web-lab crate.

I believe I've come across this crate before, particularly the function actix_web_lab::middleware::from_fn, but it didn't register with me at the time.

Drawing from the official example actix-web-lab/actix-web-lab/examples/from_fn.rs and compiler suggestions, I've successfully refactored the ad hoc middleware mentioned above into the standalone async function async fn update_return_jwt<B>(req: ServiceRequest, next: Next<B>) -> Result<ServiceResponse<B>, Error>. The screenshot below, taken from Visual Studio Code with the Rust-Analyzer plugin, displays the full source code and variable types:

Compared to the original ad hoc middleware, the code is virtually unchanged. It's worth noting that this final version is the result of my sixth or seventh attempt; without the compiler suggestions, I would not have been able to complete it. We register it with the application instance using only a single line, as per the documentation:

            .wrap(from_fn(update_return_jwt))

❹ Other minor refactorings include optimising the application instance builder code for brevity. Specifically, I've moved the code to create the CORS instance to the standalone function fn cors_config(config: &config::Config) -> Cors, and the code to create the session store to the standalone async function async fn config_session_store() -> (actix_web:🍪:Key, RedisSessionStore).

Currently, the src/lib.rs module is less than 250 lines long, housing 7 helper functions that are completely unrelated. I find it still very manageable. The code responsible for creating the server instance and the application instance, encapsulated in the function pub async fn run(listener: TcpListener) -> Result<Server, std::io::Error>, remains around 60 lines. Although I anticipate it will grow a bit more as we add more functionalities, I don't foresee it becoming overly lengthy.

❺ I am happy to have learned something new about actix-web. And I hope you find the information useful. Thank you for reading. And stay safe, as always.

✿✿✿

Feature image source:

🦀 Index of the Complete Series.

Rust: Actix-web Daily Logging -- Fix Local Offset, Apply Event Filtering

Be Hai Nguyen — Mon, 18 Mar 2024 04:46:45 +0000

In the last post of our actix-web learning application, we identified two problems. First, there is an issue with calculating the UTC time offset on Ubuntu 22.10, as described in the section 💥 Issue with calculating UTC time offset on Ubuntu 22.10. Secondly, loggings from other crates that match the logging configuration are also written onto log files, as mentioned in the Concluding Remarks section. We should be able to configure what gets logged. We will address both of these issues in this post.

🦀 Index of the Complete Series.

🚀 Please note, complete code for this post can be downloaded from GitHub with:

git clone -b v0.12.0 https://github.com/behai-nguyen/rust_web_01.git

The actix-web learning application mentioned above has been discussed in the eleven previous posts. The index of the complete series can be found here.

The code we're developing in this post is a continuation of the code from the eleventh post. 🚀 To get the code of this eleventh post, please use the following command:

git clone -b v0.11.0 https://github.com/behai-nguyen/rust_web_01.git

-- Note the tag v0.11.0.

While this post continues from previous posts in this series, it can be read in conjunction with only the eleventh post.

❶ For this post, no new modules are introduced. Instead, we will update some existing modules and files. The layout chart below shows the updated files and modules, with those marked with ★ indicating the updated ones.

.
├── .env ★
├── Cargo.toml ★
├── ...
├── README.md ★
├── src
│ ├── helper
│ │ ├── app_logger.rs ★
│ │ └── ...
│ ├── main.rs ★
│ └── ...
└── tests
    ├── common.rs ★
    └── ...

🦀 In the context of this post, our focus is solely on the RUST_LOG entry in the .env file, which we will discuss in a later section.

❷ Updates to the Cargo.toml file:

...
[dependencies]
...
time-tz = {version = "2.0", features = ["system"]}
tracing-subscriber = {version = "0.3", features = ["fmt", "std", "local-time", "time", "env-filter"]}

We added the new crate time-tz, and for tracing-subscriber, we added the crate feature env-filter. We will discuss these additions in later sections.

❸ Resolve the issue with calculating the UTC time offset to ensure reliable functionality on both Ubuntu 22.10 and Windows 10.

⓵ As mentioned in the last post:

After extensive searching, I came across this GitHub issue, Document #293 in local-offset feature description #297. It appears that even after three years, this issue remains unresolved.

Document #293 is dated December 19, 2020. Additionally, there are other relevant documents that I did not come across during my previous “extensive searching”:

November 25, 2020 -- Time 0.2.23 fails to determine local offset #296.
November 2, 2021 -- Better solution for getting local offset on unix #380.
Dec 5, 2019 -- tzdb support #193.

This reply posted on February 13, 2022 mentions the crate time-tz, which resolves the previously mentioned issue.

The parameter utc_offset: time::UtcOffset was removed from the function pub fn init_app_logger() -> WorkerGuard, and the offset calculation is now carried out internally:

    let timer = OffsetTime::new(
        localtime.offset(),
        format_description!("[year]-[month]-[day] [hour]:[minute]:[second]"),
    );

⓶ Offset calculations were accordingly removed from both the function async fn main() -> Result<(), std::io::Error> in the module src/main.rs, and the function pub async fn spawn_app() -> TestApp in the module tests/common.rs.

❸ Configuration to determine what get logged.

We use tracing_subscriber's filter::EnvFilter struct to filter which events are logged. This functionality requires the crate feature env-filter, as described above.

Event filtering is configured via the environment variable RUST_LOG. Its value can be much more sophisticated than simply trace, debug, info, warn and error. The documentation in the section Enabling logging of the env_logger crate describes the syntax of RUST_LOG with plenty of informative examples.

⓵ Implementing event filtering for the function pub fn init_app_logger() -> WorkerGuard:

    let filter_layer = EnvFilter::try_from_default_env()
        .or_else(|_| EnvFilter::try_new("debug"))
        .unwrap();

    let subscriber = tracing_subscriber::registry()
        .with(
            ...
                .with_filter(filter_layer)
        );

Please note that the code to convert the value of RUST_LOG to tracing::Level has also been removed.

For further documentation, please refer to Filtering Events with Environment Variables. The code above is from the section Composing Layers of the mentioned documentation page. As for the two functions being called, please see:

● pub fn try_from_default_env() -> Result<Self, FromEnvError>.

● pub fn try_new<S: AsRef<str>>(dirs: S) -> Result<Self, ParseError>.

And finally, the line:

            ...
                .with_filter(filter_layer)

is from the trait tracing_subscriber::layer::Layer, which is a “a composable handler for tracing events.”

⓶ As for the value of RUST_LOG, there are three cases that do not behave as I initially assumed:

The first two cases are RUST_LOG=xxxx and RUST_LOG=, where nothing gets logged. I had assumed that this error handling would default the logging event to debug:

        .or_else(|_| EnvFilter::try_new("debug"))

I attempted several times to default them to debug, but unfortunately, I was unsuccessful.

The third case is RUST_LOG, where only the RUST_LOG variable name is present in the .env file without any value assigned to it. Based on the above two instances, I expected that nothing would be logged. However, it defaults to debug!

Please note that for the next example discussion, it's important to keep in mind that the Cargo.toml file contains the following declaration, where learn_actix_web is defined:

[[bin]]
path = "src/main.rs"
name = "learn_actix_web"

Examples of some valid values:

RUST_LOG=off,learn_actix_web=debug -- Only debug logging events from the learn_actix_web crate are logged. All logging events from other crates are ignored.
RUST_LOG=off,learn_actix_web=info -- Only info logging events from the application are logged. If there are no info events in the application, nothing gets logged and the log files remain empty.
RUST_LOG=off,learn_actix_web=debug,actix_server=info -- Only debug events from the application and info events from the actix_server crate are logged.
RUST_LOG=off,learn_actix_web::middleware=debug -- Only debug events from the src/middleware.rs module of the application are logged. This middleware is triggered when accessing the GET route http://0.0.0.0:5000/helloemployee/{partial last name}/{partial first name} from an authenticated session.

A further illustration for example 4 above: Log in and click on the last button as shown in the screenshot below:

https://behainguyen.files.wordpress.com/2024/03/102-01.png

The current log file should contain the following three new lines:

2024-03-18 00:51:15 DEBUG learn_actix_web::middleware: Hi from start. You requested: /helloemployee/%chi/%ak
2024-03-18 00:51:15 DEBUG learn_actix_web::middleware: Middleware. last name: %chi, first name: %ak.
2024-03-18 00:51:15 DEBUG learn_actix_web::middleware: Hi from response -- some employees found.

This finer control demonstrates the power, utility, and helpfulness of tracking an intermittent bug that is not reproducible on staging and development environments. By enabling debug and tracing logging for specific modules, we can effectively troubleshoot such issues.

❹ Logging events should be grouped within the unique ID of the authenticated session.

For each authenticated session, there is a third-party session ID. I have conducted some studies on this cookie, and its value seems to change after each request. For further discussion of this ID under HTTPS, please refer to this discussion.

My initial plan is to group logging for each request under the value of this ID. For example:

** 2024-03-18 00:51:15 DEBUG learn_actix_web::middleware: {value of id} entered.
2024-03-18 00:51:15 DEBUG learn_actix_web::middleware: Hi from start. You requested: /helloemployee/%chi/%ak
...
** 2024-03-18 00:51:15 DEBUG learn_actix_web::middleware: {value of id} exited.

I have not yet determined how to achieve this; further study is required.

❺ We have concluded this post. I'm pleased to have resolved the offset issue and to have implemented logging in a more effective manner.

I hope you find the information useful. Thank you for reading. And stay safe, as always.

✿✿✿

Feature image source:

🦀 Index of the Complete Series.

Rust: Actix-web and Daily Logging

Be Hai Nguyen — Wed, 13 Mar 2024 05:58:20 +0000

Currently, our actix-web learning application simply prints debug information to the console using the println! macro. In this post, we will implement proper non-blocking daily logging to files. Daily logging entails rotating to a new log file each day. Non-blocking refers to having a dedicated thread for file writing operations. We will utilise the tracing, tracing-appender, and tracing-subscriber crates for our logging implementation.

🦀 Index of the Complete Series.

🚀 Please note, complete code for this post can be downloaded from GitHub with:

git clone -b v0.11.0 https://github.com/behai-nguyen/rust_web_01.git

The actix-web learning application mentioned above has been discussed in the following ten previous posts:

The code we're developing in this post is a continuation of the code from the tenth post above. 🚀 To get the code of this tenth post, please use the following command:

git clone -b v0.10.0 https://github.com/behai-nguyen/rust_web_01.git

-- Note the tag v0.10.0.

While this post continues from previous posts in this series, it can also be read independently. The logging module developed herein can be used in other projects without modification.

❶ For this post, we introduce a new module src/helper/app_logger.rs, and some other modules and files are updated. The project layout remains the same as in the last post. The layout chart below shows the affected files and modules:

-- Please note that files marked with ★ are updated, and src/helper/app_logger.rs is marked with ☆, as it is the only new module.

.
├── .env ★
├── Cargo.toml ★
├── ...
├── README.md ★
├── src
│ ├── auth_middleware.rs ★
│ ├── database.rs ★
│ ├── helper
│ │ ├── app_logger.rs ☆
│ │ └── ...
│ ├── helper.rs ★
│ ├── main.rs ★
│ ├── middleware.rs ★
│ └── ...
└── tests
    ├── common.rs ★
    └── ...

❷ An update to the .env file: a new entry has been added:

RUST_LOG=debug

The value of RUST_LOG is translated into tracing::Level. Valid values include trace, debug, info, warn and error. Any other values are invalid and will default to Level::DEBUG.

❸ Updates to the Cargo.toml file: as expected, the new crates are added to the [dependencies] section.

...
[dependencies]
...
tracing = "0.1"
tracing-appender = "0.2"
tracing-subscriber = {version = "0.3", features = ["fmt", "std", "local-time", "time"]}

❹ 💥 Issue with calculating UTC time offset on Ubuntu 22.10.

In the new code added for this post, we need to calculate the UTC time offset to obtain local time. The following code works on Windows 10:

use time::UtcOffset;

let offset = UtcOffset::current_local_offset().unwrap();

However, on Ubuntu 22.10, it doesn't always function as expected. Sometimes, it raises the error IndeterminateOffset. The inconsistency in its behavior makes it challenging to identify a clear pattern of when it works and when it doesn't.

After extensive searching, I came across this GitHub issue, Document #293 in local-offset feature description #297. It appears that even after three years, this issue remains unresolved.

This complication adds an extra layer of difficulty in ensuring both the code and integration tests function properly. In the subsequent sections of this post, when discussing the code, we'll refer back to this issue when relevant. Please keep this in mind.

❺ The src/helper/app_logger.rs module has been designed to be easily copied into other projects, provided that the Cargo.toml file includes the required crates discussed earlier.

This module contains only a single public function, pub fn init_app_logger(utc_offset: time::UtcOffset) -> WorkerGuard, which the application calls to set up the log. Please refer to the notes and documentation within this module while reading the code.

Originally, the utc_offset: time::UtcOffset parameter was not present. However, due to the issue mentioned in 💥 Issue with calculating UTC time offset on Ubuntu 22.10, the code was refactored to include this parameter, offering a bit more flexibility.

⓵ Setting up the daily log files.

    let log_appender = RollingFileAppender::builder()
        .rotation(Rotation::DAILY) // Daily log file.
        .filename_suffix("log") // log file names will be suffixed with `.log`
        .build("./log") // try to build an appender that stores log files in `/var/log`
        .expect("Initialising rolling file appender failed");

To set up the daily log files, we begin by calling the pub fn builder() -> Builder function.

We specify DAILY rotation to generate daily log files. However, it's important to note that according to the documentation, the file names are appended with the current date in UTC. Since I'm in the Australian Eastern Standard Time (AEST) zone, which is 10-11 hours ahead of UTC, there were instances where my log file names were created with dates from the previous day.

To give log files the .log extension, we use the method pub fn filename_suffix(self, suffix: impl Into<String>) -> Self.

The format of the daily log file names follows the pattern YYYY-MM-DD.log, for example, 2024-03-10.log.

We then invoke the method pub fn build( &self, directory: impl AsRef<Path>) -> Result<RollingFileAppender, InitError> to specify the location of the log files within the log/ sub-directory relative to where the application is executed. For instance:

▶️<code>Windows 10:</code> F:\rust\actix_web>target\debug\learn_actix_web.exe
▶️<code>Ubuntu 22.10:</code> behai@hp-pavilion-15:~/rust/actix_web$ /home/behai/rust/actix_web/target/debug/learn_actix_web

This results in the log files being stored at F:\rust\actix_web\log</code> and /home/behai/rust/actix_web/target/debug/learn_actix_web/log/ respectively.

⓶ We create a non-blocking writer thread using the following code:

    let (non_blocking_appender, log_guard) = tracing_appender::non_blocking(log_appender);

This is the documentation section for the function tracing_appender::non_blocking. For more detailed documentation, refer to the tracing_appender::non_blocking module. Please note the following:

This function returns a tuple of NonBlocking and WorkerGuard. NonBlocking implements MakeWriter which integrates with tracing_subscriber. WorkerGuard is a drop guard that is responsible for flushing any remaining logs when the program terminates.

Note that the WorkerGuard returned by non_blocking must be assigned to a binding that is not , as will result in the WorkerGuard being dropped immediately. Unintentional drops of WorkerGuard remove the guarantee that logs will be flushed during a program’s termination, in a panic or otherwise.

What this means is that we must keep log_guard alive for the application to continue logging. log_guard is an instance of the WorkerGuard struct and is also the returned value of the public function pub fn init_app_logger(utc_offset: time::UtcOffset) -> WorkerGuard. We will revisit this returned value in a later section.

⓷ Next, we specify the date and time format for each log line. Each line begins with a local date and time. For instance, 2024-03-12-08:19:13:

    // Each log line starts with a local date and time token.
    // 
    // On Ubuntu 22.10, calling UtcOffset::current_local_offset().unwrap() after non_blocking()
    // causes IndeterminateOffset error!!
    // 
    // See also https://github.com/time-rs/time/pull/297.
    //
    let timer = OffsetTime::new(
        //UtcOffset::current_local_offset().unwrap(),
        utc_offset,
        format_description!("[year]-[month]-[day]-[hour]:[minute]:[second]"),
    );

We've discussed local dates in some detail in this post.

🚀 Please note that this is a local date and time. In my time zone, Australian Eastern Standard Time (AEST), which is 10-11 hours ahead of UTC, the log file name for a log line that starts with 2024-03-12-08:19:13 would actually be log/2024-03-11.log.

⓸ Next, we attempt to define the tracing::Level based on the environment variable RUST_LOG discussed previously:

    // Extracts tracing::Level from .env RUST_LOG, if there is any problem, 
    // defaults to Level::DEBUG.
    //
    let level: Level = match std::env::var_os("RUST_LOG") {
        None => Level::DEBUG,

        Some(text) => {
            match Level::from_str(text.to_str().unwrap()) {
                Ok(val) => val,
                Err(_) => Level::DEBUG
            }
        }
    };

💥 I initially assumed that having RUST_LOG defined in the environment file .env would suffice. However, it turns out that we need to explicitly set it in the code.

⓹ We then proceed to “create a subscriber”, I hope I'm using the correct terminology:

    let subscriber = tracing_subscriber::registry()
        .with(
            Layer::new()
                .with_timer(timer)
                .with_ansi(false)
                .with_writer(non_blocking_appender.with_max_level(level)
                    .and(std::io::stdout.with_max_level(level)))
        );

The function tracing_subscriber::registry() returns a tracing_subscriber::registry::Registry struct. This struct implements the trait tracing_subscriber::layer::SubscriberExt. The method fn with<L>(self, layer: L) -> Layered<L, Self> from this trait returns a tracing_subscriber::layer::Layered struct, which is a:

A Subscriber composed of a Subscriber wrapped by one or more Layers.

We create the new Layer using tracing_subscriber::fmt::Layer implementation.

Note that non_blocking_appender is an instance of tracing_appender::non_blocking::NonBlocking struct. This struct implements the trait tracing_subscriber::fmt::writer::MakeWriterExt, where the method fn with_max_level(self, level: Level) -> WithMaxLevel<Self> is defined.

🚀 .and(std::io::stdout.with_max_level(level)) means that anything logged to the log file will also be printed to the console.

⓺ Next, the new Subscriber is set as the global default for the duration of the entire program:

    // tracing::subscriber::set_global_default(subscriber) can only be called once. 
    // Subsequent calls raise SetGlobalDefaultError, ignore these errors.
    //
    // There are integeration test methods which call this init_app_logger(...) repeatedly!!
    //
    match tracing::subscriber::set_global_default(subscriber) {
        Err(err) => tracing::error!("Logger set_global_default, ignored: {}", err),
        _ => (),
    }

The documentation for the function tracing::subscriber::set_global_default states:

Can only be set once; subsequent attempts to set the global default will fail. Returns whether the initialization was successful.

Since some integration test methods call the pub fn init_app_logger(utc_offset: time::UtcOffset) -> WorkerGuard more than once, we catch potential errors and ignore them.

⓻ Finally, pub fn init_app_logger(utc_offset: time::UtcOffset) -> WorkerGuard returns log_guard, as discussed above.

❻ Updates to the src/main.rs module.

⓵ Coming back to pub fn init_app_logger(utc_offset: time::UtcOffset) -> WorkerGuard, specifically regarding the returned value discussed previously, I read and understood the quoted documentation, and I believe the code was correct. However, it did not write to log files as expected. I sought help. As per my help request post, I initially called init_app_logger in the src/lib.rs module's pub async fn run(listener: TcpListener) -> Result<Server, std::io::Error>. Consequently, as soon as run went of scope, the returned WorkerGuard was dropped, and the writer thread terminated.

Simply moved it to src/main.rs's async fn main() -> Result<(), std::io::Error>, fixed this problem:

    // Call this to load RUST_LOG.
    dotenv().ok(); 

    // Calling UtcOffset::current_local_offset().unwrap() here works in Ubuntu 22.10, i.e.,
    // it does not raise the IndeterminateOffset error.
    //
    // TO_DO. But this does not guarantee that it will always work! 
    //
    let _guards = init_app_logger(UtcOffset::current_local_offset().unwrap());

Please note the call UtcOffset::current_local_offset().unwrap(). This is due to the problem discussed in the section 💥 Issue with calculating UTC time offset on Ubuntu 22.10.

⓶ The function pub fn init_app_logger(utc_offset: time::UtcOffset) -> WorkerGuard requires the environment variable RUST_LOG as discussed previously. That's why dotenv().ok() is called in async fn main() -> Result<(), std::io::Error>.

Recall that dotenv().ok() is also called in the src/lib.rs module's pub async fn run(listener: TcpListener) -> Result<Server, std::io::Error> to load other environment variables. This setup might seem clunky, but I haven't found a better solution yet!

❼ Updating integration tests. We want integration tests to be able to log as well. These updates are made solely in the tests/common.rs module.

The function pub async fn spawn_app() -> TestApp in tests/common.rs calls the src/lib.rs module's function pub async fn run(listener: TcpListener) -> Result<Server, std::io::Error> to create application server instances.

This means that spawn_app() must be refactored to call pub fn init_app_logger(utc_offset: time::UtcOffset) -> WorkerGuard and somehow keep the writer thread alive after spawn_app() goes out of scope. We manage this by:

⓵ Update TestApp struct by adding pub guard: WorkerGuard.

⓶ Update the function pub async fn spawn_app() -> TestApp with additional calls:

    // To load RUST_LOG from .env file.
    dotenv().ok(); 

    /*
    On Ubuntu 22.10, calling UtcOffset's offset methods causes IndeterminateOffset error!!

    See also https://github.com/time-rs/time/pull/297

    ...
    */

    // TO_DO: 11 is the current number of hours the Australian Eastern Standard Time (AEST)
    // is ahead of UTC. This value need to be worked out dynamically -- if it is at all 
    // possible on Linux!!
    // 
    let guard = init_app_logger(UtcOffset::from_hms(11, 0, 0).unwrap());

Note the call UtcOffset::from_hms(11, 0, 0).unwrap(). This is due to the problem discussed in section 💥 Issue with calculating UTC time offset on Ubuntu 22.10:

-- 👎 Unlike src/main.rs, where UtcOffset::current_local_offset().unwrap() works, calling it here consistently results in the IndeterminateOffset error! UtcOffset::from_hms(11, 0, 0).unwrap() works, but again, this is not a guarantee it will keep working.

👎 The value 11 is hardcoded. Presently, the Australian Eastern Standard Time (AEST) zone is 11 hours ahead of UTC. To get the AEST date and time, we need to offset UTC by 11 hours. However, 11 is not a constant value; due to daylight savings, in Southern Hemisphere winters, it changes to 10 hours (I think). This means that this code will no longer be correct.

❽ We've reached the conclusion of this post. I'd like to mention that the ecosystem surrounding tracing and logging is incredibly vast. While this post only scratches the surface, it provides a complete working example nonetheless. We can build upon this foundation as needed.

The UTC offset issue on Ubuntu 22.10, as described, must be addressed definitively. However, that task is for another day.

I'm not entirely satisfied with the numerous debug loggings from other crates. These can be filtered and removed, but that's a topic for another post, perhaps.

I hope you find the information useful. Thank you for reading. And stay safe, as always.

✿✿✿

Feature image source:

🦀 Index of the Complete Series.

Rust: actix-web JSON Web Token authentication.

Be Hai Nguyen — Tue, 27 Feb 2024 02:06:01 +0000

In the sixth post of our actix-web learning application, we implemented a basic email-password login process with a placeholder for a token. In this post, we will implement a comprehensive JSON Web Token (JWT)-based authentication system. We will utilise the jsonwebtoken crate, which we have previously studied.

🚀 Please note, complete code for this post can be downloaded from GitHub with:

git clone -b v0.10.0 https://github.com/behai-nguyen/rust_web_01.git

The actix-web learning application mentioned above has been discussed in the following nine previous posts:

The code we're developing in this post is a continuation of the code from the ninth post above. 🚀 To get the code of this ninth post, please use the following command:

git clone -b v0.9.0 https://github.com/behai-nguyen/rust_web_01.git

-- Note the tag v0.9.0.

Previous Studies on JSON Web Token (JWT)
Proposed JWT Implementations: Problems and Solutions
- Proposed JWT Implementations
- Problems with the Proposed Implementations
  - Problems when Used as an API-Server or Service
  - Problems when Used as an Application Server
- Proposed Solutions
The “Bearer” Token Scheme
Project Layout
The Token Utility jwt_utils.rs and Test test_jsonwebtoken.rs Modules
- The Token Utility src/helper/jwt_utils.rs Module
- The Test tests/test_jsonwebtoken.rs Module
The Updated Login Process
The Updated Request Authentication Process
- Code Updated in the src/auth_middleware.rs Module
- Code Updated in the src/lib.rs Module
JWT and Logout
Updating Integration Tests
Concluding Remarks

Previous Studies on JSON Web Token (JWT)

As mentioned earlier, we conducted studies on the jsonwebtoken crate, as detailed in the post titled Rust: JSON Web Token -- some investigative studies on crate jsonwebtoken. The JWT implementation in this post is based on the specifications discussed in the second example of the aforementioned post, particularly focusing on this specification:

🚀 It should be obvious that: this implementation implies SECONDS_VALID_FOR is the duration the token stays valid since last active. It does not mean that after this duration, the token becomes invalid or expired. So long as the client keeps sending requests while the token is valid, it will never expire!

We will provide further details on this specification later in the post. Additionally, before studying the jsonwebtoken crate, we conducted research on the jwt-simple crate, as discussed in the post titled Rust: JSON Web Token -- some investigative studies on crate jwt-simple. It would be beneficial to review this post as well, as it covers background information on JWT.

Proposed JWT Implementations: Problems and Solutions

Proposed JWT Implementations

Let's revisit the specifications outlined in the previous section:

🚀 It should be obvious that: this implementation implies SECONDS_VALID_FOR is the duration the token stays valid since last active. It does not mean that after this duration, the token becomes invalid or expired. So long as the client keeps sending requests while the token is valid, it will never expire!

This concept involves extending the expiry time of a valid token every time a request is made. This functionality was demonstrated in the original discussion, specifically in the second example section mentioned earlier.

🦀 Since the expiry time is updated, we generate a new access token. Here's what we do with the new token:

Replace the current actix-identity::Identity login with the new access token.
Always send the new access token to clients via both the response header and the response cookie authorization, as in the login process.

We generate a new access token based on logic, but it doesn't necessarily mean the previous ones have expired."

Problems with the Proposed Implementations

The proposed implementations outlined above present some practical challenges, which we will discuss next.

However, for the sake of learning in this project, we will proceed with the proposed implementations despite the identified issues.

Problems when Used as an API-Server or Service

In an API-like server or a service, users are required to include a valid access token in the request authorization header. Therefore, if a new token is generated, users should have access to this latest token.

What happens if users simply ignore the new tokens and continue using a previous one that has not yet expired? In such a scenario, request authentication would still be successful, and the requests would potentially succeed until the old token expires. However, a more serious concern arises if we implement blacklisting. In that case, we would need to blacklist all previous tokens. This would necessitate writing the current access token to a blacklist table for every request, which is impractical.

Problems when Used as an Application Server

When used as an application server, we simply replace the current actix-identity::Identity login with the new access token. If we implement blacklisting, we only need to blacklist the last token

🚀 This process makes sense, as we cannot expire a session while a user is still actively using it.

However, we still encounter similar problems as described in the previous section for API-like servers or services. Since clients always have access to the authorization response header and cookie, they can use this token with different client tools to send requests, effectively treating the application as an API-like server or a service.

Proposed Solutions

The above problems would disappear, and the actual implementations would be simpler if we adjust the logic slightly:

Only send the access token to clients once if the content type of the login request is application/json.
Then users of an API-like server or a service will only have one access token until it expires. They will need to log in again to obtain a new token.
Still replace the current actix-identity::Identity login with the new access token. The application server continues to function as usual. However, since users no longer have access to the token, we only need to manage the one stored in the actix-identity::Identity login.

But as mentioned at the start of this section, we will ignore the problems and, therefore, the solutions for this revision of the code.

The “Bearer” Token Scheme

We adhere to the “Bearer” token scheme as specified in RFC 6750, section 2.1. Authorization Request Header Field:

    For example:
        GET /resource HTTP/1.1
        Host: server.example.com
        Authorization: Bearer mF_9.B5f-4.1JqM

That is, the access token used during request authentication is in the format:

Bearer. + the proper JSON Web Token

For example:

Bearer.eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJlbWFpbCI6ImNoaXJzdGlhbi5rb2JsaWNrLjEwMDA0QGdtYWlsLmNvbSIsImlhdCI6MTcwODU1OTcwNywiZXhwIjoxNzA4NTYxNTA3LCJsYXN0X2FjdGl2ZSI6MTcwODU1OTcwN30.CN-whQ0rWW8IuLPVTF7qprk4-GgtK1JSJqp3C8X-ytE

❶ The access token included in the request authorization header must adhere to the "Bearer" token format.

❷ Similarly, the access token set for the actix-identity::Identity login is also a "Bearer" token.

🦀 However, the access token sent to clients via the response header and the response cookie authorization is always a pure JSON Web Token.

Project Layout

Below is the complete project layout.

-- Please note, those marked with ★ are updated, and those marked with ☆ are new.

.
├── .env ★
├── Cargo.toml ★
├── cert
│ ├── cert-pass.pem
│ ├── key-pass-decrypted.pem
│ └── key-pass.pem
├── migrations
│ ├── mysql
│ │ └── migrations
│ │     ├── 20231128234321_emp_email_pwd.down.sql
│ │     └── 20231128234321_emp_email_pwd.up.sql
│ └── postgres
│     └── migrations
│         ├── 20231130023147_emp_email_pwd.down.sql
│         └── 20231130023147_emp_email_pwd.up.sql
├── README.md ★
├── src
│ ├── auth_handlers.rs ★
│ ├── auth_middleware.rs ★
│ ├── bh_libs
│ │ ├── api_status.rs
│ │ └── australian_date.rs
│ ├── bh_libs.rs
│ ├── config.rs ★
│ ├── database.rs
│ ├── handlers.rs
│ ├── helper
│ │ ├── app_utils.rs ★
│ │ ├── constants.rs ★
│ │ ├── endpoint.rs ★
│ │ ├── jwt_utils.rs ☆
│ │ └── messages.rs ★
│ ├── helper.rs ★
│ ├── lib.rs ★
│ ├── main.rs
│ ├── middleware.rs
│ └── models.rs ★
├── templates
│ ├── auth
│ │ ├── home.html
│ │ └── login.html
│ └── employees.html
└── tests
    ├── common.rs ★
    ├── test_auth_handlers.rs ★
    ├── test_handlers.rs ★
    └── test_jsonwebtoken.rs ☆

The Token Utility jwt_utils.rs and Test test_jsonwebtoken.rs Modules

The Token Utility src/helper/jwt_utils.rs Module

In the module src/helper/jwt_utils.rs, we implement all the JWT management code, which includes the core essential code that somewhat repeats the code already mentioned in the second example:

struct JWTPayload -- represents the JWT payload, where the email field uniquely identifies the logged-in user.
JWTPayload implementation -- implements some of the required functions and methods:
- A function to create a new instance.
- Methods to update the expiry field (exp) and the last_active field using seconds, minutes, and hours.
- Four getter methods which return the values of the iat, email, exp, and last_active fields.

Additionally, there are two main functions:

pub fn make_token -- creates a new JWT from an email. The parameter secs_valid_for indicates how many seconds the token is valid for, and the parameter secret_key is used by the jsonwebtoken crate to encode the token. It creates an instance of struct JWTPayload, and then creates a token using this instance.
pub fn decode_token -- decodes a given token. If the token is valid and successfully decoded, it returns the token's struct JWTPayload. Otherwise, it returns an ApiStatus which describes the error.

Other functions are “convenient” functions or wrapper functions:

pub fn make_token_from_payload -- creates a JWT from an instance of struct struct JWTPayload. It is a "convenient" function. We decode the current token, update the extracted payload, then call this function to create an updated token.
pub fn make_bearer_token -- a wrapper function that creates a “Bearer” token from a given token.
pub fn decode_bearer_token -- a wrapper function that decodes a “Bearer” token.

Please note also the unit test section in this module. There are sufficient tests to cover all functions and methods.

The documentation in the source code should be sufficient to aid in the reading of the code.

The Test tests/test_jsonwebtoken.rs Module

We implement some integration tests for JWT management code. These tests are self-explanatory.

The Updated Login Process

In the current login process, at step 4, we note:

...
    // TO_DO: Work in progress -- future implementations will formalise access token.
    let access_token = &selected_login.email;

    // https://docs.rs/actix-identity/latest/actix_identity/
    // Attach a verified user identity to the active session
    Identity::login(&request.extensions(), String::from(access_token)).unwrap();
...

This part of the login process handler pub async fn login(request: HttpRequest, app_state: web::Data<super::AppState>, body: Bytes) -> Either<impl Responder, HttpResponse> is updated to:

...
    let access_token = make_token(&selected_login.email, 
        app_state.cfg.jwt_secret_key.as_ref(), app_state.cfg.jwt_mins_valid_for * 60);

    // https://docs.rs/actix-identity/latest/actix_identity/
    // Attach a verified user identity to the active session
    Identity::login(&request.extensions(), String::from( make_bearer_token(&access_token) )).unwrap();
...

Please note the call to make_bearer_token, which adheres to The “Bearer” Token Scheme.

This update would take care of the application server case. In the case of an API-like server or a service, users are required to include a valid access token in the request authorization header, as mentioned, so we don't need to do anything.

The next task is to update the request authentication process. This update occurs in the src/auth_middleware.rs and the src/lib.rs modules.

The Updated Request Authentication Process

The updated request request authentication involves changes to both the src/auth_middleware.rs and src/lib.rs modules.

This section, How the Request Authentication Process Works, describes the current process.

Code Updated in the src/auth_middleware.rs Module

Please recall that the src/auth_middleware.rs module serves as the request authentication middleware. We will make some substantial updates within this module.

Although the code has sufficient documentation, we will discuss the updates in the following sections.

⓵ The module documentation has been updated to describe how the request authentication process works with JWT. Please refer to the documentation section How This Middleware Works for more details.

⓶ New struct TokenStatus:

struct TokenStatus {
    is_logged_in: bool,
    payload: Option<JWTPayload>,
    api_status: Option<ApiStatus>
}

The struct TokenStatus represents the status of the access token for the current request:

When there is no token, is_logged_in is set to false to indicate that the request comes from an unauthenticated session. The other two fields are set to None, indicating that there is no error.
When there is a token, we call the pub fn decode_token(token: &str, secret_key: &[u8]) -> Result<JWTPayload, ApiStatus> function:
- If token decoding fails or the token has already expired, is_logged_in is set to false, and api_status is set to the returned ApiStatus. This indicates an error.
- If token decoding succeeds, is_logged_in is set to to true, and payload is set to the returned JWTPayload.

⓷ The function fn verify_valid_access_token(request: &ServiceRequest) -> TokenStatus has been completely rewritten, although its purpose remains the same. It checks if the token is present and, if so, decodes it.

The return value of this function is struct TokenStatus, whose fields are set based on the rules discussed previously.

⓸ The new helper function fn update_and_set_updated_token(request: &ServiceRequest, token_status: TokenStatus) is called when there is a token and the token is successfully decoded.

It uses the JWTPayload instance in the token_status parameter to create the updated access token. Then, it:

Replaces the current actix-identity::Identity login with the new updated token, as discussed earlier.

Attaches the updated token to dev::ServiceRequest's dev::Extensions by calling fn extensions_mut(&self) -> RefMut<'_, Extensions>. The next adhoc middleware, discussed in the next section, consumes this extension.

⓹ The new closure, let unauthorised_token = |req: ServiceRequest, api_status: ApiStatus| -> Self::Future, calls the Unauthorized() method on HttpResponse to return a JSON serialisation of ApiStatus.

Note the calls to remove the server-side per-request cookies redirect-message and original-content-type.

⓺ Update the fn call(&self, request: ServiceRequest) -> Self::Future function. All groundwork has been completed. The updates to this method are fairly straightforward:

Update the call to fn verify_valid_access_token(request: &ServiceRequest) -> TokenStatus; the return value is now struct TokenStatus.
If the token is in error, call the closure unauthorised_token() to return the error response. The request is then completed.
If the request is from an authenticated session, meaning we have a token, and the token has been decoded successfully, we make an additional call to the new helper function fn update_and_set_updated_token(request: &ServiceRequest, token_status: TokenStatus), which has been described in the previous section.

The core logic of this method remains unchanged.

Code Updated in the src/lib.rs Module

As mentioned previously, if a valid token is present, an updated token is generated from the current token's payload every time a request occurs. This updated access token is then sent to the client via both the response header and the response cookie authorization.

This section describes how the updated token is attached to the request extension so that the next adhoc middleware can pick it up and send it to the clients.

This is the updated src/lib.rs next adhoc middleware. Its functionality is straightforward. It queries the current dev::ServiceRequest's dev::Extensions for a String, which represents the updated token. If found, it sets the ServiceResponse authorization header and cookie with this updated token.

Afterward, it forwards the response. Since it is currently the last middleware in the call stack, the response will be sent directly to the client, completing the request.

JWT and Logout

Due to the issues outlined in this section and this section, we were unable to effectively implement the logout functionality in the application. This will remain unresolved until we implement the proposed solutions and integrate blacklisting.

-- For the time being, we will retain the current logout process unchanged.

Once blacklisting is implemented, the request authentication process will need to validate the access token against the blacklist table. If the token is found in the blacklist, it will be considered invalid.

Updating Integration Tests

There is a new integration test module as already discussed in section The Test tests/test_jsonwebtoken.rs Module. There is no new integration test added to existing modules.

Some common test code has been updated as a result of implementing JSON Web Token.

⓵ There are several updates in module tests/common.rs:

Function pub fn mock_access_token(&self, secs_valid_for: u64) -> String now returns a correctly formatted “Bearer” token. Please note the new parameter secs_valid_for.
New function pub fn jwt_secret_key() -> String
New function pub fn assert_token_email(token: &str, email: &str). It decodes the parameter token, which is expected to always succeed, then tests that the token JWTPayload's email value equal to parameter email.
Rewrote pub fn assert_access_token_in_header(response: &reqwest::Response, email: &str) and pub fn assert_access_token_in_cookie(response: &reqwest::Response, email: &str).
Updated pub async fn assert_json_successful_login(response: reqwest::Response, email: &str).

⓶ Some minor changes in both the tests/test_handlers.rs and the tests/test_auth_handlers.rs modules:

Call the function pub fn mock_access_token(&self, secs_valid_for: u64) -> String with the new parameter secs_valid_for.
Other updates as a result of the updates in the tests/common.rs module.

Concluding Remarks

It has been an interesting process for me as I delved into the world of actix-web adhoc middleware. While the code may seem simple at first glance, I encountered some problems along the way and sought assistance to overcome them.

I anticipated the problems, as described in this section and this section, before diving into the actual coding process. Despite the hurdles, I proceeded with the implementation because I wanted to learn how to set a custom header for all routes before their final response is sent to clients – that's the essence of adhoc middleware.

In a future post, I plan to implement the proposed solutions and explore the concept of blacklisting.

I hope you find this post informative and helpful. Thank you for reading. And stay safe, as always.

✿✿✿

Feature image source:

Rust: actix-web get SSL/HTTPS for localhost.

Be Hai Nguyen — Sat, 10 Feb 2024 05:58:39 +0000

We are going to enable our actix-web learning application to run under HTTPS. As a result, we need to do some minor refactoring to existing integration tests. We also move and rename an existing module for better code organisation.

🚀 Please note, complete code for this post can be downloaded from GitHub with:



git clone -b v0.7.0 https://github.com/behai-nguyen/rust_web_01.git

The actix-web learning application mentioned above has been discussed in the following six previous posts:

The code we're developing in this post is a continuation of the code from the sixth post above. 🚀 To get the code of this sixth post, please use the following command:



git clone -b v0.6.0 https://github.com/behai-nguyen/rust_web_01.git

-- Note the tag v0.6.0.

Define “to run under HTTPS”
Project Layout
The OpenSSL Toolkit
- Installation
- Windows 10, the Rust Analyzer plug-in and the OpenSSL Toolkit environment variable OPENSSL_DIR
Generate a Self-Signed Encrypted Private Key and a Certificate
Code Refactoring to Enable HTTPS
Refactor Integration Tests
- The tests/common.rs Module
- The tests/test_handlers.rs and tests/test_auth_handlers.rs Modules
Moving src/utils.rs to src/bh_libs/australian_date.rs
Concluding Remarks

❶ To run under HTTPS. That is:



https://localhost:5000/ui/login
https://192.168.0.16:5000/ui/login

❷ Project Layout.

This post introduces a self-signed encrypted private key file and a certificate file. The updated directory layout for the project is listed below.

-- Please note, those marked with ★ are updated, and those marked with ☆ are new.



.
├── Cargo.toml ★
├── cert
│ ├── cert-pass.pem ☆ -- Self-signed encrypted private key
│ └── key-pass.pem ☆ -- Certificate
├── .env
├── migrations
│ ├── mysql
│ │ └── migrations
│ │     ├── 20231128234321_emp_email_pwd.down.sql
│ │     └── 20231128234321_emp_email_pwd.up.sql
│ └── postgres
│     └── migrations
│         ├── 20231130023147_emp_email_pwd.down.sql
│         └── 20231130023147_emp_email_pwd.up.sql
├── README.md ★
├── src
│ ├── auth_handlers.rs
│ ├── auth_middleware.rs
│ ├── bh_libs
│ │ ├── api_status.rs
│ │ └── australian_date.rs ★
│ ├── bh_libs.rs ★
│ ├── config.rs
│ ├── database.rs
│ ├── handlers.rs
│ ├── helper
│ │ ├── app_utils.rs ★
│ │ ├── constants.rs
│ │ ├── endpoint.rs
│ │ └── messages.rs
│ ├── helper.rs
│ ├── lib.rs ★
│ ├── main.rs
│ ├── middleware.rs
│ └── models.rs ★
├── templates
│ ├── auth
│ │ ├── home.html
│ │ └── login.html
│ └── employees.html
└── tests
    ├── common.rs ★
    ├── test_auth_handlers.rs ★
    └── test_handlers.rs ★

❸ In this post, we are using the OpenSSL Cryptography and SSL/TLS Toolkit to generate the self-signed encrypted private key and the certificate files.

⓵ We have previously discussed its installation on both Windows 10 Pro and Ubuntu 22.10 in this section of another post.

⓶ 💥 On Windows 10 Pro, I have observed that, once we include the openssl crate, we should set the environment variable OPENSSL_DIR at the system level, otherwise the Rust Analyzer Visual Studio Code plug-in would run into trouble.

The environment variable OPENSSL_DIR indicates where OpenSSL has been installed. For example, C:\Program Files\OpenSSL-Win64. Following are the steps to access Windows 10 Pro environment variable setting dialog.

Right click on This PC ➜ Properties ➜ Advance system settings (right hand side) ➜ Advanced tab ➜ Environment Variables... button ➜ under System variables ➜ New... ➜ enter variable name and value in the dialog ➜ OK button.

The screenshot below is a brief visual representation of the above steps, including the environment variable OPENSSL_DIR in place:

We might need to restart Visual Studio Code to get the new setting recognised.

❹ Generate the self-signed encrypted private key and the certificate files using the OpenSSL Toolkit.

The OpenSSL command to generate the files will prompt a series of questions. One important question is the Common Name which is the server name or FQDN where the certificate is going to be used. If we are not yet familiar with this process, this FQDN (Fully Qualified Domain Name): What It Is, Examples, and More article would be an essential reading, in my humble opnion.

I did seek help working on this issue, please see Actix-web OpenSSL SSL/HTTPS for Localhost, is it possible please? And I was pointed to examples/https-tls/openssl/ -- and this is my primary source of reference for getting this learning application to run under HTTPS.

The command I choose to use is:



$ openssl req -x509 -newkey rsa:4096 -keyout key-pass.pem -out cert-pass.pem -sha256 -days 365

Be prepared, we will be asked the following questions:



Enter PEM pass pharse: 

Country Name (2 letter code) [AU]: 
State or Province Name (full name) [Some-State]: 
Locality Name (eg, city) []: Melbourne
Organization Name (eg, company) [Internet Widgits Pty Ltd]: 
Organizational Unit Name (eg, section) []: 
Common Name (e.g. server FQDN or YOUR name) []: 
Email Address []: 

Please enter the following 'extra' attributes
to be sent with your certificate request

A challenge password []: 
An optional company name []:

Both key-pass.pem and cert-pass.pem are in the cert/ sub-directory as seen in the Project Layout section.

💥 Please note I also use these two files on Windows 10 Pro to run the application. It works, I am not sure why yet. I need to keep an eye out for this.

❺ Code refactoring to enable HTTPS.

We are also taking the code from examples/https-tls/openssl/. In src/lib.rs, we add two private functions HttpServer object, we call method listen_openssl(...) instead of method listen(...):



...
    .listen_openssl(listener, ssl_builder())?
...

I have tested with the latest version of the following browsers: Firefox, Chrome, Edge, Opera, Brave and Vivaldi, for both:



https://localhost:5000/ui/login
https://192.168.0.16:5000/ui/login

We might get a warning of potential security risk... For example, see the Firefox warning in the below screenshot:

I just ignore the warning and choose to go ahead. Even though https:// works, but all mentioned browsers state that the connection is not secure. Please see Firefox, Chrome and Opera sample screenshots below:

❻ We have to make changes to both integration tests common code and actual test code.

⓵ It's quite obvious that we should access the routes via HTTPS. The first change would be tests/common.rs. We should set the scheme of app_url to https://:



...
    TestApp {
        app_url: format!("https://127.0.0.1:{}", port)
    }
}

I did run integration tests after making this change. They failed. Base on the error messages, it seems that reqwest::Client should “have” the certificate as well (?).

Looking through the reqwest crate documentation, pub fn add_root_certificate(self, cert: Certificate) -> ClientBuilder seems like a good candidate...

Base on the example given in reqwest::tls::Certificate, I come up with pub fn load_certificate() -> Certificate and pub fn reqwest_client() -> reqwest::Client.

I have tried to document all my observations during developing these two helper functions. They are short and simple, I think the inline documentation explains the code quite sufficiently.

-- Initially, reqwest_client() does not include .danger_accept_invalid_certs(true), resulting in a certificate error. This solution, provided in the following Stack Overflow thread titled How to resolve a Rust Reqwest Error: Invalid Certificate suggests adding .danger_accept_invalid_certs(true), which appears to resolve the issue.

💥 Base on all evidences presented so far, including the connection not secure warnings reported by browsers and the need to call .danger_accept_invalid_certs(true) when creating a reqwest::Client instance, it seems to suggest that there may still be an issue with this implementation. Or is it common for a self-signed certificate, which is not issued by a trusted certificate authority, to encounter such problems? However, having the application run under https:// addresses issues I have had with cookies. For now, I will leave it as is. We will discuss cookie in another new post.

⓶ In both integration test modules, tests/test_handlers.rs and tests/test_auth_handlers.rs, we use the pub fn reqwest_client() -> reqwest::Client function to create instances of reqwest::Client for testing purposes, instead of creating instances directly in each test method.

❼ The final task of this post involves moving src/utils.rs to src/bh_libs/australian_date.rs, as it is a generic module, even though it depends on other third-party crates. It is possible that this module will be moved elsewhere again.

The module src/bh_libs/australian_date.rs is generic enough to used as-is in other projects.

As a result, the src/models.rs module is updated.

❽ We've reached the end of this post. I'd like to mention that I also followed the tutorial How to Get SSL/HTTPS for Localhost. I completed it successfully on Ubuntu 22.10, but browsers still warn about the connection not being secure. Perhaps this is to be expected with a self-signed certificate?

Overall, it's been an interesting exercise. I hope you find the information in this post useful. Thank you for reading. And stay safe, as always.

✿✿✿

Feature image source:

Rust: actix-web endpoints which accept both application/x-www-form-urlencoded and application/json content types.

Be Hai Nguyen — Sun, 14 Jan 2024 02:01:31 +0000

We're implementing a login process for our actix-web learning application. We undertake some general updates to get ready to support login. We then implement a new /api/login route, which supports both application/x-www-form-urlencoded and application/json content types. In this post, we only implement deserialising the submitted request data, then echo some response. We also add a login page via route /ui/login.

🚀 Please note, complete code for this post can be downloaded from GitHub with:



git clone -b v0.5.0 https://github.com/behai-nguyen/rust_web_01.git

The actix-web learning application mentioned above has been discussed in the following four (4) previous posts:

The code we're developing in this post is a continuation of the code from the fourth post above. 🚀 To get the code of this fourth post, please use the following command:



git clone -b v0.4.0 https://github.com/behai-nguyen/rust_web_01.git

-- Note the tag v0.4.0.

As already mentioned in the introduction above, in this post, our main focus of the login process is deserialising both application/x-www-form-urlencoded and application/json into a struct ready to support login. I struggle with this issue a little, I document it as part of my Rust learning journey.

This post introduces a few new modules, some MySQL migration scripts, and a new login HTML page. The updated directory layout for the project is in the screenshot below:

❶ Update Rust to use latest actix-cors
❷ Add new fields email and password to the employees table
❸ Update src/models.rs
❹ New module src/auth_handlers.rs which implements routes /ui/login and /api/login
❺ Update src/lib.rs
❻ The new templates/auth/login.html
❼ Testing
- ⓵ cargo test
- ⓶ Manual tests

❶ Update Rust to the latest version. At the time of this post, the latest version is 1.75.0. The command to update:



▶️<code>Windows 10:</code> rustup update
▶️<code>Ubuntu 22.10:</code> $ rustup update

We've taken CORS into account when we started out this project in this first post.

I'm not quite certain what'd happened, but all of a sudden, it just rejects requests with message Origin is not allowed to make this request.

-- Browsers have been updated, perhaps?

Failing to troubleshoot the problem, and seeing that actix-cors is at version 0.7.0. I update it.

-- It does not work with Rust version 1.74.0. This new version of actix-cors seems to fix the above request rejection issue.

❷ Update the employees table, adding new fields email and password.

Using the migration tool SQLx CLI, which we've covered in Rust SQLx CLI: database migration with MySQL and PostgreSQL, to update the employees table.

While inside the new directory migrations/mysql/, see project directory layout above, create empty migration files 99999999999999_emp_email_pwd.up.sql and 99999999999999_emp_email_pwd.down.sql using the command:



▶️<code>Windows 10:</code> sqlx migrate add -r emp_email_pwd
▶️<code>Ubuntu 22.10:</code> $ sqlx migrate add -r emp_email_pwd

Populate the two script files with what we would like to do. Please see their contents on GitHub. To apply, run the below command, it'll take a little while to complete:



▶️<code>Windows 10:</code> sqlx migrate add -r emp_email_pwd
▶️<code>Ubuntu 22.10:</code> $ sqlx migrate add -r emp_email_pwd

❸ Update src/models.rs to manage new fields employees.email and employees.password.

If we run cargo test now, all integration tests should fail. All integration tests eventually call to get_employees(...), which does a select * from employees.... Since the two new fields've been added to a specific order, field indexes in get_employees(...) are out of order.

Module src/models.rs gets the following updates:

pub email: String field added to struct Employee.
pub async fn get_employees(...) updated to read Employee.email field. Other fields' indexes also get updated.
New pub struct EmployeeLogin.
New pub async fn select_employee(...), which optionally selects an employee base on exact email match.
New pub struct LoginSuccess.
Add "email": "siamak.bernardeschi.67115@gmail.com" to existing tests.

Please see the updated src/models.rs on GitHub. The documentation should be sufficient to help reading the code.

❹ New module src/auth_handlers.rs, where new login routes /ui/login and /api/login are implemented.

● http://0.0.0.0:5000/ui/login is a GET route, which just returns the login.html page as HTML.

● http://0.0.0.0:5000/api/login is a POST route. This is effectively the application login handler.

💥 This http://0.0.0.0:5000/api/login route is the main focus of this post:

-- Its handler method accepts both application/x-www-form-urlencoded and application/json content types, and deserialises the byte stream to struct EmployeeLogin mentioned above.

💥 Please also note that, as already mentioned, in this post, the login process does not do login, if successfully deserialised the submitted data, it'd just echo a confirmation response in the format of the request content type. If failed to deserialise, it'd send back a JSON response which has an error code and a text message.

Examples of valid submitted data for each content type:

✔️ Content type: application/x-www-form-urlencoded; data: email=chirstian.koblick.10004@gmail.com&password=password.

✔️ Content type: application/json; data: {"email": "chirstian.koblick.10004@gmail.com", "password": "password"}.

Content of src/auth_handlers.rs



#[post("/login")]
pub async fn login(
    request: HttpRequest,
    body: Bytes
) -> HttpResponse {
...
    // Attempts to extract -- deserialising -- request body into EmployeeLogin.
    let api_status = extract_employee_login(&body, request.content_type());
    // Failed to deserialise request body. Returns the error as is.
    if api_status.is_err() {
        return HttpResponse::Ok()
            .content_type(ContentType::json())
            .body(serde_json::to_string(&api_status.err().unwrap()).unwrap());
    }

    // Succeeded to deserialise request body.
    let emp_login: EmployeeLogin = api_status.unwrap();
...

Note the second parameter body, which is actix_web::web::Bytes, this is the byte stream presentation of the request body.

As an extractor, actix_web::web::Bytes has been mentioned in section Type-safe information extraction | Other. We're providing our own implementation to do the deserialisation, method extract_employee_login(...) in new module src/helper/endpoint.rs.

Content of src/helper/endpoint.rs



pub fn extract_employee_login(
    body: &Bytes, 
    content_type: &str
) -> Result<EmployeeLogin, ApiStatus> {
...
    extractors.push(Extractor { 
        content_type: mime::APPLICATION_WWW_FORM_URLENCODED.to_string(), 
        handler: |body: &Bytes| -> Result<EmployeeLogin, ApiStatus> {
            match from_bytes::<EmployeeLogin>(&body.to_owned().to_vec()) {
                Ok(e) => Ok(e),
                Err(e) => Err(ApiStatus::new(err_code_500()).set_text(&e.to_string()))
            }
        }
    });
...
    extractors.push(Extractor {
        content_type: mime::APPLICATION_JSON.to_string(),
        handler: |body: &Bytes| -> Result<EmployeeLogin, ApiStatus> {
            // From https://stackoverflow.com/a/67340858
            match serde_json::from_slice(&body.to_owned()) {
                Ok(e) => Ok(e),
                Err(e) => Err(ApiStatus::new(err_code_500()).set_text(&e.to_string()))
            }
        }
    });

For application/x-www-form-urlencoded content type, we call method serde_html_form::from_bytes(...) from (new) crate serde_html_form to deserialise the byte stream to EmployeeLogin.

-- Cargo.toml has been updated to include crate serde_html_form.

And for application/json content type, we call to serde_json::from_slice(...) from the already included serde_json crate to do the work.

These're the essential details of the code. The rest is fairly straightforward, and there's also sufficient documentation to aid the reading of the code.

💥 Please also note that there're also some more new modules, such as src/bh_libs/api_status.rs and src/helper/messages.rs, they're very small, self-explanatory and have sufficient documentation where appropriate.

❺ Register new login routes /ui/login and /api/login.

Updated src/lib.rs:



pub async fn run(listener: TcpListener) -> Result<Server, std::io::Error> {
...
            .service(
                web::scope("/ui")
                    .service(handlers::employees_html1)
                    .service(handlers::employees_html2)
                    .service(auth_handlers::login_page)
                    // .service(auth_handlers::home_page),
            )
            .service(
                web::scope("/api")
                    .service(auth_handlers::login)
            )
            .service(
                web::resource("/helloemployee/{last_name}/{first_name}")
                    .wrap(middleware::SayHi)
                    .route(web::get().to(handlers::hi_first_employee_found))
            )
...

❻ The last addition, the new templates/auth/login.html.

Please note, this login page has only HTML. There is no CSS at all. It looks like a dog's breakfast, but it does work. There is no client-side validations either.

The Login button POSTs login requests to http://0.0.0.0:5000/api/login, the content type then is application/x-www-form-urlencoded.

For application/json content type, we can use Testfully. (We could also write our own AJAX requests to test.)

❼ As this is not yet the final version of the login process, we're not writing any integration tests for it yet. We'll do so in due course...

⓵ For the time being, we've written some new code and their associated unit tests. We have also written some documentation examples. The full test with the command cargo test should have all tests pass.

⓶ Manual tests of the new routes.

In the following two successful tests, I run the application server on an Ubuntu 22.10 machine, and run both the login page and Testfully on Windows 10.

Test application/x-www-form-urlencoded submission via login page:

Test application/json submission using Testfully:

In this failure test, I run the application server and Testfully on Windows 10. The submitted application/json data does not have an email field:

It's been an interesting exercise for me. My understanding of Rust's improved a little. I hope you find the information in this post useful. Thank you for reading and stay safe as always.

✿✿✿

Feature image source:

Forem: Be Hai Nguyen

rlox: A Rust Implementation of “Crafting Interpreters” – Scanner

🦀 Index of the Complete Series.

🦀 Index of the Complete Series.

Raspberry Pi 4B: Natively Build a 64 Bit Fully Preemptible Kernel (Real-Time) with Desktop

Python & MariaDB: Which Driver? An Example of Executing a Stored Procedure That Returns Multiple Result Sets

Python FastAPI: Implementing Non-Blocking Logging with Built-In QueueHandler and QueueListener Classes

🐍 Index of the Complete Series.

Table Of Contents

🐍 Index of the Complete Series.

Python FastAPI: Integrating OAuth2 Security with the Application's Own Authentication Process

🐍 Index of the Complete Series.

🐍 Index of the Complete Series.

Python: A SQLAlchemy Wrapper Component That Works With Both Flask and FastAPI Frameworks

Rust: Actix-web -- Async Functions as Middlewares

🦀 Index of the Complete Series.

🦀 Index of the Complete Series.

Rust: Actix-web Daily Logging -- Fix Local Offset, Apply Event Filtering

🦀 Index of the Complete Series.

🦀 Index of the Complete Series.

Rust: Actix-web and Daily Logging

🦀 Index of the Complete Series.

🦀 Index of the Complete Series.

Rust: actix-web JSON Web Token authentication.

Table of contents

Rust: actix-web get SSL/HTTPS for localhost.

Table of contents

Rust: actix-web endpoints which accept both application/x-www-form-urlencoded and application/json content types.

Table of contents