Forem: Łukasz Budnik

AI-Driven Development: My Database Experiment

Łukasz Budnik — Wed, 30 Apr 2025 23:15:01 +0000

There's plenty of debate about AI's role in coding. As an engineer, I've been working with Large Language Models and GenAI since their early days. I decided to skip the debate and put AI to a practical test: build a DynamoDB-like database service from scratch with AI assistance. Because who doesn't love a good database challenge?

The idea was to build a service with the following characteristics:

DynamoDB-like API: PutItem, UpdateItem, DeleteItem, GetItem, Query, TransactWriteItems.
RocksDB Storage: Utilizing RocksDB for fast and reliable data storage.
gRPC Interface: Providing a high-performance gRPC API for client interactions.
Transport Security: Support for TLS encryption and mutual TLS (mTLS) authentication for secure client-server communication.
Efficient Data Serialization: Leveraging gRPC's Protocol Buffers for efficient data serialization.
Containerization & Orchestration: multi-arch (linux/amd64 and linux/arm64) Docker container support for easy deployment, quick-start Kubernetes manifests provided.
Observability with OpenTelemetry: Built-in OpenTelemetry integration for monitoring RocksDB metrics and operational insights.
Agile Development Simulation: Instead of providing a complete design upfront, start small and iteratively add (generate) new features. (This was partly a necessity as I worked on it primarily on weekends).
AI-Driven Development: Let AI drive the development process as much as possible.

The Results

The development approach was pragmatic and interactive. The workflow was as follows:

Begin with interactive AI sessions to outline implementation strategies using Amazon Q chat.
Review generated code and iterate until it met my standards.
Use the generated code as a foundation.
Leverage inline code generation for refinements, extensions, and new features using Amazon Q inline code generation.
Review selected concepts using Google Gemini chat.

The numbers surprised even me: approximately 80% of the application code was AI-generated, with test coverage hitting an impressive 90% (if not more).

Amazon Q demonstrated impressive knowledge across the chosen technology stack: build tools and testing frameworks (Gradle, JUnit5, Mockito), communication protocols including transport security (gRPC, Protocol Buffers, TLS 1.3), RocksDB database, and deployment technologies (Docker, Kubernetes).

I want to call out three key things from this experiment:

Development Workflow. To state the obvious: the first generated code isn't always the best. By providing feedback and requesting improvements, the quality of the generated code improved significantly.
For example, the default JUnit tests for the gRPC service weren't using mocks but instead used the underlying implementation. When I asked for mocks, I got JUnit tests with interface mocks. Once I had tests with mocks, the inline code generation followed the existing code and started generating code with mocks.
Highlight. The TransactWriteItems implementation particularly showcased AI's capabilities, handling everything from Protocol Buffer definitions, gRPC implementation, RocksDB implementation, to @FunctionalInterface Java lambda for submitting RocksDB transactions.
Initially, the inline code generation had trouble generating the correct Java builders chains for this complex operation (TransactWriteItems is a composition of a list of PutItem, UpdateItem, DeleteItem operations). In order to get the correct Java builders chains, I used the Amazon Q chat interface and provided the proto definition as the input. Once I got the working code, I completed the remaining implementation using the inline code generation. The unit test for TransactWriteItems was almost entirely implemented by inline code generation, except for mocking the Java lambda expression for submitting the RocksDB transaction, which seemed to be a challenge for the inline tool. Again, I solved that by using the chat feature, where I explained the problem and pasted the @FunctionalInterface definition for additional context. The rest of the unit test was completed by the inline code generation.
As a result, most (about 95%) of the TransactWriteItems feature (ProtoBuf definition, gRPC service implementation, RocksDB implementation, and all the unit tests) was AI-generated. Quite impressive, I have to say.
Lowlight. I have to say that not everything was perfect. There's room for improvement in handling cross-cutting concerns, like input validation.
I think that if I had provided a complete design upfront, the generated validation code would have been elegantly built-in rather than bolted-on.
The input validation code is where I spent most of my time and had to provide the most of my manual implementation.

Key takeaways

Here are my key takeaways:

AI is great at implementing well-defined patterns. The more context, the better and more accurate the generated code.
AI excels at generating boilerplate code.
AI is extremely useful for writing unit tests. The inline code generator very strictly follows the Arrange-Act-Assert pattern and very accurately predicts what and how you want to test your code.
Interactive AI sessions are great for starting from scratch or in situations where there's not (yet) enough context in the existing code.
Interactive AI sessions produce better results than quick inline generations.
Complex architectural decisions benefit from human oversight and experience.
Some aspects (cross-cutting ones) still require significant human input.

The experiment demonstrated that AI-assisted development has matured to the point where it can significantly accelerate software development.
While it's not a complete replacement for human expertise and there are some areas of the project that I'm not happy with (which I left as they were generated for other engineers to review and assess), it can handle much of the heavy lifting in software development.

I think it's a great tool for teams looking to create minimum viable products to validate ideas quickly, as well as teams looking to build more complex production-grade systems.

Want to join the experiment?

This project is open source and welcomes contributions - with one condition: pull requests should primarily consist of AI-generated code :)

For more, see: https://github.com/lukaszbudnik/roxdb.

Cloud native C

Łukasz Budnik — Thu, 19 Aug 2021 18:37:53 +0000

Low level cloud SDKs

When you think about writing a cloud native app you probably think about higher level programming languages like Java, JavaScript, .NET, Python, Ruby, etc.

But what if you have an old C app? And you are preparing to either rehost it or even better replatform it?

Well, none of the main cloud providers offer C SDKs. AWS goes as low as C++. Which is nice.

But! All main cloud providers actually do offer Go SDKs. And with Go and its cgo tool we are already at home.

c + go = cgo

I have prepared a really simple C application which calls Go functions to upload and download an object to/from AWS S3 bucket.

The code is uploaded to my GitHub repo:

lukaszbudnik / cloud-native-c

cloud-native-c shows how to use AWS Go SDK from C using cgo

The Makefile is really simple, but I'm going to quickly review it:

OS:= $(shell uname -s)

ifeq ($(OS), Linux)
    build_cmd = gcc -pthread -o app app.c storage.a
endif
ifeq ($(OS), Darwin)
    build_cmd = gcc storage.a app.c -framework CoreFoundation -framework Security -o app
endif

all: app

app: storage app.c
    $(build_cmd)

storage: storage.go
    go build -buildmode=c-archive storage.go

The storage rule compiles and builds the storage.go as a C library. It generates 2 artifacts:

storage.h - which contains definitions needed by the C program to call Go functions
storage.a - compiled C library

Other than that storage.go is just a normal go program (it has to have an empty main function). It also uses go.mod to manage dependencies (AWS SDK Go v2 libraries).

The app rule compiles and builds the app.c application. As there are small differencies between gcc on Mac and Linux I'm passing different paramters to it. For MacOS it links CoreFoundation and Security frameworks and for Linux it links pthread library. Both MacOS and Linux are built automatically on GitHub using actions running on macos-10.15 and ubuntu-20.04.

Edit: Support for Windows 2019 with MinGW was also added. Windows binaries are also built automatically on GitHub using windows-2019 runner. See repo for more details.

Source code - C

How does the application look like then?

First let's take a look at the C code:

#include <stdio.h>
#include <string.h>
#include "response.h"
#include "storage.h"

int main(int argc, char **argv)
{
  if (argc != 3)
  {
    fprintf(stderr, "This app requires exactly 2 parameters first is the path to local file and second is the S3 bucket.\n");
    fprintf(stderr, "Example:\n%s app.c my-bucket-name\n", argv[0]);
    return WRONG_PARAMETERS_ERROR;
  }

  int res = PutObject(argv[1], argv[2]);

  if (res > 0)
  {
    fprintf(stderr, "There was an error uploading the file.\n");
    return res;
  }

  char *content = GetObject(argv[1], argv[2]);
  if (content == NULL)
  {
    return FILE_DOWNLOAD_ERROR;
  }

  printf("Content of the uploaded file is:\n%s\n", content);
  Free(content);

  return SUCCESS;
}

Nothing special. I included two header files.

storage.h - generated by cgo, contains C definitions of the PutObject(), GetObject() and Free() functions. These functions take and return C data types.
response.h - contains definitions of success and error codes. This header file is also included in Go application.

As you can see, the above source code is super simple, plain old C.

Source code - Go

Now let's take a look at the Go code.

The go code is almost plain go apart of these 4 lines:

// #include <stdio.h>
// #include <stdlib.h>
// #include "response.h"
import "C"

The above directives are responsible for:

exposing any types/methods/directives defined in stdio and stdlib as C. You can reference them as: C.char (char data type) or C.free() (method to release allocated memory), etc.
exposing any types/methods/directives defined in custom response.h header as C. For example: C.SUCCESS or C.FILE_UPLOAD_ERROR.
exposing special cgo methods as C. For example to convert a *C.char to go string you can use: C.GoString() function, to create a C string (*C.char) you can use C.CString(), etc. Check cgo documentation for more.

Two points to note.

It's much easier to convert types in Go then in C. So my PutObject(), GetObject(), and Free() functions take and return C types. Thanks to this I call these functions from C just like they were native C functions.
Important thing is that all the memory allocated by C.* functions needs to be freed using C.free() function. That is why I also exposed Free() function so that all objects which were returned from Go to C app can be send back to Go to be freed.

Go lang app looks like this. The //export Function are required otherwise cgo won't include them in storage.h header file.

//export PutObject
func PutObject(filenameC, bucketC *C.char) int {
    filename := C.GoString(filenameC)
    bucket := C.GoString(bucketC)

    cfg, err := config.LoadDefaultConfig(context.TODO())
    if err != nil {
        fmt.Fprintf(os.Stderr, "configuration error: %v\n", err.Error())
        return C.CONFIGURATION_ERROR
    }
    (...)
}
//export GetObject
func GetObject(keyC, bucketC *C.char) *C.char {
    key := C.GoString(keyC)
    bucket := C.GoString(bucketC)
    (...)
    return C.CString(content)
}
//export Free
func Free(ptr *C.char) {
    C.free(unsafe.Pointer(ptr))
}

Building and running

In order to build and run the application just call:

$ make all
go build -buildmode=c-archive storage.go
gcc storage.a app.c -framework CoreFoundation -framework Security -o app
$ ./app app.c my-bucket-name
File uploaded correctly, version id: Uzp7G2ehns7AtQdwpPrQoTQriXKbeldA
File version id: Uzp7G2ehns7AtQdwpPrQoTQriXKbeldA
Content of the uploaded file is:
#include <stdio.h>
#include <string.h>
#include "response.h"
#include "storage.h"
(...)

You are now well equipped to start coding cloud native apps in C!

Building cloud native apps: Databases best practices

Łukasz Budnik — Wed, 09 Jun 2021 11:39:22 +0000

Databases are the heart of every system. They also tend to be the source of many problems.

In this post I would like to summarise best practices for implementing databases and caching for cloud native apps.

Database types

Many developers and many architects start with RDB as a default data store. RDB market is the most mature one, there are countless database frameworks available, many developers know SQL really well.

You can use RDB as key-value store, full-text search engine, JSON documents store, or time-series store.

All above is true. But does one size really fit all?

As a refresher here are some database types you should remember about when designing/implementing your cloud native app. They are highly specialised databases designed to do their job really well.

Relational
Key Value
Document-oriented
In-memory
Graph
Time-series
Ledger
Full-text search

You can find an excellent summary of those in AWS database paper Enter the Purpose-Built Database Era. The paper contains description of each of the technology, examples, pros and cons. I highly recommend you read it!

Database versioning

If you run a cloud native app, perhaps even multi-tenant one, you are for sure using infrastructure-as-code paradigm. I touched on this already in my previous article in this series: Building cloud native apps: Codebase.

Database changes and database versioning should be no different. You must not apply changes to your databases in a manual way. Further every database change should be versioned, and this is especially important if you have, for example, 6 scrum teams working on the same product.

While database schema changes happen for all database types (including NoSQL ones) in case of RDB you must make explicit changes in both database schema and application code.

For relational database migrations and versioning, I'm using migrator. It's a super lightweight and super fast database migration tool written in Go. It runs as a docker container and exposes simple yet powerful GraphQL API. All migrations applied by migrator are grouped into versions for traceability, auditing, and compliance purposes.

migrator supports PostgreSQL, MySQL (and all its flavours), and MS SQL. It supports multi-tenancy out of the box. It can read migrations from local disk, AWS S3, or Azure Blob Containers. It also out-performs other database migration frameworks like the leading Java tool FlywayDB. It can also sync already executed migrations from legacy frameworks.

Caching

I cannot stress enough how important caching is. Caching should be implemented at multiple levels. Just like CPU caches are organised into hierarchy of L1, L2, some even use L3 and L4 caches. You should investigate implementing caching strategy in your distributed applications. For a reference this is how I do it in my SaaS product:

Hibernate L1 cache - session-level cache, enabled by default;
Hibernate L2 cache - session factory-level cache, application-wide cache for data entities, we use Infinispan as a L2 cache implementation, other options include JBoss Cache2;
Application cache - application-wide cache for business objects, implemented using Guava Cache;
Distributed cache - implemented using a cluster of Redis nodes;

For completeness: a well-architected system should also include other caches like web cache (HTML/JS/CSS), client-side cache (Web Storage API/IndexDB API/Cache API, build artifacts, docker images, etc.

Caching can drastically improve performance of your application. Depending on the scale of your operations this can be hundreds of millions of operations a day!

Important thing to remember is that different types of data may require different expiration time. It's best to write down all the requirements beforehand. You don't want to have stale data in your caches, on the other hand too short expiration times will lead to performance degradation.

Connection pooling

Connection pooling is a must for session-based database like for example RDBs. Connection pooling prepares, opens, and maintains ready-to-use database connections. When application needs a database connection it fetches one from the pool and returns it when no longer needs it. Connection pooling helps to better manage the database resources, greatly removing database connection fluctuations.

If you are looking for Java examples: I used C3P0 in the past and then switched to Hikari.

For those running serverless applications (for example running thousands of functions) and using relational databases as backends the fluctuations of database connections can cause a lot of unnecessary processing on the DB side. If you are running on AWS I highly recommend you to use AWS RDS Proxy in front-of your AWS RDS database. Have your Lambdas talk to the proxy and the proxy will offload all the work required to pool and maintain database connections for your serverless application.

Monitoring

I assume you do the basic stuff already: CPU, memory, disk space, read IOPS, write IOPS.

If you see high CPU usage or free memory dropping to zero it's information that something is wrong. But these metrics won't tell you what.

You need to have a more in-depth monitoring.

We used to have a pretty detailed monitoring of our data access layer at the application level. However, it was producing inaccurate data because of Hibernate L1 and L2 caches. At the application level we were recording an event of a DB operation where in fact that operation was reading data from L2 cache.

So what we learnt is that the most accurate data regarding the database comes from the database itself.

We switched to AWS RDS Insights together with PostgreSQL native pg_stat_statement extension.

I also went one step further and asked my DevOps team to redesign VPC so that every type of the component got its own subnet. Why? With these fine-grained subnets we could generate network traffic stats from the VPC Flow Logs in a nice visual way. We had a deployment diagram on top of which a simple Python script was adding text labels with the information about network traffic exchanged by all system components.

Keeping static data next to the application

I would like to finish this post with a true story. I'm running thousands of machines on a weekly basis. All those machines needed to fetch some metadata from a database. There were different metadata for different jobs. The metadata changed only when a new version was released (they were in fact static). Fetching metadata from database for every job resulted in hundreds of millions of unnecessary database calls. This is what we did:

We started with the obvious one: implemented an application cache for metadata, it helped a lot, still didn't solve the whole problem, because we still launched a few thousands of machines and the application cache for a newly launched machine was always empty;
Second idea was to offload those operations to our Redis cluster, it helped too, our database was happy, but Redis wasn't happy that much;
Third idea was to package the metadata along the application code, it was the best of all the options, we didn't have to make any remote calls to either database nor Redis, instead we read the metadata (from Java resources) into the in-memory cache;

The takeaway here is that not everything has to be put in a database. Very often the simplest solutions are the best ones, and I'm a huge fan of the KISS principle.

Why I choose Keycloak over AWS Cognito

Łukasz Budnik — Wed, 02 Jun 2021 07:54:52 +0000

I'm responsible for delivering a secure scalable multi-tenant product that is deployed on AWS. I love AWS and my preference is to use AWS managed services everywhere I can.

AWS has a Cognito service which is a fully managed service that provides authentication, authorization, and user management.

However, for a multi-tenant SaaS product, I would go for Keycloak. And in this short article, I will tell you why. I will compare AWS Cognito with Keycloak and show you why Keycloak is still a better choice for me.

Multi-tenant setup

User pool in AWS Cognito is not multi-tenant. In order to create a multi-tenant system, you have to have a dedicated user pool per tenant. If you are a SaaS product then your tenant provisioning logic would have to be extended to automate the following actions:

Create user pool: CreateUserPool
Create user pool client: CreateUserPoolClient
Create user pool domain: CreateUserPoolDomain
Optional, to set up a more user-friendly domain that will host the sign-up and sign-in web pages, you need Route53 API calls as well

That's a lot.

Now compare it with how easy it is to add a new Keycloak tenant and client from my video. Adding a new tenant and a new client are 2 really simple calls in Keycloak.

MFA setup

OK, you can set up SMS MFA very easily in AWS Cognito. When a user registers, the verification of the phone (via SMS) is provided out of the box: the web pages and verification logic is provided by AWS Cognito.

But when you actually want to use software tokens MFA (TOTP) then you have to do the following:

Add a special OAuth scope "aws.cognito.signin.user.admin" to the app client
Pass user's JWT access token to AssociateSoftwareToken API call
Generate QR code
Once user sets up MFA on the phone, pass the user code to VerifySoftwareToken API call

I was surprised that SMS verification is supported out of the box, but for software token you have to write your own integration.

That's a shame really when you compare this with Keycloak MFA. You just set it up as a part of a single user journey with all the web pages, QR code generation, and verification logic provided by Keycloak.

Again, for reference see my video where I set up and test software tokens MFA in Keycloak:

AWS Cognito limitations

AWS Cognito isn't too flexible when it comes to some of its settings. For example, these are the settings that you cannot change after you create a user pool: sign-in options, user attributes.

AWS Cognito also has a limit of 1000 user pools per AWS account. That's actually very little for a multi-tenant applications.

AWS Cognito strong points

Before I wrap up, I would like to say that AWS Cognito is still a great choice for many apps. Its strong points are:

hands-free fully managed service
support for many events which can trigger custom Lambda functions
great support for mobile apps (Android and iOS SDKs available)
integration with AWS STS which can exchange Cognito tokens for temporary AWS credentials
integration with AWS API Gateway using Cognito User Pool Authorizer
great choice for single-tenant applications

Migrating to DynamoDB using Parquet and Glue

Łukasz Budnik — Tue, 04 May 2021 14:16:47 +0000

I am preparing to migrate 1B records from AWS RDS PostgreSQL to AWS DynamoDB. Before moving 1B records I want to build some POCs using a smaller data set (101M records) to find the most optimal way of getting those records into AWS DynamoDB.

My first choice was obvious: AWS Data Migration Service. AWS DMS supports AWS RDS PostgreSQL as a source and AWS DynamoDB as a target. This can be setup very easily.

However, it took me a few iterations, trying out different instance types, different migration task parameters, to finally get to a stage where I was fully using 20000 WCU when writing data to DynamoDB.
I also noticed that AWS DMS was using PutItem operation rather than more efficient BatchWriteItem.

The fastest of my migrations tasks was:

I watched a few videos from the online AWS re:Invent about Glue and how super efficient it was in transforming and moving data around. I also knew that Glue added a support for writing to DynamoDB.

I wanted to try it out and see if it can be of any use for my migration project.

Export tables to AWS S3 in Parquet format

I used AWS Data Migration Service to export data to AWS S3 in Parquet format. Exporting table that has 101M records was quite fast and took 54m:

Glue & Glue Studio

I haven't used Glue before. In one of the AWS re:Invent videos I saw that AWS recently added Glue Studio where you can build jobs using visual editor and for more complex jobs you can add custom code.

I built a simple job in Glue Studio, but noticed that DynamoDB is not yet supported (in Studio). Doing a quick Google search I found a sample code right from the Glue documentation: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-connect.html#aws-glue-programming-etl-connect-dynamodb.

I pasted that code into my "Transformation - Custom code" block. I only had to change variables' names as Glue Studio generates code using camelCase and the documentation was using snake_case. Anyway it was just 1 operation: a call to glue_context.write_dynamic_frame_from_options() (I hope that in the near future it will be supported out of the box in Glue Studio).

And that was it. I ran the job and it took 3h 7m to complete:

In DynamoDB I saw all my 101M records:

Environment setup and summary

The environment looked like this:

VPC with VPC Endpoints to AWS S3 and AWS DynamoDB
AWS RDS PostgreSQL running in private subnet, db.m5.large
AWS DMS instance running in private subnet, dms.c5.2xlarge
DynamoDB table provisioned with 20000 WCU
AWS DMS migration task with ParallelLoadThreads set to 130 and ParallelLoadBufferSize set to 1000 was running at 74% CPU and was generating 20000 WCU
AWS Glue job had 10 workers, dynamodb.throughput.write.percent set to 1.0 was maxing 20000 WCU

The results were the following:

	DMS RDS to DynamoDB	DMS RDS to S3	Glue S3 to DynamoDB
RDS CPU	4%	11%	-
RDS Reads IOPS	100	350	-
RDS Connections	6	6	-
DMS CPU	74%	11%	-
DynamoDB WCU	20000	-	20000
Time	2h 51m	54m	3h 7m

Direct migration between AWS RDS to AWS DynamoDB using AWS DMS was 2h 51m.

Staged migration with RDS -> S3 Parquet -> DynamoDB was: 54m + 3h 7m = 5h 1m.

After all DMS was faster than the DMS + S3 + Glue trio.

As you can see in the table above, in both approaches the pressure on AWS RDS PostgreSQL instance was minimal.

There are still some optimisations I want to try, like speeding up the import to DynamoDB by setting WCU even higher. That will require some AWS DMS task parameters tuning and probably increasing the DMS instance size as well.

DMS

Pros of using DMS are:

it's fast, even with PutItem operations
one-stop shop for your migration tasks
can run additional validation logic for your migration

Cons of using DMS are:

need to rightsize the instance and tune task parameters to max DynamoDB writes

S3 Parquet and Glue

Pros of using S3 Parquet and Glue:

serverless solution - no need to manage the infrastructure
didn't have to do any job tuning, setting dynamodb.throughput.write.percent to 1.0 was all I had to do to max DynamoDB writes
you get some bonuses, think - data lake!

Cons of using S3 Parquet and Glue:

slower than native DMS migration

Bonus #1

Since I had the data already in AWS S3 in Parquet format I wanted to play around with AWS Athena. I used Athena before and already knew it was super fast, but I never had a chance to play around with 20 columns wide 101M records data set. The experience was mind blowing. Complex queries running in just a few seconds. The same queries running on the original AWS RDS PostgreSQL instance were taking 20 minutes to complete!

Bonus #2

Since I already had the data in AWS Athena database I wanted to go one step further and easily visualise the data I had using AWS QuickSight. It took me a few minutes to set up some Pie Charts and Heat maps and publish my dashboard. All by drag and dropping columns in the QuickSight UI.

Let the bots do the releases for you while you sleep

Łukasz Budnik — Thu, 08 Apr 2021 14:45:05 +0000

When you have a high test coverage or... a high confidence in your code (in my case the former) you can setup fully automatic releases on GitHub. All driven by bots.

In this post I will show you how I set it up using my own project.

migrator is a DB migration/evolution tool written in Go. It's super fast and lightweight. It is released as a docker image on docker hub: lukasz/migrator.

migrator has a code coverage of 94% and I decided to release a new docker image every time there is a new dependency available: either a new base docker image or a new version of Go dependencies.

lukaszbudnik / migrator

Super fast and lightweight DB migration & evolution tool written in Go

migrator

Super fast and lightweight DB migration tool written in go. migrator consumes 6MB of memory and outperforms other DB migration/evolution frameworks by a few orders of magnitude.

migrator manages and versions all the DB changes for you and completely eliminates manual and error-prone administrative tasks. migrator versions can be used for auditing and compliance purposes. migrator not only supports single schemas, but also comes with a multi-schema support (ideal for multi-schema multi-tenant SaaS products).

migrator runs as a HTTP REST service and can be easily integrated into your continuous integration and continuous delivery pipeline.

The official docker image is available on docker hub at lukasz/migrator. It is ultra lightweight and has a size of 15MB. Ideal for micro-services deployments!

View on GitHub

dependabot

First, you need to enable dependabot for your project. It's super easy. Just follow the instructions detailed here: Managing vulnerabilities in your project's dependencies and Keeping your dependencies updated automatically.

dependabot can create a pull request for you whenever there is a known security vulnerability in your 3rd party dependency and there is a fix version available. You can go even further and tell dependabot to create a pull request whenever there is a new version available. I like to live on the edge and I went ahead and enabled both options.

For reference here is the link to my dependabot config file: https://github.com/lukaszbudnik/migrator/blob/master/.github/dependabot.yml.

Protip: Every time pull request is merged to target branch dependabot is rebasing all created pull requests (followed by re-running all checks). I provide different times for every check so that the number of rebases is kept to a minimum.

merge-pull-requests action

The bit missing was automated merging to target branch when all tests were passing. There is an auto-merge feature available in GitHub but dependabot doesn't support it (yet).

I decided to use GitHub Action that was available on the marketplace: GitHub Actions - Merge pull requests.

I used the default configuration file provided by authors. However, the default config was quite permissive and I needed to tweak it a little bit. I wanted to auto-merge pull requests with dependencies label (all pull requests created by dependabot have it) and (to make sure other people would not abuse the auto-merge functionality) I wanted to auto-merge pulls created by the dependabot[bot] user only.

For reference here is the link to my configuration file: https://github.com/lukaszbudnik/migrator/blob/master/.github/workflows/automerge.yml

Protip: Make sure you have "Automatically delete head branches" enabled in your repository settings. GitHub will delete merged branches automatically. To keep things nice and tidy.

project-bot app

Optional step. My CI/CD pipeline instructs docker hub to build new latest tag on every merge to main branch. However, I also wanted to roll up all small changes into official releases. I use GitHub projects (rather than milestones) to organise my work. Every time I release a new version I create a new project which acts as a placeholder for all new work. To help me organise release notes better I wanted to automatically add dependabot pull requests to an open GitHub project.

I decided to use GitHub App that was available here: GitHub Apps - project-bot. In order to include a new pull request in your project just add a new card with the correct markup in one of the columns.
For reference here is my project that uses project-bot integration: https://github.com/lukaszbudnik/migrator/projects/9.

Protip: If you create your projects using "Automated kanban" template GitHub will automatically move merged pull request to Done column. So there is even less configuration required.

bots in action

Below is a link to a sample pull request which was:

created by dependabot
added to v2021.0.1 project by project-bot
merged by merge-pull-requests
built and published by dockerhub

And it all happened at 4 AM CEST while I was in a well deserved deep sleep phase :)

Bump github.com/aws/aws-sdk-go from 1.38.14 to 1.38.15 #178

dependabot[bot] posted on Apr 08, 2021

Bumps github.com/aws/aws-sdk-go from 1.38.14 to 1.38.15.

Release notes

Sourced from github.com/aws/aws-sdk-go's releases.

Release v1.38.15 (2021-04-07)

Service Client Updates

service/accessanalyzer: Updates service API, documentation, and paginators

service/elasticache: Updates service API and documentation

This release adds tagging support for all AWS ElastiCache resources except Global Replication Groups.

service/ivs: Updates service API, documentation, and paginators

service/mgn: Updates service API, documentation, paginators, and examples

service/storagegateway: Updates service API, documentation, and paginators

File Gateway APIs now support FSx for Windows as a cloud storage.

Changelog

Sourced from github.com/aws/aws-sdk-go's changelog.

Release v1.38.15 (2021-04-07)

Service Client Updates

service/accessanalyzer: Updates service API, documentation, and paginators

service/elasticache: Updates service API and documentation

This release adds tagging support for all AWS ElastiCache resources except Global Replication Groups.

service/ivs: Updates service API, documentation, and paginators

service/mgn: Updates service API, documentation, paginators, and examples

service/storagegateway: Updates service API, documentation, and paginators

File Gateway APIs now support FSx for Windows as a cloud storage.

Commits

d9428af Release v1.38.15 (2021-04-07) (#3853)
See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR
@dependabot recreate will recreate this PR, overwriting any edits that have been made to it
@dependabot merge will merge this PR after your CI passes on it
@dependabot squash and merge will squash and merge this PR after your CI passes on it
@dependabot cancel merge will cancel a previously requested merge and block automerging
@dependabot reopen will reopen this PR if it is closed
@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

View on GitHub

Building cloud native apps: Backing services

Łukasz Budnik — Wed, 07 Apr 2021 14:11:19 +0000

Backing services as attached resources

The Twelve-Factor Apps has a great summary of backing services: https://12factor.net/backing-services.

I love it and I have very little to add. That's the essence of building a modern software.

Off-the-shelf PaaS/SaaS services

I want to make one important point. Wherever possible use integrations to existing PaaS/SaaS managed services rather than trying to build various services yourself. (Does this ring a bell Building cloud native apps: Identity and Access Management?)

The Twelve-Factor App was created by Heroku team and I will use Heroku as an example. I remember when I built my first big app on Heroku back in 2012. I used a number of different Heroku addons (now called Heroku Elements Marketplace) and it was a truly mind-blowing experience. I just attached managed services to my app and used them (endpoints and/or API keys exposed via env variables). Heroku was the avant-garde of PaaS.

The beauty of Heroku platform is that you simply can't do it wrong. You don't have access to the underlying infrastructure and there is no temptation for you to host your own Elasticsearch cluster and build your own log management solution. I know: homegrown solutions may appear cheaper, but in terms of the quality can you really match a service delivered by a highly specialized company?

What many people tend to forget is that your team's time is worth more than the costs of those services. Your team should be focusing on the business value and not trying to figure out how to best implement HA for a database cluster or troubleshoot performance issues with your log management solution.

Just look at Heroku Elements Marketplace or check out AWS, Azure, GCP list of managed services. The big cloud players offer you everything you need to develop your application: Compute, Serverless, Networking, Databases, Caches, Queues, Storage, Monitoring, Log Management, Security/Governance & Compliance/Threat Detection & Prevention, Encryption, Machine Learning, Artificial Intelligence, Internet of Things, Mobile, Game Tech, Media Services, VR & AR. AWS offers even crazy services like AWS Ground Station to control your satellites.

To wrap up:

treat backing services as attached resources;
don't build them yourself, use specialized managed services.

Building cloud native apps: Config and Toggles

Łukasz Budnik — Tue, 23 Mar 2021 15:31:45 +0000

Config for cloud native apps

In the original 12-Factor App manifest Config is listed as a third factor: https://12factor.net/config.

To state the obvious: don't hardcode any configuration settings in your application. Always set it as either configuration file or env variables.

Reduce configuration settings to bare minimum. Introduce convention over configuration and store only the settings that are absolutely necessary in your config.

Be aware that configuration settings can be:

static - which don't really change, a classic example is a DB endpoint; changing DB endpoint would require the application to be restarted to pick-up the new value;
dynamic - which can be turned on and off while the application is running.

Based on the configuration type different configuration management strategies apply.

Let's review them all.

Configuration files vs. env variables

Injecting env variables is usually much simpler than injecting configuration files. With env variables you can do fine-grained changes. With files this is more difficult as you would have to store and inject the whole (sometimes large) file. On the other hand, if your application requires 30+ configuration settings, then having to manage 30+ env variables is definitely going to be a challenge. Use configuration files in this case.

There is also a middle ground. Treat configuration file like a template and inject the key configuration settings at runtime. Look at a sample lukaszbudnik/migrator configuration file:

If your tool/framework supports env variables substitution then commit your config file to the repo, package it along with the application, and inject the actual values at runtime using env variables. This way you can have the best of the two worlds.

Metadata services

If you're deploying your app to the cloud and use services like virtual machines, then you can leverage metadata endpoints. You can query those endpoints at runtime to get a lot of useful information about the machine and its cloud environment. All big players support it: AWS EC2, Azure Virtual Machines, GCP Compute.

Internal DNS service

When you deploy your cloud native apps to AWS, Azure, GCP, Kubernetes you can leverage internal DNS services.

Internal DNS is the same for all deployments. Staging, pentest, preproduction, production. It doesn't change. Wherever your app is deployed the invoices service will always have "invoices" DNS name.

API & Secret Keys

When running on AWS, Azure, GCP, do not use API and Secret Keys. IAM roles are first class citizens even in container services. Your servers, containers, and functions can assume roles meaning you don't have to inject API and Secret Keys into them. This greatly simplifies configuration management and configuration lifecycle (one obvious benefit: no need to rotate them on a regular basis).

Configuration as a Service

Where to store configuration settings? In a dedicated Configuration as a Service solution. Configuration as a Service is a secure, highly-available, durable, encrypted storage for your configuration and secrets. Every major cloud provider offers such service. Depending on your cloud provider see: AWS Systems Manager Parameter Store,
Azure App Configuration, Azure Key Vault, or
GCP Secret Manager.

The other benefit worth mentioning is that above services come with versioning out of the box. Thanks to this you have a complete history of all the changes for auditing, governance and compliance, and/or troubleshooting purposes.

If you host your Kubernetes cluster in a cloud then I would highly recommend you to use external-secrets/kubernetes-external-secrets project to sync your Kubernetes secrets with external secrets management systems (completely transparently). kubernetes-external-secrets supports the following backends: AWS Systems Manager Parameter Store, Hashicorp Vault, Azure Key Vault, Google Secret Manager, and Alibaba Cloud KMS Secret Manager.

Feature toggles

Now that we covered static configuration, let's talk about dynamic configuration. Or as it's called feature toggles.

These are the settings that can be turned on and off dynamically at runtime.

They are very useful when you release new functionality in Alpha, Beta, GA stages for initially a small group of your customers or want to release it to your design partners first, then to a wider group, and finally make it generally available to all customers.
You can use feature toggles when you want to release new functionality using canary releases too.
Finally, feature toggles are used when you give your customers option to opt-out of some features.

Feature toggles framework that we built stores all the information in DB (in contrast to using Configuration as a Service which we use for all other configuration settings). Since we store feature toggles in DB we support the following 3 levels:

DB - we use feature toggles in stored procedures;
back-end - we load feature toggles from DB and wrap them with a Java service; this can be any language you use: Java, Go, JavaScript, Ruby, Python;
front-end - we have a REST service which exposes feature toggles to JavaScript app; JavaScript app uses it to implement different behavior and/or render different components in UI.

Tenant config vs. global system config

Assume all configs and feature toggles can be changed per customer/tenant basis.

Implement a hierarchy of configs: check if tenant-specific config exists, if yes return it, if not, fallback to the global system one.

Kubernetes

Since we are talking about cloud-native apps let me finish by some Kubernetes examples.

Kubernetes ConfigMap

Kubernetes ConfigMap is used to store configuration in the form of key-value pairs and/or files. They are stored in clear-text and should not be used to store any sensitive information. For more, see the official documentation: Kubernetes ConfigMap.

We can create a configmap from a file like this:

Later, we can inject that configmap as a volume into the pod and then mount it into the container:

Kubernetes Secrets

Kubernetes Secret is used to store sensitive information. Just like ConfigMap it can be created in the form of key-value pairs and/or files. There are built-in secret types too. For more, see the official documentation: Kubernetes Secrets.

We can create a secret using yaml file or from literal, I will use the literal here:

Later, we can reference this secret when defining an env variable in a container:

Using external-secrets/kubernetes-external-secrets

The blow gist contains a step-by-step example showing how to inject AWS Systems Manager Parameter Store secrets into Kubernetes secrets using https://github.com/external-secrets/kubernetes-external-secrets.

I post it at the bottom as it's a detailed (long) example.

Building cloud native apps: Identity and Access Management

Łukasz Budnik — Tue, 16 Feb 2021 15:02:55 +0000

Identity and Access Management for cloud native apps

Every app needs identity and access management. "No problem", I hear you say. You've done it a thousand times: users table with login and password hash.

But, is it really that simple?

Here are a few pretty standard questions you will hear from your customers and their info sec teams:

how do I enforce a particular password length?
how do I enforce lowercase, uppercase, digits, special characters in the password?
how do I enforce password change every X days?
great, I can enforce password change every X days, but you don't have password history which means I can reset the password to the old one; how do I enforce password history of X?
my user forgot his password, where's the reset password functionality?
MFA is a standard for us; oh... you don't support MFA?
since you don't support MFA we have to use our SSO to login into your app; oh... you don't support SSO?
great, you support SSO using SAML, but SAML is kinda old-school... do you support OIDC?
a minor one, hope it is not too much of a hassle: how do I add my company logo and a legal statement to the login page?
hello again, we updated risk factor for you application, and we now must use webauthn passwordless secure key device to authenticate...

By now your simple users table design got a little bit more complicated.

And now imagine you are developing a multi-tenant cloud native app and all customers come with their own security requirements.

off-the-shelf and open-source solutions

Instead of throwing yourself into development (and spending days and months on reinventing the wheel) pause for a moment. Why not use off-the-self solution? Or even better an open-source solution?

Some time ago I set myself on a mission: promote using off-the-shelf Identity and Access Management solutions.

Many architects fear of integrating other solutions into their systems. I don't understand this. Write down your requirements, do the research, write down the results, review results & pick the right solution for you, and then start building your app. Should you not be happy with the solution you can always implement it on your own... but before you do this, please scroll up and take another look at the list of only a few questions you will get from your customers.

If you choose an open-source solution and it lacks a specific feature, by reading my previous post in this series Building cloud native apps: Dependencies, you already know what to do: implement it and contribute back!

Trust me, integrating with an off-the-shelf solution (either proprietary or open-source) will save you a lot of time and money compared to building IAM solution yourself.

Keycloak

Keycloak is an open-source Identity and Access Management solution. Keycloak was initially developed by JBoss community and is curated by Red Hat now. It's also the foundation of Red Hat Single Sign-On product.

I love Keycloak. Keycloak is very easy to get you started, comes with tens of features out-of-the-box, fits nicely into multi-tenant architectures, there are a few deployment options to choose from, has nice getting started tutorials for beginners, and a very detailed documentation for those more advanced.

To encourage people to use Keycloak I decided to, instead of writing a series of posts about it, record videos where I show how to (within a few minutes) setup & test various security requirements. Check them out below.

Deploying Keycloak cluster to Kubernetes

If you want to try out all the Keycloak features yourself (and not only watch the videos - which is still fine if you're doing a research I was talking about earlier) then in this video you will learn how to deploy a Keycloak cluster to Kubernetes.

Source code is available on GitHub:

lukaszbudnik / keycloak-kubernetes

Keycloak cluster deployed to Kubernetes

Scenario #1: Custom password policies and MFA

We have a customer who wants to setup the following password policies:

at least 1 uppercase character
at least 1 lowercase character
at least 1 digit
at least 2 special characters (in video I use 2 for testing purposes)
password length of 8
password history of 10
password not a username
expire password after 90 days
use custom hashing algorithm PBKDF2-SHA256 (key stretching hashing making it less vulnerable to brute-force attacks)
enforce MFA

In this video I also show how to test above settings using a sample Keycloak app. I also show how to verify JWT tokens generated by Keycloak using JWT.io.

Scenario #2: Single Sign-On using SAML

We have a customer who wants to setup Sign Sign-On using SAML. In this case the customer has full control over identity management and can enforce additional authentication factors like authentication from corporate network only, etc.

In this video I also show how to use custom SAML assertions to import user attributes into Keycloak. I'm using JumpCloud as an SSO provider. It's free for small teams.

Scenario #3: Social Identity Providers

We have a customer who wants to setup Single Sign-On using one of the Social Identity Providers. It's very convenient to use, battle-tested, backed by biggest companies. Further, customer doesn't have to introduce yet another solution into their technology stack and simply use what the teams are already using.

As we are all developers here, the following video shows how to setup GitHub as a Social Identity Provider.

Scenario #4: User Federation using LDAP

We have a customer who is an IT dinosaur. They are not SSO ready, but they have LDAP in place. Keycloak supports User Federation and can sync with LDAP directories. Once users are in Keycloak your app can talk to Keycloak and take full advantage of JSON Web Tokens or enjoy OIDC without any changes to your app.

In Keycloak you can setup User Federation in just a few clicks. See how to sync with LDAP with custom attributes mapping in less than 7 minutes! If you need a free LDAP service check out JumpCloud.

Scenario #5: Customizing multi-tenant login pages

In the below video I show how to customize login pages. In the video I show how to add a customer logo together with a legal banner to the login page. I show how to do it in a way that it can be used for multi-tenant deployments, without having to create a dedicated login page for every customer.

Scenario #6: Custom Authentication Flows

Keycloak is highly customizable. You can not only configure password policies, MFA, SSO using SAML, SSO using OIDC, customize UI themes, customize authentication flows, but you can even write Java/JavaScript code to implement custom logic in Keycloak.

In the below video I show how to customize authentication flows and deploy a custom authenticator written in Java. The authenticator will use the user IP to either force or skip MFA step (MFA will be skipped if authentication request is coming from a trusted network).

lukaszbudnik / keycloak-ip-authenticator

Simple Custom Java Keycloak Authenticator

Scenario #7: Webauthn passwordless authentication

Keycloak supports webauthn passwordless authentication out of the box. It can be anything that your browser/system supports. In the below video I show how to setup MacOS Touch ID passwordless authentication in Keycloak.

Apart from setting up webauthn passwordless authentication I also show how to customize authentication flows.

Scenario #8: Authentication for distributed apps

This is an advanced tutorial. Still, everything is fully automated, and you can try it out on your local machine.

Source code and all the steps are available on GitHub.

lukaszbudnik / keycloak-kubernetes

Keycloak cluster deployed to Kubernetes

To complement other scenarios, Keycloak is now used as a true Identity and Access Management solution: it contains information about users (identity management) and their roles (access management).

The demo comprises of:

React front-end application authenticating with Keycloak using official Keycloak JavaScript adapter lukaszbudnik/hotel-spa
haproxy acting as an authentication & authorization gateway implemented by lukaszbudnik/haproxy-auth-gateway
mock backend microservices implemented by lukaszbudnik/yosoy
Keycloak as Identity and Access Management
ready-to-import Keycloak realm with predefined client, roles, and test users

Scenario #9: Multi-tenant JavaScript Clients

An extension of the above video where I show how to use the Keycloak JavaScript adapter in a multi-tenant fashion.

Scenarion #10: Deploying Keycloak to AWS EKS

The first video from this post shows how to deploy Keycloak cluster to local Kubernetes cluster.

See how easy it is to setup a Keycloak cluster on AWS EKS.

Scenarion #11: Transparent authentication for AWS API Gateway

AWS API Gateway has a concept of JWT Authorizers that can be attached to its resources. JWT Authorizer can be anything really. As long as it follows the JWT standard.

In the below video I show how to setup Keycloak as AWS API Gateway JWT Authorizer.

AWS CDK code for deploying a sample application is available on my github account.

lukaszbudnik / aws-cdk-items-app

API Gateway HTTP API with JWT Authorizer and Lambda and DynamoDB integration.

Scenario #12: Transparent authentication for existing apps

Let's finish with a scenario where we use Keycloak to add authorization and authentication to existing apps.

Again, this is an advanced tutorial. Still, everything is fully automated, and you can try it out on your local machine.

I use my open-source project migrator as a sample cloud-native app to secure.

Source code is available on GitHub: https://github.com/lukaszbudnik/migrator/tree/master/tutorials/oauth2-proxy-oidc-haproxy.

It also uses haproxy acting as an authentication & authorization gateway implemented by lukaszbudnik/haproxy-auth-gateway.

Just like in previous video, Keycloak is used as a true Identity and Access Management solution: it contains information about users (identity management) and their roles (access management).

In this scenario haproxy is deployed in front of migrator. haproxy verifies the JWT access token and implements access control based on user's roles to allow or deny access to underlying migrator resources.

Building cloud native apps: Dependencies

Łukasz Budnik — Thu, 21 Jan 2021 15:11:29 +0000

Dependencies management for cloud native apps

Dependency management is very important aspect of every application lifecycle management. It has its own chapter in the twelve-factor app manifest: https://www.12factor.net/dependencies.

It is a very good summary of how to approach dependency management in your project (at all levels including OS dependencies). I would like to throw in my two cents and focus more on the software development part.

Keeping dependencies up to date

Make sure that the build tool which you use has the functionality which can help you keep up with your dependencies (either out of the box or via plugins). A project comprising of several services can have tens of external dependencies. Making sure you are always up to date is not a super complex task, but (let's face it) it's a rather dull task. This task should be automated. Dependencies should be upgraded automatically every week (at minimum). Followed by a full suite of unit and integration tests.

Minor version upgrades don't break the API and what is very important they contain bug fixes and security updates.

Scanning dependencies for security vulnerabilities

Speaking of security updates. You should scan your code on a regular basis for security vulnerabilities. You can use projects like Retire.js or OWASP Dependency-Check. There are also fully featured multi-platform multi-language solutions like Dependency Track and many other.

If you are using GitHub to store your code, you get some security features out of the box.
You can setup GitHub code quality scanners to find security vulnerabilities and errors in your code: https://docs.github.com/en/github/finding-security-vulnerabilities-and-errors-in-your-code/configuring-code-scanning.
GitHub Dependabot can scan your project and let you know if there are security vulnerabilities in your dependencies. Dependabot can automatically create a pull request with a bumped version for you once a new version is available. Dependabot can be also configured to create a new pull request every time there is a newer version available (not only when there's a security vulnerability). More about Dependabot can be found here: https://docs.github.com/en/code-security/supply-chain-security/managing-vulnerabilities-in-your-projects-dependencies/configuring-dependabot-security-updates.

There are much more security-related features in GitHub. For a complete list see https://github.com/features/security.

Checking dependencies' licenses for compliance

A very important step. Check licenses for all your dependencies. Check licenses for both the back-end (maven, gradle, npm, go, gems, ...) and the front-end (npm, grunt, yarn, ...). Prepare a list of approved licenses and fail all pull requests when new dependencies with new licenses are added. You can review new licenses and if you accept them, add them to the approved list. If not, reject the pull and pick-up another library.

Share the love

There are a lot of open-source foundations, government institutions, startups, tech giants, and even banks that open-source their work. Every project that I worked on used open-source technologies. And I'm pretty sure so do you.
If you found a bug, implemented an enhancement, or maybe even added a brand new feature - please contribute back. Every contribution counts and every contribution is making the difference!

If you're on another open-source level, you may even consider sharing your own project. That's how the open-source community is changing the world around us. Share the love!

Building cloud native apps: Architecture

Łukasz Budnik — Wed, 13 Jan 2021 15:06:57 +0000

Cloud native architecture

The era of monolithic applications with one big database is long gone. These applications are terrible in every aspect of the application lifecycle management. Especially in production operations and continuous maintenance & improvement.

If you are about to start a new cloud native project, it has to be microservices. There are a number of benefits of adopting microservices architecture. The most important ones are:

from the software development point of view - you will decompose your application into smaller services which follow single responsibility pattern (reduced complexity & clear design); are faster to develop as different teams can work on different services simultaneously and release them to production more frequently (and independently of others);
from the production operations point of view - your application will be fine-grained and thus have a better availability, resiliency, and performance.

You probably seen videos from Kubeconf, AWS, Azure, Google conferences where presenters talk about hundreds of services or show dependency graphs in 10% scale so that they can fit on a slide.

For most of us this is going from microservices to nanoservices.

I do have to admit that there are cases where nanoservices are the only way of making your application work. A single function becomes a service of its own (Function as a Service). Such function is then scaled independently of other functions/services. These are the applications that process billions of events an hour. These types of applications are a totally different universe and only a few lucky ones get the chance to work with them.

So, unless you absolutely have to go nanoservices route I recommend not to do it. You risk losing all the benefits of microservices. You will spend more time on managing your services (some people call it "loving your microservices"), orchestrating releases, figuring out the dependencies between them, and trying to keep track of the costs.

When you're starting architecting your application you can start with just a few services. Break your application into functionally independent and loosely coupled services. At the design stage think of what are the potential candidates for splitting your application further. If you make these decisions (or at least think about them) at the design stage, you will be well prepared to refactor your application in the future.

OK. Let's see 2 examples of architecting microservices.

Microservices from ground up

We are developing a HR cloud native system. How to architect it as a microservices cloud native app?

the front-end application is a single page application; the application is served using CDN (note: some CDNs also support serving dynamic content or even handling PUT/POST/DELETE requests which is super useful and helps reduce latency even further);
all user requests go through the API gateway;
API gateway connects to authentication service to validate requests and passes through only the valid ones;
API gateway routes traffic to one of the functional microservices where the actual data processing and data storing happens;
functional services can be: personal information management, recruitment, time-tracking, benefits, career development, etc. (you get the drill); each of the functional service can be scaled independently (probably personal information management and time-tracking are the most popular ones);
each functional service has its own database; the database is private to its service;
services can talk to each other only via API (sync) or event bus (async);
long running processes are ran by background services; these services are broken down into functional areas too; in case of our HR system this could be reporting, payroll integrations, etc.;
you must have an audit service (seriously for every cloud native app you must have it); all actions performed in the system should be recorded; there are regulations like EU's GDPR or California's CCPA and there are customers dying to ingest your audit events for their own auditing and governance & compliance purposes;
audit service should be externally accessible to your customers so that it can be integrated with customers' SIEM solutions;

Microservices by breaking monolith

A little bit tricker scenario, but also quite popular. We have an existing application. To make things worse it's a monolith application. Can we make it microservices? Yes, we can.

Say we have an application which supports URI paths like /employees, /companies, and /invoices. They take you to different parts of your application where you can list, view, create, update, and delete resources.

Splitting the code is easy. A high-level action plan looks like this:

create 3 new services by creating 3 exact copies of the existing codebase;
at the application load balancer level create 3 routing rules (/employees, /companies, and /invoices) and point them to the newly created services;
release it;
remove the logic that does not belong to given service; from employees service remove code related to companies and invoices, perform similar exercise for companies and invoices services;
release it;
any common code left should be extracted to a shared library and referenced as a dependency;
release it.

Great! We now have 3 services. However, they all talk to the same SQL database. Breaking monolith database is not as easy, but it's still not a rocket science. A high-level action plan looks like this:

create a new database config for your every service; at first they all point to the same physical database;
release it;
list all foreign keys that cross service boundaries; a classic example is user entity which usually ends up on many other entities; hopefully there shouldn’t be too much of them;
review the database access logic to catch all the places where you need to sync data in more than one service; this is an important step because from now on you will be responsible for managing the integrity of your data; in theory there should only be scalar foreign keys (which will become ordinary int/bigint/UUID columns now); also the foreign keys (which are primary keys in their original tables) never really change so there is no need to update foreign keys when updating the original entity;
release it;
drop the FK constraints; once you complete this step you will logically split the database schema;
release it;
split the data access layer into separate libraries; group data access logic by the service functionality; refactor common code to a shared library;
release it;
next move is to physically split the database; you can do this by: using fork functionality, restoring from snapshot, restoring from point-in-time, or first creating read replicas and then promoting them - choose the approach that is the best for you; remember that based on your split strategy and the size of the database this can take some time;
update database configs to point to new databases;
release it;
the last action you need to do is (similar to what we did with the code) remove the database components that don't belong to given service (like remove all invoices and companies components from employees database and vice-versa).

Release often and in small steps to reduce the risk and gain confidence in your plan.

Deploying services

Now that we know how to build microservices we need to actually deploy them.

It should come as no surprise that for microservices I would recommend Kubernetes and serverless for nanoservices.

With Kubernetes you can easily port existing applications. I would say that every application that was written in the past 15 years can be containerized. Every major cloud provider apart of having their own managed Kubernetes service, can help you move your existing applications to the cloud: AWS App2Container, Azure Whitepaper: Containerize your apps with Docker and Kubernetes, Google Migrate Anthos.
In short: if you didn't hardcode configuration inside your code nor didn't make anything silly, you are good to be containerized.

With serverless, again, every major cloud provider has its own serverless technology stack and tons of resources: AWS Serverless, Azure Serverless Computing, and Google Serverless Computing.

Building cloud native apps: Codebase

Łukasz Budnik — Thu, 31 Dec 2020 03:24:04 +0000

Everything starts with the code

Codebase is the first factor mentioned in the twelve-factor app manifest: https://www.12factor.net/codebase.

It is the very first factor everyone should consider when building a new app. Why? Because everything starts with the code.

When I started my professional IT career I was working with CVS (Concurrent Versioning System), then I was working with Subversion, did one project in Mercurial, but fell in love with Git.

Git and GitOps

Git is the undisputed king of version control systems. Git makes the dev world go round - it is the foundation of GitOps and what we can now call everything-as-code paradigm (with the most notable examples like infrastructure-as-code, build-as-code, pipeline-as-code, documentation-as-code, and security-as-code). If you're starting a new project, it must be simply Git.

Monorepo vs. Multirepo

The twelve-factor app manifest also touches on an important matter which I would like to stress as well.

Every app should have its own codebase. If you're building your cloud native app based on microservices then every service should have its own code repository (multirepo approach). Developers have a tendency to do shortcuts, if multiple apps are stored in a single repo (monorepo approach) then there is a risk of referencing other apps/components directly and breaking the clear design and component separation. Finally, having one codebase per app simplifies a lot of: code management, release management, and everything we can label as CI/CD pipelines.

Hosting

Where to host your codebase? My preference is GitHub. The list of all the available features is mind-blowing. Starting with the basic stuff like storing your code, organising your documentation/pull requests/discussion/teams/organisations/projects, running CI/CD workflows, and last but not least running security scans for your code and 3rd party dependencies. It's where I host all my projects and with (not so) recent announcement of unlimited private repos, GitHub made life so much easier. BitBucket is another alternative. Or perhaps if you're going with AWS then you could consider AWS CodeCommit and leverage the power of the AWS platform (IAM and other AWS Code* services).

Just don't host Git server yourself. There are examples of companies that did it and it didn't go as planned.

Forem: Łukasz Budnik

AI-Driven Development: My Database Experiment

The Results

Key takeaways

Want to join the experiment?

Cloud native C

Low level cloud SDKs

c + go = cgo

lukaszbudnik / cloud-native-c

cloud-native-c shows how to use AWS Go SDK from C using cgo

Source code - C

Source code - Go

Building and running

Building cloud native apps: Databases best practices

Database types

Database versioning

Caching

Connection pooling

Monitoring

Keeping static data next to the application

Why I choose Keycloak over AWS Cognito

Multi-tenant setup

MFA setup

AWS Cognito limitations

AWS Cognito strong points

Migrating to DynamoDB using Parquet and Glue

Export tables to AWS S3 in Parquet format

Glue & Glue Studio

Environment setup and summary

DMS

S3 Parquet and Glue

Bonus #1

Bonus #2

Let the bots do the releases for you while you sleep

lukaszbudnik / migrator

Super fast and lightweight DB migration & evolution tool written in Go

migrator

Table of contents

dependabot

merge-pull-requests action

project-bot app

bots in action

Bump github.com/aws/aws-sdk-go from 1.38.14 to 1.38.15 #178

Release v1.38.15 (2021-04-07)

Service Client Updates

Release v1.38.15 (2021-04-07)

Service Client Updates

Building cloud native apps: Backing services

Backing services as attached resources

Off-the-shelf PaaS/SaaS services

Building cloud native apps: Config and Toggles

Config for cloud native apps

Configuration files vs. env variables

Metadata services

Internal DNS service

API & Secret Keys

Configuration as a Service

Feature toggles

Tenant config vs. global system config

Kubernetes

Kubernetes ConfigMap

Kubernetes Secrets

Using external-secrets/kubernetes-external-secrets

Building cloud native apps: Identity and Access Management

Identity and Access Management for cloud native apps

off-the-shelf and open-source solutions

Keycloak

Deploying Keycloak cluster to Kubernetes

lukaszbudnik / keycloak-kubernetes

Keycloak cluster deployed to Kubernetes

Scenario #1: Custom password policies and MFA

Scenario #2: Single Sign-On using SAML

Scenario #3: Social Identity Providers

Scenario #4: User Federation using LDAP

Scenario #5: Customizing multi-tenant login pages

Scenario #6: Custom Authentication Flows

lukaszbudnik / keycloak-ip-authenticator

Simple Custom Java Keycloak Authenticator

Scenario #7: Webauthn passwordless authentication

Scenario #8: Authentication for distributed apps