Forem: MediaMarktSaturn Technology

Find Product Variants

Fabian Eder — Fri, 23 Jan 2026 11:22:11 +0000

How can you identify product variants in product data? You don't need to search for them manually. We have a pipeline that finds them for us. We use a multi-step approach using hard constraints, a XGBoost model and a graph clustering algorithm.

What are product variants?

Product variants are products of the same manufacturer, that differ only slightly, for example in color or size. So, as they fulfill the same customer needs, MediaMarktSaturn shows them next to each other on the product detail page.
The definition of product variants purely relies on product attributes, and not on customer interaction. For this reason, we maintain variants in our product information management system. We put products into variant groups: products that are variants of each other are part of the same variant group. One product must only be part of 1 variant group, it cannot be in multiple variant groups.

Why do we need to improve on finding product variants?

To ensure that all product variants are visible on the product detail page, we need complete variant groups for our whole assortment. In each of the 11 countries we are operating in, the size of our assortment comprises tens of thousands of products.
There, content managers maintain variant groups manually: manufacturers provide spreadsheets with products for which variant groups need to be created, or content managers search in the product management system for similar products and create new variant groups, etc. As you can imagine, this process is cumbersome and time-consuming. As a consequence, our variant groups are never complete, and they are usually not reviewed or revised after they are created.

An automated solution to find product variants

To make product variant management easier, we have created a solution leveraging machine learning techniques. It reviews all products and all variant groups in our whole assortment to

suggest new variant groups
identify products that should be added to existing variant groups
suggest changes to existing variant groups (split a group / merge groups)

The solution comprises the following processing steps:

Identify potential product variant pairs using constraints: only products from the same manufacturer and from the same product category can be variants of each other.
Get the probability for each potential product variant pair if they could be variants of each other: from a XGBoost model (see below for further details)
Create a graph from the probabilities for each manufacturer and product category
Identify subgraphs in each graph using a graph clustering algorithm: each subgraph represents a proposed variant group (see below for further details)
Compare the proposed variant groups from the graph clustering algorithm with the existing variant groups from the product information management system

After step 5, we show the results to assortment managers and content managers. If they approve the new product variant groups and the suggested variant group changes, the new data can be imported into the product management system and distributed across the whole company from there.

The XGBoost model to assign probabilities to product pairs

XGBoost (eXtreme Gradient Boosting) is a supervised learning algorithm that needs labelled and structured data as input. That means, we need tabular data with a label column and feature columns. In our case, we have a binary classification problem: products are variants of each other (positive class) / are not variants of each other (negative class). Each potential product variant pair (the result of step 1) is one observation.

For creating the label column, we make use of the existing variant groups: product pairs from the same variant group belong to the positive class, products from different variant groups belong to the negative class. Products without a product variant group are not used for model training and model evaluation.

For creating the feature columns, we have sophisticated feature engineering in place. For example, we consider the number of product attributes that the products have in common, the similarity of their product names, their similarity with respect to system IDs and system timestamps, etc.

After we have successfully trained, validated and evaluated the classifier on training, validation and test sets, we apply it to all potential product variant pairs from the whole assortment. For each product pair, we get the prediction probability of the binary classifier. The prediction probability is a value between 0 and 1, and a higher value means that products are very likely variants of each other. We use the prediction probabilities to build graphs (see below).

Graph clustering to propose variant groups

The following image shows an example graph that we created from the prediction probabilities of the XGBoost model. Each node represents one product, and each edge shows the probability that the two products are variants of each other. Shorter and thicker edges represent a higher probability. Probabilities below 0.9 are not shown.

The Louvain algorithm has split up the graph into subgraphs, as indicated by the colors. Nodes that are strongly connected are put into the same subgraph. Each subgraph represents a proposed variant group. In other terms, the graph clustering algorithm puts products into the same variant group that are very likely to be variants of each other.

For each manufacturer and each product category, we create such a graph and apply the graph clustering algorithm on it. The algorithm proposes around 800-3000 new variant groups for 2000-9000 products for each country.

Put everything together

To execute all steps above in a coordinated manner and on a regular schedule, we have implemented a Kubeflow pipeline which we run via Vertex AI Pipelines on the Google Cloud Platform. A Streamlit app shows the variant group proposals of the algorithm to the assortment and content managers, who either approve or reject the proposals. Finally, we have set up a Cloud Run Job that supports the import of approved variants into the product management system.

Summary

There is no need to do data maintenance tasks in a purely manual manner anymore. Machine learning offers so many great opportunities to support and improve business processes.

In our case, we have successfully built a pipeline on Google Cloud Platform that suggests new product variant groups and that improves existing variant groups for thousands of products. It leverages a multi-step approach using hard constraints, an XGBoost model, and a graph clustering algorithm.

What about you? Do you face similar challenges in your business and use data science and machine learning approaches to make your life easier? Feel free to reach out to us and share your experience!

get to know us 👉 https://mms.tech 👈

How to Do MongoDB Indexes

Bernd Stübinger — Fri, 01 Aug 2025 08:24:30 +0000

You just finished building an awesome feature that involves some new database queries. You have added unit tests, you carefully validated it in your test environment, and you deployed it to production, waiting for cheers and happy feedback.

Except that... your database suddenly implodes, because one of your new queries is extremely slow, taking down the whole database cluster (and your application) with it as well.

Sound familiar? Here's how we avoid it.

What Happened?

But first of all: What was the issue, actually? You tested everything, even in a real environment, right? Right, but not with a real data volume.

In this case, your query was using a collection scan because it was not able to use an index. And while this worked locally and in your test environment, data volume on production was orders of magnitude larger, resulting in significantly higher load on the database and exponentially slower response times.

Surely, you could duplicate data from production to your test environment. But not only does that come with a certain price tag, you would also have to consider data anonymization and data access, as test environments are usually less restricted - but still must not leak production data.

So what's the better way?

Requirements

To follow our approach, you need to maintain your MongoDB indexes in your application code. This feels like a natural choice when you are already maintaining your queries and data model there, and helps to avoid other problems as well. But more on that later.

Let's assume you have a simple DataMongoRepository to manage a collection of Data documents (yes, naming things is one of the two challenges in software engineering):

class Data(val id: Long, val name: String)

class DataMongoRepository(mongoClient: MongoClient) {
    private val database = mongoClient.getDatabase("database")
    private val collection = database.getCollection<Data>("datas")

    suspend fun insert(data: Data) {
        collection.insertOne(data)
    }

    suspend fun fetchById(id: Long): Data? =
        collection.find(Filters.eq(Data::id.name, id)).firstOrNull()
}

In that case, a typical test could look like this:

class DataMongoRepositoryTest {
    private val mongoContainer = MongoDbTestContainer().also { it.start() }
    private val mongoClient = MongoClient.create(
        mongoContainer.getReplicaSetUrl("database")
    )
    private val repository = DataMongoRepository(mongoClient)

    @Test
    fun `test simple insert`() {
        runBlocking {
            // Given
            val data = Data(1, "data1")
            repository.insert(data)

            // When
            val fetched = repository.fetchById(data.id)

            // Then
            assertEquals(data, fetched)
        }
    }
}

It starts an actual MongoDB via testcontainers, inserts some test data and then fetches it to see if it is still the same.

Using an Index

Now, to ensure that this query works with higher loads, let's add an index in our DataMongoRepository:

class DataMongoRepository(mongoClient: MongoClient) {
    ...

    init {
        runBlocking {
            collection.createIndex(Indexes.ascending(Data::id.name))
        }
    }

   ...
}

If we run the test again, it still succeeds (as expected, of course).

But what if we forgot to add this index? Can we somehow make this more robust?
Yes, we can!

Ensuring Index Usage

The magic lies in MongoDB's notablescan parameter. When enabled, it prevents queries from running collection scans, and returns an error instead.

The parameter can be configured using the setParameter command on the admin database:

mongoClient
    .getDatabase("admin")
    .runCommand(
        Document.parse("{ setParameter: 1, notablescan: 1 }")
    )

In our test setup (here simplified as @BeforeTest) we enable this as default for all tests, which allows us to quickly catch new queries that would try to do a collection scan, and react accordingly:

class DataMongoRepositoryTest {
    ...

    @BeforeTest
    fun setUp() {
        runBlocking {
            mongoClient.getDatabase("admin").runCommand(
                Document.parse("{ setParameter: 1, notablescan: 1 }")
            )
        }
    }

    ...
}

Let's see how this works by adding a query by name:

class DataMongoRepository(mongoClient: MongoClient) {
    ...

    suspend fun fetchAllByName(name: String): List<Data> =
        collection.find(Filters.eq(Data::name.name, name)).toList()
}

And a corresponding test:

class DataMongoRepositoryTest {
    ...

    @Test
    fun `test fetch by name`() {
        runBlocking {
            // Given
            val data1 = Data(1, "data")
            val data2 = Data(2, "data")
            repository.insert(data1)
            repository.insert(data2)

            // When
            val fetched = repository.fetchAllByName("data")

            // Then
            assertEquals(listOf(data1, data2), fetched)
        }
    }
}

Running the new test fails with an exception:

com.mongodb.MongoQueryException: Command failed with error 291 (NoQueryExecutionPlans)

Nice, isn't it?

To make that test work, we could of course add an index for the name field. But that's not always desired, so let's see how we can disable validation for a particular test:

class DataMongoRepositoryTest {
    ...

    @Test
    fun `test fetch by name`() {
        runBlocking {
            // Allow table scan
            mongoClient.getDatabase("admin").runCommand(
                Document.parse("{ setParameter: 1, notablescan: 0 }")
            )

            // Given
            val data1 = Data(1, "data")
            val data2 = Data(2, "data")
            repository.insert(data1)
            repository.insert(data2)

            // When
            val fetched = repository.fetchAllByName("data")

            // Then
            assertEquals(listOf(data1, data2), fetched)
        }
    }
}

Et voilà.

By using a Kotlin extension function we can make this even more concise:

class DataMongoRepositoryTest {
    ...

    // Runs the given [block] with `notablescan` disabled
    private fun allowTableScan(block: suspend CoroutineScope.() -> Unit) = runBlocking {
        mongoClient.getDatabase("admin").let {
            it.runCommand(Document.parse("{ setParameter: 1, notablescan: 0 }"))
            block()
            it.runCommand(Document.parse("{ setParameter: 1, notablescan: 1 }"))
        }
    }

    @Test
    fun `test fetch by name`() {
        allowTableScan {
            // Given
            val data1 = Data(1, "data")
            val data2 = Data(2, "data")
            repository.insert(data1)
            repository.insert(data2)

            // When
            val fetched = repository.fetchAllByName("data")

            // Then
            assertEquals(listOf(data1, data2), fetched)
        }
    }
}

This can be useful for e.g. complex search queries with multiple parameters, where you either cannot or don't want to create an index for every possible combination of fields. Indexes are not for free, after all - they consume memory and need to be maintained when fields are updated, incurring additional overhead.

Unique Constraint Validation

As mentioned, indexes are not only created for performance reasons. Let's get back to our example: Isn't an id actually supposed to be unique?

The following test runs without error, so obviously there is no guarantee:

class DataMongoRepositoryTest {
    ...

    @Test
    fun `test duplicate insert`() {
        runBlocking {
            val data1 = Data(1, "data1")
            repository.insert(data)
            repository.insert(data)
        }
    }
}

In fact, we now have two Data documents in our collection with id=1:

[
  Document{{_id=688b83c7bc39ec12e9b2a15a, id=1, name=data1}},
  Document{{_id=688b83c7bc39ec12e9b2a15b, id=1, name=data1}}
]

This is especially dangerous because fetchById will simply ignore all except the first document, so it may take a while to actually notice that issue.

Let's fix the problem by making our index unique using IndexOptions().unique(true):

class DataMongoRepository(mongoClient: MongoClient) {
    private val database = mongoClient.getDatabase("database")
    private val collection = database.getCollection<Data>("datas")

    init {
        runBlocking {
            collection.createIndex(
                Indexes.ascending("id"), IndexOptions().unique(false)
            )
        }
    }

    ...
}

Now the test succeeds and we can validate the expected error code:

class DataMongoRepositoryTest {
    ...

    @Test
    fun `test duplicate insert`() {
        runBlocking {
            // Given
            val data = Data(1, "data1")
            repository.insert(data)

            // When/Then
            val exception = assertFailsWith<MongoWriteException> {
                repository.insert(data)
            }
            assertEquals(ErrorCategory.DUPLICATE_KEY, ErrorCategory.fromErrorCode(exception.code))
        }
    }
}

Summary

We now have an easy way of validating database indexes, and an automated reminder that breaks our build if we accidentally add a new query without corresponding index.

Naturally, this cannot ensure that all queries use an optimal index. But while choosing the optimal index for a given query can have significant impact on performance, it is not always trivial, and you've already come a long way when you can at least guarantee that a query uses any index in order to avoid worst case scenarios.

And sometimes, all it needs is a little reminder, so there is really no reason to not include index validation in your build - especially, if it is such a trivial implementation.

get to know us 👉 https://mms.tech 👈

An Elegant Way to Solve Multi-Tenancy

Bernd Stübinger — Tue, 30 Jan 2024 16:14:46 +0000

In this article, I will show you an elegant way to introduce multi-tenancy in an already established code base for a Kotlin/Koin stack. And while the examples are specific to this stack, the general idea should be applicable to any dependency injection framework that supports qualifiers.

History

Our product initially started below the radar to support the migration of MediaMarktSaturn's webshop to the new platform. Back then, we had to replace an existing legacy promotion system by the beginning of the Black Friday season, which meant that we already had a fixed (business) scope and a fixed deadline.

Naturally, we had to cut short on some of the more technical aspects of our code base. One of that aspects was multi-tenancy, in our case different countries and sales lines - Media Markt Germany would be a different tenant than Saturn Germany or Media Markt Austria.

Of course, we knew that we had to deal with multiple tenants at some point but since the whole migration targeted mediamarkt.de only, we could easily postpone that topic until after the first go-live. We already included the relevant parts in our data model and in our API, though, so that we wouldn't have to perform a data migration and consumers of our system could keep their existing integrations.

For our implementation we used the much easier "ignorance" strategy, i.e. we didn't deal with tenants anywhere - and in the (few) places where we needed to work with a tenant, we simply hardcoded it to Media Markt Germany.

Initial Setup

We are using a pure Kotlin reactive stack with Ktor as web framework and Koin for dependency injection and application configuration. Our product manages coupons and calculates discounts for customers, i.e assesses their baskets with respect to pre-configured promotion rules.

In its essence, our setup looked like this:

val serviceModule = module {
    single { OutletRepository() }
    single { CalculationService() }
    single { PromotionRepository() }
    single { CouponRepository() }
}

You can think of a module as a space to collect all your Koin-managed components, similar to "beans" in Spring or CDI. single declares a singleton component that will be instantiated only once and re-used for every injection point.

In our routing layer we receive basket assessment requests and forward them to our main service, the CalculationService:

routing {
    val calculationService: CalculationService = get()

    get("basket-assessments") {
        val request = call.receive<CalculationRequest>()
        call.respond(calculationService.calculateBasket(request))
    }
}

The CalculationService then fetches active promotions from the PromotionRepository and applies coupons using a CouponRepository. Naturally, this setup has different requirements to multi-tenancy: While coupons and active promotions differ between country and sales line, the core calculation logic will always stay the same and also outlet master data will be identical across all tenants. Coupons are identified by their code and thus even need to be separated on persistence level, so that multiple tenants can use identical codes.

Inside our CalculationService we use get to let Koin eagerly resolve and inject the managed instance(s) of other services:

class CalculationService {
    private val promotionRepository: PromotionRepository = get()
    private val couponRepository: CouponRepository = get()
    private val outletRepository: OutletRepository = get()
    ...
}

From Simple...

When we eventually added support for multi-tenancy, we already had an established code base, so we were facing a few challenges:

We couldn't afford (and didn't want) to rebuild everything from scratch
We had lots of functions that required to know which tenant they were working with
We also had lots of tests that didn't know anything about tenants yet

So we were looking for a solution that would allow us to keep as much of our current implementation as possible while at the same time didn't force us to introduce changes everywhere. We decided to separate all data on database level already so that our tenants could work independently from each other - I've already mentioned coupon codes above, but this can be seen as a good practice in general.

The simple approach was to determine a tenant from the request, and then add a tenant parameter to each subsequent function call. Of course, that wasn't exactly... elegant.

We had some places that experimented with the CoroutineContext to transfer tenant information but there were still a lot of places (including tests) that needed to be adapted, and back then we were just starting to understand how coroutines work. Also, we had to rely on the presence of a tenant during runtime, and actually wanted to have a bit more compile-time safety.

...to Elegant

So instead, we opted for "tenant-aware" services. We divided our existing service implementations into those that would provide common functionality and those that needed a tenant. For the latter we introduced a TenantAware interface with a single property:

interface TenantAware {
    val tenant: Tenant
}

All services that needed a tenant would now implement this interface and receive a tenant in their constructor. So our module configuration changed to:

val serviceModule = module {
    single { OutletRepository() }
    SupportedTenant.values().forEach { tenant ->
        single(named(tenant.id)) { CalculationService(tenant) }
        single(named(tenant.id)) { PromotionRepository(tenant) }
        single(named(tenant.id)) { CouponRepository(tenant) }
    }
}

This way, we had e.g. an instance of PromotionRepository for MediaMarkt Germany and one for Saturn Germany, and within that instance we had reliable access to the tenant property whenever we needed it - for example, when referring to other tenant-aware services that would now resolve the tenant-specific instance from Koin:

class CalculationService(override val tenant: Tenant) : TenantAware {
    private val promotionRepository: PromotionRepository = get(named(tenant.id))
    private val couponRepository: CouponRepository = get(named(tenant.id))
    private val outletRepository: OutletRepository = get()
    ...
}

This allowed us to limit all new tenant functionality to our routing layer, and we didn't have to change anything in our business code because every tenant-aware instance would only ever talk to other instances for the same tenant:

routing {
    get("basket-assessments") {
        val request = call.receive<CalculationRequest>()
        val calculationService = get<CalculationService>(named(request.tenant.id))
        call.respond(calculationService.calculateBasket(request))
    }
}

And by adding a tenant suffix to our database collection (we are using MongoDB) we could easily achieve data separation in our persistence as well:

class PromotionRepository(override val tenant: Tenant) : TenantAware {
    val collectionName = "promotions.${tenant.id}"
    ...
}

In our tests we only had to adapt our injection points and didn't need to touch any of the actual test methods - voila! If needed, we could easily test multiple tenants by simply injecting different instances of a service.

Extension Functions

As last polish, we used Kotlin's extension functions to simplify our module definition and dependency resolution:

val Tenant.qualifier
    get() = named(id)

inline fun <reified T> Module.singleByTenant(noinline block: Scope.(Tenant) -> T) =
    SupportedTenant.values().map { tenant ->
        single(tenant.qualifier) { block(tenant) }
    }

inline fun <reified T : Any> get(tenant: Tenant) = get<T>(tenant.qualifier)

val serviceModule = module {
    single { OutletRepository() }
    singleByTenant { CalculationService(it) }
    singleByTenant { PromotionRepository(it) }
    singleByTenant { CouponRepository(it) }
}

class CalculationService(override val tenant: Tenant) : TenantAware {
    private val promotionRepository: PromotionRepository = get(tenant)
    private val couponRepository: CouponRepository = get(tenant)
    private val outletRepository: OutletRepository = get()
    ...
}

routing {
    get("basket-assessments") {
        val request = call.receive<CalculationRequest>()
        val calculationService = get<CalculationService>(request.tenant)
        call.respond(calculationService.calculateBasket(request))
    }
}

During the initial design we spent some days tweaking our multi-tenancy implementation but since then we've had no issues and are still using it without major changes.

And with our meanwhile grown understanding of coroutines, we have also been able to use our TenantAware interface for elegant tenant-aware authorization:

suspend inline fun <T> TenantAware.requireClaim(
    claim: Claim,
    block: () -> T
): T {
    // Check if current user has required `claim` for current `tenant`
    ...
}

class CouponService(override val tenant: Tenant) : TenantAware {
    fun createCoupon() = requireClaim(Claim.COUPON_CREATE) {
        ...
    }
}

Tell me what you think and especially, if you see even more potential for improving!

get to know us 👉 https://mms.tech 👈

One Chart to rule them all

Florian Heubeck — Tue, 30 Jan 2024 09:59:30 +0000

Describing Kubernetes deployments having best practices applied is quite verbose and maintenance intensive. Doing that for multiple applications in various stages by different teams results in lots of yam(l)mer. Helm charts provide a convenient solution. For the best deployment experience, we're maintaining and using a generic application chart.

Resources over Resources

Operating applications on Kubernetes beyond basic tutorials requires a few resources:

the Deployment itself
a ServiceAccount most of the time
one to many ConfigMaps as well as Secrets
Service and Ingress if someone shall interact with your deployment
often a [Horizontal|Vertical]PodAutoscaler referring the deployment

As we're adopting GitOps practices for quite some time with great success - you may have heard that 😇 - the manifests of all those resources have to be literally written into a Git repository.

Mentioning GitOps... for automating rollouts there are even more (custom) resources required in order to instruct our GitOps controllers (Flux & Friends):

the ImageRepository targeting the container registry
a ImagePolicy declaring version patterns to automate
and eventually a Canary as no one wants to observe rollouts manually

Observation - another great topic: To announce applications to the Prometheus monitoring stack, a ServiceMonitor resource comes on top.

Well... over time you'll probably add a PodDisruptionBudget or want to have a service mesh in place, adding furthermore resources, for instance DestinationRules and VirtualServices in case of Istio.
Wait - with Istio in place, the Ingress resource gets replaced by a VirtualService.

As we're constantly evolving, our added service mesh may get superseded by the Kubernetes Gateway API, introducing Gateway or HttpRoutes in favor of the Istio CRDs.

Guess you got the point. Writing all those manifest isn't even the hardest part. They need to be maintained constantly. Kubernetes APIs come and go, with some version lifecycle in between.

Scaling pains

Of course, some team has written all those resource manifests for the first application that was deployed. And they quickly put them into a Helm chart when it was about replication to multiple stages. The Helm chart could also be reused for further applications of that team, because - honestly - all applications ("microservices") are looking very similar.

To ensure, that changes on templated manifests don't break everything immediately, the team linted their Helm chart using different tools, hinting on upcoming Kubernetes API changes, or recommending best practices.

That's all nice and works out well. If there weren't other teams doing exactly the same but different. After a while, there were countless charts around, that all served the same purpose, but varied only slightly.

Like everywhere else, reuse is practiced by duplication, but we weren't able to copy solutions, because of the minor differences in the description of our application declarations.

Combine and generalize

So we did what was the obvious solution: We took the best from all those Helm charts and brewed a generic application chart.

This chart was even our first open sourced component. It describes a typical application workload, incorporating the requirements of many, many product teams - and guess what: they're all nearly the same.

Our application chart is tested against the latest seven Kubernetes versions (1.23 to 1.29 at the time of writing) in lots of variations using the Helm chart-testing project. Each permutation is also validated using a specialized Kubernetes manifest linter for best practices and security hardening.

Every time, someone requires something not yet possible, the chart is improved for the benefit of all. In the meanwhile, there's quite an impressive feature list, which you can easily study by reading its values.yaml - being also the documentation of the chart. Some highlights are:

Init container and sidecar configuration without hassle
Google Kubernetes Engine (GKE) support on workload identity and ingress config
Automated canary deployment support using Flagger, implemented using:
Service meshes (Istio or Linkerd) as well as the Gateway API
Various configuration, secret and volume mount options
Job execution before rollout for preparation tasks like database schema updates

All in all, the following [custom] resource manifests are managed by the chart (at the time of writing):

flagger.app Canary
image.toolkit.fluxcd.io ImagePolicy
image.toolkit.fluxcd.io ImageRepository
cloud.google.com BackendConfig
networking.istio.io DestinationRule
networking.istio.io VirtualService
ConfigMap
apps Deployment
autoscaling HorizontalPodAutoscaler
gateway.networking.k8s.io HTTPRoute
networking.k8s.io NetworkPolicy
policy PodDisruptionBudge
batch Job (as pre-install,pre-upgrade Helm hook)
Secret
ServiceAccount
Service
monitoring.coreos.com ServiceMonitor

Needless to mention, that the highest API version available is also respected, so full compatibility to all supported Kubernetes releases is ensured.

Sample time

As stated earlier, this chart, which we're using for a wide spectrum of different types of application on different Kubernetes versions, is available for everyone.

The chart can be installed using the Helm cli

helm repo add mms-tech https://helm-charts.mms.tech

or Flux

---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  name: mms-tech
  namespace: default
spec:
  interval: 120m
  url: https://helm-charts.mms.tech

To give you an impression on its usage, this is how our (well-known and beloved) GitHub app Technolinator is configured:

apiVersion: helm.toolkit.fluxcd.io/v2beta2
kind: HelmRelease
metadata:
  name: technolinator
  namespace: app
spec:
  chart:
    spec:
      chart: application
      version: "~1"
      sourceRef:
        kind: HelmRepository
        name: chart-repo
        namespace: default
      interval: 60m
  interval: 10m
  values:
    image:
      repository: ghcr.io/mediamarktsaturn/technolinator
      tag: 1.48.11 # {"$imagepolicy": "app:technolinator:tag"}
      tagSemverRange: "~1"
    secretEnvFrom:
      - technolinator-config
    resources:
      requests:
        cpu: "1"
        memory: 10Gi
      limits:
        cpu: "6"
        memory: 35Gi
    container:
      port: 8080
    livenessProbe:
      path: /q/health/live
    readinessProbe:
      path: /q/health/ready
    monitoring:
      serviceMonitor: true
      metricsPath: /q/metrics
    podSecurityContext:
      runAsUser: 201
      runAsGroup: 101
      fsGroup: 101
      fsGroupChangePolicy: Always
    volumeMounts:
      - name: data
        pvcName: technolinator-data
        mountPath: /data
    configuration:
      APP_ANALYSIS_TIMEOUT: 120M
      APP_PROCESS_LOGLEVEL: DEBUG
      APP_PULL_REQUESTS_CONCURRENCY_LIMIT: 10

Again: All features are explorable in the values.yaml.
We will be happy, if this chart could also help others. And we're welcome contributions of any kinds for further improvements and use cases that we have not yet considered.

The end

There's a very divisive discussion about such generic (monster/mega) charts. Sure, unique applications, provided by their projects have their own charts, and it doesn't make any sense to squeeze them into a generic one.

But in our opinion it's different for internal business applications of a company. Using our generic chart is a matter of hardening, reuse and easy knowledge sharing on application deployment configuration. The first one demanding a certain setup crafts it once, for everyone's benefit.

Especially when it comes to repetitive complex configurations like sidecars used for integration with databases or database schema update jobs, there's no need to reinvent the wheel over and over.

Reach out (for instance using GitHub issues or discussions), if you'd like to discuss with us.

get to know us 👉 https://mms.tech 👈

Software Supply Chain Awareness at Scale

Florian Heubeck — Tue, 30 Jan 2024 09:57:22 +0000

Securing the software supply chain is hip nowadays.

About Dependencies

You know these jokes about Windows users fearing updates, MacOS disciples paying updates and Linux nerds awaiting updates?
I've always loved updates, even long before I became a Linux user. New versions ship new features, fix bugs, look better (sometimes), run faster (hopefully) and recently: provide security.

Some eye-opening samples from the newer history raised awareness, be it well-intended features becoming exploitable like with "log4shell" or npm packages hijacked like "coa".
It's obviously not only necessary to keep dependencies up-to-date, but also to observe them for potential vulnerabilities.

Software Bill of Material

To achieve supply chain security there's, first of all, need for taking inventory. We need to know which components we are depending on.
That's what the so called SBOM is about. It's the ingredients a system is made of, with lots of metadata like checksums or licenses attached.
There are two major formats out there, "SPDX" driven by the Linux foundation, and "CycloneDX" by OWASP.

OWASP Projects

The CycloneDX standard glares with better tool support, mainly because the OWASP provides lots of tooling around it. There are plugins for nearly every programming language or build framework to create an SBOM file. And there's a kind of orchestration tool named cdxgen that recently joined the official CycloneDX umbrella.

Cdxgen wraps around ~30 different things (at time of writing) to create a SBOM for nearly every project by a single command.
That's very useful for a heterogenous project zoo like in an enterprise environment.

Another great tool provided by OWASP is Dependency-Track which consumes SBOMs, visualizes projects and dependencies and makes them searchable.
But that's not all: Dependency-Track also consumes vulnerability databases from public as well as commercial services and matches CVEs to the dependencies of our projects, AND scans artifact repositories for updates of our dependencies 🤯.

That provides us with risk assessment at a glance together with precise pointers where to take action.

Connecting Ends

Building a holistic inventory of all dependencies for an entire organization would require each and every project to create and upload a SBOM. While this is of course possible, it's not very efficient and comes with some management overhead of distributing Dependency-Track API keys.

That's where our GitHub app Technolinator comes in. GitHub apps can be installed on organization level and instrument all contained repositories.
Technolinator wraps around cdxgen and gets notified on GitHub push events. For every update on a repository's default branch, a fresh SBOM is created and uploaded to Dependency-Track:

Feedback about the process is provided as commit status:

Using this approach we are able to efficiently create an inventory of all our dependencies, and in addition provide all teams some insights on potential risks.

Hello Open-Source World

Since our Dependency-Track installation shows thousands of open-source projects, and this solution of course also makes use of awesome projects, we are happy to announce that Technolinator became the first open-source software project we provide as MediaMarktSaturn Technology, together with a public Helm chart repository that contains our Dependency-Track configuration as publicly available chart.

Just have a look into Technolinator - whose name has no meaning beyond its beautiful sound. It's easy to adopt to your needs, its documentation contains already everything to get started, just start a discussion or file an issue to get your questions answered, if any.
We hope to help other organizations solving software supply chain issues using Technolinator as well, and of course welcome any contribution.

get to know us 👉 https://mms.tech 👈

Reliable Application Deployments in a GitOps Setup with Flagger

Florian Heubeck — Tue, 30 Jan 2024 09:51:06 +0000

The ultimate goal of every GitOps setup is complete automation. For a reliable, headless application rollout, the perfect supplement to our Flux managed Kubernetes resources is Flagger. In this blog, you will learn how to not worry about application deployments at any time, even on Fridays.

This article is the second of two accompanying articles to my talk about this topic on the Mastering GitOps conference.

Having continuous delivery in place means, that new versions (that passed the fully automated continuous integration (CI), of course) automatically get deployed - ideally also to production.
In the related blog article about how to monitor and harden the GitOps setup itself, our goal was to reduce risk of failure and build alerting for actual errors, so that no one has to actively monitor the rollout of changes.

All of this already relaxes operation, but anyhow we may be forced to react immediately to mitigate customer impact in case of an incident.
In this way, we want to protect our business applications even better against failures caused by changes.
There are lots of deployment strategies out there to ensure smooth changes and early problem discovery - and for applying them in our Kubernetes GitOps setup in an automated way, we're using Flagger.

Deployment sources of error

A typical business application may fail in production for mainly two reasons:

Configuration errors
Software problems

The first is obvious: Configuration is different for each environment. And most of the time, an application will face the production config first when hitting production. Adopting GitOps properly, your application configuration is hold under version control and passes the four-eyes-principle before getting applied - this reduces risk of error considerably.

Software problems indeed should be discovered far before production, you would say. True, for sure. But true as well is, that there's no place comparable with production. Data, stored and (possibly) migrated over years or decades. Traffic profiles or event series, impossible to create artificially during CI. And of course: Users. Creative and ingenious. Say no more, we're all doing our best to ensure quality, but we have to handle the unforeseeable.

When doing changes to our systems, we should apply them carefully, maybe run some validations and possibly even give them a circumspect try before rolling out completely.

Application shadows

So far, I probably haven't told you anything new. The relevant part is, that we don't want to do any of that manually. Why investing in universal automation, but risking the most important step to fail, requiring manual remediation.

Let's have a look to a typical setup in our context:

There's a deployment wrapping our business application, ConfigMap and Secret are mounted for injecting configuration. Most likely there's a HorizontalPodAutoscaler (HPA) or a ScaledObject (SO) definition as well.
The application publishes metrics, scraped by Prometheus, and of course there are metrics available from the ingress and Kubernetes itself as well.

On every update on the deployment declaration, Kubernetes runs a rolling update by default. That means, new pods are created and when they become ready, old pods are terminated. The insurance for well-being of the software has to be built into the health probes.

When configuration changes, its application depends on the way it's used. Environment values only change on pod creation or restart. File mounts are actually updated ad-hoc, but the application also needs to re-initialize them.

Since our application manifests are anyhow packaged as Helm chart, we can trigger a re-deployment on configuration change by annotating checksums of ConfigMap and Secret into the pod template of our deployment:

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    metadata:
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
        checksum/secret: {{ include (print $.Template.BasePath "/secret.yaml") . | sha256sum }}

Now, we're prepared for introducing Flagger. Flagger is configured by a so called Canary CRD that instructs it to care about a defined target, in our case the deployment:

Basically, Flagger duplicates this target, its mounted resources and autoscaler objects referenced by the Canary CR.
Eventually, when the copy became ready, Flagger instructs the Ingress to route the traffic to the copy, the so called "primary" deployment.
The last step of this initialization phase is to scale down the original deployment, now called "canary".

This seems to be very complicated, what do we gain from it?
First of all some psychological safety 😉 - all the traffic targeting our application hits a copy of our original definition. Changing the definitions has exactly zero immediate impact to our users. And don't worry, the complexity will disappear as it's transparently handled by Flagger.

The cycle of deployment life

Are you aware on how heavily frequented road or rail bridges are replace? There's a new bridge created alongside the old one and the traffic shifted over after it's complete.
The additional bridge might be a temporary one, to free up the old bridge for their reconstruction.
This is exactly how Flagger works.

On changes to the Kubernetes manifest of our Deployment, Flagger reacts by scaling it up, wait for it to get ready and runs predefined actions.
These actions are basically HTTP calls along the process of rollout. With that, Flagger can trigger for instance automated tests, validating the new deployment. And it's not only firing these webhooks, but interpreting the response code. Flagger will not proceed with the next step of the rollout, if configured webhooks did not succeed.
We're using those webhooks for request generation on non-production environments to simulate application upgrade under load.

Of course not every application is only serving requests but doing batch processing or connecting to event sources. Also these kind of deployments can be progressively rolled out using Flagger. As requests cannot be routed by modifying Ingress rules in this case, Flaggers webhooks have to be used for guiding the application through the deployment lifecycle.

Eventually, Flagger will synchronize its copy of the Kubernetes resources, completing the rollout, and scale down our deployment of origin again. This deployment strategy is referred to as blue/green and already lets us sleep very well, hence any issue with the change to be rollout out, will cause an abortion, no users will be impacted.

There are metrics provided from Flagger reporting the state of all Canary definitions, which we can use for alerting: flagger_canary_status.
We've configured a PrometheusRule that triggers a chat message containing some useful information about the failed deployment:

As you can see, there is also a notification from Flagger itself included, as it can not only trigger webhooks, but also "alert" humans.
The "CanaryRollback" rule is defined as follows and uses some custom metrics of our application to provide more details (yeah, we're proud of that PromQLs 😇):

---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
spec:
  groups:
    - rules:
        - alert: CanaryRollback
          expr: flagger_canary_status > 1
          for: 1m
          labels:
            severity: critical
          annotations:
            summary: "Canary failed"
            description: >
              Canary deployment of version
              {{ with query (printf "max_over_time(promotion_service_major_version{job=\"%s-canary\"}[2h])" $labels.name) }}{{ . | first | value | humanize }}{{ end }}.{{ with query (printf "max_over_time(promotion_service_minor_version{job=\"%s-canary\"}[2h])" $labels.name) }}{{ . | first | value | humanize }}{{ end }}.{{ with query (printf "max_over_time(promotion_service_patch_version{job=\"%s-canary\"}[2h])" $labels.name) }}{{ . | first | value | humanize }}{{ end }}
              to {{ $labels.name }}.{{ $labels.exported_namespace }} failed.
            canaryVersion: '{{ with query (printf "max_over_time(promotion_service_major_version{job=\"%s-canary\"}[2h])" $labels.name) }}{{ . | first | value | humanize }}{{ end }}.{{ with query (printf "max_over_time(promotion_service_minor_version{job=\"%s-canary\"}[2h])" $labels.name) }}{{ . | first | value | humanize }}{{ end }}.{{ with query (printf "max_over_time(promotion_service_patch_version{job=\"%s-canary\"}[2h])" $labels.name) }}{{ . | first | value | humanize }}{{ end }}'
            primaryVersion: '{{ with query (printf "max(promotion_service_major_version{job=\"%s-primary\"})" $labels.name) }}{{ . | first | value | humanize }}{{ end }}.{{ with query (printf "max(promotion_service_minor_version{job=\"%s-primary\"})" $labels.name) }}{{ . | first | value | humanize }}{{ end }}.{{ with query (printf "max(promotion_service_patch_version{job=\"%s-primary\"})" $labels.name) }}{{ . | first | value | humanize }}{{ end }}'

The proof of the pudding is in the eating

Our updated application seems to feel quite well in production by now, but why throw it into the cold water of production traffic.
As you may have guessed from the CRD name "Canary", Flaggers main purpose is the automated progressive rollout of applications.
For making use of that functionality best, let's give Flagger fine grain traffic control by adding a service mesh:

Flagger supports all major service mesh implementations and can also perform progressive rollouts using the most common ingress gateways. I've created a playground setup using Linkerd some while ago, which gives a first hands-on experience in an easy way.

With a service mesh or an ingress gateway supporting weighted routing, Flagger can route sparse traffic to our updated canary deployment and increase the amount stepwise according to the Canary definition. But Flagger not only opens the floodgates slowly, giving our new deployment a chance to warm up, it also cares about our applications' wellbeing.

Flagger analyses metrics provided by the service mesh or the ingress gateway to decide for further rolling out the new application, or better rolling back. Lots of metric definitions are already built in, and selected according to the chosen mesh or ingress implementation, but custom metrics can easily be considered as well.
If there are issues with the changed canary version, only a little amount of requests are affected, and according to the configured analysis configuration, it's rolled back quickly.

The key is, to properly express the relevant aspects in metrics, by the application itself, but also from infrastructure components like databases (e.g. open connections) and especially service mesh or ingress components, that can provide the client view on your application.
Depending on how you consider your application working great, business metrics from the application could make sense, but also comparing metrics like "the updated application must not be slower than the old one", within some tolerance:

---
apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
    name: request-duration-factor-pfifty
spec:
    provider:
        type: prometheus
        address: http://prometheus-operated.monitoring:9090
    # Factor of P50 request duration canary to primary
    query: |
        histogram_quantile(0.50,
          sum(
            irate(
              istio_request_duration_milliseconds_bucket{
                reporter="destination",
                destination_workload_namespace="{{ namespace }}",
                destination_workload="{{ target }}"
            }[{{ interval }}]
        )) by (le))
        /
        histogram_quantile(0.50,
          sum(
            irate(
              istio_request_duration_milliseconds_bucket{
                reporter="destination",
                destination_workload_namespace="{{ namespace }}",
                destination_workload="{{ target }}-primary"
            }[{{ interval }}]
        )) by (le))
---
apiVersion: flagger.app/v1beta1
kind: Canary
spec:
  analysis:
    metrics:
      - name: p50-factor
        templateRef:
          name: request-duration-factor-pfifty
        thresholdRange:
          max: 1.1

Everything is better in graphics

In the beginning, I said, no one wants to observe changes getting rolled out well, but rely on automation and alerting.
Call me a pretender, but even with all of this in place, and the confidence nothing bad will happen, I still like to watch changes getting introduced to production:

These Grafana panels visualize, how traffic gets shifted from the primary deployment (Flaggers copy of the previous version) - above the zero-line, to the updated canary deployment - below the zero-line, and back.

We were asked, why not interchanging roles of primary and canary after the traffic got shifted completely. We're not aware of an official statement by the Flagger team, but my guess is: For simplicity. For being able to having the canary become the primary after successful analysis, another abstraction would be required. The way Flagger works allows us, to just work with the plain Kubernetes app resources.

As you see in the traffic weight panel above, we shift 100% of the traffic to the canary before Flagger synchronizes its primary copy. This is not necessary and can be done at any weight - but it gives use some benefits:

No sudden traffic impact on the newly created pods of the updated primary deployment
Slow warm up of the pods, either implicit (built-in) or explicit (using Flagger webhooks)
Better resource utilization

Some words about the latter: Lets assume our application has a high resource demand, because it's heavily scaled out. During a Canary rollout, this demand would double, what might be a waste of resources, or even not possible because of a restricted cluster size. With separate autoscaling configurations (HPA/SO), the canary can extend its resource demand during the traffic shift, whereas the primary can shrink. This way the overall resources don't exceed the standard Kubernetes rolling upgrades.

Sleep well

When starting with Flagger, you need to be aware that the new and the old versions of your application will run in parallel for some time and that both will handle traffic or process data.
This can cause issues with database schema updates or server-side client state. Flagger supports conditional traffic routing like for A/B testing, but you might be restricted in which extend you do progressive rollouts.
Regarding database or API schema changes: During a default Kubernetes rolling upgrade, new and old application versions are running also in parallel, this holds the exact same issues, albeit in a shorter time frame.
Another variation of progressive delivery especially for frontend applications, where randomly providing new and old application version may have user impact, can be achieved using session affinity. In this configuration, Flagger will use cookies for sticking to the new application version once a user hit it.
By now, there's no support for StatefulSets in Flagger, but it is considered. This might extend the opportunities of adopting Flagger to more picky workloads like databases.

Retrospectively, Flagger already saved us from some outages, and there are lots of ideas to explore on how to adopt it to non-classic request/response driven workloads. In any case we are reckless to commit changes in the Friday evening.

get to know us 👉 https://mms.tech 👈

Monitoring and Hardening the GitOps Delivery Pipeline with Flux

Florian Heubeck — Tue, 30 Jan 2024 09:50:24 +0000

The ultimate goal of every GitOps setup is complete automation. For being able to operate a system hands-off, its monitoring and alerting has to be reliable and comprehensive. In this blog, you will learn how to monitor a FluxCD operated GitOps setup on Kubernetes.

This article is the first of two accompanying articles to my talk about this topic on the Mastering GitOps conference.

Update July 2025: Article was reworked for my CloudLand Festival talk and is up-to-date for Flux 2.6

About automation

With GitOps we try to eliminate any manual task from infrastructure handling and application operation. (Virtual) infrastructure is described declaratively and rolled out by Terraform or similar tools. Application manifests are templated and bundled as Helm charts or Kustomizations. And when using GitOps operators like Flux, all of these are sourced from VCS repositories (mostly Git) and pulled directly or indirectly (e.g. via Helm repositories or OCI registries) to Kubernetes.

There's nothing left to do for us but declare and observe, is it?
Well, when everything around configuration and application delivery is automated, so that no one has to get their hands dirty, there's no reason to stop automating after Kubernetes resources got (potentially) created.

In our early GitOps days back then in 2019 using Flux 1, we spent much time screening logs to find the reason of "nothing happened after change". There was Prometheus in place - then as now - but we were not happy with the provided default metrics. So we created a Kubernetes-Operator/Prometheus-Exporter thing, that attached to the log stream of chosen workloads to derive metrics from patterns applied to the log text. That was quite fun to build, but absolutely not our core business.
Fortunately the Flux stack nowadays has monitoring and alerting capabilities built in, exceeding the imagination of our former self.

Back to topic: When we're on-call duty, there should be no reason to do anything else but carry our mobile phones around. And during daily business, providing new application versions or configuration changes, it must not be necessary to observe their rollout. Any issue has to be reported, or even mitigated, by the system proactively. Silence means "everything's fine", not: "no idea what's happening".

As with GitOps there's no classic delivery pipeline that may fail, but continuous reconciliation of current and desired state, the main objectives of monitoring and hardening the setup are:

Validate desired state changes upfront
Detect and report any persistent deviation of current from desired state
Reduce the possible blast radius of problems

The first topic refers to automated checking and linting of changes targeting the desired state. That's vital, but concrete implementations are very much dependent on the setup itself. On pull/merge-requests, we use a combination of generic YAML linting, validation of Kubernetes manifests utilizing checkov and Helm chart-testing, and where possible real installations on Kind.

Subsequently, we'd like to focus on the aspects of monitoring and alerting at that point of time after configuration changes have been applied.

Flux resources and their status

Let's start with a brief overview on Flux' components, its custom resource definitions (CRDs) and their relations:

As very common, Kubernetes operators are configured by their appropriate custom resources (CRs). Hence Flux - aka the "GitOps Toolkit" - is composed from multiple operators they even exchange configuration via CRs. Not all relationships are visualized in this diagram as that's not required for its purpose.

One of the many things that I really like about Flux is its consequent and verbose reflection of errors as part of the Kubernetes resource status and events. All operators write back the result of their actions to the respective CR.

A properly fetched Helm chart may look like:

Whereas a failed Helm release shows details on its status:

In addition to the build-in readiness of Flux' CRDs, more health checks can be included into the main CR of a Flux setup, the Kustomization:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: examiner
spec:
  # (...)
  healthChecks:
    - apiVersion: helm.toolkit.fluxcd.io/v1
      kind: HelmRelease
      name: examiner
      namespace: default
    - kind: Deployment
      name: examiner
      namespace: default

On reconciliation of a Kustomization, it gets ready by itself only if all referred healthCheck resources got ready, errors provided in its status:

The health checks are evaluated on every reconciliation of the Kustomization, so that the Kustomization will become un-ready if their fosterlings get ill. That's also reflected in its status, for instance:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
status:
  conditions:
  - lastTransitionTime: "2025-06-24T10:35:59Z"
    message: 'Applied revision: main@sha1:2f41be248656587d52d0ddd4ead813df50cb5ada'
    observedGeneration: 2
    reason: ReconciliationSucceeded
    status: "True"
    type: Ready
  - lastTransitionTime: "2025-06-24T10:35:59Z"
    message: Health check passed in 5.066648877s
    observedGeneration: 2
    reason: Succeeded
    status: "True"
    type: Healthy

Updating the diagram from above, there's a status on every CR, possibly propagated among them, helping us in analyzing issues:

Monitoring and reporting resource statuses

Given the knowings about resource statuses, we can automate the feedback on problems.

Our usual monitoring setup is made of the Kubernetes Prometheus stack for metrics collection and alerting rules, and Grafana for visualization.
Both of them are fed by Flux with resource status information and reconciliation events.

There's a bunch of metrics exposed by Flux' components itself, like gotk_reconcile_duration_seconds_bucket holding latency of reconcilation by resource kind, used to observe Flux' health and its connectivity to sources like Git or OCI services.

Metrics about the custom resource statuses formerly were exposed by Flux itself as well, but in the meanwhile are externalized to components like the kube-state-metrics exporter.

The repository flux2-monitoring-example holds a configuration for the kube-prometheus-stack as well as Grafana dashboards to collect and visualize everything around Flux' CRDs. I'm not replicating the config here, but the samples shown are using it like-for-like without further customization.

This information can easily be included into your own Grafana dashboards, like shown in the dashboards provided by Flux:

That's for observability, but we're even more interested in alerts in case of any reconciliation issues. Using the Prometheus Operator for managing our Prometheus components, it may look like this (in its simplest form):

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: flux-resources
  namespace: monitoring
spec:
  groups:
    - name: readiness
      rules:
        - alert: ResourceNotReady
          expr: 'gotk_resource_info{ready!="",ready!="True"} == 1'
          annotations:
            description: '{{ $labels.customresource_kind }} {{ $labels.name }} in namespace {{ $labels.namespace }} is not ready'
          for: 2m
          labels:
            severity: critical

During the reconciliation process, this alert will become "pending", so the for duration has to be chosen slightly longer than your resources reconciliation lasts. For quicker reactions, multiple rules with different selection criteria may be necessary.

The labels of the gotk_resource_info metric provide all information required for determining the severity and maybe also the notification channel of a certain alert:

Usually, we're using different escalation paths depending on the severity of an alert. Everything of interest, that's not indicating an active incident is routed to team-internal chat, sometimes separated by technical or business relevance. Alerts that indicate customer impact and potentially affect multiple systems are routed to Opsgenie (on-call management system) causing service-degradation announcements and notifying the on-call duty person.

Prometheus Alertmanager offers a lot of integrations like for Opsgenie, and what's not supported first class can be connected with 3rd party components, for instance Prometheus MS Teams. Alerts from the PrometheusRule example above dispatched to MS Teams can look like:

Integrated observability

Although this already provides us with most of what we need for being able to react on problems, there are even more great features giving more insights, integrated into the tools we're using anyhow.

There's another Flux component alongside the others that emits detailed information about what's happening with the Flux managed resources: the notification controller:

It gets configured using Provider and Alert CRDs and supports a wide range of integrations, of those I'd like to show you my favorites.

Every change in our system originates from Git commits. The effects of a commit can be provided to the Git management system as commit status. The following example targets GitHub, but all major providers are supported.
Because of the different expectation of secret structure there's the need for an access token in addition, that is allowed to write commit statuses. This token is contained in the referenced secret, please see the Flux docs for more details:

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: github
  namespace: flux-system
spec:
  type: github
  address: https://github.com/MediaMarktSaturn/software-supply-chain-security-gitops
  secretRef:
    name: github
---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: commit-status
  namespace: flux-system
spec:
  providerRef:
    name: github
  eventSeverity: info
  eventSources:
    - kind: Kustomization
      name: '*'
      namespace: flux-system
    - kind: Kustomization
      name: '*'
      namespace: mms-system

With this configuration, a commit status is attached for every Kustomization listed in the eventSources. The Alert CRD is used for all kinds of notifications, it contains the superset of features of the notification controller. For commit status notifications, only Kustomizations events can be used, as only those carry the commit reference.

This example configuration will result in commit status in GitHub looking as follows:

The description contains a summary of the Kustomization status and may already point to error causes.

Another very nice feature makes use of Grafana annotations. With a different provider, the notification controller creates annotations using the Grafana API for being included in any dashboard, indicating what has changed at a certain point in time:

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: grafana
  namespace: monitoring
spec:
  type: grafana
  address: "http://prometheus-operator-grafana.monitoring/api/annotations"
---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: grafana
  namespace: monitoring
spec:
  providerRef:
    name: grafana
  eventSeverity: info
  eventSources:
    - kind: GitRepository
      name: '*'
      namespace: flux-system
    - kind: Kustomization
      name: '*'
      namespace: flux-system
    - kind: HelmRelease
      name: '*'
      namespace: app
    - kind: HelmChart
      name: '*'
      namespace: app

As you can see, there's support for all event sources, depending on the concrete source there is different metadata shown, commit hashes for GitRepository or Kustomization, but Helm chart version for HelmRelease.

Including annotations (that are basically just time markers) in any dashboard is quite easy:

But results in impressive vertical red lines in all panels with a time axis:

This way, all the visualizations are just connected right to the change events of the Kubernetes resources, causalities never will be overseen anymore.

Push notifications

There's a long list of "providers" delivering Flux events including a generic webhook that enables any kind of custom implementations.

Just to pick another common example, this is how to keep informed via MS Teams on everything happening:

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: msteams
  namespace: monitoring
spec:
  type: msteams
  secretRef:
    name: msteams-webhook
---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: msteams
  namespace: monitoring
spec:
  providerRef:
    name: msteams
  eventSeverity: info
  eventSources:
    - kind: GitRepository
      name: '*'
      namespace: flux-system
    - kind: Kustomization
      name: '*'
      namespace: flux-system
    - kind: HelmRelease
      name: '*'
      namespace: app
    - kind: ImagePolicy
      name: '*'
      namespace: app
    - kind: HelmRelease
      name: '*'
      namespace: dtrack
    - kind: HelmChart
      name: '*'
      namespace: dtrack
    - kind: OCIRepository
      name: '*'
      namespace: mms-system
  exclusionList:
    - "Health check passed.*"

Resulting in messages like this:

Indicating an error

Indicating a successful change

These were just some examples on how to enhance observability and monitoring of a GitOps setup using Flux, I deeply recommend to browse the Flux documentation and release notes from time to time as there's much movement in this stack.

All of this eases operation of our GitOps setup, but for our business applications we need to go further, so read on.

Not hard to handle

How can we design our GitOps setup in a way that will make it most robust? There are many aspects to consider.
First, it's the human part: We have to make careless mistakes as hard as possible.
As stated in the very beginning, proposed changes to the desired state should be validated in an automated way for basic correctness. To enforce this, no changes to the single source of truth - meaning the Git branch, Flux pulls from - can be allowed without pull/merge-request. Having automated checks on this pull/merge request in addition to the four-eyes-principle should already find the worst reckless errors.
What's less obvious but also prone to break something are merge strategies other than fast-forward. Having auto-merges of files involved generates results, that could not be checked before. Although it may be rare cases in usual source code merges, isn't it always the same places that changes in Kubernetes manifests - causing higher probability of semantically wrong merge results?

Another strong opinion of mine, without discussing in detail right now, is about the right branching model for multi-stage GitOps repositories.
Yes, Kustomize allows for really sophisticated reuse of manifests. Even if my developer heart screams DRY (don't repeat yourself), I absolutely prefer one-separate-branch-per-stage repositories.
I know all the arguments against it, and I'm also aware of how to handle single-branch setups - but the risk of breaking production with changes only intended for a non-prod environment simply doesn't justify the potential benefits of a single branch. Beside that, I argue, that pull/merge-requests targeting production can be treated differently in terms of mandatory reviews or additional validations in a very easy way, compared to a single-branch setup.
There are many ways of making one-branch-per-stage handy, like templating differences as variable substitutions, packaging commonalities into Helm charts or extracting reusable Kustomizations into common repositories.
In fact, we're able to cherry-pick even complex changes from stage to stage, that feels easier and safer to me than patching Kustomizations.

Consistency and atomicity

Regarding technical terms for the second, it's about avoiding inconsistencies and side effects.
Flux' top level resource is the Kustomization. It provides us with many configuration options to reduce the blast radius of errors:

Kustomizations should dependOn each other. We can ensure that nothing happens to downstream Kustomizations, if there are errors with CRD creation, or mandatory infrastructure changes. In combination with the readiness of a Kustomization allowing us for indirect checks (installed operator came to life), changes fail early and don't propagate to healthy components. Also consider fresh bootstraps - without defined dependencies, newly set-up systems may not be able to succeed installation.
Critical resources can be protected from deletion using the prune setting. In general, recreation of resources is no issue - in fact, we're relying on it: Some application doesn't recover from an error? Delete it and let Flux recreate - "Turn it off and on again". But this is not true for everything. For instance, external managed services like load-balancers, IP addresses, SSL certificates would probably change when getting reconfigured implicitly. So we collect all of those kind of resources in a Kustomization that gets durable using prune: false.
Another rare, but breaking case is exactly the opposite: Resources should be recreated but are immutable like Jobs. Kustomizations with force: true get us covered, but this should be used with care to not obscure unexpected issues.

Avoiding error propagation is one thing, also Helm provides us with another very useful feature: Atomic updates. Using the --atomic flag with helm upgrade will either properly install the complete package, meaning all rendered manifests, or nothing - rolling back all changes to the last revision of that Helm release.
But Flux goes even further. Detailed instructions can be configured for handling failures. Installation or upgrade of a Helm release can be retried a given number of times, and remediation actions defined like uninstall on failed installation, rollback on failed update - contrary to the default behavior of leaving the Helm release in a failed state.
Depending on the content of a Helm chart, it can be necessary to automatically rollback failed upgrades on production - but leave it failed on non-prod for analyzing the issue.

What we actually benefit from is the most beloved but also most hated feature: Templating. On any error on rendering the manifests, the Helm upgrade fails and nothing is applied at all. Using plain manifests would result in an inconsistent state - even if this risk is much lower without templating at all 😉.

But Helm isn't just templating and versioning of manifests, it's a package manager, and during the installation of a package, aka chart, there are many checkpoints we can hook in. Using this, we can prepare for an application update (like updating database schemas), implement dedicated tests, but also use these hooks to validate and optionally force a Helm upgrade to fail and rollback before the entire system is damaged further.

What I want to say: Don't just throw your manifests in Git and let Flux reconcile until it succeeds or gets stuck, but carefully design your GitOps repository considering your systems' special needs. I admit, it's science as much as art, but as always with software: It's never finished or perfect, so it should be continuously revised.

Headless deployments

Eventually, we made it. All our infrastructure components and configurations are reliable, well observable and we're actively notified on issues that have to be taken care of.

In the next article we will elaborate on how to ensure the reliable, headless (business) application deployment using Flagger, happy reading.

get to know us 👉 https://mms.tech 👈

Blogging at MMS Technology

Florian Heubeck — Tue, 30 Jan 2024 09:28:40 +0000

Daddy? How are the little blog entries made?

About Communities

Blogging about blogging - I love this kind of meta stuff.
But what I love even more are communities. Passionate, enthusiastic people, sharing an area of interest. It doesn't matter how junior or senior someone is, what experience they have, or if they are just at the beginning of their journey. The key aspect is only passion.

We always had strong communities at MediaMarktSaturn. Be it interdisciplinary departments passionate for their solution, matrix-organized chapters excited for their stack like backend developers or frontend hippies, or cross-functional renegades rehearsing the uprising to establish GitOps.
But recently we lifted this community thing to a new (meta) level.

Officially founded by engineers and supported by the top management, we proclaimed a company wide Engineering Community.
There are over 600 people in, who joined of their own accord, helping each other and discussing topics.
As always, there's much less folks very active, but enough to build at least some crews like "API", "Backend Engineering", "Cloud" or "Frontend something".

Outreach Activities

A thing that became clear to us, at the latest when we got renamed to "MMS Technology": There's a lot we have to offer.
So we aligned and established a lean process for different kinds of outreach activities like speaking on meetups or conferences.
But preparing for a conference talk is high effort and not only because of that a seldom event.

What about all the goodies that happen to us and that are achieved by us day by day? Those little things that literally drive us nuts hours and days till we figured out how easy it was after you know how.
Those solutions we built for problems which peculiarly hit us as first persons ever, at least according to the known internet.

Finally: Blogging

Guess I'm not the only one who sometimes thinks: I have to talk about... or at least write about this.
And guess what? Of course there are lots of people by my side who have value to share.

You think retail business is boring? Fun is what we make of. And we make great solutions using hottest technology, in nearly every discipline available on the free market.

Processes

Long story short: At MMS Technology every person is encouraged to write about tech. There's a very lean approval process, that is (of course) Git based. You take the template, fill in your text and create a pull request.
A group of outreach reviewers - ordinary people, just sharing their passion for tech, who are familiar with our communication guidelines, validating that you don't disclose (real) secrets and consulting you in case of obscurities.
After the pull request got approved and merged, a little GitHub Action publishes the article to medium.com and you're done.

The Beginning

Why am I telling you all of this? Because I love communities. And communities share and profit from each other. I'm absolutely convinced that everyone has valuable things to share - except for our security guys, they have vulnerabilities to share 😉.
Blogs do not need to be novels - in fact, short and precise findings or insights are the best content to provide - not like this text here.

So - happy to read you soon.

get to know us 👉 https://mms.tech 👈

LET’S GO!

Florian Heubeck — Mon, 29 Jan 2024 19:02:51 +0000

Here we go

Tech Blog Launch

We, the engineering community of MediaMarktSaturn Technology, the IT partner of the electronics retailer MediaMarkt and Saturn, will share our passion for tech with you.

You can expect any kind of tech content popping up here; we are a strong team of hundreds of passionate engineers covering all disciplines of software engineering, infrastructure management, cloud operation and much more.

And we will write about it... stay tuned.

get to know us 👉 https://mms.tech 👈