Forem: Stef van Hooijdonk

Our Next(JS) Webshop

Stef van Hooijdonk — Mon, 17 Jul 2023 07:48:00 +0000

Ownership continued

In recent posts we have shared our views on ownership and how we use that @ Coolblue to develop our software. We value ownership on a team level as also seen in re-designing / re-engineering our 'backoffice' monolith.

Since last time we wrote about our Tech Principles, we actually added a Tech Principle to our collection that should help our teams understand ownership when we look at services, events and the connections between them. Especially how we want teams to handle these interdependencies in relation to new developments and maintenance. But that is something for another post (Soon ™).

This post will focus on our journey towards the "technical design" for our next implementation of our webshop (https://coolblue.nl). Our current webshop codebase is considered a Monolith. As far as I can see in Github the first commit dates back to June 17 of 2010. Over 13 years ago.

Today we have about 10 to 15 development teams working on this single codebase to make our customers smile. Each one of those teams has a pretty specific goal. And as such have to work together in this single codebase. This makes efforts that touch on large parts of this code base, like an upgrade to the next version of PHP, a shared and a multi team problem. Our Design System is tied to this same codebase.

Secondly we mainly use PHP and Twig for the implementation of our current Webshop. Great technologies, but we want to leverage a more modern set to allow for more fluid user interactions and to keep and attract talent.

Time for a change.

From Monolith to ?

In the introduction I already mentioned a few of the concerns we have with our current Webshop implementation. Based on those, we decided to investigate how we could address them.

Team Scope versus Team Disciplines

Technology wise we are looking to “downsize” the solutions a development team owns. Reduce the overall size. Size is still non-deterministic. It is a meta-unit composed of multiple factors: complexity, lines of code, number of services, number of classes and more.
From a monolith towards multiple smaller solutions that match closer to what a team with a goal can manage, build, maintain and enhance to deliver more value for our business.

A piece of history

About 2 years ago we held our first brainstorm session "Webshop Tech vNext".

One key decision we took back then already was to start using React as the base for our back office applications and the design system for them.

About 9 months ago, we held another brainstorm session to look at options for our Webshop. At that time we noticed that NextJS, together with React, could be a replacement for our Webshop tech stack.

We also learned about microfrontends. I myself found the explanation on this page on microfrontends useful. We were already doing more and more with Services and API's in many of our other solutions and the analogy made total sense to us. Especially in light of our ideas around ownership.

Proof of Concept time

We set out to test a few topics to learn how these would work for us in a microfrontend world.

During the Proof of Concept (Q1-Q2 of this year 2023) we actually implemented a piece of the routing needed for microfrontends in our Coolblue.nl webshop production (codebase). Only to be visible and used by our developers of course;

We found no large blockers and we decided to move ahead.

Roadmap it

Rebuilding our Webshop is a tremendous effort. And that is why we created a plan in the last few weeks. And have started on this massive journey.

In a year, we hope to share more on our learnings going through this process. But looking at the plan, I am sure that if you are a customer of ours, you will have been served a page or two based on this new implementation.

Technologies that help us achieve this

If you are reading this post just to learn: "What technology is Coolblue using for their webshop?" Sorry to have kept you waiting for so long.

Requests (users) will be routed to the right microfrontend application with AWS Lambda@Edge
The Application(s) will be hosted as Serverless Containers with AWS Fargate
For our Front End we will use React and TypScript
To serve our Front End we will rely on NextJS
Our background processing and services are built mostly with C# or NextJS/TypeScript. When needed, we will consolidate services via GraphQL.

Accessibility

Stef van Hooijdonk — Mon, 05 Dec 2022 09:12:00 +0000

Web accessibility a.k.a a11y refers to the universal ability of different users and devices to access the content and features of a website, regardless of physical or cognitive ability.

In other words, web accessibility ensures that everyone can successfully use a website, including users who are blind, color blind; deaf or hard of hearing, as well as users who have difficulty using their hands or have other disabilities.

What does it mean for Coolblue

We want to adhere to the W3C’s Web Content Accessibility Guidelines (WCAG) 2.1

You can find a comprehensive checklist in Coolblue design system guidelines (internal).

Definition of done

Accessibility is considered to be part of your teams definition of done.

Security

Stef van Hooijdonk — Thu, 01 Dec 2022 09:30:00 +0000

With regards to Security it is always better to reuse proven methods than to reinvent the wheel. Therefore these principles are based on the best practices used by the Mozilla Foundation. Where applicable these have been adapted or expanded to align with the other Coolblue Prinicples and Core Values.

The “do” and “do not” used in this document are examples of controls or implementation of these principles, but do not represent an exhaustive list of possibilities. When in doubt, verify if your application, service or product aligns with the goal of the principles.

Least Privilege

Do not expose unnecessary services

Goal: Limiting the amount of reachable or usable services to the necessary minimum.

List all services presented to the network (Internet and Intranets). Justify the presence of each port or service.

Do not

OpenSSH Server (sshd) is running but no users ever login.
A web-application has a web accessible administration interface, but it is not used.
A database server (SQL) allows connections from any machine in the same VLAN, even though only a single machine needs to access it.
The administration login panel of the network switch for the office network is accessible by users of the office network.

Do not grant or retain permissions that are no longer needed

Goal: Expire user access to data or services when users no longer need them.

Use role-based access control (allows for easy granular escalation of privileges, only when necessary)
Expire access automatically when unused.
Automatically disable API keys after not having been used for a given period of time and notify the user.
Use different accounts for different role types (admin, developer, user, etc.) when no good role-based access control is available.
Routinely review user’s access permissions to ensure they’re still needed.

Do not

Grant global root access (e.g. via ‘sudo’) for all operation engineers on all systems.
Give access “just in case”.
Retain access to services that you no longer use.

Defense in Depth

Do not allow lateral movement

Goal: Make it difficult or impossible for an attacker to move from one host in the network to another host.

Prevent inbound network access to services on a host from clients that do not need access to the service through either host-based firewall rules, network firewall rules/AWS security groups, or both (which is preferred).
Clearly enforce which teams have access to which set of systems.
Alert on network flows being established between difference services.

Do not

Allow inbound OpenSSH, RDP connections from any host on any network.
Run unpatched container management services (e.g. Docker) or kernels which allow a user in one container to escape the container and affect other containers on the same host.

Isolate environments

Goal: Separating infrastructure and services from each other in order to limit the the impact of a security breach.

In cases where two distinct systems are used to govern access or authorization (e.g. AD and Okta), ensure that no single user or role has administrative permissions across both systems.
Use separate sets of credentials for different environments.
Do not
Have system administrators with access to every system/every service.
Establish service users with access to multiple services.
Allow tools remotely executing code on systems from a centralized location (single Puppet Master, Ansible Tower, Nagios, etc. instance) across multiple services.
Re-use functionality across services when not required (such as sharing load balancers, databases, etc.)

Patch Systems

Goal: Ensuring systems and software do not contain vulnerabilities when these are found in software over time.

Establish regular recurring maintenance windows in which to patch software.
-Ensure individual systems can be turned off and back on without affecting service availability.
Enable automatic patching where possible.
Check web application libraries and dependencies for vulnerabilities.

Meet Web Standards

Goal: Reduce exposure to web attacks by following the web security standards.

Achieve A or higher on Mozilla’s Observatory. (Deviaton from Mozilla standards which require a B+ or higher rating)
Follow the Web Security Standards.

Guarantee data integrity and confidentiality

Goal: Ensuring data confidentiality, integrity, and authenticity is respected throughout its lifecycle.

Details on confidentiality, integrity & availability can be found below at Explanations & Rationales

Use full-disk encryption where available on systems without physical security (laptops and mobile phones).
Encrypt credentials storage databases (Ansible Vault, Credstash, etc.)
Encrypt data in transit with TLS (during transmission).
Also encrypt data in transit inside the internal network.

Do not

Terminate TLS (e.g. with a reverse proxy or load balancer) outside a system and then transmit the data in clear-text across the rest of the network.
Use STARTTLS without also disabling clear-text connections.

Know Thy System

Fraud detection and forensics
Goal: Inspect events in real-time in order to alert on suspicious behavior, and store system behavior information in order to retrace actions after a security breach.

Audit and log system calls (e.g. with auditd or Windows Audit) made by processes when running in an operating system you control (e.g. not AWS Lambda)
Send logs off the account or system (e.g. AWS CloudTrail, system logs, etc.) outside of the account or system (different AWS account, MozDef, Papertrail, etc.)
Detect and alert on anomalous changes.

Are you at risk?

Goal: Assessing how exposed you are to danger, harm or loss.

Run Rapid Risk Assessments (RRA) for your services.
Estimate what would be the impact if your service was compromised.

Do not

Think it will never happen to you.

Inventory the Landscape

Goal: Provide an accurate, maintained catalog, or system of records for all assets.

Keep an inventory of services and service owners.
Keep an inventory of machines (e.g. ServiceNow, AWS Config, Infoblox, etc.) which is updated automatically.
Ensure that the inventory contains IP addresses of systems in particular when using IPv6 (which cannot realistically be scanned).
Never rely upon security through obscurity

Coolblue addition to Mozilla principles

No security by obscurity

Goal: To prevent substituting real security for secrecy.

Always assume an attacker with perfect knowledge

Do not

Rely on trust as a security measure

KISS - Keep It Simple and thus Secure

Goal: KISS comes from ‘Keep It Simple, Stupid’. You can only secure a system that you can completely understand.

Keep things simple. Prefer simplicity over a complex and specific architecture.
Ensure others can understand the design.
Use standardized tooling that others already know how to use.
Draw high-level data flow diagrams.
See also Code Clean & Simple.

Authentication and authorization

Require two-factor authentication

Goal: Require 2FA (or MFA) on all services internal or external to prevent attackers from reusing or guessing a single credential such as a password.

MFA (multi-factor authentication, also called 2FA for two-factors) is method of confirming a user’s claimed identity by utilising a combination of two different components such as something you know (password) and something you have (phone).

Use an SSO (Single Sign On) solution with MFA. For services that can not support SSO, use the service’s individual MFA features (e.g. GitHub). Servers carrying secrets or widespread access (or any other potentially sensitive data) should verify the user’s identity end to end, such as by prompting for an additional MFA verification when connecting to the server, even when behind a bastion host.

Use central identity management (Single Sign-On)

Goal: Minimize credential theft and identity mismanagement by minimizing the handling of user credentials such as password, MFA to a set of dedicated systems.

Use an SSO (Single Sign-On) solution that authenticates users credentials on your service’s behalf. Within Coolblue we strongly encourage the usage of SAML.
Servers update their user sessions from the SSO systems regularly to ensure the user is still active and valid.
Use authorization (e.g. group membership) data from the SSO system (possibly, in addition to your own authorization data) Do not
Accept, process, transmit or store user credentials (passwords, OTPs, keys, etc.) Let the authentication server directly handle that data.
Use direct LDAP authentication for users.

More information about Centralized User Account Management can be found below in the Explanation & Rationales section

Require strong authentication

Goal: Use credential-based authentication and user session management to grant access.

More information about Shared Passwords & Password Reuse can be found below in the Explanation & Rationales section

Use credential-based authentication and user session management where the session information is passed by the user. More info.
Use API keys for service authentication.
Prefer using asymmetric API keys with request signing (e.g. x509 client certificates, AWS Signature) over symmetric API keys (e.g. HTTP header) where possible.
Ensure that API keys can be automatically rotated in the case of a data leak.
Use a password manager to store distinct passwords for each service a user accesses.
Use purpose-built credential sharing mechanisms when sharing is required (1password for teams, LastPass, etc.)

Do not

Use easy to guess passwords or vendor default passwords.
Send your password to other individuals.
Send shared passwords over email or communication mediums other than purpose-built credential sharing mechanisms.
Use the same password for multiple services.
Trust traffic from a certain network address.
Rely on VLANs or AWS VPCs to indicate requests are safe.
Use IP ACLs as replacement for authentication.
Trust the office network for access to devices.
Use TCP Wrapper for access control.
Use machine API keys for user authentication.
Use user credentials for machine authentication.
Store API keys on devices that are not physically secure (e.g. laptops or mobile phones)
Always verify, never trust

Coolblue addition to Mozilla principles

Goal: Many security problems are caused by inserting malicious intermediaries in communication paths. Zero trust is applicable to all actors and IAAA (Identification, Authentication, Authorization and Accountability).

Deny by Default
Authenticate every transaction
Use allow lists instead of block lists

NOTE: The “do” and “do not” used in this document are example of controls or implementation of the principles, but do not represent an exhaustive list of possibilities. When in doubt, verify if your application, service or product aligns with the goal of the principles. Suggestions for do's and don'ts can be send to security@coolblue.nl.

Explanations & Rationales

Confidentiality

The confidentiality of the data depends on the type of data that needs to be protected. Personally Identifiable Information (PII) such as Names, Addresses or E-mail need a high level of protection than Publicly Available data (e.g. product information). Within Coolblue we use 4 levels of confidentiality:

Secret Data should only be accessible by a very limited amount of users. Examples of secret data are Admin / Root passwords, Business Strategy Plans, API-keys or our password databases such as Active Directory or PasswordState.

Confidential Data should only be viewed by authorized persons. Examples of confidential data are Personally Identifiable Information, order history or contracts.

Restricted Data can be freely shared within the company but not outside Examples of Restricted data are the VVV, internal processes or Demo's

Public Data can be accessed by the whole world Examples of Public data are the product information on our website, marketing ads or our vacancies.

Integrity

The integrity of data is the assurance of accuracy and consistency of the data over its entire lifecycle. A high level of integrity indicates that changes to this data should be verified and the correctness is very important. A low level of integrity indicates that changes to this data don't matter for the outcome of a process. Within Coolblue we use 3 levels of Integrity:

High Integrity data is data that should always be subjected to a 4-eye principle before a change can be made. If technically feasible, integrity checks (e.g. hashes or checksums) should be implemented to verify the data after a transaction took place. Examples of high integrity data are our source code, product pricing or financial reporting data.

Medium Integrity data is data that should only be changed by an authorized person. An authentication mechanism should be in place before changes to this type of data can be made. Examples of Medium Integrity data are our processes, Google Drive documents or data on our website.

Low Integrity data is data of which it really doesn't matter if it is changed by an unauthorized person. Examples are an order list for coffee for your team or other volatile information.

It is important to state that Data Integrity is not the same as Data Quality. Data quality in general refers to whether data is useful. Data integrity, by contrast, refers to whether data is trustworthy.

Availability

The availability of data is important to ensure timely and reliable access to information and systems. For example, the website of Coolblue requires a high availability because our customers want to be able to place an order 24/7. Next to that, we want to upkeep our promise of next day delivery. So the systems & data needed for these processes also need to have a high availability.

To determine the required availability of dta or systems, the following parameters need to be determined:

Recovery Point Objective (RPO) - The amount of data, as a measure of time, we are willing to lose during a recovery event.
Maximum Tolerable Downtime (MTD) - The amount of time we can be without the asset that is unavailable before we have to declare a disaster and call into effect our DR plan
Recovery Time Objective (RTO) - The Earliest Possible Time that we can restore/recover the asset to full functionality IF everything goes as planned, and nothing else goes wrong.
Work Recovery Time (WRT) - Determines the maximum amount of time needed to verify files, services or processes have been restored correctly and are available for use.

Because an image is better than a thousand (or in this case 107) words:

It is important to be very critical when determining these parameters. Does the system really need to be restored within 2 hours, or can business be resumed without the system fully functional (e.g. in a limited capacity or with manual workarounds).

Based on the outcome of these parameters, measures should be taken to ensure that in a case of a disaster the determined timelines can be met.

Multi-Factor Authentication (MFA)

Multi-factor authentication (MFA) is a security system that requires more than one method of authentication from independent categories of credentials to verify the user's identity for a login or other transaction.

Requiring the use of MFA for internet accessible endpoints is encouraged because by requiring not only something the user knows (a knowledge factor like a memorized password) but also something that the user has (a possession factor like a smartcard, yubikey or mobile phone) the field of threat actors that could compromise the account is reduced to actors with physical access to the user.

In cases where the possession factor is digital (a secret stored in your mobile phone) instead of physical (a smartcard or yubikey), the effect of MFA is not to reduce the field of threat actors to only those that have physical access to the user, because a secret can be remotely copied off of a compromised mobile phone. Instead, in this case, the possession factor merely makes it more difficult for the threat actor since they now need to brute force/guess your password and compromise your mobile phone. This is, however, still possible to do entirely from a remote location. In particular, storing both first and second factor on the same device (for example: mobile phone) is strongly discouraged.

Shared Passwords & Accounts

Shared passwords are passwords or/and accounts that more than one person knows or has access to.

Usage of these type of accounts is discouraged because they make auditing access difficult:

multiple users appear in audit logs as one user and different users actions are difficult to differentiate. the number of audit logs that need to be searched increases. correlation of events across different systems is impossible
if multiple people are creating event records with a single shared account across multiple systems at the same time.

Furthermore, revoking access to a subset of the users of a shared password requires a password change that affects all users.

Password Reuse

Password reuse is the practice of a single user using the same password across multiple different accounts/sites. This is contrasted with creating a different, distinct password for every account/site. Users often employ hybrid forms of password reuse like:

Using the same password for a class of accounts/sites, for example, using one single password for multiple high value financial accounts, but a different single password for multiple low value forums and wikis.
Using a consistent reproducible method of password generation for each site, for example, every account/site has a password which begins with the same characters and ends with name of the site ("rosebud0facebook", "rosebud0linkedin")

Password reuse is discouraged because:

When a site is compromised by an attacker, the attacker can easily take the user's password that has been reused on other sites and gain access to those other sites. For example if a user uses the same password on a car forum website as they use on Facebook, when that car website gets compromised, the attackers can then takeover the user's Facebook account.
Unethical administrators of any sites where a password is reused may/can gain access to accounts using the reused password.

Note that it is dangerous for a user to rely on a site being able to effectively prevent an attacker from obtaining that user's password once an attacker has compromised the site.

Since it's difficult/impossible for a user to memorize a distinct password for every account/site, a common solution is to use a password manager.

The Monolith in the Room

Stef van Hooijdonk — Mon, 21 Nov 2022 09:10:58 +0000

It is very easy to talk about your current systems and code as if they are old and legacy. "That Monolith" is legacy. It is old and the code is written in a way new joiners might not like. Even the coding language or framework might be something not in the top charts at the Github Octoverse anymore.

But more often than not, these systems, monoliths, are still the moneymaker (€€) for your company. The same for us at Coolblue.

The problem

Let me explain the problem we believe to have with at least one of our monoliths based on two concerns.

Concern: Ownership

As written earlier on our vision to have Guided Ownership as low as possible in the Development teams, having a monolith that you want to improve or needs maintenance is counter intuitive then.

Shared ownership tends to leads to no ownership

We are probably not the first department that sees this issue 1.

Some practical issues can arise also when working with multiple teams on a single solution. Most of which can be addressed with proper release management, good automation and a mature CI/CD platform.

Concern: Technology

One of our two monoliths was written almost two decades ago. We used Delphi to write an application to handle all aspects of our business processes.

Application
In itself Delphi is not the problem, even though usage in the market is declining. For us the more pressing reason to actively address this monolith is that the application is written as a desktop/windows application.

Application design
The biggest concern we have is the lack of clear and separated business logic in the application design. Logic resides in either a button click/screen or in a trigger in the data layer.

Data
We also built a single database and datamodel for this monolith to work on. Here the lack of clear ownership is becoming more and more visible. Using data from tables across by services created by other teams, making it hard to than innovate and make changes to your schema.

What are we doing about it?

Not going to hang out our laundry too much here, just setting the scene why we are moving forward with the following approach.

# what are we going to do about it?
String.Replace("monolith","guided ownership");

Replace?

The basis of our approach to solve the described problems is going to be replace by rearchitecting. Every time we want to improve or change a process, we will build a new solution and make it replace part of the monolith.

This approach allows us to work piece by piece, carving out features, processes and data as we go. Allowing for MVP like implementations first. The downside is that in many cases you are dealing with part of your logic and data living in one system (the monolith) and a newer replacement system. When you look at the data part, this means we have the constant challenge to get to an accurate and consistent data warehouse.

We made the following choices to help us do exactly this:

Domain Driven Design; Together with our tech principle design-to-encapsulate-volatility and using a pattern as Ports and Adapters allows our code to separate logic and infrastructure specific code (e.g. Oracle specific queries). This allows us to separate out the parts we are re-architecting that still need the monolith (data)access.

Events and Event Driven Architecture; to decouple the different processes and their supporting apps and services we are moving towards Event Driven Architecture. This way we can abstract the data leaving the bounded context of an application from its technical form or technology. This helps to hide the underlying situation of having two data stores during the transition period (the monolith on the one hand, and the new re-architected one on the other) from the systems receiving these events.

Kafka? Really?

We also see that having more and more apps and services emit and rely on events we will need to support that. We chose to begin with Apache Kafka as the platform to facilitate these event streams. Allowing both our Data Warehouse to tap into these streams as well as enable teams to rely on Kafka to stream between apps (bounded context).

Let me be clear, we are not going to replace all inter-service-communication with events and Kafka. For some processes a batch approach, e.g. via Airflow, is still a valid and great choice. The "it depends" strikes again.

Enabling teams with our Pillars of Guided Ownership

This brings a new problem to the table for our teams. Setting up producers, topics and consumers in this new Kafka platform. We have to help the teams here by enabling them.

Here are three examples how we have been and will be enabling our teams for this new direction.

Enable via Automation and Self-service

In a previous post I hinted already that for the year of 2023 we are looking to invest and develop the tools, templates and processes to make deploying an app or service with queues, Kafka topics and BigQuery staging tables along side the needed compute components for AWS easy and enabled for Self-service.

Enable via Skills & Knowledge

In the past few months we have deliberately done a few things to help with the knowledge needed. We sent a small delegation to the DDD Europe event and the Event Storming workshop that was held then (summer 2022). This group soaked up the knowledge and went ahead inside our tech organisation to help teams and their respective domains to perform these event storming sessions to learn how to do them and to use the output generated in the design of their next solution. This was a joint effort between Principal Developers, Data Engineering and Development teams.

Enable via Building Blocks

We also have done some research and experiments and based on that we created a solution block based on the Transactional Outbox pattern in one of our core technologies in use: dotnet/c#. And will add this to our template for creating new apps and services. This way teams can use it right away, and can see how this integrates into their new solution.

What about Tomorrow?

Going forward, we will have to keep checking if Kafka is the right fit for us to do more decoupling via Events. We chose this technology based on an proof of concept we did almost 2 years ago.

We need to keep an eye out how the carving up of this monolith is progressing. Many parts are now separate Employee Activity focussed applications already. But to turn a monolith fully off, that is the biggest challenge.

And lastly, what ever we do or change in our technology vision going forward, we need to make sure we always enable the teams. Give them the Guided Ownership they need to realise our vision.

Monitoring/Observability

Stef van Hooijdonk — Tue, 15 Nov 2022 10:10:00 +0000

Complete coverage of all production systems.

No system should be active in production (i.e. providing a service to a customer or user) without being monitored.

All monitoring/logging is public, so that everyone in Coolblue has visibility of the vitality of the system and Tech Services can monitor specific aspects without exposing sensitive data.

Monitoring means tracking errors in critical workflows, health of critical dependencies and service KPIs.

Observability

Each application we build has to be observable. That means we need to know when something is wrong, and we need to be able to determine why this is.

To be able to tell when something is going wrong with our solutions we actively monitor them and we put in place alerts for our service level objectives.

To be able to find out why things are going wrong we make sure we have the logs to do so, combined with our monitoring data when needed.

This monitoring and logging principle describes two parts:

The first one being monitoring, which is the practice that describes methods to have insights in our applications and stacks. -The other one is the practice of Logging. A practice which describes methods to register log events and give insights into the complexity of our application and stacks.

Monitoring and Alerts

You and your team actively monitor your applications. First you determine the metrics that are relevant for your application to measure. Then you should create dashboards and define alerts and service level objectives. Dashboards give insights into the recent and/or current state of your application. You use them to see at a glance what is happening. Alerts, with or without a Service Level Objective tag help you to be notified when certain thresholds you set are met. The (SLO) Alerts are also acted upon via our Tech Services Department.

Logging

Applications are hard and complex to write and manage. Problems we are solving, the abstractions we create and the implementations we choose to use, are all part of the complexity we are building. In order to shed light on that complexity we can use the practice of logging.

Logging can bring us additional insights into the operations executed in the application which can help understand the sequence of events that might have lead to a certain outcome (error or otherwise). Its an investigative tool that, if exercised correctly, can help piece together the application behaviour leading up to the outcome giving developers potentially new insights into the emergent behaviour of their systems.

Playbooks

In order to work together with those that help us action SLO Alerts when they happen, even outside of your own working day, we agree to have playbooks in place. These playbooks contain information on the SLO Alert itself, the potential underlying issue and should help and direct the reader into actions to help resolve the issue. We have a template available to write these playbooks. Please make sure the playbook is findable via the SLO Alert (make sure the SLO Alert title in our observability platform matches the SLO field in the playbook, and you can add a link to the page in the slack alert for easy access).

PII Data and Sensitive Data

We monitor and we log without exposing sensitive company data or PII data on our customers.

Definition of PII Data: Personally identifiable information (PII) is any data that can be used to identify a specific individual. Social Security numbers, mailing or email address, and phone numbers have most commonly been considered PII, but technology has expanded the scope of PII considerably. It can include an IP address, login IDs, social media posts, or digital images. Geolocation and biometric is also be classified as PII.

Guided Ownership

Stef van Hooijdonk — Mon, 14 Nov 2022 09:12:00 +0000

In our Tech department here at Coolblue, we strive to give our development teams the ownership they need to build the solutions our domains, and thus our customers, need.

We enable ownership through (at least) 6 Pillars:

With clear Tech Principles
With guidance on when to use what through a radar, architecture blocks and solution blocks
Self-service cloud infrastructure (AWS/GCP)
Observability
Grow the skills you need through our Coolblue University, our Master classes or other resources you may need
Our Domain roadmaps

Lets dive a bit deeper in each of these.

Tech Principles

Everything we design, build, deploy and run adheres to our Tech principles. I wrote about these principles in a series earlier: Our Tech Principles.

Most of these principles have been learned, the hard way. And this is our way of passing that experience along to the teams of the future.

We have made these principles part of our evaluation criteria for those that work in Tech. Making it very clear how important we think it is to work with these at every stage and at every level of development.

Radar, Architecture blocks and Solution blocks

We want teams to develop solutions freely as much as possible.

We believe that means: give the teams a lot of ownership and freedom to build these solutions.

It also means, we need to enable the teams to do so as much as possible. And by having guidance and reusability we can eliminate some tedious work, making the time and effort a Team spends focus more on the actual value delivering solution.

Reuse
We all know that reusing proven code, for a give pattern or a piece of plumbing, helps you focus on the actual solution and increase your speed of delivery.
That is why we have quite a few reusable items:

Architecture blocks; in our industry we have quite a few Design patterns and we have selected those that work well in how we develop our solutions. You can see this for instance in our Tech Principle Encapsulate for volatility and the design pattern Ports and Adapters and the Transactional Outbox pattern.
Solution blocks; actual implementations developers can find and use in their solutions. Some of which are packages/components to reuse, or templates to jumpstart a new app or service. And a Design System for building Customer facing applications, one for building Activity Focussed Employee applications and one for our Email communications. Through these building blocks we also make sure common practices, like performance and security, get addressed with solid implementations to be used and to serve as inspiration.

fyi: Architecture blocks and Solution blocks originate from TOGAF 9 - Building Blocks

Deploy and maintain
That does not mean we think having every team build with different tools and languages is the best way to do so. Every language and every development environment means learning something new, means support from our CI/CD platform and from our cloud platform(s).

It also means when people want to move teams, or solutions move to a different team, we have to deal with the knowledge needed to support, develop and run these solutions over time.
So we adopted a few core languages with an intended use case. Maybe not entirely a Radar, but it does give guidance and focus.

Self-service cloud infrastructure

Is every team SecDevOps to the full extent? No. Not yet at least, but I want our Development teams to be able to create new secure solutions and run their existing ones by themselves as much as possible.
Not only can you see that through our Principles Recovery over Perfection, Automation and Testing, but we also want the Teams to deploy, scale and fix when it suits them.
Our Hosting & Deployment teams develop and maintain the tools and core infrastructure that enabled our teams to do so. Through automated (Github) repo creation for instance. Want to build something new? Add the desired repo to Github via automation yourself. The same way to make sure our observability platform is hooked up to a newly created stack and CI/CD pipeline. We use this also to implement proven practices when looking at availability and security for our. cloud infrastructure.

If you were to ask any our 50+ development teams, if they want more self-service? they will say YES. Of course they will, it's in our corporate culture to "just do it". So we keep growing their toolset to do so.

One key area for 2023 is to invest in this area. We want our Solutions to be onboarded with more standard Cloud Infrastructure and to make sure key data can be shared with other apps (outside of the bounded context), our data warehouse and our analysts. We aim to leverage Events and Event Drive Architecture more and more, and making sure key Infrastructure is ready to use will help the teams tremendously. Think standard Queue's, topics in Kafka, Tables in BigQuery and more.

Observability

Technical Observability

Any Development Team owning a solution wants to know how it is performing and if it is performing correctly. And using Observability is a great way of doing that. Our Development Teams rely on metrics, dashboards and alerts to be in the know of their solutions. We believe this is critical for a Team to fully Own It. Acting on these insights and making sure our Employees and Customers can do what they should.

Business Observability

Another way we enable teams is to also give insights beyond the technical metrics.

Operating Costs; having insights in the total costs your App/Stack makes, triggers conscious decisions on Right-Sizing the Stacks.
Business Insights; All our Applications have a purpose: it can be process Orders, finding the right amount of Stock to keep or Sending out the right Product to a Customer. By having access to these dashboards also, the Team with their Product Owner can track if what is being done is actually Moving the Needle.

Coolblue University

Maybe an obvious Pillar, but having the right skills (ie. technical, leadership or social) will make you better at what you do. It will allow you to work better with your Team and your Stakeholders.

For that reason we have a Coolblue University with over a hundred different trainings available for you to take and use. Other online resources, IRL events, books and Class room trainings are optionally available also of course.

To understand our Business and what it is they do, we also have created Master classes. These are currently presentation-based and help any that attend to fully learn our way of working on key topics in the company.

We also share what we have learned via our internal, monthly, Tech Demo's. These offer a podium for our developers and engineers to share their learnings and insights with the rest of us in tech. By tech for tech.

Domain roadmaps

We cannot forget the reason we build Solutions/develop Software. We build it because there is a benefit for our Customers (NPS) or the Company (EBITDA). And sometimes there are more strategic reasons to build something.

Each team works with a Roadmap. We evaluate these at least 4 times a year. Always be ready to change and adjust to what we have learned. The Market, your context will always be changing. And as such we need to adjust when needed and not fall for the invested-too-much-reasoning. Agility is crucial for us as a Company and for our development efforts then also. Evident by the inclusion of Flexible in our Corporate Values.

Conclusion

This post turned out to be longer than I expected and full of corporate speak I usually try to avoid. But here we are. There are more ways we support or teams off course, but I liked the idea of focussing this on these six Pillars.

This post is mostly to share and explain how we work and think at Coolblue and how we try to create the environment for Ownership for our development teams.

Design to encapsulate volatility

Stef van Hooijdonk — Fri, 04 Nov 2022 08:45:00 +0000

Our strong belief in Clean Architecture compliments the principle of Encapsulating Volatility. If something is likely to change, make sure it's easy.

A very retro analogy can be applied to hifi systems:

Some manufacturers sold complete integrated hifi systems. These systems had everything on them. Amp, record player, double cassette deck, radio, CD player, equalizer (a graphic one if you were lucky) etc.

The problem is that as CDs were replaced by next gen technology (like minidisc, ok, like mp3 on storage and then streaming), the whole system was at risk of becoming obsolete, even though the system still served its original purpose.

However high-end manufacturers stuck to their component model. You could buy an Amp, separately from a CD player, and speakers etc. You can even mix and match components from different manufacturers.

So when CDs were replaced by mp3’s then mp3’s by streaming services, you could just add, or swap out the relevant components. The CD player, one of the ‘music suppliers’ within the system was a volatile component.

This view on things goes beyond modularisation. It informs you what modules you should have. If it’s volatile, very likely to change, encapsulate it.

Clean architecture

Regardless of whether we choose Microservices, SOA, or a monolithic approach, Clean Architecture can and should still be used. All of our systems should be built this way. It emphasises the domain at the heart of our designs (key also to DDD etc).

Dependency inversion is integral (and thus this approach supports good programming practice in general, and some specific SOLID principles). It moves things that are more susceptible to change to the edge instead of the center, making them easier to replace. Whatever we build can be built following this principle (even other architectures/patterns can be built in this manner).

SOLID

S - Single-responsibility principle
O - Open-closed principle
L - Liskov substitution principle
I - Interface segregation principle
D - Dependency Inversion Principle

You adhere to SOLID principles (no exceptions for object-oriented development). These are 5 of the basic principles of Object Oriented design and programming. They should be second nature to every OO developer at Coolblue.

Testing

Stef van Hooijdonk — Thu, 27 Oct 2022 08:20:00 +0000

The software we create should do what it was intended to do. To be sure that it does, we want our production code to be well-covered by automated tests. These tests should be runnable with the click of a button, and they should be run automatically before each release.

Although testing, when done right, should ultimately make you more productive, by virtue of having to spend less time fixing problems after you release, it does ‘slow you down’ up front, as compared to writing code without any tests. Most applications or services have a long lifecycle that warrants having extensive test coverage, but given the nature of some of the work we do, some applications or features are so trivial, short-lived, or time-critical, that having few or no tests is an acceptable situation.

The metric of ‘code coverage’ is not considered of any value to determining the quality of tests. Consequently, we should not use it to fail builds or block deployments. The reason behind this is that tests can just trigger your production code, but are not testing the right business concepts. Next to that, some units might not even need to be tested. These units can for example only properly be tested using an integration test or might be too trivial and covered by their consumers.

TDD at Coolblue means:

Process

Write a failing test first, then code until that test passes. Do not write any more production code that it is necessary to make the one failing test pass.
Red-green refactor. Don’t forget to refactor, it’s the most important part.
Everyone on board. The whole team must adopt TDD, do not partially adopt.

Writing tests

Consider the inputs, outputs, all possible weaknesses (possible errors) and strengths (successful runs).
Do not overcomplicate your tests. The test should be simple to setup and execute.
Tests should run and pass on any machine/environment. If tests require special environmental setup or fail unexpectedly, then they are not good unit tests.
Make sure your tests clearly reveal their intent. Another developer can look at the test and understand what is expected of the code.
Each test should have a limited scope. If it fails it should be obvious why it failed. It’s important to only assert one logical concept in a single test. Meaning you can have multiple asserts on the same object since they will usually be the same concept.
Keep your tests clean. They are just like your production code.
External dependencies should be replaced with test doubles in unit tests.

Running tests

Unit tests should be fast, the entire unit test suite should finish running in under a minute. Unit tests should just run with zero effort (after installing dependencies).
Ratio of number of tests in each level of testing should be balanced; pyramid model. (e.g 80% unit tests, 15% integration tests, 5% acceptance tests).
Acceptance tests should be divided in suites based on features.

Measurements of success for teams

The integration and unit tests are passing before merging.
Acceptance tests are running successfully in the Acceptance environment before code is deployed to production.
Unit/Integration tests run within CI environment on each PR.
Failing tests block deployment. All tests run successfully on a CI environment before code is deployed.

Note: ‘Coverage’, i.e. the percentage of lines covered by a unit or integration test, is not a measure of success, because it says nothing about how well the code has been tested. Do not make a specific level of test coverage a requirement, because it is hard to reach and it will cause people to start writing nonsense tests just to reach the required level of coverage.

Suggested resources

Read Test Driven Development By Example by Kent Beck
Read Clean Code by Robert C. Martin
Read xUnit Test Patterns: Refactoring Test Code by Gerard Meszaros
Check training videos at https://cleancoders.com
Working Effectively with Legacy Code by Michael Feathers

Code Clean & Simple

Stef van Hooijdonk — Wed, 26 Oct 2022 12:19:22 +0000

The next instalment of this series of our Coolblue Tech Principles is on coding clean.

We prefer code that is easy to read and simple to understand. Code isn't just written for computers, but also for humans. Team members and colleagues have to read your code and understand it. Each programming language will have its own standards defined, but in general we all comply to the following programming paradigms.

YAGNI

You aren’t going to need it (Acronym YAGNI).

This is about not adding additional functionality to a product, until deemed absolutely necessary. Keep what we make focussed on delivering value quickly. If we need a knife to eat lunch, we create a basic knife. Not a scalpel and not a Swiss army knife.

The scalpel is over-engineered. Too much time has been spent on perfecting it for a use case that isn't applicable. The Swiss army knife tries to deal with all possible scenarios of usage and fails at really satisfying the customer need. In our designs we must still consider the future, we build for today without getting in the way of tomorrow. We build the simplest thing that works, without violating any of the other principles.

KISS

Keep It Simple, Stupid.

Coolblue is growing and that means increasing numbers. Increasing numbers of customers, increasing numbers of products, increasing numbers of stakeholders and increasing numbers of developers.

But, it should be understood we’re not even close to the same order of magnitude as Amazon, Facebook, Pinterest, or even some companies closer to home. Understand where we are. Understand what we need to be and how that should affect our choices.

We should have a target of getting as close as we can to scaling linearly. In some cases, this might be a stretch. In others, it’s probably an underwhelming goal. KISS is our goal, and any service that doesn’t meet this standard needs refactoring.

Don't reinvent the wheel
As a developer, you have a responsibility to know (or discover, if you don’t know) what is already available in our landscape. Therefore, when it comes to choosing a tool/technology/framework/library etc, you have a duty to consider if a pre existing item can fulfil that need. That includes, where appropriate, commercial of the shelf software (COTS) and open source solutions. Reuse over reinvent.

Be aware of where you should be placing your effort. As guidance, consider that we could say there are 2 types of development work:

Type A: Value Add - The differentiating factor. Differentiating between what the user sees today and what they want tomorrow. It’s what the Product Owner (PO) really wants.
Type B: Plumbing and foundation. E.g., the stuff the user doesn't care about and the stuff the PO doesn't see/ask for.

Examples of Type B development work:

Libraries and tools for doing what you want (e.g., database interface)
Logging/Monitoring etc
Automated build/deployment etc
Other shared/shareable utilities

Do not spend time on ‘Type B’ unless it isn't available.

Recognise that teams have a responsibility to improve/reduce their cost of development/maintenance. For example teams could consider a Commercial Off The Shelf Product (COTS) to replace a self-built system, as the reduction in maintenance cost may outweigh the initial cost over time.

Refactoring & A bit better everyday
We aim to address refactoring as part of feature development. Not as a separate refactoring activity. That makes it part of the cost of developing a feature. Only in the most extreme circumstances should refactoring be on a team’s backlog as only refactoring work, in which case it will need significant justification.

The choice on whether to introduce technical debt and the following consequences is a joint decision between a team, its lead, and the Product Owner. The Product Owner takes final ownership of the choice and any potential consequences.

We live by this rule: "always leave the campground cleaner than you found it". The same goes for our code. That means if you change some code you take ownership for the code in the encapsulating scope. If you want to make an improvement related to this, and on that rare occasion it is not part of a feature or story PR, you can use [SCOUT] as the prefix for your PR title.

For example, when changing something in a line of code, refactor the line. Change a line of code, refactor the function. Change a function, refactor the class.. Always using your best judgment. Keep it simple. This approach supports minor refactoring and consistent maintenance of (volatile) code over time.

Automation

Stef van Hooijdonk — Mon, 12 Sep 2022 07:53:20 +0000

Automation is the backbone of what we do. It provides repeatability, reliability, scalability, and speed. Four things that allow us to recover quickly. It reduces the damage caused by imperfect software, simply because we are able to get everything back up and running quickly and easily. This oversimplifies the benefits of automation, but everyone should understand why this is necessary.

If something will be done more than once, automate it.

Here are some practical examples:

From the initial code change/pull request being approved, no human intervention should be necessary to put that change live. It should happen automatically.
No virtual machine should be deployed by a person following actions defined in a script. It should only take a button press, at most.
Applications should never require configuration by hand.
Stop toil. Toil is the kind of work tied to running a production service that tends to be manual, repetitive, automatable, devoid of enduring value, and that scales linearly (or worse) as a service grows.

Automation doesn't remove the human element, it allows humans to focus on the right element and stop performing trivial, tedious and error-prone work.

Recovery over perfection

Stef van Hooijdonk — Tue, 02 Aug 2022 10:34:50 +0000

Recovery over Perfection means that, as Tech, we would rather focus our energy on ensuring that we can recover from issues quickly and effectively, rather than try to prevent all possible failures.

This supports us in the following quest: to keep speed and momentum in our release of value to the customers and users. to avoid being held back in innovation. to seek the right balance for Coolblue between cost of development and cost of maintenance.

This philosophy leads us to the following: Recovery must be ‘instantaneous’ (or super fast, at the very least).

This means we must be able to do the following things very quickly:

Discover failure
Restore acceptable behaviour
Diagnose issues
Solve issues
Apply permanent solutions
Learn from our observations (Apart from discovering failure all of the other actions could be applied in various ways/orders).

Discovering and diagnosing failure requires effective monitoring and logging. Restoring acceptable behaviour requires easy, reproducible deployments. Solving and applying solutions requires maintainable code, accompanied by automated tests (see our other principles).

Essentially, this comes down to the following: monitoring, logging, alteration, integration, deployment and testing should be a priority. How much of each is a balance question, but in our principles we will set out minimum expectations.

It doesn't relieve you from programming or designing defensively. Networks will fail, dependencies might break down, users tend to be very creative in their input for the application. You have to expect this will happen and handle it accordingly without breaking your application. Recovery over perfection doesn't relieve you from your responsibility to handle failure gracely. It requires you to make conscious decisions around safety nets and the amount of effort needing to spend.

Go to the overview of our Tech Principles.

Tech Principles @ Coolblue

Stef van Hooijdonk — Tue, 02 Aug 2022 10:34:00 +0000

In this series I would like to share the Tech Principles we use at Coolblue Tech.

Over the years (decades) our Tech population has worked with these principles and fine-tuned them over time. Learning from our own mistakes, and learning from others. As a company we want to be flexible, and thus we adopted an agile way of working. Some of these texts are copied over from our internal sources, and as such where written by others potentially.

We believe these principles help us to deliver value quickly, allows us to make changes when needed (e.g. PlayStation 5 launches) and keeps us close to our business needs. Currently we release (and fail forward) 150-300 a day towards production.

In the end, the energy spent on investigating, exploring, and creating solutions shouldn't go to waste. It should be spent on new things. The result itself should be built upon and utilised in the future. We need to stand on the shoulders of giants and focus our energy on moving towards our real goals quicker, creating features and products that will differentiate us.

Our Tech Principles in alphabetical order are:

Accessibility
Automation
Code Clean & Simple
Data Principles
Design to encapsulate volatility
Monitoring
Recovery over perfection
Security
Testing

Over the course of the weeks, we will describe these in a little more detail. Together with this post we would like to share our "Recovery over perfection" principle post.