Forem: Guillaume Dormoy

Building Techniques: A Deep Dive into TypeScript

Guillaume Dormoy — Mon, 14 Oct 2024 09:19:05 +0000

After the fundamentals of building in part1 and a closer look into Go building phase, let’s dive into typescript.

TypeScript, a typed superset of JavaScript developed by Microsoft, has its own unique build process that involves transpilation to JavaScript. Let's explore the various aspects of building TypeScript applications.

TypeScript's Approach to Building

The TypeScript Compiler (tsc)

TypeScript comes with its compiler that translates TypeScript code into JavaScript code.

Example:

tsc main.ts

tsconfig.json

The compiler needs to know what to do, and TypeScript offers a large range of configuration options. You can add flags to the tsc command, but it's far better to use the tsconfig.json file. You can either create the file by hand or get a basic one with the command tsc --init.

Example:

{
  "compilerOptions": {
    "target": "ES2015",
    "module": "commonjs",
    "strict": true,
    "outDir": "./dist"
  },
  "include": ["src/**/*"]
}

In this file, you will be able to specify the compiler options you want and which files you want to compile. For example, you will be able to specify how you handle modules. TypeScript supports various module systems: CommonJS, AMD, UMD, ES6. If you want to learn more about modules in TypeScript, the official documentation has a whole dedicated section: https://www.typescriptlang.org/docs/handbook/modules/introduction.html

Example in tsconfig.json:

{
  "compilerOptions": {
    ...
    "module": "esnext",
    ...
  }
}

You can configure a lot more, but going through it all would take forever and would be really boring. Check out the dedicated section on TS documentation https://www.typescriptlang.org/tsconfig/.

Declaration Files (.d.ts)

Nowadays, a lot of libraries are coded straight in TypeScript, but when a library is coded in JavaScript, it can still be used in your TypeScript project if the library has a declaration file. These files ending with .d.ts are basically the typed interface of the library.

declare namespace helloer {
  function sayHello(s: string): string;
}

The TypeScript team has eased the creation of this file with the generate command: tsc --declaration

Advanced TypeScript Building Techniques

Project References

Project references in TypeScript offer significant benefits for managing large-scale applications. They provide a structured approach to organizing codebases, improving build performance, and enhancing code maintainability.

Without this reference system, when you modify your codebase, TypeScript will compile everything, and your IDE will do the same in the background for type checking. On large codebases, it can consume a lot of resources and take a few minutes.

With these references and a modular approach, you only compile what's being changed. You can have tailored compilation based on your module needs. And last but not least, it allows TypeScript to compile the modules in parallel!

Custom Transformers

I am not really into this one, but it's a really interesting feature that TypeScript allows. Custom transformers are functions that can modify the Abstract Syntax Tree (AST) of your TypeScript code during the compilation process. They allow you to add, remove, or modify nodes in the AST, enabling powerful code transformations and generation.

The reason I don't like transformers is because I feel like they hide stuff from the developers. If there's something I hate when I develop, it's when something looks like magic. When everything works, it's awesome, but when it breaks, nightmare comes.

For the sake of this article, here is a dead simple example:

import ts from 'typescript';

function myCustomTransformer(context: ts.TransformationContext) {
  return (sourceFile: ts.SourceFile) => {
    function visit(node: ts.Node): ts.Node {
      // Perform transformations here
      if (ts.isCallExpression(node) && node.expression.getText() === 'console.log') {
        return ts.factory.createCallExpression(
          ts.factory.createIdentifier('console.warn'),
          undefined,
          node.arguments
        );
      }
      return ts.visitEachChild(node, visit, context);
    }
    return ts.visitNode(sourceFile, visit);
  };
}

The configuration for TypeScript to use the transformer depends on your building system.

Monorepo Builds

We have spoken about having a modular approach in the compilation with project references; let's just take a moment to speak about some useful tools that do similar jobs, like Nx.

While Project References focuses solely on TypeScript compilation, Nx provides broader project management capabilities that can incorporate TypeScript builds alongside other development tasks in a more extensive ecosystem. It supports multiple languages and offers features like dependency visualization and smart command execution, which can really help in managing mono-repos.

Optimization Techniques

To end this post, let's go over two optimization techniques that can make your code faster.

Tree Shaking

Tree shaking is an optimization technique used in modern JavaScript bundlers to eliminate dead code. I hear you say, "It's the developer's duty to clean their code," and you are right! We developers are still human beings and therefore imperfect.

Most of the time, you already use Tree Shaking optimization without knowing it. It's in your favorite framework's production build setup.

Code Splitting

I am sure you already do this, but this one is more important than you may think. By adopting a modular approach and splitting your code into smaller chunks, you get faster loading times.

You can also use dynamic imports to import modules on the go.

—

And we are done for today! I hope you found some good techniques or interesting insights on how TypeScript software is compiled and how you can improve your loved projects.

Next, we will have a look into the history of building tools.

Until then, happy coding!

Building Techniques: A Deep Dive into Go

Guillaume Dormoy — Mon, 30 Sep 2024 09:43:01 +0000

Go has been an important language in my personal career. I built a solid audio adServer and SSP with it when I was CTO at Soundcast. Go makes networking and parallelism so easy it feels like cheating. All that with really good performance, which was perfect for the job we had to do.

In part 1, we unveiled the fundamentals of the building phase of software. In this article, we will make this more relevant with the help of a language that has a special place in my heart. Indeed, while the fundamentals of building software are similar across languages, each language ecosystem has its own unique tools and practices.

Without further ado, let's explore these, with a particular focus on Go.

Go's Approach to Building

Go, designed by Google, takes a unique approach to building that emphasizes simplicity and speed.

A Command

Go comes with a build command. This command compiles the packages named by the import paths, along with their dependencies, but doesn't install the results. It generates an executable file.

Example:

go build main.go

Cross-Compilation in Go

One of the most useful features of the Go compiler is the ability to perform cross-compilation remarkably easily. All you have to do is set the GOOS and GOARCH environment variables to target the desired platforms.

Example:

GOOS=windows GOARCH=amd64 go build main.go

This will create an executable for Windows under an amd64 architecture.

Go Modules

Go mod came a bit late because it was introduced with Go 1.11, but it's a built-in dependency management system. Before that, we had to place all our Go projects into a dedicated directory on our machine called {$GOPATH}. While it worked, Go mod gives us more flexibility now, which is great!

Usage is pretty straightforward, like everything in Go.

go mod init initializes a new module.
go.mod file tracks dependencies and their versions.

You can add conditional compilation by using build tags, but the most interesting feature, in my opinion, is that Go automatically caches build output to speed up subsequent builds (of course, you can clean the cache if needed).

Advanced Go Building Techniques

Embedding Resources

If you have static files that you want to include in your binary, you can use the //go:embed directive.

Race Detection

Go is awesome when it comes to parallelism. With its system of goroutines and channels to communicate between them, developers can easily split the workload of the software on different cores. But it's also easy to mess things up and create "races."

A data race occurs when two goroutines access the same variable concurrently and at least one of the accesses is a write. See the The Go Memory Model for details.

Go compiler can help you find them!

Use the race flag to detect race conditions:

go build -race main.go

Profiling and Optimization

One of the main things I like in Go is its performance. This language gives solid results with not much effort. But if you want to go the extra mile, the Go ecosystem also gives you some really great tools to measure your performance and optimize your code.

I am not going to go into the details, but if you want to dive into the subject, I invite you to read this gem: https://www.practical-go-lessons.com/chap-34-benchmarks

Custom Build Modes

Sometimes, you are not building software but a plugin or a library, for example. The Go team has got your back. In Go, you can define the buildmode to get the result you want!

Build shared libraries: go build -buildmode=c-shared
Build plugins: go build -buildmode=plugin
Get the full list at https://pkg.go.dev/cmd/go#hdr-Build_modes

As you can see, Go provides a lot of features in its compiler and ecosystem to allow developers to get the most out of the Go technology.

And that's it for today. I'll be releasing another article this week showcasing the building techniques for TypeScript, so subscribe or follow if you want to make sure not to miss it!

Until then, happy coding!

TSyringe and Dependency Injection in TypeScript

Guillaume Dormoy — Wed, 25 Sep 2024 09:06:48 +0000

I am not a big fan of large frameworks like NestJS; I've always liked the freedom of building my software the way I want with the structure I decide in a lightweight way. But something I liked when testing NestJS out was the Dependency injection.

Dependency Injection (DI) is a design pattern that allows us to develop loosely coupled code by removing the responsibility of creating and managing dependencies from our classes. This pattern is crucial for writing maintainable, testable, and scalable applications. In the TypeScript ecosystem, TSyringe stands out as a powerful and lightweight dependency injection container that simplifies this process.

TSyringe is a lightweight dependency injection container for TypeScript/JavaScript applications. Maintained by Microsoft on their GitHub (https://github.com/microsoft/tsyringe), it uses decorators to do Constructor injection. Then, it uses an Inversion of Control container to store the dependencies based on a token that you can exchange for an instance or a value.

Understanding Dependency Injection

Before diving into TSyringe, let's briefly explore what dependency injection is and why it's important.

Dependency injection is a technique where an object receives its dependencies from external sources rather than creating them itself. This approach offers several benefits:

Improved testability: Dependencies can be easily mocked or stubbed in unit tests.
Increased modularity: Components are more independent and can be easily replaced or updated.
Better code reusability: Dependencies can be shared across different parts of the application.
Enhanced maintainability: Changes to dependencies have minimal impact on dependent code.

Setting Up TSyringe

First, let's set up TSyringe in your TypeScript project:

npm install tsyringe reflect-metadata

In your tsconfig.json, ensure you have the following options:

{
  "compilerOptions": {
    "experimentalDecorators": true,
    "emitDecoratorMetadata": true
  }
}

Import reflect-metadata at the entry point of your application:

import "reflect-metadata";

The entry point of your application is, for example, the root layout on Next.js 13+, or it can be the main file in a small Express application.

Implementing Dependency Injection with TSyringe

Let's take the example from the introduction and add the TSyringe sugar:

Let's start with the adapter.

// @/adapters/userAdapter.ts
import { injectable } from "tsyringe"

@injectable()
class UserAdapter {
    constructor(...) {...}

    async fetchByUUID(uuid) {...}
}

Notice the @injectable() decorator? It's to tell TSyringe that this class can be injected at runtime.

So my Service is using the adapter we just created. Let's inject that Adapter into my Service.

// @/core/user/user.service.ts
import { injectable, inject } from "tsyringe"
...

@injectable()
class UserService {
    constructor(@inject('UserAdapter') private readonly userAdapter: UserAdapter) {}

    async fetchByUUID(uuid: string) {
    ...
        const { data, error } = await this.userAdapter.fetchByUUID(uuid);
    ...
    }
}

Here I also used the @injectable decorator because the Service is going to be injected into my command class, but I also added the @inject decorator in the constructor params. This decorator tells TSyringe to give the instance or the value it has for the token UserAdapter for the userAdapter property at runtime.

And last but not least, the root of my Core: the command class (often wrongly called usecase).

// @/core/user/user.commands.ts
import { inject } from "tsyringe"
...

@injectable()
class UserCommands {
    constructor(@inject('UserService') private readonly userService: UserService) {}

    async fetchByUUID(uuid) {
    ...
        const { data, error } = this.userService.fetchByUUID(uuid);
    ...
    }
}

At this point, we've told TSyringe what is going to be injected and what to inject in the constructor. But we still haven't made our containers to store the dependencies. We can do that in two ways:

We can create a file with our dependency injection registry:

// @/core/user/user.dependencies.ts
import { container } from "tsyringe"
...

container.register("UserService", {useClass: UserService}) // associate the UserService with the token "UserService"
container.register("UserAdapter", {useClass: UserAdapter}) // associate the UserAdapter with the token "UserAdapter"

export { container }

But we can also use the @registry decorator.

// @/core/user/user.commands.ts
import { inject, registry, injectable } from "tsyringe"
...

@injectable()
@registry([
    {
        token: 'UserService',
        useClass: UserService
    },
    {
        token: 'UserAdapter',
        useClass: UserAdapter
    },
])
export class UserCommands {
    constructor(@inject('UserService') private readonly userService: UserService) {}

    async fetchByUUID(uuid) {
    ...
        const { data, error } = this.userService.fetchByUUID(uuid);
    ...
    }
}

container.register("UserCommands", { useClass: UserCommands})

export { container }

Both methods have pros and cons, but at the end of the day, it's a matter of taste.

Now that our container is filled with our dependencies, we can get them from the container as needed by using the resolve method of the container.

import { container, UserCommands } from "@/core/user/user.commands"

...
const userCommands = container.resolve<UserCommands>("UserCommands")
await userCommands.fetchByUUID(uuid)
...

This example is pretty simple as each class only depends on another, but our services could depend on many, and dependency injection would really help keep everything tidy.

But wait! Don't leave me like that! How about the tests?

Testing with TSyringe

Our injections can also help us test our code by sending mock objects straight into our dependencies. Let's see a code example:

import { container, UserCommands } from "@/core/user/user.commands"

describe("test ftw", () => {
    let userAdapterMock: UserAdapterMock
    let userCommands: UserCommands

    beforeEach(() => {
        userAdapterMock = new UserAdapter()
        container.registerInstance<UserAdapter>("UserAdapter", userAdapter)
        userCommands = container.resolve<UserCommands>("UserCommands")
    });

    ...
});

Now the UserAdapter token contains a mock that will be injected into the dependent classes.

Best Practices and Tips

Use interfaces: Define interfaces for your dependencies to make them easily swappable and testable. I didn't use interfaces for the sake of simplicity in this article, but interfaces are life.
Avoid circular dependencies: Structure your code to avoid circular dependencies, which can cause issues with TSyringe.
Use tokens for naming: Instead of using string literals for injection tokens, create constant tokens:
```
export const USER_REPOSITORY_TOKEN = Symbol("UserRepository");
```
Scoped containers: Use scoped containers for request-scoped dependencies in web applications.
Don't overuse DI: Not everything needs to be injected. Use DI for cross-cutting concerns and configurable dependencies.

If you've come this far, I want to say thank you for reading. I hope you found this article instructive. Remember to always consider the specific needs of your project when implementing dependency injection and architectural patterns.

Likes and comment feedback are the best ways to improve.

Happy coding!

The Art and Science of Software Building: From Source to Executable

Guillaume Dormoy — Mon, 23 Sep 2024 08:57:37 +0000

In today's fast-paced world of software development, the journey from writing code to deploying a functional application is more complex—and more fascinating—than ever before. Whether you're a seasoned developer or just starting your coding journey, understanding the intricacies of software building is crucial for creating efficient, scalable, and robust applications.

Introduction: Why Software Building Matters

Last week, we explored what makes an effective Continuous Integration (CI) process. Now, we're diving deeper into the foundation of CI and software development as a whole: the art and science of software building. This article kicks off a series that will take you on a journey through the evolution of build processes, from simple command-line compilations to sophisticated containerized build environments.

Throughout this series, we'll uncover:

The fundamentals of software building and its critical role in the development lifecycle
Various building techniques across programming languages, with a special focus on Go and TypeScript
The game-changing role of containers in modern build processes
A historical perspective on the evolution of software building over the decades

By the end of this series, you'll have a comprehensive understanding of software building that will empower you to make informed decisions in your development projects. So, let's compile our thoughts and build our way through this fascinating aspect of software development!

Defining Software Building

Before we dive in, let's establish a clear definition:

In the world of software development, "building" is the crucial step that transforms human-readable source code into machine-executable programs. It's the bridge between the creative process of writing code and the practical reality of running software on computers.

My Journey: From University Labs to Modern Development

The Early Days: C and Manual Dependency Management

My software engineering journey began in 2006 when I started studying computer science at university. One of our first major projects was to re-implement the C library from scratch. As a 19-year-old student, I found myself grappling with the complexities of compiled languages for the first time.

Initially, compiling our C code involved a seemingly simple yet bewildering command line interface. As our projects grew more complex, we graduated to using makefiles—a significant step up, but still far from the streamlined processes we enjoy today.

Perhaps the most striking difference was dependency management. Back then, I handled dependencies manually, scouring the internet for the right libraries and importing them into my projects one by one. It was a tedious and error-prone process that consumed a significant portion of development time.

The Revelation: Automated Dependency Management

It wasn't until I left university and started working with Symfony around 2013 that I discovered the power of automated dependency management. The revelation was game-changing: tools could handle the complex web of dependencies for us, saving time and reducing errors.

Fast forward to today, and it's almost unbelievable to think of a time without robust dependency management tools. Every major programming ecosystem now boasts one or several such tools, transforming what was once a burdensome task into a seamless part of the development process.

The Modern Mantra: "Check out the dependencies, then build"

Today, whether we're working with containers or on bare metal, the software building process has been distilled to a simple yet powerful mantra: "Check out the dependencies, then build." This elegantly simple approach belies the complex evolution of build processes over the years.

The Fundamentals of Software Building

Now that we've set the stage with a bit of history, let's break down the key components and concepts of software building. Understanding these fundamentals is crucial for any developer looking to optimize their build processes and create more efficient software.

1. Compilation vs. Interpretation: The Language Spectrum

Programming languages generally fall into three categories, each with its own approach to turning code into executable instructions:

Compiled Languages: Speed at the Cost of Flexibility

Examples: C, C++, Go

Compiled languages use a compiler to translate code directly into machine code, which can be executed by the computer's hardware. This approach typically results in faster runtime performance but requires a separate build step for each target platform.

// Example of a simple Go program
package main

import "fmt"

func main() {
    fmt.Println("Hello, compiled world!")
}

To run this, you'd first compile it with go build, then execute the resulting binary.

Interpreted Languages: Flexibility and Ease of Use

Examples: Python, Ruby, JavaScript

Interpreted languages are executed line by line by an interpreter. While generally slower than compiled languages, they offer greater flexibility and ease of use. You can run the code directly without a separate compilation step.

# Example of a simple Python script
print("Hello, interpreted world!")

This script can be run directly with python script.py.

Hybrid Languages: The Best of Both Worlds?

Examples: Java, C#

Hybrid languages are compiled into an intermediate bytecode, which is then interpreted or just-in-time compiled by a virtual machine. This approach aims to balance the performance benefits of compilation with the flexibility of interpretation.

// Example of a simple Java program
public class HelloWorld {
    public static void main(String[] args) {
        System.out.println("Hello, hybrid world!");
    }
}

This would be compiled to bytecode with javac HelloWorld.java, then run with java HelloWorld.

The New Generation: Blurring the Lines

Example: TypeScript

Modern languages like TypeScript are blurring the traditional boundaries. TypeScript adds features like type annotations to JavaScript but needs to be compiled down to JavaScript to run.

// TypeScript example
let message: string = "Hello, TypeScript world!";
console.log(message);

This would be compiled to JavaScript before running in a browser or Node.js environment.

2. The Build Process: Assembling Your Software LEGO

Just as building with LEGO involves following a set of instructions to assemble bricks into a cohesive structure, software building follows a process to transform source code into a functioning application. Let's break down the key steps:

Preprocessing:
- Gather all necessary pieces (source files, dependencies)
- Prepare the build environment
Compilation:
- Translate source code to object code or bytecode
- Check for syntax errors and type mismatches
Linking:
- Connect all the compiled pieces together
- Resolve references between different parts of the code
Packaging:
- Create the final executable or distributable package
- May include bundling assets, creating installers, etc.

3. Build Tools and Dependency Management: The Right Tools for the Job

Different programming ecosystems have developed specialized tools to manage the build process and handle dependencies. Here's a quick overview of some popular ones:

Make: A classic tool for C/C++ projects
Maven/Gradle: Popular for Java projects
npm/yarn/webpack/vite: Common in JavaScript ecosystems
Go build: Go's built-in build command

Each of these tools aims to streamline the build process, making it more efficient and less error-prone.

4. Cross-Compilation: Building for Different Targets

Cross-compilation allows developers to build executables for different architectures or operating systems than the one they're working on. This is crucial for:

Developing mobile apps on desktop computers
Creating software for IoT devices
Ensuring software compatibility across different OS versions

For example, at Soundcast, I was compiling Go on a Mac using the following command to build for a Linux system:

GOOS=linux GOARCH=amd64 go build main.go

5. Optimization: Fine-tuning Performance

Build processes often include optimization steps to improve the performance of the resulting software. These can include:

Compile-time optimizations (e.g., dead code elimination)
Link-time optimizations
Profile-guided optimizations based on runtime data

Key Takeaways

Software building is a crucial step in transforming source code into executable programs.
Different language types (compiled, interpreted, hybrid) have different build processes.
The build process typically involves preprocessing, compilation, linking, and packaging.
Modern build tools and dependency management systems have greatly simplified the process.
Cross-compilation and optimization are advanced techniques that can enhance software portability and performance.

The Ever-Evolving Landscape of Software Building

As we've seen, the world of software building is rich with complexity and constantly evolving. From the early days of manual compilation and dependency management to today's containerized build environments and automated processes, the field has come a long way.

In my day-to-day work, I now leverage containers for both production and development environments. Combined with powerful package management tools like NPM, these modern approaches have transformed what was once a cumbersome process into a streamlined, efficient part of the development workflow.

As we continue this series, we'll delve deeper into specific build techniques, explore the impact of containerization, and look at how different programming languages approach the build process. Stay tuned for the next installment, where we'll focus on building in Go and TypeScript environments.

I invite you to share your own experiences with software building in the comments below. What challenges have you faced? What tools or techniques have you found most helpful? Your insights could be invaluable to fellow developers navigating this complex landscape.

Don't forget to subscribe and follow to catch the next part of this series. Until then, happy building!

Effective CI: A Comprehensive Guide

Guillaume Dormoy — Mon, 16 Sep 2024 08:36:52 +0000

Introduction: The Tale of Two CIs

As a software developer and manager, I've experienced the highs and lows of working with different Continuous Integration (CI) systems. Let me share two contrasting stories that shaped my understanding of what makes an effective CI.

The Nightmare

I once worked on a monorepo project with a massive codebase. When I first arrived, the CI system was manageable, but after months of development, feature additions, and more code and tests, it became a burden. I was leading the development team but hadn’t closely monitored how long the CI process was taking—until one day, a developer mentioned that one of his daily frustrations was waiting 40 minutes just to see his tests fail. (I spoke with this team recently and they managed to reduce it to about 10 minutes with several optimisations.)

This significant time loss impacted our productivity. Developers tried to work on other tasks while waiting for the CI, but it caused frequent context-switching, delayed code reviews, and introduced inefficiencies. In short, it was a nightmare.

The Dream

On the other hand, I once worked with a CI system so smooth that we forgot it was even there. It was fast, reliable, and delivered feedback in under five minutes. Our tests gave us confidence in the code, and code reviews were effortless. Deployments were so seamless they felt almost like cheating.

At the time, I was no stranger to CI pipelines, having built a few for personal projects and contributed to some at my previous jobs. However, seeing a team prioritize and maintain such a well-functioning CI pipeline changed my entire perspective on its importance.

These two experiences made me realize that an effective CI isn't just a nice-to-have—it's a game-changer that can dramatically improve code quality, team productivity, and overall project success. In hindsight, I view the nightmare scenario as a personal leadership failure for not addressing the CI issues sooner.

The Anatomy of an Effective CI Pipeline

Now, let's break down the key components of an effective CI pipeline and explore how to optimize each stage.

1. Code Commit and Version Control

There’s a reason most version control systems now integrate CI tools. I was even surprised it took GitHub so long to introduce GitHub Actions. You can’t have continuous integration without a proper version control system, and the workflow you adopt is just as important.

A poorly designed workflow can result in a messy history, merge conflicts, and large, unmanageable commits that make review and integration difficult. A good version control system keeps the codebase clean, with a clear history. But how can teams ensure it stays that way?

The answer lies in adopting a well-defined workflow. For many of the projects I've worked on, the GitFlow workflow has been particularly effective. One often overlooked aspect is commit messages—many developers (myself included) tend to neglect their importance. However, clear commit messages provide valuable context to reviewers, resulting in faster and better feedback. Fortunately, AI tools can now help with writing clear commit messages.

2. Build Automation

Remember my nightmare? I can honestly say, "never again." Automated builds ensure your code is compiled and packaged consistently across environments.

Now, imagine a world where builds take 50 minutes and occasionally break due to connectivity issues. Horrible, right?

To avoid this, several strategies can be applied. Containerization helps maintain consistent build environments, and you can run these images locally. Caching strategies can also be employed to prevent pulling unnecessary dependencies for every build.

For monolithic codebases, tools like NX allow you to split your build process into smaller parts, enabling parallel builds or selective builds of only the required sections of your code. This significantly improves build times.

3. Automated Testing

Automated tests are critical for catching bugs early, providing confidence in code changes, and acting as living documentation for expected behavior. They verify that code changes don’t break existing functionality and ensure requirements are met.

For tests to be effective, they need to be both fast and reliable. Flaky tests—tests that sometimes fail due to external factors—can be frustrating. I've dealt with intermittent test failures caused by improper date handling, but fortunately, they were easy to fix.

Early in my career, I thought unit tests were a waste of time because they didn’t validate database assertions. So, I focused on integration and end-to-end tests. While useful, these tests are often slow. Slow tests delay feedback, which slows everything down. Now, I prioritize unit tests with larger scopes (typically at the layer level), which allows for faster, more effective pipelines. I still use integration and end-to-end tests, but they serve as circuit breakers during staging rather than being the primary feedback loop.

Running tests in parallel is another great way to reduce execution time.

4. Code Quality Checks

Maintaining consistent coding standards and adhering to best practices improves code readability and maintainability. This is why enforcing coding standards and catching issues before they hit production is crucial, especially in today's AI-driven world.

Static code analysis tools like Sonar and Snyk.io provide valuable insights into potential security threats in both your code and dependencies. Additionally, configuring your linter to automatically clean code based on your team's style guidelines helps keep the codebase tidy.

Stopping here gives you a decent pipeline, but we can go even further with the delivery.

5. Artifact Generation and Storage

Artifacts, such as compiled binaries and container images, should be consistently built and easily accessible for deployment. Rebuilding the same code repeatedly is wasteful.

The goal is to produce and store versioned artifacts that can be deployed to any environment, with differences handled through environment variables.

In the "dream" scenario, we had artifacts for previous versions readily available, allowing us to quickly roll back when production issues occurred. Since deployment was fast, we could buy time to fix the problem without disrupting service.

6. Staging Deployment

A staging environment provides a final opportunity to catch issues before they reach production. I mentioned earlier that this is where I run end-to-end tests. Staging should closely mirror production, and automating its setup is essential.

Infrastructure-as-code tools like Terraform or Pulumi can help ensure staging environments are accurate reflections of production. Another key aspect is ensuring data consistency. Companies often struggle with staging environments because of poor or outdated data. Implementing data processing pipelines to anonymize production data can create a relevant dataset for testing.

7. Production Deployment

The ultimate goal of CI is to deliver value to end-users quickly and reliably. A CI pipeline wouldn’t be effective if it didn’t provide a safe and efficient way to release features and fixes to production.

Manual deployment slows everything down, and the pressure to get it right can lead to mistakes. Automating deployment processes within the CI pipeline mitigates these risks. With versioned artifacts ready for deployment, rolling back to a stable state is simple and stress-free.

This is the part I loved most about working on the "Dream" project—stress-free releases, knowing I could easily roll back if something went wrong.

Conclusion: Continuous Improvement in Continuous Integration

An effective CI pipeline is not a "set it and forget it" solution. It requires ongoing attention, optimization, and adaptation to meet the evolving needs of your project and team. By focusing on each stage of the pipeline and continuously seeking improvements, you can create a CI system that not only integrates code but also empowers your team to deliver high-quality software confidently.

The ultimate goal of CI is to enhance collaboration, boost productivity, and deliver better software to your users. With a well-crafted CI pipeline, you can turn the nightmare of uncertain deployments into a dream of smooth, reliable releases.