Forem: Antoine Caron

Managing Terraform Modules with Nx Monorepo

Antoine Caron — Sun, 02 Nov 2025 20:04:37 +0000

Introduction

Managing infrastructure as code at scale is challenging. When your organization grows beyond a handful of Terraform modules, you quickly encounter familiar problems: code duplication, inconsistent testing practices, unclear dependencies between modules, and the eternal question of whether to use a monorepo or split everything into separate repositories.

If you've worked with Terraform in a team environment, you've probably faced these pain points:

Running terraform fmt and terraform validate across dozens of modules manually
Uncertainty about which modules are affected by a change
Difficulty enforcing consistent standards across all modules
Time wasted running tests on unaffected modules in CI/CD
The challenge of managing dependencies between related modules

What if you could leverage the same powerful tooling that frontend teams use to manage their monorepos? Enter Nx - a smart build system that can transform how you manage Terraform modules.

Want to see this in action? I've created a complete demo repository showcasing all the concepts covered in this article, with seven Scaleway Terraform modules fully configured with Nx.

The Challenge of Managing Terraform Modules at Scale

The Multi-Repo Dilemma

Many organizations start with separate Git repositories for each Terraform module. This approach seems clean at first - each module has its own versioning, CI/CD pipeline, and release cycle. However, as your infrastructure grows, this strategy reveals its weaknesses:

Versioning Hell: When Module A depends on Module B, you need to carefully manage version pinning. Update Module B? Now you need to update Module A's version constraint, test it, and release a new version.
Cross-Module Changes: A breaking change that affects multiple modules requires coordinating PRs across multiple repositories, each with its own review and merge timeline.
Inconsistent Tooling: Each repository might have different CI configurations, different testing approaches, or even different versions of tools like tflint or terraform-docs.

The Monorepo Challenges

Moving to a monorepo solves some problems but creates others:

Build Times: Without smart tooling, CI runs tests on ALL modules even when you only changed one.
Dependency Tracking: How do you know which modules depend on each other? Manual documentation? Good luck keeping that up to date.
Task Orchestration: Running terraform fmt across 50 modules means writing bash scripts that find all the directories, loop through them, and aggregate results.
Developer Experience: Developers need clear visibility into what's affected by their changes and confidence that they're running the right tests.

The traditional approach to Terraform module management leaves teams choosing between the coordination overhead of multi-repo or the inefficiency of a naive monorepo.

Why Nx for Terraform Modules?

Nx is a smart build system originally designed for JavaScript/TypeScript monorepos, but it's fundamentally language-agnostic. At its core, Nx provides exactly what we need for managing Terraform modules:

1. Intelligent Task Orchestration

Nx understands your project graph - which modules depend on which. When you run a task, Nx:

Only executes tasks on affected projects based on your git diff
Runs tasks in the correct order based on dependencies
Executes independent tasks in parallel to maximize speed

For Terraform, this means running terraform validate only on the modules that changed, and their dependents - not the entire monorepo.

2. Dependency Graph Visualization

Run nx graph and see a visual representation of how your modules relate to each other. This is invaluable for understanding impact and planning refactors.

3. Code Generation

Nx generators let you scaffold new modules with consistent structure. Want a new Terraform module with the right directory structure, a project.json, tests, and documentation? One command.

4. Consistent Tooling Across Projects

Define tasks once, inherit them across all modules. Every module gets fmt, validate, lint, and test targets without copying configuration files everywhere.

The beauty of Nx is that it doesn't change how you write Terraform - it enhances how you manage Terraform modules at scale.

Setting Up an Nx Workspace for Terraform

Let's build a practical Nx workspace for managing Terraform modules. I'll walk you through the setup step by step.

Creating the Workspace

First, create a new Nx workspace:

npx create-nx-workspace@latest terraform-modules --preset=npm
cd terraform-modules

Choose the "npm" preset - we're not building a JavaScript application, we just want Nx's task orchestration capabilities.

Workspace Structure

Here's a recommended structure for organizing Terraform modules:

terraform-modules/
├── nx.json
├── package.json
├── modules/
│   ├── scw-vpc/
│   │   ├── project.json
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   ├── outputs.tf
│   │   └── README.md
│   ├── scw-k8s/
│   │   ├── project.json
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   ├── outputs.tf
│   │   └── README.md
│   └── scw-database/
│       ├── project.json
│       └── ...
└── tools/
    └── scripts/

Each Terraform module lives in modules/ and has its own project.json file where we define Nx tasks.

Configuring Nx for Terraform Projects

The project.json file is where Nx magic happens. Here's an example for the scw-vpc module:

{
  "name": "scw-vpc",
  "$schema": "../../node_modules/nx/schemas/project-schema.json",
  "projectType": "library",
  "sourceRoot": "modules/scw-vpc",
  "targets": {
    "fmt": {
      "executor": "nx:run-commands",
      "options": {
        "command": "terraform fmt -check",
        "cwd": "modules/scw-vpc"
      }
    },
    "fmt-fix": {
      "executor": "nx:run-commands",
      "options": {
        "command": "terraform fmt",
        "cwd": "modules/scw-vpc"
      }
    },
    "validate": {
      "executor": "nx:run-commands",
      "options": {
        "commands": ["terraform init -backend=false", "terraform validate"],
        "cwd": "modules/scw-vpc",
        "parallel": false
      }
    },
    "lint": {
      "executor": "nx:run-commands",
      "options": {
        "command": "tflint",
        "cwd": "modules/scw-vpc"
      }
    },
    "docs": {
      "executor": "nx:run-commands",
      "options": {
        "command": "terraform-docs markdown . > README.md",
        "cwd": "modules/scw-vpc"
      }
    }
  },
  "tags": ["type:terraform", "cloud:scaleway", "layer:network"]
}

The nx:run-commands executor lets us run any shell command - perfect for Terraform CLI commands.

Now you can run:

nx fmt scw-vpc          # Check formatting
nx validate scw-vpc     # Validate configuration
nx lint scw-vpc         # Run tflint
nx run-many -t validate # Run validate on all modules

Managing Dependencies Between Terraform Modules

One of Nx's most powerful features is its understanding of dependencies. But Terraform modules don't use import statements like JavaScript - so how does Nx know about dependencies?

Implicit Dependencies

Nx can infer dependencies by analyzing your Terraform code. When scw-k8s module references the scw-vpc module with a relative path:

module "vpc" {
  source = "../scw-vpc"
  # ...
}

Nx can detect this relationship. Configure implicit dependencies in your root nx.json:

{
  "pluginsConfig": {
    "@nx/dependency-checks": {
      "checkVersionMismatch": false
    }
  },
  "targetDefaults": {
    "validate": {
      "dependsOn": ["^validate"]
    }
  }
}

Explicit Dependencies

For more control, explicitly declare dependencies in your project.json:

{
  "name": "scw-k8s",
  "implicitDependencies": ["scw-vpc"],
  "targets": {
    // ...
  }
}

This tells Nx that scw-k8s depends on scw-vpc. Now when you run:

nx validate scw-k8s

Nx will first validate scw-vpc, then scw-k8s. And more importantly:

nx affected -t validate

If you change scw-vpc, Nx knows that scw-k8s is affected and will run validation on both modules.

Visualizing the Graph

See your entire module ecosystem:

nx graph

This opens an interactive browser view showing all your modules and their relationships. It's incredibly useful for:

Onboarding new team members
Planning large refactors
Understanding blast radius of changes
Identifying circular dependencies

Building a Task Pipeline

Nx really shines when you build task pipelines that automatically run in the correct order. Let's create a comprehensive set of targets for our Terraform modules.

Defining Target Dependencies

In your nx.json, use targetDefaults to create a pipeline:

{
  "targetDefaults": {
    "fmt": {
      "dependsOn": []
    },
    "validate": {
      "dependsOn": ["^validate", "fmt"]
    },
    "lint": {
      "dependsOn": ["validate"]
    },
    "security": {
      "dependsOn": ["lint"]
    }
  }
}

The ^ prefix means "dependencies of dependencies". When you run nx security scw-k8s:

Nx validates all modules that scw-k8s depends on
Then formats scw-k8s
Then validates scw-k8s
Then lints scw-k8s
Finally runs security scans on scw-k8s

All automatically, in parallel where possible.

Common Targets

Here's a complete project.json with all the targets you'll typically need:

{
  "name": "scw-vpc",
  "$schema": "../../node_modules/nx/schemas/project-schema.json",
  "projectType": "library",
  "sourceRoot": "modules/scw-vpc",
  "targets": {
    "fmt": {
      "executor": "nx:run-commands",
      "options": {
        "command": "terraform fmt -check",
        "cwd": "modules/scw-vpc"
      }
    },
    "fmt-fix": {
      "executor": "nx:run-commands",
      "options": {
        "command": "terraform fmt",
        "cwd": "modules/scw-vpc"
      }
    },
    "validate": {
      "executor": "nx:run-commands",
      "options": {
        "commands": ["terraform init -backend=false", "terraform validate"],
        "cwd": "modules/scw-vpc",
        "parallel": false
      }
    },
    "lint": {
      "executor": "nx:run-commands",
      "options": {
        "commands": ["tflint --init", "tflint"],
        "cwd": "modules/scw-vpc",
        "parallel": false
      }
    },
    "security": {
      "executor": "nx:run-commands",
      "options": {
        "command": "checkov -d .",
        "cwd": "modules/scw-vpc"
      }
    },
    "docs": {
      "executor": "nx:run-commands",
      "options": {
        "command": "terraform-docs markdown . > README.md",
        "cwd": "modules/scw-vpc"
      }
    }
  },
  "tags": ["type:terraform", "cloud:scaleway", "layer:network"]
}

Running Tasks Across Multiple Projects

# Run fmt on all modules
nx run-many -t fmt

# Run affected validations based on git changes
nx affected -t validate,lint

# Run tasks in specific order
nx run-many -t fmt,validate,lint,security

# Run on modules with specific tags
nx run-many -t validate --projects=tag:cloud:scaleway

Code Generation and Scaffolding

Manually creating project.json files for each new module gets tedious. Nx generators solve this by scaffolding consistent module structures with a single command.

Creating a Custom Generator

Create a generator in tools/generators/terraform-module/:

// tools/generators/terraform-module/index.js
const { formatFiles, generateFiles, Tree } = require("@nx/devkit");
const path = require("path");

async function terraformModuleGenerator(tree, schema) {
  const { name, cloud, layer } = schema;
  const projectRoot = `modules/${name}`;

  // Generate files from templates
  generateFiles(tree, path.join(__dirname, "files"), projectRoot, {
    name,
    cloud,
    layer,
    tmpl: "",
  });

  // Update workspace
  const projectConfiguration = {
    name,
    projectType: "library",
    sourceRoot: projectRoot,
    targets: {
      fmt: {
        /* ... */
      },
      validate: {
        /* ... */
      },
      lint: {
        /* ... */
      },
      test: {
        /* ... */
      },
    },
    tags: [`type:terraform`, `cloud:${cloud}`, `layer:${layer}`],
  };

  addProjectConfiguration(tree, name, projectConfiguration);
  await formatFiles(tree);
}

module.exports = terraformModuleGenerator;
module.exports.schema = {
  properties: {
    name: { type: "string" },
    cloud: { type: "string", enum: ["scaleway", "aws", "gcp", "azure"] },
    layer: {
      type: "string",
      enum: ["network", "compute", "storage", "security"],
    },
  },
  required: ["name", "cloud", "layer"],
};

Template Files

Create templates in tools/generators/terraform-module/files/:

# tools/generators/terraform-module/files/main.tf__tmpl__
terraform {
  required_version = ">= 1.0"

  required_providers {
    <%= cloud %> = {
      source  = "hashicorp/<%= cloud %>"
      version = "~> 5.0"
    }
  }
}

# Add your resources here

# tools/generators/terraform-module/files/variables.tf__tmpl__
variable "name" {
  description = "Name of the <%= name %> module"
  type        = string
}

# tools/generators/terraform-module/files/outputs.tf__tmpl__
output "id" {
  description = "ID of the created resource"
  value       = ""
}

Using the Generator

Now create new modules consistently:

nx g @nx/workspace:workspace-generator terraform-module \
  --name=scw-function \
  --cloud=scaleway \
  --layer=compute

# Creates:
# modules/scw-function/
# ├── project.json
# ├── main.tf
# ├── variables.tf
# ├── outputs.tf
# └── README.md

Every new module gets the same structure, targets, and tags. No copy-pasting required.

Testing Strategies

Testing infrastructure code is critical but often overlooked. With Nx, you can build a comprehensive testing pipeline that runs efficiently across all your modules.

Level 1: Static Analysis

terraform fmt & validate
Basic syntax and semantic validation:

nx run-many -t fmt,validate

tflint
Catch common mistakes and enforce best practices:

{
  "lint": {
    "executor": "nx:run-commands",
    "options": {
      "commands": ["tflint --init", "tflint --format compact"],
      "cwd": "modules/scw-vpc",
      "parallel": false
    }
  }
}

checkov
Security and compliance scanning:

nx run-many -t security  # Runs checkov on all modules

Orchestrating Static Analysis with Nx

The magic happens when you combine Nx's dependency graph with your validation pipeline:

# Only validate modules affected by your changes
nx affected -t validate,lint --base=main

# Validate a module and all its dependencies
nx validate scw-k8s --with-deps

# Run security scans in parallel (independent modules)
nx run-many -t security --parallel=3

# Run full quality checks on affected modules
nx affected -t fmt,validate,lint,security --base=main

This can reduce CI time from hours to minutes for large infrastructure repos.

CI/CD Integration

The real power of Nx shows up in CI/CD. Instead of running all checks on all modules, you only run what's needed.

GitHub Actions Example

# .github/workflows/ci.yml
name: CI

on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

jobs:
  affected:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0 # Important for nx affected

      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: "18"

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v2
        with:
          terraform_version: 1.6.0

      - name: Install dependencies
        run: npm ci

      - name: Install tflint
        run: |
          curl -s https://raw.githubusercontent.com/terraform-linters/tflint/master/install_linux.sh | bash

      - name: Derive SHAs for affected command
        uses: nrwl/nx-set-shas@v3

      - name: Run affected format check
        run: npx nx affected -t fmt --base=$NX_BASE --head=$NX_HEAD

      - name: Run affected validate
        run: npx nx affected -t validate --base=$NX_BASE --head=$NX_HEAD

      - name: Run affected lint
        run: npx nx affected -t lint --base=$NX_BASE --head=$NX_HEAD --parallel=3

      - name: Run affected security scans
        run: npx nx affected -t security --base=$NX_BASE --head=$NX_HEAD

What This Does

Only checks affected modules: If you change scw-vpc, only scw-vpc and modules that depend on it run checks
Parallel execution: Independent modules run in parallel (configured with --parallel=3)
Fast feedback: Most PRs only touch 1-2 modules, so CI completes in seconds instead of minutes

GitLab CI Example

# .gitlab-ci.yml
stages:
  - validate
  - security

.nx-affected:
  image: node:18
  before_script:
    - npm ci
    - curl -s https://raw.githubusercontent.com/terraform-linters/tflint/master/install_linux.sh | bash
  variables:
    NX_BASE: $CI_MERGE_REQUEST_DIFF_BASE_SHA
    NX_HEAD: $CI_COMMIT_SHA

affected:validate:
  extends: .nx-affected
  stage: validate
  script:
    - npx nx affected -t validate,lint --base=$NX_BASE --head=$NX_HEAD --parallel=3

affected:security:
  extends: .nx-affected
  stage: security
  script:
    - npx nx affected -t security --base=$NX_BASE --head=$NX_HEAD
  only:
    - merge_requests

Results

In a real-world scenario with 30+ Terraform modules:

Before Nx: Every PR ran checks on all 30 modules (~15 min)
After Nx: Most PRs only check 1-3 affected modules (~2 min)
Savings: 87% reduction in CI time

Managing Releases with Nx

One of the most powerful yet underrated features of Nx is its ability to manage releases in a monorepo. For Terraform modules, proper release management is crucial for versioning, changelog generation, and coordinating changes across dependent modules.

Using Nx Release

Nx provides a built-in release system that handles versioning, changelog generation, and git tagging automatically.

Initial Setup

First, configure Nx Release in your nx.json:

{
  "release": {
    "projects": ["modules/*"],
    "projectsRelationship": "independent",
    "version": {
      "generatorOptions": {
        "packageRoot": "{projectRoot}",
        "currentVersionResolver": "git-tag"
      }
    },
    "changelog": {
      "projectChangelogs": {
        "createRelease": "github",
        "file": "{projectRoot}/CHANGELOG.md",
        "renderOptions": {
          "authors": true,
          "commitReferences": true
        }
      }
    },
    "git": {
      "commit": true,
      "tag": true,
      "commitMessage": "chore(release): publish {projectName} {version}"
    }
  }
}

Creating a Release

When you're ready to release modules that have changed:

# See which modules have changed since last release
nx release --dry-run

# Release only affected modules with automatic version bump
nx release --projects=tag:cloud:scaleway

# Release a specific module with a specific version
nx release scw-vpc --specifier=1.2.0

Automated Semantic Versioning

Nx can automatically determine version bumps based on conventional commits:

# Patch version (fix: commits)
git commit -m "fix(scw-vpc): correct subnet CIDR calculation"

# Minor version (feat: commits)
git commit -m "feat(scw-k8s): add autoscaling support"

# Major version (BREAKING CHANGE: in commit body)
git commit -m "feat(scw-vpc)!: redesign network architecture

BREAKING CHANGE: VPC module now requires new subnet configuration"

# Run release
nx release --projects=tag:cloud:scaleway --dry-run
# Nx will suggest: scw-vpc: 1.2.3 → 2.0.0, scw-k8s: 1.5.0 → 1.6.0

Changelog Generation

Nx automatically generates changelogs for each module:

# modules/scw-vpc/CHANGELOG.md

## 2.0.0 (2025-11-01)

### ⚠ BREAKING CHANGES

- **scw-vpc:** VPC module now requires new subnet configuration

### Features

- **scw-vpc:** redesign network architecture ([a1b2c3d](https://github.com/org/repo/commit/a1b2c3d))

### Authors

- Antoine Caron (@Slashgear)

Release Workflow in CI

Automate releases with GitHub Actions:

# .github/workflows/release.yml
name: Release

on:
  push:
    branches: [main]

jobs:
  release:
    runs-on: ubuntu-latest
    permissions:
      contents: write
      pull-requests: write
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0

      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: "18"

      - name: Install dependencies
        run: npm ci

      - name: Configure Git
        run: |
          git config user.name "github-actions[bot]"
          git config user.email "github-actions[bot]@users.noreply.github.com"

      - name: Release
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          npx nx release --skip-publish

Tagging Strategy

Nx creates git tags for each module release:

# Tags created by Nx
scw-vpc@2.0.0
scw-k8s@1.6.0
scw-database@1.3.1

External consumers can reference specific versions:

# Using git tags to reference specific module versions
module "vpc" {
  source = "git::https://github.com/org/terraform-modules.git//modules/scw-vpc?ref=scw-vpc@2.0.0"
}

Independent vs Fixed Versioning

Nx supports two release strategies:

Independent (recommended for Terraform modules):

Each module has its own version
Modules only bump when they change
"projectsRelationship": "independent"

Fixed (useful for tightly coupled modules):

All modules share the same version
All versions bump together even if unchanged
"projectsRelationship": "fixed"

Benefits

Automatic versioning: Based on conventional commits
Generated changelogs: No manual CHANGELOG.md updates
Git tagging: Automatic and consistent
Affected awareness: Only release modules that changed
GitHub releases: Automatic release creation with notes

This makes managing dozens of Terraform modules significantly less painful than manual versioning and tagging.

Real-World Use Case

Let's walk through a concrete example: a SaaS company managing their Scaleway infrastructure with Nx. This example is based on the nx-terraform-demo repository, which you can clone and explore yourself.

The Setup

They have the following modules:

scw-vpc - Private network configuration
scw-k8s - Kubernetes Kapsule cluster (depends on vpc)
scw-database - Managed PostgreSQL (depends on vpc)
scw-object-storage - S3-compatible object storage (independent)
scw-registry - Container registry (depends on vpc)
scw-loadbalancer - Load balancer (depends on vpc, k8s)
scw-monitoring - Observability stack (depends on k8s)

The Dependency Graph

scw-vpc
  ├── scw-k8s
  │   ├── scw-loadbalancer
  │   └── scw-monitoring
  ├── scw-database
  └── scw-registry

scw-object-storage (independent)

Run nx graph to visualize this automatically.

Scenario: Updating the VPC Module

A developer needs to add a new subnet to scw-vpc. Here's what happens:

# Developer makes changes to scw-vpc
git checkout -b feat/add-subnet
# ... edit scw-vpc/main.tf ...

# Check what's affected
nx show projects --affected
# Output: scw-vpc, scw-k8s, scw-database, scw-registry, scw-loadbalancer, scw-monitoring

# Run validation only on affected modules
nx affected -t validate,lint
# Validates: vpc → k8s, database, registry → loadbalancer, monitoring

# Push and CI runs affected checks
# Only the 6 affected modules are validated, not all 7

Result: Instead of running checks on all modules, only the affected ones run. The independent module (scw-object-storage) is skipped entirely.

Scenario: Adding New Feature to Monitoring

# Developer updates scw-monitoring
nx show projects --affected
# Output: scw-monitoring (only!)

# CI only validates this one module
nx affected -t validate,lint,security --base=main

Result: All other 6 modules skipped, massive time savings.

The Impact

Before Nx:

Every PR: validate all 7 modules, run all checks (~20 min)
Developers waited, context-switched, lost productivity

After Nx:

Small changes: 1-2 modules affected (~3 min)
VPC changes: 6 modules affected (~8 min)
Clear visibility with nx graph on what breaks

Best Practices

After implementing Nx with Terraform modules across multiple teams, here are the lessons learned:

1. Use Meaningful Tags

Tags enable powerful filtering. Be strategic:

{
  "tags": [
    "type:terraform", // All terraform projects
    "cloud:scaleway", // Cloud provider
    "layer:network", // Infrastructure layer
    "team:platform", // Owning team
    "env:multi" // Supports multiple envs
  ]
}

Then run targeted commands:

nx run-many -t validate --projects=tag:team:platform
nx run-many -t security --projects=tag:cloud:scaleway

2. Keep Modules Small and Focused

Each module should do one thing well:

✅ Good: scw-vpc, scw-k8s, scw-database
❌ Bad: scw-full-infrastructure

Small modules = better reusability and easier testing.

3. Use Target Dependencies Wisely

Define a clear pipeline in nx.json:

{
  "targetDefaults": {
    "validate": {
      "dependsOn": ["^validate", "fmt"]
    },
    "test": {
      "dependsOn": ["^test", "validate", "lint"]
    }
  }
}

This ensures tasks run in the right order automatically.

4. Document with terraform-docs

Automate documentation generation:

{
  "docs": {
    "executor": "nx:run-commands",
    "options": {
      "command": "terraform-docs markdown table --output-file README.md .",
      "cwd": "modules/scw-vpc"
    }
  }
}

Run nx run-many -t docs to update all module docs at once.

5. Implement Pre-commit Hooks

Use Husky to run quick checks before commits:

npx husky install
npx husky add .husky/pre-commit "npx nx affected -t fmt-fix,validate"

Limitations and Considerations

Nx is powerful, but it's not a silver bullet. Here are honest tradeoffs to consider:

Learning Curve

Your team needs to learn:

Nx concepts (projects, targets, affected commands)
Node.js/npm basics (even though you're not writing JavaScript)
How to write project.json files

Mitigation: Start with 3-5 modules, prove the value, then expand.

Node.js Dependency

You now need Node.js in your infrastructure workflows. Some teams find this odd.

Counter-argument: Node.js is ubiquitous, lightweight, and your CI already has it. The benefits far outweigh this concern.

State Management Unchanged

Nx doesn't help with:

Terraform state files
State locking
Remote backends
Workspace management

You still need proper Terraform state hygiene. Nx is purely about workspace organization and task orchestration.

Not Ideal for Small Projects

If you have 1-3 Terraform modules that rarely change, Nx is overkill. The overhead isn't worth it.

Use Nx when:

5+ modules
Frequent changes
Multiple team members
Complex dependencies

Module Source References

Modules in the monorepo use relative paths:

source = "../scw-vpc"

External consumers need a different approach:

Publish to a module registry
Use git tags with specific refs
Or keep internal modules internal

When NOT to Use This Approach

Single module projects: Just use plain Terraform
Completely independent modules: If modules never interact, separate repos might be simpler
Team resistant to change: Cultural fit matters more than technical benefits

Be pragmatic. Nx solves specific problems - make sure you have those problems first.

Conclusion

Managing Terraform modules at scale doesn't have to be painful. Nx brings proven monorepo practices from the frontend world to infrastructure code, offering:

Intelligent task orchestration - Run only what's affected
Dependency awareness - Understand module relationships automatically
Faster CI/CD - 80%+ time savings in real-world scenarios
Better DX - Consistent tooling, code generation, and clear workflows

The initial setup requires investment - learning Nx concepts, creating project.json files, updating CI workflows. But for teams managing 5+ Terraform modules with frequent changes, the ROI is substantial.

Start small: pick 3-5 modules, prove the concept, measure the impact. If it works for you (and it likely will), expand to your full infrastructure codebase.

The future of infrastructure management is smart, graph-aware tooling. Nx brings us closer to that future today.

Resources

Demo Repository

nx-terraform-demo - Complete working example with 7 Scaleway Terraform modules managed with Nx

Nx Documentation

Terraform Tooling

terraform-docs - Automated documentation generation
tflint - Terraform linter
checkov - Security and compliance scanning

Scaleway Provider

Further Reading

Monorepo Tools - Comparison of monorepo solutions
Terraform Best Practices

Have you tried managing Terraform with Nx? I'd love to hear about your experience. Find me on Bluesky or GitHub!

How to automatically merge dependabot pull requests with Github Actions ?

Antoine Caron — Tue, 10 May 2022 00:00:00 +0000

Recently, I’ve been working a lot on industrializing this blog (maybe even a little too much). This blog is my playground, I experiment, I play, I test things. I admit that it’s over-engineered but I’m having a lot of fun with it. 😅

Since its release, I tried to use dependabot from github on my repositories. At the beginning without much conviction. As much as I thought it was great in principle, the noise generated by the dozens of open PRs per day was really annoying. As a maintainer of various open-source projects, I have been victim of dependabot PR flood several times. A real nightmare.

I had foolishly made no effort to configure it. Recently, I thought I should give it another chance. So I set up this configuration that allowed me to update my JS dependencies and used github actions in my workflows.



version: 2
updates:
  - package-ecosystem: 'npm' # See documentation for possible values
    directory: '/' # Location of package manifests
    open-pull-requests-limit: 25
    schedule:
      interval: 'daily'
  - package-ecosystem: 'github-actions' # See documentation for possible values
    directory: '/' # Location of package manifests
    open-pull-requests-limit: 25
    schedule:
      interval: 'daily'

No lie, with this configuration, on this blog developed with Gatsby, Dependabot was opening up to 20 Pull Requests per week.

⚠️ Limits

At the beginning, when you see all the PRs opening with all the dependencies updates, you are quite happy. However, managing this every week is really time consuming. Moreover, when we merge these PRs in a chain, we generate in cascade conflicts on the package.json file which prevents the merge 😒. It is therefore often necessary to wait for dependabot to rebase the PR by itself, for the tests to pass and then wait for the Pull Request to merge.

♻️ Let’s merge it automatically!

Having set up E2E tests on this blog, the check ✅ given by my continuous integration workflow is very reassuring.You can have a look at this article which describes how I set up these tests

With this automation, I’m pretty serene when I merge the dependabot update PR. So I thought, “Why not automatically merge the PRs that pass the E2E tests?”

To do this, in my continuous integration workflow, I just had to add a new job configured this way.



 dependabot:
    name: 'Dependabot'
    needs: [build, e2e] # After the E2E and build jobs, if one of them fails, it won't merge the PR.
    runs-on: ubuntu-latest
    if: ${{ github.actor == 'dependabot[bot]' && github.event_name == 'pull_request'}} # Detect that the PR author is dependabot
    steps:
      - name: Enable auto-merge for Dependabot PRs
        run: gh pr merge --auto --merge "$PR_URL" # Use Github CLI to merge automatically the PR
        env:
          PR_URL: ${{github.event.pull_request.html_url}}
          GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}

If you want to see the real example, take a look at this blog workflow.

👮‍ In the end, is it really safer?

SWithout explaining what dependabot is, one could think that all this could bring more security. Indeed, as soon as security patches are available, they would be automatically applied and deployed on my blog without any manual manipulation being necessary.

My availability to merge the PRs of dependabot being limited, one could say that setting up automation improves things. A good E2E testing strategy ensuring that I don’t break anything too important on my site seems really convenient.

But in the end, who controls the updates and what they contain? Who makes sure that they don’t contain malicious code? This blog is not a sandbox for me, so it’s not very problematic. So think carefully before setting it up.

Best practices for Web application maintenance

Antoine Caron — Sun, 05 Sep 2021 00:00:00 +0000

How not to throw away your application every two years?

Feedback based on best practices applied to the web platform developed at Bedrock Streaming

A bit of context

At Bedrock Streaming many teams develop and maintain frontend applications for our customers and users. Some of those application are not very young. In fact, the application I’m mainly working on is a website whose developments started in 2014. I have already mentioned it in different articles of this blog.

You might think: “Oh poor people, maintaining an almost 10 year old application must be hell!”

Don’t worry, it’s not the case! I have worked on projects that are much less old but where the development of new features was much more painful.

Today the project is technically up to date, we must be on the latest version of React while it had started on a version 0.x.x. In this world of web technologies often criticized (eg: the many articles on the Javascript Fatigue) whose tools and practices are constantly evolving, keeping a project “up to date” remains a real challenge.

Moreover, in the context of this project, in almost 10 years, we have had about 100 contributors. Some have only stayed a few months/years. How can we keep the maximum knowledge on “How we do things and how it works?” in such a moving human context?

This is what I would like to present you.

With the help of my colleagues, I have collected the list of good practices that still allow us to maintain this project today. With Florent Dubost, we often thought that it would be interesting to publish it. We hope you will find it useful.

Set rules and automate them

A project that stands the test of time is first and foremost a set of knowledge that is stacked one on top of the other. It’s like the Kapla tower you used to build as a child, trying to get as high as possible. A solid base on which we hope to add as much as possible before a potential fall.

From the beginning of a project, we have to make important decisions about “How do we want to do things? We think for example about “What format for our files? How do we name this or that thing?” Writing accurate documentation of “How we do things” might seem like a good idea.

However, documentation is cool, but it tends to get outdated very quickly. Our decisions evolve, but documentation does not.

“Times change but not READMEs.”

Olivier Mansour (deputy CTO at Bedrock)

Automating the checking of each of the rules we impose on ourselves (on our codebase or our processes) is much more durable. To make it simple, we avoid as much as possible to say “We should do things like that”, and we prefer “we’ll code something that checks it for us”. On top of that, on the JS side we are really well equipped with tools like Eslint that allow us to implement our own rules.

So the reflex we try to adopt is the following:

“We should try to do it like this now!”
“Ok that’s interesting, but how can we make sure we do it like that automatically with our CI (Continuous Integration)?”

Continuous Integration of a project is the perfect solution to not miss anything on every Pull Request we provide. Reviews are only easier because you don’t have to worry about all the rules that are already automated. In this model, the review is more for knowledge sharing than for typo copying and other non-compliance with the project conventions.

In this principle, we must therefore try to banish oral rules. The time of the druids is over, if all the good practices of a project have to be transmitted orally, it will only take longer to guide new developers into your team.

A project is not set in stone. These rules evolve with time. It is therefore preferable to add rules that have a script that will autofix the whole codebase intelligently. Many Eslint rules offer this, and it is a very important selection criteria when choosing new conventions.

eslint --fix

A very strict rule that will force you to modify your code manually before each push is annoying in the long run and will annoy your teams. Whereas a rule (even a very strict one) that can auto-fix itself at commit time will not be seen as annoying.

How to decide to add new rules ?

This question may seem thorny, take for example the case of <tab> / <space> in files. For this, we try to avoid the endless debates and follow the trend and rules of the community. For example, our Eslint configuration base is based on Airbnb’s which seems to have some success in the JS community. But if the rule we want to impose on ourselves is not available in Eslint or other tools, we sometimes prefer not to follow the rule rather than say “We’ll do it without a checking CI”.

The almost exhaustive list 🤞

The file format is tracked managed by Editorconfig, prettier and Eslint. We have opensourced our own configuration, if it is of any use to you.
We use a specific commit name to generate our changelog. To make sure devs follow it, a simple step in our CI checks it.
We don’t want a dev to make our JS bundles very big in production, so we track and measure their size in the CI. We use an in-house tool but we can recommend the [BuildTracker] tool (https://buildtracker.dev/).
Test coverage is not an indicator for the team, not all lines have the same need for us to be tested. Some teams at Bedrock however follow this indicator which at least has the interest to give a trend.
Our unit tests obviously run on the CI, these must pass.
Our functional tests (End to end: E2E) run on Chrome Headless, they must be green.
The logs of our E2E tests are retrieved and parsed to avoid errors or React warnings (the parsing script is however complicated to maintain)
Functional tests run in a sandbox where the whole network is proxied. We make sure that our tests do not depend on a non mocked API that could slow down their execution.
During the E2E tests we check that no image request has generated a 404.
We perform some accessibility checks with Axe during our E2E tests.
We check some rules on the CSS with Stylelint and bemlinter (we don’t use BEM anymore but there is still some style managed in SCSS that we migrate little by little in StyledComponent)
The project is a monorepo on which we try to maintain the same dependencies versions for each package. For that we have developed a tool that allows to do this check monorepo-dependencies-check
We check that our yarn.lock file has not been inadvertently modified or that it has been updated with respect to the modifications of the package.json.
Terraform is used to manage our cloud resources, we check that the file format is correct.

Test, test, test

I hope that in 2021 it is no longer necessary to explain why automatic testing of your application is essential to make it sustainable. In JS, we are rather well equipped in terms of testing tools today. However, the eternal question remains:

“What do we want to test?”

Globally if we search on the internet this question, we see that different needs make emerge very different practices and testing tools. It would be very presumptuous to think that there is a good way to automatically test your application. This is why it is preferable to define one or more test strategies that meet defined and limited needs.

Our test strategies are based on two distinct goals:

To automate the verification of the functionalities proposed to the users by putting ourselves in their place.
To provide us with efficient solutions to specify the way we implement our technical solutions to allow us to make them evolve more easily.

To do this, we perform two “types of tests” that I propose to present here.

Our E2E tests

We call them “functional tests”, they are End-to-end (E2E) tests on a very efficient technical stack composed of CucumberJS, WebdriverIO with ChromeHeadlessThis is a technical stack set up at the beginning of the project (at the time with PhantomJS for the oldest among you)

This stack allows us to automate the piloting of tests that control a browser. This browser will perform actions that are as close as possible to what our real users can do while checking how the site reacts.

A few years ago, this technical stack was rather complicated to set up, but today it is rather simple to do.The site that hosts this blog post is itself proof of this. It only took me about ten minutes to set up this stack with the WebdriverIo CLI to verify that my blog is working as expected.

I recently published an article presenting the implementation of this stack.

So here is an example of an E2E test file to give you an idea:

Feature: Playground

  Background: Playground context
    Given I use "playground" test context

  Scenario: Check if playground is reachable
    When As user "toto@toto.fr" I visit the "playground" page
    And I click on "playground trigger"
    Then I should see a "visible playground"
    And I should see 4 "playground tab" in "playground"

    When I click on "playground trigger"
    Then I should not see a "visible playground"

    # ...

And it looks like this in local with my Chrome browser!

Here is a diagram that explains how this stack works:

Today, Bedrock’s web application has over 800 E2E test cases running on each of our Pull Request and the master branch. They assure us that we are not introducing any functional regression and that’s just great!

👍 The positives

WebdriverIO also allows us to run these same tests on real devices on a daily basis through the paid SAAS service Browserstack. So we have every day a job that makes sure that our site works correctly on a Chrome last version on Windows 10 and Safari on MacOs.
These tests allow us to easily document the functionality of the application using the Gherkin language.
They allow us to reproduce cases that are far from nominal. In a TDD logic, they allow us to advance on the development without having to click for hours.
These tests allowed us not to break the old version of the site which is still in production for some customers while our efforts are concentrated on the new one.
They give us real confidence.
Thanks to our library superagent-mock, we can fixturer (plug, mock) all the APIs we depend on and thus even check the error cases. Also, mocking the browser’s XHR layer allows for a significant improvement in test execution time. 🚀
They give us access to extended uses like:
- checking accessibility rules
- check the browser console logs (to avoid introducing errors or React Warning for example)
- monitoring all network calls of the site through a proxy
- and so on…

👎 The complications

Maintaining this stack is complicated and expensive. Since few resources are published on this domain, we sometimes find ourselves digging for days to fix them 😅. Sometimes we feel quite alone in having these worries.
It is very easy to code a so-called flaky E2E test (ie: a test that can fail randomly). They make us think that something is broken. They sometimes take us a long time to stabilize. It is still much better to remove a test that will not give you a stable result.
Running all the tests takes a lot of time on our continuous integration. We must regularly work on their optimization so that the feedback they provide you is as fast as possible. These important times also cost money, because we have to run these tests on machines. For your information, the infrastructure of the website (just the hosting of our Node servers + static files + CDN) cost much less than our continuous integration. This obviously makes our Ops team smile! 😊
The new recruits in our teams have often never done this kind of testing, so there is a ~~struggle~~ phase of learning…
Some features are sometimes too complicated to test with our E2E stack (for example, payment paths that depend on third parties). So we sometimes fall back on other techniques with Jest, especially with a less unitary scope.

Our “unit” tests

To complete our functional tests we also have a stack of tests written with Jest. We call these tests unit tests because we have as a principle to try to always test our JS modules independently from the others.

Let’s not debate here about “Are these real unit tests?”, there are enough articles on the internet about this topic.

We use these tests for different reasons that cover needs that our functional tests do not cover:

to help us develop our JS modules with TDD practices.
to document and describe how a JS module works.
test very/too complicated edge cases with our E2E tests.
facilitate the refactoring of our application by showing us the technical impacts of our modifications.

With these tests, we put ourselves at the level of a utility function, a Redux action, a reducer, a React component. We rely mainly on the automock functionality of Jest which allows us to isolate our JS modules when we test.

The previous image represents the metaphor that allows us to explain our unit testing strategy to newcomers.

You have to imagine that the application is a wall made of unit bricks (our ecmascript modules), our unit tests must test one by one the bricks in total independence from the others. Our functional tests are there to test the cement between the bricks.

To summarize, we could say that our E2E tests test what our application should do, and our unit tests make sure to check how it works.

Today there are more than 6000 unit tests that cover the application and allow to limit regressions.

👍

Jest is really a great library, fast, complete, well documented.
Unit tests help us a lot to understand several years later how it all works.
We always manage to unit test our code, and it complements our E2E tests well.
The automock is really handy for breaking down tests by modules.

👎

Sometimes we found ourselves limited by our E2E test stack and couldn’t rely solely on unit tests. We were missing something to be able to make sure that the cement between the bricks worked as we wanted it to. For this, a second test stack Jest was set up called “integration test” where the automock is disabled.
The abuse of Snapshot is dangerous for your health. The use of “Snapshot testing” can save time on the implementation of your tests but can reduce the quality. Having to review a 50 line object in Snapshot is neither easy nor relevant.
With the depreciation of EnzymeJS, we are forced to migrate to React Testing Library. It is of course possible to unit test components with this new library. Unfortunately, this is not really the spirit and the way to do it.React Testing Library pushes us not to play with shallow rendering.

Our principles

We try to always follow the following rules when asking the question “Should I add tests?“.

If our Pull Request introduces new user features, we need to integrate E2E test scenarios. Unit tests with Jest can complete / replace them accordingly.
If our Pull Request aims to fix a bug, it means that we are missing a test case. We must therefore try to add an E2E test or, failing that, a unit test.

It is while writing these lines that I think that these principles could very well be automated. 🤣

The project stays, the features don’t

“The second evolution of a feature is very often its removal.”

As a matter of principle, we want to make sure that every new feature in the application does not base its activation on simply being in the codebase. Typically, the lifecycle of a feature in a project can be as follows (in a Github Flow):

a person implements on a branch
the feature is merged on master
it is deployed in production
lives its feature life (sometimes with bugs and fixes)
the feature is not needed anymore
a person unravels the code and removes it
new deployment

To simplify some steps, we have implemented feature flipping on the project.

How does it work?

In our config there is a map key/value that lists all the features of the application associated with their activation status.

const featureFlipping = {
  myAwesomeFeature: false,
  anotherOne: true,
}

In our code, we have implemented conditional treatments that say “If this feature is activated then…“. This can change the rendering of a component, change the implementation of a Redux action or disable a route in our react-router.

But what’s the point?

We can develop new evolutions progressively by hiding them behind a configuration key. We deliver features in production without activating them.
In a test environment, we can overload this config to test features that are not yet activated in production.
In the case of a white label site, we can propose these features to our customers as possible options.
Before deleting code of a feature, we deactivate it and clean it up without risk.
Thanks to an in-house tool called Applaunch, this feature flipping config can be overloaded on time in a GUI without deployment. This allows us to activate features without putting the code into production. In the event of an incident, we can deactivate features that have been degraded.

To give you a more concrete example, between 2018 and 2020 we completely overhauled the application’s interface. This graphical evolution was just a featureFlipping key. The graphical redesign was not a reset of the project, we still live with both versions (as long as the switchover of all our customers is not completed).

A/B testing

Thanks to the great work of the backend and data teams, we were even able to extend the use of feature flipping by making this configuration modifiable for sub-groups of users.

This allows us to deploy new features on a smaller portion of users in order to compare our KPI.

Decision making, technical or product performance improvement, experimentation, the possibilities are numerous and we exploit them more and more.

The future flipping.

Based on an original idea by Florent Lepretre.

We regularly had the need to activate features at ~~very~~ early hours in the future. For that we had to be connected at a precise time on our computer to modify the configuration on the fly.

To avoid forgetting to do this, or doing it late, we made sure that a configuration key could be activated from a certain date. To do this, we evolved our selector redux which indicated if a feature was activated so that it could handle date formats and compare them to the current time.

const featureFlipping = {
  myAwesomeFeature: {
    offDate: '2021-07-12 20:30:00',
    onDate: '2021-07-12 19:30:00',
  },
}

Many coffees ☕️ at 9am have been saved by future flipping.

Monitor, Measure, Alert

To maintain a project as long as bedrock’s web application, testing, documentation and rigor are not enough. You also need visibility on what works in production.

“How do you know that the application you have in production right now is working as expected?”

We assume that no functionality works until it is monitored. Today, monitoring in Bedrock on the frontend side takes the form of different tools and different stacks. I could quote NewRelic, a Statsd, a ELK stack or even Youbora for the video.

To give you an example, each time a user starts a browsing session we send an anonymous monitoring Hit to increment a counter in Statsd. We then have to define a dashboard that displays the evolution of this number in a graph. If we observe a too important variation, it can allow us to detect an incident.

Monitoring also offers us solutions to understand and analyze a bug that occurred in the past. Understanding an incident, explaining it, finding its root cause are the possibilities that are open to you if you monitor your application. Monitoring can also allow you to better communicate with your customers about the impact of an incident and also to estimate the number of impacted users.

With the multiplication of our customers, monitoring our platforms well is not enough. Too much data, too many dashboards to monitor, it becomes very easy to miss something. So we started to complement our metrics monitoring with automatic alerting. Once we have enough confidence in the metrics, we can easily set up alerts that will warn us if there is an inconsistent value.

However, we try to always trigger alerts only when it is actionable. In other words, if an alert sounds, we have something to do. Sounding alerts that do not require immediate human action generates noise and wastes time.

Limit, monitor and update your dependencies

What goes out of date faster than your shadow in a web project based on javascript technologies are your dependencies. The ecosystem evolves rapidly and your dependencies can quickly become unmaintained, out of fashion or completely overhauled with big breaking changes.

We therefore try as much as possible to limit our dependencies and avoid adding them unnecessarily. A dependency is often very easy to add but it can become a real headache to remove.

The graphic component libraries (e.g. React bootstrap, Material Design) are a good example of dependencies that we do not want to introduce. They can make integration easier at first, but they often freeze the version of your component library later on. You don’t want to freeze the React version in your application for two form components.

Monitoring is also part of our dependency management routines. Since the addition of reporting security flaws in an NPM package, it is possible to know if a project has a dependency that contains a known security flaw with a simple command. So we have daily jobs on our projects that run the yarn audit command to force us to apply patches.

Dependency maintenance is greatly facilitated by our E2E test stack which sounds the alarm if the version upgrade generates a regression.

Today, except for security flaws, we update our dependencies “when we have time”, often at the end of sprint. We are not satisfied with this because some dependencies can be forgotten. I personally use tools like yarn outdated and Dependabot on my personal projects to automate the update of my dependencies.

Accepting your technical debt

A project will always accumulate technical debt. This is a fact. Whether it is voluntary or involuntary debt, a project that resists the years will inevitably accumulate debt. Even more so, if during all these years you keep adding features.

Since 2014, our best practices, our ways of doing things have evolved well. Sometimes we decided these changes but sometimes we underwent them (an example, the arrival of functional components with React and the Hooks api).

Our project is not completely “state of art” and we assume it.

We try to prioritize our refactoring topics on the parts of the application on which we have the most concern, the most pain. We consider that a part of the application that we don’t like but on which we don’t need to work (bring evolutions) doesn’t deserve that we refactor it.

I could name many features of our application that have not evolved functionally for several years. But since we have covered these features with E2E tests since the beginning, we didn’t really have to touch them.

As said above, the next evolution of a code feature is sometimes its deactivation. So why spend time rewriting the whole application?

In any case, the code becomes “legacy”.
As long as the features are tested, nothing obliges us to refactor everything permanently so that our entire codebase is state of art.
We focus on our pain points, we re-factor what we really need to evolve.

To summarize

The best practices presented here are obviously subjective and will not be perfectly/directly applicable in your contexts. However, I am convinced that they can probably help you identify what can make your project go from fun to stale. At Bedrock we have other practices in place that I haven’t listed here but that will be the occasion for a new article sometime.

Finally, if you want me to go into more detail on some of the chapters presented here, don’t hesitate to tell me, I could try to dedicate a specific article to it.

How to setup end to end tests with WebdriverIo on Github action ?

Antoine Caron — Fri, 13 Aug 2021 06:34:27 +0000

I recently set up an end-to-end testing stack to make sure I wasn’t going to introduce any functional regressions to my blog. With several years of experience using E2E (End to End) testing technologies, I still encountered some difficulties in setting it up.

In order to save you time I give you a step by step tuto for the installation of the tools necessary to have an efficient test stack in a few minutes.

Make it run locally first

You need to install some dependencies to your project first. For me the project was a quite simple Gatsby site, but you could settle it on whatever website you develop that have a package.json file. In this case I will start from a just initialised npm package. Make sure you have a Chromium or Chrome browser installed locally. We are really lucky because WebdriverIo teams have developed a CLI topic that does this job for you.

yarn add -D @wdio/cli

Then all you need is to trigger the initialisation of a new configuration. The CLI will show you a prompt you should follow.

yarn wdio config

Here is what you should do for each question:

Select first On my local machine.
You should definitely use cucumber, Gherkin language is so great to declare humanly readable user behaviors.
Choose the compiler you need, I personally use No. You could decide if you wannt to handle Babel or Typescript depending on you habits.
For the Where are your feature files located? I like to use the default value.
Same for step definitions, use default value.
Let’s see what tests WebdriverIo will autogenerate.
Do you want to use page objects ? I wouldn’t use that if it is the first time you setup those kind of testing stack, say n.
You should select spec reporter here.
Please select chromedriver service only.
Override the base url with you local context (for example, add the port on which you expose your app locally).

Normally the CLI generates some files for you and add the missing dependencies you need.

Here is my package.json

{
  "name": "example-wdio",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "devDependencies": {
    "@wdio/cli": "^7.9.1",
    "@wdio/cucumber-framework": "^7.9.1",
    "@wdio/local-runner": "^7.9.1",
    "@wdio/spec-reporter": "^7.9.0",
    "chromedriver": "^92.0.1",
    "wdio-chromedriver-service": "^7.2.0"
  }
}

You should see a login.feature file in ./features/ folder.

Feature: The Internet Guinea Pig Website

  Scenario Outline: As a user, I can log into the secure area

    Given I am on the login page
    When I login with <username> and <password>
    Then I should see a flash message saying <message>

    Examples:
      | username | password | message |
      | tomsmith | SuperSecretPassword! | You logged into a secure area! |
      | foobar | barfoo | Your username is invalid! |

With the steps definitions in ./step-definitions/steps.js

const { Given, When, Then } = require('@cucumber/cucumber')

Given(/^I am on the (\w+) page$/, async (page) => {
  await browser.url(`https://the-internet.herokuapp.com/${page}`)
})

When(/^I login with (\w+) and (.+)$/, async (username, password) => {
  await $('#username').setValue(username)
  await $('#password').setValue(password)
  await $('button[type="submit"]').click()
})

Then(/^I should see a flash message saying (.*)$/, async (message) => {
  await expect($('#flash')).toBeExisting()
  await expect($('#flash')).toHaveTextContaining(message)
})

Let’s see what those default example tests are doing ! For that, you just have to type this in your console and tada 🎉

yarn wdio run wdio.conf.js

Here is the log it will generate thanks to the spec reporter !

------------------------------------------------------------------
[chrome 92.0.4515.131 mac os x #0-0] Running: chrome (v92.0.4515.131) on mac os x
[chrome 92.0.4515.131 mac os x #0-0] Session ID: edd73da800a210e7c677c69cd064004f
[chrome 92.0.4515.131 mac os x #0-0]
[chrome 92.0.4515.131 mac os x #0-0] » /features/login.feature
[chrome 92.0.4515.131 mac os x #0-0] The Internet Guinea Pig Website
[chrome 92.0.4515.131 mac os x #0-0] As a user, I can log into the secure area
[chrome 92.0.4515.131 mac os x #0-0] ✓ Given I am on the login page
[chrome 92.0.4515.131 mac os x #0-0] ✓ When I login with tomsmith and SuperSecretPassword!
[chrome 92.0.4515.131 mac os x #0-0] ✓ Then I should see a flash message saying You logged into a secure area!
[chrome 92.0.4515.131 mac os x #0-0]
[chrome 92.0.4515.131 mac os x #0-0] As a user, I can log into the secure area
[chrome 92.0.4515.131 mac os x #0-0] ✓ Given I am on the login page
[chrome 92.0.4515.131 mac os x #0-0] ✓ When I login with foobar and barfoo
[chrome 92.0.4515.131 mac os x #0-0] ✓ Then I should see a flash message saying Your username is invalid!
[chrome 92.0.4515.131 mac os x #0-0]
[chrome 92.0.4515.131 mac os x #0-0] 6 passing (3s)

For now the tests are not testing your application at all. You will find many ressources on how to use Cucumber JS to write great test to describe and test your application features.

Configure the Github action workflow

Now that we have managed to run E2E tests on our machine, we just need to set up a continuous integration workflow that will automatically check on your Pull Request and on your main branch that all the tests are ok.

I use for most of my projects Github Action and I’m happy with it, so the following example will use this tool. However, the principle also works with a Gitlab, Jenkins or other pipeline.

With Github Action you need to setup a Yaml file to describe you workflow. Let’s create ./.github/workflows/continuous-integration.yml file in your project !

name: Continuous Integration

on:
  push:
    branches:
      - '**'

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      # First you need to install a chromium browser in your runner
      - name: Install Chromium
        run: sudo apt-get install chromium-browser

      # You fetch the current ref
      - uses: actions/checkout@v2

      # Use Node version above 14.x you want to use
      - name: Use Node.js
        uses: actions/setup-node@v1
        with:
          node-version: 16.x

      # Install your dependencies (with yarn, npm no matter)
      - run: yarn install
        name: Install dependencies

      # This is where you could build your app
      # You could also start your server process (take a look at https://github.com/Slashgear/slashgear.github.io/blob/source/.github/workflows/continuous-integration.yml)

      # Run your test with the same command you uses locally
      - run: yarn wdio run wdio.conf.js
        name: Running E2E tests

Let’s try that ! 🚀

😭 Sadly you should face a very common error with E2E testing with chrome.

[0-0] 2021-08-12T20:34:12.293Z ERROR webdriver: Request failed with status 500 due to unknown error: unknown error: Chrome failed to start: exited abnormally.
[0-0] (unknown error: DevToolsActivePort file doesn't exist)
[0-0] (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
[0-0] 2021-08-12T20:34:12.293Z ERROR webdriver: #0 0x5631c488ba63 <unknown>

DevToolsActivePort file doesn't exist basically means your browser did not succeed to start. The main reason of the issue is the fact that you ask WebdriverIo to start Chrome browser with a Graphic User Interface on a ubuntu runner that don’t have a screen at all 😅.

We need to create a new configuration of Webdriver specific to github action that extends our basic one. Let’s create wdio-github.conf.js next to wdio.conf.js !

const basicConfig = require('./wdio.conf')

exports.config = {
  ...basicConfig.config,
  // We only need to override the Chrome configuration of capabilities
  capabilities: [
    {
      maxInstances: 5,
      browserName: 'chrome',
      acceptInsecureCerts: true,
      // We need to extends some Chrome flags in order to tell Chrome to run headless
      'goog:chromeOptions': {
        args: ['--headless', '--disable-gpu', '--disable-dev-shm-usage'],
      },
    },
  ],
}

We now just have to change the yarn wdio run command in our Github Workflow YAML file. You just need to push it and github will start it for you !

Setup multi browser configuration !

Chrome is not the only browser, and I hope it never will never be !

WebdriverIo is a great solution to run test on multiple Browsers. And I won’t show you here what great features you could use with remote running solution like BrowserStack or Saucelabs directly with WebdriverIo.

Let’s configure a Firefox locally !

Make sur to install a Java JDK 8 on your machine. Small trick for Macos users like me run brew install adoptopenjdk/openjdk/adoptopenjdk8 does the job !
Run yarn add -D @wdio/selenium-standalone-service to install selenium services working with WebdriverIo.
Make sure you have Firefox installed locally.
Now the last thing we need to do is to update our configuration to add firefox capabilities.

In wdio.conf.js, just replace capabilities and services arrays with this

{
  capabilites: [
    {
        maxInstances: 5,
        browserName: 'chrome',
        acceptInsecureCerts: true,
    },
    {
        maxInstances: 5,
        browserName: 'firefox',
        acceptInsecureCerts: true,
    }
  ],
  services: ['chromedriver', 'selenium-standalone'],
}

If you run the command yarn wdio run wdio.conf.js now, it will trigger both test on Firefox and Chrome and that is completely awesome !

The last thing we need to do is to update our Github specific configuration in order to make it work also in your continuous integration. You need to update capabilites too, in order to add Firefox and to make it boot Headless.

In wdio-github.conf.js

const basicConfig = require('./wdio.conf')

exports.config = {
  ...basicConfig.config,
  // We only need to override the Chrome configuration of capabilities
  capabilities: [
    {
      maxInstances: 5,
      browserName: 'chrome',
      acceptInsecureCerts: true,
      // We need to extends some Chrome flags in order to tell Chrome to run headless
      'goog:chromeOptions': {
        args: ['--headless', '--disable-gpu', '--disable-dev-shm-usage'],
      },
    },
    {
      maxInstances: 5,
      browserName: 'firefox',
      acceptInsecureCerts: true,
      'moz:firefoxOptions': {
        args: ['-headless'],
      },
    },
  ],
}

And that’s all folks !

Have fun covering your application with E2E tests ! Don’t hesitate to @ me on twitter if this How to helped you.

If you need to see the example application I used in this tutorial, take a look at this example Github repository.

Should we replace webpack by 🗻 Snowpack 🗻?️

Antoine Caron — Thu, 03 Jun 2021 21:27:34 +0000

After webpack, esbuild and vitejs, let’s take a look at snowpack. The latter seems to be making a name for itself in the bundler ecosystem for a few years now. So I suggest that we study its functionalities, its strong points but also its weak points.

Snowpack

What is Snowpack? It is a web application packager that offers :

a rich and powerful development experience
a production mode that offers the features necessary to optimize the assets of the site.

# npm:
npm install --save-dev snowpack
# yarn:
yarn add --dev snowpack

ESModule in the browser

Packagers such as webpack and rollup rely on the construction of a dependency tree that is analyzed and packaged at each modification. The build step of the modified files is still necessary, but the packaging operation (merging the different modules in the form of a bundle that is sent to the browser) is no longer necessary. Our browsers now know how to manage ESModules](https://caniuse.com/?search=esmodule), so it is no longer necessary to package our source modules to merge them.

This is the same idea that Evan You has taken up in Vite

With this principle, the modification of a file triggers only the build of this file, and this file only. You can have a project with thousands of javascript modules, the build time for each modification will not be affected. For dependencies (vendors), Snowpack builds them once and caches them to do it again only if they have changed.

Here is a diagram in the Snowpack doc showing the interest to avoid packaging in dev.

If you use a Create React App project, you just have to add Snowpack as a dependency, you don’t have to change anything if you didn’t extend the webpack config. I invite you to test snowpack with the template generators:

npx create-snowpack-app react-snowpack --template @snowpack/app-template-react

Don’t worry! Snowpack is not only compatible with React, you can use Vue, Svelte or just javascript.

A rich configuration (maybe too much)

Like webpack, snowpack proposes to configure its use by an object. I must admit that after playing with it quickly, I am a bit disappointed. I see a meager documentation page that seems to describe many parameterizable keys.

If you come from webpack, you won’t be lost, it’s very similar without being exactly the same.

Not touching the default configuration can be an excellent idea.

A nice collection of plugins

Snowpack is not that new. A community has built up to create a rich plugin ecosystem. Some of these plugins seem “core” because they are under the @snowpack scope but many packages are ported by a few people independent of the project. It’s reassuring without being reassuring, I’ve personally experienced webpack updates being blocked/delayed while waiting for compatibility of some plugins that were no longer maintained.

Beware of the plugins you use!

Contrary to Vite which natively offers a lot of nice features, snowpack works like webpack by relying on plugins to enrich the API. It’s a gamble, it can be complicated to keep this ecosystem of plugins up to date and efficient to continue to guarantee the interest of a migration to snowpack.

Server Side Rendering

Snowpack offers a solution to implement your applications with server-side rendering. It is clear that the need for SSR is still felt on our frontend applications for SEO or rendering performance reasons. Unfortunately, on the application packaging side, it is still complicated and it is often necessary to make two builds:

a build for the client aka the Browser
a build for the Node server

Double build, double punishment!

The technique proposed by Snowpack remains limited but it’s still correct. I propose a slight improvement of the implementation by adding the buffered server rendering mechanics.

const {readFileSync} = require('fs');
const {startServer} = require('snowpack');
const server = await startServer({ ... });
const runtime = server.getServerRuntime();

app.use(async (req, res, next) => {
  const importedComponent = await runtime.importModule('/dist/MyReactComponent.js');
  const MyReactComponent = importedComponent.exports.default;
  const html = ReactDOMServer.renderToNodeStream(React.createElement(MyReactComponent, null));
  // Directly write the head of page
  res.write(`
    <html>
    <head>
      <title>Hello 👋</title>
    </head>
    <body>
    <div id="app">
  `)
  // Render bufferized version of App
  html.pipe(res, { end: false });
  // When buffer end, we add the closing tags of page
  html.on('end', () => {
    res.write(`
      </div>
      </body>
      </html>
    `);
    res.end();
  });
});

Webpack, esbuild, quick, or snowpack ?

Clearly after this series of articles where I tried to study this new generation of tools to bundle applications, I must admit that I am very surprised. We can clearly see that the support of ES Modules in the browser marks the entry into a new era. As Sindre Sorhus reminds us in his latest article, with the end of Node 10 support and the capabilities of our current browsers, it is no longer necessary to target CJS.

Caching strategies and the use of CJS modules seem to be outdated for our needs in development environments. We can see that Vite and Snowpack offer this new mechanism which seems to be really efficient. Doing a once for all build of the libraries and of each source file is a great idea to not suffer from a too slow startup time of our big web applications.

Keep an eye on Esbuild

However, the performance of these new tools also relies heavily on Esbuild. The idea of using a more optimized stack to read, parse, combine JS or TS modules with languages that allow for finer IO and memory management is really the keystone of this new generation of tools. Even before choosing whether to stay with webpack, or use Vite and Snowpack, it is certain that Esbuild will have to be followed closely. This lib has not finished surprising us. You should also look at a tool like SWC which is a direct competitor of Esbuild. There are certainly other tools being created at the moment.

I use webpack and have configured it a lot

If you are in this situation, you may unfortunately be forced to keep webpack. This is not bad news, it is a very good tool that is far from being obsolete. It is likely that the webpack team will come up with new performance improvements, perhaps through the use of ESModules.

You can also try to use snowpack in a development environment. There is a plugin to use webpack in the snowpack prod build.

I really want to reduce the configuration of my application build

If you don’t want to keep your webpack configuration files anymore, which can be sometimes difficult to maintain, the alternative proposed by Vite can be a great option. Just keep in mind that this solution is still young.

If you are going to use Vite, I suggest that you minimize the configuration you can do to it. This will make it easier for you to keep up with the new versions that are likely to arrive in the coming months.

I use a CLI that manages my build configuration for me

You are using VueCLI, CRA, or other and you have not ejected your configuration. You don’t like to touch the build configuration of your application because the tools are complex and you don’t want to spend a lot of time configuring them. So I recommend you to stay as much as possible with the default configuration of your project as long as its performance does not bother you.

However, nothing prevents you from testing tools like Vite that work directly without configuration with your already generated projects. If the startup time of your development environment becomes too important, this can really be an interesting solution for you.

And it’s already the end?

I think I’ve covered the new tools proposed by the community to package our web applications. If you have other tools that would be interesting to look at, don’t hesitate to suggest them to me on a social network like Twitter. See you soon for new javascript peregrinations !👋

Thanks to Jérémie for the review 🤗

Vite, a new webpack killer ?!😮

Antoine Caron — Fri, 26 Mar 2021 00:00:00 +0000

After having dealt on this blog in several articles about webpack, I have started a new series of articles about new generation bundlers. It seems that we are now entering a new era of web tooling.

Having introduced esbuild and its interesting features, it seems logical to me to deal with the “little brothers” bundlers. So let’s start with Vite, a project from the VueJS community and started by Evan You.

Same starting point

With Vite we are in the same situation as with esbuild. Globally the community is happy with the current tools even if they have some performance issues.

As I said in my previous article, having a big JS project today often means having a dev and prod environment that is sometimes a bit slow. Between the choice of tools that do not sufficiently exploit parallelization or memory optimization, or repetitive operations that exploit caching very little, it is easy to identify culprits for these slowness.

NB: I have proposed some solutions to improve the performance of your webpack build in this article.

Moreover, the second generation tools (like webpack, Rollup, Parcel) could not handle from the beginning recent features of our browsers like ESmodules.

What is the idea?

The revolutionary idea of Vite is to combine two tools for two different needs in order to optimize the build to be as fast as possible. The two tools that make up Vite are esbuild and Rollup, so nothing new. But why two bundling tools? Basically, for two reasons:

Our dependencies don’t change often, so re-evaluating their module tree at each build is not necessary. We can generate the bundle of our vendor once and for all with an optimized tool like esbuild. This bundler is very fast and allows a quick start of the server.
The modules in our source code are subject to a lot of changes unlike the dependencies. So Vite uses another treatment based on ESM that works natively on recent browsers.

Feel free to read this doc page for more details.

In fact, how does it work?

In order to play a little with the tool, I propose a small presentation through an example project. Let’s start by creating an example project. I show you here how to create it and put you the link of the github repository in which I published it.

mkdir "example-vite"
cd "example-vite"

git init
yarn init -y

Installing Vite is very easy, you just need a dependency.

yarn add -D vite

For the purpose of this example, I propose an example with React (there are already many examples with Vue 😉 )

yarn add react react-dom

Let’s add 3 usual commands to launch Vite in the package.json file

package.json

{
  "scripts": {
    "dev": "vite",
    "build": "vite build",
    "serve": "vite preview"
  }
}

Finally, we need some small source files for Vite to play with.

index.html

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <title>Example Application with Vite</title>
  </head>
  <body>
    <div id="app"></div>
    <script type="module" src="/src/main.jsx"></script>
  </body>
</html>

src/main.jsx

import React from 'react'
import ReactDOM from 'react-dom'

ReactDOM.render(
  <React.StrictMode>
    <h1>Hello world dear readers ! </h1>
  </React.StrictMode>,
  document.getElementById('app')
)

Now you just have to start the development server with this command.

yarn dev

🎉 Normally, within milliseconds Vite has started a server and if you go to https://localhost:3000 that presents you with this beautiful application.

Let’s not lie, it’s a bit sad, let’s see how Vite does if we add some CSS. Let’s put a beautiful color chartreuse to this title. Let’s add the following stylesheet.

src/index.css

h1 {
  color: chartreuse;
}

Then we just need to add an import to this new file.src/main.jsx

import './index.css'

There, now you have a beautiful color that clearly lacks contrast to be accessible!

If you now try to run the yarn build command, you can see that Vite will build you a dist folder. Without any settings I have these different resources ready to be deployed on a static server.

We can observe that natively Vite exports 2 javascript bundles (1 for the sources, 1 for the vendors/dependencies) and a CSS bundle that exports the style that has been imported in your application. And this is clearly a big plus of Vite compared to the competition of other tools (although parcel offers some of the same logic). The build is extremely fast and does what you would expect it to do without having to configure it. Sorry but I think it’s great!

I don’t know if you know react-refresh, the official React package that allows you to optimize the auto-refresh of a React application. This package allows you to update your React components on the fly without them losing their state.Vite even though it was born out of the VueJS community, is not specifically oriented towards a frontend framework. Tools like react-refresh are therefore not included by default. So you have to define it in the configuration. Unfortunately, Vite doesn’t fare any better than the other tools; we are forced to define yet another config file at the root of the project.

So let’s install the plugin:

yarn add -D @vitejs/plugin-react-refresh

vite.config.js

import { defineConfig } from 'vite'
import reactRefresh from '@vitejs/plugin-react-refresh'

export default defineConfig({
  plugins: [reactRefresh()],
})

Now I wanted to test some of the more advanced features that you can expect from a quality bundler. So I set up a single page application that uses lazy loading. I won’t show you how I did it in this article, it would be too long but you can go directly to test it in your browser.

So clearly for lazy loading it’s easy with Vite, I’m amazed! The tool immediately detects my use of the dynamic import import() to generate a separate chunk for the JS but also the CSS.

const Content = React.lazy(() => import('./lazy'))

The strengths of Vite

It is clear that Vite has many nice features and advantages. Besides its incredible speed, I would like to note that this bundler offers a really well thought out autoconfiguration.

In the demo I gave you earlier, I didn’t show you that Vite handles natively and without configuration static files, Web Workers, WASM binaries. But it doesn’t stop there, we have to admit that this magical tool also natively supports JSX and Typescript.

When it comes to style management, Vite is not to be outdone. Without any plugin or configuration, it allows you to manage CSS @imports, preprocessors like SASS and LESS, CSS modules and even the postprocessor PostCSS (if you define a configuration).

More anecdotally, Vite knows how to manage your .env file to manage your environment variables thanks to dotenv.

But the feature that completely blew me away was the rather simple setup of the SSR.

This is the first time I’m talking about a bundler that natively handles Server Side Rendering. I use other tools in production for the different applications I develop. Unfortunately, it is still very complicated to set up such an architecture (even with tools like Webpack).

So we can see that developers are mainly turning to turnkey solutions like Next and Nuxt to manage these issues for them. This is not a bad thing in itself. However, I think that it is sometimes necessary in some projects to take control of this functionality for business needs. So we can only be happy that tools like Vite have thought about it. I invite you to go read this page of the documentation of Vite to understand how to implement this.

So we stop using webpack?

After this laudatory presentation of this tool, one could ask the question yes. However, you should not forget a simple rule.

Everything that a tool tends to do magically for you often becomes much more complicated to customize.

The many choices Vite seems to make to improve the developer experience worry me a bit. I’m a bit afraid that all this default configuration will be complicated to maintain by the Vite teams.

Contrary to esbuild which has the motto “I want to offer a tool that does few things but does them very well”, we have here a tool that makes a lot of promises. However, we must recognize that Vite also offers to use and define plugins to extend its functionalities without making them native to the main tool.

Moreover, it should not be forgotten that Vite is also based on Rollup, a second generation bundler that benefits from a rich ecosystem of plugins that are mostly compatible. But the Rollup configuration is very complicated to edit and maintain, so I hope you won’t have to touch it if you are tempted to test Vite on your applications.

I would like to point out that some tools like VuePress offer today an alternative Vitepress which uses Vite as a bundler.

Before jumping on the Vite solution, I suggest you to test another third generation bundler which is much talked about: Snowpack

Stay tuned!

Esbuild, the incredibly fast 💨 and promising bundler 📈!

Antoine Caron — Sat, 13 Mar 2021 00:00:00 +0000

I’ve been playing with JS bundlers for several years. Still convinced of the necessity of using these tools (don’t let me believe that you don’t package your JS modules in production 😅), I played a lot with webpack. Especially for performance, optimization and custom plugins usage issues.

I still think that in 2021, webpack is the most industrial and successful solution to bundle my web applications. I hear that tools like parcel and rollup are still good alternatives. However, webpack probably has the biggest community and is used by many projects.

But let’s face it, today we are satisfied with these bundling tools despite their poor performance. I work every day on a project with several thousands of “modules” solved by webpack and it is sometimes a pain 🥱.

Despite an intensive use of cache and workers, webpack shows some limitations to package large applications.

Why does esbuild look interesting ?

I can’t think of an easier way to express it than to explain it to you simply:

The first time I ran esbuild on my test web app, I thought it crashed when in fact it ran at an absolutely insane speed.

To install it, it’s not complicated:

yarn add -D esbuild

npm install -D esbuild

Or even with NPX

npx esbuild --version

Being written in Go, a WASM version and binaries for the main architectures are available.esbuild bets on native Go to take advantage of a maximum of parallelization solutions and a better memory management.

A lean API by design

Globally the API of esbuild is really simple, in 30 minutes you have read all the docs of possible settings. This is far from the 3-4 hours needed to read the whole documentation of a webpack for example. In spite of a configuration that might seem limited, I am still pleasantly surprised. I have the impression that we are really close to having the “right grammar” that we need to do bundling.

esbuild offers 3 consumption modes:

CLI

esbuild app.jsx --bundle --minify --sourcemap --target=chrome58,firefox57,safari11,edge16

GO

package main

import "github.com/evanw/esbuild/pkg/api"
import "os"

func main() {
  result := api.Build(api.BuildOptions{
    EntryPoints: []string{"app.jsx"},
    Bundle: true,
    MinifyWhitespace: true,
    MinifyIdentifiers: true,
    MinifySyntax: true,
    Engines: []api.Engine{
      {api.EngineChrome, "58"},
      {api.EngineFirefox, "57"},
      {api.EngineSafari, "11"},
      {api.EngineEdge, "16"},
    },
    Write: true,
  })

  if len(result.Errors) > 0 {
    os.Exit(1)
  }
}

JS

require('esbuild').buildSync({
  entryPoints: ['app.jsx'],
  bundle: true,
  minify: true,
  sourcemap: true,
  target: ['chrome58', 'firefox57', 'safari11', 'edge16'],
  outfile: 'out.js',
})

In my opinion, the CLI is still very practical for testing things, but in a more “industrial” use, we still prefer the JS or GO format.

Plugin mechanics

Evan Wallace the creator and core maintainer of esbuild makes no secret of the fact that he doesn’t want his tool to meet 100% of the needs that one can have in the web world. However, this doesn’t mean that we can’t use this tool in specific cases.

As we can see with other bundlers, esbuild offers the mechanics of plugins that allow you to do many things. To avoid maintaining all these specific needs, the creator relies on the community to create all the plugins you could want. And clearly, the community is there, I let you see this page that lists some plugins.

The most interesting features

I’m not going to list here the features that seem to me the heart of a Web bundler like code splitting, injection, minification. However, I was surprised by some features that are not found elsewhere.

An easy to understand architecture

Clearly, what makes the strength of esbuild compared to its competitors is its architecture which can be summarized simply. It is easy to understand that by combining the parallelization of the build steps and the reduction of the number of readings of the AST. I invite you to read more explanations in the doc.

Browser targets

By default esbuild allows you to define the target of your build. What level of javascript do you want to achieve?

Usually we use a suite of tools like @babel/preset-env and a browserlist to make sure we generate the JS compatible with our targeting. Babel is great, I use it every day, but piling up different tools for bundling is clearly not a good solution in my eyes. It adds a lot of complexity:

instead of learning to use a simple bundler tool, I have to learn a targeted transpilation tool on top of that
I have to maintain two dependencies
going through a third party librairy can reduce performances (this is a bit the bet of esbuild)

The server mode

esbuild is so fast that it can afford to expose you an HTTP server on a folder that contains the result of your compilation on each request. Other tools usually rely on a watch mode that watches for files that change to start a build.

The watch mode also exists with esbuild, but the serve mode seems to me even nicer because you just have to refresh your browser to have the latest version of your application locally.

require('esbuild')
  .serve(
    {
      servedir: 'www',
    },
    {
      entryPoints: ['src/app.js'],
      outdir: 'www/js',
      bundle: true,
    }
  )
  .then(server => {
    // Call "stop" on the web server when you're done
    server.stop()
  })

But then we stop everything and go on it?

Let’s save time, the answer for me is clearly no.

As the creator says in the FAQ of the doc in all honesty, the project is not to be considered as being in alpha. However, the tool itself does not yet have all the features that would make it a good replacement for the previous generation bundlers. I’m thinking in particular of the absence of native HMR, or of a perfectible splitting code.

However, one should not remain closed on this question. Clearly esbuild has very strong points that are missing in the current ecosystem. The community, still in its infancy, is rather active and the exchanges in the Issues and PR of the repo are very interesting.

What I really appreciate in this project are the parts taken: a focus on performance, an API that remains simple. Finally, for once a bundler doesn’t have 1000 dependencies and add 100Mb in my node_modules folder, it’s nice enough to note.

I’ll finish by saying that esbuild is not the only alternative that is offered to us in this new generation of bundler. I intend to do this kind of analysis on tools like Vite or Snowpack.

🎄 Advent of Code 2020 🎄: my solutions with JS

Antoine Caron — Fri, 04 Dec 2020 00:00:00 +0000

I decided this year to try the adventofcode2020 challenge in JS to see. I published the solutions I found in a public repository on Github. Not sure I will find solution for the upcoming challenges. 😅

Slashgear / advent-of-code

Personal implementation in JS of https://adventofcode.com/ challenge, don't hesitate to take a look if your are stuck

advent-of-code by Slashgear

In this repository you can find all the advent-of-code challenges that I passed. All solutions are in JS only (You can run them with Node). Feel free to look at my solutions for inspiration.

Solutions are sorted by year and by day in the packages folder.

packages/
  2019/
    day-1/
      ..
    day-2/
      ..
  2020/
     day-1/
      ..
    day-2/
      ..
    day-3/
      ..
  ..

View on GitHub

Project is a lerna monorepo structure showing solution by years and days.

packages/
  2019/
    day-1/
      ..
    day-2/
      ..
  2020/
     day-1/
      ..
    day-2/
      ..
    day-3/
      ..
    day-4/
      ..
  ..

For now, you can find solution for:

Github Action: How to parallelize tests dynamically by folder?

Antoine Caron — Tue, 01 Dec 2020 00:00:00 +0000

Once again, I gave myself a challenge to learn how to use Github Action. I tried to see if it was possible to dynamically launch a job for each folder in a directory.

We often want to run tests in parallel to reduce the time of continuous integration. Often tools like Jest or CucumberJS have parallelization features, but these often face the limits of the power available on the machines on which you want to run them.

tests/
  user/
    foo.feature
    bar.feature
  account/
    example.feature
  live/
    a.feature
    video.feature

I wondered if it wouldn’t be more efficient to run the tests in parallel directly with the CI tool so that only small runners would be needed. In the previous example, I would like to be able to run in parallel on a separate runner for each folder in test/.

A Solution with Github action

name: CI

on:
  push:
    branches: [main]

jobs:
  directories: # Job that list subdirectories
    runs-on: ubuntu-latest
    outputs:
      dir: ${{ steps.set-dirs.outputs.dir }} # generate output name dir by using inner step output
    steps:
      - uses: actions/checkout@v2
      - id: set-dirs # Give it an id to handle to get step outputs in the outputs key above
        run: echo "::set-output name=dir::$(ls -d */ | jq -R -s -c 'split("\n")[:-1]')"
        # Define step output named dir base on ls command transformed to JSON thanks to jq
  loop:
    runs-on: ubuntu-latest
    needs: [directories] # Depends on previous job
    strategy:
      matrix:
        dir: ${{fromJson(needs.directories.outputs.dir)}} # List matrix strategy from directories dynamically
    steps:
      - run: echo ${{matrix.dir}}

It’s up to you to adapt it to your needs!

Github Action: How to dynamically run step on each PR label ?

Antoine Caron — Sun, 29 Nov 2020 00:00:00 +0000

For some time now I’ve been playing with github actions to see if I can easily reproduce behaviors I apply in my current CI tool. I wanted to parallelize tasks on each label of a pull request. In this article, I will show you a possible way to do it!

In my work at Bedrock, we currently use Jenkins to manage our ongoing application integration and deployment. Beyond the various concerns of using groovy with pipeline syntax 😅, we have to admit that we manage to do some pretty complex workflows.

Deciding to see if it would be easy for me to reproduce the steps of these pipelines with github action, I decided to try its features. Among other things by publishing a Github Action to check the title of a PR according to a regexp.

Solution

First you need to create a workflow based on pull_request events and you can even specify the types of events.

name: Example workflow
on:
  pull_request:
    types: [opened, labeled, unlabeled, synchronize]

Now add a job that will set up a parallelization strategy based on the labels of the github event.

jobs:
  build:
    strategy:
      matrix:
        label: ${{github.event.pull_request.labels.*.name}}
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Run a one-line script
        run: echo Hello, world from "${{matrix.label}}"!

This will automatically run the steps in parallel on all the labels of each Pull Request. 🎉 However, strategy.matrix.label does not accept an empty array as a value.😢

The job should therefore be conditional so that it is only run if ${{github.event.pull_request.labels.*.name}} is not empty. There is an if parameter in the jobs that allows to do this, but unfortunately it is interpreted after checking the value of strategy.matrix.label.

The solution I found is to do this check in a previous job. This is an example of a workflow that dynamically launches actions on all the labels of each PR.

name: Example workflow

on:
  pull_request:
    types: [opened, labeled, unlabeled, synchronize]

jobs:
  checkLabels:
    runs-on: ubuntu-latest
    if: ${{github.event.pull_request.labels.*.name[0]}}
    steps:
      - run: echo Check labels is not empty
  build:
    needs: [checkLabels]
    strategy:
      matrix:
        label: ${{github.event.pull_request.labels.*.name}}
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Run a one-line script
        run: echo Hello, world from "${{matrix.label}}"!

It’s up to you to adapt it to your needs!

How to deploy on Github Pages with Travis CI?

Antoine Caron — Tue, 26 May 2020 00:00:00 +0000

Github Pages is a solution to host static websites directly in your Github repository by pushing site files to gh-pages branch.I use this feature to host this blog.It is free and very useful for your own site or open source documentation.Here is a tutorial of how to publish Gatsby website with travis.

Make sure to have a public github repository with your static site and a travis-ci.org account.

1. Add travis file to your project

Define basic travis-ci configuration to build and push you site on gh-pages page on each new commit on master.Add .travis-ci.yml file in the root directory of your git project.

Edit this file with your build configuration.

# This is the required part containing the build step for gatsby
language: node_js
node_js:
  - '12'
cache:
  yarn: true
  directories:
    - public
    - .cache
script:
  - yarn build # Generate static HTML files

# Here is the magic part
deploy:
  # Use Github pages deploy process
  provider: pages
  # Keep builded pages
  skip-cleanup: true
  # Directory where your generated files are located
  local_dir: public
  # Github security/auth token
  github-token: $GITHUB_TOKEN
  # Incremental commit to keep old build/files from previous deployments
  keep-history: true
  # Git branch on which it should deploy (master, gh-pages, foo...)
  target_branch: gh-pages
  on:
    # Which branch on commit/push will trigger deployment
    branch: master

Take a look at all deploy providers available with travis.

2. Generate Github personal token

Generate a Github token on this page with repository permission and keep it safe.

3. Configure your travis job

Toggle your github repository to travis listed project on this page.Then on your project travis page, go to More options > Settings.

In the Environment Variables section, add variable named GITHUB_TOKEN and use generated token from previous step.

4. Activate Github Pages feature for your repository

In your github repository settings page, be sure to activate github pages feature in Github Pages section.

Save it, and now it is done !You are good to go, your next commit pushed on master branch will trigger deployment of your site.

Hope this will help you configure travis deployment for your project.

If you liked it, I’ll try to do the same tutorial with Github Action.

Announcing use-reduced-motion

Antoine Caron — Mon, 04 May 2020 12:01:05 +0000

Everything always starts with an idea

Last week, as I was setting up the dark mode on my personal website, I got motivated to work on a new package for the React community.

" Well, somebody shared a hook to handle the prefers-color-scheme feature in browsers, that's super handy."

In just a few minutes, I was able to integrate this great feature without getting in my head.
That's when I thought of that awesome blog post about a new feature for accessibility.

At the time, while reading this article, I found out that some users may feel uncomfortable consulting web pages that "wiggle".
Indeed, elements of a page that move, zoom or change colors can greatly disturb the use and understanding of the content of our pages.
This may not be the case for you, but many users are in this situation.

In order to allow you to better understand the problem, I suggest a small experiment with a very short video.
Focus on the people dressed in white and try to count the number of passes.

Do you understand now?

The "superb onboarding animation" may be very beautiful, but it is very disturbing for some of your users.
They find themselves in the same situation as you do with this video.
The moving elements make it difficult to capture all the information you want to convey.

Fortunately, OS and browsers have become aware of accessibility issues and now provide tools.

It is therefore our responsibility today to integrate these solutions to avoid putting some of our users in a situation of handicap.

A media query allows you to stop your animations for users who wish to do so.

/*
  If the user has expressed their preference for
  reduced motion, then don't use animations on buttons.
*/
@media (prefers-reduced-motion: reduce) {
  button {
    animation: none;
  }
}

The support for this feature is even very correct.

Here is a video demo of how this feature works, taken from the article quoted above.

https://storage.googleapis.com/web-dev-assets/prefers-reduced-motion/prefers-reduced-motion.mp4

Where is the package and how to use it?

It's very nice this media query but in some cases, the animations I use on my sites are managed by JavaScript.
Fortunately, we have solutions to track the use of the media query in the browser.

To make them easier to use with React, I've integrated them into a Hook in the way of "use-dark-mode".

https://www.npmjs.com/package/use-reduced-motion

To install it in your project, nothing could be simpler:

npm install use-reduced-motion
# or
yarn add use-reduced-motion

Then you just have to use it in one of your components:

import React from 'react'
import { useReducedMotion } from 'use-reduced-motion'
import { AnimatedDiv } from '../somewhere'

export const MyExampleComponent = () => {
  const prefersReducedMotion = useReducedMotion()
  return <AnimatedDiv pause={prefersReducedMotion} />
}

I invite you to test here with your browser/OS, the following animation will stop automatically.

Feel free to share this article if you liked it, any contribution to the package is welcome.

Slashgear / use-reduced-motion

React Hook to detect option that reduce browser animation and motion for A11Y

Welcome to use-reduced-motion 👋

React hook to detect user reduced motion feature

Install

npm install use-reduced-motion
yarn add use-reduced-motion

Usage

Use directly in your component. Pass the boolean value to your JS animated component to stop it.

import React from "react";
import { useReducedMotion } from "use-reduced-motion";
import { AnimatedDiv } from "../somewhere";

export const MyExampleComponent = () => {
  const stopMotion = useReducedMotion();
  return <AnimatedDiv pause={stopMotion} />;
};

Author

👤 Antoine CARON antoine395.caron@gmail.com

Show your support

Give a ⭐️ if this project helped you!

📝 License

This README was generated with ❤️ by readme-md-generator

View on GitHub

Image credit unDraw