Forem: Dimitri Ivashchuk

How to debug GitHub actions. Real-world example

Dimitri Ivashchuk — Wed, 27 Mar 2024 06:51:52 +0000

GitHub Actions is the modern standard of CI/CD for both indie hackers & enterprises. With Actions one can achieve practically anything in a very easy and declarative way whether it be running tests on your code, deploying your production bundles, or even executing some code health checks or AI PR reviews!

GitHub Actions due to their declarative nature could be written by not only DevOps experts but also by frontend & backend engineers. One thing that you will immediately notice though as a person who does not have Ops experience is the difference in feedback loop. It’s much harder to see what has gone wrong on the CI machine compared to your local machine!

In this short guide, we will focus on the main techniques to try and answer the hot question - “How to debug GitHub actions?”

Setup fronted application with Vite & React

As an example, we will run a simple GitHub action on our front-end project. GitHub action will be the bare minimum and will execute some open-source visual regression tests on a single webpage.

Let’s create a simple Vite project:

npm create vite@latest

After selecting the following parameters our code shall be bootstrapped:

✔ Project name: … vite-project
✔ Select a framework: › React
✔ Select a variant: › TypeScript

We only need to install dependencies and run our application:

npm install && npm run dev

Now let’s add a basic testing setup so we have something to work with in our GitHub Action.

Setup open-source visual regression testing with lost-pixel

To get started we need to install the library & generate the basic config:

npm install lost-pixel && npx lost-pixel init-ts

Let’s modify the config so it can test plain web-pages:

import type { CustomProjectConfig } from "lost-pixel";

export const config: CustomProjectConfig = {
  pageShots: {
    baseUrl: "localhost:5173",
    pages: [
      {
        path: "/",
        name: "app",
      },
    ],
  },
  generateOnly: true,
  failOnDifference: true,
};

While running our frontend application let’s execute lost-pixel tests locally:

npx lost-pixel update

This command will crawl all of the listed pages(just one / page in our case) and create baseline snapshots that all the future snapshots will be compared to.

This is the expected output of this command:

Version: 3.16.0
---
✅ Found config file: /Users/d-ivashchuk/projects/lost-pixel-workspace/blogposts/vite-gh-actions/lostpixel.config.ts
🚀 Starting Lost Pixel in 'generateOnly' mode
Running lost-pixel in update mode. Baseline screenshots will be updated
📂 Creating shot folders
📸 Creating shots
Removing 0 files from .lostpixel/current/
Removing 0 files from .lostpixel/difference/

=== [Page Mode] http://localhost:5173 ===

Prepared 1 pages for screenshots on chromium
[1/1] Taking screenshot of 'app ' (app)
[1/1] Screenshot of 'app' taken and saved to '.lostpixel/current/app.png' in 2.669s (app)
Screenshots done!
Creating shots took 3.142 seconds
🔍 Checking differences
Comparing 1 screenshots using 'pixelmatch' as compare engine
[1/1] Comparing 'app' (app)
[1/1] Baseline image missing. Will be treated as addition. (app)
Comparison done!
Removing 0 files from .lostpixel/baseline/
👋 Exiting process with 0 found differences & 1 baselines to update

Now we have our frontend tests set up as well, let’s try to run this on GitHub Action.

Setup GitHub

To run everything that we were already able to run locally we need to create a GitHub Action declaration file in the root of the workspace .github/workflows/visual-testing.yml:

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Setup Node
        uses: actions/setup-node@v3
        with:
          node-version: 18.x
          cache: "npm"

      - name: Install dependencies
        run: npm ci

      - name: Build Vite app
        run: npm run build

      - name: Run Vite app
        run: npm run start &

      - name: Lost Pixel
        uses: lost-pixel/lost-pixel@v3.16.0

The above code pulls the code of the repository on the CI, installs dependencies, builds our application, serves it, and runs lost-pixel on top of it.

It is a literal match of what we have done locally, but now it is automated. If you push this code and your GitHub action runs you will see that lost-pixel step fails with the following error:

That’s not something that we expect and at this point, we have no clue what is going on. Let’s finally see the easiest way of debugging GitHub actions.

Debugging GitHub Actions

When it comes to debugging something(CI or locally, does not matter) two things matter the most:

having a plan to debug
having a tool to execute the plan

In our case we already have a good deal of understanding of what’s going on from the error that we saw on our CI(GitHub Actions):

[1/1] ❌ Page loading failed page.goto: net::ERR_CONNECTION_REFUSED at http://localhost:5173/

Our first guess is that something is wrong with our frontend application and we need to verify it. Locally you would just go to localhost:5173 and see it for yourself. On CI it is trickier and we need to be smart with the tooling, as there is no way for us to open something that runs on the GitHub Actions in our browser.

The go-to method of debugging GitHub Actions is tmate. With tmate we can connect to our running Action terminal and see what is going on there by executing some simple commands!

As the problem occurs between Vite run & lost-pixel run let’s strategically place a tmate step in our Action yml file:

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Setup Node
        uses: actions/setup-node@v3
        with:
          node-version: 18.x
          cache: "npm"

      - name: Install dependencies
        run: npm ci

      - name: Build Vite app
        run: npm run build

      - name: Run Vite app
        run: npx serve dist &

      - name: Setup tmate session
        uses: mxschmitt/action-tmate@v3

      - name: Lost Pixel
        uses: lost-pixel/lost-pixel@v3.16.0

Tmate will stop our GitHub Action from running further and expose an ssh command that you could use to connect from your terminal to the GitHub Action running container:

ssh GSvE5PGwZkU33j9jwCGr3vzd9@nyc1.tmate.io

It will look similar to the above command, after executing it you should be in and able to debug your Action properly.

You will also have an option to copy a URL of Web Shell that will allow you to connect directly from your browser.

If you run ls command you should see that Vite has been successful in building our application indeed as we have dist folder listed.

Let’s see if we can access anything served(we are looking for our frontend application) by executing the following command:

curl http://localhost:5173

This should return correct html for the page meaning that our app is running just fine, but lost-pixel has trouble accessing it.

This gives enough clues to investigate the docs of the GitHub actions and lost-pixel to see that we need to change the localhost to the IP address of the local network running on GitHub actions. There is also an answer on Stack Overflow that can lead us in the right direction! Lost Pixel action running in docker simple couldn’t reach localhost it needed a concrete IP address in that case.

Let’s fix it in our lost-pixel.config.ts

import type { CustomProjectConfig } from "lost-pixel";

export const config: CustomProjectConfig = {
  pageShots: {
    baseUrl: "http://172.17.0.1:5173",
    pages: [
      {
        path: "/",
        name: "app",
      },
    ],
  },
  generateOnly: true,
  failOnDifference: true,
};s

This resulted in the successful run of our CI, meaning we fixed the bug on the CI with the help of tmate debugging.

Afterthoughts

GitHub Actions is a powerful tool that allows you to write your CI/CD pipelines in declarative plain language. When it comes to debugging - tmate is one of the most popular options due to the simplicity & enjoyable developer experience it brings.

When it comes to the alternatives to tmate, there is another great debugging tool that you could check out. It is called act and it allows you to run GitHub Actions code on your local machine making debugging even easier. It has its own limitations and some learning curve but overall it is another tool you should use if you can’t fix the CI bugs by connecting directly into the running action with the tmate.

Simplifying Next.js visual testing

Dimitri Ivashchuk — Mon, 22 Jan 2024 10:42:18 +0000

Status quo

I am currently building lost-pixel, an open-source visual regression testing tool and so far we have tried to make the developer experience click well by natively integrating with Storybook, Ladle & Histoire, - the most prominent component development tools in the industry.

We've recognised right from the start that apart from testing the components, we, as the frontend engineers are also interested in making sure that those components when used to form a cohesive picture on our websites. This is also a huge shame when that cohesiveness breaks and we have a poor UI experience for the users :D Hence, we've also added pages mode to lost-pixel which allows users to specify an array of pages to test in a simple but extensible format for any frontend framework(in this example we will be using Next.js)

 pages: [{ path: '/app', name: 'app' }, {path:'/pricing', name: 'pricing'}],

I started using it myself extensively on our landing page, inside of our SaaS platform, and everywhere where visual tests were making sense. But it always felt like something was missing, as there was still a bunch of manual work for engineers to do to create those lists of pages.

We came up with another abstraction over it! Allowing users to come up with their way of informing us about what they want to visually test in their application:

pagesJsonUrl: './lost-pixel-pages.json',

With pagesJsonUrl users got the option to generate the list of pages in the format that we expect(thank you typescript for making it safe) and just point us to the file or URL where this JSON is available for us to consume.

Yet, while being super useful in projects like Gatsby where there is a way of nicely getting the list of the website pages from their internal API it was still too big of a hustle for projects where outputting the pages was not so straightforward.

There came my good friend, Oleg, who is also the founder of Webstudio, an open-source Webflow alternative and he suggested something that flipped the game for us by adding a simple feature that took less than an hour to write!

Solution

As with everything web platform-related, given the plethora of options out there it is very easy to overlook some things that make sense. I felt stupid that I overlooked it and in retrospect, this feature was crucial to add at the same time we introduced the pagesJsonUrl

Enter npx lost-pixel page-sitemap-gen 🎉

Most modern websites have sitemaps that are dynamically generated which is an easy fit for next.js websites and those sitemaps usually have the list of the most important pages that you care about on your website.

By reading the sitemap, parsing it & creating the list of pages to be consumed by the pagesJsonUrl I finally closed the circle! Let me show you an example of a simple GitHub action:

      ...

      - name: Build next
        run: pnpm run build 
      - name: Start next
        run: pnpm run start &

      - name: Generate Sitemap
        run: npx lost-pixel page-sitemap-gen http://172.17.0.1:3000/sitemap.xml "./lost-pixel-pages.json"

      - name: Lost Pixel
        uses: lost-pixel/lost-pixel@v3.10.0
        env:
          LOST_PIXEL_API_KEY: ${{ secrets.LOST_PIXEL_API_KEY }}

export const config: CustomProjectConfig = {
    pageShots: {
        pagesJsonUrl: './lost-pixel-pages.json',
        baseUrl: 'http://172.17.0.1:3000',
    },
    lostPixelProjectId: 'clb1xlaeo1299101qqkt94cn43',
    apiKey: process.env.LOST_PIXEL_API_KEY,
};

What happens here is simple, but yet very powerful for the maintainability & simplicity of your visual tests.

We build the Next.js app(the sitemap is dynamically generated so we always have a fresh version of it)
We run npx lost-pixel page-sitemap-gen to take the fresh version of the sitemap and generate the array of pages for lost-pixel to snapshot
We run lost-pixel over that array

Suddenly instead of manually adding the page to the array every time we add something new it is fully automated. The best thing about it all - it can be recreated for any website that has a sitemap, either dynamic or static 🚀
Here I've added 50 pages to our visual testing suite for our landing page in less than 5 minutes. Lost Pixel Platform can better showcase it but you can also use it through our open-source lost-pixel and easily achieve the same functionality without spending a dime!

Ending thoughts

I am pretty excited about this little feature as it unlocks a new level of developer experience improvement within lost-pixel. I would be excited to hear your thoughts and see if I have missed some other awesome use cases with this feature 😀

Visual regression testing of shadcn UI

Dimitri Ivashchuk — Mon, 30 Oct 2023 12:25:24 +0000

Introduction

This is a practical guide to Tailwind, shadcn-ui Visual Regression Testing with Storybook. It will walk you through the basics of integrating Tailwind & shadcn-ui, managing components with Storybook, and ensuring visual consistency using Lost Pixel's open-source visual regression testing. Perfect for developers at any level, this guide offers a straightforward approach to enhancing your Next.js or any other projects using different frameworks. Let's dive in!

Getting started

Clone the next.js template from shadcn-ui-next-js template and download it on your machine. We start with npm install to get all of the dependencies running. npm run dev starts the next project and sets us up for the next steps.

What are we going to do:

Add couple of components using shadcn-ui
Add storybook to manage the components we created
Add lost-pixel open-source visual regression testing flow to make sure our components are always looking as expected

Extending our component base

We will create the patterns folder where we will put our new components that we will want to test. Don't forget to addon all of the primitives like label,switch and other trough amazing shadcn cli

Our first component will be <Cookies/>

"use client"

import { Button } from "../ui/button"
import {
  Card,
  CardContent,
  CardDescription,
  CardFooter,
  CardHeader,
  CardTitle,
} from "../ui/card"
import { Label } from "../ui/label"
import { Switch } from "../ui/switch"

export function DemoCookieSettings() {
  return (
    <Card>
      <CardHeader>
        <CardTitle>Cookie Settings</CardTitle>
        <CardDescription>Manage your cookie settings here.</CardDescription>
      </CardHeader>
      <CardContent className="grid gap-6">
        <div className="flex items-center justify-between space-x-2">
          <Label htmlFor="necessary" className="flex flex-col space-y-1">
            <span>Strictly Necessary</span>
            <span className="font-normal leading-snug text-muted-foreground">
              These cookies are essential in order to use the website and use
              its features.
            </span>
          </Label>
          <Switch id="necessary" defaultChecked />
        </div>
        <div className="flex items-center justify-between space-x-2">
          <Label htmlFor="functional" className="flex flex-col space-y-1">
            <span>Functional Cookies</span>
            <span className="font-normal leading-snug text-muted-foreground">
              These cookies allow the website to provide personalized
              functionality.
            </span>
          </Label>
          <Switch id="functional" />
        </div>
        <div className="flex items-center justify-between space-x-2">
          <Label htmlFor="performance" className="flex flex-col space-y-1">
            <span>Performance Cookies</span>
            <span className="font-normal leading-snug text-muted-foreground">
              These cookies help to improve the performance of the website.
            </span>
          </Label>
          <Switch id="performance" />
        </div>
      </CardContent>
      <CardFooter>
        <Button variant="outline" className="w-full">
          Save preferences
        </Button>
      </CardFooter>
    </Card>
  )
}

Second component will be <CreateAccount/>

"use client"

import { Icons } from "@/components/icons"

import { Button } from "../ui/button"
import {
  Card,
  CardContent,
  CardDescription,
  CardFooter,
  CardHeader,
  CardTitle,
} from "../ui/card"
import { Input } from "../ui/input"
import { Label } from "../ui/label"

export function DemoCreateAccount() {
  return (
    <Card>
      <CardHeader className="space-y-1">
        <CardTitle className="text-2xl">Create an account</CardTitle>
        <CardDescription>
          Enter your email below to create your account
        </CardDescription>
      </CardHeader>
      <CardContent className="grid gap-4">
        <div className="grid grid-cols-2 gap-6">
          <Button variant="outline">
            <Icons.gitHub className="mr-2 h-4 w-4" />
            Github
          </Button>
          <Button variant="outline">
            <Icons.twitter className="mr-2 h-4 w-4" />
            Twitter
          </Button>
        </div>
        <div className="relative">
          <div className="absolute inset-0 flex items-center">
            <span className="w-full border-t" />
          </div>
          <div className="relative flex justify-center text-xs uppercase">
            <span className="bg-background px-2 text-muted-foreground">
              Or continue with
            </span>
          </div>
        </div>
        <div className="grid gap-2">
          <Label htmlFor="email">Email</Label>
          <Input id="email" type="email" placeholder="m@example.com" />
        </div>
        <div className="grid gap-2">
          <Label htmlFor="password">Password</Label>
          <Input id="password" type="password" />
        </div>
      </CardContent>
      <CardFooter>
        <Button className="w-full">Create account</Button>
      </CardFooter>
    </Card>
  )
}

Adding those to the page should look similar to this:

Now let's add Storybook to our project and add our patterns & primitives to our storybook so we could develop them in isolation, document & most importantly test them.

Adding storybook

Storybook is indespensable tool for modern frontend development as it allows to document the components, develop them in isolation & finally easily test them for visual regression testing.

Let's run this command to automatically enable the storybook in our project npx sb init --builder webpack5. Next we need to make sure that storybook works well with the tailwind which is used under the hood of shadcn.ui.

npx storybook@latest add @storybook/addon-styling-webpack

We will override our font in globals.css so that our fonts in both storybook and application are the same, so this tutorial is simpler to follow.

@layer base {
  * {
    @apply border-border;
  }
  body {
    @apply bg-background text-foreground;
    font-feature-settings: "rlig" 1, "calt" 1;
    font-family: "Inter", sans-serif;
  }
}

Let's add 3 stories of the tailwind component that we have. One primitive component - <Button/> and two pattern components - <Cookies/> & <CreateAccount>.

In the stories folder that we have automatically created let's delete all of the generated files & add our own.


import type { Meta, StoryObj } from "@storybook/react"

import { DemoCreateAccount } from "../components/patterns/create-account"

const meta: Meta<typeof DemoCreateAccount> = {
  component: DemoCreateAccount,
}

export default meta
type Story = StoryObj<typeof DemoCreateAccount>

/*
 *👇 Render functions are a framework specific feature to allow you control on how the component renders.
 * See https://storybook.js.org/docs/react/api/csf
 * to learn how to use render functions.
 */
export const Primary: Story = {
  render: () => <DemoCreateAccount />,
}


import type { Meta, StoryObj } from "@storybook/react"

import { DemoCookieSettings } from "../components/patterns/cookies"

const meta: Meta<typeof DemoCookieSettings> = {
  component: DemoCookieSettings,
}

export default meta
type Story = StoryObj<typeof DemoCookieSettings>

/*
 *👇 Render functions are a framework specific feature to allow you control on how the component renders.
 * See https://storybook.js.org/docs/react/api/csf
 * to learn how to use render functions.
 */
export const Primary: Story = {
  render: () => <DemoCookieSettings />,
}


import type { Meta, StoryObj } from "@storybook/react"

import { Button } from "../components/ui/button"

const meta: Meta<typeof Button> = {
  component: Button,
}

export default meta
type Story = StoryObj<typeof Button>

/*
 *👇 Render functions are a framework specific feature to allow you control on how the component renders.
 * See https://storybook.js.org/docs/react/api/csf
 * to learn how to use render functions.
 */
export const Primary: Story = {
  render: () => <Button>Primary button</Button>,
}

Note that our primitive component is used in both of the pattern components, this is important for the visual regression testing, as with growth of your component library the biggest risk lies in changing the primitives & affecting your patterns where those primitives are used.

Our end result should look like this:

Let's add a simple visual regression testing for our tailwind & shadcn-ui.

Adding Lost Pixel

Let's add Lost Pixel package to our repo:

npm add lost-pixel && npx lost-pixel init-ts

Our lostpixel.config.ts should me slightly modified to ensure we target storybook at the correct place.

import type { CustomProjectConfig } from "lost-pixel"

export const config: CustomProjectConfig = {
  storybookShots: {
    storybookUrl: "./storybook-static",
  },
  generateOnly: true,
  failOnDifference: true
}

Believe it or not - this is the basic working visual testing setup with Lost Pixel, the only thing that we need to do is to build our storybook & run lost-pixel!

npm run build-storybook && npx lost-pixel update

This will build our storybook & then run lost-pixel tests over it & place the current component visual snapshots into .lost-pixel folder.

Now let's try to change the <Button/> primitive and rerun our tests, let's change default background from primary to destructive

default: "bg-destructive text-primary-foreground hover:bg-primary/90"

npm run build-storybook && npx lost-pixel

Comparing 3 screenshots using 'pixelmatch' as compare engine

[1/3] Comparing 'button--primary' (button--primary)
[2/3] Comparing 'cookies--primary' (cookies--primary)
[3/3] Comparing 'createaccount--primary' (create-account--primary)

[1/3] Difference of 4726 pixels (0.51%) found. Difference image saved to: .lostpixel/difference/button--primary.png (button--primary)
[2/3] No difference found. (cookies--primary)
[3/3] Difference of 47364 pixels (5.14%) found. Difference image saved to: .lostpixel/difference/create-account--primary.png (create-account--primary)

Comparison done!

We can see that in both of the pattern components that were using the <Button/> with primary variant we have the spotted regressions. All the relevant to the run images are saved in .lost-pixel folder.

This is a very basic setup that allows you to run visual regression tests locally. Let's add a simple GitHub actions integration to make sure we can catch the visual regressions of our tailwind components continuously & effortlessly.

Running visual regression testing on GitHub Actions

To start running Lost Pixel on CI we need to create an action workflow file. Let's create .github folder, and place workflows folder inside. Inside workflows let's create vrt.yml file with the following contents:

on: [push]
jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Setup Node
        uses: actions/setup-node@v3
        with:
          node-version: 18.x
          cache: "npm"

      - name: Install dependencies
        run: npm install

      - name: Build Storybook
        run: npm run build-storybook

      - name: Lost Pixel
        uses: lost-pixel/lost-pixel@v3.7.4

This code checks out repository, installs dependencies, builds storybook & finally runs Lost Pixel. If there will be visual regressions the check will fail.

Whenever you make a change to your stories that should update the baselines - just run npx lost-pixel docker update. This will run lost-pixel in docker to make sure we eliminate inconsistencies in operating systems between your machine & CI, it will also update all of the visual snapshots in the baseline folder of .lost-pixel making your next CI run successful.

That's the result we are aiming for here for when you are pushing some code that changes your primitive components and influences other patterns down the dependency graph:

If you want to extend your storybook visual tests & add your Next.js pages - it's super trivial with Lost Pixel as well, you can follow this tutorial that focuses on holistic tests

Holistic Visual Regression Testing

Dimitri Ivashchuk — Thu, 09 Mar 2023 13:48:08 +0000

Introduction to Holistic Visual Testing 🖼️

As front-end development continues to evolve, it's more important than ever to have a robust testing strategy in place to ensure that your applications look and function as expected. One of the most effective ways to do this is through visual regression testing, which allows you to catch and fix visual bugs before they reach production.

Holistic visual regression testing is an approach that aims to test the entire application, including both layout and interactions, as a whole.

In this example, we will set up a complex visual testing suite for our Next.js application. Here is the list of tools we will use:

Storybook
Playwright
GitHub actions
Lost Pixel Platform

As the final result, we want to make sure that the following parts of our applications are looking the way they are intended to look:

Individual components - visual unit tests
Pages - visual integration tests
Results of playwright/cypress tests - visual e2e tests

Lost Pixel makes it easy to compose all of this in a single place to make the management of your visual testing a breeze. Here is the final version of the code for the whole integration:

Here is the final version of the code on GitHub

Setting up Next application 🔼

We will clone the app-playground repo from Next.js, it has rudimentary parts of the actual application, and that's precisely what we need.

git clone https://github.com/vercel/app-playground.git

pnpm install

pnpm dev

Visual unit tests 🖼️

We will use Storybook in combination with Lost Pixel to make sure our app components are looking the way we expect them to look.

Let's install Storybook:

npx storybook@next init && pnpm install postcss-loader style-loader @storybook/addon-postcss

Inside .storybook/main.js we need to make Storybook & Tailwind.css play well with each other :D

const path = require('path');
module.exports = {
  stories: ['../stories/**/*.mdx', '../stories/**/*.stories.@(js|jsx|ts|tsx)'],
  addons: [
    '@storybook/addon-links',
    '@storybook/addon-essentials',
    '@storybook/addon-interactions',
    {
      name: '@storybook/addon-postcss',
      options: {
        cssLoaderOptions: {
          // When you have splitted your css over multiple files
          // and use @import('./other-styles.css')
          importLoaders: 1,
        },
        postcssLoaderOptions: {
          // When using postCSS 8
          implementation: require('postcss'),
        },
      },
    },
  ],
  framework: {
    name: '@storybook/nextjs',
    options: {},
  },
  docs: {
    autodocs: 'tag',
  },
};

Inside .storybook/preview.js we need to import tailwind styles on top of the file:

import '../styles/globals.css';

export const parameters = {
  actions: { argTypesRegex: '^on[A-Z].*' },
  controls: {
    matchers: {
      color: /(background|color)$/i,
      date: /Date$/,
    },
  },
};

Inside stories/Button.stories.tsx let's write our first story that we will use in tests:

import type { Meta, StoryObj } from '@storybook/react';
import React from 'react';

import Button from 'ui/Button';

// More on how to set up stories at: https://storybook.js.org/docs/7.0/react/writing-stories/introduction
const meta: Meta<typeof Button> = {
  title: 'Example/Button',
  component: Button,
};

export default meta;
type Story = StoryObj<typeof Button>;

export const Default: Story = {
  render: () => <Button kind="default">Default</Button>,
};

export const Error: Story = {
  render: () => <Button kind="error">Error</Button>,
};

Now we run our storybook to see our components displayed in an isolated manner:

npm run storybook

Our stories will serve as the basis for the unit visual tests. Let's integrate this first step with Lost Pixel Platform to see how it will work there.

Lost Pixel Platform ⚙️

You can use the quickstart documentation from lost-pixel.com to set up the platform to work with your repositories via installing the official Lost Pixel GitHub app. In this guide, we will focus on setting up the right configuration, assuming you have done the platform setup already!

You can use Lost Pixel Onboarding to easily set up the project by copying & pasting the respective files.

In our lostpixel.config.ts at the root of our project, let's define the tests for Storybook

import { CustomProjectConfig } from 'lost-pixel';

export const config: CustomProjectConfig = {
  storybookShots: {
    storybookUrl: './storybook-static',
  },

  lostPixelProjectId: 'clde9r3rh00v3m50vlq8y0k78',
  apiKey: process.env.LOST_PIXEL_API_KEY,
};

In our .github/workflows/vrt.yml at the root of our project let's define our CI/CD job:

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    name: Lost Pixel

    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Setup Node
        uses: actions/setup-node@v3
        with:
          node-version: 16.x

      - name: Install dependencies
        run: npm install --legacy-peer-deps

      - name: Build Storybook
        run: npm run build-storybook

      - name: Lost Pixel
        uses: lost-pixel/lost-pixel@v3.0.4
        env:
          LOST_PIXEL_API_KEY: ${{ secrets.LOST_PIXEL_API_KEY }}

With this simple setup, we are already covering the visual unit tests 🙌🏼

Let's make it really holistic now!

Visual integration tests 🖼️ ↔️ 🖼️

Sometimes we care not only about how our components will be used in isolation but also about how they will look on the page when they are combined with a plethora of other components & styles. Let's explore how we can easily expand our initial setup to monitor our pages continuously!

Let's extend our lostpixel.config.ts to include our page shots.

import { CustomProjectConfig } from 'lost-pixel';

export const config: CustomProjectConfig = {
  storybookShots: {
    storybookUrl: './storybook-static',
  },
  pageShots: {
    pages: [
      { path: '/layouts/books/fiction', name: 'fiction-books' },
      { path: '/layouts/books/biography', name: 'biography-pages' },
    ],
    baseUrl: 'http://172.17.0.1:3000',
  },
  lostPixelProjectId: 'clde9r3rh00v3m50vlq8y0k78',
  apiKey: process.env.LOST_PIXEL_API_KEY,
};

We also need to extend .github/workflows/vrt.yml to build & run our Next.js app so we can access pages in Lost Pixel:

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    name: Lost Pixel

    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Setup Node
        uses: actions/setup-node@v3
        with:
          node-version: 16.x

      - name: Install dependencies
        run: npm install --legacy-peer-deps

      - name: Build Storybook
        run: npm run build-storybook

      - name: Build Next app
        run: npm run build

      - name: Run Next app
        run: npm run start &

      - name: Lost Pixel
        uses: lost-pixel/lost-pixel@v3.0.4
        env:
          LOST_PIXEL_API_KEY: ${{ secrets.LOST_PIXEL_API_KEY }}

Now we are talking 🚀 Our visual tests are covering not only unit parts with individual components but also the integration part where we are making the screenshots for the whole pages.

Let's make it an ultimate setup with a magic flavour of the feature of Lost Pixel called custom shots 🪄✨✨✨

Visual E2E tests 🖼️ ⚙️ 🖼️ ⚙️ 🖼️

What if I told you that on any given CI run, you can create the screenshots using your logic & they will be available immediately on Lost Pixel Platform for the visual regression testing? Enter custom shots 📦

We will be using Playwright to interact with the page & make sure that after some set of interactions, we are always looking at the same UI.

npm install --save-dev @playwright/test

Let's also extend our package.json with test:e2e command:

...
"scripts": {
  "dev": "next dev",
  "build": "next build",
  "start": "next start",
  "test:e2e": "playwright test",
}

Now we will create our E2E test that will not do much, but will give you a good understanding of how you could integrate Lost Pixel into any test like this!

In e2e/example.spec.ts :

import { test } from '@playwright/test';

test('lost-pixel e2e', async ({ page }) => {
  // Perform some action on the page, like clicking the button.
  // 📒 http://172.17.0.1:3000 is already fit for GitHub action runtime
  await page.goto('http://172.17.0.1:3000/context');
  await page.click('data-test-id=context-click-counter');
  await page.click('data-test-id=context-click-counter');
  // Make a shot for Lost Pixel to test
  // 📒 Path is arbitrary, but I advise you to store all of the custom shots in a single folder, e.g. lost-pixel. 
  await page.screenshot({ path: 'lost-pixel/a.png', fullPage: true });
});

Our ContextClickCounter component gets a new data-test-id, so the code above works:

      <button
        // Here is the important part
        data-test-id="context-click-counter"
        onClick={() => setCount(count + 1)}
        className="rounded-lg bg-gray-700 px-3 py-1 text-sm font-medium tabular-nums text-gray-100 hover:bg-gray-500 hover:text-white"
      >
        {count} Clicks
      </button>

Now we are ready to integrate our E2E with Lost Pixel! Let's extend our lostpixel.config.ts to include our custom shots.

import { CustomProjectConfig } from 'lost-pixel';

export const config: CustomProjectConfig = {
  storybookShots: {
    storybookUrl: './storybook-static',
  },
  pageShots: {
    pages: [
      { path: '/layouts/books/fiction', name: 'fiction-books' },
      { path: '/layouts/books/biography', name: 'biography-pages' },
    ],
    baseUrl: 'http://172.17.0.1:3000',
  },
  customShots: {
    // 📒 currentShotsPath should be matching the one above from playwright test
    currentShotsPath: './lost-pixel',
  },
  lostPixelProjectId: 'clde9r3rh00v3m50vlq8y0k78',
  apiKey: process.env.LOST_PIXEL_API_KEY,
};

And finally, we extend our GitHub Action declaration file by running Playwright tests:

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    name: Lost Pixel

    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Setup Node
        uses: actions/setup-node@v3
        with:
          node-version: 16.x

      - name: Install dependencies
        run: npm install --legacy-peer-deps

      - name: Build Storybook
        run: npm run build-storybook

      - name: Build Next app
        run: npm run build

      - name: Run Next app
        run: npm run start &

      - name: Playwright tests
        run: npx playwright install --with-deps && npm run test:e2e

      - name: Lost Pixel
        uses: lost-pixel/lost-pixel@v3.0.4
        env:
          LOST_PIXEL_API_KEY: ${{ secrets.LOST_PIXEL_API_KEY }}

After Lost Pixel Platform runs this time, you should see something similar to this:

After interacting with our page & clicking the button two times, we made a screenshot of a page to test visually. Sometimes it's very useful because apart from asserting the presence or absence of some elements on the page that E2E tests traditionally rely on, the layout could be broken after a particular interaction and here visual regression tests will catch this for us.

Summary 🔼

We have built a holistic visual testing framework over an existing Next.js application using the tools suitable for every task: Storybook for visual unit tests, Pages for visual integration tests & Playwright for visual E2E tests. We have use GitHub Actions to make sure our setup runs continuously on every push. Finally, we have used the Lost Pixel Platform to manage our visual tests in one central place & notify us whenever visual regression is found.

If you liked the read and enjoyed setting up your visual tests, you might as well like our Visual Regression Testing Discord. There you can find help with any issues, chat about your setup & discuss the good practices for visual tests.

Next.js visual regression testing made easy

Dimitri Ivashchuk — Thu, 08 Sep 2022 09:46:08 +0000

Why visual regression testing

Your app code changes quite frequently - so does the UI. The problem with this flow is that sometimes unintentional UI changes result from incorrect code. We often forget to check the deep corners of the application that users might see for every potential regression; moreover, we don't have time for that, right 😅 Ultimately as with other testing types the goal is to ship with more confidence and o*wn the changes you make to the code*!

| Luckily, we can quickly establish automation that can help us with that

In this tutorial we will learn how to:

setup a new next.js app
use lost-pixel to enable CI visual regression testing

At this time, you might wonder what visual regression tests even are.

On top are two snapshots taken before/after a code change, below is how the engine compares those two to identify the visual regression.

Without further ado, let's jump into the code, and don't worry, as we will delve into some low-level dev-ops you don't need any specific knowledge, we will figure everything on the fly.

For the final code you could check out this repo

Next setup

This tutorial covers the most basic Next.js app, so you can follow official docs for the simple instructions for setting it up. First, let's add the *app page * and move the content from the index there so we have a clear path we will test!

Adding Lost Pixel

So far, our setup has the frontend application & the tooling to observe the frontend components/pages. However, to have complete control over the changes to the code and their repercussions on the UI, we use an open-source visual regression testing tool called Lost Pixel.

To set up the first tests, you need to add the lost-pixel.config.js or lost-pixel.config.ts at the root of your project by running one of the below commands:

npx lost-pixel init-js
or
npx lost-pixel init-ts

Don't forget to include lostpixel.config.ts file into your tsconfig when using typescript version of config so it's picked up correctly. npm install lost-pixel -D is as well required for the types.

This will generate the config that powers Lost Pixel. As we are setting up the open-source version which persists the snapshots of your components in your repo, you need to add the generateOnly key into your config - this will ensure that lost-pixel will not try to upload the images to the cloud, failOnDifference flag will exit the GitHub action with the error. Here is how your config should look like:

import { CustomProjectConfig } from 'lost-pixel';

export const config: CustomProjectConfig = {
  pageShots: {
    pages: [
      { path: '/app', name: 'app', id: 'app' },
    ],
    pageUrl: "http://172.17.0.1:3000",
  },
  generateOnly: true,
  failOnDifference: true
};

Note the 172.17.0.1 IP address, which is specific to the GitHub actions run.

GitHub actions setup

With our current setup, it's pretty straightforward to enable the visual-regression testing on the CI. We will use GitHub actions that have native integration with lost-pixel for this example. Still, lost-pixel also being a docker container, it's possible to integrate it into the CI platform of your choice.

Let's get to business!

In .github/workflows/test.yml

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Setup Node
        uses: actions/setup-node@v3
        with:
          node-version: 16.x
          cache: 'npm'

      - name: Build Next app
        run: npm run build

      - name: Run Next app
        run: npm run start &

      - name: Lost Pixel
        uses: lost-pixel/lost-pixel@v2.15.0

Let's see what this workflow file does in a little more detail:

on: [push] // we want to execute this action on every push to our repository

jobs:
  build: // we set up one build job 
    runs-on: ubuntu-latest // it runs on ubuntu

    steps: // it will execute the steps outlined below
      - name: Checkout
        uses: actions/checkout@v2 // get the code of our repo

      - name: Setup Node
        uses: actions/setup-node@v2
        with:
          node-version: 16.x
          cache: 'npm' // get node to work in our process 

      - name: Install dependencies
        run: npm ci // install dependencies defined in package.json

      - name: Build Next app
        run: npm run build // build next.js application

      - name: Run Next app
        run: npm run start & // run next.js application in the background mode

      - name: Lost Pixel
        uses: lost-pixel/lost-pixel@v2.15.0 // run lost-pixel visual regression tests

As you commit this code, we will see that GitHub has already picked the job and is executing the flow!
We don't yet have anything in .lostpixel baselines folder which means the action will fail and notify us that there are missing baselines. You can learn more about baselines and flow to update them here. So let's add another workflow to make sure we always have an easy way to update baselines!

In .github/workflows/test.yml


on: push

jobs:
  lost-pixel-update:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Setup Node
        uses: actions/setup-node@v3
        with:
          node-version: "16"

      - name: Install dependencies
        run: npm ci --legacy-peer-deps

      - name: Build Next app
        run: npm run build

      - name: Run Next app
        run: npm run start &

      - name: Lost Pixel
        uses: lost-pixel/lost-pixel@v2.15.0
        env:
          LOST_PIXEL_MODE: update

      - name: Create Pull Request
        uses: peter-evans/create-pull-request@v4
        if: ${{ failure() && steps.lp.conclusion == 'failure' }}
        with:
          token: ${{ secrets.GH_TOKEN }}
          commit-message: update lost-pixel baselines
          delete-branch: true
          branch: "lost-pixel-update/${{ github.ref_name }}"
          title: "Lost Pixel update - ${{ github.ref_name }}"
          body: Automated baseline update PR created by Lost Pixel

You must have already guessed that this workflow file is not much different from the previous one with one little catch - it will automatically create a new PR. Don't forget to add the update mode env var so we have our baselines updated!

    - name: Lost Pixel
        uses: lost-pixel/lost-pixel@v2.11.0
        env:
          LOST_PIXEL_MODE: update // <--- this one here is really important for updater action to run

For this exercise we need a personal access token from github. To ensure that the action has access to the GitHub token which you, by the way, should never expose to the public - we will place it in the secrets of the repo.

Let's run the updater action:

If there are no baselines it will prompt a PR creating the Lost Pixel baselines, if there are existing baselines and there were some differences found it will as well prompt a new PR. In this case we see that there is one open PR with the new baseline. Let's verify it looks good and merge the pull request!

After PR merge we can see that Lost Pixel has ran the tests and all of them are passing 🍏🍏🍏

Amazing job, you now have visual regression testing running on your CI/CD pipeline 🚀

Closing thoughts

Whenever the Lost Pixel fails on CI, you need to decide if the change was intentional or not, which you could read up here. If it has been a deliberate change - you need to run update your baselines on CI. Another strategy could be generating the artefacts from the failed run and putting those manually into .lostpixel/baselines folder!

The implementation is finished, you could try making some changes to your /app page, like changing fonts, layout and color - as we have visual regression tests setup, we will be able to catch all the changes and ship with more confidence.

For the final code you could check out this repo as previously mentioned!

Build telegram bot with node

Dimitri Ivashchuk — Wed, 04 Dec 2019 12:51:06 +0000

Originally published on divdev.io

Table of tutorials contents

Setting up the project
Working with messages - WIP
Adding commands to the bot, working with the special messages and keyboards - WIP
Persisting state between sessions - WIP
Using telegraf middleware - WIP

Motivation

As Telegram in justified way replaced all of my other messengers for personal and professional use I’ve started to look for ways of automating things with it. The main reason for that is that bot features of telegram are already enough for most of the user facing apps that contain input/output flows. In simple words you get your UI for free and only need to care about business logic.

Choice of technologies

Coming from JavaScript background I didn’t think much about the choice of the server framework for the telegram application - simple and clean express.js. It was harder to choose between two of the most popular node frameworks for building bots for telegram. The main players there are Telegraf.js and telegram-node-api. As I’ve scanned the docs and tutorials I decided to go for Telegraf, so you can expect that the whole series will be written with it. Be sure to check both and form your own opinion as they offer quite similar functionalities!

What you will learn?

As I was building my small army of minio... bots, I’ve faced various problems and found different solutions to them. I want to provide a broad understanding of the tools that bots can offer to you, meaning that after finishing the series you will be ready to solve your problems and automate tasks with telegram bots.

What this series is not about?

I will not go deep into details how telegram works, or how Telegraf successfully abstracts the most complicated logic providing the wrappers over most useful telegram APIs. On the contrary, I will focus on the real world problems that I at some point solved and personally think they are the strongest blockers when it comes to writing the bot, and figuring our not fantastically (still very decently though :) ) written telegram docs.

Talk is cheap, let's get started

In this first part of the series I want to show you how easy the whole setup is and how you can get the things rolling with less than 50 lines of code!

Telegraf framework requires a server to operate on top of so the first and obvious thing for us to do is setting up express server. Don't forget to install the needed packages with npm install express telegraf or yarn add express telegraf.

const express = require('express')

const app = express()

const PORT = '8888'

app.listen(PORT, async () => {
  console.log(`🚀 server is running`)
  require('./bot')
})

As telegram needs a way of authorizing bot actions every bot must be created via @botfater - telegram owned bot that acts as a setup wizard for all the bots that you will be creating.

The whole process is fast and frictionless. Go to the bot and run /newbot either from message input field or from the command list Dropdown.

Choose the name of your marvelous creation and you will get a congrats message from telegram containing the first thing we need for making our bot work - Access token.

Now we are ready to go trough setup of telegraph. Provide your access token to the configuration object of telegraf (be sure to keep it secret and concealed in your .env file which should not go to your public version control like github or gitlab. For the purposes of this tutorial I include the token directly as a string, which is not production safe!

Telegram api can operate in two modes - webhook mode and long polling mode. For the simplicity of development and later deployment of your bot into the wild we will be using long polling mode.

const Telegraf = require('telegraf')

const bot = new Telegraf('996917:AAF7ZAhEFxTc7WMSnNuO_VHx4YePZzhm20Y')

bot.telegram.deleteWebhook().then(success => {
  success && console.log('🤖 is listening to your commands')
  bot.startPolling()
})

We are ready to launch our bot. To do this you need to start the server. You should see the messages that confirm our server to be running and our bot to be listening.

You would probably be surprised but we are just one step away from giving our bot life and giving it ability to engage into simple interactions. Let’s teach it to greet the users when they open the bot.

const Telegraf = require('telegraf')

const bot = new Telegraf('996917685:AAF7ZAhEFxTc7WMSnNuO_VHx4YePZzhm20Y')

bot.telegram.deleteWebhook().then(success => {
  success && console.log('🤖 is listening to your commands')
  bot.startPolling()
})

bot.start(async ctx => {
  const name = ctx.from.first_name
  ctx.reply(`Hello ${name ? name : 'friend'}! You managed to run me!`)
})

Congrats, you made some good progress and now can bootstrap a bot to prepare the project for new functionalities. Speaking of which, it will be coming in next part of the series where we will do a deep dive into messages and telegraf/telegram APIs for working with them.

In the next part of the series you will learn how to work with messages in telegram bot, how to react to them listening for regex patterns and simple strings and other useful tips for making the user love the interaction with your
bots!

If you liked the post check my other posts and videos at divdev.io and subscribe to my twitter not to miss any important updates! Thanks for reading, see you!