Forem: chlorine

Execute E2E Test Cases Using Natural Language with Intelli-Browser

chlorine — Sun, 10 Nov 2024 15:51:34 +0000

Introduction

End-to-end (E2E) testing ensures that your application works correctly from start to finish. Writing and maintaining E2E test cases can be complex and time-consuming, especially when dealing with intricate user interactions.

What if you could write E2E tests using simple natural language instructions? This would not only make the tests more readable but also easier to maintain.

The Intelli-Browser is an innovative project that allows you to do exactly that. In this article, we will walk you through how to use it to execute E2E test cases using natural language.

Inspired by Claude-3.5-Sonnet Computer Use

Intelli-Browser draws inspiration from advanced language models Claude-3.5-Sonnet, known for their new ability to use computers the way people do. By interacting with the large model, Intelli-Browser accepts actions , simulates browser usage and combines it with E2E testing scenarios.

Video Demo

Here we have an user task:

Click search and input "Web API", press "arrow down" once to select the second result. Then press "ENTER" to search it. Find "Keyboard API" nearby title "K" and click it

And a video demo is in: MDN Demo Video

How It Works

User Prompt and Page Information

When you write a natural language test case, the user prompt and the current page information are sent to a large language model (LLM) integrated within Intelli-Browser. The LLM analyzes the page content and interactive elements, understanding the context and the required actions.

Action Planning

Based on the analysis, the LLM plans a sequence of actions to achieve the goal specified in the natural language test case. These actions might include clicking buttons, filling out forms, navigating through pages, and verifying certain elements on the webpage.

Execution and Feedback Loop

Intelli-Browser executes the planned actions within a real or simulated browser environment and provides feedback to the LLM on the success or failure of each action. This feedback loop helps in adjusting the subsequent actions based on the real-time status of the webpage.

Termination Conditions

The process continues until either no more actions are required, achieving the task's goal, or it becomes evident that the goal cannot be achieved due to an error or an unexpected page state. This intelligent handling ensures robustness and adaptability in test case execution.

Getting Started

Installation

You can install Intelli-Browser via npm, yarn, or pnpm. Here are the commands for each package manager:

# use npm
npm install @intelli-browser/core

# use yarn
yarn add @intelli-browser/core

# use pnpm
pnpm add @intelli-browser/core

API Reference

To start using Intelli-Browser, you need to import it and create a client instance. Here is how you can do it:

import { IntelliBrowser } from '@intelli-browser/core';

const client = new IntelliBrowser({
  apiKey: '',  // add apiKey or provide ANTHROPIC_API_KEY in .env file
});

// Example of executing natural language instructions
await client.run({
  page,  // playwright Page instance
  message: 'Click search and input "Web API", press "arrow down" to select the second result. then press "ENTER" to search it',  // user prompt
});

Writing Your First Natural Language E2E Test

Let’s create a basic test case that verifies a search functionality on a web application

import { IntelliBrowser } from '@intelli-browser/core';
import { chromium } from 'playwright';

(async () => {
  // Launch a browser instance
  const browser = await chromium.launch();
  const page = await browser.newPage();

  // Navigate to the homepage
  await page.goto('https://example.com');

  const client = new IntelliBrowser({
    apiKey: '',  // add apiKey or provide ANTHROPIC_API_KEY in .env file
  });

  // Execute the user prompt
  await client.run({
    page,
    message: 'Click search and input "Web API", press "arrow down" to select the second result. then press "ENTER" to search it',
  });

  // Close the browser
  await browser.close();
})();

Generating Traditional E2E Test Cases

If you want to generate traditional E2E test cases after executing the natural language prompt, you can retrieve the return data from the client.run method.

import { IntelliBrowser } from '@intelli-browser/core';
import { chromium } from 'playwright';

(async () => {
  // Launch a browser instance
  const browser = await chromium.launch();
  const page = await browser.newPage();

  // Navigate to the homepage
  await page.goto('https://example.com');

  const client = new IntelliBrowser({
    apiKey: '',  // add apiKey or provide ANTHROPIC_API_KEY in .env file
  });

  // Execute the user prompt and generate E2E cases
  const e2eCases = await client.run({
    page,
    message: 'Click search and input "Web API", press "arrow down" to select the second result. then press "ENTER" to search it',
  });

  console.log(e2eCases);
  // Example output:
  // [
  //   'await page.mouse.move(1241.61, 430.2)',
  //   'await page.waitForTimeout(2266)',
  //   'await page.mouse.down()',
  //   'await page.mouse.up()',
  //   'await page.waitForTimeout(3210)',
  //   "await page.mouse.type('Web API')",
  //   'await page.waitForTimeout(3064)',
  //   "await page.keyboard.press('ArrowDown')",
  //   'await page.waitForTimeout(2917)',
  //   "await page.keyboard.press('Enter')",
  //   'await page.waitForTimeout(6471)',
  //   "await page.keyboard.press('PageDown')",
  //   'await page.waitForTimeout(7021)',
  //   'await page.mouse.move(687.39, 923.4)',
  //   'await page.waitForTimeout(4501)',
  //   'await page.mouse.down()',
  //   'await page.mouse.up()'
  // ]

  // Close the browser
  await browser.close();
})();

At End

The Intelli-Browser project simplifies the process of writing and executing E2E test cases by using natural language instructions and advanced language model capabilities. This approach makes tests more understandable and maintainable, especially for teams with less focus on coding expertise.

For more information and advanced configurations, check out the Github repository. Feel free to open an issue or submit a pull request.

Happy testing!

How does pnpm work

chlorine — Fri, 31 Mar 2023 15:40:03 +0000

As one of the very popular package managers, pnpm is mainly characterized by fast speed and saving disk space. I will introduce how pnpm works to help you understand the principle of pnpm.

Introduction

The meaning of pnpm is performant npm. From the benchmarks in pnpm website, we can see in many scenarios pnpm has good performance advantages compared with npm/yarn/yarn_pnp.

Directory structure of node_modules

Nested structure

In earlier versions of npm@2, corresponding to Node.js 4.x and previous version, node_modules is a nested structure while installing.

A simple case here, both demo-foo and demo-baz depend on demo-bar. When demo-foo and demo-baz are installed in the same repository, we can get the following node_modules structure:



node_modules
└─ demo-foo
   ├─ index.js
   ├─ package.json
   └─ node_modules
      └─ demo-bar
         ├─ index.js
         └─ package.json
└─ demo-baz
   ├─ index.js
   ├─ package.json
   └─ node_modules
      └─ demo-bar
         ├─ index.js
         └─ package.json

Although the directory structure is relatively clear at this time, each dependent package will have its own node_modules directory, and the same dependency has not been reused. For the above example, the same dependency demo-bar has been installed twice.

Another problem is the Maximum Path Limitation of windows. In some complex cases when the project has a deep dependency level, the dependent path often exceeds the length limit.

Flat structure

In order to solve the above problems, yarn proposed a flat structure design: flattening all dependencies in node_modules. And the implementation of the later npm v3 version is similar, so use yarn or npm@3+ to install the above example, you will get the following flat directory structure:



node_modules
└─ demo-bar
   ├─ index.js
   └─ package.json
└─ demo-baz
   ├─ index.js
   └─ package.json
└─ demo-foo
   ├─ index.js
   └─ package.json

In addition, for different versions of the same dependency in this way, only one of them will be hoisted, and the remaining versions will still be nested in the corresponding packages.

For instance, if we upgrade the above demo-bar to v1.0.1 (its dependency demo-foo is also v1.0.1), you will get the following structure, which version will be hosited to the top depends on the order of installation:



node_modules
└─ demo-bar
   ├─ index.js
   └─ package.json
└─ demo-baz
   ├─ index.js
   ├─ package.json
   └─ node_modules
      └─ demo-bar
         ├─ index.js
         └─ package.json
└─ demo-foo
   ├─ index.js
   ├─ package.json
   └─ node_modules
      └─ demo-bar
         ├─ index.js
         └─ package.json

Problems with flat structures

The flat solution is not perfect, and accompanies with some new problems:

Phantom dependencies

Phantom dependencies means the dependencies that are not declared in package.json but can be directly used in your project. This issue is caused by the flat structure, and the dependencies of the dependencies will also be hoisted to the top level of node_modules, so that you can reference it directly in the project. And if some day this sub-dependency is no longer a dependency of the reference package, there will be problems with the references in the project.

For example, in the project containing demo-foo and demo-baz as the dependencies, demo-bar also appears in node_modules as a dependent dependency:



node_modules
└─ demo-bar
   ├─ index.js
   └─ package.json
└─ demo-baz
   ├─ index.js
   └─ package.json
└─ demo-foo
   ├─ index.js
   └─ package.json

Npm doppelgangers

NPM doppelgangers refer to different versions of the same dependency, due to the hoist mechanism, only one version will be hoisted, and other versions may be installed repeatedly. The same example as above, when we upgrade demo-bar to v1.0.1, the 1.0.0 version that demo-baz and demo-foo depends on will be repeatedly installed in a nested way:



node_modules
└─ demo-bar // v1.0.1
   ├─ index.js
   └─ package.json
└─ demo-baz
   ├─ index.js
   ├─ package.json
   └─ node_modules
      └─ demo-bar // v1.0.0
         ├─ index.js
         └─ package.json
└─ demo-foo
   ├─ index.js
   ├─ package.json
   └─ node_modules
      └─ demo-bar // v1.0.0
         ├─ index.js
         └─ package.json

How to solve

First of all, pnpm installs the dependencies to the global store, and then use symbolic link and hard link to organize directory structure. It links the global dependencies into the project, and links the direct dependencies into top level of node_modules directory. All dependencies are flattened under the node_modules/.pnpm directory, which realizes the global dependencies shared of all projects, and solves the problem of phantom dependencies and NPM doppelgangers.

Symbolic link and hard link

Link is the way of file sharing in the operating system, where the one is symbolic link, also known as soft link, and the onther one is hard link. From the point of view of use, there is no difference between them, both supporting reading and writing, and if it's an excutable file, it can also be excuted directly. The main difference is the core principle:

Hard link

Hard links don't create a new index node, the source file and the hard link point to the same index node
Hard links don't support directories, only file level, and don't support cross disk partitions
The file is not actually deleted until the source file and all hard links are deleted

Symbolic link

The symbolic link stores the path of the source file, pointing to the source file, similar to the shortcut of Windows
The symbolic link supports directories and files, which are different files from source files. It has different inode values and different file types, so the symbolic link can be accessed across partitions
After deleting the source file, the symbolic link still exists, but the source file cannot be accessed through it

How to create links through command



# symbolic ink
ln -s myfile mysymlink

# hard link
ln myfile myhardlink

How does pnpm work

At first, pnpm will installs the dependencies into <home dir>/.pnpm-store in the current partition. The current store location can be obtained by the following command：



pnpm store path

And then hard link the required packages from node_modules/.pnpm into the store path. Finally, symbolically link the top-level dependencies and dependent dependencies in node_modules to node_modules/ .pnpm, an example that depends on demo-foo@ 1.0.1 and demo-baz@ 1.0.0, the node_modules structure is:



node_modules
└─ .pnpm
   └─ demo-bar@1.0.0
      └─ node_modules
         └─ demo-bar -> <store>/demo-bar
   └─ demo-bar@1.0.1
      └─ node_modules
         └─ demo-bar -> <store>/demo-bar
   └─ demo-baz@1.0.0
      └─ node_modules
         ├─ demo-bar -> ../../demo-bar@1.0.0/node_modules/demo-bar
         └─ demo-baz -> <store>/demo-baz
   └─ demo-foo@1.0.1
      └─ node_modules
         ├─ demo-bar -> ../../demo-bar@1.0.1/node_modules/demo-bar
         └─ demo-foo -> <store>/demo-foo
└─ demo-baz -> ./pnpm/demo-baz@1.0.0/node_modules/demo-baz
└─ demo-foo -> ./pnpm/demo-baz@1.0.1/node_modules/demo-foo

Here is a screenshot of the pnpm website to help you better understand how symbolic link and hard link are organized in the project structure:

For the actual usage of the link in pnpm, the following is the relevant source code:



function createImportPackage (packageImportMethod?: 'auto' | 'hardlink' | 'copy' | 'clone' | 'clone-or-copy') {

  // this works in the following way:

  // - hardlink: hardlink the packages, no fallback

  // - clone: clone the packages, no fallback

  // - auto: try to clone or hardlink the packages, if it fails, fallback to copy

  // - copy: copy the packages, do not try to link them first

  switch (packageImportMethod ?? 'auto') {

  case 'clone':

    packageImportMethodLogger.debug({ method: 'clone' })

    return clonePkg

  case 'hardlink':

    packageImportMethodLogger.debug({ method: 'hardlink' })

    return hardlinkPkg.bind(null, linkOrCopy)

  case 'auto': {

    return createAutoImporter()

  }

  case 'clone-or-copy':

    return createCloneOrCopyImporter()

  case 'copy':

    packageImportMethodLogger.debug({ method: 'copy' })

    return copyPkg

  default:

    throw new Error(Unknown package import method </span><span class="p">${</span><span class="nx">packageImportMethod</span> <span class="k">as</span> <span class="kr">string</span><span class="p">}</span><span class="s2">)

  }

}

Other abilities

At present pnpm can be installed and used out of Node.js runtime. And pnpm can also manage Node.js version through pnpm env, similar to nvm. For a full feature comparison with npm/yarn see feature-comparison

Limitations

Due to compatibility issues with symbolic links in certain scenarios, pnpm cannot currently be used on applications deployed on Electron and Lambda, as outlined in: discussion

By adding node-linker=hoisted to .npmrc, a flat node_modules directory without symbolic links can be created, which is similar to the directory structure created via npm/yarn.

As the same store is shared globally, modifying the contents within node_modules will directly affect the corresponding content in the global store, which will also have an impact on other projects.

For the above issue, the most recommended approach is to use clone(copy-on-write). By default, multiple references point to the same file, and only when the user needs to modify it, a copy is made so that it will not affect other references that are reading the content of the source file.

However, not all operating systems support this. By default, pnpm attempts to use clone. If it is not supported, it falls back to using hard link. You can also manually set the package reference method by specifying package-import-method in your .npmrc configuration file.

For other limitations, see: https://pnpm.io/limitations

Other tools

bun: https://github.com/oven-sh/bun
- bun is a JS runtime written in Zig. It also offers a package management tool, but it may encounter some compatibility issues.
Volt：https://github.com/dimensionhq/volt
- volt is a Node.js package manager written in Rust, which is known for its extremely fast performance. It is currently in beta, but it has not been updated for several months.
tnpm: https://github.com/cnpm/npminstall
- tnpm implements a virtual mapped directory by utilizing the features of Filesystem in Userspace(FUSE) and OverlayFS.

Reference

Getting started with creating a frontend project quickly

chlorine — Sun, 20 Nov 2022 14:38:32 +0000

How to create a frontend project from zero quickly? Let's talk about the steps of creating a new project since 2022.

Preparation

First, the installations of Node.js and package manager are required in your local environment. pnpm is recommanded, and pnpm can also be used directly as Node.js version manager.

Install pnpm：

# Mac or Linux
curl -fsSL https://get.pnpm.io/install.sh | sh -

# Windows
iwr https://get.pnpm.io/install.ps1 -useb | iex

Install the LTS version of Node.js using pnpm:

pnpm env use --global lts

Create a project

I will show how to create a project using React and TypeScript as an example.

Scaffold

For a new project, using vite is highly recommanded. We can create a inital project based on vite by running the command:

pnpm create vite my-react-app --template react-ts

We can see the following directory structure：

.
├── public
├── src
├── index.html
├── package.json
├── tsconfig.json
├── tsconfig.node.json
└── vite.config.ts

ESLint

ESint is very useful and necessary with checking errors in your code by static analysis. Here we choose the aribnb style eslint rules which are very popular in the community. Install the corresponding packages via the following command：

pnpm add eslint eslint-config-airbnb-base eslint-plugin-import -D

And then add a new file named .eslintrc.json in your root directory:

{
  "extends": [
    "eslint:recommended",
    "airbnb-base",
  ],
  "plugins": [
    "import"
  ],
}

Support TypeScript and React

For a project using typescript and react, additonal parser and plugin are required to support, install them first:

# TypeScirpt eslint parser
pnpm add @typescript-eslint/parser @typescript-eslint/eslint-plugin -D

# React eslint plugin
pnpm add eslint-plugin-react eslint-plugin-react-hooks -D

Then add configurations about it in .eslintrc.json:

{
   "extends": [
    "eslint:recommended",
    "airbnb-base",
    "plugin:react/recommended",
    "plugin:react/jsx-runtime",
    "plugin:react-hooks/recommended",
    "plugin:@typescript-eslint/eslint-recommended",
    "plugin:@typescript-eslint/recommended"
  ],
  "plugins": [
    "import",
    "@typescript-eslint",
    "react",
  ],
  "parser": "@typescript-eslint/parser"
}

Finally we add the script in package.json to run eslint:

{
  "scripts": {
     "lint": "eslint --fix --quiet --ext .ts,.tsx src"
  }
}

Prettier

Prettier is a tool for code formatting, it can help us to manage our code style, for instance code indent and blank lines. Install prettier:

pnpm add prettier -D

And then add a new file named .prettierrc.json in your root directory：

{
  "printWidth": 80,
  "tabWidth": 2,
  "semi": true,
  "singleQuote": true
}

Use prettier with eslint

We can use the ESLint plugin of prettier to check the rules of prettier code style synchronously when checking ESLint rules:

pnpm add eslint-plugin-prettier eslint-config-prettier -D

And then update configurations in .eslintrc.json，Note that you need to set the prettier/prettier rules to error：

{
   "extends": [
    "eslint:recommended",
    "airbnb-base",
    "plugin:react/recommended",
    "plugin:react/jsx-runtime",
    "plugin:react-hooks/recommended",
    "plugin:@typescript-eslint/eslint-recommended",
    "plugin:@typescript-eslint/recommended",
    "prettier"
  ],
  "plugins": [
    "import",
    "@typescript-eslint",
    "react",
    "prettier"
  ],
  "parser": "@typescript-eslint/parser",
  "rules": {
    "prettier/prettier": "error",
  }
}

Husky and lint-staged

After configuring ESLint and prettier, you need a workflow to trigger the related checks. Here we choose the commonly used combination of husky and lint-staged:

pnpm add husky lint-staged -D

Add configurations to package.json in the root directory. It will execute ESLint and prettier fixes when matching files with ts or tsx suffixes:

{
  "lint-staged": {
    "**/*.{ts,tsx}": [
      "eslint --fix --quiet",
      "prettier --write",
      "git add"
    ]
  }
}

You are also supposed to add a precommit file under .husky directory to trigger the behavior of lint-staged:

#!/bin/sh
. "$(dirname "$0")/_/husky.sh"

npx lint-staged

At last, adding the inital script of husky to package.json to ensure the above hooks can be triggered normally:

{
  "scripts": {
    "prepare": "husky install"
  }
}

If everything works fine. Every time when you commit, husky will trigger the precommit hook first. And then use lint-staged to match file rules. Finally run the related scripts such as eslint and prettier fixes.

Vitest

Unit testing is a very important part of project development. Through it, the code quality and logical integrity can be guaranteed to a certain extent. For project created with vite, vitest is more compatiable with it to write test cases. Let's install them first:

pnpm create vitest jsdom -D

And then we add configuration jsdom as test environment. After we add the type declaration of vitest at the top, we can configure vitest directly in vite.config.ts. The plugins and configurations are shared and there is no need to add a new vitest.config.ts file:

/// <reference types="vitest" />
import { defineConfig } from 'vite';
import react from '@vitejs/plugin-react';

export default defineConfig({
  plugins: [react()],
  test: {
    testTimeout: 20000,
    environment: 'jsdom',
  },
});

A simple example of a test case written using @testing-library/react can be found in: react-typescript

Github workflow

CI is a important part of project automation. Through CI, we can run some tasks automatically. Take github as an instance, we configurate a workflow, its main function is to run eslint checks, typescript type checks and test cases when you push or pull request your code to github.

At first we need to create .github/workflows directory in the root, and then create a file named ci.yml under it. Its main content is:

name: CI
on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main
jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Set node v14
        uses: actions/setup-node@v3
        with:
          node-version: '14'
      - name: Install
        run: npm install
      - name: Lint
        run: npm run lint
  typecheck:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Set node v14
        uses: actions/setup-node@v3
        with:
          node-version: '14'
      - name: Install
        run: npm install
      - name: Typecheck
        run: npm run typecheck
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Set node v14
        uses: actions/setup-node@v3
        with:
          node-version: '14'
      - name: Install
        run: npm install
      - name: Build
        run: npm run test

Here we create three jobs: lint/typecheck/test. They will automatically execute the commands defined in package.json after triggering the push or pull_request operation. By the way, the script typecheck is the same as tsc.

How to create faster

For current frontend projects, the above typescript, eslint, prettier, husky and etc are basically standard. But it's time-consuming if you re-do such a series of configurations every time you create a new project. So I have developed a personal program that can help you quickly create a project with the above configurations. Only a line of command is needed:

pnpm create @tooltik/cap my-cap-app --template react-ts

The project can be found at: github, any issues, pull requests and suggestions are highly welcomed!