Forem: Nikola

AI-Powered Test Automation: How Playwright Agents Plan, Write, and Fix Tests for Us

Nikola — Tue, 14 Oct 2025 19:08:52 +0000

There’s a point in every QA engineer’s journey when maintaining automated tests becomes harder than writing them. You fix one flaky selector, another breaks - a small refactor ripples through half the suite.

We’ve spent years making automation smarter with patterns, abstractions, and better reporting - but structure alone can’t keep up with change.

Now, with Planner, Generator, and Healer, Playwright can analyze your UI, plan test cases, fix broken steps, and explain issues in plain English. We’re entering a phase where AI acts like a test engineer that never sleeps - one that learns from every failure.

In this post, I’ll show how I used Playwright Agents on a simple Dockerized Juice Shop app - how they plan, write, and fix tests on their own, and what that means for QA.

Setting the Stage: Starting with Structure

Before generating anything, I wanted the AI to understand my folder logic - a clean structure is half the battle in Playwright automation.

To help the Agent learn that structure, I first created the folders manually inside VS Code Insider - one for each part of the framework: components, fixtures, pages, selectors, tests, and utils. Once the structure was ready, I opened the workspace and prompted the Agent to explore it. From that point, every generated file automatically followed the same naming and placement pattern I had defined.

💡 Tip: Always define your folder structure before running the Planner. This gives the AI context about your framework organization, so instead of creating random files, it will follow your layout and naming conventions.

PLAYWRIGHT-AGENT-DEMO
components/  = reusable UI logic and visual parts
fixtures/    = shared test data and setup helpers
pages/       = page objects with user flows
selectors/   = locators and test IDs
tests/       = main test specs
utils/       = helper methods and utilities

Setting Up Playwright Agents

With the structure ready, it’s time to bring the Agents to life.

1️⃣ Use VS Code Insiders

Playwright Agents need VS Code Insiders for full automation - the regular version can only suggest, not act.
👉 https://code.visualstudio.com/insiders

2️⃣ Initialize Playwright Agents

Run this inside your project:

npx playwright init-agents --loop=vscode

Creates all necessary AI agent files inside .github/chatmodes/
Sets up three roles:Planner,Generator, and Healer
Re-run after major Playwright updates to refresh definitions

Enabling File Editing in Chat Mode

Before using the Agents, give them permission to edit your workspace:

Command Palette → Chat → Configure Tools

✅ Edit workspace files
✅ Run commands
✅ Search files

This allows the Agents to actually create and modify files, not just suggest code.

Planner: From Prompt to Login Test Plan

Prompt I used for the Planner: this is the message I sent to the Agent to generate a test plan

Plan a concise login test for Juice Shop (http://localhost:3000) - happy path + invalid creds, with ACs and selector/fixture notes. Output to /specs/login-plan.md.

Planner explored the login screen at http://localhost:3000
Read my workspace and recognized folder structure
Produced /specs/login-plan.md with ACs and implementation notes

Generator: From a Simple Plan to a Complete Login Suite

Prompt I used for the Generator: the message I sent to make the Agent build my entire login flow - pages, selectors, fixtures, tests, and the config - all structured and ready to run.

Generate all login files per /specs/login-plan.md into my folders (components/fixtures/pages/selectors/tests/utils) and include/update playwright.config.ts

What the Generator did:

Created login.components.ts for form logic
Added users.json fixture with test data
Generated LoginPage.ts (page object)
Defined login.selectors.ts for stable locators
Created four spec files:
- login-happy-path.spec.ts
- login-invalid-credentials.spec.ts
- login-security.spec.ts
- login-smoke.spec.ts
⚙️ Updated playwright.config.ts and package.json

Note: The output depends on how detailed your test plan is. Since mine was for the OWASP Juice Shop app, the Agent also generated some security-related tests. They weren’t needed for my scope, but I let it run just to see what it could do.

Healer: Fixing Broken Tests

To test reliability, I intentionally broke a selector in login.selectors.ts - changing #email to #mail. The Healer analyzed the DOM, found the mismatch, and fixed it automatically.

This was a small example, but it showed real potential. Beyond broken selectors, the Healer can also refactor - remove redundant code, simplify methods, and clean up repetition.

Observation: the Healer isn’t always consistent - sometimes a one-line fix becomes a long refactor. Still, it gets the job done, tirelessly, while you focus on bigger problems.

✅ Running the Complete Test Suite

After the Planner created the test plan, the Generator built all the files, and the Healer fixed a few initial issues - I hadn’t written a single line of code myself.

Everything was planned, generated, and healed automatically by the Playwright Agents.

When I finally ran the full test suite, this is how it looked in the terminal:

All login-related scenarios - valid, invalid, and security - executed successfully across multiple browsers. Seeing all tests green without touching a single spec file really shows how far automation has come.

💭 Final Thoughts

After using these Playwright Agents, I can say this is a big step forward for smarter, faster testing. They widen coverage, reduce time on repetitive tasks, and make automation feel more collaborative.

Everything depends on how clearly you describe and structure things - the better your setup, the better the results. I wanted this post to be more hands-on than theoretical - similar to what Debbie O’Brien showed in the Playwright Live session.

The Generator does a solid job but still needs guidance - sometimes repeating logic or overcomplicating simple functions. That’s where we, as QA engineers, come in: to fine-tune, guide, or rewrite when needed. For example, it often adds a .goto('/') in every test - not ideal, but easy to fix and impressive for a start.

I also noticed that sometimes even simple tests get generated in a broken state - for example, with incorrect assertions or missing waits. However, the Healer usually steps in and fixes them automatically once you run the suite. Still, it’s clear that without our QA intuition and manual oversight, these agents wouldn’t deliver tests in the exact way we expect them to behave. A quick look at a failing report is often enough to spot what went wrong and understand how the AI reasoned its way to that error - which is fascinating in itself.

The Healer can overdo things but always finds a way back. Later, you can run the Generator again to tidy up the result.

Recommendation: try these Agents, even on a demo app. You’ll quickly see how much time they save and how AI can become your QA teammate.

Thanks for reading! Feel free to share your thoughts or questions in the comments - let’s keep learning and stay in step with the future of testing. 🚀

Dive into API Testing with Playwright in Java

Nikola — Thu, 08 Aug 2024 18:15:08 +0000

🎯 Overview

Welcome to the world of API testing with Playwright in Java!
If you’re tired of wrestling with boilerplate code and tedious setup processes, you’re in luck. I’ve already set up everything for you in a neat, ready-to-use GitHub repository. All you need to do is clone the project and follow along.

In this post, we’ll explore the ins and outs of the framework I’ve crafted. You’ll learn how to use it, understand its components, and see how it all fits together to make API testing a breeze.

Framework Structure Overview

The repository contains a detailed README file that explains the project structure in depth. Here’s a brief overview:

playwright-api-testing-framework
├─ src
│  ├─ main
│  │  └─ java
│  │     └─ io
│  │        └─ ndenic
│  │           └─ apitesting
│  │              ├─ service
│  │              ├─ model
│  │              └─ utils
│  └─ test
│     ├─ java
│     │  └─ io
│     │     └─ ndenic
│     │        └─ apitesting
│     │           ├─ tests
│     │           └─ util
│     └─ resources
│        ├─ testdata
│        └─ schemas
├─ test-suite
│  └─ testng.xml
└─ pom.xml

For a detailed explanation of each component and directory, please refer to the README.md file in the root of the repository. It includes specific details on the framework structure and how to use each part effectively.

Key Components

src/main/java/io/ndenic/apitesting/service: Contains the API service classes.
src/main/java/io/ndenic/apitesting/model: Houses the POJO classes. The rest-countries branch includes the Country class used for parsing API responses.
src/main/java/io/ndenic/apitesting/utils: Utility classes to support the testing framework.
src/test/java/io/ndenic/apitesting/tests: Where the test cases are located.

🚀 Getting Started

1. Clone the Repository

First things first, grab the project from GitHub. Head over to this repository and clone it to your local machine:

cd Playwright-API-testing-framework-template
git clone https://github.com/ndenic/Playwright-API-testing-framework-template.git

2. Switch to the `rest-countries` Branch

To explore the Country POJO class and the main setup, switch to the rest-countries branch:

git checkout rest-countries

3. Install Dependencies

Next, navigate to the project directory and install the necessary dependencies. The project uses Maven for dependency management, so you’ll need to run:

mvn clean install

This command will pull in all the libraries and plugins we use for API testing.

4. Explore the Structure

Here’s a brief overview of the project structure and key components:

src/main/java/io/ndenic/apitesting: Contains the core code for interacting with APIs.
- service: This is where the magic happens. It contains the APIService class that handles the API requests and responses.
- model: Includes the POJO classes representing the data structures you’ll interact with.

src/test/java/io/ndenic/apitesting/tests: Houses your test cases.
- CountryTests.java: This is where you’ll write your test cases. It uses Playwright to send requests and validate responses.

src/test/resources: Contains configuration and test data.
- config.dev.properties: Configuration for the development environment.
- config.prod.properties: Configuration for the production environment.

testng.xml: Defines the test suite and includes Allure for reporting.

5. Run Tests

To run the tests, simply execute the following command:

mvn test

This will run the tests defined in CountryTests.java and generate reports using Allure.

6. Configuration and Customization

The project is configured to use different environments and tags. You can customize the environment by setting the ENV parameter:

mvn test -DENV=dev

Similarly, run tests by tags using:

mvn test -Dtags=smoke

7. Generating Allure Reports

Once you've successfully run your tests and want to see those beautiful reports that give you insights into your API test results, it's time to generate the Allure report. Allure reports provide a detailed and visual overview of your test results, making it easier to spot issues and track progress.

After running your tests with mvn test, you can generate the Allure report by running the following command:

mvn allure:report

This command will create an Allure report in the target/site/allure-maven-plugin directory. To view the report in your browser, you can serve it using the following command:

mvn allure:serve

This will start a local server and open the report in your default browser. You'll see a comprehensive breakdown of your test cases, complete with execution history

📚 Conclusion

Setting up an API testing framework with Playwright in Java doesn’t have to be overwhelming. With this guide and the provided framework, you can jumpstart your API testing journey and focus on what truly matters—writing and executing effective tests.

For a detailed breakdown of the project structure and more information on usage, visit the GitHub repository and check out the README.md file. Feel free to reach out with any questions or feedback!

Happy testing!

Set up your test automation project with Playwright using Typescript

Nikola — Mon, 11 Jul 2022 07:04:35 +0000

Playwright is a powerful framework, which provides cross-browser automation through a single API, but this post is not entirely about it, but about how to set up a project in a good way. I will use Playwright for this purpose

What you will not find in this post is an initial level explanation of the playwright framework. I think there is already a lot about it on the internet. This is pure practice, something that will serve you well if you are the first to break the ice in terms of setting up a project.

You started working on a new project (or an old one) and a request came or you decided to use Playwright yourself. It's ok to start writing tests just after you install playwright, but in fact, this kind of start will affect your later more complex tests, where your code will be difficult to maintain and even more difficult to add new tests. I will try to make this part easier for you and to help you set good roots at the very beginning.

What you will learn from this post is:

Project setup starts with a good folder structure
Which design pattern is good for that purpose
A step-by-step explanation of what you need so that you can just write the tests afterwards without a headache

Warm-up

If you haven't written a single basic test or at least spent a few hours in this framework, I suggest you first visit their official site (https://playwright.dev/) and look around a bit and write some tests. It's not difficult, and believe me if you deal with test automation at work, you will like this.
I have been using this framework for a year or so. I can tell you that the transition from Selenium/Java to Playwright/TypeScript was not the best.
I'm not saying it was difficult, but I wasn't the happiest at that moment.

But we won't talk about that, now I efficiently use that tool to finish all the work with quality and speed. So in this post, we are working with Playwright using Typescript. These are some general things and can be applied with different tools and it's not related only to the playwright.

Blasting off

We will start with setting up the project. Of course, we need nodejs (https://nodejs.org/en/)
We will create an empty directory, for example, pw-test-automation and after that, we will open an IDE, some code editor. I use VSCode for this purpose, so I will show an example using that code editor

Then we will need to open terminal and do
npm init

After that, we will add the dependencies that are basic for our work, which is only Playwright. You can add everything else as you wish, test runners or any other dependencies.
So next things are
npm i -D @playwright/test

npm i -D playwright

After we have successfully installed the dependencies, we will see the versions of the playwright in the package.json file.

Design pattern

Ok, let's move on to something more serious. We will use what some call the Business-Layer Page-Object Pattern, where the tests will have a clear structure of what is happening there. So we don't catch the selectors in tests, we don't go through the lists, but we call the functions that contain it and which are named enough that someone who doesn't have that much knowledge in that area can understand what's happening there.
Don't worry, it will become clearer to you soon if it hasn't already.
This is my way of implementing this design pattern and it works well for me.

Let's create inside pw-test-automation folder (root folder)
services folder, then inside we will create two more - pages and steps
We need one more folder inside the root folder and that's
tests. Where we will create our test files

Config files

We will quickly create a tsconfig.json file (in root folder), to use better the TS features and avoid some JS syntax errors which will appear without a config file.

Those are just some basic compilerOptions , but of course you can expand it.

Next, we'll create a playwright config file so we can configure our tests to run the way we want. And of course we have the possibility to expand if the project requires it.
(https://playwright.dev/docs/test-configuration)

Here we have some simple things configured.
As u can see for key baseURL we just added URL, but in real projects we won't have one environment, this means that we will have to change the URL as soon as we want to run our tests on another environment.
Now let's change that so that we can change the environment based on the entered input parameters.

I have created one more JSON file which is for environments and it contains two keys. One key is prod, other is dev. But that's all up to you and your project. It depends which environments you got and how are they called.

Okay we added process environment variable ENV, so how to specify that and when?
Next step , we should open package.json and change scripts -> test
"scripts": { "test": "ENV=$npm_config_ENV playwright test" },

Now we just set that if in the future we wanted to run a test based on some environment, we would type, for example, npm run test --ENV=prod or dev in our case. At the moment we don't have a single test, so we have nothing to run, but no rush.

When setting up a project, you have to do it thoroughly and think many steps ahead. Now let's do it again as for the environment, but this time we need to run the tests by tags as well. For example run all regression tests

And then update also test script in package JSON file with latest changes
"scripts": { "test": "ENV=$npm_config_ENV TAGS=$npm_config_TAGS playwright test" },

Ok, I think you get the point. A lot of things can be configured this way. I'll give you a hint for further options, what if we want to run by browser type?

I didn't spend time explaining to you what each key in the config file means individually because I think there is no better explanation than the documentation itself.

Selectors

Let's go to the next section.
We will need to store selectors, somewhere. My advice is to have separate file for that purpose. There are many options -> create only one selectors.json file or have folder selectors which contains JSON files for each Page. I prefer first option , because we have enough freedom to create nested JSON object and store selectors in similar way as to create N amount of files.

Services

Now we have the selectors, we have the config file, we have the basic structure according to the Business-Layer Page-Object Pattern and we are starting to create, for example, the HomePage.

The point is that here we use the selectors that we saved in the json file later in the functions and return them as a Locator object that we can work with.

Let's create steps for HomePage

Now you can already understand that we will only call those steps in the test files. This is just an example and in reality we will have a lot of steps/functions here.

Test data

And now you will definitely need Test data, so for example you pass some data to the function with which you want to test
For this purpose, my suggestion is to create a test-data folder in the root folder and create as many JSON (or any other type) files with test data as you need.

As you can see, the project setup seems almost complete. Everything is clearly separated into units.
But of course CI/CD integration is also needed here, which is not difficult at all, but it takes this story to the other side. I'll leave that up to you, and I'll be there to help you if you need it, of course don't hesitate to ask a question.

It's not over yet, hold on :)

The last thing in that magic circle of POM model is the Test file. So we created a folder in the start for that purpose -> tests.

In that folder we will create, for example, the
home-page-tests.spec.ts file

In the test file, in this case, we have beforeEach, but don't pay attention to that now, because it could also be beforeAll, and in that case, it would be desirable to have afterEach or afterAll as well.
We initialize the page in the before method so that we can pass it to the constructor of the steps class.

We do not use the HomePage functions in tests, unless there is an overwhelming need for it and in some cases it is unavoidable. We already used only Steps functions that do all the work. For example, we have inputTextWithTestData() and after that we add a function for assertion in steps and name it appropriately.

What do we get with this kind of structure ?
Easier maintenance of tests (more precisely steps)
Code redundancy is avoided (we don't have to repeat ourselves, we can use the same step in several test cases if there is a need for it)

Do you need more?

At the end of the day, someone who doesn't understand the code that much will understand what the test is about. You get an almost Gherkin-like syntax, if you look at the "before" method as GIVEN and the rest of the action WHEN insertInputTextWithTestData , THEN assertion

So let me repeat the most important

Service folder contains Page and Steps
Page functions are functions that return Locators or Promises<> that we solve later. We use those functions in the Steps class.
The Steps class literally contains the steps that we will do in the test files to execute a test case (add a book to the cart, open the cart, confirm that the book is in the cart).
We don't want it all to be in one function, but several smaller ones, so that they can be used again if we need to perform the same actions again and also so that the name of the function itself is short and clear.
In the test file, we initialize page and steps, write test() and call functions from the steps class. And let the game begin

Summary

This is how it might look in the end. I added the pipeline YML file, .jenkinsfile and the helper class
Now you can start writing your e2e/UI tests or eventually expand that framework by your project needs.

And yeah, finally, this would be an example of how we will run the tests
npm run test --ENV=dev --TAGS=regression
but we will probably have more parameters to manage what and how we will run in our tests.

Thanks for reading, I hope this will help you if you find yourself in the situation of setting up a project from scratch and of course shoot your questions if you have any.