Have you ever run a Playwright test locally, watched it pass with flying colors, and then pushed your code—only for it to fail on CI? Debugging these inconsistencies can be tricky, but there’s a simple trick that can help you figure out what’s going wrong.
The Key to Diagnosing Flaky Tests
When a test passes consistently on your local machine but fails on CI, the issue is often related to timing, race conditions, or resource constraints. To expose these hidden problems, you can use Playwright’s --repeat-each flag.
Step 1: Run Your Test Multiple Times
Use the following command to run your test 100 times:
npx playwright test file-with-failing-test.spec.ts:20 --repeat-each=100
This will execute the test repeatedly in multiple workers, increasing the chances of catching intermittent failures. If there’s a race condition or an unstable dependency, it’s more likely to show up during repeated runs.
Step 2: Stress Test with More Workers
Sometimes, running the test repeatedly isn’t enough, especially if the failure is triggered by CPU or memory constraints on CI. In this case, you can stress your system by maximizing the number of workers:
npx playwright test --repeat-each=100 --workers=10
If your local machine has 10 CPU cores, Playwright will utilize them to simulate the load your test might face in a CI environment. This can help uncover concurrency issues and race conditions that may not appear in single-run scenarios.
Step 3: Stop on First Failure
Running a test 100 times can take a while, but you don’t have to wait for all runs to complete. Use the -x flag to stop execution as soon as a failure occurs:
npx playwright test --repeat-each=100 -x
This will save you time by immediately identifying when a test starts failing under stress.
Final Check: Ensure Stability Before Pushing
Once you’ve identified and fixed the root cause of the flaky test, rerun the test with --repeat-each=100 to confirm stability. If all 100 tests pass, you can push your changes with confidence, knowing it’s far less likely to fail on CI.
Conclusion
Debugging flaky Playwright tests on CI doesn’t have to be a guessing game. By leveraging --repeat-each, --workers, and -x, you can reliably reproduce failures locally and fix them before they cause problems in your pipeline.
If you found this guide helpful, be sure to share it with your team and check out more Playwright testing tips. 🚀 Happy testing!
🔹 Want to see this in action? Check out our latest video:
Top comments (0)