Forem: Yaniv

How I Built a 12-Step CI/CD Pipeline That Spins Up MySQL, Flask, and Playwright From Scratch

Yaniv — Sat, 11 Apr 2026 14:44:47 +0000

Setting up CI for a web app is straightforward. Setting up CI for a test automation framework that needs a real database, two backend servers, and a headless browser — all starting from zero on every run — is a different problem.

This is how I built a GitHub Actions pipeline that provisions the entire infrastructure, runs 37 tests across 3 layers, and deploys a historical test report — every time I push to main.

The Problem: "Works On My Machine" Doesn't Scale

Locally, my test framework needs:

MySQL 8.0 with a specific schema loaded
A JSON Server running on port 3000
A Flask API server running on port 5000
Playwright with Chromium installed
Environment variables for DB credentials and API keys

Running pytest locally assumes all of this is already set up. In CI, nothing exists. Every run starts from an empty Ubuntu container.

The challenge isn't running the tests — it's building the world they need to run in.

The 12 Steps

Here's the full pipeline, and why each step exists:

Steps 1-3: Foundation

- name: 1. Checkout Code
  uses: actions/checkout@v4

- name: 2. Set up Python 3.13
  uses: actions/setup-python@v5
  with:
    python-version: '3.13'

- name: 3. Set up Node.js 18
  uses: actions/setup-node@v4
  with:
    node-version: '18'

Python for the test framework. Node.js for JSON Server. Nothing surprising here, but note the explicit version pinning — 3.13, not 3.x. CI flakiness often starts with "we let the runner pick the version."

Step 4-5: Dependencies

- name: 4. Install Python Dependencies
  run: |
    python -m pip install --upgrade pip
    pip install -r requirements.txt

- name: 5. Install Playwright Browsers
  run: playwright install chromium --with-deps

--with-deps is critical. Without it, Playwright installs the browser binary but not the OS-level libraries it needs (libgbm, libasound, etc.). The test run will fail with a cryptic error about missing shared objects.

Step 6: Database Schema

services:
  mysql:
    image: mysql:8.0
    env:
      MYSQL_ROOT_PASSWORD: root_password
      MYSQL_DATABASE: expense_test_db
      MYSQL_USER: test_user
      MYSQL_PASSWORD: test_password
    ports:
      - 3306:3306
    options: >-
      --health-cmd="mysqladmin ping"
      --health-interval=10s
      --health-timeout=5s
      --health-retries=5

# In steps:
- name: 6. Initialize MySQL Schema
  run: |
    sudo apt-get install -y mysql-client
    mysql -h 127.0.0.1 -u test_user -ptest_password expense_test_db < data/init_mysql.sql

The MySQL service container starts alongside the job. The health-cmd ensures the container is actually ready before we try to load the schema. Without health checks, you'll hit "Connection refused" errors roughly 30% of the time.

The schema itself is minimal — one table with a CHECK constraint:

CREATE TABLE IF NOT EXISTS expenses (
    id INT PRIMARY KEY AUTO_INCREMENT,
    expense_name VARCHAR(255),
    amount DOUBLE CHECK (amount >= 0),
    date VARCHAR(50),
    category VARCHAR(100)
);

That CHECK constraint is actually a test target — one of my E2E tests validates that negative amounts get rejected at the DB level even though the UI accepts them.

Steps 7-9: Server Orchestration

- name: 7. Install & Start JSON Server
  run: |
    npm install -g json-server
    json-server --watch json-server/db.json --port 3000 &

- name: 8. Start Flask Server
  env:
    DB_TYPE: mysql
    MYSQL_HOST: 127.0.0.1
  run: |
    python server/app.py &

- name: 9. Wait for Servers to be Ready
  run: |
    curl --retry 10 --retry-delay 2 --retry-connrefused http://localhost:3000/expenses
    curl --retry 10 --retry-delay 2 --retry-connrefused http://localhost:5000/expenses
    echo "All servers are up and running!"

The & at the end of each server command runs it in the background. Step 9 is the safety net — it polls both servers with retry logic until they respond, or fails after 20 seconds.

This is a pattern I see skipped in a lot of CI setups. People start a server and immediately run tests, then wonder why they get intermittent connection errors. Always add a readiness check.

Step 10: The Actual Tests

- name: 10. Run Tests (exclude Mobile)
  env:
    GROQ_API_KEY: ${{ secrets.GROQ_API_KEY }}
  run: |
    pytest -m "not mobile" --alluredir=allure-results --ai-analysis

-m "not mobile" excludes tests that require a physical Android device. The remaining 37 tests cover Web (Playwright), API, Database, and cross-layer E2E.

--ai-analysis triggers an optional Groq LLM call on test failures to classify the root cause. The API key is stored as a GitHub Secret.

Steps 11-12: Reporting

- name: 11. Generate Allure Report
  uses: simple-elf/allure-report-action@master
  if: always()
  with:
    allure_results: allure-results
    allure_history: allure-history
    keep_reports: 20

- name: 12. Deploy Allure Report to GitHub Pages
  uses: peaceiris/actions-gh-pages@v3
  if: always()
  with:
    github_token: ${{ secrets.GITHUB_TOKEN }}
    publish_dir: allure-history

if: always() is key — the report is generated even when tests fail. Without this, failures produce no report, which is exactly when you need one most.

keep_reports: 20 maintains the last 20 runs, so you get historical trend analysis in Allure — pass rates over time, flakiness detection, duration trends.

What I Learned Building This

1. Health checks prevent 80% of CI flakiness. The MySQL health-cmd and the curl retry loops in step 9 eliminated almost all intermittent failures. Before adding them, roughly 1 in 4 runs failed due to timing issues.

2. if: always() on reporting steps is non-negotiable. The whole point of CI reports is to understand failures. If the report step only runs on success, it's useless.

3. Service containers beat docker-compose in GitHub Actions. I originally tried running docker-compose up inside the workflow. It works, but it's slower and harder to debug. Native service containers integrate better with the runner's networking.

4. Pin your versions. Python 3.13, Node 18, MySQL 8.0 — not latest, not 3.x. A version bump in a dependency should be a deliberate commit, not a surprise in CI.

The Docker Alternative

For local development, the same test suite runs via Docker Compose with a single command:

docker-compose up --build

This uses a custom entrypoint script that replicates the CI steps — waits for MySQL, starts both servers, runs pytest:

[1/5] Waiting for MySQL...        ✓
[2/5] Starting JSON Server...     ✓
[3/5] Starting Flask Server...    ✓
[4/5] Waiting for servers...      ✓
[5/5] Running tests...
========================= 34 passed, 3 xfailed =========================

Same tests, same infrastructure, same results — whether it runs in GitHub Actions or on a developer's laptop.

Full Source

The complete workflow file and Docker setup are in the repo:

GitHub: Financial-Integrity-Ecosystem

The CI config is at .github/workflows/ci.yml. The Docker setup is in Dockerfile, docker-compose.yml, and docker-entrypoint.sh.

This is part 2 of a series on building a multi-layer test automation framework. Part 1 covered using Set Theory for cross-layer data integrity validation.

Yaniv Metuku — QA Automation Engineer

How I Used Set Theory to Catch Bugs That Unit Tests Miss

Yaniv — Fri, 10 Apr 2026 16:40:02 +0000

Most test automation tutorials teach you to test layers in isolation: UI tests check buttons, API tests check status codes, DB tests check records. But the bugs that actually cost money in production? They live between the layers.

I learned this the hard way while building a test automation framework for a financial expense tracker. This post is about one specific technique — Set Theory validation — that catches data integrity bugs that no single-layer test will ever find.

The Problem: Everything Passes, But Data Is Wrong

Imagine this scenario:

A user creates an expense for $100 through the Web UI
The UI shows a success message ✅
The API returns 201 Created ✅
The database has... $0. Or two records. Or nothing.

Every individual layer test passes. The UI test confirms the success message appeared. The API test confirms the status code. But nobody verified that the actual data made it through the entire pipeline correctly.

In financial applications, this is not a cosmetic bug — it's a silent data inconsistency that can go unnoticed until an audit.

The Approach: Database State as a Mathematical Set

Instead of checking "does a record exist?", I treat the entire database table as a mathematical set, and use set difference to prove exactly what changed.

Here's the concept:

# Step 1: Capture DB state BEFORE the action
old_set = {(id, name, amount) for each row in expenses}
old_sum = SUM(amount) from expenses

# Step 2: Perform the action (create expense via UI or API)

# Step 3: Capture DB state AFTER the action
new_set = {(id, name, amount) for each row in expenses}
new_sum = SUM(amount) from expenses

# Step 4: Validate using set difference
isolated_record = new_set - old_set

assert len(isolated_record) == 1          # Exactly ONE new record
assert new_sum - old_sum == expected_amount  # Amount is correct

This is powerful because:

It's operation-independent. Whether the expense was created via UI, API, or direct SQL — the validation is the same.
It catches duplicates. If a bug causes two records to be inserted, len(isolated_record) will be 2.
It catches phantom data. If some other process modified the table during the test, the set difference will include unexpected records.
It catches amount drift. If $100 was entered but $99.99 was stored (floating point issues, rounding bugs), the sum check catches it.

Real Implementation

In my framework, this looks like this across the stack:

DB helper that captures state:

@staticmethod
@allure.step("DB: Get all expenses as set")
def get_all_expenses_as_set(cursor):
    cursor.execute("SELECT id, expense_name, amount FROM expenses")
    return {(row[0], row[1], row[2]) for row in cursor.fetchall()}

@staticmethod
@allure.step("DB: Get sum of amounts")
def get_sum_of_amounts(cursor):
    cursor.execute("SELECT COALESCE(SUM(amount), 0) FROM expenses")
    return cursor.fetchone()[0]

Cross-layer E2E test that uses it:

def test_api_create_reflects_in_db(self):
    # Capture pre-state
    old_set = DBActions.get_all_expenses_as_set(cursor)
    old_sum = DBActions.get_sum_of_amounts(cursor)

    # Act: Create expense via API
    response = APIActions.post(session, url, payload)
    APIVerification.verify_status_code(response, 201)

    # Capture post-state
    new_set = DBActions.get_all_expenses_as_set(cursor)
    new_sum = DBActions.get_sum_of_amounts(cursor)

    # Validate integrity
    diff = new_set - old_set
    assert len(diff) == 1, f"Expected 1 new record, got {len(diff)}"
    assert new_sum - old_sum == expected_amount

The same pattern applies to update (set difference shows one record with changed values) and delete (old_set - new_set shows the removed record).

Where This Caught Real Bugs

While building this, the Set Theory approach caught two issues that single-layer tests completely missed:

1. MySQL CHECK constraint silently rejecting negative amounts.
The UI happily accepted -50 as an expense amount. The API returned 201. But MySQL's CHECK (amount >= 0) constraint blocked the INSERT — so the record never existed in the DB. The set difference was empty when it should have contained one record. Without the cross-layer test, this would have looked like a perfectly passing test suite.

2. VARCHAR(255) overflow truncation.
A 300-character expense name was entered through the UI. The API accepted it. MySQL truncated it to 255 characters silently. The set difference caught the mismatch because the stored record didn't match the expected data.

The Full Picture

This technique is one piece of a larger framework I built with 53 tests across 4 layers (Web/Playwright, API/Flask, Mobile/Appium, Database/MySQL). The cross-layer E2E tests that use Set Theory are a small percentage of the total test count, but they catch the highest-risk bugs.

The architecture enforces strict separation:

Tests → Workflows → Actions/Verifications → Page Objects + Data

Every layer has one job. Tests never call raw UI or API actions directly — they go through workflows that compose actions into business flows. This keeps the set theory validation reusable across different test scenarios.

When You Should (and Shouldn't) Use This

Use it when:

Your application handles financial data, inventory, or any domain where data accuracy matters more than UI polish
Data flows through multiple systems (frontend → backend → database → reporting)
You've had production bugs where "the UI said X but the DB had Y"

Don't use it when:

You're testing a static website or content-only app
The DB is behind a well-tested ORM with strong constraints and you trust the abstraction
Test execution time is critical and you can't afford the extra DB queries

Try It Yourself

The full framework is open source:

GitHub: Financial-Integrity-Ecosystem

You can run the entire suite with a single command:

git clone https://github.com/Yaniv2809/Financial-Integrity-Ecosystem.git
cd Financial-Integrity-Ecosystem
docker-compose up --build

This spins up MySQL, Flask, JSON Server, and Playwright — runs all 37 non-mobile tests automatically.

The cross-layer E2E tests are in tests/api/test_e2e_api_db_expense.py and tests/test_e2e_web_api_db.py if you want to see the set theory pattern in action.

I recently shared this project on r/QualityAssurance and got valuable feedback that led to several improvements, including adding a testing philosophy section that explains the business risk behind each layer. If you have feedback or have used similar patterns in production, I'd genuinely like to hear about it.

Yaniv Metuku — QA Automation Engineer