Forem: Om Narayan

HIPAA Mobile QA Checklist: Your Testing Pipeline is a Compliance Risk

Om Narayan — Sun, 28 Dec 2025 23:31:28 +0000

In HealthTech, "move fast and break things" is not a strategy. It is a lawsuit.

In 2024, healthcare data breaches compromised 275 million patient records. The Change Healthcare ransomware attack alone affected 192 million individuals—the largest healthcare breach in history. OCR closed 22 HIPAA investigations with financial penalties, and 2025's enforcement initiative has one laser focus: Risk Analysis Failures.

They are no longer just auditing production systems. They are auditing everything.

Including your testing pipeline.

The Myth of "Fake" Data

"We don't need to worry—we use synthetic patient data."

I hear this every week. It sounds logical. If the data is not real, HIPAA does not apply, right?

Wrong.

Security research found that 77% of mHealth apps contained hardcoded API keys. Even if your patient names are "John Doe," your application binary contains:

Hardcoded API Keys granting access to production-adjacent environments
Business Logic revealing how you process diagnoses and claims
Staging Endpoints connecting to real (or realistic) patient data
Accidental Leaks from "anonymized" production dumps that were not scrubbed properly

Upload that binary to a third-party cloud. That cloud gets breached. Now explain to an auditor why your proprietary health algorithms and PHI system credentials were on a server in another country.

The "Shared Responsibility" Trap

Cloud providers operate on a Shared Responsibility Model:

The Vendor secures the physical infrastructure
You are responsible for the data you put on it

When you run a test on a cloud device, you generate artifacts that live in vendor storage for 30-90 days:

Screenshots of patient dashboards
Video recordings of intake flows
HTTP logs containing JSON bodies with medical records
Crash dumps showing memory states

If you cannot guarantee that every artifact is scrubbed of PHI, you are at risk. And "the vendor handles security" is not a compliant answer.

Does Your Cloud Provider Even Offer a BAA?

Provider	BAA Available?	Reality
AWS Device Farm	Yes (via AWS BAA)	Only if configured correctly with encryption + audit logs
BrowserStack	Enterprise only	"Contact Sales"—expect 5-10x standard pricing
Sauce Labs	Enterprise only	Requires explicit request
LambdaTest	No public BAA	Not suitable for PHI
Firebase Test Lab	No	Not HIPAA eligible

Critical: A BAA does not make you compliant. AWS explicitly states compliance is "conditional on services being configured correctly by the customer." Misconfigure, and the BAA does not protect you.

The 5-Point HIPAA QA Audit

Your testing infrastructure must pass these five checks based on the HIPAA Security Rule:

1. Data Residency Control

✅ Pass: Data never leaves your corporate firewall
❌ Fail: Binaries uploaded to vendor S3 buckets in Virginia/Frankfurt

2. Transmission Security

✅ Pass: Test commands travel over encrypted tunnels you control
❌ Fail: Binaries travel over public internet to shared cloud infrastructure

3. Right to Audit

✅ Pass: You can physically inspect devices running your tests
❌ Fail: You trust a vendor's SOC2 report but cannot audit their device wiping

4. Data Destruction

✅ Pass: You factory reset devices immediately after tests
❌ Fail: You rely on vendor cleanup scripts (which often miss cached logs)

5. Access Logging

✅ Pass: Every engineer has unique credentials; all access is logged
❌ Fail: Team shares a generic cloud login (violates HIPAA unique user ID requirement)

Score 5/5? You are probably audit-ready.

Score 3-4? You have gaps to address.

Below 3? Schedule a risk analysis review immediately.

Real Enforcement: What Gets Organizations Fined

Every organization penalized in early 2025 had the same finding:

"Failure to conduct a compliant risk analysis."

A risk analysis under HIPAA requires identifying every system that creates, receives, or transmits ePHI. This includes:

Production databases (obviously)
Backup systems (usually covered)
Developer laptops (sometimes covered)
Testing infrastructure (almost never covered)

When an auditor asks "Where does PHI exist?" and you do not list your QA pipeline, you have a gap. When that gap involves third-party cloud infrastructure, you have a $3 million problem—the penalty one national supplier paid in 2025 for risk analysis failures.

The Cost Math

Violation Tier	Penalty Range
Tier 1: Unknowing	$141 - $71,162 per violation
Tier 2: Reasonable cause	$1,424 - $71,162 per violation
Tier 3: Willful neglect (corrected)	$14,232 - $71,162 per violation
Tier 4: Willful neglect (not corrected)	$71,162 - $2,134,831 per violation

A single test run with improperly handled PHI, multiplied by affected records, generates catastrophic liability.

The Three Paths to Compliance

Path 1: De-Identified Data Only

Never use PHI in testing. Remove all 18 HIPAA identifiers.

Reality check: Most healthcare apps cannot fully test this way. Patient matching, insurance verification, and clinical workflows require realistic data structures.

Path 2: Cloud Testing with Full Controls

Get a signed BAA. Configure encryption. Enable audit logging. Verify device wiping. Document everything.

Reality check: Enterprise tiers cost 5-10x standard pricing. And your binary—with embedded keys and logic—still sits on a server you do not control.

Path 3: Zero-Upload On-Premise Testing

Keep everything on infrastructure you control.

[Your CI/CD] ◄──P2P Encrypted──► [Your Devices]
     │                                 │
     ▼                                 ▼
 Your Logs                        Your Storage

[DeviceLab Cloud] ◄── Signaling only
                      (no PHI, no binaries)

Result:

✅ PHI Exposure: None external
✅ BAA Required: No
✅ Audit Scope: Your existing infrastructure only

With DeviceLab, your binary moves directly from CI/CD to your devices via encrypted P2P. We handle signaling only—no PHI, no binaries, no test data ever touches our servers.

We are the pipe, not the bucket.

The BAA Headache (Solved)

If you use a cloud vendor with PHI, you need a Business Associate Agreement:

Standard cloud plans ($150/month) do not include BAAs
Enterprise plans with BAAs often cost 5-10x more
Negotiating a BAA takes 2-4 weeks minimum

With DeviceLab, you do not need a BAA with us. We never process your data. Your PHI stays on your network, streaming P2P between your devices.

Nothing to negotiate. Nothing to declare. Nothing to audit beyond your existing infrastructure.

Week 1 Action Items

Days 1-2: Inventory

List every system touching test builds
Identify all third-party vendors in your QA pipeline
Document what data type (PHI, de-identified, synthetic) each test uses

Days 3-4: Gap Analysis

Run through the 5-point audit
Identify missing BAAs
Flag unencrypted PHI transmission points

Days 5-7: Remediation

Choose your path: de-identified, compliant cloud, or on-premise
If cloud: initiate BAA process (expect 2-4 weeks)
If on-premise: order hardware
Update your risk analysis to include testing infrastructure

The Bottom Line

You would not host your production database on a public FTP server.

Do not host your test infrastructure on a public device cloud.

OCR is auditing more aggressively than ever. Your testing pipeline is either:

In scope and documented → Compliant
In scope and undocumented → Violation
Out of scope because PHI never leaves your network → Ideal

DeviceLab helps you achieve option 3. No BAA required. No PHI on our servers. No new audit scope.

The safest place for patient data—even mock data—is inside your house.

275 million records breached. $3 million penalties. 22 enforcement actions. Your auditor will ask about your testing pipeline.

Will you have an answer?

Build a Zero-Trust Lab →

Read the Security Architecture →

See the Certified Hardware List →

Disclaimer: This article is for informational purposes only and does not constitute legal advice. You should contact your attorney to obtain advice with respect to any particular issue or problem.

Learn More

Renting Test Devices is Financially Irresponsible. Here's the Math.

Om Narayan — Sun, 28 Dec 2025 23:28:40 +0000

Your finance team would never approve renting laptops at a 300% markup. So why are you doing it with test devices?

Every month, engineering teams wire thousands of dollars to cloud testing providers for infrastructure they could own outright. Not because the cloud is better—but because nobody ran the numbers.

Let's run them.

The Math Your CFO Needs to See

Here is the actual cost breakdown for a standard 10-device lab:

Cost Factor	Cloud (BrowserStack)	DeviceLab (Own Hardware)
Monthly cost	$2,500	$891 (9 paid + 1 free)
Annual cost	$30,000	$10,692
Hardware (one-time)	$0	$779 (Mac Mini + Hub + Cables)
Year 1 Total	$30,000	$11,471
Year 2 Total	$60,000	$22,163
Year 3 Total	$90,000	$32,855

3-Year Savings: $57,145

That is $19,000+ per year saved on just 10 phones. Not a rounding error—enough to hire a junior engineer or upgrade every developer's laptop.

The 100-Device Reality

When you scale this to a full enterprise lab, the "Cloud Tax" becomes indefensible:

Devices	Cloud Cost (3 Years)	DeviceLab Cost (3 Years)	You Keep
10	$90,000	$32,855	$57,145
25	$225,000	$83,940	$141,060
50	$450,000	$167,880	$282,120
100	$900,000	$335,760	$564,240

At 100 devices, you are saving $188,000 per year. That is pure bottom-line profit.

"But We Need Instant Access to Any Device"

No, you don't.

Look at your test logs. 90% of your runs happen on the same 15 devices. The iPhone 15. The Galaxy S24. The Pixel 8. Your core regression suite does not need a Finnish Nokia from 2019.

The smart play is not all-or-nothing. It is the Hybrid Model:

┌─────────────────────────────────┐  ┌──────────────┐
│                                 │  │              │
│     CORE REGRESSION (90%)       │  │ EDGE CASES   │
│                                 │  │    (10%)     │
│         DeviceLab               │  │ BrowserStack │
│                                 │  │              │
│         $0/minute               │  │  $$$/minute  │
│                                 │  │              │
│   • Daily CI/CD                 │  │ • Rare OS    │
│   • Smoke tests                 │  │ • Old devices│
│   • Regression suite            │  │ • One-offs   │
│                                 │  │              │
└─────────────────────────────────┘  └──────────────┘

Keep your cloud subscription for the weird stuff. Run your daily workload on hardware you own.

You are not killing your safety net. You are right-sizing it.

The Hidden Costs Nobody Talks About

The Parallelism Tax

Want to run 10 tests simultaneously? That is 10 parallel slots. At $250 each. Per month.

Your developers are waiting in queues because you are rationing access to save money. That is not a testing problem—it is a procurement problem.

With owned devices, parallelism is free. Run 50 tests on 50 devices. No slots. No queues. No artificial scarcity.

The Latency Problem

Cloud devices sit in Virginia or Frankfurt. Your staging server sits in your VPC.

Every test command travels: Your CI → Cloud Provider → Their Device → Back.

That is 100-500ms of latency per interaction. Multiply by thousands of UI actions. Your test suite is slow because of geography, not code.

Local devices respond in under 50ms. That is not incrementally better—it is a different experience entirely.

The Security Liability

Every test run uploads your unreleased binary to someone else's server. Your staging credentials. Your test data. Your intellectual property.

"But they're SOC2 compliant!"

SOC2 means they have security policies. It does not mean your data is not sitting on shared infrastructure, passing through their network, visible in their logs.

With on-premise devices, your binary never leaves your building. Zero trust is not a marketing term—it is an architecture decision.

What You Actually Need to Buy

Stop overcomplicating this. A production-ready 10-device lab costs less than one month of cloud:

Item	Product	Cost
Host	Mac Mini M4 (16GB)	$599
USB Hub	Sabrent HB-BU10	$60
Cables	Anker PowerLine III (10x)	$120
Total		$779

No server room. No dedicated IT. No ongoing hardware maintenance contracts.

Plug it in. Install DeviceLab. Run tests.

→ See the complete certified hardware list

The Objections (And Why They're Wrong)

"We don't have DevOps bandwidth for hardware"

Modern device labs are software-defined. DeviceLab auto-recovers disconnected devices, monitors battery health, and reboots hung phones. You are not hiring someone to babysit USB cables.

"What about device coverage?"

You are already testing on 15 devices. You will still test on 15 devices. The only difference is who owns them.

Buy refurbished. An iPhone 13 tests iOS apps just fine—and costs $400 instead of $1,200.

"Our enterprise security team won't approve random hardware"

They will approve it faster than they will approve uploading proprietary code to a third-party cloud. On-prem is the security team's preferred answer.

"Scaling is easier in the cloud"

Scaling cloud costs is easy. Just add more zeros to the invoice.

Scaling owned hardware means buying another Mac Mini. Once. Then it is yours forever.

Calculate Your Exact Savings

Every team is different. Here is how to run your own numbers:

Count your current cloud devices (check your invoice)
Multiply by $250/month (conservative cloud cost)
Compare to DeviceLab: $99/device/month + one-time hardware
Calculate 3-year difference

Or use this formula:

Annual Savings = (Devices × $250 × 12) - (Devices × $99 × 12) - Hardware Cost

For 25 devices with $2,800 hardware:

= (25 × $250 × 12) - (25 × $99 × 12) - $2,800
= $75,000 - $29,700 - $2,800
= $42,500 saved in Year 1

Year 2 and beyond? That $2,800 hardware cost disappears. You save $45,300 every year after.

Who Already Made the Switch

Teams running 50+ devices on DeviceLab instead of cloud:

Fintech apps that cannot upload binaries to third-party servers
Healthcare platforms with HIPAA data in test scenarios
Enterprise teams tired of explaining cloud costs to finance
Startups that refused to accept "testing is expensive" as a given

They did not switch because they hate BrowserStack. They switched because the math stopped making sense.

What Happens Next

Option 1: Keep paying the cloud tax. Budget $30,000/year for 10 devices. Explain to finance why testing costs more than the developers writing the tests.

Option 2: Spend $779 on hardware. Pay $99/device/month. Save $18,500+ in Year 1 alone. Redeploy that budget to hiring, features, or literally anything else.

The cloud made sense in 2015 when the alternative was building a data center. It does not make sense in 2025 when the alternative is a Mac Mini under your desk.

Frequently Asked Questions

How much does BrowserStack cost per device?

BrowserStack charges approximately $250-300 per device per month for dedicated real device access, translating to $3,000+ per device annually.

What is a cloud exit in mobile testing?

A cloud exit is the strategic move from renting cloud-based device testing infrastructure to owning your own private device lab, reducing costs by over 60%.

How long does it take to recoup device lab hardware costs?

Most teams break even within 2 months. A 10-device lab saves over $1,600/month compared to cloud rentals, paying off the hardware investment in roughly 45 days.

What hardware do I need to build a 10-device lab?

A Mac Mini M4 ($599), Sabrent HB-BU10 USB hub ($60), and 10 Anker cables ($120). Total: $779 one-time cost.

Is the hybrid model better than full cloud exit?

For many teams, yes. Keep 10% of devices on cloud for rare OS versions and edge cases, move 90% of daily regression to owned hardware. You get cost savings without losing coverage.

Your test infrastructure is either an asset or an expense. Renting makes it an expense. Owning makes it an asset. The math is simple. The decision should be too.

Get Started

Sub-50ms Latency: The Physics of Fast Mobile Automation

Om Narayan — Sun, 28 Dec 2025 23:15:06 +0000

Have you ever watched an Appium test run on a cloud provider?

You see the command in your terminal.
...pause...
...pause...
The button on the screen finally clicks.

That pause isn't your code. It isn't "processing time." It's the speed of light, and it's killing your test stability.

If you're running tests on GitHub Actions targeting a cloud device farm, you're fighting a losing battle against physics.

This is the deep dive into why—and the architectural fix that makes tests run 3x faster.

The Human Perception Threshold

In 1968, IBM researcher Robert Miller established response time thresholds that still guide UX design today. Jakob Nielsen popularized them in 1993, and they remain the gold standard:

Latency	Human Perception
< 100ms	Feels instantaneous—direct manipulation
100-300ms	Noticeable delay, but flow uninterrupted
300-1000ms	System feels sluggish, user waits
> 1000ms	Context switch—user loses focus

The 1982 Doherty Threshold research went further: productivity soars when response time drops below 400ms, and continues improving down to 50ms.

For interactive tasks—dragging, swiping, debugging gestures—recent research shows users can perceive latencies as low as 16-33ms in direct touch interactions. The 100ms "threshold" was always a ceiling, not a target.

Why this matters for testing: When you're debugging a swipe-to-delete gesture, you need to feel the physics. A 200ms delay between your swipe and the animation makes it nearly impossible to diagnose timing issues. You're debugging blind.

The Cloud Latency Stack

When you interact with a cloud device farm, your input travels through:

Your Input → Local Network → ISP → Cloud Data Center → Server Processing → Device → Screen Capture → Video Encoding → Return Path → Your Screen

Round-trip: 150-400ms (typical)

Each hop adds latency:

Component	Added Latency
Geographic distance (speed of light)	30-100ms
ISP routing and peering	10-50ms
Data center network	5-20ms
Server processing	10-30ms
Video encoding	10-30ms
Video decoding (your browser)	5-15ms
Display rendering	5-15ms

Best case: A developer in Virginia testing on Virginia-based cloud devices might see 80-120ms.

Typical case: A developer in Bangalore testing on US-based cloud infrastructure sees 200-350ms.

Worst case: Complex routing, congested networks, or transcoding issues can push latency to 400ms+.

This is physics. No amount of engineering can make light travel faster through fiber.

The Protocol Problem: Why WebDriver is "Chatty"

To understand the bottleneck, you have to look at the WebDriver Protocol.

Appium doesn't send your entire test script to the device at once. It sends individual HTTP commands for every single action. Synchronous. Blocking. One at a time.

A typical "Login" flow looks like this:

1. POST /element (Find 'username')  → Request ✈️ Cloud → Response
2. POST /element/value (Type text)  → Request ✈️ Cloud → Response  
3. POST /element (Find 'password')  → Request ✈️ Cloud → Response
4. POST /element/value (Type text)  → Request ✈️ Cloud → Response
5. POST /element (Find 'submit')    → Request ✈️ Cloud → Response
6. POST /element/click (Click it)   → Request ✈️ Cloud → Response

Six commands. Six round-trips. If your runner is in Virginia (Azure US-East) and the device cloud is in California (US-West), that's ~80ms round-trip latency per command.

The Math of Slowness:

Test Complexity	Commands	RTT @ 100ms	RTT @ 20ms (Local)
Simple login flow	50	5 seconds	1 second
Standard regression	500	50 seconds	10 seconds
Full E2E suite	2,000	3.3 minutes	40 seconds

This is why your local test runs in 2 minutes, but the cloud test takes 8 minutes. 60% of that time is pure network waiting.

Over thousands of test runs per month, this compounds into hours of wasted compute time—and slower feedback loops for your team.

The 3 Architectures: Good, Better, Best

Most teams assume "CI/CD" just means "GitHub Actions Cloud." But where you put the Runner—the computer executing the test script—defines your speed.

Architecture A: The "Double Hop" (Legacy Cloud) 🔴

Setup: GitHub-Hosted Runner (Azure) → BrowserStack/Sauce Labs (AWS/GCP)

GitHub Actions (Azure US-East)
        ↓ ~50ms
BrowserStack Hub (AWS US-West)
        ↓ ~30ms
Device Data Center
        ↓ ~20ms
Actual Device
        ↓
[Response travels back: +100ms]

Latency per command: 150-300ms

Your command leaves Microsoft's cloud, travels across the open internet to the vendor's cloud, hits the device, and travels all the way back. Packet loss and jitter cause frequent "flaky" failures.

The hidden cost: You can't control which Azure region GitHub picks. One day your runner is in US-East, the next it's in US-West. Latency variance causes tests to pass one run and timeout the next.

Architecture B: The "Hybrid" (GitHub Cloud + DeviceLab) 🟡

Setup: GitHub-Hosted Runner (Azure) → DeviceLab Tunnel → Your Office

GitHub Actions (Azure US-East)
        ↓ ~80ms (internet tunnel)
Your Mac Mini (Office/Home)
        ↓ ~5ms (USB)
Your Devices

Latency per command: 80-150ms

Your test script runs in GitHub's cloud. Commands travel over the internet (via DeviceLab's secure tunnel) to your Mac Mini, which executes them on the phone.

The verdict: It works, and it's convenient because you don't manage runners. But you still pay the "Tunnel Tax"—physics dictates that a command from Azure to your office takes time.

Still faster than Architecture A (no shared resource contention, no vendor queue times), but not instant.

Architecture C: The "Edge Runner" (God Mode) ⚡🟢

Setup: Self-Hosted GitHub Runner on the same Mac Mini hosting the devices

GitHub sends "Start Job" signal
        ↓
Mac Mini picks up job
        ↓
Test runs on localhost:4723
        ↓ ~2ms (USB cable)
iPhone/Android Device

Latency per command: <5ms

Network hops: 0
Stability: Near-perfect

You install the GitHub Actions Runner agent directly on your Mac Mini. GitHub triggers the job, but execution happens locally. The Appium command travels over a USB cable, not the internet.

┌─────────────────────────────────────┐
│  YOUR MAC MINI (Device Node)        │
│                                     │
│  [GitHub Runner] ──► [Appium] ──────┼──► USB ──► iPhone 15
│                                     │
└─────────────────────────────────────┘

Round-trip latency: <5ms

The math changes completely:

Architecture	Latency/Cmd	500-Cmd Test Overhead	Flakiness
A: Double Hop	200ms	100 seconds	High
B: Hybrid	100ms	50 seconds	Medium
C: Edge Runner	5ms	2.5 seconds	Near-zero

Architecture C is 40x faster on network overhead alone.

Screen Streaming: WebRTC vs MJPEG

The architecture diagrams above focus on commands. But there's another latency source: screen video.

Legacy cloud farms stream device screens via MJPEG—essentially a series of JPEG images over HTTP. This adds 500ms-2 seconds of video latency on top of command latency.

DeviceLab uses WebRTC (the same protocol powering Zoom/Google Meet):

Protocol	Latency	Why
MJPEG	500ms-2s	TCP, buffered, server-transcoded
WebRTC	<50ms	UDP, P2P, no transcoding

For manual testing, this matters enormously. Debugging a gesture on a 2-second delayed video is impossible. With WebRTC, the device screen feels like it's plugged into your monitor.

What Sub-50ms Testing Feels Like

At sub-50ms latency:

Manual debugging: You swipe, and the screen responds. The device feels like it's in your hand. You can debug gesture physics, animation timing, and scroll behavior by feel, not by inference.

Interactive sessions: App review, QA walkthroughs, and stakeholder demos run smoothly. No awkward pauses explaining "there's a bit of lag."

Automated tests: Appium commands execute immediately. Test suites run 20-40% faster just from eliminating network overhead.

CI pipelines: Faster test execution means faster feedback. Developers find out about failures in minutes, not hours.

The difference isn't subtle. Teams that switch from cloud to local consistently report that testing "feels different"—like upgrading from a video call to being in the same room.

The Latency Measurement

Don't take claims at face value. Measure it yourself.

For manual testing, use this rough approach:

Start a screen recording on your computer
Tap a button that triggers a visual change
Frame-by-frame, count the time between your tap and the screen update
30fps video = 33ms per frame; 60fps = 16ms per frame

For automated testing, instrument your test code:

import time

start = time.time()
driver.find_element(by, value).click()
end = time.time()

print(f"Command latency: {(end - start) * 1000:.0f}ms")

Run this on cloud infrastructure and local infrastructure. The difference will be stark.

What you'll likely see:

Setup	Measured Latency
Cloud device farm (cross-region)	180-350ms
Cloud device farm (same region)	100-180ms
DeviceLab P2P (cross-network)	50-100ms
DeviceLab P2P (same network)	20-50ms
USB-connected device	10-30ms

DeviceLab on the same network approaches USB-connected performance—without the cable.

When Cloud Latency Is Acceptable

Let's be fair: not every use case requires sub-50ms latency.

Cloud latency is fine for:

Screenshot comparison tests (no interaction timing)
Smoke tests that verify basic flows
Device coverage testing (checking if it works on Device X)
Batch execution where speed isn't critical

Cloud latency hurts for:

Debugging gesture interactions
Testing animations and transitions
Interactive QA sessions
Manual exploratory testing
CI pipelines where feedback speed matters
Any test where you're diagnosing why something fails

If your testing is purely "does it crash?"—cloud latency is fine. If you're trying to understand behavior, you need responsive devices.

Setting Up the Edge Runner

Moving to the Edge Runner architecture doesn't require rearchitecting your test suite.

Step 1: Hardware

A Mac Mini M2 and USB hub can host 10-15 devices. Total cost: ~$800. (See the Certified Hardware List)

Step 2: Install Self-Hosted Runner

# Download and configure GitHub Actions runner
./config.sh --url https://github.com/your-org/your-repo \
            --token YOUR_RUNNER_TOKEN
./run.sh

Step 3: Install DeviceLab Node

DeviceLab's agent runs alongside the GitHub runner. Your devices appear in a browser dashboard, accessible from anywhere—for interactive sessions while CI runs locally.

Step 4: Update Workflow

jobs:
  test:
    runs-on: self-hosted  # ← Changed from 'ubuntu-latest'
    steps:
      - uses: actions/checkout@v4
      - run: npm test  # Appium hits localhost:4723

Same Appium scripts. Same Maestro flows. Same XCUITest suites.

You're changing where they execute, not how they're written.

Frequently Asked Questions

Why is Appium slow on BrowserStack/Sauce Labs?

The WebDriver protocol is synchronous and chatty. A single test involves hundreds of HTTP requests. When each request must travel 3,000 miles to a cloud server and back, network latency adds minutes to execution time.

What is the fastest GitHub Actions setup for mobile testing?

The "Edge Runner" architecture. Install a Self-Hosted GitHub Runner directly on the Mac Mini hosting the phones. This reduces network latency to <5ms per command.

Can I use GitHub-Hosted Runners with DeviceLab?

Yes. This is the "Hybrid" setup. It's slower than Edge Runners due to the "Tunnel Tax" (commands travel from Azure to your office), but still faster than cloud farms because you eliminate shared-resource contention and queue times.

What is the latency difference between MJPEG and WebRTC?

Legacy cloud farms stream video via MJPEG (500ms-2s latency). DeviceLab uses WebRTC P2P, delivering <50ms screen latency—which feels like the device is plugged into your monitor.

How much faster is the Edge Runner architecture?

For a 500-command test: Cloud Double-Hop adds 100 seconds of network overhead. Edge Runner adds 2.5 seconds. That's 40x less waiting.

The Bottom Line

Cloud device testing adds 150-400ms of latency per command. That's not a bug—it's physics. The WebDriver protocol is chatty, and every HTTP request must travel thousands of miles.

The three architectures:

Double Hop (GitHub Cloud → Cloud Farm): 200ms/command, high flakiness
Hybrid (GitHub Cloud → DeviceLab Tunnel): 100ms/command, medium flakiness
Edge Runner (Self-Hosted → Local Devices): 5ms/command, near-zero flakiness

The Edge Runner is 40x faster on network overhead. For a 500-command test suite, that's the difference between 100 seconds of waiting and 2.5 seconds.

If your tests are slow and flaky on cloud infrastructure, you're not doing anything wrong. You're just fighting the speed of light.

Stop fighting. Move the runner to the edge.

Debugging gestures through 300ms of lag is like conducting surgery wearing oven mitts. Your tools should feel like extensions of your hands, not obstacles between you and your work.

Disclaimer: Latency measurements vary based on network conditions, geographic location, and device configuration. The figures in this article represent typical ranges observed in real-world testing scenarios.

Set Up an Edge Runner

Binary Sovereignty: Stop Uploading Your Unreleased App to Strangers

Om Narayan — Sun, 28 Dec 2025 23:08:23 +0000

Here is a contradiction I see every day in enterprise engineering:

Security Team: Locks down the staging environment. Enforces strict VPNs. Rotates API keys weekly. Mandates 2FA for every repo. Scrutinizes every npm package.

DevOps Team: Takes the compiled, unreleased binary—containing all those keys, logic, and intellectual property—and uploads it to a public cloud server owned by a third-party vendor.

We call this "Cloud Testing." In any other context, we would call it a data leak.

The "Secure Tunnel" Fallacy

Most teams justify this workflow because they use a "Secure Tunnel" (like BrowserStack Local or Sauce Connect). They believe that because the traffic between the device and their staging server is encrypted, they are safe.

They are missing the elephant in the room.

The tunnel protects the network requests. It does not protect the Application Binary.

To run an Appium or Espresso test on a cloud device, you must first upload your .apk or .ipa file to the vendor's storage. That binary sits on their servers. It is installed on their devices.

What is actually in that binary?

Even with ProGuard or R8 obfuscation, your binary is a treasure map of your business. As the OWASP Mobile Security Testing Guide warns, reverse engineering is trivial:

Staging API Endpoints: Often hardcoded or in config files
Feature Flags: Unreleased features your competitors would kill to see
Third-Party Keys: API keys for analytics, crash reporting, and internal tools
Business Logic: The actual algorithms that make your product unique

You are handing the keys to the castle to a vendor, simply because you do not want to manage a USB hub.

"But They Are SOC2 Compliant!"

I hear this defense constantly. "The vendor has a SOC2 Type II report! They are secure."

Let's be clear about what SOC2 means.

SOC2 means the vendor has processes in place. It means they have promised not to look at your data. It means they have background checks for their employees.

SOC2 is a legal shield, not a physical barrier. For a deeper analysis of cloud provider security risks, see our cloud device lab security guide.

It does not change the physics of the situation: Your proprietary code is sitting on a hard drive you do not own, accessible by admins you do not know, in a data center you cannot visit.

If that vendor gets breached (and vendors get breached every week), your unreleased app is compromised. If a rogue employee at the vendor wants to inspect your binary, the only thing stopping them is a policy document.

Ask your cloud provider these questions:

How long is my binary retained after a test run?
Who at your company can access uploaded binaries?
Are devices wiped between customers, or just between sessions?
Where exactly are my test logs stored?
Can you provide a data flow diagram for my uploads?

Most cannot answer all five. That should concern you.

The Credentials Problem

Your app does not run in isolation. Tests need to authenticate.

Every test run includes:

Staging API keys
Test user credentials
Internal endpoint URLs
OAuth tokens
Push notification certificates

These appear in:

Environment variables (logged)
CI/CD environment variables (often dumped in logs on failure)
Network traffic (captured)
Test output (stored)
Crash reports (uploaded)

Cloud providers capture this data for debugging features—the same features you use to figure out why a test failed. That means your staging credentials exist in their logging infrastructure.

"We mask sensitive data in logs."

Masking catches known patterns. It does not catch custom credential formats, hardcoded tokens in debug builds, internal URLs that reveal architecture, or error messages that leak implementation details.

Security theater looks like security. It is not the same thing.

The Alternative: Zero Trust Architecture

True security is not about trusting a vendor's "pinky promise." It is about architecture that makes trust unnecessary.

We built DeviceLab on a Zero Trust / Peer-to-Peer (P2P) model.

How It Works

When you run a test with DeviceLab, the architecture is fundamentally different:

Your CI/CD Runner (GitHub Actions, Jenkins) has the binary
Your Device Node (the Mac Mini under your desk) has the phone
DeviceLab establishes a direct, encrypted P2P pipe between them

The command adb install app.apk happens inside your network. The binary moves from your build server to your test device.

[YOUR INFRASTRUCTURE]                        [DEVICELAB CLOUD]

┌─────────────┐                              ┌─────────────┐
│  CI/CD      │                              │             │
│  Runner     │ ◄─────── (Signaling) ──────► │ Orchestrator│
│ (Has Binary)│                              │ (No Binary) │
└──────┬──────┘                              └─────────────┘
       │
       │  ◄────── (P2P Encrypted Pipe) ──────►
       │          Binary Transfer
       ▼
┌─────────────┐
│ Device Node │
│ (Mac Mini)  │
└─────────────┘

It never touches DeviceLab's cloud.

We could not steal your app if we wanted to. We do not have the storage for it. We do not have the access rights. We simply broker the handshake, then get out of the way.

Binary Sovereignty for Regulated Industries

For most startups, this is a matter of IP protection. For FinTech and HealthTech, it is often a matter of compliance.

The Banking Scenario

If you are building a banking app, your test build often points to a "Staging" environment that mimics production. While you scrub customer data, the mechanisms of your fraud detection and transaction logic are in that binary.

Uploading that to a public cloud is a massive unnecessary risk.

Many fintech security teams prohibit cloud device testing entirely. The compliance exposure under PCI-DSS, SOX, and state banking regulations is simply not worth it. Learn more about why fintech teams can't use shared device clouds.

The HIPAA Scenario

Healthcare apps often contain hardcoded test user credentials for E2E scenarios. Even if these are "fake" patients, the structure of the data and the API calls reveal how you handle PHI (Protected Health Information).

Keeping this traffic and data strictly on-premise is the only way to ensure 100% compliance. See our complete HIPAA mobile QA checklist for detailed requirements.

Can you prove under audit that no real patient data ever leaked into a test build? With on-premise testing, the answer is simple: the data never left your network.

The Competitive Intelligence Scenario

Your unreleased features are competitive intelligence. A test build reveals:

Upcoming features (UI strings, new screens)
Architecture decisions (API endpoints, service names)
Performance characteristics (what you are optimizing)
Bug patterns (what breaks under test)

This information has value. Uploading it to shared infrastructure is a choice.

The "Crown Jewels" Test

Ask your CISO this simple question:

"Would you be comfortable emailing our unreleased source code to a vendor?"

The answer is always "No."

So why are we comfortable uploading the compiled version of that code to the same vendor?

The binary is the source code. A decompiler turns it back into readable Java or Swift in seconds.

Making the Switch

Moving to zero-trust testing is not complicated. This is why enterprises use private device labs.

Hardware: Mac Mini + USB hub + your devices (~$1,500 for 10 devices). See our certified hardware list for tested configurations.
Software: DeviceLab connects your devices to your CI
Network: Devices on your network, or P2P tunnels for remote access
Process: Same Appium/Maestro/XCUITest scripts, different execution target

Your test code does not change. Your CI configuration changes slightly. Your security posture changes completely.

Frequently Asked Questions

Is BrowserStack secure for banking apps?

Public clouds like BrowserStack are SOC2 compliant, but they require you to upload your binary to their servers. For strict FinTech compliance, keeping binaries on-premise via DeviceLab is safer.

What is Binary Sovereignty?

Binary Sovereignty is the practice of keeping your application compiled code (APK/IPA) strictly within your own infrastructure, never uploading it to third-party vendors for testing.

Does DeviceLab store my app?

No. DeviceLab uses a peer-to-peer architecture. Your binary streams directly from your CI/CD to your device nodes. We physically cannot access or store your application.

What data is exposed during cloud device testing?

Your unreleased binary, staging API credentials, test user data, OAuth tokens, push notification certificates, and debug logs with internal endpoints—all uploaded to third-party infrastructure.

Can someone decompile my uploaded APK/IPA?

Yes. A decompiler turns your binary back into readable Java or Swift in seconds. Your binary IS your source code—just in a different format.

The Bottom Line

Every cloud test is a binary upload to third-party infrastructure. That upload includes your unreleased app, your credentials, and your business logic.

You can accept that risk. Many companies do.

Or you can eliminate it entirely by testing on devices you own, connected through infrastructure you control.

It is time to take your binaries back.

Your competitors would love to see your unreleased features. Your auditors want to know where your test data lives. One of these groups can access cloud testing infrastructure. Think about which one.

Build a Zero-Trust Lab in 5 Minutes

Maestro Flakiness: Source Code Analysis

Om Narayan — Thu, 25 Dec 2025 07:36:17 +0000

Maestro markets itself as a test framework that "embraces the instability of mobile applications." But what does that actually mean in code? I dug into the source to find out.

The Marketing Promise

From Maestro's documentation:

"UI elements will not always be where you expect them, screen tap will not always go through, etc. Maestro embraces the instability of mobile applications and devices and tries to counter it."

"No need to pepper your tests with sleep() calls. Maestro knows that it might take time to load the content and automatically waits for it (but no longer than required)."

Sounds great. Let's see what the code actually does.

1. Element Finding: The Hardcoded 17-Second Timeout

When you write tapOn: "Login", Maestro doesn't look once and fail. It polls continuously. But for how long?

Source: Orchestra.kt

class Orchestra(
    private val maestro: Maestro,
    private val lookupTimeoutMs: Long = 17000L,        // Hardcoded: 17 seconds
    private val optionalLookupTimeoutMs: Long = 7000L, // Hardcoded: 7 seconds for optional
)

What this means:

Every element lookup waits up to 17 seconds by default
Optional elements wait 7 seconds
You cannot change this per-command

Want to wait 30 seconds for a slow API response? Too bad. Want to fail-fast in 3 seconds for performance testing? Also no.

2. The Polling Mechanism: Simple but Rigid

Source: MaestroTimer.kt

fun <T> withTimeout(timeoutMs: Long, block: () -> T?): T? {
    val endTime = System.currentTimeMillis() + timeoutMs

    do {
        val result = block()
        if (result != null) {
            return result
        }
    } while (System.currentTimeMillis() < endTime)

    return null
}

This is a tight polling loop with no configurable delay between iterations. It just hammers the view hierarchy until it finds the element or times out.

Compare to Appium's FluentWait where you can set custom timeout, custom polling interval, and exceptions to ignore.

3. Tap Retries: Only 2 Attempts, Non-Configurable

Maestro has a clever feature: if a tap doesn't change the UI, it retries. But how many times?

Source: Maestro.kt

private fun getNumberOfRetries(retryIfNoChange: Boolean): Int {
    return if (retryIfNoChange) 2 else 1  // That's it. Just 2.
}

Two retries. Hardcoded. Not configurable.

And the explicit retry command in YAML? Also capped:

Source: Orchestra.kt

private const val MAX_RETRIES_ALLOWED = 3

So even if you write retry: 10 in your YAML, you get 3 maximum.

4. Screenshot Diff Threshold: Magic Number 0.5%

When Maestro decides whether a tap "worked," it compares screenshots pixel-by-pixel:

Source: Maestro.kt

private const val SCREENSHOT_DIFF_THRESHOLD = 0.005  // 0.5% pixel difference

If less than 0.5% of pixels changed, Maestro assumes the tap failed and retries.

Problems:

A small spinner animation might trigger false "success"
A full-screen color change of 0.4% would be considered "no change"
You can't adjust this threshold

5. Wait For App To Settle: Fixed 200ms Polling

After interactions, Maestro waits for the UI to stabilize:

Source: ScreenshotUtils.kt

repeat(10) {  // Poll up to 10 times
    val hierarchyAfter = viewHierarchy(driver)
    if (latestHierarchy == hierarchyAfter) {
        val isLoading = latestHierarchy.root.attributes
            .getOrDefault("is-loading", "false").toBoolean()
        if (!isLoading) {
            return hierarchyAfter
        }
    }
    latestHierarchy = hierarchyAfter

    MaestroTimer.sleep(MaestroTimer.Reason.WAIT_TO_SETTLE, 200)  // Fixed 200ms
}

Hardcoded values:

10 iterations maximum
200ms between polls
Total maximum wait: 2 seconds

Fast app? You're wasting 200ms per poll. Slow app? 2 seconds might not be enough.

6. waitUntilVisible: 10 Seconds, Take It or Leave It

Source: Maestro.kt

private fun waitUntilVisible(element: UiElement): ViewHierarchy {
    var hierarchy = ViewHierarchy(TreeNode())
    repeat(10) {  // 10 attempts
        hierarchy = viewHierarchy()
        if (!hierarchy.isVisible(element.treeNode)) {
            LOGGER.info("Element is not visible yet. Waiting.")
            MaestroTimer.sleep(MaestroTimer.Reason.WAIT_UNTIL_VISIBLE, 1000)  // 1 second
        } else {
            LOGGER.info("Element became visible.")
            return hierarchy
        }
    }
    return hierarchy
}

10 polls × 1 second = 10 seconds maximum. Not configurable.

7. Platform Differences: Android vs iOS

Maestro handles settling differently per platform:

Android (AndroidDriver.kt):

// Checks if window is still updating
val windowUpdating = blockingStubWithTimeout.isWindowUpdating(...)

iOS (IOSDriver.kt):

// Uses screenshot comparison to detect animation end
val didFinishOnTime = waitUntilScreenIsStatic(SCREEN_SETTLE_TIMEOUT_MS)

Different approaches, different reliability characteristics, same lack of configurability.

Summary: The Hardcoded Reality

Parameter	Value	Configurable?
Element lookup timeout	17,000ms	❌ No
Optional element timeout	7,000ms	❌ No
Tap retry attempts	2	❌ No
Max retry command	3	❌ No
Screenshot diff threshold	0.5%	❌ No
Settle polling interval	200ms	❌ No
Settle max iterations	10	❌ No
waitUntilVisible timeout	10,000ms	❌ No

The Verdict

Maestro's "built-in flakiness handling" is real—it does more than raw XCUITest or Espresso. But it's a one-size-fits-all solution with hardcoded values.

The code is clean and the approach is sound. The problem is the lack of escape hatches. When the defaults don't work for your app, you're stuck.

This isn't necessarily bad—it's a trade-off for simplicity. But the marketing implies more intelligence and adaptability than the code delivers.

Key Source Files

If you want to explore further:

Orchestra.kt — Command execution and timeouts
Maestro.kt — Tap logic and retries
MaestroTimer.kt — Polling primitives
ScreenshotUtils.kt — Screenshot comparison
Commands.kt — Command definitions

We're Not Just Pointing Out Problems

We love Maestro's YAML syntax—it's the best thing to happen to mobile test automation in years. Simple, readable, version-control friendly.

But the execution engine has real limitations. Hardcoded timeouts. No configurability. Platform inconsistencies.

So we're building something to fix it.

An open-source engine that runs Maestro YAML tests on Appium's battle-tested infrastructure. Configurable timeouts. Real device support. No magic numbers.

Watch this space.

Learn More

BrowserStack Alternative: Your Own Devices

Om Narayan — Thu, 25 Dec 2025 07:30:07 +0000

TL;DR: BrowserStack is great for quick access to thousands of devices, but the costs add up fast, latency can hurt interactive testing, and some teams can't upload their app binaries to third-party servers. If you already own devices or need a private testing cloud, there's another approach: turn your own hardware into a device farm with P2P streaming. No rentals, no tunnels, no third-party uploads.

The Real Cost of Cloud Device Testing

BrowserStack has become the default choice for cross-device testing, and for good reason. Access to 20,000+ real devices without buying hardware sounds compelling. But let's look at what teams actually pay.

According to Vendr's internal transaction data, the average BrowserStack contract runs about $32,433 per year, with enterprise deals reaching into six figures. Even smaller teams face significant costs:

Plan	Monthly Cost	Annual Cost
App Automate (1 parallel)	$199-$249	~$2,400-$3,000
5 parallels	~$1,000	~$12,000
10 parallels	~$2,000+	~$24,000+

For a team running CI tests on every pull request, those parallel slots fill up fast. And when they do, tests queue—or fail.

What Users Actually Say

Capterra's analysis of BrowserStack reviews (Value for Money rating: 4.2/5, below the 4.5 category average) surfaces recurring themes:

"Browserstack is very convenient and powerful, but the price is prohibitive for me as I use it sparingly."

"It can be costly to use BrowserStack, particularly for small businesses or lone developers."

"The cost can quickly escalate, especially when additional licenses or concurrent sessions are needed."

The pattern is clear: BrowserStack works well, but teams consistently cite pricing as a barrier—especially for startups, smaller QA teams, and high-frequency testing scenarios.

Three Problems Cloud Testing Creates

Beyond cost, cloud-based device farms introduce structural challenges that no pricing plan can fix.

1. Latency You Can't Engineer Away

When you interact with a device in BrowserStack's data center, every tap travels from your machine to their servers and back. BrowserStack's own documentation acknowledges this:

"This additional routing introduces multiple network hops, which increase page load and interaction response times."

For automated tests, this manifests as slower execution times. For manual testing, the lag makes interactive debugging frustrating—especially for gesture-heavy apps or games requiring responsive input.

The physics are simple: a device in Mumbai can't respond as quickly to a tester in Bangalore if every interaction routes through a US data center. BrowserStack has added regional hubs, but network latency remains fundamentally bound by distance.

2. Local Testing Requires Tunnels

Testing your localhost:3000 development server on BrowserStack requires their Local binary—a tunnel that routes traffic through your machine. This works, but introduces complexity:

Port conflicts (45690/45691 commonly collide with other services)
Proxy and firewall configuration (corporate networks often block tunnel connections)
VPN incompatibility (many enterprise VPNs break tunnel functionality)
Binary version mismatches (keeping the local binary updated across team machines)

Their support documentation includes extensive troubleshooting guides for these exact issues. The tunnel approach is a necessary workaround for cloud architecture—not a feature.

3. Your App Binary Lives on Their Servers

To test your iOS or Android app, you upload the .ipa or .apk to BrowserStack's infrastructure. For many teams, this is fine. For others, it's a dealbreaker.

Who can't use cloud device farms:

Banking and financial apps — CISOs often prohibit uploading app binaries to third-party servers
Healthcare apps — HIPAA compliance may restrict where PHI-handling code can reside
Government contractors — Security clearance requirements may mandate on-premise testing
Pre-release apps — Competitive sensitivity around unreleased features
Apps with embedded secrets — API keys, certificates, or proprietary algorithms in the binary

BrowserStack offers SOC 2 compliance and security certifications, which satisfy many enterprise requirements. But compliance frameworks differ, and some security policies simply prohibit third-party code hosting regardless of certifications.

The Alternative: Build Your Own Device Farm

What if instead of renting device time, you used hardware you already have?

Most mobile development teams accumulate devices: old test phones, devices from bug reports, the drawer of "we might need this model someday" hardware. These devices typically sit unused between manual test sessions.

DeviceLab turns your existing Android and iOS devices into a private testing cloud.

The core idea is simple: instead of uploading your app to a data center, stream the device screen to your browser while the device stays on your desk, in your office, or at a teammate's home.

How It Works

DeviceLab uses peer-to-peer WebRTC connections—the same technology powering video calls—to stream device screens with sub-50ms latency. The architecture is fundamentally different from cloud device farms:

Aspect	Cloud Device Farm	DeviceLab P2P
Device location	Data center (1000s of miles away)	Your desk/office
Interaction latency	200-500ms+	Sub-50ms
App binary	Uploaded to third-party servers	Stays on your device
Localhost testing	Requires tunnel binary	Just works (same network)
Device availability	Shared pool, may queue	Dedicated to your team
Hardware cost	$0 (rental model)	One-time purchase
Monthly cost	Per-minute or per-parallel	Per-device flat rate

Zero-Trust Architecture

With DeviceLab, your app binary never leaves your network. The P2P connection streams video frames from device to browser—the actual .ipa or .apk stays installed locally on hardware you control.

For compliance-constrained teams, this eliminates the third-party upload question entirely. Your CISO doesn't need to evaluate another vendor's security posture because your code never touches their servers.

No Tunnels for Localhost

Because your device and development machine are on the same network (or connected via WebRTC), testing localhost:3000 requires no configuration. The device can reach your local server directly, just like a USB-connected device would.

No binary to install. No ports to configure. No firewall rules to add.

Cost Comparison: Rental vs. Ownership

Let's model a realistic scenario: a team needing 10-device concurrency for CI runs.

Cloud Rental (BrowserStack)

10 parallel slots on App Automate: ~$24,000/year
Additional costs for Live testing, Percy, etc.
Costs scale linearly with usage

Own Devices (DeviceLab)

10 used/refurbished phones: ~$2,000-$4,000 one-time
DeviceLab subscription: $99/device/month = $990/month = $11,880/year
Year 1 total: ~$14,000-$16,000
Year 2+ total: ~$12,000

The math shifts further in ownership's favor as you scale:

Devices	BrowserStack (Annual)	DeviceLab (Annual after Y1)
5	~$12,000	~$6,000
10	~$24,000	~$12,000
20	~$48,000+	~$24,000
50	Custom pricing	~$60,000

And with owned devices, you get unlimited test runs. There's no metering, no queue, no waiting for a device to become available.

When Cloud Testing Still Makes Sense

To be fair, BrowserStack and similar services solve real problems that DeviceLab doesn't address:

Choose cloud device farms when:

You need access to hundreds of device models you don't own
One-off testing on obscure devices (that specific Samsung tablet from 2019)
You have no physical devices and don't want to buy any
Your security policies allow third-party binary uploads
You need features like network simulation across global regions

Choose own-device testing when:

You already have devices sitting unused
You run high-frequency CI tests (cost scales better)
Your security team prohibits third-party app uploads
Latency matters (games, gesture-heavy apps, interactive debugging)
You need guaranteed device availability (no queuing)
Your team is distributed and devices are in multiple locations

The approaches can also complement each other. Use BrowserStack for broad compatibility checks across 50 device models. Use DeviceLab for your CI pipeline's core test suite on the 5-10 devices that matter most.

Getting Started

If you're exploring alternatives to cloud device testing, here's a practical path:

1. Audit Your Existing Hardware

Most teams have more devices than they realize. Check desk drawers, old test stations, and that box in the storage closet. Functional devices from the last 3-4 years are usually sufficient for testing.

2. Calculate Your Current Costs

What are you actually paying for cloud device testing? Include:

Base subscription costs
Overage charges from parallel limits
Time spent waiting for device availability
Engineering time troubleshooting tunnel issues

3. Run a Pilot

Start small. Connect 2-3 devices to DeviceLab and run your existing test suite. Compare:

Execution time (latency impact)
Reliability (flakiness rates)
Developer experience (interactive testing feel)

4. Evaluate Security Fit

If data privacy is a driver, involve your security team early. The zero-trust architecture may simplify their review compared to evaluating a cloud vendor's entire infrastructure.

Framework Support

Framework	Android	iOS
Appium	✅	✅
Maestro	✅	✅ (only platform with real device support)
Espresso	✅	—
XCUITest	—	✅
WebDriverIO	✅	✅
Manual Testing	✅	✅

FAQ

Can I use DeviceLab with Appium?

Yes. DeviceLab supports Appium, Espresso, WebDriverIO, XCUITest, and Maestro. Your existing test scripts work without modification.

What about Maestro?

DeviceLab supports Maestro on both Android and iOS—including real iOS devices, which no other platform currently offers.

Do I need devices connected to my machine?

No. Devices can be anywhere—connected to any machine running the DeviceLab agent. Team members can contribute devices from home offices, and you access them all through your browser.

What happens if a device goes offline?

The dashboard shows device status in real-time. Devices automatically reconnect when they come back online. For CI, you can configure test distribution to skip unavailable devices.

Is there a free tier?

We offer a free trial to connect your first device and run tests. See devicelab.dev for current plans.

Summary

BrowserStack built the market for cloud device testing, and their product works well for many teams. But the rental model has inherent tradeoffs: recurring costs that scale with usage, latency from remote devices, tunnel complexity for local testing, and binary uploads that some security policies prohibit.

If you own devices—or are willing to buy them once—there's now an alternative. DeviceLab's P2P architecture delivers low-latency testing on hardware you control, with no third-party binary uploads and no tunnels for localhost.

The question isn't whether cloud device testing is bad. It's whether it's the right model for your team's constraints.

Try DeviceLab Free

Migrate BrowserStack to DeviceLab: Appium

Om Narayan — Thu, 25 Dec 2025 07:21:17 +0000

Your Appium tests work. You've spent months building a reliable test suite. It runs against local devices without issues.

Then you moved to BrowserStack.

Suddenly you're dealing with browserstack.yml, bstack:options, app upload APIs, capability generators, and vendor-specific configurations. Your clean test code now has BrowserStack scattered throughout.

Here's how to undo all of that.

What BrowserStack Made You Do

Option A: The SDK Approach

BrowserStack's "recommended" approach requires a browserstack.yml file:

# browserstack.yml
userName: YOUR_USERNAME
accessKey: YOUR_ACCESS_KEY
framework: testng
app: bs://j3c874f21852ba57957a3fdc33f47514288c4ba4

platforms:
  - platformName: android
    deviceName: Samsung Galaxy S23
    platformVersion: '13.0'
  - platformName: android
    deviceName: Google Pixel 8
    platformVersion: '14.0'

parallelsPerPlatform: 2
browserstackLocal: true

buildName: Regression Suite
projectName: MyApp Android

debug: true
networkLogs: true
deviceLogs: true
appiumLogs: true
video: true

browserStackLocalOptions:
  forcelocal: true
  localIdentifier: randomstring

Plus their SDK installed in your project:

<!-- pom.xml -->
<dependency>
    <groupId>com.browserstack</groupId>
    <artifactId>browserstack-java-sdk</artifactId>
    <version>1.13.3</version>
</dependency>

Plus environment variables:

export BROWSERSTACK_USERNAME="your_username"
export BROWSERSTACK_ACCESS_KEY="your_access_key"

Option B: Direct Capabilities

If you're not using the SDK, your test code looks like this:

public class BrowserStackTest {
    private AndroidDriver driver;

    @BeforeMethod
    public void setUp() throws Exception {
        DesiredCapabilities capabilities = new DesiredCapabilities();

        // Standard Appium capabilities
        capabilities.setCapability("platformName", "android");
        capabilities.setCapability("appium:platformVersion", "13.0");
        capabilities.setCapability("appium:deviceName", "Samsung Galaxy S23");
        capabilities.setCapability("appium:automationName", "UiAutomator2");

        // BrowserStack-specific capabilities
        HashMap<String, Object> browserstackOptions = new HashMap<>();
        browserstackOptions.put("userName", "YOUR_USERNAME");
        browserstackOptions.put("accessKey", "YOUR_ACCESS_KEY");
        browserstackOptions.put("appiumVersion", "2.4.1");
        browserstackOptions.put("projectName", "MyApp");
        browserstackOptions.put("buildName", "Build 1.0");
        browserstackOptions.put("sessionName", "Login Tests");
        browserstackOptions.put("debug", true);
        browserstackOptions.put("networkLogs", true);
        browserstackOptions.put("video", true);
        capabilities.setCapability("bstack:options", browserstackOptions);

        // App uploaded to BrowserStack
        capabilities.setCapability("app", "bs://j3c874f21852ba57957a3fdc33f47514288c4ba4");

        // BrowserStack's hub URL
        driver = new AndroidDriver(
            new URL("https://hub-cloud.browserstack.com/wd/hub"),
            capabilities
        );
    }
}

Before you can run tests, you also need to upload your app:

curl -u "YOUR_USERNAME:YOUR_ACCESS_KEY" \
  -X POST "https://api-cloud.browserstack.com/app-automate/upload" \
  -F "file=@/path/to/app.apk"

# Response:
# {"app_url":"bs://j3c874f21852ba57957a3fdc33f47514288c4ba4"}

Then reference that bs:// hash in your capabilities. Every time you update your app, you need a new hash.

What DeviceLab Requires

Your Test Code

Same as your local test code. Just comment out the app capability:

public class DeviceLabTest {
    private AndroidDriver driver;

    @BeforeMethod
    public void setUp() throws Exception {
        DesiredCapabilities capabilities = new DesiredCapabilities();

        // Same capabilities as local testing
        capabilities.setCapability("platformName", "Android");
        capabilities.setCapability("appium:automationName", "UiAutomator2");

        // Comment out app capability - DeviceLab handles it via CLI
        // capabilities.setCapability("app", "./app.apk");

        // Connect to localhost - same as local
        driver = new AndroidDriver(
            new URL("http://localhost:4723/wd/hub"),
            capabilities
        );
    }
}

That's it. Same code you use locally. Comment one line. No SDK. No bstack:options. No app upload API.

Running Tests

# Terminal 1: Start DeviceLab test node
curl -fsSL https://app.devicelab.dev/test-node/KEY | sh -s -- \
  --framework appium \
  --app ./app.apk

# Wait for: ✅ Appium server ready on http://localhost:4723

# Terminal 2: Run your tests
mvn clean test

DeviceLab handles app transfer to the device, app installation, Appium server setup, and port tunneling to localhost:4723.

You just run your tests against localhost like you always did locally.

Side-by-Side Comparison

Capabilities

Capability	BrowserStack	DeviceLab
`platformName`	Yes	Yes
`automationName`	Yes	Yes
`deviceName`	Yes	No (selected via CLI)
`platformVersion`	Yes	No
`app`	Yes (`bs://hash`, upload first)	No (passed via CLI)
`bstack:options`	Yes (10+ keys)	No
`userName/accessKey`	Yes	No (in org key)

Server URL

	BrowserStack	DeviceLab
URL	`https://hub-cloud.browserstack.com/wd/hub`	`http://localhost:4723/wd/hub`

Same URL you use for local testing.

Files Changed

	BrowserStack	DeviceLab
`browserstack.yml`	Yes	No
`pom.xml` (SDK)	Add dependency	No changes
Test code	Add 15+ lines of caps	Remove `app` capability
CI/CD	Configure secrets	One secret (org key)

The Migration

Step 1: Remove BrowserStack Dependencies

pom.xml:

<!-- DELETE THIS -->
<dependency>
    <groupId>com.browserstack</groupId>
    <artifactId>browserstack-java-sdk</artifactId>
    <version>1.13.3</version>
</dependency>

Delete:

browserstack.yml
Any BrowserStack-specific configuration files

Step 2: Clean Up Capabilities

Before (BrowserStack):

DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability("platformName", "android");
capabilities.setCapability("appium:platformVersion", "13.0");
capabilities.setCapability("appium:deviceName", "Samsung Galaxy S23");
capabilities.setCapability("appium:automationName", "UiAutomator2");

HashMap<String, Object> browserstackOptions = new HashMap<>();
browserstackOptions.put("userName", "YOUR_USERNAME");
browserstackOptions.put("accessKey", "YOUR_ACCESS_KEY");
browserstackOptions.put("appiumVersion", "2.4.1");
browserstackOptions.put("projectName", "MyApp");
browserstackOptions.put("buildName", "Build 1.0");
browserstackOptions.put("sessionName", "Login Tests");
browserstackOptions.put("debug", true);
browserstackOptions.put("networkLogs", true);
browserstackOptions.put("video", true);
capabilities.setCapability("bstack:options", browserstackOptions);

capabilities.setCapability("app", "bs://j3c874f21852ba57957a3fdc33f47514288c4ba4");

driver = new AndroidDriver(
    new URL("https://hub-cloud.browserstack.com/wd/hub"),
    capabilities
);

After (DeviceLab):

DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability("platformName", "Android");
capabilities.setCapability("appium:automationName", "UiAutomator2");

driver = new AndroidDriver(
    new URL("http://localhost:4723/wd/hub"),
    capabilities
);

Lines of capability code: BrowserStack: 22 lines → DeviceLab: 6 lines

Step 3: Update CI/CD

Before (GitHub Actions with BrowserStack):

name: Mobile Tests
on: [push]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Java
        uses: actions/setup-java@v4
        with:
          java-version: '17'
          distribution: 'temurin'

      - name: Upload App to BrowserStack
        run: |
          APP_URL=$(curl -u "${{ secrets.BROWSERSTACK_USERNAME }}:${{ secrets.BROWSERSTACK_ACCESS_KEY }}" \
            -X POST "https://api-cloud.browserstack.com/app-automate/upload" \
            -F "file=@app/build/outputs/apk/debug/app-debug.apk" \
            | jq -r '.app_url')
          echo "APP_URL=$APP_URL" >> $GITHUB_ENV

      - name: Run Tests
        env:
          BROWSERSTACK_USERNAME: ${{ secrets.BROWSERSTACK_USERNAME }}
          BROWSERSTACK_ACCESS_KEY: ${{ secrets.BROWSERSTACK_ACCESS_KEY }}
          APP_URL: ${{ env.APP_URL }}
        run: mvn clean test

After (GitHub Actions with DeviceLab):

name: Mobile Tests
on: [push]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Java
        uses: actions/setup-java@v4
        with:
          java-version: '17'
          distribution: 'temurin'

      - name: Start DeviceLab
        run: |
          curl -fsSL https://app.devicelab.dev/test-node/${{ secrets.DEVICELAB_ORG_KEY }} | sh -s -- \
            --framework appium \
            --app ./app/build/outputs/apk/debug/app-debug.apk &
          sleep 30  # Wait for Appium server

      - name: Run Tests
        run: mvn clean test

Secrets needed: BrowserStack: 2 → DeviceLab: 1

What You Gain

1. No More "Works Locally, Fails in Cloud" Debugging

This is where teams lose the most time with BrowserStack.

The BrowserStack reality:

Your staging environment is behind a firewall. You need to open firewall rules to BrowserStack IPs (security risk), or run BrowserStack Local tunnel on every CI machine. Tunnel drops mid-test, causing random failures. DNS resolution works differently in their cloud. SSL certificates fail for internal services. Your staging server rate-limits BrowserStack IPs.

Result: Teams spend more time debugging environment issues than actual test failures. The test isn't flaky—the connection to your staging API is.

The DeviceLab reality:

Your devices sit on your network. They already have access to your staging servers, internal APIs, VPNs, and test databases. Same access as your laptop.

No firewall changes. No tunnels to configure or maintain. No "works on my machine" debugging. Same network, same DNS, same access.

If your test works locally, it works on DeviceLab.

2. Your Data Stays on Your Network

BrowserStack:

Your APK → BrowserStack servers → Their devices → Logs in their cloud

DeviceLab:

Your APK → Your device (via WebRTC P2P) → Results on your machine

DeviceLab never sees your app, your test data, or your credentials. We route the connection; we never see the content.

3. No App Upload Dance

Every time you update your app with BrowserStack:

# Upload new build
curl -u "user:key" -X POST ".../upload" -F "file=@new-app.apk"
# Get new hash: bs://abc123
# Update capabilities or browserstack.yml with new hash
# Or use custom_id and hope it picks the right build

With DeviceLab:

# Just pass the new APK
curl ... --app ./new-app.apk

No hashes. No upload API. No wondering if you're testing the right build.

4. Faster Feedback

BrowserStack flow: Upload APK (30s-2min depending on size) → Wait for processing → Queue for device → Run test

DeviceLab flow: APK transfers directly to your device → Run test

Your devices. No queue. No shared infrastructure.

5. Test on Your Actual Devices

BrowserStack gives you devices in their data center. They're real, but they're not your devices.

DeviceLab lets you test on:

The exact devices your users have
Devices with your company's MDM profiles
Devices on your network with your backend access
That weird Android 8 phone your CEO insists on using

6. Any Appium Version, Any Plugin

BrowserStack: Limited to Appium versions 1.21.0 through 2.19.0. The current Appium release is 3.1.2. No custom plugins allowed.

DeviceLab: You control the Appium server. Run version 3.1.2, pin to 2.x for stability, or use any version you need. Install any Appium plugins—images, gestures, or your own custom plugins.

Your test infrastructure, your rules.

Device Selection

BrowserStack:

capabilities.setCapability("appium:deviceName", "Samsung Galaxy S23");
capabilities.setCapability("appium:platformVersion", "13.0");

DeviceLab:

# Specific device
curl ... --device-names "Samsung Galaxy S23"

# Multiple devices (parallel)
curl ... --device-count 5

# Any available device
curl ...  # DeviceLab picks one

Device selection moves from code to CLI. Your test code stays clean.

Parallel Testing

BrowserStack:

# browserstack.yml
parallelsPerPlatform: 5
platforms:
  - deviceName: Samsung Galaxy S23
    platformVersion: '13.0'
  - deviceName: Google Pixel 8
    platformVersion: '14.0'

DeviceLab:

curl ... --device-count 5

Same result. Less configuration.

Common Migration Issues

"I need video recordings and logs"

DeviceLab captures test artifacts locally by default. You can optionally enable cloud storage in settings, but no logs are uploaded unless you choose to.

"I have tests spread across multiple files with different capabilities"

With DeviceLab, capabilities are minimal. Just ensure all tests point to http://localhost:4723/wd/hub and don't set the app capability.

"What about iOS?"

Same approach. DeviceLab supports iOS devices:

curl -fsSL https://app.devicelab.dev/test-node/KEY | sh -s -- \
  --framework appium \
  --platform ios \
  --app ./MyApp.ipa

Your XCUITest-based Appium tests work the same way.

Migration Checklist

Step	Action
1	Remove `browserstack-java-sdk` from `pom.xml`
2	Delete `browserstack.yml`
3	Remove all `bstack:options` from capabilities
4	Remove `app` capability (or leave blank)
5	Change server URL to `http://localhost:4723/wd/hub`
6	Update CI/CD to use DeviceLab test node
7	Connect your devices with device node
8	Run tests

Total time: 30 minutes if your test code is well-organized.

Set Up Your Devices (Once)

Before running tests, connect your devices to DeviceLab:

# On the machine with your devices
curl -fsSL https://app.devicelab.dev/device-node/KEY | sh

Devices appear in your dashboard. They're now available for testing from anywhere.

Summary

	BrowserStack	DeviceLab
Setup time	Hours (SDK, caps, yml)	Minutes
Code changes	20+ lines of capabilities	Remove 1 line
App handling	Upload API → bs:// hash	`--app` flag
Server URL	Their hub URL	localhost:4723
Appium version	1.21.0–2.19.0 only	Any (including 3.x)
Plugins	Not allowed	Any plugin
Your data	On their servers	Never leaves your network
Your devices	Theirs (shared)	Yours (owned)

Your tests already work. Stop adapting them for someone else's platform.

Try DeviceLab

Migrate BrowserStack to DeviceLab: Maestro

Om Narayan — Thu, 25 Dec 2025 06:59:50 +0000

Your Maestro flows work. Clean YAML, reliable tests, runs perfectly on your local device.

Then you tried to run them on BrowserStack.

Suddenly you're uploading apps via REST API, zipping test suites with specific folder structures, making multiple API calls, and tracking build IDs. What was maestro test flow.yaml became a multi-step orchestration.

Here's how to get back to simplicity.

What BrowserStack Made You Do

Step 1: Upload Your App

curl -u "YOUR_USERNAME:YOUR_ACCESS_KEY" \
  -X POST "https://api-cloud.browserstack.com/app-automate/maestro/v2/app" \
  -F "file=@/path/to/app.apk" \
  -F "custom_id=SampleApp"

Response:

{
  "app_name": "app.apk",
  "app_url": "bs://c8ddcb5649a8280ca800075bfd8f151115bba6b3",
  "app_id": "c8ddcb5649a8280ca800075bfd8f151115bba6b3",
  "uploaded_at": "2025-01-05 14:52:54 UTC",
  "custom_id": "SampleApp",
  "expiry": "2025-02-05 14:52:54 UTC"
}

Note that app_url. You'll need it.

Step 2: Zip Your Test Suite (Correctly)

BrowserStack requires a specific folder structure:

sample_parent_folder/           ← Must have a parent folder
├── flow1.yaml                  ← Root flows run by default
├── flow2.yaml
├── common/
│   └── login.yaml              ← Won't run unless specified
└── subflows/
    └── checkout.yaml

Critical: If you upload a .zip without a parent folder, tests fail.

# Wrong (will fail)
zip -r tests.zip *.yaml

# Right
mkdir test_suite
cp -r *.yaml common/ subflows/ test_suite/
zip -r tests.zip test_suite

Then upload:

curl -u "YOUR_USERNAME:YOUR_ACCESS_KEY" \
  -X POST "https://api-cloud.browserstack.com/app-automate/maestro/v2/test-suite" \
  -F "file=@tests.zip" \
  -F "custom_id=SampleTest"

Step 3: Execute the Build

curl -u "YOUR_USERNAME:YOUR_ACCESS_KEY" \
  -X POST "https://api-cloud.browserstack.com/app-automate/maestro/v2/android/build" \
  -H "Content-Type: application/json" \
  -d '{
    "app": "bs://c8ddcb5649a8280ca800075bfd8f151115bba6b3",
    "testSuite": "bs://89c874f21852ba57957a3fdc33f47514288c4ba1",
    "project": "My_Project",
    "devices": ["Samsung Galaxy S23-13.0", "Google Pixel 8-14.0"],
    "networkLogs": "true",
    "deviceLogs": "true"
  }'

Step 4: Check Results

Poll the API or check the dashboard.

Total API calls: 3 minimum (upload app, upload tests, execute)

Files modified: Your YAML? None. Everything else around it? Everything.

What DeviceLab Requires

Your Flows

Same YAML. No changes.

# login_flow.yaml
appId: com.example.myapp
---
- launchApp
- tapOn: "Sign In"
- tapOn: "Email"
- inputText: "test@example.com"
- tapOn: "Password"
- inputText: "password123"
- tapOn: "Login"
- assertVisible: "Welcome"

Running Tests

curl -fsSL https://app.devicelab.dev/test-node/KEY | sh -s -- \
  --framework maestro \
  --app ./app.apk \
  --tests ./flows/

That's it. One command. Same YAML files. No uploads. No zipping. No build IDs.

Side-by-Side Comparison

The Workflow

Step	BrowserStack	DeviceLab
Upload app	REST API call → get bs:// URL	Included in command
Upload tests	Zip correctly → REST API call	Included in command
Execute	REST API call with both URLs	Same command
Track results	Poll API / dashboard	CLI output
Total API calls	3	0

Your Files

	BrowserStack	DeviceLab
YAML flows	No changes	No changes
Folder structure	Must have parent folder	Use as-is
Zip file	Required	Not needed
Build scripts	Multi-step orchestration	One command

The Migration

Step 1: Remove BrowserStack Scripts

If you have CI/CD scripts like this:

#!/bin/bash
# upload_and_run.sh

# Upload app
APP_URL=$(curl -u "$BS_USER:$BS_KEY" \
  -X POST "https://api-cloud.browserstack.com/app-automate/maestro/v2/app" \
  -F "file=@$APK_PATH" | jq -r '.app_url')

# Zip tests
mkdir -p test_suite
cp -r flows/* test_suite/
zip -r tests.zip test_suite

# Upload tests
TEST_URL=$(curl -u "$BS_USER:$BS_KEY" \
  -X POST "https://api-cloud.browserstack.com/app-automate/maestro/v2/test-suite" \
  -F "file=@tests.zip" | jq -r '.test_suite_url')

# Execute
curl -u "$BS_USER:$BS_KEY" \
  -X POST "https://api-cloud.browserstack.com/app-automate/maestro/v2/android/build" \
  -H "Content-Type: application/json" \
  -d "{
    \"app\": \"$APP_URL\",
    \"testSuite\": \"$TEST_URL\",
    \"devices\": [\"Samsung Galaxy S23-13.0\"]
  }"

# Cleanup
rm -rf test_suite tests.zip

Delete it.

Step 2: Replace With One Command

curl -fsSL https://app.devicelab.dev/test-node/KEY | sh -s -- \
  --framework maestro \
  --app ./app.apk \
  --tests ./flows/

Step 3: Update CI/CD

Before (GitHub Actions with BrowserStack):

name: Maestro Tests
on: [push]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Upload App to BrowserStack
        id: upload-app
        run: |
          RESPONSE=$(curl -u "${{ secrets.BS_USER }}:${{ secrets.BS_KEY }}" \
            -X POST "https://api-cloud.browserstack.com/app-automate/maestro/v2/app" \
            -F "file=@app.apk")
          APP_URL=$(echo $RESPONSE | jq -r '.app_url')
          echo "app_url=$APP_URL" >> $GITHUB_OUTPUT

      - name: Prepare Test Suite
        run: |
          mkdir -p test_suite
          cp -r flows/* test_suite/
          zip -r tests.zip test_suite

      - name: Upload Tests to BrowserStack
        id: upload-tests
        run: |
          RESPONSE=$(curl -u "${{ secrets.BS_USER }}:${{ secrets.BS_KEY }}" \
            -X POST "https://api-cloud.browserstack.com/app-automate/maestro/v2/test-suite" \
            -F "file=@tests.zip")
          TEST_URL=$(echo $RESPONSE | jq -r '.test_suite_url')
          echo "test_url=$TEST_URL" >> $GITHUB_OUTPUT

      - name: Run Tests
        run: |
          curl -u "${{ secrets.BS_USER }}:${{ secrets.BS_KEY }}" \
            -X POST "https://api-cloud.browserstack.com/app-automate/maestro/v2/android/build" \
            -H "Content-Type: application/json" \
            -d '{
              "app": "${{ steps.upload-app.outputs.app_url }}",
              "testSuite": "${{ steps.upload-tests.outputs.test_url }}",
              "devices": ["Samsung Galaxy S23-13.0"]
            }'

After (GitHub Actions with DeviceLab):

name: Maestro Tests
on: [push]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run Tests
        run: |
          curl -fsSL https://app.devicelab.dev/test-node/${{ secrets.DEVICELAB_ORG_KEY }} | sh -s -- \
            --framework maestro \
            --app ./app.apk \
            --tests ./flows/

Lines of YAML: BrowserStack: 45 lines → DeviceLab: 15 lines

What You Gain

1. No More "Works Locally, Fails in Cloud" Debugging

This is where teams lose the most time with cloud testing services.

The BrowserStack reality:

Your staging environment is behind a firewall
You need to open firewall rules to BrowserStack IPs (security risk)
Or configure BrowserStack Local tunnel
Tunnel drops mid-test, causing random failures
DNS resolution works differently in their cloud
SSL certificates fail for internal services

Result: Teams spend more time debugging environment issues than actual test failures.

The DeviceLab reality:

Your devices sit on your network. They already have access to your staging servers, internal APIs, VPNs, and test databases. Same access as your laptop.

No firewall changes
No tunnels to configure or maintain
No "works on my machine" debugging
Same network, same DNS, same access

If your test works locally, it works on DeviceLab.

2. No Upload Dance

BrowserStack: Every build requires uploading app → uploading tests → executing. App URLs expire in 30 days.

DeviceLab: Pass the file path. Done.

3. No Zip Structure Anxiety

BrowserStack's docs warn:

"Incorrect folder structure may cause Maestro to fail in locating your flow files, resulting in test execution failures."

DeviceLab: Point to your flows directory. It works.

# Your actual folder structure
flows/
├── login.yaml
├── checkout.yaml
└── profile/
    └── update.yaml

# DeviceLab command
curl ... --tests ./flows/

4. iOS Physical Device Support

Maestro officially supports:

✅ Android devices
✅ Android emulators
✅ iOS simulators
❌ iOS physical devices

BrowserStack: Uses their own closed-source fork of Maestro. Last checked, it was based on version 1.39—the current open-source version is 2.0.10.

DeviceLab: You choose your Maestro version. Plus, DeviceLab runs Maestro on your physical iOS devices.

curl -fsSL https://app.devicelab.dev/test-node/KEY | sh -s -- \
  --framework maestro \
  --platform ios \
  --app ./MyApp.ipa \
  --tests ./flows/ \
  --device-names "John's iPhone 15"

5. Your Data Never Leaves

BrowserStack flow:

Your APK → Their servers (uploaded)
Your YAML → Their servers (uploaded)
Test execution → Their devices
Results → Their dashboard

DeviceLab flow:

Your APK → Your device (P2P transfer)
Your YAML → Your device (P2P transfer)
Test execution → Your device
Results → Your machine

DeviceLab never sees your app, your flows, or your test results.

6. Faster Iteration

BrowserStack:

Make a change to flow.yaml
Re-zip the test suite
Upload new test suite
Execute build
Wait for results

DeviceLab:

Make a change to flow.yaml
Run command
See results

No re-uploading. No waiting for processing. Your changes run immediately.

Device Selection

BrowserStack:

{
  "devices": ["Samsung Galaxy S23-13.0", "Google Pixel 8-14.0"]
}

DeviceLab:

# Specific device
curl ... --device-names "Samsung Galaxy S23"

# Multiple devices (parallel)
curl ... --device-count 3

# Specific OS version
curl ... --os-version "14"

Device selection is a CLI flag. Your test code stays clean.

Common Migration Issues

"I use execute parameter to run specific flows"

BrowserStack:

{
  "execute": ["path/to/flow1.yaml", "path/to/flow2.yaml"]
}

DeviceLab:

# Run specific flows
curl ... --tests ./flows/flow1.yaml

# Run a directory
curl ... --tests ./flows/smoke/

# Run multiple
curl ... --tests ./flows/login.yaml --tests ./flows/checkout.yaml

"My flows use environment variables"

Both support them:

BrowserStack (via config.yaml):

env:
  USERNAME: user@example.com
  API_KEY: 12345

DeviceLab:

curl ... --env USERNAME=user@example.com --env API_KEY=12345

# Or use a .env file
curl ... --env-file .env

"What if a flow fails?"

DeviceLab shows real-time output in your terminal:

✅ login.yaml - PASSED (12.3s)
❌ checkout.yaml - FAILED (8.1s)
   └── assertVisible "Order Confirmed" failed
✅ profile.yaml - PASSED (6.2s)

Results: 2/3 passed

Migration Checklist

Step	Action
1	Delete BrowserStack upload scripts
2	Remove zip/folder structure handling
3	Update CI/CD to single DeviceLab command
4	Connect your devices with device node
5	Run `curl ... --framework maestro`

Total time: 15 minutes

Set Up Your Devices (Once)

Before running tests, connect your devices to DeviceLab:

# On the machine with your devices
curl -fsSL https://app.devicelab.dev/device-node/KEY | sh

For iOS physical devices:

curl -fsSL https://app.devicelab.dev/device-node/KEY | sh -s -- \
  --apple-team-id YOUR_TEAM_ID

Summary

	BrowserStack	DeviceLab
API calls	3 (upload app, upload tests, execute)	0
Zip handling	Required, specific structure	Not needed
App reference	`bs://` hash (expires)	Local path
YAML changes	None	None
Maestro version	Closed fork (1.39)	You choose (latest 2.x)
iOS physical	No	Yes
Your data	On their servers	Never leaves your network

Local to DeviceLab: Zero Changes

Local Maestro:

maestro test flow.yaml

DeviceLab Maestro:

curl -fsSL https://app.devicelab.dev/test-node/KEY | sh -s -- \
  --framework maestro \
  --app ./app.apk \
  --tests ./flows/

Same YAML. Different device. Your network.

Your Maestro flows already work. Stop orchestrating around someone else's infrastructure.

Try DeviceLab

Getting Started with DeviceLab (5-Min Setup)

Om Narayan — Thu, 25 Dec 2025 06:54:58 +0000

Running automated tests on real devices shouldn't require complex infrastructure. With DeviceLab, you connect your own devices and start testing in minutes—no SDKs to install, no agents to configure. Just two curl commands.

How It Works

DeviceLab has two concepts:

Device Node — Makes your physical devices available for testing
Test Node — Runs your tests on those devices

Both run via curl commands. No installation required.

Step 1: Start a Device Node

Connect your Android or iOS device via USB, then run:

curl -fsSL https://app.devicelab.dev/device-node/KEY | sh

Replace KEY with your team key from the DeviceLab dashboard.

This automatically detects your connected devices (physical, emulators, simulators) and keeps running until you stop it with Ctrl+C.

Step 2: Run Tests

Download a sample test and run it:

# Download sample
curl -O https://app.devicelab.dev/samples/maestro-android-sample.zip
unzip maestro-android-sample.zip && cd maestro-android-sample

# Run tests
curl -fsSL https://app.devicelab.dev/test-node/KEY | sh -s -- \
  --framework maestro \
  --platform android \
  --app ./TestHiveApp.apk \
  --tests ./maestro-tests

That's it. Your tests run on your connected device—the app binary transfers directly via WebRTC, never touching DeviceLab's servers.

Other Frameworks

DeviceLab supports multiple test frameworks:

Maestro — Shown above
Appium (Java) — See quick start guide
Appium (Python) — See quick start guide

FAQ

What devices does DeviceLab support?

DeviceLab supports Android phones and tablets, iPhones and iPads, Android emulators, and iOS simulators. Physical iOS devices require a Mac with Xcode for initial setup.

Do I need to install anything on my mobile device?

No app installation is needed on the device. You just enable Developer Mode/USB Debugging and connect via USB. DeviceLab agent runs on the host machine.

Can I run tests from a different machine than where devices are connected?

Yes. Once devices are connected to a machine running the DeviceLab agent, they can be accessed from any machine through the DeviceLab dashboard.

What's Next?

Your first device is free. Get started with DeviceLab.

Maestro on Real iOS Devices: Working Guide

Om Narayan — Mon, 22 Dec 2025 05:48:24 +0000

TL;DR: Maestro doesn't officially support physical iOS devices. We built maestro-ios-device to fix that. Your existing YAML flows work unchanged on real iPhones.

The Problem Everyone Hits

You wrote your Maestro tests. They pass on the iOS Simulator. Now you want to run them on a real iPhone.

You discover: Maestro doesn't support physical iOS devices.

This has been the most requested feature since January 2023. Three years of GitHub issues. The Maestro team says official support won't land until sometime next year.

If you're here, you probably don't want to wait.

Does Maestro Support Real iOS Devices?

Not officially. As of December 2025:

Platform	Simulators	Real Devices
Android	✅	✅
iOS	✅	❌

Cloud providers like BrowserStack offer workarounds by running Maestro on their infrastructure. But if you want to test on your own devices — the ones sitting on your desk — you're stuck.

Until now.

The Solution: maestro-ios-device

We built a bridge that connects Maestro to physical iOS devices.

What it does:

Runs your existing Maestro YAML flows on real iPhones
No changes to your test files
Supports parallel testing on multiple devices
Works with iOS 16.x through 18.x

What it doesn't do:

Require a cloud subscription
Send your app or test data anywhere
Change how you write Maestro tests

Quick Start

1. Install the tool:

curl -fsSL https://raw.githubusercontent.com/devicelab-dev/maestro-ios-device/main/setup.sh | bash

2. Start the device bridge:

maestro-ios-device --team-id YOUR_TEAM_ID --device DEVICE_UDID

3. Run your tests:

maestro --driver-host-port 6001 --device DEVICE_UDID --app-file /path/to/app.ipa test flow.yaml

Your existing flows work unchanged:

appId: com.example.app
---
- launchApp
- tapOn: "Login"
- inputText: "user@example.com"
- tapOn: "Submit"
- assertVisible: "Welcome"

Parallel Testing on Multiple iPhones

Here's something that wasn't possible before: running Maestro tests on multiple real iOS devices simultaneously.

The original Maestro codebase had hardcoded port limitations. We removed them.

Terminal 1:

maestro-ios-device --team-id ABC123 --device IPHONE_12_UDID --driver-host-port 6001

Terminal 2:

maestro-ios-device --team-id ABC123 --device IPHONE_14_UDID --driver-host-port 6002

Now run your test suite split across devices. Cut your CI time in half (or more).

How It Works

Maestro's iOS driver was built for simulators. We extended it to recognize and communicate with physical devices.

Four things happen under the hood:

Device Detection — Recognizes physical devices with Developer Mode enabled
XCTest Runner — Builds and deploys the test driver to actual hardware (requires Apple Developer certificate)
Port Forwarding — Bridges localhost to the device's HTTP server
Session Management — Handles real-device constraints gracefully (e.g., clearState via app reinstall)

What Works, What Doesn't

We're transparent about limitations. Some are ours, most are Apple's:

Command	Status	Notes
`launchApp`	✅	—
`tapOn`	✅	—
`inputText`	✅	—
`assertVisible`	✅	—
`clearState`	✅	Reinstalls app (requires `--app-file`)
`setLocation`	⚠️	Requires additional setup
`addMedia`	❌	iOS platform restriction

Why We Built This

We run DeviceLab — a platform that lets teams test on their own physical devices. Our customers kept asking for Maestro support.

We looked at the Maestro codebase. The foundation for iOS device support existed — infrastructure PRs had been merged, but the last mile was missing.

So we finished it.

We submitted PR #2856 to give it back to the community. But PRs take time to merge, and teams need solutions now.

That's why we released maestro-ios-device as a standalone tool.

Compatibility

Maestro Version	Status
2.0.10	✅ Tested
2.0.9	✅ Tested
Other versions	Not tested

Requirements:

macOS (Xcode required for code signing)
Apple Developer account (free or paid)
iOS device with Developer Mode enabled

FAQ

Is this official Maestro?

No. This is an unofficial community tool. Not supported by mobile.dev or the Maestro team. When official iOS device support ships, we recommend switching to it.

Will my existing tests work?

Yes. If your YAML flows run on the iOS Simulator, they'll run on real devices with no changes.

Do I need to pay for anything?

The tool is free. You need an Apple Developer account (the free tier works for personal devices).

How is this different from BrowserStack/LambdaTest?

Those services run Maestro on their devices in their cloud. This runs on your devices in your office. No data leaves your network.

OpenSTF is Dead: The Best Alternative for Mobile Device Labs in 2025

Om Narayan — Tue, 16 Dec 2025 09:25:45 +0000

If you've been running OpenSTF (Open Smartphone Test Farm) for your mobile testing infrastructure, you've probably hit a wall. Your Android 14 and 15 devices aren't working. Updates stopped years ago. And you're stuck wondering: what now?

You're not alone. Thousands of QA teams and developers built their device labs on OpenSTF, and now they're scrambling for alternatives that actually work with modern devices.

What Happened to OpenSTF?

OpenSTF was originally developed at CyberAgent in Japan to manage their growing collection of 160+ test devices. It became the go-to open-source solution for teams wanting to build their own device labs.

But in July 2020, the core team announced they were abandoning the project:

"This project along with other ones in OpenSTF organisation is provided as is for community, without active development."

The last official OpenSTF release was v3.4.1, which only supports up to Android 9. If you're trying to test on Android 10, 11, 12, 13, 14, or 15 devices—you're out of luck.

The DeviceFarmer Fork: Not the Answer

Development supposedly moved to a community fork called DeviceFarmer. But here's the reality:

156+ open issues sitting on GitHub
Volunteer-maintained with no funding or dedicated team
Slow Android version support—Android 14/15 support is incomplete
Still no iOS support—same limitation as original OpenSTF
Same ancient tech stack—RethinkDB, Node.js 8.x, ZeroMQ

The project openly acknowledges: "Development is still largely funded by individual team members and their unpaid free time, leading to slow progress."

One developer on Hacker News captured the sentiment perfectly: "They dropped support on it and moved to another product (DeviceFarmer) which never seems to have materialised at all. We just kept running OpenSTF until it stopped working."

Why OpenSTF's Architecture Was Always Flawed

1. The 12-Container Problem

To run OpenSTF in production, you needed to orchestrate:

RethinkDB (database)
nginx (reverse proxy)
stf-app (web interface)
stf-auth (authentication)
stf-processor (device processing)
stf-triproxy-app (message routing)
stf-triproxy-dev (device routing)
stf-reaper (cleanup)
stf-storage-plugin-apk
stf-storage-plugin-image
stf-storage-temp
stf-websocket
stf-api
stf-provider (one per host machine)
adbd (ADB daemon)

Setting this up took days. Maintaining it was a part-time job.

2. No Data Privacy Architecture

The OpenSTF team acknowledged this in their own docs:

"We have made certain assumptions about the trustworthiness of our users. As such, there is little to no security or encryption between the different processes."

And:

"Devices do not get completely reset between uses, potentially leaving accounts logged in or exposing other sensitive data."

3. iOS Support: Never Happening

iOS support exists through a separate project (stf_ios_support), but it's barely functional:

Only 1 iOS device per Mac machine
1 FPS hardcoded framerate
Clicking is slow due to Apple API limitations
Devices frequently get stuck in "preparing" state
WebDriverAgent has memory leaks requiring restarts every 4 hours

4. Test Automation Gap

Here's what surprises many teams: OpenSTF is not a test automation platform.

OpenSTF provides remote device access—you can view the screen, tap, type, and install apps. But it doesn't run your Appium tests, Espresso suites, XCUITest cases, or Maestro flows.

The Modern Alternative: P2P Architecture

After running enterprise device labs for 12+ years serving companies like Disney+Hotstar, Airtel, Swiggy, and Jio, we saw the gap clearly:

Small teams have the same security concerns as enterprises
Cloud device farms require trusting third parties with sensitive data
OpenSTF is dead and DeviceFarmer isn't a real solution

Zero-Trust Architecture Difference

OpenSTF Architecture:

Your Device → OpenSTF Server → Your Browser
           (all data flows through central server)

Cloud Device Farm Architecture:

Your Test Data → Cloud Provider Servers → Test Results
              (they promise not to look)

P2P Architecture:

Your Device ↔ WebRTC P2P ↔ Your Browser
        (data flows directly, never touches third-party servers)

With P2P, your test data, app binaries, and device interactions flow directly between your devices and your browser via WebRTC peer-to-peer connections.

This isn't "we promise not to look at your data." This is "we architecturally cannot access your data."

What Modern Solutions Should Solve

Problem	OpenSTF	Modern Solution
Android 14/15 support	Broken	Full support
iOS support	Never worked well	Native support
Data privacy	No encryption	Zero-trust P2P
Infrastructure complexity	12+ containers	Single agent install
Maintenance burden	Significant	Managed updates
Test automation	Separate setup	Built-in

Full Test Framework Support

Unlike OpenSTF which only provides remote access, modern solutions should support all major frameworks:

Framework	OpenSTF	Modern Solution
Appium	Requires separate setup	Built-in
Espresso	Manual integration	Built-in
XCUITest	Complex setup	Built-in
Maestro	Not supported	Native support

Migration from OpenSTF

If you're currently running OpenSTF or DeviceFarmer, migration should be straightforward:

What You Keep

Your physical devices
Your Appium test suites
Your Espresso and XCUITest projects
Your CI/CD integrations
Your USB hubs and host machines

What You Gain

iOS device support
Android 14/15 support (and future versions)
All test frameworks built-in
Zero-trust data privacy
No more RethinkDB maintenance
No more minicap/minitouch debugging

What You Leave Behind

RethinkDB administration
Node.js 8.x security vulnerabilities
The anxiety of running abandoned software
Android-only limitations
12+ Docker containers
Manual Appium server setup

When OpenSTF Still Makes Sense

To be fair, OpenSTF/DeviceFarmer isn't wrong for everyone:

OpenSTF might fit if you:

Have dedicated DevOps resources with STF expertise
Only need Android devices (older versions)
Don't need integrated test automation
Have unlimited time for setup and maintenance
Run in an isolated network with no security requirements

Frequently Asked Questions

Is OpenSTF still maintained?

No. The original OpenSTF project was abandoned in July 2020. The last official release (v3.4.1) only supports Android 9. Development moved to DeviceFarmer, which is volunteer-maintained with limited resources.

What testing frameworks should modern solutions support?

Look for built-in support for Appium, Espresso, XCUITest, and Maestro. Your existing test suites should work without modification.

How should data privacy be handled?

Look for WebRTC peer-to-peer connections. Your device screen, inputs, and test data should flow directly between your device and your browser—never through third-party servers.

Can I migrate gradually from OpenSTF?

Yes. You can run new solutions alongside your existing OpenSTF setup during evaluation. Many teams migrate device-by-device over a few weeks.

The Bottom Line

OpenSTF was revolutionary for its time. It proved that teams could build their own device labs instead of paying cloud testing premiums. But it's 2025, and OpenSTF is dead.

The DeviceFarmer fork is on life support. Cloud alternatives require trusting third parties with your data.

Modern solutions should offer:

WebRTC P2P architecture (not 2015-era message queues)
Zero-trust privacy (your data never touches third-party servers)
iOS and Android support
All frameworks built-in (Appium, Espresso, XCUITest, Maestro)
Actual maintenance and support

If you're still running OpenSTF or DeviceFarmer, it's time to upgrade.

We built DeviceLab to solve these exact problems. First device free forever.

Went in to support Maestro for physical devices. Came out with even more.

Om Narayan — Mon, 08 Dec 2025 08:55:49 +0000

The Backstory

iOS real device support has been one of Maestro's most requested features since January 2023.

Almost three years of developers asking for the same thing. Issue after issue. Comment after comment.

We looked at the codebase. Surprisingly, a lot of groundwork was already there. Infrastructure PRs had been merged. The foundation existed.

It just needed someone to finish the last mile.

Why Us?

At DeviceLab, we run tests on real devices every day. It's literally what we do.

When our customers started asking for Maestro support on physical iOS devices, we had the context—and the motivation—to build it.

So we did.

We submitted PR #2856 to give it back to the community.

But here's the thing: The Maestro team has indicated official iOS real device support won't land until next year. No committed timeline. Our PR won't help anyone until it's merged—or until they build their own solution.

We didn't want teams to wait that long.

That's why we packaged a ready-to-use tool.

What It Looks Like

Your existing Maestro flows. Real iPhones. No changes.

appId: com.example.app
---
- launchApp
- tapOn: "Login"
- inputText: "user@example.com"
- tapOn: "Submit"
- assertVisible: "Welcome"

Same YAML. Same commands. Just real devices.

Tested on: iOS 26.x and iOS 18.x, including parallel execution on multiple devices.

Under the Hood

Getting Maestro to talk to real iOS devices required four pieces:

1. Device Detection

Maestro's iOS driver was built for simulators. We taught it to recognize physical devices with Developer Mode enabled.

2. XCTest Runner

The XCTest driver needed to be built and deployed to actual hardware—not just the simulator. This requires code signing with a valid Apple Developer certificate.

3. Port Forwarding

Simulators run on localhost. Real devices don't.

We bridge that gap:

localhost:6001 → device:22087

4. Session Management

Real devices have constraints simulators don't. We handle them gracefully—like clearState working via app reinstall instead of simctl commands.

The Architecture

┌─────────────┐     ┌──────────────────┐     ┌─────────────┐
│   Maestro   │────▶│ maestro-ios-device│────▶│ iOS Device  │
│  (patched)  │     │  (port forward)  │     │ (XCTest)    │
└─────────────┘     └──────────────────┘     └─────────────┘
     :6001                                        :22087

maestro-ios-device builds and installs the XCTest runner on your device
The runner starts an HTTP server on the device
Port forwarding connects your local Maestro to the device
Your tests run exactly as they would on a simulator

Bonus: Parallel Testing

While we were in there, we unlocked something else.

Running tests on multiple devices simultaneously? Wasn't possible before. Hardcoded port limitation.

Now it is:

# Terminal 1
maestro-ios-device --team-id ABC123 --device DEVICE_1 --driver-host-port 6001

# Terminal 2
maestro-ios-device --team-id ABC123 --device DEVICE_2 --driver-host-port 6002

Parallel testing on real iOS devices. Finally.

The Honest Limitations

We're not claiming this is perfect. Some iOS platform restrictions apply:

Command	Status	Notes
`clearState`	✅ Works	Via app reinstall (requires `--app-file`)
`setLocation`	⚠️ Limited	Requires additional setup
`addMedia`	❌ Not supported	iOS restriction

These are Apple's limitations, not ours.

Try It

Install:

curl -fsSL https://raw.githubusercontent.com/devicelab-dev/maestro-ios-device/main/setup.sh | bash

Start the device bridge:

maestro-ios-device --team-id YOUR_TEAM_ID --device DEVICE_UDID

Run your tests (in another terminal):

maestro --driver-host-port 6001 --device DEVICE_UDID --app-file /path/to/app.ipa test flow.yaml

Full docs: github.com/devicelab-dev/maestro-ios-device

Compatibility

Maestro Version	Status
2.0.10	✅ Supported
2.0.9	✅ Supported
Other versions	❌ Not tested

⚠️ Disclaimer

This is an unofficial community tool. Not supported by mobile.dev or the Maestro team. Use at your own risk.

When Maestro releases official iOS device support, we recommend switching to the official version.