Forem: bing yu

Training Your Mouse Behavior Clone: Make AI Browser Agents Move Like You

bing yu — Sun, 17 May 2026 09:39:18 +0000

In May 2026, a paper titled "FP-Agent: Fingerprinting AI Browsing Agents" was published on arXiv. The research team measured 7 mainstream AI browsing agents and found that their mouse trajectories, typing rhythms, and other behavioral features form a distinctive fingerprint -- not only distinguishing AI from humans, but also differentiating between different agent frameworks.

What's more concerning: the behavioral consistency across existing schemes makes them easy to classify as automated traffic.

This article explores, from a browser automation developer's perspective, how to use deep learning to make AI agent mouse operations learn your style instead of applying a generic "humanization" template.

The Problem: Why Does "Humanization" Become a Fingerprint?

Most mainstream browser automation frameworks handle "humanization" of mouse movements like this:

def move_mouse(x, y):
    points = bezier_curve(current_pos, x, y)
    for px, py in points:
        mouse.move(px, py)
        time.sleep(random.uniform(0.01, 0.03))

Seems reasonable? But here's the problem:

Everyone using this framework generates the same class of Bezier curves
Random jitter follows the same distribution
Overshoot trigger probability is the same fixed value

If you run 1,000 automation instances and cluster their mouse trajectories, you'll find they heavily overlap. This creates a behavioral fingerprint — detect one pattern, identify all instances using that framework.

The irony: the more "humanization" features you add, the less human it becomes — because there's no unified "human" pattern. If all your automation instances share the same "humanization" parameters, those parameters themselves become a massive collective fingerprint.

A Different Approach: Learn "You", Not "Human"

If the model is trained on your personal mouse operation data and generates trajectories with your personal style, the situation changes completely:

Dimension	Generic Humanization	Personal Behavior Clone
Trajectory Distribution	Shared by all users	Unique per person
Detection Difficulty	One cluster identifies all	Requires per-person modeling
Model Size	N/A	~2MB

The core shift: instead of using more complex rules to simulate "human", let the model learn "you" from your own data.

Data Collection: Record Your Habits Non-Invasively

To do behavior cloning, first you need your personal mouse trajectory data.

The implementation is simple — a Tampermonkey userscript that listens to mousemove events and records the complete trajectory from mouse movement to click. Movements shorter than 20px are treated as stationary clicks and discarded. We care about movement patterns, not the click itself.

The data format is straightforward:

{
  "viewport": {"w": 2018, "h": 1075},
  "trajectory": [
    {"x": 1480, "y": 322, "t": 0},
    {"x": 1504, "y": 317, "t": 31},
    {"x": 1501, "y": 319, "t": 69}
  ],
  "target": {"tag": "DIV", "text": "Code"}
}

x/y are viewport coordinates, t is the time offset in milliseconds relative to the trajectory start. The target HTML tag is included because clicking a button vs. clicking a link produces genuinely different trajectories — buttons have larger target areas and more casual movements; links have smaller targets with more cautious end phases.

After a few days of normal browsing, you'll have hundreds to thousands of trajectories. Then use the Tampermonkey menu to export a .jsonl file and drop it into the project's data/ directory.

Architecture: Three Models, Each Handles One Thing

With the data collected, the initial approach was to train a large GRU model for end-to-end spatial and temporal prediction. But experiments showed this didn't work well — the model would either learn overly smooth spatial trajectories (losing personalized arc styles) or uniform timing (losing acceleration/deceleration rhythms).

The solution was to decompose the problem into three modules:

Bezier (Skeleton)  →  NoiseModel (Spatial Deviation)  →  GRU (Timing)

Bezier Curve generates the macroscopic skeleton -- it's a fixed algorithm, doesn't learn anything, just guarantees a reasonable path from start to end.
NoiseModel is a small GRU (~166KB) that receives Bezier control points and outputs your personalized (x, y) path. It learns how you deviate from the ideal path -- what arcs you prefer, how much you jitter.
GRU (~2MB) takes the NoiseModel's spatial path and predicts only when each point is reached -- where you speed up, slow down, or hesitate.

Decomposing this way worked much better. Intuitively, the arc you take and when you accelerate/decelerate are two separate things. Learning them independently keeps each model cleaner.

How NoiseModel Learns Spatial Deviation

NoiseModel's input is 8 Bezier curve parameters (start point, end point, two control point coordinates, normalized to viewport). It autoregressively generates a series of (x, y) points.

During training, the first 5 epochs use pure teacher forcing with real trajectory points, then gradually reduce until the model fully self-predicts. This way it learns real data patterns while being able to generate paths independently at inference time.

One detail: points in the final 20% of the trajectory get 4x loss penalty during training. The deceleration and fine-tuning phase near the target is where human vs. machine differences are most pronounced -- machines tend to arrive straight-on, while humans show subtle overshoot and correction.

How GRU Learns Timing

GRU's input is per-step "relative space" features: distance remaining to target, how far the last step moved, how long the last step took, current progress. Not absolute coordinates — because when humans move a mouse, the brain processes "the target is still that direction, roughly how far away", not "the cursor is at pixel coordinates on screen".

There's a gotcha with time handling: in the raw data, time differences between adjacent points range from 8ms to 3494ms (extreme outliers). Direct training would be dominated by these outliers. The fix is a log1p transform -- compressing the range to [0, 8.8], then using expm1 inverse transform after training.

Another practical finding: the model tends to predict slower times than actual. Adding a 0.70 scaling factor after training makes the generated total duration match the real median.

Why Not Transformer

You might ask, why GRU when everyone's using Transformers now? Three reasons:

Trajectories are strongly continuous unidirectional sequences — GRU's inductive bias naturally matches this
Inference requires autoregressive generation, each step depends only on the previous one — GRU is much lighter than Transformer
Small dataset (hundreds of samples) — GRU generalizes better on small data

Training: Are a Few Hundred Samples Enough?

Honest answer: a few hundred is barely enough. The ideal is thousands per person.

In practice, both models are trained entirely on real data, no synthetic data dependency. The training strategy is to upweight the few hundred real samples (default ×10), telling the optimizer "prioritize fitting these real samples first".

Training speed is also decent: GRU single epoch on GPU is about 45 seconds, 200 epochs under 3 minutes. CPU works too, just takes a bit longer.

Final output: NoiseModel ~166KB, GRU ~2MB. Two files totaling under 3MB, CPU inference latency under 5ms.

An Overlooked Detail: Event Rate

GRU-generated trajectories typically have only ~30 points, with ~27ms intervals — roughly 37Hz sampling rate.

But what's the real mouse sampling rate? Browser-captured mousemove is usually around 60Hz (~16ms). Actual mouse hardware sampling rate is typically 125Hz (~8ms).

30 points looks too sparse. More critically: if your automation always produces exactly 30 points, that's an obvious artificial fingerprint. Real human mouse movements have variable event counts — a 700px movement might generate 20 effective events when moving fast, or 80 when moving slowly with fine adjustments.

So a resampling layer is added after GRU output — "translating" the model's predicted timing contour into real mouse event rates:

Adaptive intervals: sparse during fast phases (~14ms), dense during deceleration (~4ms)
Plus-minus 3ms time jitter: simulates hardware sampling noise
Plus-minus 0.3px spatial jitter: simulates sensor accuracy
Variable point count: same start/end generates 20-80+ events each time

This step only does "translation", it doesn't change the model's predicted velocity curve. When the model says "this action takes 800ms, accelerates in the middle, decelerates at the end", resampling just breaks that into irregularly-spaced events.

Results: How Does It Look

Let's look at the comparison directly.

Left: three-stage pipeline generated trajectory (Bezier skeleton + NoiseModel spatial deviation + GRU timing). Right: pure Bezier curves. The pipeline trajectory isn't a smooth mathematical curve -- it has personalized arc deviations, uneven speed, and subtle hesitation near the end.

The dynamic comparison is more intuitive:

Mechanical (gray): instant teleport to target, pure algorithm
Bezier (blue): smooth uniform glide, still algorithmic
GRU Model (gold): acceleration, hesitation, end-phase micro-adjustment — style learned from real data

Quick Start

# 1. Install dependencies
pip install torch numpy matplotlib

# 2. Place your exported .jsonl files in the data/ directory

# 3. Train NoiseModel (spatial personalization)
python training/generate_trajectories.py --train-noise --epochs 100

# 4. Train GRU (timing personalization)
python training/train_mouse_model.py --epochs 200

# 5. Visualize results
python examples/demo.py --save output.png
python examples/animate_demo.py --save output.gif

Integrate into browser automation:

from mouse_controller import move_to_humanized

# Before clicking
await move_to_humanized(page, target_x, target_y, tag="BUTTON")
# Then execute click
await page.click(target_x, target_y)

If the model fails to load, the framework automatically falls back to Bezier curves — no breaking changes.

Limitations

A few limitations worth being transparent about:

Data volume. A few hundred samples is barely sufficient; the ideal is thousands per person. The good news is this is a positive flywheel — more data means a better clone of you.

NoiseModel's assumption. The current "real = Bezier + residual" assumption is strong. A better approach would use a generative model (diffusion or VAE) to generate full trajectories directly.

Multimodal. Mouse trajectory is just one dimension of behavioral fingerprinting. Keyboard rhythm, scroll patterns, mouse dwell time aren't modeled yet. For keyboard rhythm, add a keyboard.jsonl under data/ (same format as trajectory) and adapt the scripts under training/ to reuse the existing GRU pipeline.

These limitations are also directions for future iteration.

Closing Thoughts

The AI browser agent ecosystem in 2026 is maturing rapidly. Operation accuracy is no longer the core bottleneck. The next bottleneck is trust — platforms don't trust AI traffic, users don't trust mechanical operations.

The future of AI operating computers shouldn't be "a generic robot simulating a generic human" — it should be your AI assistant, operating in your style, with your decision preferences, and behavioral characteristics that match you.

All code and data in this project is for technical research and personalized AI research use only.

Project code: https://github.com/YuBing-link/mouse-behavioral-clone

Paper reference: FP-Agent: Fingerprinting AI Browsing Agents, arXiv:2605.01247, May 2026

Hexagonal Architecture is Not a Layered Architecture: Topology, Safety, and When to Walk Away

bing yu — Mon, 11 May 2026 18:33:06 +0000

Most introductory articles draw it as an onion

Every article about Hexagonal Architecture ends up with the same diagram: three concentric circles. Domain on the inside, Application in the middle, Infrastructure on the outside. Onion Architecture. Clean Architecture. Layered Architecture with a different hat.

This is not what Ports & Adapters is.

The difference is not in the number of layers—it's in the direction of coupling.

In a layered architecture, dependency flows vertically: Controller calls Service, Service calls Repository. Each layer depends on the one below it. The problem is not that this doesn't work—it works fine for many projects. The problem is that business logic and infrastructure still couple in the same direction. Your UserService calls UserRepository (an interface), but the implementation carries JPA annotations, transaction annotations, and caching annotations. Testing the Service layer still means dealing with these implicit dependencies.

In Hexagonal Architecture, dependency flows radially inward: Adapters depend on Ports, Ports belong to the Domain. There is no "upper layer" and "lower layer"—only "inside" and "outside."

// Layered Architecture: dependency flows downward
Controller → Service → Repository (impl) → JPA/MySQL

// Hexagonal Architecture: dependency flows inward
Controller → [Port In] ← ApplicationService → [Port Out] ← Adapter
                            ↓
                    DomainService → [Port Out] ← Adapter

The key difference: in layered architecture, the Service layer depends on the Repository implementation directly (even through an interface, the impl lives in the same layer). In hexagonal architecture, the business logic has no knowledge of who implements Port Out. It could be MyBatis, an in-memory Map, or a gRPC call. This "not knowing" is Dependency Inversion applied at the architectural level, not just the class level.

Alistair Cockburn's original 2005 article described two "ports" (inbound and outbound) with "adapters" translating between the port protocol and whatever technology sits outside. There were exactly two ports, not three, four, or five layers. The proliferation of inner rings in modern interpretations comes from Clean Architecture's influence, not from Cockburn's original model. This matters because each additional ring creates a new boundary where objects must be mapped—and each mapping is a cost.

Why not another architecture?

I evaluated three alternatives before settling on Ports & Adapters. Each had a different cost structure.

Layered architecture is the natural starting point. The project began this way and shipped in three days. But as the codebase grew, the problem wasn't "does it work"—it was "how much code must I read to make a change." In a layered architecture, a feature crosses three layers with no formal contract between them. Adding a field means verifying the Controller DTO, the Service parameter, and the Mapper mapping all changed. Nothing prevents a Controller from injecting a Mapper directly—and code review missed it multiple times. Layered architecture's constraints are soft constraints. They depend on team discipline in an environment where frameworks like Spring actively encourage cross-layer injection through @Autowired.

Clean Architecture shares the same inward-dependency topology as Hexagonal, but introduces more named layers (Entities, Use Cases, Interface Adapters, Frameworks). The naming imposes a cognitive cost: "why is this class an Entity but that one isn't?" "What's the boundary between a Use Case and a Domain Service?" Every team member needs to internalize these distinctions before they can make consistent architectural decisions. For a project that was already running, introducing Clean Architecture meant introducing terminology debates before we could fix the actual coupling problems. Hexagonal's vocabulary is smaller—port, adapter, domain—and leaves the internal structure of the hexagon as an implementation detail rather than prescribing it.

CQRS + Event Sourcing is appropriate when the read/write ratio is extreme or when audit requirements demand a complete event log. Neither condition held for this project. The read and write models share the same shape, and the cost of maintaining a separate event store would have exceeded its value.

Hexagonal Architecture won for three project-specific reasons:

Multiple inbound protocols, multiple outbound technologies. The project doesn't just serve HTTP requests—it streams via SSE, receives Stripe webhooks, authenticates Chrome extension devices. It doesn't just use a database—it calls translation engines (HTTP), searches vectors (Redis), processes payments (Stripe), sends emails (SMTP). Hexagonal treats "inbound" and "outbound" as first-class architectural concepts. Adding a protocol means adding an adapter, not modifying business logic.
The translation engine is the most volatile dependency. Translation providers change frequently—vendor switches, A/B tests, local vs cloud deployment. If engine selection leaks into business code, every switch becomes surgery across multiple files. Port Out + Adapter turns "switch engine" into "swap adapter."
Testing business logic doesn't need infrastructure. The translation pipeline depends on Python services, vector search depends on Redis, payments depend on Stripe SDK. In layered architecture, testing the Service layer means starting or mocking all of these. In hexagonal architecture, domain tests mock Port Out interfaces—no Spring context, no Redis, no WireMock. This matters in CI: domain tests run without any external service.

This is not a universal recommendation. If your project has one inbound protocol (a standard HTTP API) and one outbound technology (a single database), layered architecture has a lower overhead cost and clearer payoff. Hexagonal's interface count and adapter classes are pure overhead in this scenario.

The direction of a Port determines the architecture's nature

Many articles describe Ports as "just interfaces" where Inbound means "called by Controller" and Outbound means "implemented by Repository."

The critical property of a Port is not "it's an interface"—it's who owns it.

Port In belongs to the Application layer—it defines "what the system can do," implemented by ApplicationService.
Port Out belongs to the Domain layer—it defines "what the domain needs," implemented by Adapters.

This is Dependency Inversion rendered at the architectural level. The domain defines the interfaces it needs; infrastructure implements them.

I misunderstood this twice. The first time, I treated Port as "a place to define interfaces" without distinguishing who defined them. The second time, I placed Port Out inside the adapter package, thinking "domain-owned outbound interfaces" was DDD dogma. The practical consequence: when an interface needed to change, I couldn't tell whether the business requirement had changed (Port Out change means the domain's contract with the outside world changed) or the implementation had changed (adapter-only change, no business impact).

Port ownership determines change impact scope. Changing a Port Out signature means the domain's expectation of the outside world has changed—not just that a repository implementation has been swapped.

Domain layer "zero dependency" is a matter of degree

Hexagonal Architecture advocates often say "the domain layer has zero framework dependencies." Strictly speaking, this is a matter of degree, not an absolute.

Does using java.time.LocalDateTime count as a framework dependency? Technically yes (JSR 310), but nobody cares about this level. Does using Lombok @Data count? It's an annotation processor with no runtime impact—most projects accept it.

But using Spring @Transactional, Jackson @JsonProperty, or Hibernate @Entity introduces runtime behavior changes that require framework context for testing. These should not appear in the domain package.

Our domain layer uses Lombok and nothing else. No @Service, no @Entity, no @Value. The test is not "does it have annotations" but "can the domain logic still compile and test if the framework is removed from the classpath."

Adapters are not just translators—they are security boundaries

The conventional view of an Adapter is protocol translation: HTTP request to Java call, Java object to SQL statement, Java exception to HTTP status code.

Adapters have an implicit second responsibility: input sanitization and exception classification.

External systems are untrusted. A translation engine can be DNS-hijacked. Stripe callbacks can be replayed. Cached data in Redis can have the wrong format (cross-version deployment).

If an adapter only translates protocols without validating inputs, the problem propagates to the domain layer. And the domain layer, by design, knows nothing about the outside world—making it poorly equipped to distinguish "valid data" from "data that happens to be valid-looking."

The boundary between adapter validation and domain validation follows a useful heuristic:

Format validation (JSON syntax, field type, non-empty string) — adapter layer.
Semantic validation (order amount must be positive, status transition must be legal, user cannot delete self) — domain layer.

If the validation rule remains valid after replacing the technology implementation, it belongs in the domain layer.

Code boundaries are not deployment boundaries

Hexagonal Architecture solves a code organization problem: the dependency topology between different concerns.

It does not solve a deployment problem: all adapters and the domain layer compile into one JAR, run in one JVM, and share memory and thread pools.

If one adapter leaks memory (loading all translation cache into the process), it affects the entire application. If one adapter's HTTP connection pool is exhausted, it blocks other adapter requests.

This is not a flaw in Hexagonal Architecture—it never promised to solve deployment isolation. But in a containerized environment, this is a separate concern that must be designed independently. Code boundaries and process boundaries can align or diverge depending on deployment strategy, not code structure.

The security semantics of Converters

Domain models and persistence entities are separate classes, connected by Converters. This is not redundancy—it's isolation.

Domain models reflect business concepts; persistence entities reflect database structure. They can legitimately differ—the entity may store concatenated values that the domain splits apart, or the domain may use enums while the database stores integers.

Converters have a property that is easy to overlook: different directions have different safety constraints.

Converting from persistence entity to domain model requires validation—the data may have been written by an older version that accepted values now considered invalid. Converting from domain model to persistence entity assumes domain constraints have already been enforced—validation should not be delegated to the converter layer.

How to tell when Hexagonal Architecture is overkill

Small projects don't need it. Not because it's wrong for small projects, but because the ROI scales with complexity.

When your system has 3 Controllers, 5 Services, and 2 tables, Hexagonal Architecture's indirection is a concrete cost with an abstract benefit. Maintaining six files (Port interface, ApplicationService, Domain Model, Adapter, Converter, Entity) is measurably more expensive than maintaining two files (Controller, Service). The benefit—future-proofing against change—is speculative.

When the system grows to multiple inbound protocols, multiple outbound technologies, and a complex domain, the clear dependency boundaries start paying back. Not because there's more code, but because the cost of reasoning about a change decreases. When you know "translation engine changes go to adapter/out/translate" and "billing logic changes go to domain/service," you don't need to read the entire codebase to assess impact.

This is a threshold problem. The threshold depends on team size, change frequency, and technology stack complexity. There is no universal answer.

Summary

Hexagonal Architecture is not layered architecture rearranged. It's a different dependency topology—radially inward instead of vertically downward.

Why choose it? Not because layered architecture is wrong, but because this project has:

More than one inbound protocol and more than one outbound technology.
A translation engine that changes frequently—the most volatile external dependency.
Business logic that needs to be testable without infrastructure dependencies.

What it solves: business logic should know about technology, but should not depend on it.

What it doesn't solve: deployment isolation, automatic input validation, or security hardening. These are separate concerns that must be designed independently.

The most useful thought to take away from this is not about Hexagonal Architecture specifically—it's about what architecture is for. Architecture patterns serve change management. The question to ask of any pattern is not "is this pattern good" but "what does changing a requirement cost in this structure—how many files must I read, how many tests must I run, what risks am I taking."

Hexagonal Architecture has advantages here and costs here. Knowing both clearly matters more than picking the right label.

Project source:

YuBing-link / noveltrans

AI-powered SaaS translation platform for web novels — batch translate with RAG memory, multi-agent collaboration, Stripe billing, Chrome extension & REST API. Java 21 + React 19 + Python.

NovelTrans

A production SaaS backend for AI-powered novel/document translation — multi-engine orchestration, RAG translation memory, Stripe subscription management, team collaboration, and multi-tenant data isolation.

中文版

Overview

NovelTrans is a full-stack translation platform built for web novel authors and translators. It replaces the traditional "copy-paste into Google Translate" workflow with an intelligent pipeline that understands context, preserves character name consistency, and learns from past translations — all while reducing LLM API costs through RAG-based semantic reuse.

Three client channels share the same backend:

React web dashboard — DeepL-style interface with real-time chapter preview
Chrome extension (MV3) — three modes: full-page, reader mode, text selection
External REST API — API-key authenticated, for third-party integrations

Features

Multi-engine AI orchestration — Routes translation requests across LLM (Python FastAPI + OpenAI SDK) and local engines (MTranServer) with probability-based load balancing using rolling 60-second performance windows; MTranServer serves dual purpose — fast translation mode for instant…

View on GitHub

Available for backend architecture and payment system design.