Forem: Holger Imbery

Building On‑Prem AI Agents with Azure Local, Foundry Local, and Microsoft Agent Framework

Holger Imbery — Sat, 11 Apr 2026 07:26:18 +0000

Summary Lede

Cloud-native architecture belongs on-premises too

On-premises and cloud-native are not contradictions — they are complementary. While enterprises have spent years building cloud-native practices in the cloud, those same principles—containerization, orchestration, API-driven integration, and infrastructure-as-code - deliver even greater value when deployed on-premises. This guide shows you how to build production AI agents that (must) run locally and using cloud native deployment schemas with Azure Local, Foundry Local, and Microsoft Agent Framework - this is proving that cloud-native excellence is not constrained by your network boundary.

If you operate in regulated industries, manage constrained connectivity, or face data residency requirements, this architecture gives you the operational consistency of the cloud without leaving your premises.

This article is the second in a series on Azure Local:

Azure Local, Foundry Local, and Microsoft 365 Local: A Comprehensive Guide for IT Architects and Decision-Makers.

Enterprise teams are moving beyond “chatbots” toward agents that can retrieve internal knowledge, call tools, orchestrate workflows, and produce outcomes aligned to real business processes. The challenge is that many agent reference designs assume always‑on cloud connectivity and cloud-hosted inference. That assumption does not hold everywhere.

In regulated industries, in plants and branches with constrained connectivity, or in environments where latency and data locality are non‑negotiable, the architecture has to follow the use case. This post describes a pragmatic design you can implement today by combining:

Azure Local as the on‑prem infrastructure substrate, managed through Azure Arc.
AKS on Azure Local as the standardized Kubernetes runtime for agent services and supporting components.
Foundry Local (preview) as the local inference runtime exposing an OpenAI‑compatible REST interface for model calls.
Microsoft Agent Framework (MAF) as the agent and workflow layer, including tool integration, session/state management, middleware, and telemetry patterns.

A critical insight: cloud-native architecture and practices are not limited to cloud deployments. The principles—containerization, orchestration, infrastructure-as-code, API-driven integration, observability, and declarative state management—are equally valuable on‑premises. In fact, they become more essential when your infrastructure cannot scale elastically or rely on the implicit redundancy of cloud regions. By applying cloud-native architecture to on‑prem agent deployments, you gain consistent operational models across locations, faster iteration, clear boundaries between layers, and the ability to treat infrastructure changes as routine rather than exceptional.

One design choice drives everything that follows: separate the agent runtime from the model runtime. You want the agent layer (routing, tools, workflows, state, observability) to evolve independently from inference, especially when local inference is in preview and can change.

The architecture in one picture (logical view)

A practical baseline pattern is “local inference, centralized orchestration.”

This separation keeps your application surface stable by establishing a clear boundary between stateless agent logic and stateful model inference. Because the agent layer and model runtime are decoupled, you can update agent code, refine routing logic, add new tools, or adjust middleware without touching the inference layer. Tools can be added safely behind constrained proxies or API gateways, allowing you to apply fine-grained network controls and audit trails at the integration boundary. Governance policies, observability hooks, and logging patterns remain consistent across agent operations regardless of where inference is placed. Simultaneously, inference becomes a managed dependency that can scale, relocate, or upgrade independently of application code. This architectural separation is particularly valuable in regulated environments where model serving and application logic often require separate operational controls, hardware isolation, or audit commitments. By decoupling these layers, you achieve the flexibility to place inference close to hardware accelerators (GPUs, NPUs) and data sources without forcing agent code to depend on infrastructure choices that are still evolving, especially when the inference runtime is in preview status and subject to API changes or performance tuning.

Where this approach fits (and where it does not)

This is a good fit when

This pattern becomes the right choice when one or more of the following constraints are fundamental to your deployment environment:

Data residency and regulatory compliance are hard boundaries. When regulations, industry standards, or organizational policy require that prompts, retrieved context, and inference results remain physically within an on‑premises boundary—whether for financial data, healthcare records, or proprietary intelligence—local inference becomes non‑negotiable. Cloud-based APIs, even with encryption and data-deletion assertions, may not satisfy audit requirements or legal obligations in certain jurisdictions. In these cases, the agent architecture must be designed to keep the full inference pipeline local while still benefiting from cloud-based observability and control planes, where appropriate, via segregated connections.

Latency is a direct measure of operational usability. In manufacturing plants, field service operations, retail branches, or other environments where agents serve human users on the shop floor or remote locations, response time is not a performance metric—it is a functional requirement that affects whether the agent is used at all. When users are waiting for a troubleshooting recommendation or a work instruction, a response that takes tens of seconds to traverse cloud networks is often abandoned. Local inference, combined with local agent orchestration, ensures that the slowest part of the response pipeline is your own internal network and compute capacity, not external connectivity.

Connectivity cannot be assumed to be always available and high-bandwidth. Many operational environments have constrained connectivity: scheduled outbound traffic windows, rate-limited connections, air-gapped subnets, or intentional network fragmentation for security isolation. The agent needs to function usefully within these constraints rather than degrade into a pass-through to cloud APIs. Azure Local supports this by enabling local execution and local state, while Arc provides a control-plane integration path when connectivity is available, rather than requiring continuous connectivity.

You want cloud-native operational practices applied on‑premises. This includes containerized deployments, Kubernetes orchestration for workload management, infrastructure-as-code for reproducibility, GitOps-driven delivery pipelines, policy enforcement at the runtime boundary, and standardized telemetry and logging. These practices are not exclusive to cloud deployments; they provide the same benefits on‑premises—clear separation of concerns, predictable deployments, and auditability—but require an infrastructure platform like Azure Local to realize them consistently.

When this pattern becomes problematic

Reconsidering a local-first architecture is warranted in several practical scenarios. If your inference workload demands elastic horizontal scaling and you cannot predict peak capacity without overprovisioning on-premises infrastructure, then chasing elastic scale with local hardware becomes economically and operationally inefficient. Building auto-scaling logic that manages standby capacity across stateful models would contradict the efficiency argument for locality. Similarly, if your operational environment requires production-grade stability guarantees from the inference API layer with minimal risk of breaking changes between deployments, the current maturity of local inference runtimes (such as Foundry Local, which remains in preview) presents a material risk. Preview components introduce uncertainty regarding backward compatibility, performance-tuning recommendations, and troubleshooting depth, which may not align with production SLAs. Finally, if the problem you are solving is fundamentally deterministic—where steps follow a fixed sequence, validation rules are static, and branching logic is known in advance—a structured workflow orchestration tool or a conventional microservice often provides clearer observability, simpler debugging, and lower operational overhead than an agent. Not every problem with tools and state management requires agentic behavior; sometimes explicit choreography is both simpler and more reliable.

These are the constraints that inform the "whether" decision. The next section moves to the "why Azure Local" specifically, grounded in use-case context rather than abstract on-premises philosophy.

Why Azure Local makes sense here (the use case drives locality)

Azure Local is not the point of the architecture. It is the platform choice that becomes rational when the agent pattern has to follow the environment: where data and tools live, what network rules allow, what latency targets are required, and what failure modes are acceptable.

1) The agent needs to live where the tools and data already are

High‑value agents are typically tool-heavy, and the distribution of that tooling directly affects where the agent runtime should run. The model call itself—the inference step that generates a response — is only one part of the agent interaction. The larger portion of the interaction involves retrieving documents from internal repositories, querying operational databases, validating business rules and constraints, and writing outcomes back to systems of record. Each of these operations carries a latency cost, integration overhead, and often data governance implications.

When the authoritative data sources and the systems that perform work live on‑premises—whether that is an ERP system, a manufacturing execution system, a document repository, or a service dispatch platform—moving the agent runtime closer to those systems becomes pragmatically necessary rather than architecturally optional. A remote agent calling back into on‑premises tools over the network incurs not only the latency of each call but also the operational complexity of maintaining secure, reliable network pathways between cloud and on‑premises infrastructure, managing retry logic for transient failures across that boundary, and reasoning about whether a failure in the agent's response came from the model inference or from a tool integration issue.

Placing the agent runtime on‑premises reduces integration complexity by collapsing tool interactions into local function calls with minimal network hops. It also materially shrinks the trust boundary. Data that would otherwise traverse a cloud service boundary—even with encryption in transit and assertions of deletion—can remain inside the perimeter where it originated. Azure Local provides a consistent, repeatable substrate on which to host the runtime within the organizational boundary while still enabling cloud-native operational practices such as containerization, orchestration, and declarative configuration management that teams have come to expect.

2) Latency is a functional requirement, not a nice-to-have

In operational scenarios, predictable response time is not an optimization target—it is a functional requirement embedded in the task itself. When a field technician, floor supervisor, or support worker invokes an agent for a troubleshooting recommendation or work instruction, they are typically performing a task that cannot proceed until they receive guidance. A response that arrives within seconds fits naturally into human workflow and decision-making; the user can act on it immediately and move forward. A response that takes tens of seconds—or worse, becomes non-deterministic depending on cloud API load—exceeds the mental context window of the task. Users abandon the agent, fall back to phone calls or manual lookups, or proceed without the agent's input entirely, making the agent operationally irrelevant regardless of its intelligence.

The latency problem compounds when the agent's response is not a single inference call. A typical operational agent orchestrates multiple steps: retrieving context from a document repository, querying a database to validate prerequisites, calling an external service to fetch the current state, and then synthesizing a response. Each of these operations incurs round-trip time. When those dependencies live on-premises, and the agent runtime lives in a cloud region thousands of kilometers away, the baseline latency floor is determined by geography and internet backbone capacity, not by the execution speed of any individual component. You cannot optimize away the speed of light or your ISP's rate limiting. Placing the agent runtime and its tooling in the same local environment—where internal networks typically offer latency in the single-digit millisecond range—ensures that the slowest element becomes your own infrastructure capacity, which you can measure, predict, and scale. This transforms latency from an externality you absorb to a variable you control.

Azure Local supports this placement strategy by providing AKS-hosted agent services and local model-serving infrastructure in a shared operational footprint. The inference engine, the agent orchestration layer, and the tool integrations all run in the same data center or facility where the authoritative systems live. This collapse of distance translates directly into a collapse of latency, which translates into usability in environments where response time affects task completion.

3) Connectivity constraints are a design input

Many environments are not "cloud-connected" as cloud reference architectures assume. The assumption embedded in most cloud-native architecture guidance is that outbound connectivity is available, reliable, and incurs acceptable latency and throughput. In practice, many operational environments operate under very different constraints. Outbound traffic to public cloud endpoints may be restricted by security policy or rate-limited by egress gateways. Connectivity may be scheduled—available only during specific windows or subject to maintenance blackouts. In other cases, network segments may be deliberately disconnected by design: operations technology networks in manufacturing facilities, isolated domains in financial institutions, or intentionally air-gapped environments in highly regulated sectors all follow this pattern. Even when connectivity exists, it may be mediated by proxies, firewalls, or VPNs, adding latency and complexity to troubleshooting when the agent's inference or tool calls fail.

Azure Local enables local execution of the agent runtime and inference engine regardless of whether upstream cloud connectivity is available. Simultaneously, it aligns with Azure's control-plane concepts and governance models via Azure Arc when connectivity is available. This dual capability means you can design and operate an agent system that functions reliably in disconnected or intermittently-connected scenarios without abandoning cloud-native operational practices. When connectivity is available, Arc can be used for centralized observability, policy enforcement, and update orchestration. When connectivity is unavailable, the local agent continues to function using local tools and data. This gives you an operational path that respects the actual constraints of your environment rather than forcing an architecture that assumes away those constraints or requires workarounds to compensate for them.

4) You can keep cloud-native operations without reinventing on‑prem deployment

Teams generally want repeatable delivery, policy enforcement, and consistent observability. The conventional tension between on-premises deployments and cloud-native operations has historically forced a false choice: either accept the operational discipline and automation of cloud platforms at the cost of moving workloads outside your perimeter, or keep infrastructure on-premises and revert to manual configuration management, bespoke deployment scripts, and fragmented observability tooling.

Azure Local plus AKS on Azure Local severs that coupling. Containerized deployments, GitOps-driven configuration management, Kubernetes namespaces, and declarative rollout strategies work identically whether your agent runtime is in a public cloud region or in your own data center. The infrastructure boundary becomes transparent to operational practices. Teams can maintain the same deployment pipelines, policy engines, and observability systems they have built for cloud workloads and apply them without modification to on-premises clusters. This continuity of tooling and process significantly reduces the operational friction that typically accompanies on-premises agent deployments. The "local" decision becomes a deployment location decision—a choice about where to run proven, familiar infrastructure patterns—rather than a return to bespoke server management, manual patching, and isolated monitoring infrastructure that would otherwise characterize traditional on-premises deployments.

5) Local inference forces you to manage capacity and hardware intentionally

If inference is local, capacity planning and acceleration hardware become first-class concerns that demand explicit decision-making rather than outsourced abstraction. When inference runs in a public cloud service, capacity is nominally infinite—or at least, the perception of infinity is maintained through multi-tenancy and auto-scaling tiers that obscure the underlying hardware realities. Costs accumulate by token count and API call frequency, but the physical infrastructure remains opaque. The tradeoff is acceptable if your workload is occasional or bursty; the cost volatility is a known variable you can budget for.

When inference runs locally, however, the hardware economics become tangible. A single GPU accelerator costs tens of thousands of dollars upfront, requires power and cooling infrastructure, and has a finite lifespan. Acquiring that hardware is no longer a usage-based charge smoothed into monthly billing; it is a capital expenditure that sits in your facility and has opportunity cost. This visibility forces intentional capacity planning: you must understand your typical inference load, peak throughput requirements, model sizes, and acceptable latency percentiles, and then purchase hardware that meets those requirements with some headroom for growth. You cannot simply add capacity by changing a tier or waiting for auto-scaling to provision more instances; you provision intentionally.

Azure Local provides a platform to run and govern those resources, allowing you to isolate inference nodes, stage updates, and enforce change control without coupling the inference lifecycle to the agent code lifecycle. You can reserve specific nodes for specific models, apply resource quotas to prevent one workload from starving another, and manage hardware refreshes independently from application deployments. This separation of concerns means you can upgrade your inference engine or swap model versions without draining the entire cluster, and you can plan hardware replacement without triggering emergency application refactorings. The operational rigor this imposes is not a burden—it is an alignment of technical decision-making with the actual cost structure of your infrastructure.

6) The architecture stays incremental and reversible

By separating agent runtime from model runtime, you establish a deployment boundary that allows you to make infrastructure decisions independently from application logic. This separation is critical in practice because it decouples two sources of change that typically move at different velocities: the agent orchestration layer (tools, workflows, routing, state management) tends to evolve rapidly as teams refine business logic and respond to operational feedback, while the inference runtime makes infrequent but high-impact decisions around model selection, hardware acceleration strategy, and inference node topology that are capital-intensive and difficult to reverse.

Starting small means you can pilot with a single inference node running a small quantized model, then grow to multiple specialized nodes—some optimized for latency-sensitive operations, others for throughput—without requiring changes to the agent code itself. The agent layer continues to interact with inference through the same OpenAI-compatible API boundary, indifferent to whether a single GPU or a distributed cluster backs that endpoint. You can keep your agent API stable while swapping models; if a new quantization or a different model family becomes available, you can stage it on a secondary node and route traffic to validate behavior before completing the migration. You can change inference node placement by adjusting scheduling constraints or moving nodes between racks without triggering a redeploy of agent services. This mobility is not possible when agent code and inference are tightly coupled—for example, when inference decisions are embedded in application code or when the agent layer depends on model-specific features or tokenization strategies.

Azure Local supports this incremental expansion by providing a consistent Kubernetes control plane and standard scheduling mechanisms that treat compute resources as fungible. Your initial pilot might span a single machine running AKS on Azure Local in a branch or regional office; as you validate the model and prove business value, you can expand to a small cluster in your primary data center. Each step remains operationally routine because you are not changing how workloads are deployed or managed—you are only changing the scale and distribution of resources. A pilot deployment and a production cluster follow the same GitOps patterns, use the same artifact promotion pipelines, and respond to the same observability signals, allowing you to graduate from proof-of-concept to production without a redesign of your delivery model or a learning curve on unfamiliar operational practices.

Practical decision test: Azure Local tends to be the right call when most of these are true: the authoritative tools/data are on‑prem, prompts and retrieved context must remain local, latency is a requirement, connectivity is constrained, and you want cloud-native operations in the same footprint.

With that context, we can move from "why" to "how".

Step‑by‑step implementation runbook

Phase 0 — Define boundaries: agent vs workflow, and what "done" means

Write the outcome in business terms.

Define success in measurable outcomes, not in model features. Examples include reduced downtime, faster triage, fewer escalations, shorter handling time, or improved compliance auditability.
Classify steps as agent or workflow.
- Use an agent for open-ended interpretation, conversational assistance, flexible tool use, and summarization.
- Use a workflow for deterministic steps, routing, approvals, checkpoints, and auditable state transitions.
Produce a tool inventory and trust boundary map.

For each tool, define authentication, authorization, validation, allowed destinations, and audit requirements.

Operational gotcha: Teams often prototype by giving agents broad access "to move fast." That security debt becomes expensive later. Start with constrained proxies and allow-lists from day one.

Phase 1 — Platform baseline: Azure Local + Arc + AKS on Azure Local

1.1 Establish baseline assumptions

Decide upfront:

Topology: pilot node, datacenter cluster, or distributed sites.
OS mix: Linux nodes, Windows nodes, or mixed.
Acceleration: CPU only vs GPU/NPU inference nodes.
Connectivity mode: connected, constrained, or partially disconnected.

Operational gotcha: Constrained connectivity changes everything about artifact flow. Treat "how will nodes pull images and models?" as a first-class requirement (private registry, artifact promotion, caching).

1.2 Build a minimal AKS baseline (repeatable)

Include:

Namespaces for separation (platform, agents, tools, observability).
Ingress and certificate strategy.
Secrets management strategy.
Logging/metrics pipeline.
Network policies and egress controls.

Example namespace baseline:

apiVersion: v1
kind: Namespace
metadata:
  name: agents
  labels:
    pod-security.kubernetes.io/enforce: "restricted"
---
apiVersion: v1
kind: Namespace
metadata:
  name: tools
  labels:
    pod-security.kubernetes.io/enforce: "restricted"

Operational gotcha: Without early namespace boundaries and baseline policies, your cluster becomes a collection of special cases that are hard to govern and hard to migrate.

Phase 2 — GitOps delivery (recommended even for pilots)

2.1 Repository layout pattern

A structure that scales:

clusters/<cluster-name>/ for cluster-specific overlays
platform/ for shared add-ons (ingress, monitoring, policy)
workloads/agents/ for agent services
workloads/tools/ for tool proxies and connectors

2.2 Kustomization pattern (example)

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: agents
  namespace: flux-system
spec:
  interval: 5m
  path: ./workloads/agents/overlays/prod
  prune: true
  sourceRef:
    kind: GitRepository
    name: platform-repo
  timeout: 5m

Operational gotcha: GitOps only reduces drift if "kubectl apply in production" is the exception with a documented break-glass process.

Phase 3 — Foundational services: state, memory, and observability

Make state explicit and intentional:

Conversation state (threads, session context) belongs in agent stores designed for that purpose.
Business state (work items, approvals, tickets) belongs in systems of record.

Common supporting components on AKS:

Redis for caching and rate limiting
PostgreSQL (or equivalent) for durable state
A vector store if you implement local RAG
OpenTelemetry collector for traces/metrics/logs

Operational gotcha: Agent telemetry can explode. Define retention, sampling, and content redaction policies early. In regulated environments, you often cannot log raw prompts or retrieved text.

Phase 4 — Install Foundry Local (preview) on inference nodes

Treat Foundry Local as a managed runtime dependency.

4.1 Placement and isolation

Prefer dedicated inference nodes where possible.
Place them where the acceleration hardware lives.
Segment networking so AKS can reach them reliably while keeping exposure minimal.

4.2 Endpoint discovery (avoid hard-coded ports)

Prefer one of these:

Discovery service pattern: publish the current base URL into a config store that your agent services read.
Gateway pattern: place a stable internal proxy in front of Foundry Local to normalize routing and policies.

Operational gotcha: Hard-coded ports work in a lab and fail after reboots, upgrades, or runtime changes. Build discovery or stable routing into the design.

Phase 5 — Network, TLS, and identity between AKS and Foundry Local

5.1 Connectivity options

Common choices:

Direct HTTPS from agent pods to Foundry node IP/DNS
Internal L4/L7 proxy for stable routing and policy
Service mesh for mTLS and telemetry (only if you already operate one)

5.2 TLS strategy

Use your standard PKI approach, if possible, and ensure that clients validate certificates by default.

Operational gotcha: "Works with curl -k" is a warning sign, not a milestone. Fix trust chains early so insecure shortcuts do not become permanent.

Phase 6 — Implement the inference adapter in your MAF service

Design goal: agent code calls a model client abstraction, not a concrete endpoint.

6.1 Configuration pattern (ConfigMap + Secret)

apiVersion: v1
kind: ConfigMap
metadata:
  name: agent-config
  namespace: agents
data:
  FOUNDRY_BASE_URL: "https://foundry-local.internal.example"
  FOUNDRY_MODEL: "local-chat-model"
  INFERENCE_TIMEOUT_SECONDS: "30"
  INFERENCE_MAX_RETRIES: "2"
---
apiVersion: v1
kind: Secret
metadata:
  name: agent-secrets
  namespace: agents
type: Opaque
stringData:
  FOUNDRY_API_KEY: "placeholder-if-required-by-client"

Deployment consuming it:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: maf-agent-api
  namespace: agents
spec:
  replicas: 2
  selector:
    matchLabels:
      app: maf-agent-api
  template:
    metadata:
      labels:
        app: maf-agent-api
    spec:
      containers:
      - name: api
        image: registry.local/agents/maf-agent-api:1.0.0
        envFrom:
        - configMapRef:
            name: agent-config
        - secretRef:
            name: agent-secrets
        resources:
          requests:
            cpu: "250m"
            memory: "512Mi"
          limits:
            cpu: "2"
            memory: "2Gi"
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 10

6.2 Client policy (timeouts, retries, circuit breakers)

Start with conservative defaults:

Timeout: 20–60s depending on model/prompt size
Retries: 1–2 for transient failures only
Circuit breaker: open after repeated failures to prevent cascading latency
Concurrency limits: protect inference nodes from overload

Operational gotcha: Without explicit backpressure, a single busy agent route can saturate inference and degrade every workload that shares the runtime.

Phase 7 — Tool integration with constrained proxies

Do not give agents direct access to sensitive systems.

Recommended approach:

Deploy tool proxy services in a dedicated namespace.
Restrict outbound connectivity to approved destinations only.
Enforce authorization, validation, and allow-lists in the proxy.
Log every invocation with correlation IDs.

A default-deny egress policy concept:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: tools-default-deny-egress
  namespace: tools
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress: []

Operational gotcha: If you skip network controls early, you will discover "mystery dependencies" later when tools call endpoints that were never approved.

Phase 8 — Observability: correlate agent → tools → inference

Minimum requirements:

Correlation ID propagated across inbound request, tool calls, inference calls, and response
Latency breakdown (tool time vs inference time vs orchestration time)
Error classification by category (tool failure, inference failure, policy block, timeout)
Token/prompt size metadata if available

Operational gotcha: Decide what is safe to log. For many environments, metadata and hashes are acceptable, but raw prompts and retrieved snippets are not.

Phase 9 — Hardening: safety, governance, regression testing

Hardening checklist:

Prompt and tool regression tests for critical flows
Golden conversations for validation after runtime updates
Tool schemas and allow-lists are enforced centrally
Timeouts on every external call
Rate limits per user and per route
Graceful degradation when inference is unavailable (fallback to workflow/human)

Operational gotcha: Preview inference runtimes can introduce behavior changes that are not "errors" but still break user expectations. Without regression tests, you will find out in production.

Phase 10 — Operations: versioning, rollouts, and capacity planning

10.1 Independent update cadencess

Operate on separate cadences:

Agent services: frequent updates via CI/CD
Inference runtime: cautious updates via staged rollout
Cluster/platform: regular maintenance windows

10.2 Rollout strategy

Canary agent changes with a small traffic slice and compares latency/error rates
Pin inference runtime versions and validate with representative load before expanding rollout

10.3 Capacity planning

Define explicit SLOs:

p95 latency target for a representative prompt
maximum concurrent sessions per inference node
acceptable queueing delay under peak load

Operational gotcha: Size for peaks and recovery scenarios. A thundering herd is common when shifts start, sites reconnect, or batch processes trigger.

A practical “day‑1 to day‑30” plan

Day 1–3: Foundation

Define business outcomes and agent/workflow boundaries
Stand up AKS baseline namespaces, ingress, and GitOps scaffolding
Deploy telemetry pipeline and basic dashboards

Day 4–10: Inference integration

Install Foundry Local on inference nodes
Implement endpoint discovery and TLS trust
Add inference adapter in the MAF service with externalized configuration

Day 11–20: Tools and data

Build constrained tool proxies with allow-lists and audit logs
Implement retrieval paths that keep data inside the boundary
Add correlation IDs end-to-end

Day 21–30: Hardening and operations

Add regression tests and golden conversations
Implement rollouts, version pinning, and canary strategy
Load test and finalize a capacity plan and operational runbooks

Conclusion

This stack is not about "on‑prem versus cloud." It is about aligning the agent pattern with the constraints imposed by the use case: data locality, tool proximity, latency targets, and network realities. Azure Local provides a consistent on‑prem platform for that pattern; AKS keeps operations cloud-native; Foundry Local enables local inference; and Agent Framework provides the application layer to build agents and workflows that map to real business outcomes. By following this architecture and implementation runbook, you can deliver production‑grade AI agents that run locally, proving that cloud-native excellence is not constrained by your network boundary.

Testing Copilot Agents: When to Use Agent Evaluation vs. the Copilot Studio Kit

Holger Imbery — Sun, 05 Apr 2026 07:29:14 +0000

Microsoft's Agent Evaluation GA announcement on March 31, 2026, update to Testing Copilot Studio Agents: Copilot Studio Kit vs. Agent Evaluation (Preview)

Summary Lede
Agent Evaluation and the Copilot Studio Kit are not competing tools—they represent a layered quality-assurance strategy. Agent Evaluation provides fast, AI-assisted behavioral validation embedded directly in Copilot Studio, ideal for iteration and rapid feedback. The Copilot Studio Kit delivers deterministic, enterprise-grade verification for production gates, compliance, and governance. This article breaks down what each tool does, when to use them, and how to adopt both as your agent quality matures.

Why read this
If you're building or scaling Copilot agents in your organization, you need clarity on testing strategy. This article cuts through the positioning and provides a practical decision framework for when to reach for Agent Evaluation versus the Copilot Studio Kit, with real-world scenarios showing how mature teams layer both tools across their development lifecycle.

What Microsoft shipped with Agent Evaluation (GA)

On March 31, 2026, Microsoft announced the general availability of Agent Evaluation, marking a significant milestone in Copilot Studio's testing and validation capabilities. Agent Evaluation is now generally available and built directly into Copilot Studio. Its goal is to make agent quality visible, repeatable, and scalable without requiring external tools or setup. This GA release represents the culmination of Microsoft's efforts to democratize agent quality assurance, bringing evaluation capabilities previously limited to advanced setups directly into the hands of everyday agent makers in the Copilot Studio authoring environment.

Core characteristics (as of 31. March 2026)

Integrated directly into the Copilot Studio authoring experience
Agent Evaluation is not a separate tool or external service. It lives within the Copilot Studio interface, where agents are built, allowing makers to validate their agents without context-switching or complex integrations. This tight integration reduces friction and encourages frequent validation during development cycles.

Designed to answer the production question:
"Can we trust this agent to behave correctly, consistently, and safely?"
This core question drives the entire design philosophy. Agent Evaluation focuses on behavioral confidence—whether the agent produces appropriate, consistent, and safe responses across diverse scenarios and user inputs.

Replaces unscalable manual testing and spot‑checking
Before Agent Evaluation, agent validation relied heavily on manual testing: individually testing scenarios, reviewing responses, and hoping coverage was adequate. This approach doesn't scale with agent complexity or usage volume. Agent Evaluation automates and scales this process through AI-assisted evaluation and reusable test sets.

Intended to be used before launch and continuously after changes
Agent Evaluation is not a one-time gate. It's designed for continuous validation: before initial launch, before deploying updates, and continuously as conversations flow through production. This shift from ceremonial testing to continuous validation aligns with modern DevOps practices.

Evaluation capabilities

Agent Evaluation allows makers to:

Create evaluation sets from:
- Manually added questions
- Imported test sets
- AI‑generated queries derived from agent metadata and knowledge sources
Choose flexible evaluation methods, including:
- Exact/partial match
- Semantic similarity
- Intent recognition
- Relevance and completeness scoring
Mix AI‑generated and human‑defined scenarios to balance breadth and depth
Reuse evaluations over time and run them via APIs for lifecycle testing

Key framing:
Agent Evaluation is positioned as a lightweight, AI‑assisted validation layer that fits naturally into everyday agent authoring and iteration. Unlike heavy external testing frameworks that require context switching and specialized infrastructure, Agent Evaluation operates within Copilot Studio itself, where agents are built. This embedded approach acknowledges that agent makers are iterating rapidly, testing comprehensively at each step, and need validation feedback within their authoring flow rather than as a post-production bottleneck. The AI-assisted scoring means makers don't need to hand-write every test case or define complex rubrics upfront; they can generate relevant test scenarios from their agent's own knowledge sources and metadata, then refine them. This makes evaluation accessible to makers of all skill levels and scales with agent complexity.

What the Copilot Studio Kit provides for testing

The Copilot Studio Kit (Power CAT) is a separate, solution‑based toolkit that augments Copilot Studio with enterprise‑grade testing, governance, and analytics. Developed by the Microsoft Power CAT (Patterns and Practices) team, the Kit represents a mature, production-ready framework built for organizations requiring rigorous quality assurance, regulatory compliance, and scalable CI/CD integration. While Agent Evaluation addresses everyday iteration and behavioral confidence within the authoring canvas, the Copilot Studio Kit provides the structural backbone for organizations that need deterministic verification, audit trails, multi-layer testing orchestration, and governance enforcement across large deployments.

Explicit testing capabilities

The Kit supports structured, deterministic, and multi‑layer testing, including:

Response Match (exact or conditional text comparison)
Attachment Match (Adaptive Cards/files)
Topic Match (requires Dataverse enrichment)
Generative Answer evaluation using AI Builder and rubrics
Multi‑turn tests running in a shared conversation context
Plan Validation for generative orchestration (verifying which tools/actions are invoked, not just what the agent says)

Execution and automation

Tests are executed via Copilot Studio APIs (Direct Line)
Bulk creation and maintenance via Excel import/export
Detailed run‑level telemetry:
- Pass/fail
- Latency
- Observed responses
- Aggregated metrics
Results can be enriched with:
- Azure Application Insights
- Dataverse conversation transcripts

Enterprise extensions beyond testing

The Kit also includes:

Conversation KPIs for Power BI
Prompt Advisor
Agent Inventory
Agent Review Tool
Compliance Hub with policy enforcement and SLA‑driven reviews

Key framing:
The Copilot Studio Kit is built for verification, regression testing, production gates, and governance at scale. Unlike Agent Evaluation's lightweight, AI-assisted approach that lives within the authoring canvas, the Kit functions as an enterprise testing backbone designed for organizations that require deterministic verification, full audit trails, and regulatory compliance enforcement. It bridges the gap between development-time validation and production-readiness, enabling structured quality gates that align with enterprise DevOps pipelines. The Kit's emphasis on exact response matching, topic validation, and orchestration plan verification makes it essential for mission-critical deployments where agent behavior must be predictable, traceable, and compliant.

Direct comparison

Dimension	Agent Evaluation (GA)	Copilot Studio Kit
Where it lives	Built into Copilot Studio UI	Separate Power CAT solution
Primary purpose	Behavioral validation	Functional verification
Setup effort	Minimal	Higher (Dataverse, AI Builder, App Insights optional)
Test creation	Manual, import, AI‑generated	Manual + Excel bulk
AI‑assisted scoring	Yes (core feature)	Yes (Generative Answers via AI Builder)
Deterministic checks	Limited	Strong (exact match, topic, attachments)
Multi‑turn scenarios	Not explicitly documented	Explicitly supported
Orchestration plan validation	Not documented	Explicitly supported
CI/CD & quality gates	Implicit / API‑based	Explicit pipeline integration
Governance & compliance	Not in scope	First‑class feature

How they relate (this is the key insight)

Microsoft is not replacing the Copilot Studio Kit with Agent Evaluation.
Instead, the sources show a clear layering strategy:

Agent Evaluation
→ Fast, AI‑assisted, in‑product validation
→ Ideal for early feedback, iteration, and continuous confidence
Copilot Studio Kit
→ Deep, deterministic, automatable verification
→ Ideal for release gates, regression testing, orchestration correctness, and governance

This positioning is also explicitly reflected in community and Microsoft guidance that frames Agent Evaluation as filling the gap that manual testing cannot scale, while the Kit remains the system‑level quality backbone.

Practical takeaway for enterprise teams

Based on what is explicitly documented:

When to use each tool

Agent Evaluation is best suited for:

Rapid iteration cycles during agent development
Early-stage quality validation before formal review
Continuous behavioral checks without infrastructure complexity
Scenarios where AI-assisted, semantic evaluation is sufficient
Teams prioritizing speed of feedback over deterministic guarantees
Questions like: "Is this agent generally behaving well after my last change?" → Use Agent Evaluation.

Copilot Studio Kit is best suited for:

Production release gates and formal deployment approval
Regression testing before pushing updates to production
Regulatory and compliance-driven scenarios requiring audit trails
Mission-critical agents where deterministic verification is mandatory
Complex orchestration scenarios requiring plan and tool invocation validation
Multi-turn conversations that need end-to-end correctness
Questions like: "Did we break anything? Are the topics correct? Are the tools invoked? Can this ship?" → Use the Copilot Studio Kit.

How they complement each other

In mature setups, the tools are complementary, not competitive:

Development phase: Agent Evaluation provides fast feedback loops for iteration
Pre-production phase: Copilot Studio Kit enforces deterministic verification gates
Production phase: Both tools support continuous monitoring—Agent Evaluation for behavioral trends, the Kit for functional regression detection
Governance phase: The Kit's compliance and KPI tracking provide the enterprise audit trail and policy enforcement layer

Organizations scaling from single-agent projects to enterprise deployments should expect to adopt both tools at different maturity stages, using them in sequence rather than as either/or choices.

Conclusion

Agent Evaluation and the Copilot Studio Kit represent Microsoft's thoughtful answer to the agent testing maturity curve. As organizations build, iterate, and scale agents from proof-of-concept to mission-critical systems, both tools play essential roles at different stages of the lifecycle.

Agent Evaluation brings quality validation into the authoring experience, reducing friction in everyday iteration and making behavioral confidence accessible to all agent makers. Its AI-assisted approach acknowledges the reality of rapid development cycles and the need for fast feedback loops.

The Copilot Studio Kit, by contrast, provides the deterministic backbone that enterprises require—exact verification, governance enforcement, regulatory compliance, and the audit trails necessary for mission-critical deployments.

The key insight is that these tools are not competitors but complementary. Teams should adopt them in sequence, starting with Agent Evaluation during development for rapid iteration, then layering in the Copilot Studio Kit as the agent approaches production. Organizations serious about agent quality at scale will ultimately adopt both, using them to build confidence at every stage from ideation to production and beyond.

Azure Local, Foundry Local, and Microsoft 365 Local: A Comprehensive Guide for IT Architects and Decision-Makers

Holger Imbery — Sat, 04 Apr 2026 07:56:37 +0000

Summary Lede

Cloud Capabilities Without Leaving Premises

As regulatory demands tighten, latency requirements become critical, and data sovereignty moves from a nice-to-have to a must-have, Microsoft has engineered a comprehensive answer: Sovereign Private Cloud. This three-pillar platform—Azure Local infrastructure, Microsoft 365 Local productivity, and Foundry Local AI—enables organizations to operate complete, intelligent cloud systems entirely within their boundaries. Whether you're managing classified government systems, running millisecond-critical manufacturing operations, sustaining teams in air-gapped locations, or processing sensitive AI workloads behind regulatory firewalls, this guide walks you through architectures, deployment strategies, and real-world patterns for implementing on-premises cloud at enterprise scale.

Reasons to Read This Article:

Complete Platform Understanding: Grasp all three components of this Sovereign Private Cloud approach, how they integrate, and which combination matches your operational model (connected, intermittently connected, or fully offline).
Deployment Confidence: Learn the hardware requirements, licensing models, connectivity tolerances, and planning phases required to deploy Azure Local (hyperconverged or disconnected), Microsoft 365 Local, and Foundry Local in production.
Use Case Alignment: Identify whether your organization fits one of the key scenarios—government/defense data sovereignty, manufacturing low-latency control, retail edge compute, isolated locations, or confidential AI—with architectural patterns and reference implementations.
Agentic AI on Premises: Discover how to build multi-agent AI systems using Microsoft Agent Framework + Foundry Local + Azure Local infrastructure, enabling autonomous reasoning and automation with zero cloud dependency.
Risk Mitigation and Best Practices: Understand connectivity tolerance, failover strategies, backup approaches, and testing protocols to ensure your on-premises cloud operates reliably and compliantly.
Evaluation Path: Explore trial options (60-day Azure Local eval, free Foundry Local, partner-delivered M365 Local pilots) tailored to your budget and risk profile.

Azure Local is Microsoft's distributed infrastructure solution — formerly known as Azure Stack HCI — that extends Azure capabilities to customer-owned environments. It enables local deployment of both modern and legacy applications across distributed or sovereign locations, using Azure Arc as the unifying control plane. Azure Local is the foundation of Microsoft's Sovereign Private Cloud offering, which unifies three components: Azure Local (infrastructure), Microsoft 365 Local (productivity), and Foundry Local (AI inference) to deliver a full-stack private cloud that operates at any connectivity level — connected, intermittently connected, or fully disconnected. This article provides a comprehensive overview of these offerings, their use cases, deployment options, and best practices for IT architects and decision-makers considering on-premises Azure solutions.

Why Azure Local: Hyperconverged and Disconnected

Azure Local addresses a set of business requirements for which the public cloud alone is insufficient: compute that must remain on-premises, mission-critical application resiliency, low-latency decision-making, and specific compliance mandates. Microsoft positions it as part of the adaptive cloud approach — bringing the cloud to the customer so they can build and innovate anywhere.

Hyperconverged Deployments (Connected Mode)

A hyperconverged deployment of Azure Local consists of one machine or a cluster of machines connected to Azure. Clusters support 1 to 16 physical machines with hyperconverged storage (up to 8 machines in rack-aware configurations). The architecture is built on proven technologies: Hyper-V, Storage Spaces Direct, and Failover Clustering.
In connected mode, the Azure cloud serves as the management plane. Administrators use the Azure Portal, Azure CLI, or PowerShell to view, monitor, and manage individual Azure Local instances or an entire fleet. Azure Local includes a secure-by-default configuration with more than 300 security settings, providing a consistent security baseline and a drift-control mechanism.
Connectivity tolerance: If internet connectivity is lost, all host infrastructure and existing VMs continue to run normally. However, features that directly rely on cloud services become unavailable. Azure Local must successfully sync with Azure at least once every 30 consecutive days. If that window is exceeded, the cluster enters a reduced-functionality mode — existing VMs continue running, but new VMs cannot be created until connectivity is restored.

Disconnected Operations (Full Offline Mode)

For environments where any cloud connectivity is undesired or impossible, disconnected operations bring the entire Azure control plane on-premises. Organizations can deploy and manage Azure Local instances, build VMs, and run containerized applications using select Azure Arc-enabled services from a local control plane that provides a familiar Azure Portal and Azure CLI experience — all without a connection to the Azure public cloud.
Key constraint: Disconnected mode requires extra capacity for a dedicated management cluster to host the local control plane appliance.

This management cluster has the following minimum hardware requirements:

Specification	Minimum Configuration
Number of nodes	3 nodes
Memory per node	96 GB (appliance alone needs ≥64 GB)
Cores per node	24 physical cores
Storage per node	2 TB SSD/NVMe
Boot disk drive storage	960 GB SSD/NVMe

Disconnected operations are intended for organizations that cannot connect to Azure due to connectivity issues or regulatory restrictions. To procure this capability, a valid business justification and a Microsoft Customer Agreement for Enterprises (MCA-E) (or other eligible agreement type) are required.

Connected vs. Disconnected: Decision Framework

Decision Factor	Connected (Hyperconverged)	Disconnected
Cloud dependency	Requires outbound HTTPS to Azure ≥ once per 30 days	Zero cloud dependency; local control plane
Management plane	Azure public cloud (Azure Portal, Arc)	On-premises Azure Portal and CLI replica
Hardware overhead	Workload cluster only (1–16 nodes)	Workload cluster + dedicated 3-node management cluster
Eligibility	Any Azure subscription	Requires MCA-E and business justification
Best for	Hybrid scenarios, branch offices, edge with periodic connectivity	Air-gapped facilities, classified environments, remote sites without Internet

Typical Use Cases

Azure Local, Foundry Local, and Microsoft 365 Local serve scenarios in which the traditional public cloud alone cannot meet operational, regulatory, or latency requirements. The following use cases emerge from Microsoft's documentation and partner ecosystem:

Government and Defense (Data Sovereignty)

Organizations in government, defense, and intelligence sectors require that data, operations, and control remain within organizational boundaries. Azure Local enables sovereign private clouds in which all workloads run locally. Microsoft 365 Local adds core collaboration tools — Exchange Server, SharePoint Server, and Skype for Business Server — that run entirely within the customer's sovereign operational boundary, keeping teams productive even when disconnected from the cloud. In disconnected mode, data residency and sovereign requirements are met without relying solely on public sovereign cloud controls.

Manufacturing and Industrial Operations (Low Latency & Reliability)

Azure Local targets control systems and near real-time operations with extreme latency requirements — manufacturing execution systems, industrial quality assurance, and production line operations that must continue through network outages. On-premises compute clusters enable decisions in milliseconds without cloud round-trip delays. Azure Local's integration with Azure IoT Operations (deployed on AKS clusters enabled by Azure Arc on Azure Local) provides a turnkey approach for managing and processing IoT data at the edge.

Retail and Branch Offices (Edge Compute)

Azure Local supports single-machine deployments through full clusters, making it suitable for distributed retail or branch scenarios where local AI inference at the source is needed — for example, self-checkout systems and loss-prevention applications in retail stores. The hyperconverged design ensures that even if WAN connectivity to central services drops, local operations continue uninterrupted.

Remote and Isolated Locations

Industries operating in areas with limited network infrastructure — oil rigs, mining sites, rural clinics, and vessels at sea — benefit from operating in disconnected environments. Azure Local lets them use Azure Arc services and run workloads without relying on internet connectivity. Foundry Local extends this by enabling on-device inference of AI models in offline or bandwidth-constrained environments.

Confidential AI and Data Processing

Organizations that need to run AI on sensitive data without exposing it to third-party clouds can combine Azure Local with Foundry Local. This enables local AI inferencing, where data is processed at the source. Foundry Local supports chat completions (text generation) and audio transcription (speech-to-text) through a single runtime that runs entirely on-device, with no cloud dependency for inference. Foundry Local now supports large multimodal models on Azure Local infrastructure, using the latest GPUs from partners like NVIDIA, so you can run advanced AI inference in sovereign environments.

What Is Available

Azure Local Core Infrastructure

Azure Local is a full-stack infrastructure software running on validated hardware in customer facilities. It supports VMs, containers, and select Azure services locally while maintaining Azure-consistent management through Azure Arc.

Features and architecture of hyperconverged deployments:

Feature	Description
Hardware	Validated hardware from Microsoft partners; 1–16 machines per instance (max 8 for rack-aware clusters)
Storage	Storage Spaces Direct; external SAN storage in preview for qualified opportunities
Networking	Customer-managed with physical switches and VLANs; optional software-defined networking (SDN)
Azure Local Services	VMs for general-purpose workloads; AKS enabled by Azure Arc for containerized workloads
Azure Management	Azure Policy, Azure Monitor, Microsoft Defender for Cloud, and others via Azure Arc
Observability	Metrics and logs sent to Azure Monitor and Log Analytics for infrastructure and workload resources
Management Tools	Azure Portal, CLI, ARM/Bicep/Terraform (cloud); PowerShell, Windows Admin Center, Hyper-V Manager, Failover Cluster Manager (local)
Disaster Recovery	Azure Backup, Azure Site Recovery, and non-Microsoft partners
Security	300+ security settings for consistent baseline and drift control; Trusted Launch for VMs; Microsoft Defender for Cloud integration

Common Azure services on Azure Local:

Use Case	Service
Virtual machines	Azure Local VMs enabled by Azure Arc (Windows/Linux, with Trusted Launch support)
Virtual desktops	Azure Virtual Desktop (AVD) session hosts on-premises
Container orchestration	Azure Kubernetes Service (AKS) enabled by Azure Arc
Arc-enabled services	Select Azure services for hybrid workloads via Azure Arc
High-performance databases	SQL Server on Azure Local with extra resiliency
Media analytics	Azure AI Video Indexer enabled by Azure Arc
AI chat assistants	Azure Edge RAG (Preview) — turnkey RAG solution for custom chat over private data
IoT management	Azure IoT Operations on AKS clusters on Azure Local

Disconnected operations support a subset of these services via the local control plane:

Service	Description
Azure Portal	Local portal experience similar to Azure Public
Azure Resource Manager (ARM)	Subscriptions, resource groups, ARM templates, CLI
RBAC	Role-based access control for subscriptions and resource groups
Managed identity	System-assigned managed identity for supported resource types
Arc-enabled servers	VM guest management for Azure Local VMs
Azure Local VMs	Windows or Linux VMs via disconnected operations
Arc-enabled Kubernetes (Preview)	CNCF Kubernetes cluster management on Azure Local VMs
AKS enabled by Arc (Preview)	AKS on Azure Local in disconnected mode
Azure Local device management	Create and manage instances, add/remove nodes
Azure Container Registry	Store and retrieve container images and artifacts
Azure Key Vault	Store and access secrets
Azure Policy	Enforce standards and governance on new resources

Deployment types include hyperconverged deployments, multi-rack deployments (in preview), Microsoft 365 on Local, and disconnected operations. Multi-rack deployments support larger configurations with prescriptive hardware BOMs featuring pre-integrated racks containing SAN storage, servers, and network devices; re-use of existing hardware is not supported for multi-rack at this time.

Microsoft 365 Local

Microsoft 365 Local runs Exchange Server, SharePoint Server, and Skype for Business Server on Azure Local infrastructure that is entirely customer-owned and managed. It supports both hybrid and fully disconnected deployments and provides an Azure-consistent management experience with a unified control plane.

Core capabilities:

Exchange, SharePoint, and Skype for Business: Enterprise-grade email, document management, and unified communications on-premises, addressing stringent data residency requirements.
Certified and validated solutions: Deployed on Azure Local Premier Solutions from hardware partners, guaranteeing compatibility for sovereign deployments.
Full-stack validated reference architecture: Prescriptive guidance for networking, storage, compute, and identity integration based on best practices for optimal performance and resiliency.
Sovereign Private Cloud capabilities: Azure-consistent management with enhanced security features (encryption, access controls, compliance mechanisms) aligned with local regulatory frameworks.
Hybrid or fully disconnected support: Connected mode uses Azure as the cloud control plane; disconnected mode uses a local control plane for complete isolation and air-gapped operations.

Example large-scale server role allocation (connected mode):

3 servers configured as a three-node Azure Local instance for SharePoint Server and SQL Server workloads.
4 servers each as single-node Azure Local instances for Exchange Server mailbox roles.
2 servers each as single-node Azure Local instances for Exchange Server edge transport roles.

Microsoft 365 Local is now generally available and must be deployed through a Microsoft-certified solution partner. Microsoft has committed to supporting these on-premises productivity server workloads through at least 2035.

Foundry Local

Foundry Local is an on-device AI inference solution (currently in public preview) that enables local execution of AI models through a CLI, SDK, or REST API. It provides an OpenAI-compatible REST endpoint running entirely on-device, meaning prompts and model outputs are processed locally without being sent to the cloud.

System requirements:

Requirement	Details
OS	Windows 10 (x64), Windows 11 (x64/ARM), Windows Server 2025, macOS
Minimum hardware	8 GB RAM, 3 GB free disk space
Recommended hardware	16 GB RAM, 15 GB free disk space
Optional acceleration	NVIDIA GPU (2000 series+), AMD GPU (6000 series+), AMD NPU, Intel iGPU, Intel NPU (32 GB+ memory), Qualcomm Snapdragon X Elite (8 GB+ memory), Qualcomm NPU, Apple silicon

Supported AI capabilities:

Component	Description
Foundry Local Service	An OpenAI-compatible REST server providing a standard interface for inference. The endpoint is dynamically allocated when the service starts.
ONNX Runtime	Executes optimized ONNX models on CPUs, GPUs, or NPUs; supports multiple hardware providers (NVIDIA, AMD, Intel, Qualcomm) and quantized models for faster inference.
Model Management	CLI and cache system for downloading, listing, and managing AI models locally.

Key architectural components:

Foundry Local Service: An OpenAI-compatible REST server providing a standard interface for inference. The endpoint is dynamically allocated when the service starts.
ONNX Runtime: Executes optimized ONNX models on CPUs, GPUs, or NPUs; supports multiple hardware providers (NVIDIA, AMD, Intel, Qualcomm) and quantized models for faster inference.
Model Management: CLI and cache system for downloading, listing, and managing AI models locally.

No Azure subscription is required to use Foundry Local on a device; it runs on local hardware with no recurring cloud costs for inference.
For sovereign environments requiring heavier AI workloads, the integration of Foundry Local with Azure Local supports large-scale models utilizing the latest GPUs from NVIDIA, with Microsoft providing comprehensive support for deployments, updates, and operational health.

Prerequisites and Planning for Deployment

Hardware and Catalog Selection

Azure Local runs exclusively on validated hardware configurations listed in the Azure Local Solutions Catalog. Hardware solutions fall into three categories: Validated Nodes, Integrated Systems, and Premier Solutions. Premier Solutions delivers deep integration and validation for a smooth end-to-end experience. For hyperconverged deployments, you can reuse existing hardware only if it matches a supported configuration in the catalog; otherwise, upgrades or new hardware are required.

Each Azure Local machine in a hyperconverged cluster must meet system requirements for CPU, memory, storage, and network. For planning, the Azure Local Catalog and available sizing tools help estimate hardware requirements for the intended workload profile. Networking must be designed for redundancy and performance—typically using 10–25 GbE or higher links, physical switches, and VLANs. Optional SDN services can be enabled for software-defined networking.

For disconnected operations, plan additional capacity for the management cluster as detailed in Section 1 (3 nodes, 96 GB RAM/node, 24 cores/node, 2 TB SSD/node, 960 GB boot disk/node).

For Microsoft 365 Local, hardware must be an Azure Local Premier Solution that specifically meets the M365 Local requirements listed in the Azure Local Solutions Catalog. Please work with your authorized Microsoft partner to size the deployment appropriately. We have reference architectures for small-, mid-, and large-scale configurations tailored to your needs.

Azure Subscription and Licensing

An Azure subscription is required for Azure Local. The billing model charges a per-physical-core fee on on-premises machines, plus consumption-based charges for any additional Azure services used. All charges roll up to the existing Azure subscription. For disconnected operations, an eligible enterprise agreement (such as MCA-E) is also needed, and qualification must be discussed with the Microsoft account team before procurement.

Additional licensing considerations include:

OS licenses for workload VMs (e.g., Windows Server)
Microsoft 365 server licenses if deploying M365 Local (Exchange, SharePoint, Skype)
Foundry Local requires no Azure subscription and has no RBAC role requirements when running solely on-device.

Network Connectivity Planning

In connected mode, each machine must have outbound HTTPS connectivity to well-known Azure endpoints at least every 30 days. If SDN is planned, review the SDN overview before deployment. Network and host requirements must be met per Microsoft's published specifications.

In disconnected mode, the local management cluster must be networked to the workload clusters within the customer's environment, but no external internet is required post-deployment (only registration data is exchanged during initial deployment, registration, and license renewal).

Assessment and Planning Phases

A structured planning process reduces risk. Microsoft and its partners typically follow phased engagement for Azure Local projects, especially for M365 Local:

Phase	Description
Assessment	Analyze organizational requirements, compliance needs, and desired outcomes
Planning	Define hardware configurations, software solutions, migration, and integration strategies
Acquisition	Procure necessary hardware, software, and licenses
Deployment	Execute the planned rollout in accordance with best practices

For disconnected operations, organizations must additionally identify workloads and application requirements for disconnected operation, and staff (or partners) with the capability to deploy and operate disconnected environments.

Deploying Azure Local: Steps and Best Practices

Cluster Installation and Registration

For hyperconverged deployments, the Azure Local operating system can be downloaded from the Azure Portal, which includes a free 60-day trial. Alternatively, pre-integrated systems from OEM partners arrive with Azure Local pre-installed. After installing the OS on each server node and configuring the cluster (using Storage Spaces Direct for storage and Failover Clustering for high availability), the cluster must be registered with Azure Arc to enable cloud management through the Azure Portal and Arc tools.

Hardware can be purchased from any Microsoft hardware partner listed in the Azure Local Catalog, and the available sizing tool can help estimate hardware requirements before purchase.

Post-Deployment Configuration

Once registered, the Azure Local instance appears in the Azure Portal as a manageable resource. Post-deployment steps include:

Enabling Arc-enabled services: Configure AKS clusters, Arc-enabled data services, or other platform services as needed for workload requirements.
Applying governance policies: Use Azure Policy to enforce compliance standards across the on-premises environment, and configure Microsoft Defender for Cloud to assess and improve security posture.
Setting up monitoring: Configure Azure Monitor and Log Analytics for metrics and log collection from both infrastructure and workloads.
Keeping the environment current: Azure Local provides Solution Updates that simplify keeping the entire stack up to date across OS, firmware, and drivers.

For disconnected deployments, these management services are configured on the local control plane appliance rather than through Azure public endpoints. The local Azure Portal and CLI provide an equivalent experience for managing policies, deploying VMs, and monitoring infrastructure within the isolated environment.

Deploying Microsoft 365 Local
M365 Local must be deployed through a Microsoft-certified solution partner. The partner follows the reference architecture to provision the required Azure Local instances and configure Exchange, SharePoint, and Skype for Business server roles. The reference architectures include prescriptive guidance for networking and security — covering virtual networks, network security groups, and load balancers to segment, isolate, and secure access to workloads. In connected mode, architectures use Azure as the cloud-connected control plane; in disconnected mode, they use a local control plane.

Organizations can contact their Microsoft account team or visit the Microsoft 365 Local General Availability sign-up page for information about authorized partners.

Testing and Validation

Thorough validation after deployment is critical:

Cluster validation: Run the built-in validation tools to confirm hardware, storage, and network configurations meet requirements.
VM and failover testing: Create test VMs, perform live migrations between nodes, and simulate node failures to verify high availability.
Connectivity resilience (connected mode): Simulate internet outages to confirm workloads continue uninterrupted and that the cluster correctly reconnects and syncs within the 30-day window.
Disconnected mode testing: Verify that the local management portal supports all required operations (VM provisioning, policy enforcement, monitoring) without any external connectivity.
Backup and recovery validation: Test backup and restore procedures using Azure Backup, Azure Site Recovery, or third-party solutions.

Planning and Deploying VM Workloads

Capacity Planning

Unlike the elastic scaling of public Azure, on-premises capacity is finite. IT architects must right-size VMs based on the physical resources available in the Azure Local cluster, while maintaining headroom for peak loads and failover overhead. Consider future growth when sizing: adding capacity requires purchasing and deploying new server nodes — a slower process than cloud scaling. The Azure Local Catalog and sizing tools assist with estimating how many VMs of given sizes a cluster configuration can support.

Creating VMs via Azure Arc

Azure Local manages VMs as Azure resources through the Azure Arc Resource Bridge. VMs can be created using the Azure Portal, Azure CLI, ARM templates, Bicep, or Terraform. The creation workflow through the Azure Portal involves:

Navigate to the Azure Local cluster resource and select + Create VM.
Specify project details: subscription, resource group.
Configure instance details: VM name, custom location (associated with the Azure Local cluster), security type (Standard or Trusted Launch), storage path, OS image, administrator account, vCPU count, memory allocation (static or dynamic — cannot be changed post-deployment).
Optionally enable Guest Management for Arc extensions integration, Domain Join for Active Directory, and additional data disks.
Configure networking: attach at least one network interface with appropriate IP allocation (DHCP or static).
Review and create.

This information is based on the documented VM deployment process for Azure Local environments.

Image management: Custom VM images (VHDs) can be uploaded or imported as templates. Preparing golden images — pre-hardened with security agents, configurations, and required software — streamlines consistent provisioning across the fleet.

Security for VM Workloads

Trusted Launch: Supported for Azure Local VMs, enabling secure boot and virtual TPM (vTPM). The vTPM state automatically transfers within a cluster, and attestation confirms whether the VM started in a known-good state.
Microsoft Defender for Cloud: Can assess and improve the security posture of both the Azure Local instance and individual VMs.
Arc guest management: Extensions can be deployed inside VMs for configuration management, monitoring, and security agent installation.

GPU Workloads

For AI or graphics-intensive workloads, Azure Local supports GPU-equipped servers. GPUs can be made accessible to VMs through direct pass-through or shared via GPU partitioning (GPU-P), which allows a single physical GPU to be divided into multiple virtual GPUs for different workloads simultaneously. This is valuable when multiple AI inference services, rendering tasks, or data processing workloads need GPU acceleration concurrently. NVIDIA GPUs (such as A-series models) are validated for Azure Local deployments.

Tryout and Evaluation Options

Option	Details
Azure Local 60-Day Trial	Download the Azure Local OS from the Azure Portal for a free 60-day evaluation for proof-of-concept deployments on your own hardware. Even a single validated server can be used to test core features. Microsoft's Azure Arc Jumpstart project provides step-by-step demo scenarios.
Foundry Local (Preview)	Free, no Azure subscription required. Install via `winget install Microsoft.FoundryLocal` (Windows) or `brew tap microsoft/foundrylocal && brew install foundrylocal` (macOS). Run a model immediately: `foundry model run qwen2.5-0.5b`. Experiment with text generation and speech-to-text on existing hardware. Alternatively, download the installer from the Foundry Local GitHub repository.
Microsoft 365 Local	No standalone trial download; engagement through Microsoft or a certified solution partner is required for proof-of-concept or pilot deployments. Contact your Microsoft account team or visit the M365 Local GA sign-up page. Hardware requirements are significant (enterprise-scale server configurations), so evaluations typically take place in partner labs or test environments.
Learning Resources	Microsoft Learn modules, tutorials, and the Azure Arc Jumpstart provide guided lab experiences. Community blogs, partner solution briefs (from Dell, HPE, Lenovo, etc.), and the Microsoft Tech Community contain implementation case studies and architectural guidance.

Tradeoff consideration for trials: The 60-day Azure Local trial enables self-service evaluation of the core hyperconverged platform and VM management. However, testing disconnected operations requires the dedicated management cluster hardware and MCA-E eligibility, which limits ad hoc experimentation. For M365 Local, the partner-delivered model ensures proper configuration, but it means organizations cannot independently test before engaging commercially. Foundry Local, by contrast, offers the lowest barrier to entry — it runs on a standard laptop or desktop with no cloud dependencies.

Appendix: Building Agentic AI Solutions with Azure Local, Microsoft Agent Framework, and Foundry Local

Conceptual Overview

Modern AI applications increasingly follow an agentic pattern — multiple specialized AI agents that reason, communicate, and act to perform complex tasks. Microsoft provides tools to develop and run these solutions entirely on local infrastructure by combining three components:

Azure Local — the on-premises infrastructure providing compute, storage, networking, and (optionally) GPU acceleration.
Foundry Local — the on-device AI inference runtime serving LLM and other models via an OpenAI-compatible API endpoint.
Microsoft Agent Framework (MAF) — an open-source framework (Python and .NET SDKs) for building, orchestrating, and deploying AI agents and multi-agent workflows.

The Agent Framework was introduced as an open-source project by Microsoft and is hosted on GitHub at microsoft/agent-framework with over 8,300 stars and 1,400 forks. The latest release at the time of research was python-1.0.0rc5 (dated 2026-03-19).

Architecture Pattern

A concrete reference implementation was published on the Microsoft Developer Community Blog, demonstrating real-world AI automation with Foundry Local and MAF — described as running with "no cloud subscription, no API keys, no internet required". The system uses four specialized agents orchestrated by MAF:

Agent	Function	Latency
PlannerAgent	Sends user commands to the Foundry Local LLM and produces a structured JSON action plan	4–45 seconds
SafetyAgent	Validates actions against workspace bounds and schema constraints	< 1 ms
ExecutorAgent	Dispatches validated actions to the target system (e.g., robotics simulator for inverse kinematics and gripper control)	< 2 seconds
NarratorAgent	Produces a template-based summary of actions taken (with optional LLM elaboration)	< 1 ms

The orchestration flow follows a sequential pipeline: User → Orchestrator → Planner → Safety → Executor → Target System, with the Narrator providing observability.

In this reference, the PlannerAgent uses Foundry Local as its AI backend, invoking a local model (e.g., qwen2.5-coder-0.5b) via the standard OpenAI Python client pointing to the Foundry Local endpoint:


from foundry_local import FoundryLocalManager
import openai

manager = FoundryLocalManager("qwen2.5-coder-0.5b")
client = openai.OpenAI(
    base_url=manager.endpoint,
    api_key=manager.api_key,
)

This pattern — structured JSON output from an LLM, validated by a safety layer, dispatched to a domain-specific engine — generalizes beyond robotics to home automation, game AI, CAD, lab equipment, and any domain requiring safe, structured control.

Deployment Patterns on Azure Local

For production deployment of agentic AI on Azure Local infrastructure, the following layered architecture applies:

Layer 1 — AI Model Hosting: One or more Azure Local VMs (or containers) running Foundry Local to serve AI models. For small models, a standard CPU-equipped VM suffices. For large multimodal models, VMs with dedicated GPU access on Azure Local infrastructure leverage the latest NVIDIA GPUs for high-throughput inference. Foundry Local automatically selects the best execution provider (NPU > GPU > CPU) for the available hardware.
Layer 2 — Agent Orchestration: The Microsoft Agent Framework runs as a service (in a container on AKS or in a VM) and orchestrates the multi-agent pipeline. It handles agent-to-agent communication, memory management, tool integrations, and calls to the Foundry Local inference endpoint. Domain-specific engines (simulation environments, database connectors, control system APIs) can be integrated as tools that agents invoke during execution.
Layer 3 — Application Interface: A custom frontend (web application, dashboard, CLI, or API gateway) through which users submit tasks and receive results. This can be hosted on the same Azure Local cluster. All inter-layer communication occurs over the cluster's internal network, keeping data fully on-premises and latency to a minimum.

Applicable Scenarios

The combination of Azure Local + Foundry Local + MAF enables agentic AI solutions where:

Industrial automation: Agents interpret natural-language operator commands, plan machine actions, validate safety constraints, and execute robotic or process-control operations — all on the factory floor without cloud dependency.
Sovereign AI assistants: Multi-agent systems that collate local data, reason using on-device LLMs, and provide decision support in classified or regulated environments (defense, finance, healthcare) where data must never leave the premises.
Edge intelligence: IoT-connected environments where agents monitor sensor data streams, use local AI for anomaly detection and root-cause analysis, and actuate responses in real time — applicable to energy infrastructure, transportation systems, or smart facilities.
Offline automation: Field operations, shipboard systems, or disaster-response scenarios where internet connectivity is unavailable but sophisticated AI reasoning and automation are still required.

Key Advantages and Tradeoffs

Advantages: Running agentic AI entirely on Azure Local provides data sovereignty (all prompts, model outputs, and orchestration data remain local), low latency (no network hops to cloud endpoints), deterministic cost (no per-token API charges), and operational resilience (functions without internet).
Tradeoffs: On-device models are constrained by local GPU memory and compute — the largest cloud-hosted models (e.g., GPT-4 at full scale) may not be runnable locally without significant GPU investment. Model updates require manual download and deployment rather than automatic cloud-side updates. Additionally, Foundry Local remains in public preview, meaning features and supported models are still evolving and may have limitations before general availability. Organizations should evaluate whether the models available for local inference meet their quality bar for production use, and plan for a path to larger models as Foundry Local's support for large-scale models on Azure Local with NVIDIA GPUs matures.

Conclusion

Azure Local, Foundry Local, and Microsoft 365 Local together form a cohesive platform for organizations seeking sovereign, on-premises cloud capabilities without compromise. As data residency, regulatory compliance, and operational resilience become non-negotiable requirements across industries, Microsoft's investment in distributed infrastructure and local AI inference reflects a fundamental shift in how enterprises architect their digital ecosystems.

The combination of Azure Local (providing edge-aware infrastructure and hybrid compute), Microsoft 365 Local (delivering productivity and collaboration on-premises), and Foundry Local (enabling local LLM inference) addresses the long-standing tension between cloud agility and data sovereignty. Whether your organization operates in a connected, intermittently connected, or fully disconnected environment, these solutions let you innovate locally without sacrificing the governance, scale, or intelligence that cloud-native architectures offer.

For IT architects and decision-makers, the path forward is clear: evaluate your specific regulatory, latency, and data residency requirements; prototype on a small cluster or Azure Local Appliance; and progressively expand as organizational confidence and operational maturity grow. The learning curve is manageable, the economics are favorable for regulated industries, and the competitive advantage in markets demanding data sovereignty is significant.

As Foundry Local and Azure Local move toward general availability and mature their feature sets, the case for Sovereign Private Cloud becomes stronger. The future of enterprise computing is not "cloud vs. on-premises" — it is a thoughtfully designed hybrid architecture that respects both business logic and the regulatory terrain in which that logic operates.

Practical Guideline: How to Move Agents Beyond POCs and Deliver Real Enterprise Value

Holger Imbery — Sat, 21 Mar 2026 08:29:29 +0000

Summary Lede

I hear the same question repeatedly from customers exploring agent or broader AI adoption: "How do we escape the endless POC phase and actually deliver real business value?" Most organizations get stuck prototyping broadly instead of executing narrowly, trapped in cycles of experimentation that never reach production. This practical guideline distills ten core principles proven to move agents from ideation into measurable enterprise impact. Read on to discover how to anchor initiatives in real processes, maintain scope discipline, connect agents to live input channels, enforce production-grade behavior from day one, integrate with mission-critical systems early, deliver in short iteration cycles, create lightweight review processes, commit to real usage within 30 days, use multiple small agents, and plan for long-term flexibility—transforming your AI investment from experimentation into sustainable value delivery.

Organizations often get stuck in endless prototyping cycles because they experiment broadly rather than execute narrowly. This guideline distills the core principles that move agents from ideation to measurable business impact.

1. Anchor the Initiative in a Single Real Process

Begin by identifying a single operational workflow where your organization currently loses productive time on a recurring basis. This workflow should exhibit characteristics such as repetitive manual steps, rule-based decision logic, or intensive data manipulation. Avoid starting with abstract experimentation or exploratory prototypes that lack connection to actual business operations.

The economic rationale for this approach is straightforward. When you ground your agent development in a concrete, existing process, you force alignment with real data sources, actual system dependencies, and measurable business outcomes. This concrete anchoring eliminates the disconnection that is characteristic of laboratory environments and proof-of-concept work, which often remain isolated from production constraints and real-world variability.

To establish this baseline understanding, you should document the current state of the process by answering these questions. First, identify what data inputs currently exist and where they originate within your organization. Second, determine which specific steps within the workflow consume the most human effort and therefore represent the highest opportunity for efficiency gains. Third, establish what quantifiable outcome should improve as a result of agent implementation, whether measured in terms of time savings per transaction, reduction in human errors, or increase in process throughput.

2. Keep the First Agent Extremely Narrow

The most decisive factor in moving beyond proof-of-concept phases is maintaining scope discipline. Organizations frequently fail to operationalize agents because they attempt to expand functionality too broadly before achieving stable baseline performance in a narrow domain. This expansion pattern increases complexity exponentially while simultaneously distributing development resources across multiple problem dimensions.

The essential discipline requires that you define the agent's responsibilities in a single, unambiguous sentence of the form: "This agent is responsible for X and nothing beyond X." This constraint forces explicit trade-offs between capability breadth and implementation depth, ensuring that resources concentrate on achieving reliable performance in one well-defined function rather than fragmented performance across multiple functions.

Consider these concrete examples of appropriately scoped initial deployments. An agent might be chartered to look up customer pricing from a master database and return one verified result, without attempting to negotiate, modify, or recommend alternative pricing. Another agent might be restricted to extracting structured fields from incoming documents and validating them against schema requirements, without attempting interpretation or applying business rules. A third agent might be limited to classifying incoming inquiries into exactly three predefined categories, without attempting subcategories or fuzzy classifications.

This discipline against over-engineering serves multiple economic functions. It reduces the surface area for defects, shortens the time to reach measurable operational impact, and simplifies the governance model for operating the agent in production environments. By deferring expansion until baseline performance is established, organizations create a foundation of operational reliability upon which additional capabilities can be layered incrementally.

3. Connect the Agent to a Real Input Channel Early

Manual testing through a studio UI creates an isolated environment that does not reflect operational reality. The agent is evaluated against synthetic inputs, clean data structures, and predetermined response patterns—conditions that rarely occur in production systems. Real business value emerges only when the agent receives actual operational requests that contain the variability, ambiguity, and edge cases inherent in genuine work.

Operational input channels include the following mechanisms through which work currently flows into your organization: forwarded email messages containing unstructured customer inquiries, Teams chat messages that combine urgent questions with side conversations, CRM cases that reference prior interactions and incomplete context, and uploaded documents that may contain inconsistent formatting or missing required fields. Each of these channels introduces distinct data quality challenges and user expectations.

The economic rationale for early channel integration stems from the principle of revealed preference through actual behavior. When users interact with the agent through their existing workflow channels rather than a lab environment, their usage patterns reveal which capabilities create genuine value and which create friction. Synthetic testing cannot substitute for this behavioral signal. Furthermore, exposure to live variability from the first iteration accelerates learning about edge cases and failure modes that would otherwise remain hidden until full production deployment.

Action: Select one input channel that currently delivers the highest volume of work into your target process and route genuine operational requests through the agent beginning in the first development cycle. This approach ensures that early versions contend with real data distributions and authentic user patterns rather than idealized test scenarios.

4. Enforce Production‑Grade Behavior From Day One

Development environments and production environments typically operate under fundamentally different constraints and enforcement mechanisms. In many organizations, agents developed during the proof-of-concept phase operate with suspended governance controls, synthetic data, and permissive access policies that would never be acceptable in operational systems. This separation creates a structural barrier to production adoption because moving an agent from a development environment to production then requires a complete redevelopment of its data connectors, compliance controls, and operational behaviors.

The efficient approach eliminates this artificial separation by imposing production-grade constraints from the initial development phase. This requires the agent to use the actual data sources employees rely on for their daily work, rather than sanitized copies or test databases. The agent must apply existing data access restrictions and compliance controls that govern access to sensitive information within your organization, rather than running with elevated or unrestricted permissions. The agent must maintain consistent tone, content, and response handling in line with organizational standards, rather than developing ad hoc response patterns during development that would later require modification. The agent must draw on approved knowledge sources that align with organizational information governance policies, rather than accessing ad hoc files or unvetted external data.

This approach reduces the economic cost of production deployment by eliminating the need to redesign and re-implement governance controls at transition time. Additionally, exposing the agent to genuine constraints during development accelerates the identification of edge cases and failure modes that would otherwise remain hidden until production deployment, when they would be far more costly to address.

Action: Establish the expectation that the first version of the agent operates under the same governance framework and data access policies as the final production system. This mindset collapses the artificial gap between proof-of-concept development and production readiness.

5. Integrate With One Mission‑Critical System Early

From an operational perspective, a proof-of-concept implementation that remains disconnected from your organization's core systems generates negligible business value regardless of how well the agent performs in isolation. The critical transformation occurs when the agent gains the ability to read from or write to systems that directly affect your operational workflows, such as customer relationship management platforms, enterprise resource planning systems, document management repositories, or human resources information systems. At that point, the agent transitions from a theoretical capability into a practical tool that produces measurable outcomes within your existing business processes.

The economic principle underlying this requirement is straightforward: manual handoff steps between system boundaries represent a fundamental source of friction and delay. When an agent completes its analysis but requires a human to manually transfer its output into another system, you have failed to eliminate the bottleneck that prompted the agent's development in the first place. Conversely, when the agent can directly query information from, or write results to, systems where decisions take effect, the entire workflow collapses into a unified operational flow that removes intermediate steps.

Your implementation approach should prioritize identifying which single system integration would eliminate the greatest volume of repetitive manual work, and then building only the minimal version of that integration during the initial development phase. This targeted approach might manifest as the agent querying a system to retrieve structured reference data that previously required manual lookup, writing a record to a system to capture the decision the agent has reached, extracting and processing a document that originated from a system's document repository, or triggering an automated workflow in a system that would otherwise require manual initiation. Even a single integration point of this magnitude, when implemented at the outset rather than deferred until later phases, serves as a forcing function that exposes the real constraints your agent must navigate within your organization's operational environment.

6. Deliver in Short Iteration Cycles

Extended design phases pose a structural impediment to effective agent development by deferring real-world validation and prolonging the time horizon before measurable feedback becomes available. Organizations attempting comprehensive upfront design face two competing failures: either they design systems that do not align with operational reality once implemented, or they extend the pre-implementation phase so long that organizational priorities shift before deployment occurs. Agents improve most rapidly through short cycles of genuine operational usage because each cycle generates concrete evidence of performance gaps and behavioral mismatches that cannot be anticipated through laboratory analysis alone.

The organizational practice that supports this principle involves establishing a defined release rhythm of 7 to 10 days as the baseline cadence for detecting problems, gathering behavioral feedback, and incorporating refinements. This rhythm creates a predictable organizational rhythm while ensuring sufficient time for both development work and operational assessment. Within this structured cycle, the work proceeds through sequential phases: during the initial week, the focus concentrates on delivering a working version of the agent that operates within the narrowly defined scope established by principle two. During the second week, attention shifts toward integrating the agent with the single mission-critical system identified during principle five, which forces the agent to operate under genuine operational constraints. During the third week, the team prioritizes incorporating feedback-driven refinements based on direct observations of how the agent performs under real operational patterns and edge cases. By the fourth week, the agent transitions into daily-use rotation, becoming a standard component of the operational workflow rather than an experimental capability.

This structured iteration discipline transforms theoretical value into practical, measurable improvements by compressing the feedback loop between hypothesis and evidence to a manageable timeframe. Organizations that maintain shorter iteration cycles identify defects and misalignments exponentially faster than those that attempt extended design phases, resulting in a substantially faster path to production-grade performance.

7. Create a Lightweight Review and Quality Model

Heavyweight governance structures—comprehensive architecture review boards, multi-stage approval processes, and extensive documentation requirements—impose transaction costs that delay feedback cycles and create organizational friction. These formal processes were designed for environments where deployment cycles measured months and the cost of errors remained relatively stable. Agent development operates under fundamentally different constraints: deployment cycles measure days, and the cost of a minor behavioral inconsistency in an agent can compound over hundreds of interactions before detection.

The economic principle underlying lightweight review processes is that not all decisions require the same deliberative overhead. Operational decisions about individual agent behaviors—how to handle edge cases, whether a response meets quality standards, or how the agent should escalate undefined requests—benefit from frequent, lightweight validation rather than formal approval hierarchies. Conversely, decisions about expanding an agent's scope or integrating new system connectors require structured deliberation, but this deliberation should remain episodic rather than continuous.

The practical implementation of this principle involves establishing separate review cadences calibrated to the urgency of decisions. Weekly operational reviews should examine direct evidence of agent performance, specifically documented failures, observed edge cases that the agent failed to handle correctly, and user experience friction points that emerged during actual operational usage. These reviews operate without approval authority; they serve as diagnostic sessions that generate recommendations for refinement. Monthly functional expansion decisions should convene stakeholder representatives to evaluate whether the agent's scope should be widened, which integration points to add next, or whether the agent should be split into multiple specialized agents. These decisions operate with explicit approval authority because scope decisions determine resource allocation for the subsequent month's development work.

Standard templates for agent instructions, response formats, and escalation procedures ensure consistency across agents without requiring case-by-case review. A template encodes learned patterns from prior agent implementations into repeatable structures that new agents can adopt immediately, reducing both development time and the likelihood of behavioral inconsistencies between agents.

This calibrated review model reduces unnecessary transaction costs while maintaining deliberative oversight for decisions that require it, so alignment occurs without creating a delivery bottleneck.

8. Commit to Real Usage Within 30 Days

The principle of time-bound value realization addresses a fundamental problem in agent adoption: organizations frequently defer the transition from development to operational deployment indefinitely, justifying continued laboratory work with incremental improvements that never add up to genuine business impact. The economic cost of this deferral compounds over time because development resources consumed during extended proof-of-concept phases represent opportunity costs that could have been deployed toward other organizational priorities.

A disciplined commitment to a fixed time horizon solves this problem by establishing an explicit deadline for demonstrating measurable operational value. The specific timeframe of thirty days aligns with the typical organizational planning cycle, allowing early agent deployment results to inform resource allocation decisions for the subsequent planning period. This timeframe is sufficiently compressed to prevent indefinite deferral while remaining realistic for narrowly scoped agents integrated with single system connectors.

The operational rule is straightforward: if the agent is not delivering quantifiable value within 30 days of initial deployment to a real operational channel, the scope must be simplified rather than expanded. ** This is not a judgment of development competence but rather a signal that the current scope-to-resource ratio has become misaligned. Value delivery failure indicates that either the scope remains too broad to achieve stability within the available development effort, or the integration points do not connect to work patterns that generate sufficient transaction volume to demonstrate impact. In either case, the remedy is to reduce scope further rather than to invest additional effort in the current design.

This discipline creates accountability structures that prevent laboratory research from consuming indefinite organizational resources while also forcing difficult conversations about scope alignment early in the adoption cycle, before significant resource commitments have been made.

9. Use Multiple Small Agents Instead of One Overloaded One

As operational processes expand and additional requirements pile up, assigning more responsibilities to a single agent compounds performance issues and makes the governance framework needed to maintain operational consistency much harder to manage. Each additional responsibility you layer onto an existing agent increases the dimensionality of the state space the agent must navigate, exponentially expanding the set of edge cases and behavioral scenarios that must be designed for, tested against, and monitored in production.

From an economic perspective, this multifaceted complexity imposes two distinct costs. First, development velocity decreases substantially as the cognitive burden of managing interdependencies between distinct responsibilities grows. When an agent handles both classification and task execution, modifications to classification logic require careful analysis of how those changes cascade through task execution behavior. Second, operational failure modes become increasingly difficult to isolate and remediate because a performance problem observed at the system boundary may originate from any of several distinct layers of responsibility.

The principle of agent specialization addresses this problem by establishing the discipline of splitting responsibilities across multiple focused agents as operational scope expands. Rather than expanding a single agent to handle routing decisions, classification decisions, domain-specific task execution, and document processing in sequence, you would instead deploy four distinct agents, each responsible for a single function. The routing agent receives incoming work and determines which specialized agent should handle the request. The classification agent processes the routed work and assigns it to the appropriate category within a predefined taxonomy. The domain-specific task agent performs the operational work within that category, calling back-end systems and generating results. The document processing agent extracts structured information from unstructured documents and prepares it for downstream task agents.

This decomposition yields multiple benefits that justify the additional engineering required to orchestrate multiple agents. Small, specialized agents reach production stability faster because each agent operates within a constrained state space with fewer edge-case combinations. Governance remains explicit and traceable because each agent has a single defined responsibility, making it straightforward to document expected behavior and audit actual behavior against that standard. Failure isolation becomes tractable because a performance degradation can be attributed to a specific agent component rather than requiring analysis across all bundled responsibilities. When a specific agent begins exhibiting unexpected behavior, the blast radius of potential impact remains constrained to the specific function that agent performs, rather than cascading through multiple dependent responsibilities.

Over extended operational timelines, this modular architecture provides additional economic value through reduced cost of capability evolution. When organizational requirements change, you can modify or replace a single specialized agent without requiring a redesign of the entire set of responsibilities. This flexibility allows organizations to adapt their agent ecosystem as operational priorities change.

10. Plan for Long‑Term Flexibility

Long-term organizational success with agent systems depends on architectural decisions that preserve future optionality without imposing excessive upfront complexity. Adoption frameworks and industry analysis show that organizations with modular architectures, rather than monolithic designs, have significantly lower total cost of ownership over multi-year operational timelines. The economic principle underlying this requirement is that modular systems distribute change costs across smaller component boundaries, whereas monolithic systems concentrate change costs across tightly coupled dependencies.

Your agent architecture should prioritize flexibility in integrating capabilities by establishing well-defined interfaces between agents and external systems, rather than embedding system-specific logic directly into agent instructions or prompts. This approach means that when your organization adopts a new CRM platform or replaces a document management system, you can update the system integration layer without requiring redesign of agent behavior specifications. Additionally, the architecture should remain protocol-driven, meaning that agents communicate with each other and with external systems through standardized APIs and message formats rather than through proprietary connectors. This discipline ensures that as your organization's technology infrastructure evolves, your agent ecosystem can adapt without requiring wholesale redevelopment.

The practical implication of this principle is that your initial agent deployment should incorporate extensibility patterns from the outset rather than deferring architectural considerations until later phases. When you define how an agent accesses your customer database, design that access pattern to accommodate a future change in the database platform without requiring modifications to the agent's core logic. When you establish how agents communicate with business systems, use standardized protocols and well-documented interfaces that would allow additional agents to access those same systems without requiring new connector development. This forward-looking engineering discipline imposes modest additional design effort during initial implementation but eliminates expensive rearchitecting work later as organizational requirements evolve and technology infrastructure changes.

Conclusion: The Fast‑Path to Production

Organizations that successfully transition agents from proof-of-concept phases into sustained operational deployment share a consistent pattern of implementation discipline. These ten principles represent a synthesis of organizational practices that have demonstrated measurable results across diverse operational contexts.

The foundational requirement is to anchor agent development in a specific, real operational process rather than pursue abstract experimentation. This grounding in actual business workflows ensures that agent capabilities connect directly to measurable organizational problems. Building on this foundation, maintaining scope discipline through narrowly defined initial agent responsibilities creates the conditions for rapid stabilization and early demonstration of operational value. The agent should then receive genuine operational input through the channels where work currently flows into the organization, exposing the agent to real data distributions and authentic user behaviors from the initial development phase.

Throughout the development cycle, applying production-grade governance controls, data access policies, and behavioral standards from day one eliminates the artificial gap between development and production environments. Simultaneously, integrating with at least one mission-critical system early in the development process forces the agent to operate under genuine operational constraints rather than remaining isolated in a laboratory environment. The development methodology should employ short iteration cycles measured in weeks rather than months, which compresses the feedback loop between hypothesis and evidence, enabling rapid identification of misalignments between designed behavior and operational reality.

Supporting this development rhythm requires establishing lightweight review processes calibrated to the urgency of decisions, and separating continuous operational assessments from episodic capability expansion decisions. Organizations must enforce time-bound value realization through a commitment to deliver measurable operational results within thirty days, which prevents indefinite deferral of production deployment and forces disciplined conversations about scope alignment. As operational requirements expand, maintaining modular architectures that distribute capabilities across multiple specialized agents rather than accumulating responsibilities within single agents preserves development velocity and simplifies operational governance. Finally, planning for long-term flexibility through well-defined interfaces and standardized protocols enables the agent ecosystem to adapt as organizational technology infrastructure and business requirements evolve.

These principles work together to create implementation patterns that compress the transition from conception to the delivery of operational value.

Microsoft 365 E7: Why Microsoft's New License Is a Logical Step for Agent‑Driven Enterprises

Holger Imbery — Sat, 14 Mar 2026 08:13:21 +0000

Summary Lede

Microsoft's announcement of Microsoft 365 E7 in March 2026 marks a watershed moment in enterprise technology strategy. For the first time in over a decade, Microsoft introduced a new top-tier enterprise license—not to add incremental features, but to fundamentally reconceptualize how organizations govern both human workers and autonomous AI agents as integrated components of the workforce. At $99 per user per month, E7 bundles Microsoft 365 E5, Microsoft 365 Copilot Wave 3 with agentic capabilities, the Microsoft Entra Suite, and the newly introduced Agent 365 control plane. This consolidation signals that AI agents have transitioned from experimental pilots to production-grade organizational resources requiring enterprise-grade identity, access, compliance, and auditability frameworks.

Why You Should Read This

If you lead enterprise technology strategy, manage cloud infrastructure, evaluate AI adoption roadmaps, or determine software licensing budgets, E7 represents a critical inflection point in how enterprises will architect their IT operating models over the next five years. This article explains not just what E7 includes, but why Microsoft built it—addressing the architectural gaps E5 left exposed as organizations scale agent deployment from hundreds of thousands to tens of millions of instances. You'll understand the economic logic, the governance infrastructure, and the strategic positioning underlying this licensing evolution, enabling you to make informed decisions about whether E7 aligns with your organization's agent deployment trajectory and control requirements.

Introduction: From Productivity Suites to Agent Platforms

Historical Context and Evolution

Microsoft's enterprise licensing strategy has traditionally centered on supporting productivity and organizational efficiency through cloud services and security infrastructure. The Microsoft 365 E5 tier, introduced in 2015, represented the established enterprise standard, designed to address the comprehensive security, compliance, productivity, and governance requirements of large organizations during the cloud adoption phase. For the next 11 years, E5 served as the highest-tier enterprise licensing option within the Microsoft 365 portfolio.

The March 2026 Announcement

On 9 March 2026, Microsoft announced the availability of Microsoft 365 E7, designated as the Frontier Suite, representing the first introduction of a new top‑tier enterprise license since the E5 tier was originally established in 2015. This announcement signals a deliberate architectural evolution in how Microsoft structures enterprise licensing and organizational governance at scale.

Composition and Technical Structure

The Microsoft 365 E7 offering, priced at $99 per user per month, consolidates multiple previously distinct components into a unified licensing structure. This bundled approach encompasses Microsoft 365 E5, Microsoft 365 Copilot, the complete Microsoft Entra Suite, and the newly introduced Agent 365 control plane. Each component addresses specific operational and governance requirements within the modern enterprise technology infrastructure.

Fundamental Architectural Shift

The introduction of E7 should not be interpreted as a simple price adjustment or repackaging of existing capabilities. Rather, E7 represents a substantive architectural shift in Microsoft's strategic positioning and technical philosophy. Microsoft is fundamentally repositioning Microsoft 365 from a platform optimized for human-centric productivity and security to a comprehensive control plane that manages and governs an integrated, mixed workforce comprising both human workers and autonomous artificial intelligence agents. This transition reflects evolving organizational requirements as enterprises move beyond pilot implementations of AI technologies toward systematic, organization-wide agent deployment at scale.

What Microsoft 365 E7 Actually Includes

Microsoft 365 E7 consolidates four distinct product components, previously available as separate subscription offerings, into a unified licensing structure. This architectural consolidation reflects Microsoft's strategic decision to bundle interdependent capabilities that are increasingly required for enterprise-scale deployment of autonomous AI agents. The following sections provide detailed technical specifications for each included component.

Microsoft 365 E5: Foundation Layer

Microsoft 365 E5 is the foundational component of the E7 licensing tier, providing core productivity, compliance, security, and identity management capabilities. These capabilities encompass the complete Microsoft Office productivity suite, including Exchange Online for messaging infrastructure, SharePoint Online for content management and collaboration, Teams for unified communications, OneDrive for business cloud storage, Microsoft Defender for comprehensive threat protection, Microsoft Intune for mobile and device management, Microsoft Purview for data governance and compliance, and Power BI Pro for business analytics and visualization. These capabilities provide the fundamental infrastructure required for enterprise productivity, data protection, and organizational governance.

Microsoft 365 Copilot (Wave 3): Advanced AI Integration

The E7 tier includes Microsoft 365 Copilot at the Wave 3 release level, which represents a significant evolution in AI integration across the Microsoft 365 application portfolio. Copilot is embedded across Microsoft Word, Excel, PowerPoint, Outlook, Teams, and the Loop workspace collaboration platform. Beyond traditional copilot assistance functions, Wave 3 introduces expanded agentic capabilities that enable autonomous planning, decision-making, and action execution. Additionally, Wave 3 extends multi-model support to integrate with multiple language model providers, specifically OpenAI and Anthropic Claude, providing organizations with flexibility in selecting the underlying AI model infrastructure based on specific organizational requirements, performance characteristics, or policy constraints.

Microsoft Entra Suite: Identity and Access Management

The E7 offering includes the complete Microsoft Entra Suite, an expanded product tier that goes beyond the standard Entra ID P2 offering. The Entra Suite encompasses advanced identity verification, comprehensive access governance frameworks, Zero Trust network access architecture for conditional connectivity, and sophisticated conditional access policy enforcement mechanisms. These capabilities provide an enterprise-grade identity management and access control infrastructure necessary to manage both human and non-human (agent-based) organizational identities within a unified framework.

Agent 365: Governance and Control Infrastructure

Agent 365 represents a newly introduced governance and security layer specifically designed to manage autonomous AI agents at an organizational scale. Agent 365 provides centralized inventory tracking across both Microsoft-native and third-party AI agent frameworks; comprehensive observability and monitoring capabilities; policy enforcement mechanisms specific to agent behavior and resource utilization; and lifecycle management functionality, including agent provisioning, update orchestration, and controlled retirement procedures. This component addresses the operational requirement for centralized governance of non-human autonomous entities executing within enterprise systems.

Economic Analysis and Bundle Composition

When these four components are purchased individually through separate licensing arrangements after July 2026, the aggregate monthly cost per user is projected to be approximately $117 USD. The E7 consolidation bundle is offered at $99 USD per user per month, representing an aggregate cost reduction of approximately 15–17% when compared to the sum of individually purchased components. This pricing structure reflects both the operational efficiency gains from unified licensing administration and Microsoft's strategic intent to incentivize adoption of the consolidated governance framework for agent deployment.

Why E7 Exists: E5 Was Built for the Cloud Era, Not the Agentic Era

Microsoft executives have been explicit that E5 was designed "pre‑agentic".

E5 assumes:

humans are the primary actors,
automation is largely scripted,
identities map cleanly to employees.

Modern enterprises increasingly violate all three assumptions.

AI agents today:

act autonomously,
access mailboxes, calendars, files, and APIs,
execute multi‑step workflows over time,
are often created outside central IT using low‑code or no‑code tools.

Agent 365: The Missing Control Plane Enterprises Have Been Lacking

Agent 365 is the genuinely new element in E7, and the main reason E7 is more than a repackaged bundle.
Agent 365 provides:

Centralized agent inventory across Microsoft and third‑party frameworks
Identity and access controls via Entra
Security monitoring via Defender XDR
Compliance and auditability via Purview
Lifecycle management (provisioning, update, retirement)

Crucially, Agent 365 does not build or host agents. It governs them.
Compute and execution remain consumption‑based via Copilot Studio, Azure AI Foundry, or partner platforms.
This mirrors how enterprises already separate:

application development,
runtime infrastructure,
identity and governance.

Why E7 Is a Good Move for Enterprises Leveraging Agents

It Normalizes Agents as Enterprise Identities

Microsoft is treating agents as digital workers, subject to the same identity, access, and policy frameworks as humans.
This is a necessary prerequisite for scaling agents beyond experimentation.

It Reduces Architectural Fragmentation

Prior to E7, organizations had to stitch together:

E5,
Copilot add‑ons,
Entra extensions,
emerging agent governance tools.

E7 consolidates these into a single, coherent enterprise architecture aligned with Zero Trust principles.

It Shifts AI from "Assistance" to "Execution".

Wave 3 of Copilot introduces agentic capabilities that plan, act, and execute, not just summarize or draft.
E7 provides the governance layer required to allow that execution safely.
Without E7‑level controls, many organizations would be forced to block these capabilities entirely.

It Aligns Cost Models with Reality

While $99 per user appears high, E7 reflects:

the true cost of enterprise security,
identity governance for both humans and agents,
reduced overhead compared to managing multiple SKUs.

Importantly, Microsoft is signaling that agents will be licensed like users, potentially with hybrid subscription and consumption models over time.

Microsoft 365 Enterprise Comparison

E5 vs E5 + Copilot vs E7 (Frontier Suite)

Status (March 2026)

Microsoft 365 E7 GA: May 1, 2026

Pricing shown is list price (USD, per user/month)

Consumption costs for building/running agents are not included in any plan

Dimension	E5	E5 + Copilot	E7 (Frontier Suite)
What’s included	Full Microsoft 365 E5 (Office apps, Exchange/SharePoint/OneDrive, Teams*, Defender, Intune, Purview, Power BI Pro). No Copilot, no Agent 365, and no full Entra Suite.	E5 (as left) plus Microsoft 365 Copilot add‑on. No Agent 365, no full Entra Suite.	E5 + Copilot + Entra Suite + Agent 365 in one SKU; positioned as the “Frontier Suite” for agent‑at‑scale scenarios.
List price (USD/user/month)	$60 (from July 1, 2026).	$90 (E5 $60 + Copilot $30).	$99 (bundle). With Teams‑excluded option reported at $90.45.
Copilot (Wave 3) agentic capabilities	Not included.	Included (via add‑on). Multi‑model (OpenAI + Anthropic) support arrives with Wave 3.	Included by default with Wave 3 agentic features (planning, acting across Microsoft 365).
Agent 365 (agent governance & control plane)	Not included.	Not included by default (can be added at $15/user/month).	Included; GA on May 1, 2026.
Entra Suite (beyond Entra ID P2)	Not included (E5 includes Entra ID P2 but not the broader Entra Suite).	Not included.	Included (e.g., Private Access, Internet Access, ID Governance/Protection, Verified ID).
Security & compliance posture for agents	Human‑centric controls only; no unified agent inventory/observability.	Adds creation/use of Copilot agents but without centralized agent governance unless Agent 365 is added.	Unified agent inventory, policy enforcement, auditability across Defender/Entra/Purview.
Bundle economics	Baseline plan.	A la carte add‑on model. E5 + Copilot = ~$90. Adding Entra Suite (+$12) and Agent 365 (+$15) pushes to ~$117.	~$99 vs ~$117 à la carte → ~15–17% discount; simpler procurement/governance.
Availability date	Available now.	Available now (Copilot GA prior to E7; Wave 3 rolling out).	GA May 1, 2026.
Who it fits	Organizations prioritizing core productivity/security without near‑term agent scale‑out.	Teams piloting Copilot or limited agentic use cases, willing to bolt on governance later.	Enterprises standardizing on agents (cross‑department), requiring identity‑first governance, zero‑trust access, and consolidated risk controls.

Notes

Teams availability depends on regional licensing rules; E7 is also offered as a *“without Teams”** SKU in some regions.*
Agent execution costs (LLM tokens, orchestration runtime, long‑running workflows) are not included in any license and must be budgeted separately.
E7 is the first Microsoft 365 SKU designed explicitly for the agentic AI era, not just productivity.

Important Caveat: E7 Is Not the Full Cost of an Agentic Enterprise

E7 does not include:

agent execution compute,
LLM consumption,
orchestration runtime costs.

These remain variable and are billed separately via Copilot Studio, Azure AI Foundry, or partner services.

Enterprises should therefore view E7 as:

the governance and control foundation, not the entire AI budget.

Conclusion: E7 as an Architectural Statement

Microsoft 365 E7 represents a significant shift in how enterprises conceptualize their licensing strategy and operational architecture. Rather than functioning primarily as a licensing vehicle, E7 serves as a declaration that Microsoft 365 is positioned as the foundational operating system for enterprises that intend to operate within an agentic computing paradigm.

For organizations that anticipate the need to execute a comprehensive deployment strategy involving autonomous AI agents across multiple business functions, integrate these agents into mission-critical business processes, and maintain the requisite levels of security governance, compliance attestation, and auditability, the scope and depth of capabilities provided by E7 are not superfluous. Instead, these capabilities represent structural and architectural necessities that must be addressed to enable safe and controlled agent deployment at an organizational scale.

The evolution from E5 to E7 reflects a fundamental recalibration of enterprise platform design philosophy. The E5 licensing tier was optimized and engineered for the cloud-centric era, where enterprises sought to modernize their infrastructure, data management, and collaboration mechanisms through cloud-native services. E7, by contrast, is optimized for an organizational context in which the workforce composition includes both human workers and autonomous AI agents, each requiring appropriate identity governance, access controls, security monitoring, and compliance instrumentation within an integrated control plane.

This architectural shift acknowledges that managing agents as first-class organizational entities—rather than as peripheral or experimental capabilities—requires the same level of systematic governance, policy enforcement, and observability that enterprises have come to expect from their core identity and security infrastructure.

Introducing MATE: A Modular Testing Environment for AI Agents

Holger Imbery — Sat, 07 Mar 2026 08:07:02 +0000

Summary Lede

As AI agents become integral to business processes, reliable and repeatable testing is essential for confidence in deployment. This article introduces the Multi-Agent Test Environment (MATE) – an enterprise-grade framework for automated testing of AI agents across platforms and frameworks – and explains how its modular design addresses key challenges in agent testing. We explore why testing AI agents is critical, delve into MATE's architecture and features, compare MATE with alternative testing approaches, and outline MATE's roadmap including red-team testing, enhanced cloud deployment, and support for emerging agent frameworks.

Importance of Testing AI Agents

AI agents built with Microsoft Copilot Studio are powerful but complex systems. They combine natural language understanding, generative AI, and business logic, often operating in critical scenarios (customer support, data retrieval, workflow automation, etc.). Ensuring these agents work correctly and safely under diverse conditions is as important as testing traditional software – if not more so. Key reasons why rigorous agent testing is essential include:

Reliability and Consistency: Unlike deterministic software, AI agents can produce different answers to the same question due to their probabilistic nature. Without structured tests, one might only catch issues by manually typing questions and hoping for the right answer, a fragile approach. Automated testing provides consistency – the same test can be run repeatedly to ensure the agent’s behavior remains reliable after updates.
Enterprise-Grade Quality: In enterprise deployments, an untested agent can lead to incorrect or even unsafe outputs, damaging user trust or violating compliance. Ad-hoc testing that “relies on intuition instead of structured testing” doesn’t scale for enterprise needs. Organizations require repeatable, at-scale test processes to validate that agents meet quality standards (accuracy, relevance, safety) consistently before and after release.
Complex Multi-turn Interactions: Copilot Studio agents often handle multi-turn conversations, maintaining context across multiple user and agent turns. Testing these multi-step dialogues manually is time-consuming and error-prone. Automated test suites allow developers to simulate complex conversation flows (with varying user inputs, branching dialogs, tool invocations, etc.) and verify the end-to-end behavior in one run. This ensures that the agent can handle scenario-based conversations robustly, from greeting to task completion.
Nondeterministic and Generative Responses: When agents use generative AI capabilities, they might produce creative or unexpected phrasing. Verifying such responses is not as simple as exact string matching. Effective testing must evaluate responses on semantic correctness, completeness, and compliance, even if wording varies. This introduces a challenge: how do you automatically judge an AI-generated answer’s quality? We’ll see how MATE tackles this with an AI-based “judge” component.
Frequent Updates and Continuous Integration: Agents are rarely static – their underlying prompts, skills, and knowledge sources evolve. Without automation, re-testing the agent after each change or on a schedule (for example, to catch drift or regressions) would be prohibitively labor-intensive. A good agent testing framework enables continuous integration (CI) pipelines and nightly runs, so that any breaking change or quality degradation is caught early. This is crucial for scaling up the number of agents in production while keeping maintenance overhead low.
Transparency and Debugging: When a test fails, developers need insights into why. For example, did the agent retrieve the wrong data because of an intent misclassification? Or did it produce a partially correct answer that was marked as a failure due to a strict check? Good testing tools provide detailed reporting – conversation transcripts, logs, and metrics – to help pinpoint the root cause of failures. This accelerates debugging and continuous improvement of the agent.

In summary, robust testing of agents is the linchpin for trustworthy AI deployments. It allows teams to validate functionality, accuracy, robustness, and safety in a systematic way. This need has driven Microsoft to introduce solutions like the Power CAT Copilot Studio Kit (a Power Platform solution for agent testing) and, more recently, the built-in Agent Evaluation feature in Copilot Studio (now in preview). However, each solution comes with certain limitations or prerequisites, which the new MATE aims to overcome. Before comparing approaches, let’s first introduce MATE and how it works.

Introducing MATE: A Modular Testing Framework for AI Agents

Link to MATE GitHub Repository

Link to MATE Wiki

Multi-Agent Test Environment (MATE) is an internal project and framework designed to provide automated, comprehensive testing for AI agents, initially focusing on Microsoft Copilot Studio agents. MATE was created to address the challenges above by combining enterprise-grade tooling with a modular, extensible architecture. In essence, MATE allows developers and testers to connect to a running Copilot Studio agent, simulate conversations, evaluate the agent’s responses against expected outcomes using AI, and produce detailed metrics and reports – all in an automated fashion.

MATE’s approach can be seen as bringing many of the benefits of the Copilot Studio Kit into a single, code-first testing environment. Rather than a Power App solution, MATE is a pure .NET 9 application that you can run in a container stack. This design choice means MATE operates outside the constraints of the Power Platform, giving developers more flexibility in how and where they run their tests.

Let’s break down how MATE works and how it addresses key testing challenges:

Direct Line Integration (Live Agent Testing): MATE connects to the agent through its Direct Line API endpoint. This is the same interface used by real chat channels (like Teams or a custom web chat). By using Direct Line, MATE ensures it's testing the deployed agent exactly as end-users experience it. The tool can send a sequence of user messages and receive the agent’s replies in turn, thereby automating full multi-turn conversations. This addresses the challenge of multi-turn flows by allowing complex scenario scripts to be executed automatically. It’s effectively like an automated “test chat” but running dozens or hundreds of predefined conversations unattended.
Test Case Definition and Multi-turn Flows: In MATE, you can define a test case with multiple steps of user input (representing a conversation) and the expected outcomes. Expected outcomes can include:
- Expected Intent and Entities – i.e., which topic or action the agent should trigger and which key data (entities) it should extract.
- Acceptance Criteria – specific conditions that constitute a pass/fail for the test (for example, certain keywords must appear in the answer, or a certain API call must be made).
- Reference Answer – an ideal answer text or outline for comparison. Each test case can be labeled with a priority or category, useful for organizing large test suites (e.g., “P1 critical flows”, “Edge cases”, etc.). By supporting multi-step conversations in test cases, MATE ensures you can test end-to-end agent behavior, not just isolated single-turn Q&A.
“Model-as-a-Judge” Evaluations: One of MATE’s most powerful features is using an AI model to evaluate the quality of the agent’s response. Rather than relying only on hard-coded checks (exact matches or simple contained keywords), MATE sends the agent’s answer along with the reference answer and validation criteria to a Large Language Model (LLM) – for instance, an Azure OpenAI GPT-4 model – which acts as an impartial judge. This AI Judge scores the response across multiple evaluation dimensions:
- Task Success: Did the agent fulfill the user’s request or solve the user’s problem?
- Intent Match: Did the agent correctly understand what the user was asking for?
- Factuality: Is the information provided true and accurate (no hallucinations or incorrect data)?
- Helpfulness/Completeness: Is the answer complete, well-structured, and does it address the user’s need effectively?
- Safety/Compliance: Does the response avoid policy violations (no sensitive data exposure, no disallowed content)? Each of these dimensions is scored (e.g. 0.0 to 1.0), and MATE can apply configurable weightings to decide if a test passes or fails overall. For example, you may require 0.9+ on Task Success and Intent Match, tolerate a lower score on style metrics like Helpfulness, and demand a perfect score on Safety. This approach directly tackles the challenge of evaluating nondeterministic generative answers: even if the agent’s wording differs from the expected answer, the AI Judge can still determine that the answer is essentially correct and useful. Conversely, if the agent’s response is irrelevant or contains errors, the AI Judge will assign low scores, causing the test to fail. This method provides a nuanced, context-aware evaluation that traditional automated tests struggle to achieve. (Internally, the AI Judge uses prompt-based prompting of an LLM with the expected answer or criteria to get these scores.)
In Addition, MATE also supports other judge types, such as:
- RubricsJudge – A fully deterministic judge that evaluates responses using explicit rules such as Contains, NotContains, and Regex, making it ideal for compliance, safety, and reproducible pass/fail checks.
- HybridJudge – A cost‑efficient combination judge that first gates responses with deterministic rubrics and then applies an LLM for deeper qualitative scoring only where needed.
- CopilotStudioJudge – A Copilot‑Studio‑specific LLM judge that is citation‑ and grounding‑aware, aligning evaluations with Copilot Studio’s default reasoning and response patterns:
- GenericJudge – A lightweight, zero‑cost judge based on simple keyword and regex matching, intended for fast smoke tests and offline or CI scenarios
Automated Test Generation from Documentation: Authoring a comprehensive set of test cases can be labor-intensive. MATE addresses this by allowing you to upload documents (PDFs or text files) that are relevant to your agent’s domain or knowledge base. It then automatically:
- Extracts text content from the documents (using a PDF parser).
- Indexes and chunks the content for semantic analysis (using a Lucene-based index).
- Generates potential questions and answers from the content using an LLM. The outcome is a set of suggested Q&A pairs or even multi-turn conversation scenarios derived from the documentation. For example, if you upload a product FAQ PDF, MATE can generate likely customer questions and the correct answers from that PDF. These can be reviewed and added to your test suites. This feature helps broaden test coverage automatically, ensuring the agent is tested on real knowledge it’s supposed to have, and catching gaps where it might not respond correctly. It’s an intelligent way to keep tests in sync with content. (Notably, Copilot Studio Kit in the Power Platform also introduced an AI-based test generation in preview, which uses the agent’s topics and knowledge to generate example questions. MATE provides a similar capability but on external docs and with full control of the generated cases.)
Detailed Reporting and Analysis: After executing tests, MATE provides rich metrics and logs. In the Web Dashboard, you can see overall pass rates, success trends over time, and drill down into individual test runs. Each test run retains the transcript of the conversation and the scores for each evaluation dimension, so you can inspect exactly where a particular test failed. This addresses transparency: instead of just “Test 5 failed”, you can see that it failed because, say, Factuality scored low (perhaps the agent gave a wrong detail), and even read the conversation to diagnose the issue. MATE’s Runs view lets you compare results between runs – useful for spotting regressions after an update. All test data (test cases, results, transcripts, etc.) are stored in a local PostgreSQL database for quick retrieval and can be queried or exported for additional analysis.
Web UI and CLI for Different Use Cases: MATE offers two interfaces:
- A Web Application (built with ASP.NET Blazor Server) for an interactive experience. This is ideal for exploratory testing, configuring your test suites, and reviewing results. The UI includes a setup wizard for initial configuration (entering your agent’s Direct Line credentials and your AI model info) to generate the necessary settings file. Testers can use the web UI to kick off test runs on-demand, monitor progress, and view results in real time.
- A Command-Line Interface (CLI) tool for automation. The CLI allows you to run tests as part of scripts or pipelines. For example, you can incorporate dotnet run --suite "Regression Suite" into a DevOps or GitHub Actions pipeline, so that whenever the agent’s bot is updated or its content changes, the test suite runs and verifies everything still works. The CLI returns an exit code indicating success or failure (0 if all tests passed, non-zero if any test failed), which CI systems can use to pass/fail a build. This enables true CI/CD for AI agents – a failed test can halt a deployment, preventing flawed agent versions from going live.
Containerized and Extensible Architecture: MATE is designed to be run in a self-hosted manner, giving teams full control. It doesn’t require a SaaS backend or a Dataverse environment – you just need a machine that can reach the internet for calling the agent service and the AI model endpoint. This avoids many of the Power Platform licensing constraints associated with the Copilot Studio Kit (discussed later). The architecture is modular by design, with separate components (projects) for domain logic, data storage, core services, web UI, and CLI. This modularity not only enforces clean separation of concerns, but also sets the stage for supporting multiple types of agents in the future. In fact, MATE’s roadmap includes extending support to other agent frameworks beyond Copilot Studio – the core logic (test execution, AI judging, etc.) can be adapted to different agent APIs by swapping out the integration layer. Early code commits already hint at multi-agent support being developed and even an upcoming “red teaming” module for adversarial testing (there are structural hooks in the codebase for this, though the feature is not yet implemented). This means MATE is not a one-off tool but a growing platform for comprehensive AI agent testing across the board.

MATE Architecture at a Glance

Internally, MATE is built with a modern software architecture using the latest Microsoft stack:

.NET 9 with C# – providing performance and cross-platform support.
ASP.NET Core Blazor Server for the web front-end – delivering a rich interactive UI for managing tests and viewing results.
Entity Framework Core (with PostgreSQL) – for the local database that stores test cases, results, transcripts, etc., ensuring persistence without requiring an external DB server.
Azure AI OpenAI SDK – to connect to the AI Judge model hosted on Azure’s AI services (Azure OpenAI “Foundry”). This is how MATE queries an LLM for evaluation of answers.
Lucene.NET – used for full-text indexing in the document-driven test generation feature, to find relevant content in uploaded docs for question generation.
PDF processing libraries (e.g., UglyToad.PdfPig) – to extract text from PDF documents for test generation.
Serilog – for structured logging of events and errors, helping with diagnosing issues in test executions.

The solution is divided into several components (projects) reflecting a modular design: a Domain layer for core models and interfaces, a Data layer for database access, a Core services layer (implementing the judge logic, execution engine, etc.), a WebUI for the front-end, and a CLI project for the command-line interface. This modularity makes it easier to maintain and extend specific parts (for example, adding a new agent connector could be done by introducing a new service in the Core or a new API integration, without touching the UI or data layers).

Overall, MATE is engineered to be a scalable, extensible test harness for AI agents. It’s currently focused on Copilot Studio agents, but its principles apply broadly. Next, we’ll compare MATE to the Copilot Studio Kit – the established testing solution from Microsoft’s Power CAT team – to understand their differences and use cases.

Comparing MATE with Copilot Studio Kit

Microsoft’s Power CAT Copilot Studio Kit is an existing solution aimed at testing and managing Copilot Studio agents. It’s a Power Platform solution (managed package) that provides a canvas app or model-driven app interface, along with Dataverse entities and Power Automate flows, enabling test case creation, automated test runs via the Direct Line API, and analytics (such as conversation transcripts, dashboards, etc.). The Copilot Studio Kit was instrumental in early adoption of agent testing – it allowed makers to do things like bulk import test cases (via Excel), run them from a UI, and even integrate with Azure DevOps pipelines via Power Platform build tools.

However, the Copilot Studio Kit has some inherent characteristics stemming from its Power Platform foundation. Below is a comparison of MATE vs. Copilot Studio Kit across key dimensions:

Aspect	MATE (Multi-Agent Test Environment)	Copilot Studio Kit (Power CAT)
Deployment Model	.NET application (containarized). Runs locally or in cloud; launched via web UI or CLI on demand.	Power Platform managed solution. Deployed to a Dataverse environment; accessed via Power Apps interface.
Licensing & Costs	source-available - CC BY-NC 4.0 . Requires .NET runtime and an Azure OpenAI endpoint (for AI Judge) which may incur usage costs. No special Power Platform licensing needed beyond having a Copilot Studio agent to test.	Provided by Microsoft Power CAT as a sample solution (available on GitHub). However, requires Power Platform premium licenses: a Dataverse environment, and for certain features, AI Builder credits (for generative answer analysis). Usage of Dataverse and Power Automate in the kit might consume capacity or require specific licenses.
Technology Stack	Modern .NET 9 stack; Blazor Web UI, CLI tool, local PostgreSQL DB. Integrates with Azure services (OpenAI) for evaluation. Highly customizable and extendable by developers (source code available).	Low-code Power App + Dataverse. Relies on standard Power Platform tech (model-driven app or canvas app, Dataverse tables, Power Automate flows, AI Builder for some AI tasks). Customization is limited to what Power Platform allows.
Test Creation	Supports manual creation of test cases via UI or by defining JSON/CSV, etc., and auto-generation of test cases from documents using LLMs. Test cases can include multi-turn dialogues in one case. Organized into test suites for batch execution.	Supports manual test case input (through the app or via Excel import/export). Also supports multi-turn test cases and offers some AI-assisted generation of test questions from agent topics/knowledge (in Preview, via the Agent Evaluation integration). Test cases stored in Dataverse; grouping of tests supported (by test set).
Test Execution	Runs tests externally by connecting to the agent’s Direct Line channel (or Web Channel with secret & bot ID). Offers a CLI for headless execution (suitable for CI pipelines) and a web interface for interactive runs. Test results are stored locally and displayed in the web UI with analytics.	Executes tests through Copilot Studio’s Direct Line API as well, orchestrated by Power Automate flows under the hood. Typically run on-demand from the app’s interface. There is integration for pipelines via Power Platform build tools, though this is more complex to set up. Results are stored in Dataverse and can be viewed via in-app dashboards or exported.
Evaluation Methodology	AI-driven semantic evaluation: uses a GPT-based AI Judge to score responses on multiple quality dimensions (task success, intent match, factual correctness, etc.). This allows flexible, semantic comparisons rather than simple exact matches. Configurable pass thresholds provide fine-grained control. Also supports explicit pass/fail rules (acceptance criteria) where needed.	Rule-based and some AI: supports exact or partial response matching, checking for expected keywords or presence of attachments, etc. For generative answers, the kit uses AI Builder to compare the agent’s answer with a reference answer for similarity. It also retrieves telemetry from Application Insights to help explain failures. Plan validation tests examine the agent’s action plan against expected tools (for orchestration scenarios).
Modularity & Extensibility	Designed to be modular and extensible. The core can be extended to new agent types (plans to support other AI agent frameworks are in progress). The evaluation component (AI Judge) can be pointed to different models or adapted with different prompts. Being source-available, organizations can modify or extend MATE (e.g., add custom evaluation metrics, integrate with other data sources) to fit their needs.	Focused scope, limited extensibility. The Copilot Studio Kit is specific to Copilot Studio agents and deeply tied to the Power Platform environment structure. It’s not architected to test arbitrary other agents. Customizing it generally means modifying the Power App or creating new Dataverse fields/flows, which requires Power Platform expertise.
Data & Infrastructure	Stores test artifacts and results in a local database within the application. No cloud infrastructure needed to get started; data stays within the user’s environment. For scaling up, the application could be hosted on a server or in Kubernetes (containerization support is under development). Because it’s self-hosted, data sovereignty and privacy can be managed internally.	Relies on Dataverse for storing tests and results, and optionally uses other services (App Insights, SharePoint, etc.) for logs and knowledge management. This provides seamless integration if you’re already within the Microsoft ecosystem, but it requires that all data be in the Power Platform cloud.

Table: Comparison of MATE vs. Copilot Studio Kit across key aspects of testing functionality and usage.

As seen above, MATE and Copilot Studio Kit share the same goal – improving agent quality through automated testing – but they differ in implementation approach. MATE is more developer-oriented, offering flexibility, openness, and extensibility, whereas the Copilot Studio Kit is maker-friendly, integrated in the Power Platform with a ready-to-use interface but comes with platform constraints.

From a Microsoft perspective, the Copilot Studio Kit was a bridge solution to empower agent creators with testing capabilities before deeper platform features were available. Now, with Agent Evaluation built directly into Copilot Studio (currently in preview), some capabilities of the kit are being absorbed into the product itself – for instance, AI-generated test queries and built-in execution of test sets. Still, the Kit provides additional tooling (like dashboards, inventory, governance features) that are useful in complex environments.

see articles:

Ship Copilot Studio Agents with Confidence: Master Automated Testing with the Copilot Studio Kit

Testing Copilot Studio Agents: Copilot Studio Kit vs. Agent Evaluation (Preview)

MATE, on the other hand, is an independent effort to provide a robust testing harness that can evolve fast and go beyond what the closed-source product features offer. It is not limited by Power Platform’s boundaries (for example, one could imagine integrating MATE with other LLM evaluation criteria, or hooking it up to monitor backend APIs invoked by the agent). Additionally, MATE’s modular nature means it could incorporate other agent types into the same testing dashboard. For example, if you have a fleet of different AI bots – some built in Copilot Studio, some using Azure OpenAI Orchestration, some third-party – MATE could theoretically be extended to test them all in one place, whereas the Copilot Studio Kit is only for Copilot Studio agents.

When to use which? If you are a Power Platform maker or IT admin who wants a straightforward, supported way to test Copilot Studio agents and you’re already comfortable with Power Apps and Dataverse, the Copilot Studio Kit is a solid choice. It integrates nicely with the environment (and your data, logs, etc.) and doesn’t require coding to use. However, you’ll need the necessary licenses and some patience to configure the environment, and you won’t be able to easily customize how tests are evaluated beyond what Microsoft provides.

If you are a developer or dev team looking for a more flexible, code-driven approach – especially if you want to integrate agent testing into a DevOps pipeline or extend testing to specialized scenarios – MATE is very appealing. It does require .NET and some setup, but it gives you full control. You can run it locally for rapid iteration, include it in automated builds, and tweak it to your needs. There’s also no dependency on having a Power Platform environment or any particular license. You do need access to an Azure OpenAI service (or you could swap in another LLM API if desired) to leverage the AI judge, but that is relatively straightforward for most enterprise scenarios.

Roadmap and Future Enhancements in MATE

MATE is an evolving project, and there are several notable enhancements on the roadmap:

Support for Additional Agent Types: As of now, MATE supports testing Microsoft Copilot Studio agents exclusively, because it specifically uses Copilot’s Direct Line API and related assumptions (like the concept of “topics” and Dataverse knowledge base) in its current version. However, the architecture is being extended to accommodate other agent platforms. Future versions are expected to introduce modules for other agent types – for example, the ability to test Microsoft Agent Framework agents, or even agents built with entirely different frameworks. This will broaden MATE’s applicability across various “agentic AI” solutions used within Microsoft and beyond, making it a one-stop testing hub for heterogeneous AI systems.
Integrated Red Teaming: In addition to “blue team” style functional testing (checking that the agent does what it’s supposed to), MATE aims to incorporate “red teaming” capabilities. Red teaming in AI refers to attacking or stress-testing the agent with malicious or unexpected inputs to probe its defenses and safety measures. This can include testing the agent’s response to prompt injections, inappropriate content requests, or attempts to trick the agent into breaking rules. The goal is to ensure the agent is robust against misuse or adversarial users. The MATE codebase already contains the foundation for a Red Teaming module, but this is currently just a skeleton (non-functional in the current release). Once completed, this feature will allow users to run a suite of adversarial tests (perhaps using predefined malicious prompts or common attack patterns) against their agents and get a report on vulnerabilities or policy compliance issues. This is a critical part of AI system testing, especially for enterprise scenarios, and its inclusion will differentiate MATE further by offering a more comprehensive safety evaluation than what is currently possible with the Copilot Studio Kit or built-in Agent Evaluation (which, so far, focus on correctness and performance rather than adversarial robustness).
Cloud and Scalable Deployment: Presently, MATE runs as a local docker stack. Looking forward, the project plans to simplify deployment on Azure, likely via containerization. Kubernetes support is on the roadmap, meaning you might be able to deploy MATE as a set of containers (web app, background worker, etc.) in an AKS (Azure Kubernetes Service) or similar environment. This will enable team-wide usage at scale – multiple testers or developers could share a MATE instance, run tests concurrently, and store results in a central location, much like a web application service. Cloud deployment will also facilitate integration with other services (for example, connecting to Azure DevOps for automatic test triggers, or scaling out the AI Judge component).
UI/UX and Usability Improvements: As an source-available project, MATE will continue to refine its user interface and ease of use. Features on the horizon could include richer test editing experiences (perhaps a visual conversation flow editor), more analytics dashboards (trend of agent performance over time, flakiness of certain tests, etc.), and integration with agent design tools (for example, pulling in agent topics or suggesting tests based on recent real user conversations – aligning with how built-in Agent Evaluation reuses Test Pane interactions).

Conclusion

Testing AI agents is no longer optional – it’s a necessity for any organization that wants to confidently deploy AI solutions. Agent Development Solutions empower the creation of sophisticated AI Agents, but ensuring these agents function correctly, safely, and efficiently requires going beyond manual testing or one-off trials. This is where testing frameworks like Copilot Studio Kit and MATE come into play.

MATE (Multi-Agent Test Environment) represents a next-generation approach to agent testing. It addresses the limitations of earlier tools by adopting a fully modular, code-first architecture that can keep pace with the rapidly changing AI landscape. By using MATE, developers and testers gain the ability to thoroughly automate conversations with their agents, evaluate responses with the help of AI, generate tests from existing knowledge, and integrate all this into continuous delivery pipelines. The outcome is a higher degree of assurance that your Copilot Studio agent will perform as expected when it’s in production – responding correctly to user queries, using the right tools, and staying within the guardrails.

In comparison to the Power Platform-based Copilot Studio Kit, MATE offers more flexibility, extensibility, and independence. You won’t be constrained by specific licensing or environment setups, and you can tailor the tool to your needs. On the other hand, it’s a more technical solution that may require developer effort to set up and maintain, whereas the Copilot Studio Kit is more turn-key if you’re already within Microsoft’s ecosystem. It’s encouraging to see both approaches available, as they cater to different audiences.

Ultimately, MATE’s importance goes beyond just testing Copilot Studio agents. It signifies an evolving philosophy in the AI agent world: that testing and evaluation should be first-class citizens in the development lifecycle of AI systems, just as they are in traditional software development. With AI models and agents becoming increasingly central to applications, tools like MATE help ensure we can trust these systems through systematic validation. MATE’s deep integration of AI for testing (using an AI to test another AI) is an innovative approach that can significantly enhance the rigor of evaluations.

In summary, MATE enables teams to ship Copilot Studio agents (and, in the future, other AI agents) with greater confidence. It provides the means to catch issues early, improve agents iteratively based on test feedback, and guard against regressions as agents evolve. By combining the power of automation with the wisdom of AI judging, MATE exemplifies a “test smarter” strategy for the era of generative AI – ensuring that our intelligent agents are not only smart, but also reliable, safe, and effective when they go to work for us.

Testing Copilot Studio Agents: Copilot Studio Kit vs. Agent Evaluation (Preview)

Holger Imbery — Sat, 14 Feb 2026 07:09:12 +0000

Summary Lede

Copilot Studio agents deserve testing at scale—but which tool fits your team? Agent Evaluation brings lightweight, AI-powered checks into the Studio authoring UI for rapid iteration, while Copilot Studio Kit delivers enterprise-grade multi-turn validation, plan verification, and telemetry for production gates. This guide cuts through the hype and shows you exactly when to reach for each tool—and why the best teams use both together.

Why Read This Choosing the right testing tool can make or break your Copilot Studio agent quality strategy. This article goes deeper into how Copilot Studio Kit and Agent Evaluation complement—and differ in—their testing approaches. If you’ve read my earlier piece on Ship Copilot Studio Agents with Confidence: Master Automated Testing with the Copilot Studio Kit (2026-01-31), which focused on Kit capabilities alone, this comparison will help you decide when to use Kit, when to use Agent Evaluation, and when to use both together. Perfect for teams scaling from dev iteration to production release gates.

Validation vs. Verification in AI Agent Testing

Before diving into the two tools, it’s worth clarifying a critical distinction in quality assurance: verification and validation address different concerns when testing conversational agents. Verification answers the question “Did we build it right?” It focuses on ensuring your agent behaves as designed, follows the intended logic flows, and produces outputs that meet specifications. In practical terms, verification tests check that your agent’s instructions are correctly implemented, that topics route to the right handlers, and that expected responses are generated for known inputs. Validation , by contrast, asks “Did we build the right thing?”—it assesses whether your agent actually meets user needs, provides accurate and helpful information, and performs well in real-world scenarios. Validation is inherently broader and more subjective; it often involves human judgment, user feedback, and quality metrics like relevance, groundedness, and user satisfaction. The Copilot Studio Kit excels at verification through structured test cases and telemetry analysis, while Agent Evaluation bridges both worlds by enabling quality assessments that combine deterministic checks (verification) with AI-based graders that approximate validation concerns. Most production-grade agent programs require both verification to catch implementation bugs and ensure consistency, and validation to ensure the agent truly solves the problem it was designed to solve.

Microsoft provides two complementary approaches to test Copilot Studio agents:

Copilot Studio Kit (Power CAT): A Dataverse-backed, installable toolkit that runs batch tests via the Direct Line API , enriches results with Application Insights / Dataverse , and supports multi‑turn and plan validation scenarios, plus Excel import/export and dashboards. (Kit overview, Test capabilities, Run tests, GitHub repo)
Agent Evaluation (preview): A built‑in Copilot Studio experience for creating evaluation sets , generating prompts with AI, selecting test methods (exact/partial, similarity/intent, AI‑judged quality metrics), and running structured checks directly from the authoring UI. (Microsoft Copilot Blog announcement, Preview documentation, What’s new)

Used together, Agent Evaluation accelerates inner‑loop quality checks during design, while Copilot Studio Kit anchors enterprise‑grade testing, telemetry, and CI/CD gates prior to production.

What Each Option Provides

Copilot Studio Kit (Power CAT)

The Copilot Studio Kit is a comprehensive, installable testing framework built on Microsoft’s Power CAT (Power Customer Advisory Team) architecture. It extends Copilot Studio’s native capabilities by providing infrastructure for large-scale, automated agent validation.

How it works:

The Kit operates through a structured workflow: you define test cases within Agents , organize them into Test Sets , and execute these sets as Test Runs on demand or via scheduled processes. Test execution occurs through the Direct Line API , which simulates authentic user interactions with your published agent. Results are enriched with telemetry from Application Insights and Dataverse , capturing detailed diagnostic information, including which topics were invoked, confidence scores for intent matching, and end-to-end latency measurements. This multi-layer instrumentation enables root-cause analysis of test failures and performance anomalies.

Test types:

The Kit supports multiple validation approaches to accommodate different agent behaviors.

Response Match tests verify that agent outputs match expected text or patterns.

Attachment/Adaptive Card tests validate that rich response elements (cards, files, structured outputs) are generated correctly.

Topic Match tests confirm that conversations trigger the intended dialog flows.

Generative Answers tests assess responses from generative models embedded in agents.

Beyond single-turn exchanges, the Kit handles Multi‑turn conversations - sequences of user inputs and agent responses within a single session - and Plan validation , which verifies that generative orchestration components select and invoke the correct tools, actions, and connected agents in the proper sequence.

Run management:

Once you execute a Test Run, the Kit provides tools for iterative validation work. You can duplicate previous runs to establish baselines, re-run enrichment steps (regenerating Application Insights correlations and Dataverse transcripts) without re-executing the entire test, and analyze aggregate pass/fail statistics or drill down into individual case results to identify failure patterns and trends.

Artifacts and maintenance:

The Kit’s source code, configuration schemas, and runbooks are maintained in the Power CAT GitHub repository, where you can review implementation details, file issues, and contribute improvements.

Agent Evaluation (Preview) in Copilot Studio

Agent Evaluation is a feature integrated directly into the Copilot Studio authoring environment, designed to facilitate the systematic evaluation of conversational agents during development. Unlike external toolkits or custom test harnesses, Agent Evaluation operates natively within the Studio UI, allowing authors to create, manage, and execute test cases without leaving the design context. This approach is intended to reduce the barrier to agent testing and encourage more frequent, iterative validation within the authoring workflow.

How it works: Agent Evaluation introduces the concept of evaluation sets , which are collections of test prompts and expected responses. Authors can create these sets manually, import them from CSV files, or generate them automatically using AI based on the agent’s metadata or knowledge sources. This flexibility supports both targeted test case authoring and rapid bootstrapping of test coverage. Additionally, logs from the Test Pane - the interactive testing interface within Copilot Studio - can be reused to create evaluation sets, streamlining the conversion of ad hoc tests into structured checks.

For each test case, authors define the user prompt, the expected answer, and the success criteria. The evaluation process can then be run directly within Studio, producing aggregate and per-case results that are immediately accessible for inspection and troubleshooting. This tight integration with the authoring environment is intended to support a rapid feedback loop, enabling authors to identify and address issues early in the development cycle.

Test methods: Agent Evaluation supports several methods for assessing agent responses:

Lexical matching : This includes exact and partial string matching between the agent’s response and the expected answer. Lexical methods are useful for scenarios where deterministic outputs are required, such as FAQ responses or compliance-driven answers.
Similarity/Intent matching : These methods use semantic similarity algorithms or intent classification to determine whether the agent’s response is sufficiently close in meaning to the expected answer, even if the wording differs. This is particularly relevant for conversational agents that employ generative models or paraphrasing.
AI-judged quality metrics : Agent Evaluation can apply AI-based graders to assess qualitative aspects of responses, such as relevance, completeness, and groundedness. These metrics provide a more nuanced view of agent performance, especially in open-ended or knowledge-intensive scenarios.

Notes: Please note that Agent Evaluation is currently in public preview. As such, its feature set, supported test methods, and grading criteria may evolve based on user feedback and ongoing development. The tool is not intended to replace Responsible AI (RAI) or safety reviews, which remain essential for production deployments of conversational agents. Instead, Agent Evaluation is best viewed as a complementary capability for improving agent quality during the iterative design and testing phases. (Preview docs)

Side‑by‑Side Comparison

Dimension	Copilot Studio Kit	Agent Evaluation (preview)
Workspace	Separate model‑driven app (Dataverse) you install	Built directly into Copilot Studio authoring UI
Test data authoring	Dataverse entities; Excel import/export	Create/import CSV; reuse Test Pane; AI‑generate prompts
Execution	Via Direct Line API with cloud flows/enrichment	Run directly in Studio
Test methods	Response/Attachment/Topic match; Generative Answers ; Multi‑turn ; Plan validation	Exact/Partial match; Similarity/Intent ; AI quality graders
Observability	Enrichment with App Insights + Dataverse (topics, intent scores, latencies)	Aggregate pass/fail and scores with drill‑downs in Studio
CI/CD fit	Strong for release gates, duplication, and re‑runs	Strong for inner loop; roadmap for broader scenarios
Adjacent features	Part of a broader kit (Compliance Hub, KPIs, SharePoint sync)	Focused on evaluation inside Studio
Maturity	Generally available toolkit (maintained by Power CAT)	Public preview (subject to change)

When to Use What

Prefer Agent Evaluation (preview) when

You want fast, in‑Studio feedback while iterating on instructions, topics, or knowledge.
Your scenarios are single‑turn (FAQ‑style) and benefit from lexical/semantic/quality scoring.
You need AI‑generated test prompts to bootstrap coverage quickly.

Prefer Copilot Studio Kit when

You must validate multi‑turn, end‑to‑end flows in one conversation context.
You need to verify generative orchestration plans (correct tools/actions/connected agents).
You require deep telemetry (topics, intent scores, latencies) and App Insights correlation.
You manage release gates with repeatable Test Runs and re‑runs of enrichment steps.

Practical Scenarios

Inner‑loop tuning for HR FAQ → Use Agent Evaluation with Similarity and AI quality graders; generate 25 prompts from metadata, add a few gold answers, iterate on failures.
Pre‑release regression for tool‑using policy agent → Use Copilot Studio Kit : Multi‑turn + Plan validation ; App Insights + Dataverse enrichment to diagnose routing/tool‑selection issues.
Semantic drift watch → Weekly Agent Evaluation runs for relevance/groundedness ; investigate dips by reviewing knowledge changes.
CI pipeline across multiple agents → Nightly Kit Test Runs with Excel‑managed test sets; export failures for triage; duplicate runs after fixes; investigate latencies via App Insights.

Strengths & Limitations

Copilot Studio Kit - strengths

Multi‑turn, plan validation, deep observability (App Insights + Dataverse), and release‑friendly run management.

Copilot Studio Kit - limitations

Separate installation/governance; relies on Direct Line and cloud flows; AI Builder needed for AI‑based answer analysis.

Agent Evaluation - strengths

Native UI, AI‑generated test sets, flexible lexical/semantic/AI graders; quick to adopt.

Agent Evaluation - limitations

Public preview; multi‑turn on the roadmap; does not replace RAI/safety reviews.

Decision Guide (Quick Reference)

Need fast Studio‑native checks while editing? → Agent Evaluation (preview)
Need enterprise regression, multi‑turn, plan validation, telemetry? → Copilot Studio Kit
Mature teams: Use both - Evaluation for inner loop; Kit for outer loop, and promotion gates.

Helpful How‑Tos & Deep‑dives (Clickable Links)

Copilot Studio Kit: Overview · Test capabilities · Run tests · GitHub repo · Community guides: Forward Forever, Matthew Devaney (MS Auth setup)
Agent Evaluation (preview): Blog announcement · Preview docs · What’s new · Community guides: How‑to, Tips & pitfalls

Conclusion

Testing conversational agents in Microsoft Copilot Studio requires a nuanced approach, as development team needs can vary significantly by project stage and agent complexity. The Copilot Studio Kit and Agent Evaluation (preview) are two distinct tools that address different aspects of the testing process.

The Copilot Studio Kit is designed for scenarios requiring comprehensive, repeatable, and instrumented testing. It is particularly well-suited for validating multi-turn conversations, verifying orchestration plans, and collecting detailed telemetry for analysis. Because it operates as a separate, installable solution and integrates with Dataverse and Application Insights, it is most appropriate for teams that need to enforce quality gates before production releases or require historical tracking of test results. The Kit’s support for Excel-based test management and its extensibility through source code access make it a practical choice for organizations with established DevOps practices or those managing multiple agents at scale.

Agent Evaluation (preview), on the other hand, is integrated directly into the Copilot Studio authoring environment. Its primary focus is to provide rapid feedback during agent development and tuning, especially for single-turn or FAQ-style interactions. The ability to generate test prompts with AI and apply a range of grading methods (from exact match to semantic similarity and quality metrics) makes it accessible to authors who may not have a background in test automation. However, as a preview feature, its capabilities and scope are still evolving, and it does not currently address multi-turn or orchestration scenarios in depth.

In practice, many teams will find value in using both tools: Agent Evaluation for quick, iterative checks during agent design, and Copilot Studio Kit for more thorough validation prior to deployment. Understanding each tool’s intended use cases, technical requirements, and current limitations will help teams select the most appropriate approach for their workflow. As the Copilot Studio platform continues to evolve, the boundaries between these tools will likely shift, but the need for systematic, context-aware testing will remain constant to deliver reliable conversational agents.

Microsoft Frontier Agents: A Deep Technical Overview

Holger Imbery — Sat, 07 Feb 2026 10:57:11 +0000

Summary Lede

Microsoft Frontier Agents represent a fundamental shift in enterprise automation, moving from rigid rule-based systems to reasoning-capable AI agents that can gather information across organizational boundaries, synthesize complex data, and execute multi-step workflows autonomously. By operating within your existing Microsoft 365 infrastructure and security frameworks, Frontier Agents enable organizations to reimagine how work gets accomplished without requiring separate governance infrastructure or bypassing compliance controls.

Read on , if you are an enterprise architect, IT administrator, or business leader evaluating how to incorporate AI-based reasoning and automation into your organization, this article provides the technical and strategic foundation you need to understand what Frontier Agents actually do, how they differ from traditional automation approaches, and what you should consider before implementing them. You will learn the specific capabilities of Frontier Agents, the governance mechanisms that control them, and the workflow redesign required to realize genuine value - not just the technology deployment itself. Whether you are preparing to participate in the Frontier program or evaluating whether agent-based automation makes sense for your organization, this deep technical overview will help you make more informed decisions about where and how to invest in this emerging technology.

What Is the Frontier Program?

The Frontier program provides organizations with a structured mechanism to participate in early evaluation of emerging artificial intelligence capabilities within their existing Microsoft 365 infrastructure. Rather than deploying untested features across production systems, Frontier allows selected teams and administrators within an organization to voluntarily work with experimental AI features in a contained manner. These experimental features include various types of AI-powered agent systems - software entities capable of autonomous or semi-autonomous reasoning and action - as well as extensions to the Copilot assistant interface and specialized agent modes tailored for specific applications. Importantly, all of these experimental capabilities operate within the same security and data boundaries as the organization’s regular production Microsoft 365 environment. This design ensures that experimental features automatically inherit the organization’s established security policies, identity and access management through Entra ID, data location requirements, and compliance obligations. Organizations do not need to build separate infrastructure or bypass their existing governance frameworks to participate in these early evaluations.

Key characteristics of the Frontier program include:

Native Tenant Integration: Frontier agents operate seamlessly within your existing Microsoft 365 tenant, leveraging established security postures, Entra ID identity controls, data residency requirements, and compliance frameworks without requiring separate infrastructure or governance bypass.
Controlled Experimental Access: Organizations can grant selective access to preview capabilities through granular admin controls, ensuring that experimental features remain isolated to designated teams while maintaining full visibility and auditability.
Feedback-Driven Iteration: Microsoft actively incorporates operational data and user insights to refine, enhance, or retire Frontier capabilities, creating a responsive feedback loop that shapes agent behavior and feature maturity.
Licensing and Activation Requirements: Access to Frontier agents typically requires valid Microsoft 365 and Microsoft 365 Copilot licenses, with explicit administrator enablement through the Microsoft 365 Admin Center to control program participation at the organizational level.

Frontier Agents in Microsoft 365 Copilot

What Are Frontier Agents?

Frontier Agents are software systems that are built directly into Microsoft 365 applications and the Copilot experience. These systems work by gathering information from across your organization, analyzing it step by step using logical reasoning, and then executing tasks that require multiple sequential steps.

When you interact with a Frontier Agent, here is what actually happens beneath the surface: The agent first receives some information or a request from you. It then searches and retrieves data from various sources across your organization, including your emails, stored documents, calendar entries, meeting transcripts, and other business systems. Rather than simply retrieving information through a traditional search, the agent analyzes and reasons with the information it has gathered. It considers how different data elements relate to one another, understands the context in which that data exists, and then breaks down your original request into smaller, sequential steps that it can execute.

Two concrete examples of Frontier Agents currently available are the Researcher and Analyst agents. The Researcher agent, for instance, can accept a specific question from you and then search through the documents, emails, and other sources available within your organization to locate relevant information. After gathering this information, it synthesizes the pieces to provide a comprehensive answer to your question. The Analyst agent operates on similar principles but is designed to examine data from multiple sources and identify patterns, trends, or themes that emerge across that data, rather than being built to answer a specific user question.

An important distinction to note is that these agents are not limited to a single predefined action. Rather, they can take a single complex request from a user - such as “prepare a written summary of all customer feedback that arrived via email during the last quarter” - and break it down into multiple sequential steps. The agent might begin by identifying which emails and documents actually contain customer feedback. Following that, it might classify the feedback into categories by topic or theme. After categorization, it would extract the most significant points from each category. Finally, it would compile all of this into a single coherent summary document. Instead of a human staff member manually executing each step in sequence, the agent can reason through the problem and execute them in a logical order.

Real-World Scenarios

To better understand how these agents function in practical situations, consider the following concrete examples of work that they can perform:

Analyzing customer feedback across multiple communication channels: Organizations receive customer feedback through various channels - email messages, support tickets, conversation logs, and direct messages. Rather than having a team member manually review months of accumulated customer communications to identify recurring problems, an agent can systematically examine all these disparate sources of customer input across your organization, identify which specific issues or concerns appear multiple times, and surface those recurring themes for leadership. This process would otherwise require significant manual effort to collate information from systems that were never designed to be analyzed together.
Constructing comprehensive business review documents by synthesizing information from heterogeneous sources: Businesses often need to prepare periodic review documents that combine information from multiple different places - strategic documents that were written and stored in your document systems, records of what was discussed and decided during meetings throughout the review period, and email conversations where important decisions and discussions occurred. An agent can gather these distinct types of information across different systems and synthesize them into a coherent narrative that presents a unified view of what occurred during the period in question.
Locating decision points that remain unresolved in organizational communications: In any organization, communications - whether through email, chat systems, or meeting notes - often contain discussion about decisions that need to be made, but the outcome of those discussions remains unclear or incomplete. An agent can review your organization’s communications, identify discussions where no clear resolution was documented, and compile a summary that clarifies which decisions remain pending and who should be involved in resolving them.
Evaluating patterns in historical meeting content and suggesting corresponding workflow adjustments: Organizations hold many meetings over time, and the topics, themes, and concerns that emerge across multiple meetings often reveal something meaningful about where real work challenges lie. An agent can review the captured records from your organization’s past meetings, identify recurring themes and concerns across meetings and groups, and use that analysis to suggest where the organization might redirect its focus or restructure its processes to address the underlying issues that keep arising.

Microsoft Agent 365: The Control Plane Behind Frontier Agents

Agent 365 is the administrative and governance layer that supports AI agents across an enterprise.

Key Capabilities

Agent Discovery and Inventory Management: Agent 365 maintains a comprehensive record and catalog of all AI agents operating within an organization. This inventory function allows administrators and operators to understand which agents exist, where they are deployed, what purposes they serve, and their operational status at any given time. Rather than having agents scattered across the organization with no central visibility, this inventory system provides a unified view of the agent ecosystem.
Identity-Based Resource Access Controls: Agent 365 works in conjunction with Entra ID (Azure’s identity management system) to establish which specific resources, data sources, and systems that individual agents are allowed to access. Instead of granting agents unlimited access across an organization’s systems, Agent 365 enforces granular permission boundaries, ensuring each agent can access only the data, applications, and services necessary to perform their assigned functions. This principle of least privilege prevents an agent from inadvertently or maliciously accessing sensitive data or systems outside its defined scope.
Standardized Integration Interfaces and Development Kits: Agent 365 provides standardized software development kits (SDKs) and application programming interfaces (APIs) that establish consistent patterns for how new agents can be built and integrated with existing organizational systems. Rather than building each agent in isolation using different approaches, these standardized interfaces ensure that agents built by different teams or vendors can work together and communicate in predictable ways, following established patterns.
Relationship Mapping and System Visualization: Agent 365 provides administrators and architects with visual representations that illustrate how different agents relate to one another, how they connect to and interact with people in the organization, what data sources they access or modify, and how they fit into broader business workflows. This visualization capability helps organizations understand the dependencies and interactions across their entire agent estate.
Integrated Security Operations and Threat Detection: Agent 365 incorporates monitoring and alerting functions that work in coordination with Microsoft Defender (security threat detection) and Microsoft Purview (data governance and compliance). This integration enables security teams to monitor agent behavior for signs of anomalous or suspicious activity and compliance teams to track whether agents are handling regulated data appropriately.
SDK Components and Development Framework Support: Agent 365 supplies developers with pre-built components, software libraries, and reference implementations that accelerate the process of creating new agents or extending existing ones. It also supports industry-standard protocols, such as the Model Context Protocol (MCP), which establishes common patterns for how components can exchange information and coordinate their behavior.

The Role of Frontier Agents in Enterprise Transformation

As organizations implement Frontier Agents, they are fundamentally reconsidering how work gets accomplished. Rather than viewing these systems as replacements for human workers, the most effective deployments treat agents as tools that handle specific, well-defined operational tasks while people focus on decision-making, judgment calls, and work that requires contextual understanding.

This shift requires more than simply enabling a new technology. Organizations need to redesign their operational workflows to leverage agents’ capabilities. This means identifying which portions of existing processes are primarily repetitive information gathering, data synthesis, or sequential task execution - the kinds of work that agents handle effectively - and then separating those portions from the work that genuinely requires human judgment, client interaction, or strategic thinking. Once this separation occurs, you can structure a workflow where the agent handles the mechanical portions while a person reviews and takes responsibility for the decisions that matter.

One practical implication worth considering: when an agent executes multiple steps in sequence to complete a task, there are often opportunities to build in checkpoints that allow a human to review the agent’s work before it moves to the next stage. This prevents compounding errors and allows people to correct the agent’s reasoning when it goes astray. Rather than treating agent output as completely reliable or completely unreliable, organizations learn to understand where agents tend to make mistakes and design their approval workflows accordingly.

Another important consideration is that moving work to agents does not necessarily mean work disappears from the organization. Instead, the work’s character changes. Where someone previously spent time reading through emails to find relevant information, they now spend time reviewing the agent’s findings to verify that the needed information was captured. Where someone previously spent time copying data from one system into a report, they now spend time checking whether the agent’s synthesis of that data makes sense. This redistribution of work can free up capacity for higher-value activities, but only if organizations deliberately redirect that freed-up capacity toward them rather than simply eliminating positions and expecting the same output from fewer people.

Examples of Frontier Experiences Available Today

The Frontier ecosystem includes:

Experimental Copilot agents in the Agent Store.
Agent Mode in Excel for the web.
Agents for Dynamics 365 workloads.
Experimental functionalities in Word, PowerPoint, and Copilot Chat.
AI-enhanced Windows 365 Cloud PC capabilities.

How Administrators Enable and Manage Frontier Agents

Administrators must:

Confirm licensing prerequisites.
Enable Frontier in the Admin Center.
Assign Frontier access to users.
Use Agent 365 tools to manage lifecycle, visibility, and compliance.
Provide SDK access for development.

Why Frontier Agents Matter for Enterprise Architects

Understanding the Shift from Rule-Based to Reasoning-Based Systems

Traditionally, enterprise automation has relied on rule-based systems - tools that follow explicit, predetermined logic written by engineers. These systems work well when the tasks they handle are predictable and straightforward: if condition A is true, then execute action B. However, rule-based systems struggle when circumstances change unexpectedly or when situations don’t fit neatly into predefined categories. When an edge case arises that no one anticipated, the rule-based system either fails or performs the wrong action, requiring human intervention or system modification.

Frontier Agents introduce a different approach. Rather than following rigid if-then-else logic, these agents can examine a situation, understand context and nuance, reason through the problem using information available to them, and adapt their approach as they encounter new information. This capability is critical for enterprise architects because it enables them to automate and streamline work that was previously too variable or context-dependent to automate. Enterprise architects should understand where agents can genuinely improve their organizations and where the investment in agent systems would not yield meaningful returns.

Practical Reasoning Capabilities and Their Implications

Agents can perform multi-step reasoning that mirrors how skilled humans approach complex problems. An agent can gather information from multiple sources, recognize patterns and relationships within that information, adjust its understanding as it discovers new details, and modify its approach based on what it learns. This capability matters for enterprise architects because it changes which business processes become candidates for automation. Instead of limiting automation to straightforward, fixed-sequence tasks, architects can now consider automating substantially more nuanced work. However, architects need to think carefully about which problems benefit from this capability and which would be simpler to solve through other means. Not every situation requires sophisticated reasoning - sometimes simpler solutions serve organizations better.

Unified Governance Without Separate Infrastructure

One of the most significant practical benefits for enterprise architects is that Frontier Agents operate within existing organizational boundaries. Rather than requiring architects to design separate systems, approval processes, or compliance frameworks for agent-based work, agents inherit your organization’s existing security, identity, and compliance structure. This simplifies governance - architects do not need to build parallel systems. However, architects considered how to ensure that existing governance frameworks accommodate agent operations and that identity and access controls properly constrain what agents can do.

Hybrid Human–Agent Workflows and Redesigned Work Processes

Implementing agents require architects and process owners to rethink how work flows through the organization. It is not sufficient to add an agent to an existing workflow. Instead, organizations need to identify where human judgment and decision-making are essential, where human oversight is valuable but not essential, and where mechanical task execution can be delegated entirely to an agent. This often means redesigning workflows substantially. An enterprise architect should expect that workflow redesign will be more challenging and time-consuming than deploying agent technology alone and should budget time and expertise accordingly.

Incremental Adoption and Learning-Based Deployment

Rather than requiring organizations to commit to large-scale transformation immediately, agents can be introduced gradually. This allows organizations to learn how agents affect their work, identify which applications genuinely improve outcomes, develop troubleshooting skills, and build internal expertise. Enterprise architects should view this as an extended learning period in which the organization creates an understanding of what agents can do within their specific context, rather than a one-time implementation that commits the organization to a comprehensive agent-based transformation immediately. This approach carries risks - it means initial projects may not yield dramatic returns - but it also creates space for learning and course correction.

Conclusion

The Frontier program is a practical approach organizations can use to test experimental AI-based systems that perform multi-step reasoning and task execution within their existing Microsoft 365 environments. Rather than requiring organizations to establish entirely separate infrastructure, sign off on different governance frameworks, or bypass critical requirements, the Frontier program enables them to experiment while remaining within their organizations’ existing security and identity boundaries.

Automate Frontier Agents - software systems built into Microsoft 365 applications that gather information from multiple sources, reason with it, and execute sequential tasks to accomplish complex objectives - provides a concrete technology that organizations can use to explore how AI-based reasoning and automation might apply to their own workflows. Agent 365, the administrative layer beneath these agents, provides the inventory management, access controls, monitoring, and coordination mechanisms organizations need to maintain visibility and control over agent operations at scale.

In practice, organizations that choose to implement Frontier Agents will need to do more than enable new technology features. They will need to rethink which parts of their business processes benefit from having agents handle information gathering and task execution, how to structure approval workflows that allow human staff to focus on judgment and decision-making, and how to ensure that the technology actually produces better outcomes rather than simply shifting the character of the work without improving results. This redesign work is often more demanding than the technical implementation itself, but it is also where genuine value is created.

Organizations that approach Frontier Agents as an extended learning opportunity - where initial projects serve as experiments to develop organizational understanding of what agents can accomplish within specific contexts - are likely to make more thoughtful implementation decisions than organizations that pursue large-scale deployment without this learning period. This incremental approach has both benefits and drawbacks: initial projects may not deliver dramatic returns on investment, but the organization will develop practical knowledge of where agents are genuinely helpful, where they create problems, and how to design workflows that leverage agent capabilities effectively.

Ship Copilot Studio Agents with Confidence: Master Automated Testing with the Copilot Studio Kit

Holger Imbery — Sat, 31 Jan 2026 12:35:30 +0000

Summary Lede

Shipping Copilot Studio agents without systematic, automated testing is risky: large language models (LLMs) are non‑deterministic, topic routing can drift, and integrations fail in ways casual “chat tests” won’t catch. Microsoft’s Copilot Studio Kit adds structured, repeatable testing (including multi‑turn and generative answer validation), integrates with Power Platform pipelines for gated deployments, and provides analytics and compliance tooling—so you can test as software teams do and ship with confidence.

Why Read This Article : AI agents powered by LLMs are unpredictable by nature. Casual testing—clicking through a chat interface a few times—is not enough to catch latent quality issues, topic confusion, or integration failures that emerge under real-world conditions. This guide walks you through a comprehensive, systematic approach to testing Copilot Studio agents that mirrors enterprise software practices: from defining repeatable test cases with the Copilot Studio Kit, to embedding quality gates in your deployment pipelines, to monitoring compliance and performance post-launch. If you are shipping agents to production—or planning to do so—you need structured, auditable testing to reduce risk and build stakeholder confidence.

Why agent testing is different (and non‑optional) in Copilot Studio

Conversational agents in Copilot Studio operate under fundamentally different constraints than traditional software. Where conventional applications follow deterministic code paths and produce consistent output for given inputs, conversational agents integrate multiple layers: natural language processing, topic routing, prompt-based reasoning, and knowledge retrieval across enterprise systems. This architectural difference creates testing challenges that cannot be addressed by conventional unit testing or ad hoc verification approaches.

The non-deterministic nature of these systems stems from several sources. Large language models (LLMs) introduce inherent variability in response generation even when given identical inputs. Topic routing may inconsistently classify user intents, particularly for edge cases or ambiguous queries. Knowledge retrieval systems may return different result sets depending on index state, search ranking, or temporal data updates. When agents compose multiple actions—such as querying multiple data sources, applying filters, and synthesizing responses—the combinations of these variations compound, making manual testing insufficient to validate behavior across the full range of user interactions.

Microsoft’s published guidance emphasizes that this variability necessitates systematic, large-scale evaluation rather than point-in-time chat testing. A developer might test an agent through a few interactive sessions and conclude it functions correctly based on limited exposure. However, when the agent encounters hundreds or thousands of real-world queries—including paraphrased variations, context-dependent follow-ups, and edge cases—latent quality issues, topic confusion, and integration failures emerge. Automated testing frameworks assess correctness and performance across representative input sets, providing measurable confidence in agent behavior.

It is important to note that while automated evaluation effectively identifies accuracy and performance issues, Microsoft explicitly documents that this approach cannot replace responsible AI reviews or the safety filters built into governance processes. Automated testing validates the agent’s logical correctness and consistency, but human review remains essential for assessing potential harms, compliance with organizational policies, and alignment with responsible AI principles. These complementary activities must both occur during the release process.

The testing toolbox at a glance

Copilot Studio Kit (Power CAT)

The Copilot Studio Kit is an open‑source, solution‑aware extension that adds a formal testing and analysis layer to Copilot Studio. At its core, the Kit lets you define agents, tests, and test sets and then run batch tests against your agent through the Direct Line API. Results are not limited to raw strings; they can be enriched with Dataverse conversation transcripts (to expose the exact triggered topic and intent scores) and Azure Application Insights (for telemetry and failure diagnostics). The Kit supports deterministic checks (e.g., response matching and Attachment comparisons) as well as LLM-assisted validation for Generative answers using AI Builder—a critical capability when answers are non-deterministic. For complex scenarios, you can compose multi‑turn tests to validate end‑to‑end flows in a single conversation context, and use plan validation to ensure that agents with generative orchestration select the expected tools/actions above a configured threshold. In enterprise rollouts, the Kit’s managed solution model, Excel import/export for bulk test authoring, and optional Compliance Hub significantly shorten the path to repeatable, auditable test evidence.

When to use :

You need scalable, repeatable test execution with artifacts you can retain and trend over time.
You must validate both deterministic paths (topic routing, attachments) and non‑deterministic LLM outputs (generative answers).
You want to instrument tests with Dataverse and Application Insights to explain why a test passed/failed.

Agent evaluation (in‑product, preview)

Agent evaluation is a built-in preview feature of Copilot Studio that lets you author or generate test sets and run automated evaluations directly in the product. It is designed to measure answer quality and coverage at scale and can generate tests from your agent’s topics/knowledge or import them from a file. Evaluations are executed with a designated test user profile, which is essential when tools and knowledge sources require authentication. Because this feature is in preview, treat it as a complementary capability to the Kit: use it for fast, in-product evaluations and keep the Kit’s runs and exports for long-term retention and pipeline gating.

When to use :

Early iterative cycles where makers want quick, in‑canvas evaluation runs.
Seeding a broader test corpus by auto‑generating questions from topics/knowledge, then curating.
Validating that a specific test identity can access the same tools/knowledge as target users.

Power Platform pipelines

Power Platform pipelines provide the native ALM path to move solution-packaged agents across Dev → Test → Prod with approvals, audit trails, and environment isolation. Critically, pipelines can be extended to run Copilot Studio Kit tests as a pre-deployment quality gate: the deployment is paused, tests execute, results are evaluated against pass thresholds, and only then does the pipeline promote to the next stage. This pattern converts “publish from dev” into a governed release with repeatable validation, versioning, and rollback via managed solutions. In practice, you configure a pipeline host environment, wire cloud flows + Dataverse events to call the Kit’s test runner, and enforce gate criteria before production.

When to use :

Any production‑bound agent; manual promotion without a gate is a risk.
Teams that need approvals, traceability, and the ability to block a release on failed tests.
Organizations standardizing on solution‑based ALM across Power Platform assets.

Direct Line performance testing

Functional correctness is not sufficient if the agent cannot meet performance objectives. Microsoft’s guidance documents how to load and performance-test Copilot Studio agents using Direct Line over WebSockets (preferred for realistic behavior) or HTTP GET polling when WebSockets are not feasible. Test harnesses should track response times at the stages that affect user experience: Generate Token, Start Conversation, Send Activity, and Receive/Get Activities. These measurements, collected under realistic concurrency and payload conditions, enable you to baseline and detect regressions as topics, tools, and knowledge sources evolve.

When to use :

Before go‑live and on every significant change to prompts, tools, or knowledge that could affect latency.
When you must validate SLA adherence (for example, first token and full‑response times).

CoE & Compliance

Testing is part of a broader governance and compliance posture. Microsoft’s Phase 4 guidance emphasizes structured testing, deployment, and launch practices, including final security and compliance checks, telemetry enablement, and controlled rollout. The Copilot Studio Kit’s Compliance Hub complements this by continuously evaluating agent configurations captured via Agent Inventory against configurable thresholds, creating compliance cases, and supporting SLA-driven triage (manual review, quarantine, or delete). Together with Managed Environments, DLP policies, and CoE Starter Kit telemetry, these controls provide continuous post-deployment oversight of agents, reducing configuration drift and helping teams keep production behavior within approved boundaries.

When to use :

Organization‑wide programs where multiple business units ship agents and you need consistent review and enforcement.
Environments with strict regulatory or data‑handling requirements that require continuous configuration posture checks.

Use the Copilot Studio Kit for structured, repeatable tests (including multi-turn, generative-answer validation, enrichment, and exports). Use Agent evaluation (preview) for in-product, fast iteration. Enforce release quality with Power Platform pipelines by gating promotion on automated test results. Validate scalability and user-perceived latency with Direct Line performance testing. Finally, operate agents within a governed framework using Phase-4 practices and Compliance Hub to maintain compliance and configuration integrity over time.

Phase-4 testing (detailed guidance and checklist)

Phase 4 refers to Microsoft’s “Testing, deployment, and launch” stage in the Copilot Studio governance and security best-practices sequence. It defines what must happen after build-time design and before (and immediately after) a production release. In practical terms, Phase 4 defines the quality gates, security checks, controlled rollout, and post-release monitoring you should apply to every Copilot Studio agent before it serves real users.

Below is a concise, implementation-oriented summary of Phase 4 practices and their execution.

Testing and validation

Goal : Prove the agent’s functional behavior and non‑deterministic answer quality with repeatable, automated tests—not manual chats.

Automated scenario testing : Use the Copilot Studio Kit to define tests and test sets (response/attachment/topic/generative, including multi-turn and plan validation) and run batches against the agent. Enrich results with Dataverse transcripts and Application Insights for root-cause analysis.
CI/CD readiness : Maintain test artifacts and runs as part of your release process. Microsoft’s Phase 4 guidance explicitly recommends automated testing and evaluation as a prerequisite for deployment.
Quality gates in pipelines : Integrate Kit test runs into Power Platform pipelines so a deployment pauses, executes tests, evaluates pass thresholds, and only then promotes to the next stage. This delivers an auditable “test-before-deploy” control.

Final security and compliance checks

Goal : Ensure the production environment enforces the right data and access boundaries and that all Azure resources are approved.

Data policies and RBAC : Verify environment-level policies (e.g., DLP), role assignments, and connection security in the production environment—not just in Dev/Test. This prevents accidental connector drift at go-live.
Azure resource review : Confirm approvals for app registrations, networks, keys, and endpoints associated with the agent’s external dependencies. Use secure secret storage and rotate keys.
Production knowledge sources : Point the agent to the production document libraries and data sets (many teams test with separate SharePoint paths or sample data; Phase 4 requires verification of the production bindings).

Controlled production rollout

Goal : Promote a versioned, solution‑packaged agent to production via ALM—not via ad‑hoc publish.

Deploy via pipelines : Package the agent in a Power Platform solution and promote Dev → Test → Prod with approvals, audit trail, and environment isolation. This is the supported, governed path for Copilot Studio.
Pre-deployment steps : In the pipeline, add hooks to run Kit tests and evaluate pass rates as a quality gate before import into the target environment.
Launch plan : Communicate availability and usage guidance to the intended audience and stakeholders as part of the release checklist.

Enable monitoring and ongoing governance

Goal : Operate the agent as a managed service with telemetry, compliance posture tracking, and corrective workflows.

Telemetry : Configure Azure Application Insights for usage, performance, and error logging. This supports regression detection and incident response after go‑live.
Compliance operations : Use the Copilot Studio Kit’s Compliance Hub to continuously evaluate agent configurations against policy thresholds, raise review cases, and track SLA‑bound remediation (review, quarantine, delete).
CoE integration : Leverage the Power Platform CoE Starter Kit to inventory agents, watch adoption/health signals, and maintain platform‑wide governance routines.

Why Phase 4 matters

Phase 4 formalizes the last mile from “it works in development” to “it is safe to expose to users.” It replaces manual spot checks and one‑click publishing with automated validation, governed deployment, and observable operations—the baseline expected of enterprise‑grade AI agents. By following Phase 4 practices, teams reduce the risk of production incidents, data leaks, and compliance violations. They gain confidence that agents behave as intended under real‑world conditions and that any deviations are detected and addressed promptly. Ultimately, Phase 4 transforms Copilot Studio agents from experimental prototypes into reliable, governed components within the organizational AI landscape.

Test strategy for low‑code/no‑code agents

Test types you should plan for:

Conversational/functional (does the response match expectations for known intents?) – Use Kit’s Response match and Topic match.
Generative answer validation (LLM output quality and guardrails) – Use AI Builder-based Generative answers with Application Insights context.
End-to-end scenarios across multiple turns and tools – Use Multi-turn and Plan validation for generative orchestration to ensure the plan contains the expected tools/actions.
Integration (Dataverse, connectors, actions) – Validate topic routing via Dataverse enrichment and attachment payloads (e.g., Adaptive Cards).
Performance & reliability under load – Use Direct Line guidance to capture token/start/send/receive latencies.
Safety/compliance – Follow governance phase guidance and consider Kit’s Compliance Hub to flag configuration policy issues.
What to automate first : high-volume intents, critical business workflows (e.g., authentication-gated actions), and generative answers that must adhere to strict constraints. Then expand to long-tail intents and exploratory questions using generated test sets (Agent evaluation) and bulk import (Kit).

Implementing automated testing with Copilot Studio Kit

Set up Copilot Studio Kit for automated testing

Install the Kit from Marketplace or GitHub as a managed solution in your chosen environment; complete post‑deployment connection references.
Configure the agent connection for testing:
Set agent base configuration (name, region/token endpoint), and Direct Line channel security as applicable.
Enable Dataverse enrichment to analyze conversation transcripts (topic names, intent scores).
Enable Application Insights enrichment for diagnostics and negative tests (e.g., moderated/no result cases).
Configure AI Builder as the LLM provider for Generative answers validation.

If your agent uses Microsoft Authentication, register an app and configure the Kit’s test automation so the test runner can authenticate as a user.

Design test cases and test sets

Supported test types (Kit):

Response matches with string operators (equals/contains/starts/ends).
Attachments comparison (JSON array) or AI Validation for structure/semantics.
Topic match (requires Dataverse enrichment).
Generative answers (requires AI Builder + optional Application Insights).
Multi-turn (compose several tests into one conversation).
Plan validation (ensure the generated plan includes expected tools/actions above a threshold).

Bulk authoring/import: Use Kit’s Excel import/export to create or modify multiple tests efficiently. In-product test sets (Agent evaluation, preview): Create up to 100 test cases per set; generate questions from agent description/topics/knowledge or import from a file; run with a selected test user profile that has the right connections/authentication.

Run, analyze, and iterate

Run test sets against your agent; Kit records observed responses and latencies, and aggregates them. Enrichment adds topic routing and detailed diagnostics. Export results (CSV) for long‑term retention or integration with other tools. Use Kit’s analytics artifacts (e.g., Analyze Test Results) and conversation KPIs in Dataverse for trend analysis beyond the built‑in Copilot Studio analytics.

Automate “test → deploy” with pipelines

Treat agents as solution components and move them with Power Platform pipelines; add automated Kit test runs as a pre‑deployment gate so only solutions that meet pass thresholds are deployed to Test/Prod. Microsoft’s guidance and Kit docs provide a pattern using cloud flows and Dataverse to pause the pipeline, run tests, evaluate the pass rate, and decide whether to continue or stop. This replaces the brittle “publish from dev” habit with auditable CI/CD and quality gates aligned to enterprise ALM.

Performance and reliability testing via Direct Line

For load testing, simulate real user behavior using Direct Line with WebSockets where possible; otherwise, use HTTP GET polling. Track and report response times for Generate Token, Start Conversation, Send Activity, and Receive/Get Activities to understand user‑perceived latency under load. Microsoft’s documentation provides detailed guidance on setting up these tests.

Governance: beyond the build—compliance, environments, and monitoring

Microsoft’s governance phase recommends validating security and compliance, using ALM pipelines for controlled rollout, and enabling telemetry (Application Insights). The CoE Starter Kit and Kit’s Compliance Hub can help continuously evaluate agent posture and enforce approvals/quarantines when violations occur.

Anti-patterns: why “chat a bit and publish” fails

Clicking Publish pushes the current agent state to channels; it is not a deployment pipeline. Without environments, solutions, and quality gates, you lack versioning, rollback, and documented test evidence—unacceptable for enterprise operations. Use solutions, pipelines, and automated tests instead.

Practical checklist for Phase-4 testing and deployment

Pre-requisites

Agent in a solution; environments for Dev/Test/Prod; pipeline host environment configured.
Copilot Studio Kit installed and connected; App Insights + Dataverse enrichment; AI Builder available.

Test design

Identify top intents and critical flows; write Response/Topic/Attachment tests.
Add Generative answers tests for non‑deterministic responses (with validation instructions and sample answers).
Compose Multi‑turn scenarios for end‑to‑end paths; add Plan validation for generative orchestration.

Automation

Create test sets in Kit and (optionally) in Agent evaluation to complement coverage; export results to CSV for retention.
Wire pre‑deployment pipeline steps to run Kit tests and enforce pass thresholds (quality gate).

Performance

Execute load tests through Direct Line, prefer WebSockets, and track the specific latencies Microsoft recommends.

Governance

Run security/compliance checks; monitor with CoE and Compliance Hub after go‑live.

Conclusion

Automated testing is essential for Copilot Studio agents due to their non‑deterministic nature and complex integrations. Microsoft’s Copilot Studio Kit provides a robust framework for defining, executing, and analyzing tests, enabling teams to validate both deterministic and generative behaviors at scale. By integrating these tests into Power Platform pipelines, teams can enforce quality gates that ensure only validated agents reach production. Complementing this with performance testing via Direct Line and ongoing governance through telemetry and compliance tools establishes a comprehensive lifecycle for reliable, enterprise-grade AI agents. Adopting these practices transforms Copilot Studio agents from experimental prototypes into trusted components of the organizational AI landscape, capable of delivering consistent value while adhering to security and compliance standards.

Migrating from Agent Builder (Copilot Studio Lite) to Copilot Studio

Holger Imbery — Sat, 24 Jan 2026 00:00:00 +0000

Technical overview and configuration guide (January 2026)

Summary Lede Moving your agents from Agent Builder(aka Copilot Studio lite) — the simplified creation tool embedded in Microsoft 365 Copilot — to the full-featured Copilot Studio environment represents a significant step forward in how you manage and deploy conversational AI within your organization. While Agent Builder excels at helping teams quickly prototype and launch straightforward agents for internal use, Copilot Studio provides the comprehensive tooling needed when those agents mature into reliable, enterprise-scale solutions.

Why read it? Are you looking to scale your Copilot agents from small-team tools to enterprise-ready solutions? This article explains the compelling benefits, migration process, and critical considerations—so you can confidently elevate your agent strategy.

This article uses the terms “Agent Builder” and “Copilot Studio Lite” interchangeably to refer to the same simplified agent-creation tool in Microsoft 365 Copilot. The full-featured environment is called “Copilot Studio.” Microsoft is continuously changing naming conventions, so please refer to the latest official documentation for the most current terminology.

Introduction

The migration process itself is straightforward: you essentially create a copy of your existing agent in a Power Platform environment where Copilot Studio operates. However, the real value lies in what becomes available after migration. You gain access to proper version control mechanisms that let you track changes over time, stage updates across multiple environments, and roll back to previous configurations if something goes wrong. Analytics capabilities expand dramatically, giving you detailed insights into how users interact with your agents, where conversations succeed or fail, and which knowledge sources prove most valuable.

From a governance perspective, the difference is substantial. Copilot Studio integrates with Power Platform’s security model, allowing you to implement role-based access controls, enforce data loss prevention policies, and maintain detailed audit trails of who changed what and when. You can configure connectors to external systems with appropriate guardrails, manage how agents access sensitive data, and ensure everything complies with your organization’s compliance requirements.

The migration also opens the door to more sophisticated agent behaviors. You can implement conditional logic that routes conversations based on context, integrate with external APIs and services beyond what Agent Builder supports, and leverage Dataverse as a structured data layer for your agent’s knowledge and state management. If your organization runs separate development, testing, and production environments—as most enterprises do — Copilot Studio’s multi-environment support becomes essential for maintaining stable operations while continuously improving your agents.

This article walks through the practical aspects of performing this migration: understanding which components transfer automatically and which require manual reconfiguration, meeting the licensing and environment prerequisites, and establishing a workflow that minimizes disruption to users who depend on your existing agents.

Why Migrate?

Agent Builder is great for quickly building conversational agents within Microsoft 365 Copilot. But when you need advanced features like:

Version control, staged rollouts, and rollback options
Deep usage analytics and dashboards
Enterprise governance: role-based access, data policies, audit trails
Complex customization: conditional logic, external API integrations
Multi-environment management (dev, test, prod)
Broader connectors and Dataverse integration

Copilot Studio is designed for these needs—making it the scalable, secure next step for production-ready bots.

What Transfers in the Migration

Understanding the Migration Mechanism

When you’ve developed an agent using Agent Builder within Microsoft 365 Copilot and reach the point where you require capabilities that exist only in the complete Copilot Studio platform, Microsoft provides a direct migration path through the Copy to Copilot Studio function.

How to Initiate the Copy

You’ll find this option in the More options (…) menu when viewing your agent in Agent Builder. The purpose of this function is to create a duplicate of your existing agent in the Power Platform environment where Copilot Studio operates, eliminating the need to recreate it manually.

Important: This Is a Copy, Not a Move

It’s important to understand that this is a copy operation , not a move or synchronization. Your original agent in Agent Builder remains unchanged and continues to function exactly as before.

What you get is an independent snapshot of that agent’s configuration at the moment you initiated the copy process. Any subsequent changes you make to either version—the original in Agent Builder or the copy in Copilot Studio—will not affect the other.

This design allows you to:

Continue using the Agent Builder version while you configure and test the Copilot Studio version
Ensure no disruption to users who depend on the existing agent
Validate the migrated version before fully transitioning

What Gets Transferred Automatically

The copy operation handles a specific subset of your agent’s configuration; not everything is copied over, especially uploaded files. Understanding exactly what transfers automatically helps you plan the remaining setup work after migration.

Select the target environment for your Copilot Studio agent during the copy process.

Basic identification and behavior elements that copy over include the agent’s name, the description that appears to users, and the core instructions that define how the agent should respond and behave. These static text fields define the foundation of your agent’s personality and purpose. The suggested prompts you configured to help users get started with the agent also transfer, maintaining that initial user experience guidance.

Visual identity is preserved through the automatic transfer of your agent’s icon, ensuring brand consistency between the Agent Builder and Copilot Studio versions.

Knowledge sources represent a more nuanced aspect of the migration. SharePoint files, folders, and entire sites that you designated as knowledge sources in Agent Builder will transfer to Copilot Studio. In the same way, any websites you added as knowledge sources make the transition. These knowledge sources are the foundation of your agent’s ability to provide accurate, contextually relevant responses based on your organization’s specific information.

What Requires Manual Reconfiguration

Several configuration categories do not transfer automatically during the copy operation. These require deliberate action on your part once the agent exists in Copilot Studio.

Enterprise knowledge accessed through Copilot connectors represents a significant gap in the automatic transfer process. In contrast, basic SharePoint and website knowledge sources transfer; enterprise knowledge that relies on specialized Copilot connectors must be manually reconfigured in Copilot Studio after migration. This includes setting up the appropriate connectors and reestablishing the connection to your enterprise data sources. The underlying reason relates to how these connectors authenticate and authorize access—these security configurations don’t automatically replicate across environments.

Scoped Copilot connectors pose a limitation in the current Copilot Studio implementation. If your Agent Builder agent uses scoped connectors that restrict data access based on specific criteria, you should know that Copilot Studio doesn’t currently support this capability. You’ll need to evaluate whether alternative approaches exist for obtaining similar data scoping within Copilot Studio’s connector framework, or determine whether this is a blocking issue for your migration timeline.

Embedded files you uploaded directly to Agent Builder don’t transfer during the copy operation. The system doesn’t migrate the binary file content itself; it only references external knowledge sources, such as SharePoint locations. After migration, you’ll need to upload these files again directly to Copilot Studio. This manual step lets you verify that files remain current and relevant for the Copilot Studio environment.

Teams chats and meetings that served as knowledge sources in Agent Builder require a different implementation approach in Copilot Studio. Instead of the embedded access mechanism used in Agent Builder, you’ll add the Power Platform connector for Microsoft Teams within Copilot Studio. This connector provides the access pathway to Teams content, though you may need to reconfigure specific permissions and scope settings to match your original Agent Builder configuration.

Email content from Outlook follows a similar pattern to Teams integration. If your Agent Builder agent drew knowledge from emails, you’ll need to add the Power Platform connector for Outlook in Copilot Studio and reconfigure which mailboxes or folders the agent can access as knowledge sources.

Code generation, document creation, and chart generation capabilities require enabling the code interpreter functionality in Copilot Studio. While Agent Builder may have included these features by default or with simpler configuration, Copilot Studio treats the code interpreter as an explicit capability you enable in the agent’s settings. Navigate to the appropriate settings section and activate the code interpreter to restore this functionality.

Image generation from text prompts represents a functional gap between Agent Builder and Copilot Studio. If your agent can generate images from user descriptions in Agent Builder, this capability isn’t currently available in Copilot Studio. Basic charts and graphs remain possible through the code interpreter capability, but photorealistic image generation or creative visual content creation from prompts isn’t supported. You’ll need to assess whether this limitation affects your agent’s core value proposition and plan accordingly—either by removing image generation from your agent’s advertised capabilities or by maintaining a parallel Agent Builder version for use cases that require it.

Configuration	Action
Enterprise knowledge (Copilot connectors)	Set up connectors after you copy your agent.
Scoped Copilot connectors	Not currently supported in Copilot Studio.
Embedded files	Upload the files again in Copilot Studio.
Teams chats and meetings added as knowledge	Add the Power Platform connector for Teams in Copilot Studio.
Emails added as knowledge	Add the Power Platform connector for Outlook in Copilot Studio.
Create documents, charts, and code	Add code interpreter via agent settings in Copilot Studio.
Create images from prompts	Not currently supported in Copilot Studio. Basic charts and graphs are part of the code interpreter capability.

Licensing & Environment Requirements

You need either a Microsoft 365 Copilot or Copilot Studio license. Trials may be available depending on tenant policies.
The migration process depends on Power Platform environments:
- Must support Dataverse and be in a supported region
- You need appropriate security roles and environment accessibility
Admin-managed governance, data, and connector policies are managed in Power Platform Admin Center; sharing limits may apply

Post-Migration Considerations

Once copied, the new agent is independent—subsequent edits in Agent Builder won’t sync
Every “Copy” operation creates a new agent instance
Retire or archive old agents after ensuring continuity
Periodically revisit environment-specific policies and data retention setups

Conclusion

The process of moving your agents from Agent Builder into the complete Copilot Studio environment represents more than a simple platform upgrade—it fundamentally changes how your organization approaches conversational AI as a strategic capability rather than a tactical tool. This transition acknowledges that what began as a straightforward internal assistant has demonstrated sufficient value to warrant the infrastructure, governance, and operational discipline that enterprise software demands.

From a practical standpoint, the migration provides access to capabilities that become essential as usage scales beyond small teams. Version control mechanisms let you track exactly what changed in your agent’s configuration over time, which is critical for troubleshooting unexpected behavior and understanding how the agent evolved to meet changing business needs. The ability to maintain separate development, testing, and production instances means you can experiment with improvements without disrupting users who depend on the current working version. Staged rollouts allow you to gradually introduce changes to subsets of users, monitoring for issues before full deployment.

The analytics infrastructure in Copilot Studio offers visibility not available in Agent Builder. You gain detailed metrics on conversation patterns—where users struggle, which knowledge sources are most valuable, which questions the agent can’t answer effectively, and how response quality varies across user populations. This data becomes the foundation for continuous improvement, transforming agent optimization from guesswork into an evidence-based process.

Governance capabilities matter increasingly as agents handle sensitive information or influence essential business processes. Copilot Studio integrates with Power Platform’s security model, providing role-based access controls to determine who can modify agent configurations, audit trails to document every change for compliance purposes, and data loss prevention policies to prevent agents from inadvertently exposing protected information. These controls become non-negotiable when regulatory requirements or corporate policies govern how AI systems must operate.

The expanded customization options available in Copilot Studio enable more sophisticated agent behaviors that address complex business scenarios. You can implement conditional logic to route conversations based on context, integrate with external systems via a comprehensive connector ecosystem, and leverage Dataverse as a structured repository for your agents’ knowledge and state management. These capabilities transform agents from simple question-answering tools into intelligent participants in broader business workflows.

Understanding the migration process — what transfers automatically versus what requires manual reconfiguration—helps you avoid surprises and allocate appropriate time and resources. The copy operation handles core elements like instructions, knowledge sources from SharePoint and websites, and visual identity. Still, it leaves connector-based enterprise knowledge, embedded files, and specific advanced capabilities requiring deliberate setup in the new environment. Planning this work ensures a smooth transition without extended periods of functionality unavailability.

Please remember that the migration makes an independent copy rather than moving your agent throughout this process. This design allows you to maintain operational continuity with the existing Agent Builder version while you configure, test, and validate the Copilot Studio version. After confirming that the migrated agent is working correctly and that users have transitioned successfully, would you mind retiring the original to avoid a service availability gap?

Power Fx in Copilot Studio - The Computational Backbone of Enterprise Agents

Holger Imbery — Sat, 17 Jan 2026 00:00:00 +0000

Summary Lede

Building production-ready (conversational) agents in Microsoft Copilot Studio requires mastering a critical architectural principle: the LLM handles the words, Power Fx handles the computation. While large language models excel at understanding natural language and generating human-like responses, they cannot guarantee the deterministic, auditable calculations that enterprise automation requires. This article explains how Power Fx functions as the computational backbone of your agents, providing repeatable calculations, precise data transformations, and enforceable business rules—while the LLM manages the conversational flow. Whether you are implementing financial workflows, compliance-sensitive processes, or complex agent flows that integrate with Dataverse and Azure AI, understanding this separation of concerns transforms experimental chatbots into reliable enterprise automation solutions.

Read on to explore the practical formulas, variable scoping rules, and architectural patterns that make Power Fx essential for anyone building autonomous agents in the Microsoft ecosystem.

As autonomous agents and agent flows in Microsoft Copilot Studio become more capable and increasingly central to enterprise automation, understanding Power Fx is no longer optional-it is essential. The landscape of enterprise software development has undergone a fundamental transformation in recent years, with conversational interfaces and intelligent automation moving from experimental features to core components of business operations. Within this evolving ecosystem, Power Fx serves a critical architectural role in executing at-scale formula evaluation.

Power Fx acts as the deterministic backbone of your agent’s logic, providing the predictable, rule-based execution layer that complements the reasoning and language interpretation performed by the LLM. This relationship between Power Fx and the LLM represents a carefully considered architectural pattern: the language model excels at interpreting natural language, understanding context, and generating human-like responses, while Power Fx handles the precise computational work-calculations, data transformations, orchestration, and state management—that demands absolute consistency and auditability.

Consider the fundamental nature of how these systems operate. When a user interacts with an agent in Copilot Studio, their message arrives as unstructured natural language. The LLM processes this input, identifying intent, extracting entities, and determining which conversational flow to activate. However, once that intent is understood, the agent must often perform specific operations, such as retrieving data from a database, calculating values based on business rules, validating input against regulatory constraints, or determining which subsequent action to take based on complex conditions. These operations cannot rely on probabilistic reasoning or approximate outputs. They require exactness, repeatability, and transparency-qualities that define Power Fx as a formula language.

This distinction becomes particularly significant in enterprise contexts where agents handle sensitive operations, financial transactions, compliance-related decisions, or integration with systems of record. In such environments, every calculation must produce identical results under identical conditions, every data transformation must be traceable, and every decision point must be explicable to auditors or regulatory bodies. Power Fx provides this foundation, creating a clear separation between the uncertainty inherent in language understanding and the precision required for executing business logic.

For enterprise makers-especially those working with Dynamics 365, Azure AI, Dataverse, and embedded automation—Power Fx empowers you to:

Implement precise decision logic rather than relying solely on probabilistic reasoning.
Ensure repeatable behavior for compliance-sensitive processes.
Manipulate data, variables, collections, and system values in a predictable low-code expression language that aligns with existing Power Platform experience.
Control how topics communicate with each other, how state flows through the conversation, and how the agent orchestrates automation.

If you are building autonomous agents, approvals, orchestrations, or contextual AI inside business applications, then familiarity with Power Fx is a core competency. This article introduces Power Fx in the context of Copilot Studio agent flows, provides examples, and covers advanced concepts using Microsoft’s official materials.

Power Fx in Copilot Studio - A Structured Language for Agent Logic

Power Fx in Copilot Studio is fully integrated into the authoring canvas and can be used in:

Set a variable nodes
Message and Question nodes
Condition nodes
Action nodes
Adaptive Card logic
Question behavior configurations
and last but not least in Agents Flows

This allows you to compute values, evaluate conditions, transform user input, and shape your agent’s responses. The placement of Power Fx within these specific nodes reflects a deliberate architectural decision: computational logic resides at the points where data enters the system, transforms during processing, or influences routing decisions. Rather than scattering imperative code throughout the conversational design, Power Fx expressions encapsulate discrete computational operations within well-defined boundaries, making the agent’s behavior more comprehensible during both initial development and subsequent maintenance.

Copilot Studio supports a subset of Power Fx functions, not the complete Power Apps function library. This distinction matters for practitioners migrating existing formulas or designing complex computational logic. Many core table, text, date, and conditional functions behave identically to their Power Apps counterparts, allowing experienced Power Platform developers to transfer their existing knowledge directly into agent development. Functions governing table manipulation-such as Filter, ForAll, and AddColumns-generally operate as expected. Similarly, text processing functions, including Concatenate, Left, Right, and Text formatting operations, maintain their familiar semantics. Date and time calculations through functions such as DateAdd, DateDiff, and Now provide consistent behavior across the platform.

Example: Using Power Fx to Modify and Display Dates

A foundational example from Microsoft’s documentation shows how Power Fx can improve user-facing output. In the tutorial, the agent receives an order number and wants to communicate an estimated delivery date.

Step-by-step

Create a new variable in a Set a variable value node.
Switch to the Formula tab.
Insert this formula to compute the delivery date:

Text(
     DateAdd(
         Now(),
         3,
         TimeUnit.Days
     ),
     DateTimeFormat.LongDate
 )

This takes the current datetime, adds three days, and formats the result as a long date string, which is then stored in the Topic variable Date:

This example highlights why Power Fx is necessary: your agent performs deterministic date transformation without involving the LLM, ensuring accuracy and consistency.

Understanding Variable Prefixes in Copilot Studio

Unlike Power Apps, variables in Copilot Studio require specific prefixes in Power Fx formulas. These prefixes are crucial for correct data access. They indicate the variable’s scope and context:

Topic. for topic-level variables (e.g., Topic.UserMessage)
System. for system-level variables (e.g., System.CurrentDateTime)
Global. for agent-level variables (e.g., Global.AgentName)

These prefixes make the flow of state inside an agent explicit, which is essential for multi-step logic and orchestrations.

Use literal values in a formula

When working with Power Fx formulas, you are not restricted to referencing variables alone. The language also permits the direct inclusion of literal values-that is, fixed data values that you explicitly write into the formula itself. When you choose to embed a literal value within a formula expression, it is necessary to express that value according to the syntax conventions that correspond to its underlying data type. The table below provides a detailed enumeration of the data types supported in Power Fx, along with illustrative examples that demonstrate the specific formatting requirements for representing literal values of each type.

Type	Format examples
String	`"hi"`, `"hello world!"`, `"copilot"`
Boolean	Only `true` or `false`
Number	`1`, `532`, `5.258`, `-9201`
Record and Table	`[1]`, `[45, 8, 2]`, `["cats", "dogs"]`, `{ id: 1 }`, `{ message: "hello" }`, `{ name: "John", info: { age: 25, weight: 175 } }`
DateTime	`Time(5,0,23)`, `Date(2022,5,24)`, `DateTimeValue("May 10, 2022 5:00:00 PM")`
Choice	Not supported
Blank	Only `Blank()`

Common Power Fx formulas

The following table lists data types and Power Fx formulas you can use with each data type. Links point directly to Microsoft’s official documentation for your reference.

Type	Power Fx formulas
String	Text function Concat and Concatenate functions Len function Lower, Upper, and Proper functions IsMatch, Match, and MatchAll functions EndsWith and StartsWith functions Find function Replace and Substitute functions
Boolean	Boolean function And, Or, and Not functions If and Switch functions
Number	Decimal, Float, and Value functions Int, Round, RoundDown, RoundUp, and Trunc functions
Record and Table	Concat and Concatenate functions Count, CountA, CountIf, and CountRows functions ForAll function First, FirstN, Index, Last, and LastN functions Filter, Search, and LookUp functions JSON function ParseJSON function
DateTime	Date, DateTime, and Time functions DateValue, TimeValue, and DateTimeValue functions Day, Month, Year, Hour, Minute, Second, and Weekday functions Now, Today, IsToday, UTCNow, UTCToday, IsUTCToday functions DateAdd, DateDiff, and TimeZoneOffset functions Text function
Blank	Blank, Coalesce, IsBlank, and IsEmpty functions Error, IfError, IsError, IsBlankOrError functions

Power Fx (Copilot Studio) Cheat Sheet

Numbers & Math

Function	Description	Example → Result
Abs	Absolute value of a number	Abs(-55) → 55
Round	Round to the specified number of digits	Round(3.14159, 2) → 3.14
RoundDown	Round down to the specified digits	RoundDown(3.9, 0) → 3
RoundUp	Round up to the specified digits	RoundUp(3.1, 0) → 4
Trunc	Truncate a number toward zero	Trunc(-3.9) → -3
Int	Integer part of a number	Int(7.8) → 7
Value	Convert text to number	Value(“42”) → 42
Decimal	Convert string to decimal number	Decimal(“12.5”) → 12.5
Average	Average of a table expression or set of numbers	Average([1,2,3,4]) → 2.5
Degrees / Acos / Asin / Atan / Atan2 / Cos / Cot	Trig functions (radians) and conversions	Degrees(Acos(0.5)) → 60

Text

Function	Description	Example → Result
Concatenate / Concat	Join strings or project and join from a table.	Concatenate(“Hello “, “Holger”) → “Hello Holger”
Text	Convert values (like numbers/dates) to text with formatting.	Text(1234.5, “[$-en-US]#,##0.00”) → “1,234.50”
Len	Length of a string.	Len(“Copilot”) → 7
Lower / Upper / Proper	Case conversion.	Proper(“microsoft copilot”) → “Microsoft Copilot”
Find	Find substring position (1‑based).	Find(“pilot”,”Copilot”) → 3
Replace / Substitute	Replace text by position or substring.	Substitute(“a-b-c”,”-“,”/”) → “a/b/c”
StartsWith / EndsWith	Prefix/suffix check.	StartsWith(“Copilot”,”Co”) → true
EncodeUrl / EncodeHTML	URL/HTML escape text.	EncodeUrl(“a b”) → “a%20b”
Char	Return character from code.	Char(10) → “\n” (newline)

Regex/Match note: Some community posts mention gaps for Match/MatchAll support in Copilot Studio; validate in your environment.

Logical & Conditional

Function	Description	Example → Result
And / Or / Not	Boolean logic (&& also supported for And).	And(5>3, 2<1) → false
If	If/elseif/else branching. (Power Fx general)	If(Score>=50,”pass”,”fail”) → “pass” (when Score=80)
Switch	Multi-branch by matching a value. (Power Fx general)	Switch(“B”,”A”,1,”B”,2,0) → 2
Coalesce	First non‑blank value.	Coalesce(Blank(), “fallback”) → “fallback”
Boolean	Convert text/number/dynamic to Boolean.	Boolean(“true”) → true
Blank	Produce a blank (NULL-like) value.	Blank() → blank

Date & Time

Function	Description	Example → Result
Now / Today / UTCNow / UTCToday	Current local/UTC date-time or date. (Power Fx general)	Text(UTCNow(), “yyyy-mm-ddTHH:MMZ”) → e.g., “2026-01-16T13:11Z”
Date / DateTime / Time	Construct date/time values.	Date(2026,1,16) → 2026-01-16
DateValue / TimeValue / DateTimeValue	Parse date/time from text.	DateTimeValue(“May 10, 2022 5:00 PM”) → 2022-05-10 17:00
Day / Month / Year / Hour / Minute / Second / Weekday	Extract parts of a date-time.	Year(Date(2026,1,16)) → 2026
DateAdd	Add days/months/quarters/years.	DateAdd(Date(2026,1,16), 30, Days) → 2026-02-15
DateDiff	Difference between two dates in a unit.	DateDiff(Date(2026,1,1), Date(2026,1,16), Days) → 15
EDate	Add/subtract months, preserving day-of-month when possible.	EDate(Date(2026,1,31), 1) → 2026-02-29 (leap handling varies by calendar)
TimeZoneOffset	Minutes difference from UTC. (Power Fx general)	TimeZoneOffset(Now()) → e.g., 60 for CET winter.

Tables, Records & Collections

Function	Description	Example → Result
AddColumns	Return a table with computed columns added.	AddColumns([1,2,3], “Squared”, Value*Value) → [{Value:1,Squared:1}, …]
DropColumns	Remove one or more columns.	DropColumns(Table, “Temp”) → table without Temp
Count / CountA / CountIf / CountRows	Count numbers, non‑blank, conditional, or rows.	CountIf([1,2,3,4], Value>2) → 2
Filter	Filter records by a predicate. (Power Fx general)	Filter(Orders, Amount>1000) → high‑value orders
Search	Full/partial text search on columns. (Power Fx general)	Search(Products, “pro”, “Name”) → matching rows
LookUp	First record matching a condition. (Power Fx general)	LookUp(Users, Email=”a@b.com”) → record or blank
Distinct	Unique values from a column/table.	Distinct([1,1,2]) → [{Value:1},{Value:2}]
First / Last / FirstN / LastN / Index	Access first/last/Nth items. (Power Fx general)	Index([10,20,30],2) → 20
ForAll	Iterate over a table and evaluate an expression. (Power Fx general)	ForAll([1,2,3], Value*2) → [2,4,6]
AsType	Treat a record reference as a specific table type.	AsType(AnyRecord, MySchema) → typed record
Column / ColumnNames	Retrieve column names/values from a Dynamic value.	ColumnNames(DynamicVar) → [“Col1”,”Col2”,…]

JSON & Dynamic

Function	Description	Example → Result
JSON	Convert a value to a JSON text string. (Power Fx general)	JSON({a:1}) → “{“a”:1}”
ParseJSON	Parse JSON text to an untyped object. (Power Fx general)	ParseJSON(“{"x":5}”) → dynamic with x=5

Colors (useful for UI-rich responses or cards)

Function	Description	Example → Result
ColorValue	Parse CSS name or hex color.	ColorValue(“#0078D4”) → color value
ColorFade	Fade a color by a percentage.	ColorFade(ColorValue(“#000000”), 0.5) → 50% lighter

Web/Encoding Utilities

Function	Description	Example → Result
EncodeUrl	URL-encode special characters.	EncodeUrl(“a+b c”) → “a%2Bb%20c”
EncodeHTML	Escape characters for HTML context.	EncodeHTML(“”) → “”

Conversions & Misc

Function	Description	Example → Result
Dec2Hex	Convert number to hexadecimal string.	Dec2Hex(255) → “FF”
Boolean	Convert text/number/dynamic to true/false.	Boolean(1) → true

Copilot Studio usage tips (quick)

Variables : Always prefix (System., Global., Topic.).

Example in a condition:

If(Global.TotalSales > 100000, “High”, “Normal”)

Literals : Enter values in the proper literal format (strings in quotes, tables in [], records in { }).

Number formatting : Remember US decimal and comma separators when entering parameters and numeric literals.

Why Power Fx Complements the LLM

As discussed throughout this article, the architectural relationship between Power Fx and the large language model in Copilot Studio reflects a deliberate separation of concerns that addresses the distinct computational requirements of conversational agents in production environments.

Power Fx provides several specific technical capabilities that prove essential when building agents that must operate reliably within enterprise constraints:

Deterministic logic for calculations, data manipulations, conditional flows, and business rules. When an agent needs to calculate tax amounts, apply discount rules, validate input against business constraints, or execute multi-step conditional logic, these operations must produce identical results every time the same inputs are provided. Power Fx expressions execute predictably, returning the same output for the same input state, which allows developers to reason about agent behavior with confidence. This repeatability becomes particularly important when agents handle financial calculations, regulatory compliance checks, or data transformations that feed into downstream systems, where inconsistencies could cause operational failures or data integrity issues.

Control over tables and collections, enabling reproducible filters, aggregations, and mappings. Agents frequently work with structured data retrieved from Dataverse, SharePoint lists, SQL databases, or external APIs. Power Fx provides a functional programming model for manipulating these datasets, filtering records based on specific criteria, transforming record structures through projection operations, aggregating values across collections, and joining data from multiple sources. These operations occur within the agent runtime without requiring external service calls, reducing latency and simplifying the execution path. The table manipulation functions in Power Fx allow developers to express complex data operations concisely while maintaining clarity about what transformations occur at each step in the conversational flow.

A predictable mechanism for Adaptive Cards, Instructions, and Tool execution, integrating logic with user interface and automation components. Adaptive Cards often display dynamic content computed from variables, user input, or system state. Power Fx expressions embedded within Adaptive Card definitions calculate values for display, control visibility of card elements, and format data appropriately for presentation. Similarly, when configuring instructions that guide the LLM’s behavior or when invoking tools and actions that connect to external systems, Power Fx expressions prepare parameters, validate preconditions, and transform results. This integration keeps computational logic close to the context where it executes, making the agent’s behavior more transparent during development and troubleshooting.

This architectural division-where the language model handles natural language understanding, intent recognition, and response generation, while Power Fx handles computational operations, data manipulation, and business rule execution—establishes a foundation for building agents that are both conversationally capable and operationally reliable. The LLM component addresses the inherent ambiguity and variability of human language. In contrast, the Power Fx component ensures that the agent’s interactions with data and systems follow defined, testable, and auditable rules. This separation allows organizations to deploy conversational agents for production workloads where behavior must be consistent, outcomes must be traceable, and operations must comply with regulatory or security requirements.

Practical Importance in Agent Flows

Agent flows in Copilot Studio represent the execution layer where autonomous agents interact with external systems and coordinate multi-step processes. These flows handle operations such as invoking HTTP endpoints, querying databases through connectors, triggering approval processes in Power Automate, and coordinating data exchange between Dataverse and third-party systems. Within this execution context, Power Fx functions as the computational layer that bridges conversational state and system integration requirements.

Specifically, Power Fx expressions within agent flows address several technical requirements:

Data validation before action execution. Before invoking an external API or updating a database record, the agent must verify that input parameters meet expected formats, fall within acceptable ranges, and satisfy business constraints. Power Fx expressions embedded in condition nodes or within action input configurations evaluate these constraints, preventing invalid data from reaching external systems. For example, validating that a submitted order total matches the sum of line items, or confirming that a date falls within an acceptable scheduling window.
Parameter computation for system integrations. External actions often require specific data formats, calculated identifiers, or transformed values. Power Fx formulas prepare these parameters by concatenating strings into required formats, converting data types, performing mathematical operations on numeric inputs, or extracting subsets of structured data. This transformation occurs within the agent flow itself, reducing the need for intermediate services or custom code.
Routing decisions based on runtime state. Agent flows frequently branch based on system variables, user permissions, data lookup results, or environmental conditions. Power Fx expressions in condition nodes evaluate these factors to determine execution paths, directing high-value transactions to approval workflows while auto-approving standard requests, routing requests to different backend systems based on geographic region, or selecting between cached and real-time data sources based on freshness requirements.
Consistency across repeated executions. Agent flows execute repeatedly across many user sessions, often handling similar requests with varying input parameters. Power Fx ensures that identical inputs produce identical computational results, making flow behavior predictable and testable. This consistency allows developers to validate flow logic through test cases and maintain confidence that production behavior matches development expectations.

The combination of conversational agents, agent flows, and Power Fx creates a technical architecture in which natural-language interfaces trigger structured automation processes. Understanding Power Fx becomes necessary when building agents that must reliably interact with enterprise systems, enforce business rules during automated processes, or maintain data integrity across conversational and transactional boundaries.

Conclusion

As autonomous agents become increasingly central to enterprise automation, Power Fx has emerged not merely as a convenient formula language but as an essential architectural component that defines the boundary between conversational flexibility and computational precision. The relationship between Power Fx and the large language model in Copilot Studio represents a carefully considered separation of concerns: the LLM excels at interpreting natural language and managing conversational context, while Power Fx provides the deterministic execution layer for calculations, data transformations, business rule enforcement, and state management.

This architectural division addresses fundamental requirements that enterprise agents must satisfy. When agents handle financial transactions, compliance-sensitive decisions, or integrations with systems of record, they require computational operations that produce identical results under identical conditions, logic that remains auditable and traceable, and behavior that remains consistent across thousands of user interactions. Power Fx delivers these guarantees through its functional programming model, explicit variable scoping with Topic, System, and Global prefixes, and comprehensive libraries for manipulating strings, dates, numbers, tables, and records.

For enterprise makers working with Dynamics 365, Azure AI, Dataverse, and agent flows, Power Fx competency directly impacts the reliability, maintainability, and regulatory compliance of deployed solutions. Understanding how to implement conditional logic in condition nodes, transform data before action execution, validate input against business constraints, and prepare parameters for external system integrations determines whether conversational agents can transition from experimental prototypes to production workloads that handle critical business processes.

The examples and patterns discussed throughout this article-from basic date formatting to complex table manipulations, from variable scoping rules to integration with Adaptive Cards and agent flows—form the foundation for building agents that are both conversationally capable and operationally reliable. As the Power Platform continues to evolve and autonomous agents take on increasingly sophisticated responsibilities, deep familiarity with Power Fx will remain a core competency for anyone designing enterprise-grade conversational automation within Microsoft’s ecosystem.

Building Intelligent, Agentic Applications in VS Code - A Technical Deep Dive into the AI Toolkit Extension Pack

Holger Imbery — Sat, 27 Dec 2025 05:38:53 +0000

Technical overview and configuration guide (December 2025)

Summary Lede The Visual Studio Code AI Toolkit Extension Pack now delivers a complete local-to-cloud development workflow for intelligent, agentic applications—from interactive prototyping and multi-agent visualization to single-click deployment on Microsoft Foundry.

Recent updates expand the model catalog (including Anthropic Claude variants), introduce graph-based workflow visualization, and enable seamless conversion between declarative YAML and code-first agent implementations.

Combined with integrated tracing, evaluation frameworks, and MCP tool support, the toolkit transforms VS Code into a production-grade environment for building AI systems that are observable, testable, and operationally ready.

Why read this article?

If you’re building AI agents or intelligent applications, this deep dive shows you how to move from local experimentation to production deployment without changing tools or rewriting workflows.

You’ll learn the technical architecture behind:

Model selection
Agent orchestration
MCP tool integration
Evaluation pipelines
Runtime tracing

Complete with concrete YAML-to-code examples in Python and C#.

Whether you’re prototyping your first agent or scaling multi-agent systems, this guide maps the capabilities that reduce friction, improve quality assurance, and accelerate time-to-value.

Why This Matters Technologically

Modern intelligent applications are rarely monolithic; they are composed of multiple agents, external tools, datasets, and evaluation loops. The latest update explicitly targets this reality by enabling local-to-cloud continuity : developers can author agents, run and trace them locally in VS Code, and then deploy to Microsoft Foundry with one click—preserving workflow context for orchestration, visualization, and evaluation in the cloud.

This unification reduces friction between prototyping and production, so the same artifacts (YAML workflows, agent code, test scaffolds, traces) remain coherent across environments.

The Intelligent Apps documentation complements that with a detailed breakdown of:

Model Tools
Agent & Workflow Tools
MCP Workflow

This makes it clear how the extension organizes:

Resources (models, agents, MCP servers)
Operations (playground, bulk runs, evaluation, fine‑tuning, conversion, tracing)

The effect is a predictable, composable development experience built on VS Code conventions, so teams can adopt AI features without retooling their entire stack.Download the extension

Business Benefits: From Experimentation to Operability

Faster Time-to-Value :

Local debugging, hosted agent playgrounds, unified visualization, and single‑click deployment shorten the path from experimentation to a hosted, observable agent system. This reduces handoffs and rework, which are typical cost centers in AI projects transitioning to production.

Governance and Evaluability :

The toolchain supports structured evaluation (metrics, tasks, datasets) and integrates tracing and graph visualization for multi‑agent workflows. This makes quality assessment and incident response more systematic, improving confidence in releases and reducing operational risk.

Model Choice and Portability :

The model catalog spans hosted providers and local runtimes (ONNX, Ollama) and includes recent Anthropic models. Organizations can align model selection to privacy constraints, latency targets, and cost envelopes without changing development surfaces.

Incremental Adoption :

YAML‑based declarative workflows can be converted into code aligned with the Microsoft Agent Framework—enabling teams to start simple and progressively customize. This staged path mitigates the risk of over‑engineering early while preserving the option to extend later.

A Detailed Overview of the VS Code AI Toolkit Extension

The extension exposes a structured VS Code view with distinct sections that map cleanly to an AI app’s lifecycle. Below is a technical tour of the major components.

Resources

Models : Lists deployed and available models you can use in projects. It is the anchor for selecting runtime backends before entering playgrounds or builders.

Agents : Displays agents you have created or deployed through the toolkit. This centralizes agent artifacts used by downstream tools such as bulk runs and tracing.

MCP Servers : Enumerates Model Context Protocol servers you’ve added, which provide tool‑use capabilities (databases, APIs, services). This turns agents from pure language generators into action‑taking systems.

Model Tools

Model Catalog : A unified browser over models from GitHub, ONNX, Ollama, OpenAI, Anthropic, Google, and others. Engineers can compare options and evaluate tradeoffs before binding a model to an agent. The Ignite update explicitly calls out Anthropic Claude variants now accessible here.

Model Playground : An interactive chat/test surface for prompts, parameters, and multimodal inputs. It’s designed for rapid hypothesis testing—ideal before formalizing agent instructions.

Conversion : Utilities for converting, quantizing, and optimizing pretrained models for local execution (CPU/GPU/NPU). This aids portability and cost control, particularly for edge scenarios.

Fine‑tuning : Workflows to adapt foundation models using custom datasets either locally (GPU) or in Azure Container Apps (GPU). This allows domain specialization without abandoning the VS Code environment.

Agent & Workflow Tools

Agent Builder : Previously known as Prompt Builder, this now emphasizes agent construction—authoring instructions, integrating tools (MCP servers), and emitting production‑ready code with structured outputs. Engineers can scaffold agents in Python or .NET, add function/tool calls, and iterate interactively.

Bulk Run : Batch execution across multiple models or prompts to compare outputs at scale. This is essential for regression testing and prompt robustness analysis.

Evaluation : Dataset‑driven assessment using metrics such as relevance, similarity, coherence, and task‑specific criteria. It complements bulk runs to form a measurable confidence baseline.

Tracing : Runtime telemetry over reasoning steps, tool calls, and latency hotspots. When paired with multi‑agent visualization, tracing enables efficient root‑cause analysis and performance tuning.

MCP Workflow

Add MCP Server / Create New MCP Server : Discover featured MCP servers, validate local Node/Python environments, and scaffold new servers. This extends agents with secure, composable capabilities to query systems, invoke APIs, and manipulate external state.

Visualization and Round‑Trip Development

Graph Visualizer : The Ignite update introduces interactive visualization of multi‑agent workflows, illuminating node execution and connections. This makes debugging and understanding complex orchestration feasible within the editor.

Local‑to‑Cloud Roundtrip : Seamless switching between VS Code and the Foundry portal for YAML workflows, playgrounds, and templates—plus single‑click deployment. This aligns dev, ops, and evaluation artifacts without format drift.

Development Workflow: From YAML to Code and Back

A pragmatic pattern for teams is to begin with declarative YAML workflows in Foundry for clarity and quick iteration. When customization demands increase—e.g., custom tool‑use, advanced control flow, or instrumentation—convert YAML to Agent Framework code and continue in Agent Builder. GitHub Copilot scaffolding helps produce maintainable code with tracing hooks and test scaffolding. This reduces context switching and preserves lineage from declarative prototypes to code‑first implementations.

Evaluation, Quality, and Operability

Robust AI systems demand measurable quality. The toolkit’s Evaluation module standardizes dataset‑based comparisons and metric computation, and the Tracing module surfaces runtime behaviors, tool invocations, and timing. In multi‑agent setups, the graph visualizer further supports operational debugging. Collectively, these capabilities enable repeatable tests, incident triage, and performance baselining before and after deployment.

Model Strategy and Local Execution

The extension’s catalog allows mixing hosted providers with local execution via ONNX and Ollama. Local modes are valuable for privacy (data residency), latency (edge), and cost (avoiding per‑token charges for specific workloads). Where domain‑specific performance is needed, fine‑tuning in local GPUs or Azure Container Apps helps align responses to proprietary data without a wholesale platform shift. Model conversion ensures the operational footprint matches target hardware, including NPU acceleration paths.

Practical Onboarding

Installation is straightforward through the Marketplace; once installed, the AI Toolkit icon appears in the Activity Bar. The Get Started walkthrough introduces you to playground usage and guides you through key views. From there, the usual order of operations is: pick a model in the Catalog, validate prompts in the Playground, construct an agent in Agent Builder (adding MCP tools where necessary), test at scale with Bulk Run, evaluate with datasets and metrics, trace runtime behavior, and finally deploy to Foundry for managed orchestration and visualization.

YAML to Code Example: Customer Support Agent with MCP Tools

A minimal, illustrative example that shows how a declarative YAML workflow (as you’d author in Microsoft Foundry and open in VS Code) can be “converted” into a code-first agent implementation in Python using the same concepts—model selection, agent instructions, tool use via MCP servers, tracing, and structured outputs.

Note : The YAML schema for Foundry agent workflows can evolve; the sample below is intentionally simplified to make the mapping clear. In practice, you’d open YAML workflows in VS Code and iterate or move to code using the AI Toolkit’s Agent Builder with Copilot-assisted scaffolding. The latest update explicitly highlights YAML workflows and code-first customization, MCP tool integration, and multi-agent visualization.

YAML (declarative) — simplified agent workflow

# file: customer_support_agent.yaml
version: 0.3
workflow:
  name: customer_support_agent
  description: >
    Respond to customer queries by retrieving account/order context
    and grounding the answer in internal knowledge and web documentation.

runtime:
  model: anthropic/claude-sonnet-4.5 # Example model from the catalog
  temperature: 0.2
  max_tokens: 1024
  trace: true

inputs:
  - name: user_query
    type: string
    required: true
  - name: user_id
    type: string
    required: true

instructions:
  role: system
  content: |
    You are a customer support agent. Always ground answers in retrieved data.
    If data is missing or uncertain, ask a clarifying question.
    Return a structured JSON with fields: `answer`, `sources`, `confidence`.

tools:
  - type: mcp
    name: mcp-postgres
    title: "Customer DB"
    actions:
      - id: fetch_user_context
        description: "Get profile, subscription status, recent orders."
        parameters:
          user_id: string
  - type: mcp
    name: mcp-knowledgebase
    title: "Internal KB"
    actions:
      - id: kb_search
        description: "Search internal support articles"
        parameters:
          query: string
  - type: http
    name: web_docs
    base_url: "https://example-docs.company.com/api/search"
    actions:
      - id: external_doc_search
        parameters:
          query: string

orchestration:
  steps:
    - id: get_context
      call: fetch_user_context
      with:
        user_id: ""
    - id: kb
      call: kb_search
      with:
        query: ""
    - id: docs
      call: external_doc_search
      with:
        query: ""
    - id: answer
      llm:
        prompt: |
          User question: 
          Context: 
          KB hits: 
          External docs: 
          Compose an actionable answer. Include citations and confidence score.
      output_schema:
        type: object
        properties:
          answer: { type: string }
          sources: { type: array, items: { type: string } }
          confidence: { type: number }

policies:
  grounding: required
  pii_handling: redaction

outputs:
  from_step: answer

What this conveys:

Model/runtime settings (model name, temperature, tracing).
Inputs the agent needs (user query, user ID).
System instructions guiding behavior.
Tools (two MCP servers + an HTTP tool) and action signatures.
Orchestration that calls tools, then composes a final LLM answer with a structured schema.
Policies for grounding and PII handling.

These elements align with the AI Toolkit guidance on agent instructions, MCP tool use, and structured outputs; YAML workflows are explicitly highlighted in the latest update to facilitate easy editing and conversion for code-first customization.

Equivalent code (Python) — agent, tools, tracing, structured output

Below is a conceptual Python implementation that mirrors the YAML’s behavior. It registers tools (including MCP-backed actions), defines agent instructions, handles orchestration, and returns a typed result. In VS Code, you’d typically scaffold this in Agent Builder and refine it with Copilot.

# file: customer_support_agent.py
from dataclasses import dataclass
from typing import List, Dict, Any, Optional

# --- Model wrapper (conceptual) ---
class LLM:
    def __init__ (self, name: str, temperature: float = 0.2, max_tokens: int = 1024, trace: bool = True):
        self.name = name
        self.temperature = temperature
        self.max_tokens = max_tokens
        self.trace = trace

    def invoke(self, system: str, prompt: str, tools_context: Dict[str, Any]) -> Dict[str, Any]:
        """
        Invoke the model with system instructions + composed prompt.
        Tracing hooks would capture tool calls, token usage, latency, etc.
        In AI Toolkit, tracing is integrated; here we show conceptually.
        """
        # ... integrate with your runtime (e.g., Anthropic/OpenAI via AI Toolkit)
        # Return a structured JSON per output schema.
        return {
            "answer": "Here is a grounded response with steps and references...",
            "sources": tools_context.get("sources", []),
            "confidence": 0.82
        }

# --- MCP tool interfaces (conceptual stubs) ---
@dataclass
class MCPAction:
    id: str
    description: str

class MCPServer:
    def __init__ (self, name: str, title: str, actions: List[MCPAction]):
        self.name = name
        self.title = title
        self.actions = {a.id: a for a in actions}

    def call(self, action_id: str, **kwargs) -> Dict[str, Any]:
        # In VS Code, you configure MCP servers in Agent Builder; the runtime validates
        # your Node/Python environment and lets agents call tools programmatically.
        # Here we simulate dispatch.
        # e.g., fetch_user_context(user_id=...), kb_search(query=...), etc.
        if action_id == "fetch_user_context":
            return {"user": {"id": kwargs["user_id"], "tier": "Pro"}, "orders": [{"id": "O-1001", "status": "shipped"}]}
        if action_id == "kb_search":
            return {"hits": [{"id": "KB-42", "title": "Reset password"}, {"id": "KB-7", "title": "Refund policy"}]}
        raise KeyError(f"Unknown action: {action_id}")

# --- HTTP tool (simplified) ---
class HttpTool:
    def __init__ (self, base_url: str):
        self.base_url = base_url
    def get(self, endpoint: str, **params) -> Dict[str, Any]:
        # Replace with real HTTP calls or use an MCP wrapper that performs HTTP.
        return {"hits": [{"url": "https://example-docs.company.com/articles/abc", "title": "Policy overview"}]}

# --- Structured output schema ---
@dataclass
class AgentOutput:
    answer: str
    sources: List[str]
    confidence: float

# --- Agent implementation ---
class CustomerSupportAgent:
    def __init__ (self, model_name: str = "anthropic/claude-sonnet-4.5"):
        self.llm = LLM(name=model_name, temperature=0.2, max_tokens=1024, trace=True)
        # Tool registry mirrors YAML:
        self.db = MCPServer(
            name="mcp-postgres",
            title="Customer DB",
            actions=[MCPAction(id="fetch_user_context", description="Get profile, subscription status, recent orders.")]
        )
        self.kb = MCPServer(
            name="mcp-knowledgebase",
            title="Internal KB",
            actions=[MCPAction(id="kb_search", description="Search internal support articles")]
        )
        self.web = HttpTool(base_url="https://example-docs.company.com/api/search")

        # System instructions (from YAML 'instructions'):
        self.system_instructions = (
            "You are a customer support agent. Always ground answers in retrieved data. "
            "If data is missing or uncertain, ask a clarifying question. "
            "Return a structured JSON with fields: `answer`, `sources`, `confidence`."
        )

    def run(self, user_query: str, user_id: str) -> AgentOutput:
        # Orchestration steps:

        # 1) fetch_user_context
        context = self.db.call("fetch_user_context", user_id=user_id)

        # 2) kb_search
        kb_hits = self.kb.call("kb_search", query=user_query)

        # 3) external_doc_search (HTTP, simplified)
        docs_hits = self.web.get("/search", query=user_query)

        # Compose prompt (YAML step 'answer.llm.prompt')
        prompt = (
            f"User question: {user_query}\n"
            f"Context: {context}\n"
            f"KB hits: {kb_hits}\n"
            f"External docs: {docs_hits}\n"
            "Compose an actionable answer. Include citations and confidence score.\n"
        )

        # Invoke the LLM with tracing enabled (per runtime.trace)
        result = self.llm.invoke(
            system=self.system_instructions,
            prompt=prompt,
            tools_context={
                "sources": [
                    *(h.get("id") for h in kb_hits.get("hits", [])),
                    *(hit.get("url") for hit in docs_hits.get("hits", []))
                ]
            }
        )

        # Validate against the expected schema
        return AgentOutput(
            answer=result["answer"],
            sources=result.get("sources", []),
            confidence=float(result.get("confidence", 0.0))
        )

# --- Example usage ---
if __name__ == " __main__":
    agent = CustomerSupportAgent()
    output = agent.run(user_query="How do I update my billing address?", user_id="U-12345")
    print(output)

Key Takeaways

System Instructions and Runtime Configuration

The system instructions and runtime parameters (model selection, temperature, tracing) translate directly from YAML declarations into code constructor arguments or instance state.

MCP Tool Integration

MCP tools defined in YAML become instantiated server objects with callable actions in your code. VS Code’s Agent Builder validates the execution environment and assists with configuring featured MCP servers.

Orchestration Flow

The orchestration steps declared in YAML map to explicit, ordered method calls in code. These calls retrieve context from tool results and ultimately compose an LLM invocation.

Structured Output Enforcement

The YAML output schema corresponds to a typed data structure (such as an AgentOutput dataclass in Python or record in C#), ensuring type safety and schema compliance at the agent boundary.

How you’d do this inside VS Code AI Toolkit

Open the YAML workflow in VS Code (via Foundry extension) and review steps, tools, and instructions. The latest update enables round‑trip editing and single‑click deployment between VS Code and Foundry.
Use Agent Builder to scaffold an agent :
- Paste the system instructions
- Set model/runtime parameters
- Attach MCP servers (featured MCP servers are discoverable and validated)
Translate orchestration steps :
- For each YAML step, create corresponding code functions/method calls
- Copilot can generate the scaffolds and function signatures aligned with best‑practice patterns
Run locally with tracing and iterate in the Hosted Agents Playground:
- Use the graph visualizer to understand multi‑agent flows
- Debug step execution with integrated tracing
Evaluate using datasets and metrics :
- Test relevance, similarity, coherence, or task‑specific criteria
- When the agent meets acceptance criteria, deploy to Foundry to manage orchestration and enterprise hosting

Optional: C# skeleton for the same agent

If your runtime preference is .NET, here’s a skeleton showing the exact mapping. In VS Code, you’d choose the language when scaffolding the agent.

// file: CustomerSupportAgent.cs (conceptual)
public sealed class CustomerSupportAgent
{
    private readonly Llm _llm;
    private readonly McpServer _db;
    private readonly McpServer _kb;
    private readonly HttpTool _web;
    private readonly string _systemInstructions =
        "You are a customer support agent... Return JSON with `answer`, `sources`, `confidence`.";

    public CustomerSupportAgent(string modelName = "anthropic/claude-sonnet-4.5")
    {
        _llm = new Llm(modelName, temperature: 0.2, maxTokens: 1024, trace: true);
        _db = McpServer.Create("mcp-postgres")
                       .WithAction("fetch_user_context", "Get profile, subscription status, recent orders.");
        _kb = McpServer.Create("mcp-knowledgebase")
                       .WithAction("kb_search", "Search internal support articles");
        _web = new HttpTool("https://example-docs.company.com/api/search");
    }

    public AgentOutput Run(string userQuery, string userId)
    {
        var context = _db.Call("fetch_user_context", new { user_id = userId });
        var kbHits = _kb.Call("kb_search", new { query = userQuery });
        var docs = _web.Get("/search", new { query = userQuery });

        var prompt = $@"User question: {userQuery}
Context: {Serialize(context)}
KB hits: {Serialize(kbHits)}
External docs: {Serialize(docs)}
Compose an actionable answer. Include citations and confidence score.";

        var result = _llm.Invoke(_systemInstructions, prompt, toolsContext: new {
            sources = ExtractSources(kbHits, docs)
        });

        return new AgentOutput(result.answer, result.sources, (double)result.confidence);
    }
}

Mapping cheat‑sheet (YAML → Code)

YAML Element	Code Mapping
`runtime.model`, `temperature`, `max_tokens`, `trace`	LLM constructor/fields
`instructions.role: system` / `content`	Agent’s system prompt string passed as `system`
`tools.type: mcp` and `actions`	MCP server objects with callable actions
`orchestration.steps`	Ordered function/method calls culminating in `llm.invoke(...)`
`output_schema`	Dataclass/DTO enforcing structured return types

Conclusion

The VS Code AI Toolkit Extension Pack, especially with the latest update, provides a comprehensive environment for building intelligent, agentic applications that span local experimentation to cloud deployment.

By supporting declarative YAML workflows alongside code-first implementations in Python and C#, it offers flexibility for teams at different maturity levels.

The integration of MCP tools, tracing, evaluation, and multi-agent visualization addresses key challenges in operability and quality assurance.

This deep dive illustrated how to leverage these capabilities effectively, empowering developers to create robust AI systems with confidence and efficiency.