How to Use Microsoft Scope: The Open-Source AI Coding Agent Evaluation Platform
CodexFuse Team · 2026-11-06T11:52:16.363Z · 20 min read min read
How to Use Microsoft Scope: The Open-Source AI Coding Agent Evaluation Platform
Learning how to use Microsoft Scope starts with a change of mindset: you are not installing a coding assistant, you are installing a laboratory for testing one. Microsoft Scope is a self-service, open-source platform for measuring the agentic coding experience of your product surfaces, skills, MCP servers, and extensions, and it runs the same coding task across different agents, tools, and models so you can compare the results side by side. This microsoft scope tutorial follows the official local-development path from Microsoft's own documentation, as of November 2026, and it works through one complete loop: install the stack, run one task, read the Judge's report, and fix what the evidence shows.
You need Docker before anything else, because the local stack runs in containers, and our Docker beginner guide covers the runtime this guide assumes. Scope is a research preview, announced on 19 October 2026, only 18 days before this guide was written, so expect rough edges and moving documentation. There is no signup, no hosted tier, and no free trial: Scope is software you deploy yourself, and that single fact shapes everything that follows.
What Is Microsoft Scope?
Microsoft Scope is a self-service platform for measuring how AI coding agents use your software. It submits a coding task to an agent of your choice, watches the run, evaluates the outcome against criteria you define, and stores the evidence so you can compare runs side by side. The agents it drives today are GitHub Copilot, Claude Code, and VS Code with the Copilot driver extension, and every one of them runs through the Agent Client Protocol, which keeps the comparison honest across different tools.
Scope ai agent evaluation is a measurement discipline, not a leaderboard. Results describe the tasks and configurations you tested, not a universal ranking of agents, and Microsoft says so explicitly in its own documentation.
That framing matters because a single run can mislead you: agents are non-deterministic, and the same task can produce different behavior twice. Scope's answer is repeatability, the same task, the same criteria, and the same profile, run as often as you need.
What Is DevAx, and Why Do Agents Need Evaluating?
DevAx, short for Developer Agent Experience, is the practice Scope was built to measure. The idea is simple: AI coding agents are the newest users of your software, and they choose stacks, call APIs, and follow documentation just as human developers do. When your docs are ambiguous or your SDK is awkward, an agent fails quietly where a human would ask a question, and your product loses a user it never sees.
Scope is the DevAx evaluation platform that turns that idea into repeatable experiments. Microsoft's DevRel team started measuring this in the fall of 2025, running hundreds of agent sessions across Azure, Cosmos DB, SPFx, and Microsoft 365 Copilot extensions with the kinds of prompts real developers use.
The failure patterns were consistent: agents ignored available CLI tools and edited files by hand, anchored on the wrong docs section, and loaded extension tools but never called them, relying on training data two SDK versions behind. Those observations became the AX Practitioner Playbook, open-sourced alongside Scope on 19 October 2026, and the playbook's three outcomes are evaluate your technology, diagnose failures, and drive improvements.
What Scope Is Not
Scope is not a hosted IDE, and it is not an agent runtime you embed in your own products. It is a measurement platform that drives existing agents against tasks you control, and it never writes code for you directly. If you want a coding assistant, install one of the agents Scope can evaluate; Scope itself only watches and scores.
Scope is also not a secure sandbox for untrusted code, despite how some third-party coverage frames it. The local worker configuration mounts the Docker socket so agents can run containers, and a Docker socket can give extensive control over the host. Run experimental agents on a dedicated machine or virtual machine, keep the development stack off the public internet, and keep unrelated credentials and sensitive repositories out of that environment.
Scope is released under the MIT License, so it is free to use and extend, and there is no price to quote and no paid tier to unlock. The only costs are the ones your model and provider usage may incur, and the documentation states plainly that provider usage may incur charges or consume quotas, without giving dollar figures.
What You Need Before You Start
The local stack needs six tools, and the documentation pins two of them to exact versions. Node.js must be version 22 to match continuous integration, and pnpm must be version 10.29.1, pinned in the repository's package.json. Docker with Compose v2 runs the stack, mkcert creates trusted HTTPS certificates for the local sign-in emulator, and the GitHub CLI supplies the token for the Copilot quick start.
| Tool | Requirement |
|---|---|
| Git | Clone the repository; fork it first if you plan to contribute |
| Node.js | Version 22, matching continuous integration |
| pnpm | Version 10.29.1, pinned in package.json |
| Docker with Compose v2 | Run the local stack; use a current version with Compose Watch support |
| mkcert | Create trusted HTTPS certificates for the local sign-in emulator |
| GitHub CLI | Obtain a token for the Copilot quick start with gh auth login |
The one requirement that catches people out is the Copilot entitlement. Authenticating with gh auth login alone does not grant Copilot access, because the Copilot worker needs an active GitHub Copilot entitlement on the account that supplies its token.
You also need credentials with access to the models the Judge and other AI features use, and the local backing services themselves need no Azure subscription. If you plan to use the Claude Code worker instead, that worker needs its own credentials and model access, and our GitHub beginner guide covers the account and authentication side of that setup.
How to Install Microsoft Scope Locally
The install is seven steps, and the rest of this section walks through each one. No version number exists for Scope, because the repository has zero releases as of 6 November 2026, so this guide follows the current main branch and the commands below come from Microsoft's local-development documentation.
- Clone the repository and enter it with git clone https://github.com/microsoft/scope.git, then cd scope.
- Enable Corepack and install the dependencies with corepack enable, then pnpm install --frozen-lockfile.
- Authenticate with GitHub using gh auth login, from an account that has an active Copilot entitlement.
- Start the Copilot worker stack with pnpm docker:dev:copilot, which builds and starts the Portal, API, scheduler, Judge, token manager, gateway, and local dependencies.
- Approve the local certificate trust prompt when mkcert installs its local certificate authority on first use.
- Open the Portal with pnpm open:portal, which resolves to http://localhost:5100, and sign in with a seeded emulator user such as [email protected].
- Confirm the stack is healthy by building the CLI with pnpm build:cli, pointing it at the local API, and listing your projects.
If you came here looking for a scope install docker compose shortcut, the honest answer is that the official path runs Docker Compose through pnpm scripts, not through a bare docker compose up command. Third-party guides circulating since the announcement show commands such as npm install -g @microsoft/scope-cli and scope eval run, but those commands do not appear in Microsoft's official documentation, and copying them will waste an afternoon. Every command in this guide comes from the docs site, not from a roundup.
git clone https://github.com/microsoft/scope.git cd scope corepack enable pnpm install --frozen-lockfile gh auth login pnpm docker:dev:copilot pnpm open:portal
The first build downloads several images and can take some time, and database migrations plus development agent registration run automatically during startup. For persistent overrides, copy .env.local.example to .env.local and edit that file, and never put credentials in .env.base or commit them. The startup scripts generate the local environment file from .env.base and configure local sign-in over HTTPS.
How to Run Your First Evaluation in the Portal
Your first evaluation is a small hello-world task, and the docs use the same one: ask the agent to create a Node.js HTTP server and evaluate whether its source implements a GET /health route returning JSON. Start by selecting or creating a project to keep your evaluation data together, then create the task and at least one observable evaluation criterion. Open the run submission page, select the running Copilot agent and an available model, and attach your task and criteria, saving the agent configuration as a reusable profile if you plan to repeat it.
Submit the request and watch the run's live logs stream in from the worker. The status badge moves from pending to queued to processing to done, and the terminal outcome is succeeded, failed, or finished. If no models are available, check your provider access and the model-scanner logs before submitting, and remember that a registered agent is not necessarily a running one, so choose the worker your Compose command actually enabled.
Open the Report tab when the run finishes. It holds one entry per criterion with a pass or fail and a short rationale from the Judge, and for the hello-world set you should expect entries such as a package.json with express as a dependency, a main entry file that creates an Express server, and a GET / route that returns a hello-world response. If a criterion fails, the rationale explains why, which is often more useful than the verdict itself, because it tells you whether the task, the criteria, or the agent's instructions need fixing.
How to Write Criteria That Actually Measure Something
Criteria are the heart of Scope, and each one is a single, observable statement the Judge evaluates against the run's output. A good criterion is observable, meaning a human reading the same output would reach the same conclusion, atomic, meaning it checks one thing, and outcome-focused, meaning it describes the result rather than the path. The server returns JSON on /health is a good criterion; code is high-quality is not, because no two readers would score it the same way.
Criteria form a directed acyclic graph, because each criterion can declare parents through dependsOn, and a criterion is evaluated only when all of its parents pass. If a parent fails, its descendants are skipped and reported as such, which stops the Judge from scoring work that was never built. A graph with no dependencies at all is still a DAG, and cycles in dependsOn are rejected outright.
| Gate | Order | What it checks | What happens on failure |
|---|---|---|---|
| Requirements | First | Whether the task itself is understood and the required pieces exist | The pipeline stops; later gates are recorded as skipped |
| Build | Second | Whether the agent's code compiles and its dependencies resolve | The pipeline stops; later gates are recorded as skipped |
| Test | Third | Whether the built code behaves as the criteria demand | Dependent criteria inside the gate are skipped |
Gates are the stages a run passes through, in the order Requirements, Build, and Test, and each gate has its own iteration budget. If a gate does not pass before its budget runs out, the pipeline stops and later gates are recorded as skipped, and a failed criterion separately causes its dependents to be skipped inside the same gate. Criteria are mutable, and editing one replaces its stored definition, but past reports were computed from the criteria as they stood at submission time, so edits never rewrite history.
- id: has_package_json prompt: Has a package.json with express as a dependency - id: has_express_server prompt: Has a main entry file that creates an Express server dependsOn: [has_package_json] - id: has_root_route prompt: GET / returns a hello world response dependsOn: [has_express_server] - id: configurable_port prompt: The server listens on a configurable port dependsOn: [has_express_server]
The Judge reads the has_package_json criterion first, then has_express_server only if it passed, and then has_root_route and configurable_port in parallel only if has_express_server passed. When you create a criterion, Scope can generate the underlying evaluation prompt for you using GitHub Models, and you can edit the generated prompt before saving. MCP servers, skills, and extensions are all properties of the profile rather than the criteria, and our what is MCP guide explains the protocol those servers speak.
How to Read the Judge's Report Without Fooling Yourself
The automated Judge can make mistakes, so important conclusions need human review, and Microsoft says that in the same breath as it describes the Judge. The Judge is an LLM-based service that evaluates a completed run against its criteria and produces a structured pass/fail report with a per-criterion rationale. Treat that report as a starting point for inspection, not as proof, and always read the generated files, the workspace snapshot, and the captured agent activity behind any verdict that matters.
Repeatability is what turns a report into evidence. Run the same task under the same profile more than once, because repeatable configurations do not guarantee identical agent behavior, and a result that appears once may not appear twice.
When the Judge's rationale and the artifacts disagree, trust the artifacts, fix the criterion or the task prompt, and run again. That discipline is the whole point of an evaluation platform: it replaces arguments about what an agent did with a record you can both read.
How to Compare Two Agents on the Same Task
Comparison is where Scope earns its keep, and the reason teams use it to evaluate AI coding agents is that it holds everything constant except the one variable under test. Keep the task and the criteria constant while you swap the worker, the model, the MCP servers, or the skills, and save each configuration as a versioned profile so the comparison can be repeated exactly. Every saved profile version is immutable, so version 2 stays frozen forever once it is created, and that immutability is what makes a comparison reproducible weeks later.
Profiles bundle the worker, the model, the agent version, the MCP servers, the skills, and the extensions, and the VS Code Copilot worker is the only one that supports VS Code extensions. Our VS Code beginner guide covers the editor side if you want to evaluate that worker, and the other two workers are the GitHub Copilot CLI and the Claude Code CLI, both driven through the Agent Client Protocol.
Prompt features add a second axis to comparisons, because Scope detects characteristics of your task prompt such as asking for an API, asking for TypeScript, or asking for a database. Runs can then be grouped by feature, so you can compare agents across heterogeneous tasks instead of only across identical ones. That grouping is how a team moves from one-off experiments to a shared, reproducible measurement suite.
How to Use Scope from the CLI and REST API
The Portal is the easiest way to start, and the CLI and REST API are the way to automate. The scope CLI gives you the same submit-and-inspect workflow as the Portal, with real-time log streaming, flexible output formats, and easy integration into shell scripts. Install it with the documented installer, which pipes the repository's install script to bash, and point it at your local API before doing anything else.
curl --fail --location https://raw.githubusercontent.com/microsoft/scope/main/install-cli.sh | bash export SCOPE_API_URL=http://localhost:3100 scope run submit -m "Create a Hello World Node.js / Express REST API." -w coder-acp-copilot -c has_package_json has_express_dependency has_get_route scope run logs -i <request-id> scope run status -i <request-id> scope run list scope run get -i <run-id> scope run download -i <request-id> scope run retry -i <request-id>
Submit a run with scope run submit, passing the task message, the worker type, and the criterion IDs, or pass a scenario file and a saved profile instead. Logs stream in real time by default, and you can add --no-stream to submit silently and poll later, or pass -o json to get machine-readable output for a pipeline. Pin the agent version and a fixed profile version in automated pipelines, because reproducibility is the point.
The REST API exposes everything the Portal does under /api/v1, with an OpenAPI explorer at /api-docs on your deployment. Submit a request with a POST to /api/v1/requests, stream its logs over server-sent events, and create a report on demand with a POST to /api/v1/reports. One behavior to remember is that request submission is not idempotent: every POST creates a new request, even if you send the same payload twice, so a retrying client can accidentally double-submit.
How to Self-Host Scope on Kubernetes
Local Docker Compose is for development, and Kubernetes is the path for a shared deployment. Scope self-hosts on Kubernetes with the deploy/helm/scope chart from the repository, and the Azure option provisions the cluster and its data services with a Bicep template from deploy/azure. The chart installs six core services into the scope namespace, and each one has a distinct job in the evaluation pipeline.
| Service | Role |
|---|---|
| API | REST API, server-sent-event log streaming, and configuration and run management |
| Portal | Web interface for submitting runs and reading reports |
| Judge | Criteria evaluation against each completed run |
| Token Manager | Provider credential storage, validation, and round-robin allocation |
| Scheduler | Priority-aware dispatch of pending runs to workers |
| Report Generator | Background generation of run reports |
Worker scaling is queue-driven through KEDA, which watches the matching queue for each worker, and the current manifests scale from 2 Claude Code replicas to 10 ACP Copilot replicas. Our Kubernetes beginner guide covers the cluster side of that deployment, including the kubectl and Helm prerequisites the docs list. The chart has limits worth knowing before you commit: it does not yet template the AI Gateway or DevProxy sidecar or Kubedock, so scenarios needing HAR capture or in-task Docker access are not supported, and it installs no ingress controller, DNS, TLS, cert-manager, autoscaling, or observability stack.
The production reference architecture adds node-pool scaling and a FluxCD-applied Kustomize configuration on top of the quickstart chart, with External Secrets Operator supplying secrets from Key Vault. Worker pods run benchmark-controlled agent code, so the worker identity gets access to Blob Storage and queues only, with no Key Vault role, and credentials should be registered through the Portal's Secrets tab or the Token Manager API. That separation is deliberate: the backend identity can reach Key Vault, Redis, Blob Storage, and queues, while the worker identity cannot.
What Microsoft Learned From Its Own Evaluations
Scope was not built in the open and then handed to strangers: Microsoft's DevRel team ran it internally for months before the announcement, and the fixes it produced are the strongest argument for using it. The repository was created on 28 July 2026, 101 days before this guide, and the announcement followed on 19 October. For the Azure Cosmos DB Agent Kit alone, the evaluations led to 20 Agent Kit issues and 27 documentation issues being resolved, 47 issues in total, and one targeted Agent Kit update moved hierarchical partition-key adoption from 47 percent to 100 percent, a 53-point gain, while OFFSET/LIMIT usage fell from 47 percent to zero, a 47-point drop.
| Result | Figure |
|---|---|
| End-to-end scenarios evaluated | 8 |
| Languages covered | 5 |
| Agent Kit issues resolved | 20 |
| Documentation issues resolved | 27 |
| Hierarchical partition-key adoption | 47 percent to 100 percent |
| OFFSET/LIMIT usage | 47 percent to 0 percent |
# Numbers from Microsoft's own Scope evaluations, computed 6 November 2026 from datetime import date announcement_age = (date(2026, 11, 6) - date(2026, 10, 19)).days # 18 repo_age = (date(2026, 11, 6) - date(2026, 7, 28)).days # 101 issues_total = 20 + 27 # 47 partition_gain = 100 - 47 # 53 offset_drop = 47 - 0 # 47 spfx_before = 0 / 4 * 100 # 0.0 spfx_after = 4 / 5 * 100 # 80.0 keda_span = 10 - 2 # 8 print(announcement_age, repo_age, issues_total, partition_gain, offset_drop, spfx_before, spfx_after, keda_span) # Output: 18 101 47 53 47 0.0 80.0 8
Those gains describe the Cosmos DB scenarios Microsoft evaluated, not a general ranking of agents, and the team stresses that scoping every time it quotes the numbers. A smaller fix shows the same loop at work: SPFx release notes used filenames like release-1.22.md, and agents assumed that covered version 1.22.2 and never looked for the version-specific page.
Renaming the file to release-1.22.0.md moved the result from none of 4 runs reaching the right page to 4 of 5 runs doing so, a jump from 0 percent to 80 percent that cost one filename change. That is the DevAx loop in miniature: measure, diagnose, fix the docs, and rerun.
CodexFuse offers no Microsoft Scope tool, and this guide does not pretend otherwise: Scope is Microsoft's platform, and the work it asks of you is your own. What it gives back is evidence about how agents actually use your software, which is the one thing no amount of reading your own documentation can provide. Start with the local stack, run one task, read one report, and let the evidence decide what you fix first.
Sources
Microsoft Scope documentation, What is Scope? microsoft.github.io/scope/introduction/what-is-scope/. Fetched 6 November 2026.
Microsoft Scope documentation, Local development. microsoft.github.io/scope/getting-started/local-development/. Fetched 6 November 2026.
Microsoft Scope documentation, Concepts. microsoft.github.io/scope/introduction/concepts/. Fetched 6 November 2026.
Microsoft Scope documentation, Defining criteria. microsoft.github.io/scope/guides/defining-criteria/. Fetched 6 November 2026.
Microsoft Scope documentation, Submitting requests with the CLI. microsoft.github.io/scope/guides/submitting-requests-cli/. Fetched 6 November 2026.
Microsoft Scope documentation, Submitting requests with the REST API. microsoft.github.io/scope/guides/submitting-requests-api/. Fetched 6 November 2026.
Microsoft Scope documentation, Access. microsoft.github.io/scope/getting-started/access/. Fetched 6 November 2026.
Microsoft Scope documentation, Your first run. microsoft.github.io/scope/getting-started/first-run/. Fetched 6 November 2026.
Microsoft Scope documentation, Kubernetes self-hosting overview. microsoft.github.io/scope/self-hosting/overview/. Fetched 6 November 2026.
Microsoft Scope system architecture. github.com/microsoft/scope/blob/main/docs/architecture/system-architecture.md. Fetched 6 November 2026.
Introducing Scope: Open-source evaluation for AI coding agents. Microsoft Open Source Blog, 19 October 2026. opensource.microsoft.com/blog/2026/10/19/introducing-scope-open-source-evaluation-for-ai-coding-agents/. Fetched 6 November 2026.
The game has changed: AX Practitioner Playbook. microsoft.github.io/agent-experience/playbook/the-game-has-changed/. Fetched 6 November 2026.
AX Handbook and Project Scope. Patrick Chanezon, 19 October 2026. blog.chanezon.com/2026/10/19/ax-handbook-and-project-scope.html. Fetched 6 November 2026.
microsoft/scope repository. GitHub, fetched 6 November 2026.
Microsoft Scope AI coding agent evaluation self-hosted guide. aicybr.com, 26 October 2026. aicybr.com/blog/microsoft-scope-ai-coding-agent-evaluation-self-hosted-guide. Fetched 6 November 2026.
Microsoft Open-Sources Scope: Benchmarking Developer Agent Experience (DevAx) And AI Coding Agent Security. undercodetesting.com, 3 November 2026. Fetched 6 November 2026; its install commands were checked against the official docs and are not used in this guide.
Frequently Asked Questions
Is Microsoft Scope a hosted service I can sign up for?
No. Scope is software you deploy yourself, either as a local Docker Compose stack for development or on Kubernetes for a shared deployment. There is no signup page, no hosted tier, and no free trial, and you bring your own model and provider credentials.
Do I need a GitHub Copilot subscription to use Scope?
You need one for the default Copilot worker, because that worker requires an active GitHub Copilot entitlement on the account supplying its token. Signing in with gh auth login alone is not enough, and if you have no Copilot access, use the Claude Code worker instead, which needs its own credentials.
What is the difference between a criterion and a gate in Scope?
A criterion is a single reusable check, such as whether the server returns JSON on /health, while a gate is a stage of the run, such as Requirements, Build, or Test, that groups criteria in order. Criteria live in a dependency graph, and a failed criterion skips its dependents, while a gate that exhausts its iteration budget stops the pipeline and records later gates as skipped.
Can I trust the Judge's pass/fail verdict on its own?
No, not for important conclusions. The Judge is an LLM-based evaluator, and automated judges can make mistakes, so Microsoft's own guidance is to use representative tasks, repeated runs, and human review of the evidence. Always inspect the generated files and the workspace snapshot behind a verdict before acting on it.
How do I install the Scope CLI?
Run the documented installer, curl --fail --location https://raw.githubusercontent.com/microsoft/scope/main/install-cli.sh | bash, which installs the scope binary without needing GitHub authentication. Then set SCOPE_API_URL to your deployment's API address, for example http://localhost:3100 for local development, and confirm it with scope run list.
