Microsoft open-sources Scope, a self-hosted way to test how coding agents use your SDK or docs

Microsoft announced Scope on October 19. You give it a task, a starting codebase if you need one, and an agent profile. Profiles are versioned and record the worker, model, agent version, skills, MCP servers and extensions, so you can change one thing and rerun the same task.

Success criteria form a dependency graph, so a runtime check can depend on the build passing. An LLM-based Judge service scores each run against those criteria. Microsoft's own post says automated judges can make mistakes and that decisions still need a human to review the evidence. Each run keeps tool calls, generated files and workspace snapshots, so you can trace a failed check back to the step that caused it.

The example in the post is the Azure Cosmos DB team. Testing eight scenarios in five languages, it says it fixed 20 Agent Kit issues and 27 documentation issues, and after one Agent Kit update, use of hierarchical partition keys went from 47% to 100%. Scope is built on Kubernetes, runs locally with Docker Compose, and has a web portal, a CLI and a REST API. It isn't a hosted service, so you bring your own model and agent provider access.