Tester

GA npm version CI License: MIT

Testing, evals, and conformance for Model Context Protocol (MCP) servers. MST is published as @gleanwork/mcp-server-tester, and its CLI is mst.

Write deterministic Playwright tests against your MCP tools, or run data-driven eval datasets — including LLM-based evaluation of tool discoverability.

Playwright Tests

The mcp Playwright fixture connects to your MCP server (stdio or HTTP) and exposes a high-level API for calling tools and asserting responses. Custom matchers keep assertions readable.

import { test, expect } from '@gleanwork/mcp-server-tester/fixtures/mcp';

test('read_file returns file contents', async ({ mcp }) => {
  const result = await mcp.callTool('read_file', { path: '/tmp/test.txt' });
  expect(result).toContainToolText('Hello, world');
  expect(result).not.toBeToolError();
});

test('server exposes required tools', async ({ mcp }) => {
  const tools = await mcp.listTools();
  expect(tools.map((t) => t.name)).toContain('read_file');
});

Playwright tests are fast, deterministic, and designed for CI. Use them for regression testing, schema validation, and protocol conformance. The framework includes built-in conformance checks for the MCP spec. mst generate records tool calls as a spec file you can edit (see CLI).

Available matchers:

Matcher	Description
toMatchToolResponse	Response exactly matches expected value (deep equal)
toContainToolText	Response contains expected substrings
toMatchToolSchema	Response validates against a Zod schema
toMatchToolPattern	Response matches a regex pattern
toMatchToolSnapshot	Response matches a saved baseline
toBeToolError	Response is (or is not) an error
toHaveToolResponseSize	Response size is within bounds
toSatisfyToolPredicate	Response satisfies a custom function
toHaveToolCalls	LLM called the expected tools
toHaveToolCallCount	LLM made N tool calls
toPassToolJudge	LLM evaluates response quality against a rubric

Eval Datasets

Eval datasets define cases as JSON: an input the client under test acts on, and assertions about what it did. Run them with runEvalDataset() in a Playwright test, on the mst client, or with mst run on any client (Claude Code, Cowork, ChatGPT). Checks on a single tool's response belong in Playwright tests, above.

{
  "name": "file-ops",
  "cases": [
    {
      "id": "read-config",
      "input": "What version is set in the app's config file?",
      "assertions": {
        "toolsTriggered": {
          "calls": [
            {
              "name": "read_file",
              "required": true
            }
          ]
        },
        "containsText": "1.0.0"
      }
    },
    {
      "id": "summarize-readme",
      "input": "Summarize the README in two sentences",
      "assertions": {
        "toolsTriggered": {
          "calls": [
            {
              "name": "read_file",
              "required": true
            }
          ]
        }
      },
      "judges": [{ "type": "rubric", "rubric": "groundedness" }]
    }
  ]
}

import { test, expect } from '@gleanwork/mcp-server-tester/fixtures/mcp';
import { loadEvalDataset, runEvalDataset } from '@gleanwork/mcp-server-tester';

test('file operations eval', async ({ mcp }, testInfo) => {
  const dataset = await loadEvalDataset('./data/evals.json');
  const result = await runEvalDataset(
    { dataset, client: 'mst', model: 'claude-haiku-4-5', defaultTrials: 5 },
    { mcp, testInfo }
  );
  expect(result.passed).toBe(result.total);
});

Supported assertions:

Assertion	Description
containsText	The client's answer includes expected substrings
matchesPattern	The client's answer matches a pattern
toolsTriggered	The client called the expected tools
toolCallCount	The client made a number of tool calls in a range

Judges go in the case's judges list, beside assertions: each scores the answer against a rubric or reference, such as { "type": "rubric", "rubric": "groundedness" } or a plugin judge "acme/judge/completeness".

LLM client mode

In LLM client mode, a real LLM receives your server's tool list and a natural language prompt, then decides which tools to call. This tests whether your tool names, descriptions, and input schemas are clear enough for autonomous use — a different question from whether the tools return correct output.

A case with input runs on the client. In a Playwright test, runEvalDataset names the client and model ({ dataset, client: 'mst', model: 'claude-haiku-4-5' }), and the mst client uses the test's MCP connection; an eval config names them with client and model. A case can set its own client, model and clientOptions.

{
  "id": "find-config",
  "input": "Find the application config file and return its contents",
  "assertions": {
    "toolsTriggered": {
      "calls": [
        {
          "name": "read_file",
          "required": true
        }
      ]
    }
  }
}

LLM client mode makes real API calls and produces non-deterministic results. Use trials to run a case multiple times and measure pass rate rather than expecting 100% on a single run. See the LLM Client Guide for configuration and cost management.

Installation

Requires Node.js 22+. These docs describe 2.0, which is in beta:

npm install --save-dev @gleanwork/mcp-server-tester@beta @playwright/test

npm install @gleanwork/mcp-server-tester installs 1.x, whose docs are at v1.1.1. Upgrading: Migration guide (v1.x → v2.0).

LLM-as-judge with the default Anthropic judge needs the Anthropic SDK:

npm install --save-dev @anthropic-ai/sdk

The SDK client's packages (ai and the @ai-sdk/* providers) are optional dependencies, installed by default; add them yourself if you install with --omit=optional.

Quick Start

npx @gleanwork/mcp-server-tester@beta init

The CLI wizard creates a playwright.config.ts, example tests, and a sample eval dataset configured for your server. See the CLI Guide for all options.

Configuration

Point the framework at your MCP server in playwright.config.ts:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  reporter: [['list'], ['@gleanwork/mcp-server-tester/reporters/mcpReporter']],
  projects: [
    {
      name: 'my-server',
      use: {
        mcpConfig: {
          transport: 'stdio',
          command: 'node',
          args: ['server.js'],
        },
      },
    },
  ],
});

For HTTP servers, set transport: 'http' and serverUrl. For servers that require OAuth, see the Transports Guide and CLI Guide for authentication setup, including CI/CD token management.

Plain stdio is supported by Claude CLI, Vercel AI SDK, ChatGPT Work/Codex, and Cowork evaluation clients, including Cowork on macOS and Linux. Cowork also supports private file-backed native proxies and mixed HTTP/stdio server sets. See evaluation client transport support for client-specific fields, environment rules, and setup ownership.

Documentation

• Quick Start — detailed setup and configuration
• Assertions — all assertion types including snapshot sanitizers
• LLM Client Simulation — tool discoverability testing
• LLM Gateways — routing MST's LLM calls through a gateway
• Traces — sending each trial to an OpenTelemetry collector
• API Reference
• Transports — stdio and HTTP configuration, OAuth
• Protocol Versions — legacy and 2026-07-28, protocol matrices, era-aware conformance
• Agent Skills — testing and evaluating skills served over MCP (SEP-2640)
• CLI Commands — init, generate, login, token, auth, datasets, judges, run, grade, batch, open, cowork
• Evaluation Framework — eval configs, plugins and their extension kinds, the run format
• Cowork and ChatGPT Desktop — running evals on desktop clients
• Design walkthrough and explainer — how evals are meant to run end to end, and what is built so far
• Architecture decisions — why MST is built as it is, including detached runs: a run on an environment that keeps going with the laptop off
• Glossary — the terms MST uses
• The run report: the report every eval run writes
• Development — contributing and building
• Migration Guide (v1.x → v2.0) — MCP SDK v2 and protocol versions
• Migration Guide (v0.12 → v1.0) — upgrading from pre-1.0 releases

AI Skills

Install AI skills to help your coding assistant write tests and evals, and improve tool metadata:

npx skills add -g gleanwork/mcp-server-tester

This installs skills globally so they're available across all your projects. Four skills are included:

Skill	Description
mcp-tester-guide	Reference: vocabulary, config, fixtures, matchers, eval configs, CLI, 1.x renames
write-mcp-test	Write Playwright tests that call tools directly and check them with matchers
write-mcp-eval	Write eval datasets and eval configs: cases a client and model act on, and variants
optimize-mcp-tool-metadata	Find tool-metadata changes that measurably help, with runToolOptimization

Compatible with Claude Code, Cursor, Windsurf, Copilot, and 40+ other AI agents.

Examples

The examples/ directory contains complete working examples:

• filesystem-server/ — Test suite for Anthropic's Filesystem MCP server: 5 Playwright tests, 11 eval dataset cases, Zod schema validation.
• sqlite-server/ — Test suite for a SQLite MCP server: 11 Playwright tests, 14 eval dataset cases.
• basic-playwright-usage/ — Minimal Playwright patterns.

Eval configs

For projects with multiple datasets, use an eval config. Dataset paths are shorthand for tagged file sources, while clients, metrics, judges, result stores, and other extensions are built-ins or come from plugins (referenced as <namespace>/<kind>/<name>, such as acme/judge/completeness):

{
  "name": "tool-selection-search",
  "datasets": [
    {
      "type": "file",
      "path": "evalsets/search.json"
    }
  ],
  "servers": {
    "prod": {
      "transport": "http",
      "serverUrl": "https://example.com/mcp"
    }
  },
  "client": "mst",
  "clientOptions": {
    "provider": "anthropic"
  },
  "metrics": ["passed", "tool_count"],
  "results": {
    "store": {
      "type": "file",
      "dir": ".mcp-test-results"
    }
  }
}

Run one eval config or a bounded batch:

npx mst run \
  --config eval.json \
  --plugins ./plugins \
  --dry-run

npx mst batch \
  --config-dir configs \
  --workers 4 \
  --skip-existing \
  --dry-run

Use variants to compare server sets or client configurations. A variant can override servers, client, model, client options, tool maps, input templates, metrics, and judges. The canonical execution primitives remain EvalDataset, EvalCase, MCPConfig, and runEvalDataset.

Plugins add dataset sources, clients, judges, pairwise judges, metrics, result stores, connectors, environments and shared configs. A plugin is a plain default-exported object in ESLint's shape, and its extensions are referenced as <namespace>/<kind>/<name>. See Plugins. Secrets remain environment-variable or plugin-owned runtime inputs and do not belong in committed eval configs.

Known Limitations

These MCP protocol features are not currently supported. These are deliberate scope decisions, not bugs:

• MCP prompts (listPrompts, getPrompt)
• Server-to-client notifications
• Streaming tool responses (callTool waits for the complete response)

If any of these affect your use case, please open an issue.

License

MIT