Elevenlabs Call

A remote MCP server that lets Poke place real outbound phone calls through ElevenLabs Conversational AI (over Twilio). Tell Poke "Call the dentist and book a cleaning next week" and an AI voice agent makes the call and reports back.

Runs on Cloudflare Workers (TypeScript). ElevenLabs + Twilio own the real-time voice streaming; this server is the control plane that starts calls and relays results.

Architecture

Poke  ──MCP (HTTP/JSON-RPC)──▶  Worker  ──REST──▶  ElevenLabs ConvAI ──▶ Twilio ──▶ ☎ callee
  ▲                              │  ▲
  └────push notification─────────┘  └──post-call webhook──┘
        (Poke inbound API)

• place_call provisions a per-call ElevenLabs agent tailored to the objective, triggers POST /v1/convai/twilio/outbound-call, and returns a call_id immediately (calls run for minutes — the tool never blocks).
• Completion is observed two ways: the get_call_status tool polls GET /v1/convai/conversations/{id}, and an ElevenLabs post-call webhook pushes a summary back to the user via the Poke inbound API. Both funnel through one idempotent finalize routine (notify once).

MCP tools

Tool	Purpose
place_call	Start an outbound call pursuing a natural-language objective to a to_number (E.164), with optional caller_context.
get_call_status	Status + transcript + outcome summary for a call_id.
list_calls	The user's recent calls, most recent first.
cancel_call	Best-effort cancel of an in-flight call.

Project layout

src/
  index.ts        HTTP router: /mcp, /webhook, /health
  mcp.ts          MCP JSON-RPC handler (initialize, tools/list, tools/call)
  tools.ts        The four tools + input validation (zod)
  elevenlabs.ts   Thin ElevenLabs ConvAI client
  poke.ts         Poke inbound-API client (push-back)
  finalize.ts     Idempotent reconcile/finalize (notify-once, agent cleanup)
  webhook.ts      Post-call webhook verification + processing
  store.ts        KV-backed CallStore (per-user scoping, rate limit, indexes)
  summarize.ts    Transcript → short outcome summary
  auth.ts / env.ts / types.ts
test/             vitest unit tests

Prerequisites / accounts

• ElevenLabs account + API key, with a Conversational AI phone number linked to Twilio (gives an agent_phone_number_id).
• Twilio account + a voice-capable number imported into ElevenLabs.
• Poke account + inbound API key (for push-back notifications).
• Cloudflare account (Workers + KV).

Setup

npm install

# 1. Create the KV namespace and paste the id into wrangler.toml
wrangler kv namespace create CALLS

# 2. Set secrets (never committed)
wrangler secret put ELEVENLABS_API_KEY        # ElevenLabs xi-api-key
wrangler secret put TWILIO_PHONE_NUMBER_ID    # ElevenLabs agent_phone_number_id (Twilio-linked)
wrangler secret put POKE_API_KEY              # Poke inbound API key
wrangler secret put WEBHOOK_SECRET            # shared secret for the post-call webhook HMAC
wrangler secret put MCP_BEARER_TOKEN          # token Poke must present to call /mcp

# 3. (optional) tune non-secret vars in wrangler.toml
#    DESTINATION_ALLOWLIST="+14155551234,+442071234567"   # empty = allow any (not recommended)
#    RATE_LIMIT_PER_HOUR="10"
#    AGENT_LLM="claude-haiku-4-5"            # any ElevenLabs-supported LLM enum value
#    TTS_MODEL_ID="eleven_v3_conversational" # realtime v3; plain "eleven_v3" is gated (see table below)
#    TTS_VOICE_ID=""                         # specific voice, or blank for the default
#    AGENT_LANGUAGE="en"

# 4. Deploy
npm run deploy

Configure the ElevenLabs post-call webhook

Post-call webhooks are configured in the ElevenLabs dashboard at the workspace level — they are NOT registered by this server's agent-creation code. Without this, calls still run but completion notifications never fire.

1. ElevenLabs dashboard → Settings → Webhooks → create a webhook with URL:

   https://<your-worker>.workers.dev/webhook
   

2. Copy the signing secret ElevenLabs generates for that webhook and set it as your Worker secret:

   wrangler secret put WEBHOOK_SECRET   # paste the ElevenLabs signing secret
   

   The Worker verifies the ElevenLabs-Signature header (t=<ts>,v0=<hmac-sha256> of "<ts>.<body>") against this secret, so the two must match or webhooks are rejected with 401.
3. In Agents → Settings, enable the post-call transcription webhook (and, if your workspace uses per-agent overrides, enable it on the agent too).

Because agents are created per-call by this server, prefer enabling the post-call webhook workspace-wide so every generated agent inherits it.

Agent model selection

Each call provisions a fresh agent using the model settings from wrangler.toml [vars]:

Var	Default	Notes
AGENT_LLM	claude-haiku-4-5	Any supported LLM enum value.
TTS_MODEL_ID	eleven_v3_conversational	Realtime v3 voice. Heads up: the plain eleven_v3 id is the gated "expressive" model and returns expressive_tts_not_allowed on accounts without that feature — use the _conversational variant for live calls (or fall back to eleven_turbo_v2_5 / eleven_flash_v2_5).
TTS_VOICE_ID	(default)	A specific voice id, or blank for the workspace default.
AGENT_LANGUAGE	en	Agent language.

These map to conversation_config.agent.prompt.llm, conversation_config.tts.model_id / voice_id, and conversation_config.agent.language.

Register the server in Poke

Either the web app or CLI:

# Web: poke.com/integrations/new  → name + URL https://<your-worker>.workers.dev/mcp
# CLI:
npx poke@latest mcp add https://<your-worker>.workers.dev/mcp -n "Phone Calls" -k "<MCP_BEARER_TOKEN>"

Poke sends Authorization: Bearer <token> and X-Poke-User-Id on every request; the Worker enforces both (bearer = auth, user id = scoping).

Local development

npm run dev                                   # wrangler dev on http://localhost:8787
npx poke@latest tunnel http://localhost:8787/mcp -n "Phone Calls (dev)"

Testing

npm test          # vitest unit tests (store scoping, rate limit, finalize idempotency,
                  # status mapping, summary, webhook signature, MCP dispatch)
npm run typecheck

Logs

Workers observability is enabled ([observability] in wrangler.toml), so invocation logs are retained and browsable in the Cloudflare dashboard (Workers → elevenlabs-call-mcp → Logs). For a live stream:

npx wrangler tail elevenlabs-call-mcp --format pretty

Useful log lines: webhook: finalized <call_id> -> done (full loop succeeded), place_call failed for <id>: … (raw upstream error), and webhook rejected: … (signature verification failed — usually a WEBHOOK_SECRET mismatch).

Safety notes

This server places real phone calls that cost money and reach real people. Guardrails:

• Bearer-token auth on the MCP endpoint; HMAC-verified webhook.
• Per-user rolling-hour rate limit (RATE_LIMIT_PER_HOUR).
• Optional destination allowlist (DESTINATION_ALLOWLIST) — strongly recommended in production.
• Per-user call isolation via X-Poke-User-Id; cross-user reads return not-found.
• Secrets are never echoed in tool output, logs, or errors.

Status & known limitations

Verified working end-to-end: place_call → per-call agent (Claude Haiku 4.5 + eleven_v3_conversational) → Twilio outbound → ElevenLabs processing → signed post-call webhook → /webhook (HMAC verified) → push to Poke.

Remaining caveats:

• Poke push: the inbound-API call returns 2xx (see src/poke.ts for the exact endpoint/payload); user-visible delivery still worth a manual confirm if you change the payload.
• cancel_call is best-effort — ElevenLabs exposes no first-class outbound-call kill switch; the server marks the record cancelled and tears down the agent, but an already-connected call may take a moment to drop.
• TTS model is account-gated — eleven_v3_conversational works here; plain eleven_v3 requires the Expressive TTS feature.
• Rate limit is a soft limit — the per-user KV counter isn't transactional, so it can be slightly exceeded under bursty concurrency.