Support Copilot an unofficial Mastra demo

Unofficial demo — not affiliated with or endorsed by Mastra.

A one-line prompt edit, and the support agent stopped citing its sources.

Same Mastra agent. Same tool. Same model. The only difference between these two versions is the system prompt below — the kind of change a team makes for snappier replies, not realizing what it costs. The diff is real. What it does to the agent's answers is what the eval suite further down exists to catch.

agents/support-copilot.instructions v1 baselinev2 — "snappier replies" 
Loading prompt diff…

Did it regress?

Eight support questions, run against both versions through a real @mastra/core Agent with one tool (a docs search over Mastra's own documentation), scored with three real @mastra/evals scorers: tool-call accuracy (did it search the docs first — deterministic, no model call), faithfulness (is the answer actually supported by what it retrieved), and answer relevancy (does it address the question).

Loading eval results…

Try it yourself

Ask the live agent something a Mastra user might ask. Pick which version answers — the baseline that grounds and cites, or the "snappier" one that doesn't. Calls a Cloudflare Workers AI model live, through the same tool and prompts shown above.

Ask a question below, or pick a suggestion above.

Max 280 characters. Rate-limited per visitor and capped for the whole demo per day.

How this is built

  1. Docs knowledge base

    13 short, faithful extracts from Mastra's real public docs (agents, tools, memory, workflows, evals, observability, deployment), rewritten as clean prose with links back to the source pages — shared/docs-corpus.mjs.

  2. One tool, one agent, two versions

    A single searchDocs tool (plain keyword search over the corpus, not a vector DB — an honest MVP choice) wired into a real @mastra/core Agent. Two versions share the model and tool and differ only in system prompt — scripts/run-eval.mjs.

  3. Real scorers, real runEvals

    Mastra's own runEvals() harness runs both versions against 8 test questions with createToolCallAccuracyScorerCode, createFaithfulnessScorer, and createAnswerRelevancyScorer from @mastra/evals.

  4. Model: Cloudflare Workers AI

    Both the agent and the LLM-judge scorers use @cf/meta/llama-3.3-70b-instruct-fp8-fast via workers-ai-provider — no OpenAI/Anthropic keys, everything runs on the free Workers AI tier.

  5. Live chat mirrors the same config

    The chat above calls Workers AI directly from a Cloudflare Pages Function (functions/api/chat.js), reusing the exact same system prompts, tool, and model as the precompute — not wrapped in the Agent class, to keep the edge-runtime bundle simple within the build timebox. Same behavior, different call path — see the README for why.

Methodology & limits

  • n = 8 questions. A smoke test of the regression, not a statistically powered benchmark. Six are answerable from the docs; two are deliberate "trick" questions where the honest answer is "the docs don't cover that" — good bait for an ungrounded agent to hallucinate on.
  • The judge isn't independent. Faithfulness and answer-relevancy both use the same model that generates the answers (llama-3.3-70b-instruct-fp8-fast, the only reliably tool-calling model available on this account's free tier). A model grading its own family's output is a known weakness of this setup, not hidden from it.
  • Retrieval is keyword search, not embeddings. searchDocs scores token overlap over 13 short documents. It's what a customer engineer would wire up first, before reaching for a vector DB — it is not a RAG benchmark.
  • The regression flag is a fixed threshold. An item is flagged "regressed" when v2's faithfulness or answer-relevancy drops by 0.25 or more versus v1. Arbitrary, stated plainly, and visible per-item so you can judge the raw numbers yourself.
  • Cloudflare's free Workers AI tier caps at 10,000 neurons/day account-wide, shared with other projects on this account. See "What's real vs precomputed" below for what that means for this deploy.

What's real vs precomputed

Real and precomputed

Every score and answer in "Did it regress?" above — when that section isn't showing a pending state — came from an actual scripts/run-eval.mjs run: a real Mastra Agent, real createTool, real runEvals(), real @mastra/evals scorers, against the real Workers AI REST API. Nothing there is hand-written or simulated. The raw run (data/raw-eval-runs.json) and the shaped result (public/data/eval-results.json) are both committed.

Real and live

The chat above calls Workers AI on every message, through functions/api/chat.js, using the exact prompts and tool shown in the diff and in "How this is built".

Not the Agent class, live

The live chat calls the AI SDK + workers-ai-provider directly rather than instantiating @mastra/core's Agent inside the Cloudflare Pages Function. Mastra documents Cloudflare as a supported runtime for a full Mastra server deployment (@mastra/deployer-cloudflare), but that's a separate deploy target from a single lightweight Pages Function — wiring it in was out of scope for this build's timebox. The precompute script is where the real Agent/Scorer classes run end to end.

About this demo

Built by Ulf Faldt Reifenberg for an application to Mastra's Customer Engineer role. The job spans customer-facing agent work and core framework improvements, with reliability, eval strategy, and monitoring named explicitly — so this demo tries to show that loop directly: a realistic regression, a real Mastra eval suite that catches it, and the actual prompts and code behind both.

Source: github.com/AlwaysBlind/mastra-customer-engineer-demo. Unofficial — not affiliated with or endorsed by Mastra.