Which MCP server actually works today?
Last updated:
A listing in an MCP registry shows that a server can be found. It does not show that the server connects, authenticates and answers a real tool call today, and the official registry says to assume minimal-to-no moderation. To know whether it works, run one representative call yourself, record when and how, and keep "it failed" separate from "we could not run it". For the two tool categories it covers, CanaryIndex publishes exactly that kind of dated, raw evidence.
Facts on this page were checked against the linked sources and CanaryIndex's published data on October 4, 2026.
What a registry listing does and does not tell you
- The official MCP Registry is permissive by design. Its moderation policy says: "The MCP Registry is quite permissive! We only remove illegal content, malware, spam, and completely broken servers." It also says consumers "should assume minimal-to-no moderation". It does not remove "low-quality or buggy servers" or servers with security vulnerabilities (moderation policy).
- Choosing a server is your responsibility. MCP's security policy says "the decision to connect to and use a server rests with the user or administrator" (SECURITY.md).
- Entries can be listed but unusable. A client developer who validated registry entries found that "about half of the servers have no configuration". Of those that do, he found "only about a third" produce a valid server configuration (registry discussion #635, October 2025).
- Catalogs carry broken entries too. Open issues in Docker's MCP catalog include a server that does not start (#1720) and an entry whose SSE endpoint returns HTTP 410 (#4735). Catalog metadata goes stale too: six source links return 404, though that issue says the images themselves were live, so it is not evidence those servers are down (#4564).
How to check a server yourself
- Connect with the protocol version it speaks. Under the current MCP specification (2026-07-28), clients negotiate the version on each request. Servers on 2025-11-25 and earlier expect an
initializehandshake first. A good client handles both (versioning). - Authenticate the way a real user would. An open endpoint that works for you without credentials tells you nothing about the authenticated path.
- Call
tools/listand confirm that the tool you need is there, with the input schema you expect (tools). - Make one representative
tools/callwith an input whose correct answer you know. A JSON-RPC error is a protocol failure. A result withisError: trueis a tool failure. A successful result can still be wrong, so compare it with the expected answer. - Record the date, client, network location and account you used, and repeat. One success is an observation, not a reliability figure.
The official MCP Inspector (npx @modelcontextprotocol/inspector --cli) is a scriptable client for exactly these calls.
Three outcomes, not two
| Outcome | Meaning | Counts against the tool? |
|---|---|---|
| Passed | The call completed and the output met the expected answer | No |
| Failed | The call completed but returned an error or a wrong answer | Yes |
| Failed to observe | The checker could not run the call: plan limits, credentials, rate limits or its own network | No. It says nothing about the tool. |
Merging the last two is the most common way a scorecard misleads.
What CanaryIndex publishes today
CanaryIndex is a narrow, public scorecard, not an MCP directory:
- What it covers: comparable Apify Actors in two categories, PDF text-layer to Markdown and direct audio/video URL to transcript. Apify Actors can also be called through Apify's MCP server. Each tool gets the same fixed public fixture weekly (Mondays, 08:17 UTC, on GitHub-hosted runners), under a $5 monthly budget cap.
- How a tool passes: when the run succeeds and the objective score reaches the category threshold. For transcripts, the threshold is 0.70, where the score is 1 − word error rate.
- What the latest run found: that run (October 2, 2026, 22:49 UTC) recorded 7 tools:
- The publisher's own PDF tool passed with all expected tokens. - The publisher's own transcriber completed but did not pass (word error rate 0.4211, score 0.58). - All 5 third-party Actors are failed to observe. Apify returned HTTP 403 because the publisher's plan cannot run other publishers' public Actors. That is a limit of the checker's account, not a failure of those tools.
- Recommendations are currently empty. A recommendation needs a current pass and a public listing, and nothing met both at that run. Read it yourself from recommendations.json or the raw latest observation.
- What it does not certify: security, privacy, legal compliance, universal correctness or future availability. Its observations come from one fixture, one account and one runner location at one time (method).
FAQ
Does a server being in the official MCP Registry mean it works?
No. The registry removes completely broken servers when they are found, but it does not test entries and does not remove buggy ones. Test the server yourself before an agent depends on it.
Is a reliable MCP server also a safe one?
No. Reliability says the server answered correctly when called. Whether to trust it with your data and credentials is a separate security review.
Which MCP server should my agent use for PDF-to-Markdown or transcription?
Check CanaryIndex's recommendations.json. If the list for your category is empty, as it was in the latest run, no tool currently has public passing evidence there. Run your own representative call before relying on one.