Use HumanJudge in Claude Desktop — compare AI models with real human reviews
Add HumanJudge as a free MCP server in Claude Desktop. Ask Claude to compare AI models, surface flag patterns, or check content quality — backed by 16,668+ blind human reviews. Setup in 30 seconds.
If you use Claude Desktop for research, analysis, or decision-making about AI tools, you're probably running into the same wall: Claude can summarize what's on the web, but it can't tell you what real humans actually think of GPT-5.4's marketing copy versus Grok's.
The HumanJudge MCP server fixes that. Add one URL as a connector and Claude gets 5 native tools backed by 16,668 blind human reviews of 15 AI models. Ask the comparison question — Claude answers with real review data, not a web summary.
Who this is for
- • Researchers and analysts evaluating AI tools for adoption decisions
- • PMs and CTOs picking models for production deployments
- • Consultants advising clients on AI tool selection
- • Anyone who already lives in Claude Desktop and wants quality data without leaving
What you can ask Claude after connecting
| Question pattern | What Claude returns (via MCP tools) |
|---|---|
| "Which AI is best at marketing content?" | Ranked leaderboard from 16K+ reviews |
| "Compare Claude Opus and GPT-5.4 head to head" | Side-by-side scores + reviewer quotes |
| "What flag patterns appear most in Grok's outputs?" | Top flag categories with examples |
| "Check this draft against real human reviews" | Semantic match against 913 evaluated outputs |
| "What's the latest human verdict on each major model?" | Live feed of recent reviews |
Setup (30 seconds)
- 1
Open Settings → Connectors
In Claude Desktop, go to Settings → Connectors.
- 2
Add custom connector
Click "Add custom connector" and fill in:
Name:
HumanJudgeURL:
https://api.humanjudge.com/mcp - 3
Click Connect and sign in
Your browser will open. Sign in with your HumanJudge account and click Allow. Free account, takes 30 seconds.
- 4
Enable tools
Make sure the connector is enabled and tools are allowed under Tool permissions.
🔔 Turn off Web search for cleaner answers. Claude may prefer web search over the HumanJudge tools if both are enabled.
Example questions to try right now
"Which AI is best at marketing content writing?"
Ranked leaderboard with pass rates and flag counts.
"Compare GPT-5.4 vs Claude Opus 4.6 for blog content"
Side-by-side scores with where they differ and reviewer quotes.
"What patterns do humans flag most in Grok's outputs?"
Flag categories with frequency and examples.
"Check this draft: [paste your AI's output]"
Semantic match against 913 evaluated outputs.
FAQ
What is the HumanJudge MCP server? ▾
A free MCP (Model Context Protocol) server that lets Claude Desktop query 16,668+ blind human evaluations of AI models. Add it as a custom connector — Claude gets 5 native tools to compare models, see flag patterns, and check content quality.
How do I install it? ▾
Settings → Connectors → Add custom connector → paste https://api.humanjudge.com/mcp → click Connect → sign in with your HumanJudge account in the browser. Total setup time: about 30 seconds.
Do I need Claude Pro or Max? ▾
Yes. Custom connectors require a Claude Pro or higher plan from Anthropic. The HumanJudge MCP server itself is free.
Why turn off web search? ▾
Claude may prefer web search over MCP tools when both are enabled. Turning off web search forces Claude to use the HumanJudge connector for AI quality questions, which gives you real human review data instead of secondhand web summaries.
What models can I get data on? ▾
GPT-5.4, GPT-5.2, Claude Opus 4.6, Claude Sonnet 4.6, Gemini 3.1 Pro, Gemini 3 Flash, Grok 4, Grok 4.1 Fast, Qwen3 VL 235B, gpt-oss-120b, Marketeam.ai, Palmyra Creative, Jasper, Kimi K2.6 — 15 models in the marketing benchmark, more across other arenas.
Is the data real-time? ▾
Yes. Each tool call hits the live HumanJudge API. New reviews submitted today show up in your queries today.
What if the OAuth sign-in fails? ▾
Make sure popups aren't blocked, you have a HumanJudge account, and the URL is exactly https://api.humanjudge.com/mcp (no trailing slash, https not http).
The 5 tools Claude gets
Once connected, Claude has these native MCP tools available:
get_model_scores — Leaderboard with pass rates and flag counts
compare_models — Head-to-head comparison of 2–3 models
get_flags — Flag patterns and categories per model
check_content — Match text against evaluated AI outputs
get_latest — Most recent human verdicts
Want this in your other tools?
Last updated: 2026-05-10 · Data refreshes live from humanjudge.com