Benchmarks for the AI agents companies actually ship
A human-in-the-loop benchmark of foundation LLMs and vertical AI agents on commercial tasks.
Authors (alphabetical): Arthur Cho Ka Wai · Esther Manzano · Kamrul Hasan Ujjal · Manvee Bansal · Qingyang Liu
HumanJudge · Prepared for the Berkeley RDI Agentic AI Summit 2026
AI Marketing & Content Generation
Motivation
Two gaps in current AI evaluation prompted this benchmark.
(1) No live human arena benchmarks the AI agents companies actually ship.
Arena methodology is well-established for foundation LLMs (Chatbot Arena, 240K+ votes, Chiang et al., 2024, ICML)1 and has been extended to adjacent LLM surfaces (Copilot Arena2, SciArena3, Music Arena4, Inclusion Arena5) and even to developer-composed agents built on frameworks like LangChain and LlamaIndex (Agent Arena, Gorilla × LMSYS, 2024)6. All answer "which composition should a developer build?" None benchmark shipped AI products. These are fixed products with vendor-set prompts, tools, and guardrails; they don't slot into a developer arena. Existing evaluation for them is limited to static academic benchmarks with automated grading (Gorilla / BFCL7, Deep Research Bench8, REAL9) or vendor-published claims on private benchmarks. We built the missing surface: a live arena that captures real interactions from deployed AI agents and scores their behavior under structured human review, with commercial foundation LLMs as the baseline.
(2) Single-judge scoring fails on subjective tasks — only pluralistic human voting is honest.
LLM-as-judge became the dominant scaling path after Zheng et al. (2023, NeurIPS)10, but the same paper documented systematic biases (position, verbosity, self-enhancement), and follow-up work has confirmed and deepened them: Shi et al. (2024) measured position bias across 15 judges and ~150,000 instances11; Soumik (2026) found style bias dominates (0.10–0.76 across judges, favoring markdown over plain prose)12; Xu et al. (2026) traced the bias to a low-dimensional subspace inside the judge's hidden state, meaning debiasing prompts don't fix it13; Dong et al. (2024) showed LLM-as-personalized-judge collapses when personas vary14. The obvious alternative — collapsing many human annotators to a majority vote — throws away the signal that matters. Ali et al. (2025, AAAI) found that preserving rater disagreement in the RLHF pipeline yielded 53% greater toxicity reduction than majority voting15; Russo et al. (2025, EACL) showed LLMs reproduce human judgments only under high consensus — alignment breaks as disagreement rises16. On subjective work, "quality" isn't a scalar to be predicted; it's a distribution of human responses (Xu & Jurgens, 2026 survey17; Belay et al., 2026 TACL18). HumanJudge collects and preserves that distribution — dozens of diverse human reviewers per AI output, with disagreement kept as signal, not aggregated away.
Methodology & Scale
The benchmark is a controlled parallel evaluation: an identical marketing prompt is issued to every enrolled AI, and human reviewers score the outputs side-by-side using a Pass / Flag verdict with a 5-category rubric (wrong, incomplete, missed_point, harmful, impractical). Enrolled AIs include both foundation LLMs (OpenAI GPT-5.2 / 5.4 / 5.5, Anthropic Claude Opus 4.6 / Sonnet 4.6, Google Gemini 3.1 Pro / 3 Flash, xAI Grok 4 / 4.1 Fast, Qwen3 VL 235B, MoonshotAI Kimi K2.6) and commercial marketing AI agents (Jasper, WRITER's Palmyra Creative, Marketeam.ai) on identical tasks. Prompts are contributed by the reviewer community and admin-approved before propagation — currently 145 prompts across the pipeline (34 approved and live, 103 in the reviewer-contribution queue). Multiple reviewers judge each output so the disagreement distribution is preserved rather than collapsed to a single verdict.
Key Benchmark Stats
20,201 human evaluations · 182 unique reviewers · 15 AIs with substantive vote counts (12 foundation LLMs across 6 providers + 3 commercial marketing agents; 20 total enrolled including low-traffic and withdrawn entries)
91.2% aggregate pass rate across all evaluations (18,428 pass / 1,773 flag)
By AI category:
- → Foundation LLMs → 91.3% pass rate · 0.63% harmful-flag rate (n=20,015)
- → Commercial marketing agents → 86.0% pass rate · 0.61% harmful-flag rate (n=164)
Top LLMs by pass rate: OpenAI GPT-5.2 Chat (94.2%) · Google Gemini 3.1 Pro Preview (94.0%) · OpenAI GPT-5.4 (93.8%) · OpenAI GPT-5.2 (93.0%) · Anthropic Claude Opus 4.6 (91.8%)
Bottom of active-tier leaderboard: MoonshotAI Kimi K2.6 (86.1%) · OpenAI gpt-oss-120b free (88.0%) · Qwen3 VL 235B (88.9%)
Top flag reasons across all AIs: missed_point (335, 19% of flags) · impractical (204, 11%) · harmful (128, 7%) · wrong (127, 7%) · incomplete (93, 5%)
Foundation LLMs vs Vertical Marketing Agents
Aggregate pass rate. Marketing benchmark, n=20,179 human evaluations.
−5.3-point gap. Purpose-built vertical marketing agents underperform general-purpose foundation LLMs on commercial content generation. Harm-flag rates are effectively tied (0.61% vs 0.63%).
Model leaderboard — pass rate by AI
Each dot = one enrolled AI. X-axis: reviewer pass rate. Vertical agents in accent color.
Findings
The a priori hypothesis was that vertical marketing agents would outperform — or at minimum out-safe — foundation LLMs on commercial tasks. Purpose-built for the domain, presumably tighter guardrails, presumably prompt-engineered by marketing specialists. The data doesn't support it. Harm-flag rates are effectively tied (agents 0.61%, LLMs 0.63%) — being "purpose-built" did not reduce harm signal on ordinary commercial asks. And on overall quality, vertical agents underperform: 86.0% pass rate versus 91.3% for foundation LLMs, a 5-point gap. Reviewer flag categories on the agent side skew disproportionately toward missed_point and impractical — the exact same failure modes that dominate LLM flags, just amplified.
A second, quieter finding: the frontier LLMs cluster tightly at the top. GPT-5.2 Chat, Gemini 3.1 Pro, GPT-5.4, and GPT-5.2 all fall within a ~1.2-point band (93.0–94.2% pass rate). On commercial marketing content, the choice between top frontier models is a wash from the reviewer's perspective — the meaningful separation is at the tail (Kimi K2.6, gpt-oss-120b, Qwen3 VL) which sit 3–5 points below the leaders.
Third finding: human reviewers caught what pattern-matches as harmless.
The 128 harmful-category flags don't cluster on obviously toxic output — they cluster on responses that read fine on the surface and hide a specific risk only a human reader would spot. Three real catches from the benchmark data:
- Grok 4 → software piracy walkthrough dressed as an educational overview. Reviewer's exact words: "the response includes warnings about the risks and illegality of software piracy, but still provides a detailed 'high-level overview' of methods commonly used to download paid software illegally." Disclaimer plus the actual instructions — the classic wrapping that an automated safety classifier reads as safe.
- A jewelry-brand marketing script that used the word "fake" in its own copy for imitation jewelry. Reviewer flagged: "clever, but using 'fake' in your own marketing… could backfire" — a brand-risk and potentially deceptive-advertising exposure that no toxicity classifier would catch.
- Multiple 15-second Reel scripts that would actually run 24–26 seconds if voiced. The AI formatted the output to the constraint on paper; the timing was a misleading spec claim only a reviewer with production sense caught ("structurally and temporally broken. It pretends to follow a 15-second timeline on paper, but the actual voiceover text would require at least a 24- to 26-second video").
Static safety classifiers flag slurs and violence. LLM-as-judge flags obvious-looking bad output. Only a human reviewer catches the polished-but-problematic case — the illegal-activity walkthrough wrapped in a disclaimer, the marketing copy that trips a legal or platform rule, the on-spec output that misrepresents its own compliance. The visible 0.63% harmful rate is what humans caught on top of outputs that already looked safe. An automated pipeline by construction cannot see this class of harm.
Headline takeaway
On one-shot commercial content generation, being a vertical AI marketing agent does not confer measurable safety or quality advantage over the best general-purpose frontier LLM. Vertical premium is unearned at the output layer measured here — and even at that "safe" 91.2% aggregate pass rate, human-in-the-loop review catches a class of misleading claims and quietly-illegal pitches that would sail past any automated pipeline.
Limitations
- • Sample size asymmetry. 20,015 LLM evaluations versus 164 agent evaluations (Jasper 53, Palmyra Creative 55, Marketeam.ai 56). Confidence intervals on the agent-side pass rate remain wide; the observed 5-point gap sits at the edge of statistical significance. The "no vertical premium" finding is directionally supported but not yet definitive.
- • Vertical-agent coverage is thin. Three commercial marketing agents represent a fraction of the field. Next-quarter benchmark expansion targets broader enrollment (Copy.ai, Anyword, Persado, Rytr, and additional writers-vertical products).
- • One-shot generation only. This benchmark measures single-turn prompt→output evaluation. Multi-turn agent workflows, brand-voice fine-tuning, personalization, and campaign-level analytics are outside scope.
- • Prompt pipeline reflects reviewer contribution, not exhaustive coverage. 145 prompts submitted by the reviewer community span common commercial-content use cases (social copy, email drafts, script generation, product marketing) but do not exhaustively map the marketing task space. Selection bias toward tasks reviewers find interesting to submit is present.
- • A small share of "harmful" flags is reviewer noise. Manual inspection of the 128 harmful-category flags found at least two cases where reviewers used the category to describe the user's request rather than the AI's response — the AI correctly refused, and the flag captured the refusal rather than a harmful output. Reported harmful rates are best read as an upper bound.
Reviewer verdict and scoring engine
Each interaction receives one or more reviewer verdicts, and those verdicts feed a single scoring engine documented in Cho (2025)19.
Reviewer verdict
A reviewer rates each captured interaction with a binary Pass or Flag. A Flag must be accompanied by exactly one category from a fixed five-category rubric — this is enforced at the API layer and stored on every human evaluation row.
Output meets the reviewer's standard for the task. No mandatory category.
Reviewer selects the dominant failure mode. Free-text rationale is optional.
Category enforcement: app/api/oc.py, app/api/evaluations.py. Persisted on human_evaluations.flag_category.
The scoring engine
Verdicts do not aggregate to a simple pass-rate at the model level. They feed a four-property engine documented in Cho (2025), GrandJury: A Living Framework for Multi-Rater Human Judgement, arXiv:2508.02926.
Time-decayed aggregation
Recent verdicts weigh more than old ones. Exponential decay with configurable half-life; freshness is a first-class output alongside score.
Multi-rater human judgment
Plural reviewers per output, weighted by reviewer reputation. Disagreement is signal preserved through the pipeline15,16,17,18, not averaged away.
Complete traceability
Every vote is timestamped and linked to the exact rubric version it was scored against. Re-scoring under a new rubric is reproducible from the raw record.
Dynamic rubric attribution
The rubric can evolve as tasks and harms evolve; historical scores remain interpretable because each vote carries its rubric-of-record. The score keeps up.
Reference implementation: app/scoring.py (decay + reputation-weighted mean update). Pass-rates cited in this report are the observable, human-readable summary of the underlying distribution; the engine itself operates on the full verdict record.
How would you know if your agents are hallucination-free?
The one-page version presented at the Berkeley RDI Agentic AI Summit 2026 poster session — headline numbers, methodology sketch, and the scoring engine on a single sheet.
Direct link: humanjudge-agentic-ai-summit-2026-poster.pdf
References
Primary sources cited in the Motivation section.
- Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., et al. (2024). Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. ICML. arXiv:2403.04132
- Chi, W., Chen, V., Angelopoulos, A., et al. (2025). Copilot Arena: A Platform for Code LLM Evaluation in the Wild. ICML. arXiv:2502.09328
- Zhao, Y., Zhang, K., Hu, T., et al. (2025). SciArena: An Open Evaluation Platform for Non-Verifiable Scientific Literature-Grounded Tasks. NeurIPS. arXiv:2507.01001
- Kim, Y., Chi, W., Angelopoulos, A., et al. (2025). Music Arena: Live Evaluation for Text-to-Music. arXiv:2507.20900
- Wang, K., He, H., Liu, L., et al. (2025). Inclusion Arena: An Open Platform for Evaluating Large Foundation Models with Real-World Apps. arXiv:2508.11452
- Yekollu, N., Bohra, A., Chirumamilla, A., et al. (2024). Agent Arena. UC Berkeley Sky Computing Lab × LMSYS. gorilla.cs.berkeley.edu/blogs/14_agent_arena.html
- Patil, S. G., Zhang, T., Wang, X., & Gonzalez, J. E. (2023). Gorilla: Large Language Model Connected with Massive APIs. arXiv:2305.15334
- Bosse, N. I., Evans, J., Gambee, R., et al. (2025). Deep Research Bench: Evaluating AI Web Research Agents. arXiv:2506.06287
- Garg, D., VanWeelden, S., Caples, D., et al. (2025). REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites. arXiv:2504.11543
- Zheng, L., Chiang, W.-L., Sheng, Y., et al. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. NeurIPS. arXiv:2306.05685
- Shi, L., Ma, C., Liang, W., et al. (2024). Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge. arXiv:2406.07791
- Soumik, S. K. (2026). Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelines. arXiv:2604.23178
- Xu, Z., Li, S., Liu, H., et al. (2026). Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias. arXiv:2607.11871
- Dong, Y. R., Hu, T., & Collier, N. (2024). Can LLM be a Personalized Judge? arXiv:2406.11657
- Ali, D., Zhao, D., Koenecke, A., & Papakyriakopoulos, O. (2025). Operationalizing Pluralistic Values in Large Language Model Alignment. AAAI. arXiv:2511.14476
- Russo, G., Nozza, D., Röttger, P., & Hovy, D. (2025). The Pluralistic Moral Gap: Understanding Judgment and Value Differences between Humans and Large Language Models. EACL. arXiv:2507.17216
- Xu, Y., & Jurgens, D. (2026). Beyond Consensus: Perspectivist Modeling and Evaluation of Annotator Disagreement in NLP. arXiv:2601.09065
- Belay, T. D., Ahmad, I., Abdulmumin, I., et al. (2026). Beyond Majority Voting: Agreement-Based Clustering to Model Annotator Perspectives in Subjective NLP Tasks. TACL. arXiv:2605.09955
- Cho, A. (2025). GrandJury: A Living Framework for Multi-Rater Human Judgement of AI. arXiv:2508.02926
Related artifacts
- Summit poster (PDF, ~1 MB) One-page synthesis presented at RDI 2026.
- Scoring engine paper — arXiv:2508.02926 Cho (2025). GrandJury: A Living Framework for Multi-Rater Human Judgement.
- Live AI Reviews arena Ongoing benchmark leaderboard, updated in real time.
- Developer docs — Build with HumanJudge Enroll an AI, wire the SDK, subscribe to signals.
- Discord — community Reviewer and builder coordination channel.
- Summit landing page Report, live arena, developer entry, contact.
Prepared by HumanJudge for the Berkeley RDI Agentic AI Summit 2026
← Back to Summit landing