+30
Point Spread
identical model · different harness
12
Frameworks
7 design philosophies
37%
Lab → Prod Gap
CLEAR framework average
95%
Pilot Failure Rate
never reach production
The 30-Point Spread
GAIA Benchmark · Claude Opus 4 · identical model
Orchestration scaffold contribution · Princeton HAL / Q1 2026
Framework selection is a performance decision masquerading as an infrastructure one.
no harness
44.8%
HAL scaffolding
64.9%
best harness
74.6%
Benchmark Leaderboard
Q1 Lushbinary · multi-step accuracy · cost per task
Framework
Accuracy
Cost/task
Stars
Signal
All 12 Frameworks
400+ companies · 34.5M PyPI/mo · 40–50% LLM call reduction via stateful caching. Steep 1–2 week learning curve. Production ceiling highest of any framework tested. [2]
10M+ agents/month · prototype in 2–4 hrs · AMP Suite $120K/yr. No checkpointing. Control flow encoded in prompts — not inspectable state. [24]
Evolved from Swarm · MCP added v0.7 · session-based checkpoints (2026) · 100+ LLMs. Lock-in risk: 6-month state history tied to OpenAI Stateful Runtime. [20]
Reframed as execution framework Feb 2026 · graph-based engine · GitHub/Jira/MongoDB connectors · Vertex AI one-command deploy · Kotlin support. GCP-native teams only. [1]
From the Gatsby team · 3,300+ models · 94 providers · time-travel debugging · suspend/resume HITL · first-class MCP. TypeScript-only — no Python. [27]
HuggingFace · action primitive is generated Python code · 40 lines vs 120 for equivalent LangGraph agent · leads in local LLM support. Minimal orchestration surface. [1]
Strict types · dependency injection · structured responses. FastAPI ergonomics — immediately familiar to Python web teams. Smaller multi-agent orchestration surface than LangGraph. [1]
Replaces AutoGen + Semantic Kernel · 5 orchestration patterns · 7 LLM providers · MCP first-class · Python + .NET. Inherits 58.7k-star AutoGen community. [18]
Community fork of AutoGen v0.2 · MCP v0.12 · event-driven async. GroupChat: 20+ LLM calls per interaction — token costs balloon fast in group deliberation scenarios. [1]
From Amazon production systems · steering handlers: 100% vs 82.5% prompt-only accuracy · Bedrock/Anthropic/OpenAI/Ollama. Orchestrator is a single point of failure — bad routing propagates. [17]
Data framework for LLM apps · Workflows with checkpointing and human-in-the-loop · RAG-optimized agent design. Best when retrieval is the primary agent action. [14]
C# + Azure-native plugin planner. Superseded by MS Agent Framework GA April 2026. One-year v1.x maintenance commitment. Migrate now. [19]
7 Design Schools
Philosophy determines ceiling, not just style
Control-explicit graph
LangGraph · Google ADK
Write the graph. Highest accuracy ceiling (94%). Rewarded with 40–50% LLM call savings via state checkpoints. 1–2 week ramp.
Role-declarative
CrewAI
Role-based teams; prototype in 2–4 hrs. 3× token overhead vs LangGraph, no checkpointing, control flow lives in prompts — not inspectable code.
Model-driven minimalism
Strands Agents
LLM decides orchestration. Steering handlers close 82.5% → 100% accuracy gap. Orchestrator is a single point of failure with no graph-level guard on bad routing.
Code-first
Smolagents
Action primitive = generated Python code. 40-line agent vs 120 for LangGraph equivalent. Best for local-LLM research workflows. Limited multi-agent orchestration.
Type-safe composition
PydanticAI · Mastra
Strict typed interfaces and DI throughout. FastAPI ergonomics (Python) or Vercel AI SDK patterns (TypeScript). Predictable — type errors surface at compile time, not runtime.
Managed runtime
OpenAI SDK · MS Agent Framework
Platform-native patterns, fastest onboarding. Lock-in risk is structural: 6-month state history cannot migrate without rebuilding the memory layer from scratch.
Event-driven async
AG2
Community fork of AutoGen v0.2, preserving stable API. Top quality score (91%) at highest cost ($0.45/task). GroupChat = 20+ LLM calls per interaction.
Production Risk Signals
⚠ Reality Check
40–60%
Of total agent operating cost is LLM API calls
A 3× token-overhead framework nearly doubles the largest line item · [2]
Research Sub-topics
4 surveys · 84 citations total
survey 01 / 04
The Contenders & Their Philosophies
Seven design philosophies — control-explicit graphs to model-driven minimalism — define the 2026 landscape, with harness choice causing measurable performance variance on identical models.
survey 02 / 04
Head-to-Head Comparison Matrix
15-dimension matrix across 12 frameworks — LangGraph leads on production control, CrewAI on time-to-prototype, Mastra on TypeScript teams; MS Agent Framework replaces AutoGen and Semantic Kernel.
survey 03 / 04
When Each Wins + The Risk
Framework choice alone swings benchmark scores 30 points on identical models — when each of the seven agent harnesses wins, and what breaks under production load.
survey 04 / 04
Demo & Eval Methodology
How to structure a live comparison session — fix the model, tasks, and environment; score at three levels (end-to-end, trajectory, component); separate pass@k from pass^k and utility from security.
Agent Harness Toolkit Showdown 2026 · 84 citations · 4 surveys ·
Lushbinary Q1 Benchmark ·
GAIA 2026 ·
Princeton HAL ·
Anthropic Evals Guide