500 human-validated GitHub issues from real Python repositories. A model has to produce a patch that makes the project's own hidden tests pass.
End-to-end software engineering: reading a codebase, locating the bug, writing a working patch.
Results depend heavily on the agent scaffold around the model, not just the model. The scaffold is shown next to each score.
| # | Model | Lab | Score | Measured | Method | Source |
|---|---|---|---|---|---|---|
| 1 | Claude Opus 4.7 | Anthropic | 83.5% | 2026-04-20 | independent | Epoch AI Benchmarking Hub |
| 2 | GPT-5.5 | OpenAI | 80.6% | 2026-04-24 | independent | Epoch AI Benchmarking Hub |
| 3 | Gemini 3.5 Flash | Google DeepMind | 79.3% | 2026-06-01 | independent | Epoch AI Benchmarking Hub |
| 4 | Claude Opus 4.5 | Anthropic | 79.2% | 2025-12-15 | independent · live-SWE-agent | SWE-bench leaderboard |
| 5 | Doubao-Seed-Code | ByteDance | 78.8% | 2025-09-28 | independent · TRAE | SWE-bench leaderboard |
| 6 | GLM-5.2open | Z.ai | 78.7% | 2026-06-25 | independent | Epoch AI Benchmarking Hub |
| 7 | DeepSeek-V4-Proopen | DeepSeek | 77.6% | 2026-06-18 | independent | Epoch AI Benchmarking Hub |
| 8 | Gemini 3 Pro Preview | Google DeepMind | 77.4% | 2025-11-20 | independent · live-SWE-agent | SWE-bench leaderboard |
| 9 | Qwen3.7-Max | Alibaba | 77.3% | 2026-06-18 | independent | Epoch AI Benchmarking Hub |
| 10 | GPT-5.4 | OpenAI | 76.9% | 2026-03-06 | independent | Epoch AI Benchmarking Hub |
| 11 | Claude Sonnet 4 | Anthropic | 76.8% | 2025-08-04 | independent · EPAM AI/Run Developer Agent | SWE-bench leaderboard |
| 12 | Multiple | Anthropic | 76.8% | 2025-09-02 | independent · Atlassian Rovo Dev | SWE-bench leaderboard |
| 13 | Kimi K2.6open | Moonshot AI | 76.7% | 2026-05-08 | independent | Epoch AI Benchmarking Hub |
| 14 | Qwen 3.6 Max (Preview) | Alibaba | 76.7% | 2026-05-28 | independent | Epoch AI Benchmarking Hub |
| 15 | Gemini 3 Flash | Google DeepMind | 75.8% | 2026-02-17 | independent · mini-SWE-agent | SWE-bench leaderboard |
| 16 | MiniMax M2.5open | MiniMax | 75.8% | 2026-02-17 | independent · mini-SWE-agent | SWE-bench leaderboard |
| 17 | Claude Opus 4.6 | Anthropic | 75.6% | 2026-02-17 | independent · mini-SWE-agent | SWE-bench leaderboard |
| 18 | Gemini 3.1 Pro | Google DeepMind | 75.6% | 2026-02-24 | independent | Epoch AI Benchmarking Hub |
| 19 | Claude Sonnet 4.6 | Anthropic | 75.2% | 2026-02-21 | independent | Epoch AI Benchmarking Hub |
| 20 | Claude Sonnet 4.5 | Anthropic | 74.8% | 2025-11-03 | independent · Sonar Foundation Agent | SWE-bench leaderboard |
| 21 | GPT-5.3 Codex | OpenAI | 74.8% | 2026-02-25 | independent | Epoch AI Benchmarking Hub |
| 22 | GPT-5 | OpenAI | 74.4% | 2025-10-15 | independent · Prometheus-v1.2.1 | SWE-bench leaderboard |
| 23 | GLM-5.1open | Z.ai | 74.2% | 2026-05-15 | independent | Epoch AI Benchmarking Hub |
| 24 | Claude Opus 4.1 | Anthropic | 73.3% | 2026-02-11 | independent | Epoch AI Benchmarking Hub |
| 25 | Claude Opus 4 | Anthropic | 73.2% | 2025-05-22 | independent · Tools | SWE-bench leaderboard |
| 26 | GLM-5open | Z.ai | 72.8% | 2026-02-17 | independent · mini-SWE-agent | SWE-bench leaderboard |
| 27 | GPT 5.2 Codex | OpenAI | 72.8% | 2026-02-19 | independent · mini-SWE-agent | SWE-bench leaderboard |
| 28 | GPT-5.2 | OpenAI | 72.8% | 2026-02-17 | independent · mini-SWE-agent | SWE-bench leaderboard |
| 29 | Kimi K2 | Moonshot AI | 71.2% | 2025-10-14 | independent · Lingxi v1.5 | SWE-bench leaderboard |
| 30 | Kimi K2.5open | Moonshot AI | 70.8% | 2026-02-17 | independent · mini-SWE-agent | SWE-bench leaderboard |
| 31 | Undisclosed | Other | 70.6% | 2025-05-19 | independent · TRAE | SWE-bench leaderboard |
| 32 | DeepSeek-V3.2open | DeepSeek | 70.0% | 2026-02-17 | independent · mini-SWE-agent | SWE-bench leaderboard |
| 33 | Gemini 3 Pro | Google DeepMind | 69.6% | 2026-02-26 | independent · mini-SWE-agent | SWE-bench leaderboard |
| 34 | Qwen3-Coder-480B-A35B-Instruct | Alibaba | 69.6% | 2025-08-05 | independent · OpenHands | SWE-bench leaderboard |
| 35 | GLM-4.6open | Z.ai | 68.2% | 2025-09-30 | independent · Undisclosed | SWE-bench leaderboard |
| 36 | Claude Haiku 4.5 | Anthropic | 66.6% | 2026-02-17 | independent · mini-SWE-agent | SWE-bench leaderboard |
| 37 | Claude 3.7 Sonnet | Anthropic | 66.4% | 2025-05-14 | independent · Aime-coder v1 | SWE-bench leaderboard |
| 38 | GPT 5.1 Codex | OpenAI | 66.0% | 2025-11-24 | independent · mini-SWE-agent | SWE-bench leaderboard |
| 39 | GPT-5.1 | OpenAI | 66.0% | 2025-11-20 | independent · mini-SWE-agent | SWE-bench leaderboard |
| 40 | o1-preview | OpenAI | 64.6% | 2025-01-17 | independent · W&B Programmer O1 crosscheck5 | SWE-bench leaderboard |
| 41 | o4-mini | OpenAI | 64.6% | 2025-05-03 | independent · PatchPilot-v1.1 | SWE-bench leaderboard |
| 42 | GLM-4.5open | Z.ai | 64.2% | 2025-07-28 | independent · Undisclosed | SWE-bench leaderboard |
| 43 | Claude 3.5 Sonnet | Anthropic | 63.4% | 2025-02-06 | independent · AgentScope | SWE-bench leaderboard |
| 44 | Kimi K2 Thinkingopen | Moonshot AI | 63.4% | 2025-12-10 | independent · mini-SWE-agent | SWE-bench leaderboard |
| 45 | Claude 3.7 Sonnet w/ Review Heavy | Anthropic | 62.4% | 2025-02-25 | independent · SWE-agent | SWE-bench leaderboard |
| 46 | swe-search | Anthropic | 62.2% | 2024-12-21 | independent · CodeStory Midwit Agent | SWE-bench leaderboard |
| 47 | MiniMax M2open | MiniMax | 61.0% | 2025-11-24 | independent · mini-SWE-agent | SWE-bench leaderboard |
| 48 | 4x Scaled | Other | 60.8% | 2025-02-03 | independent · OpenHands | SWE-bench leaderboard |
| 49 | Qwen3-Coder-30B-A3B-Instructopen | Alibaba | 60.4% | 2025-09-01 | independent · EntroPO + R2E | SWE-bench leaderboard |
| 50 | DeepSeek V3.2 Reasoner | DeepSeek | 60.0% | 2025-12-01 | independent · mini-SWE-agent | SWE-bench leaderboard |
| 51 | GPT-5 mini | OpenAI | 59.8% | 2025-08-07 | independent · mini-SWE-agent | SWE-bench leaderboard |
| 52 | TTS(Bo16) | OpenAI | 58.8% | 2025-06-29 | independent · DeepSWE-Preview | SWE-bench leaderboard |
| 53 | o3 | OpenAI | 58.4% | 2025-07-26 | independent · mini-SWE-agent | SWE-bench leaderboard |
| 54 | Qwen 3.6 Plus | Alibaba | 57.9% | 2026-05-14 | independent | Epoch AI Benchmarking Hub |
| 55 | Gemini 2.5 Pro (Jun 2025) | Google DeepMind | 57.6% | 2026-02-13 | independent | Epoch AI Benchmarking Hub |
| 56 | Devstral Small (2512) | Mistral | 56.4% | 2025-12-09 | independent · mini-SWE-agent | SWE-bench leaderboard |
| 57 | Devstral (2512) | Mistral | 53.8% | 2025-12-09 | independent · mini-SWE-agent | SWE-bench leaderboard |
| 58 | Frogboss 32B 2510 | Other | 53.6% | 2025-11-10 | independent · FrogBoss-32B-2510 | SWE-bench leaderboard |
| 59 | Gemini 2.5 Pro | Google DeepMind | 53.6% | 2025-07-26 | independent · mini-SWE-agent | SWE-bench leaderboard |
| 60 | Kimi K2 Instruct | Moonshot AI | 53.4% | 2025-08-04 | independent · CodeSweep - SWE-agent | SWE-bench leaderboard |
| 61 | CodeAct v2.1 (claude-3-5-sonnet-20241022) | Anthropic | 53.0% | 2024-10-29 | independent · OpenHands | SWE-bench leaderboard |
| 62 | Gemini 2.0 Flash (v20241212-experimental) | Google DeepMind | 52.2% | 2024-12-12 | independent · Google Jules | SWE-bench leaderboard |
| 63 | Claude 3.5-Sonnet-20241022 | Anthropic | 51.6% | 2025-01-22 | independent · AutoCodeRover-v2.1 | SWE-bench leaderboard |
| 64 | TTS(Bo8) | Alibaba | 47.0% | 2025-06-16 | independent · Skywork-SWE-32B | SWE-bench leaderboard |
| 65 | DevStral Small 2505 | Mistral | 46.8% | 2025-05-20 | independent · OpenHands | SWE-bench leaderboard |
| 66 | Co-PatcheR | Other | 46.0% | 2025-05-28 | independent · PatchPilot | SWE-bench leaderboard |
| 67 | Frogmini 14B 2510 | Other | 45.0% | 2025-11-10 | independent · FrogMini-14B-2510 | SWE-bench leaderboard |
| 68 | Gemini 2.0 Flash (Experimental) | Google DeepMind | 44.2% | 2025-01-18 | independent · CodeShellAgent | SWE-bench leaderboard |
| 69 | Amazon.nova Premier v1:0 | Amazon | 42.4% | 2025-05-27 | independent · Amazon Nova Premier 1.0 | SWE-bench leaderboard |
| 70 | o3-mini | OpenAI | 42.4% | 2025-02-14 | independent · Agentless Lite | SWE-bench leaderboard |
| 71 | DeepSWE-Preview | Other | 42.2% | 2025-06-29 | independent · R2E-Gym | SWE-bench leaderboard |
| 72 | DeepSeek V3 0324open | DeepSeek | 42.0% | 2025-08-06 | independent · SWE-Exp | SWE-bench leaderboard |
| 73 | Llama3-SWE-RL-70B | Meta AI | 41.2% | 2025-02-26 | independent · Agentless Mini | SWE-bench leaderboard |
| 74 | Claude 3.5 Haiku | Anthropic | 40.6% | 2024-10-22 | independent · Tools | SWE-bench leaderboard |
| 75 | SWE-agent-LM-32B | Alibaba | 40.2% | 2025-05-11 | independent · SWE-agent | SWE-bench leaderboard |
| 76 | GPT-4.1 | OpenAI | 39.6% | 2025-07-26 | independent · mini-SWE-agent | SWE-bench leaderboard |
| 77 | GPT-4o | OpenAI | 38.8% | 2024-10-28 | independent · Agentless-1.5 | SWE-bench leaderboard |
| 78 | DevStral Small 2507 | Mistral | 38.0% | 2025-07-25 | independent · SWE-agent | SWE-bench leaderboard |
| 79 | Qwen2.5 Coder 32B Instructopen | Alibaba | 38.0% | 2025-06-16 | independent · Skywork-SWE-32B | SWE-bench leaderboard |
| 80 | GPT-5 nano | OpenAI | 34.8% | 2025-08-07 | independent · mini-SWE-agent | SWE-bench leaderboard |
| 81 | Qwen2.5 (7B + 72B) | Alibaba | 32.8% | 2025-03-06 | independent · SWE-Fixer | SWE-bench leaderboard |
| 82 | Lingma SWE-GPT 72b (v0925) | OpenAI | 28.8% | 2024-10-02 | independent · Lingma Agent | SWE-bench leaderboard |
| 83 | Gemini 2.5 Flash | Google DeepMind | 28.7% | 2025-07-26 | independent · mini-SWE-agent | SWE-bench leaderboard |
| 84 | gpt-oss-120bopen | OpenAI | 26.0% | 2025-08-07 | independent · mini-SWE-agent | SWE-bench leaderboard |
| 85 | Lingma SWE-GPT 72b (v0918) | OpenAI | 25.0% | 2024-09-18 | independent · Lingma Agent | SWE-bench leaderboard |
| 86 | GPT-4.1 mini | OpenAI | 23.9% | 2025-07-20 | independent · mini-SWE-agent | SWE-bench leaderboard |
| 87 | MCTS Refine 7B | Other | 23.2% | 2025-06-27 | independent · MCTS-Refine-7B | SWE-bench leaderboard |
| 88 | GPT-4 (1106) | OpenAI | 22.4% | 2024-04-02 | independent · SWE-agent | SWE-bench leaderboard |
| 89 | Llama 4 Maverick Instruct | Meta AI | 21.0% | 2025-07-20 | independent · mini-SWE-agent | SWE-bench leaderboard |
| 90 | Lingma SWE-GPT 7b (v0925) | OpenAI | 18.2% | 2024-10-02 | independent · Lingma Agent | SWE-bench leaderboard |
| 91 | Claude 3 Opus | Anthropic | 15.8% | 2024-04-02 | independent · SWE-agent | SWE-bench leaderboard |
| 92 | Gemini 2.0 Flash | Google DeepMind | 13.5% | 2025-07-26 | independent · mini-SWE-agent | SWE-bench leaderboard |
| 93 | Lingma SWE-GPT 7b (v0918) | OpenAI | 10.2% | 2024-09-18 | independent · Lingma Agent | SWE-bench leaderboard |
| 94 | Llama 4 Scout Instruct | Meta AI | 9.1% | 2025-07-20 | independent · mini-SWE-agent | SWE-bench leaderboard |
| 95 | Claude 2 | Anthropic | 4.4% | 2023-10-10 | independent · RAG baseline | SWE-bench leaderboard |
| 96 | SWE-Llama 7B | Meta AI | 1.4% | 2023-10-10 | independent · RAG baseline | SWE-bench leaderboard |
| 97 | SWE-Llama 13B | Meta AI | 1.2% | 2023-10-10 | independent · RAG baseline | SWE-bench leaderboard |
| 98 | GPT-3.5 | OpenAI | 0.4% | 2023-10-10 | independent · RAG baseline | SWE-bench leaderboard |