Research / evaluation
Merbench
Can a language model write a diagram, check its work, and fix its mistakes?
Merbench evaluates how well language models write and debug Mermaid diagrams using an MCP validation tool. Models have up to five attempts per test case across three difficulty levels. Success reflects both valid diagram syntax and effective use of feedback.
Costs are the recorded US dollar values from these historical runs, not current provider prices. A recorded zero does not establish that a model is free to run. Missing costs are shown as N/A.
- Evaluation runs
- 2,057
- Models evaluated
- 46
- Difficulty levels
- 3
Providers in this snapshot: Amazon, Anthropic, Google, OSS, OpenAI.
Evaluation source ↗How to read the results
- Success rate
- The percentage of runs that produced a valid Mermaid diagram.
- Cost / run
- Mean recorded cost of a complete run, including attempts and tool feedback.
- Cost / success
- Mean cost divided by success rate. Unavailable when there are no successful runs or no recorded cost.
- Duration
- Mean elapsed time per run in seconds.
- Tokens
- Mean total token usage per run.
- Runs
- The number of evaluation runs included after filtering.
Model leaderboard
Select a column heading to sort. Scroll the table horizontally to see all metrics.
| Rank | ||||||||
|---|---|---|---|---|---|---|---|---|
| 1 | gpt-5.2-codex (medium) | 100.0% | $0.1419 | $0.1419 | 54.06s | 64,101 | 45 | OpenAI |
| 2 | gpt-5.2 (medium) | 100.0% | $0.1653 | $0.1653 | 43.19s | 69,768 | 45 | OpenAI |
| 3 | claude-opus-4.5 | 100.0% | $0.5252 | $0.5252 | 58.24s | 93,280 | 45 | Anthropic |
| 4 | gpt-5.1-codex-max (medium) | 97.8% | $0.1247 | $0.1275 | 54.27s | 78,085 | 45 | OpenAI |
| 5 | gemini-3-pro-preview | 97.8% | $0.2480 | $0.2536 | 79.49s | 80,741 | 45 | |
| 6 | claude-sonnet-4.5 | 95.6% | $0.2596 | $0.2717 | 58.42s | 75,860 | 45 | Anthropic |
| 7 | gpt-5 (medium) | 93.3% | $0.1381 | $0.1480 | 76.25s | 74,458 | 45 | OpenAI |
| 8 | gpt-5.1-codex (medium) | 93.3% | $0.1399 | $0.1499 | 63.58s | 89,230 | 45 | OpenAI |
| 9 | o3 | 93.3% | $0.1502 | $0.1609 | 118.85s | 64,968 | 45 | OpenAI |
| 10 | gpt-5.1 (medium) | 91.1% | $0.1061 | $0.1164 | 53.77s | 61,876 | 45 | OpenAI |
| 11 | gpt-5.2 (none) | 86.7% | $0.1305 | $0.1506 | 21.41s | 57,198 | 45 | OpenAI |
| 12 | gpt-5-codex | 80.0% | $0.0811 | $0.1013 | 59.10s | 47,452 | 45 | OpenAI |
| 13 | gemini-3-flash-preview | 77.8% | $0.0488 | $0.0628 | 40.33s | 69,554 | 45 | |
| 14 | o4-mini | 62.2% | $0.0833 | $0.1338 | 60.44s | 59,667 | 45 | OpenAI |
| 15 | gpt-5.1 (none) | 55.6% | $0.0343 | $0.0617 | 33.31s | 17,784 | 45 | OpenAI |
| 16 | gemini-2.5-flash-preview-09-2025 | 45.7% | $0.0332 | $0.0728 | 46.23s | 46,035 | 46 | |
| 17 | gpt-4.1 | 42.2% | $0.0147 | $0.0348 | 16.41s | 4,300 | 45 | OpenAI |
| 18 | gpt-5-mini | 35.6% | $0.0079 | $0.0223 | 51.49s | 8,576 | 45 | OpenAI |
| 19 | gemini-2.5-pro | 35.6% | $0.1048 | $0.2948 | 52.74s | 31,300 | 45 | |
| 20 | gpt-5.1-codex-mini (medium) | 33.3% | $0.0119 | $0.0356 | 31.13s | 32,398 | 45 | OpenAI |
| 21 | gemini-2.5-pro-preview-06-05 | 27.1% | $0.0387 | $0.1427 | 36.92s | 8,246 | 48 | |
| 22 | gemini-2.5-pro-preview-05-06 | 26.7% | $0.1308 | $0.4904 | 49.85s | 19,754 | 45 | |
| 23 | gemini-2.5-pro-preview-03-25 | 22.9% | $0.1133 | $0.4942 | 57.17s | 16,394 | 48 | |
| 24 | gpt-4.1-mini | 20.0% | $0.0028 | $0.0139 | 23.43s | 3,733 | 45 | OpenAI |
| 25 | gpt-5-nano | 13.3% | $0.0037 | $0.0275 | 88.50s | 11,963 | 45 | OpenAI |
| 26 | gemini-2.5-flash | 13.3% | $0.0118 | $0.0887 | 10.15s | 6,991 | 90 | |
| 27 | qwen3-30b-a3b-thinking-2507-mlx | 10.3% | $0.0017 | $0.0168 | 92.27s | 8,167 | 39 | OSS |
| 28 | seed-oss-36b-instruct-mlx | 6.3% | $0.0009 | $0.0150 | 396.96s | 3,054 | 16 | OSS |
| 29 | gemini-2.5-flash-lite-preview-06-17 | 4.4% | $0.0008 | $0.0189 | 4.40s | 5,234 | 45 | |
| 30 | gemini-2.5-flash-preview-05-20 | 4.4% | $0.0080 | $0.1789 | 9.26s | 5,120 | 45 | |
| 31 | gemini-2.5-flash-preview-04-17 | 4.4% | $0.0233 | $0.5237 | 24.15s | 10,493 | 45 | |
| 32 | gemini-2.5-flash-lite | 3.3% | $0.0013 | $0.0382 | 5.90s | 9,507 | 90 | |
| 33 | gpt-oss-20b | 2.2% | $0.0002 | $0.0111 | 47.90s | 3,897 | 45 | OSS |
| 34 | nova-premier-v1:0 | 2.2% | $0.0287 | $1.2916 | 58.14s | 7,520 | 45 | Amazon |
| 35 | nova-micro-v1:0 | 0.0% | $0.0001 | N/A | 19.31s | 1,798 | 45 | Amazon |
| 36 | nova-lite-v1:0 | 0.0% | $0.0003 | N/A | 24.21s | 3,091 | 45 | Amazon |
| 37 | gemini-2.0-flash | 0.0% | $0.0003 | N/A | 4.21s | 1,326 | 60 | |
| 38 | xlam-2-32b-fc-r | 0.0% | $0.0004 | N/A | 171.86s | 8,211 | 15 | OSS |
| 39 | qwen3-coder-30b-a3b-instruct-mlx | 0.0% | $0.0005 | N/A | 21.61s | 4,384 | 45 | OSS |
| 40 | qwen3-30b-a3b-2507 | 0.0% | $0.0006 | N/A | 23.04s | 4,278 | 45 | OSS |
| 41 | gpt-4.1-nano | 0.0% | $0.0007 | N/A | 18.99s | 3,920 | 45 | OpenAI |
| 42 | gemma-3-27b | 0.0% | $0.0007 | N/A | 120.44s | 6,955 | 45 | OSS |
| 43 | gemini-2.5-flash-lite-preview-09-2025 | 0.0% | $0.0011 | N/A | 5.68s | 5,688 | 45 | |
| 44 | nova-pro-v1:0 | 0.0% | $0.0011 | N/A | 49.14s | 905 | 45 | Amazon |
| 45 | llama-xlam-2-70b-fc-r | 0.0% | $0.0027 | N/A | 238.06s | 8,592 | 15 | OSS |
| 46 | magistral-small-2509-mlx | 0.0% | $0.0042 | N/A | 582.19s | 4,439 | 15 | OSS |
Performance & efficiency
Each point represents a model. The leaderboard lists success rate, cost, duration, and total tokens for the selected runs.
Chart loads as you approach. Table alternatives remain available.
Performance by difficulty
Compare model success rates across the selected difficulty levels. Exact percentages are available in the data table.
Chart loads as you approach. Table alternatives remain available.
View performance by difficulty data table
| Model | Easy success rate | Hard success rate | Medium success rate |
|---|---|---|---|
| gemini-2.0-flash | 0% | 0% | 0% |
| gemini-2.5-flash | 40% | 0% | 0% |
| gemini-2.5-flash-lite | 10% | 0% | 0% |
| gemini-2.5-flash-lite-preview-06-17 | 13.3% | 0% | 0% |
| gemini-2.5-flash-lite-preview-09-2025 | 0% | 0% | 0% |
| gemini-2.5-flash-preview-04-17 | 13.3% | 0% | 0% |
| gemini-2.5-flash-preview-05-20 | 13.3% | 0% | 0% |
| gemini-2.5-flash-preview-09-2025 | 66.7% | 6.3% | 66.7% |
| gemini-2.5-pro | 66.7% | 6.7% | 33.3% |
| gemini-2.5-pro-preview-03-25 | 68.8% | 0% | 0% |
| gemini-2.5-pro-preview-05-06 | 73.3% | 0% | 6.7% |
| gemini-2.5-pro-preview-06-05 | 68.8% | 0% | 12.5% |
| gemini-3-flash-preview | 100% | 33.3% | 100% |
| gemini-3-pro-preview | 100% | 93.3% | 100% |
| gemma-3-27b | 0% | 0% | 0% |
| gpt-4.1 | 100% | 0% | 26.7% |
| gpt-4.1-mini | 53.3% | 0% | 6.7% |
| gpt-4.1-nano | 0% | 0% | 0% |
| gpt-5 (medium) | 93.3% | 86.7% | 100% |
| gpt-5-codex | 93.3% | 46.7% | 100% |
| gpt-5-mini | 53.3% | 6.7% | 46.7% |
| gpt-5-nano | 40% | 0% | 0% |
| gpt-5.1 (medium) | 86.7% | 86.7% | 100% |
| gpt-5.1 (none) | 100% | 6.7% | 60% |
| gpt-5.1-codex (medium) | 100% | 80% | 100% |
| gpt-5.1-codex-max (medium) | 100% | 93.3% | 100% |
| gpt-5.1-codex-mini (medium) | 26.7% | 33.3% | 40% |
| gpt-5.2 (medium) | 100% | 100% | 100% |
| gpt-5.2 (none) | 100% | 60% | 100% |
| gpt-oss-20b | 6.7% | 0% | 0% |
| llama-xlam-2-70b-fc-r | 0% | 0% | 0% |
| magistral-small-2509-mlx | 0% | 0% | 0% |
| nova-lite-v1:0 | 0% | 0% | 0% |
| nova-micro-v1:0 | 0% | 0% | 0% |
| nova-premier-v1:0 | 6.7% | 0% | 0% |
| nova-pro-v1:0 | 0% | 0% | 0% |
| o3 | 100% | 93.3% | 86.7% |
| o4-mini | 86.7% | 33.3% | 66.7% |
| gpt-5.2-codex (medium) | 100% | 100% | 100% |
| claude-opus-4.5 | 100% | 100% | 100% |
| claude-sonnet-4.5 | 86.7% | 100% | 100% |
| qwen3-30b-a3b-2507 | 0% | 0% | 0% |
| qwen3-30b-a3b-thinking-2507-mlx | 23.1% | 0% | 7.7% |
| qwen3-coder-30b-a3b-instruct-mlx | 0% | 0% | 0% |
| seed-oss-36b-instruct-mlx | 6.7% | 0% | 0% |
| xlam-2-32b-fc-r | 0% | 0% | 0% |
Token usage
Average request and response tokens per model, shown separately in the data table.
Chart loads as you approach. Table alternatives remain available.
View token usage data table
| Model | Average request tokens | Average response tokens |
|---|---|---|
| claude-opus-4.5 | 90,339.5 | 2,939.6 |
| gpt-5.1-codex (medium) | 85,990.7 | 3,239.2 |
| gpt-5.1-codex-max (medium) | 74,987.2 | 3,097.7 |
| gemini-3-pro-preview | 72,089.4 | 5,372.6 |
| claude-sonnet-4.5 | 73,187.6 | 2,671.7 |
| gpt-5 (medium) | 69,307.6 | 5,150.2 |
| gpt-5.2 (medium) | 66,242.6 | 3,525.3 |
| gemini-3-flash-preview | 63,931.4 | 3,746.2 |
| o3 | 61,596.1 | 3,371.3 |
| gpt-5.2-codex (medium) | 61,671 | 2,429.2 |
| gpt-5.1 (medium) | 58,592.3 | 3,283.2 |
| o4-mini | 54,318.9 | 5,348.1 |
| gpt-5.2 (none) | 54,716.2 | 2,480.9 |
| gpt-5-codex | 44,965 | 2,487 |
| gemini-2.5-flash-preview-09-2025 | 34,330.6 | 7,158 |
| gpt-5.1-codex-mini (medium) | 30,254.3 | 2,143.3 |
| gemini-2.5-pro | 23,790.4 | 4,732.8 |
| gpt-5.1 (none) | 16,403 | 1,380 |
| gpt-5-nano | 3,208.6 | 8,754 |
| gemini-2.5-flash-lite | 8,427.3 | 1,079.4 |
| gemini-2.5-pro-preview-05-06 | 7,631.8 | 1,165.7 |
| llama-xlam-2-70b-fc-r | 7,792.8 | 798.5 |
| gpt-5-mini | 5,279.2 | 3,296.7 |
| xlam-2-32b-fc-r | 7,465.5 | 744.7 |
| qwen3-30b-a3b-thinking-2507-mlx | 3,086.7 | 5,080.1 |
| nova-premier-v1:0 | 6,529.6 | 990.3 |
| gemma-3-27b | 5,798.4 | 1,156.1 |
| gemini-2.5-pro-preview-03-25 | 5,791.1 | 1,079 |
| gemini-2.5-pro-preview-06-05 | 5,005.8 | 1,314.6 |
| gemini-2.5-flash-lite-preview-09-2025 | 4,065 | 1,622.9 |
| gemini-2.5-flash-lite-preview-06-17 | 4,179.7 | 1,053.7 |
| magistral-small-2509-mlx | 2,497.2 | 1,941.1 |
| qwen3-coder-30b-a3b-instruct-mlx | 3,351.9 | 1,031.5 |
| gpt-4.1 | 3,286.8 | 1,012.8 |
| qwen3-30b-a3b-2507 | 3,242.4 | 1,035.5 |
| gemini-2.5-flash-preview-04-17 | 3,260.2 | 871.5 |
| gpt-4.1-nano | 2,884 | 1,035.8 |
| gpt-oss-20b | 2,716.6 | 1,179.4 |
| gpt-4.1-mini | 2,659 | 1,073.7 |
| gemini-2.5-flash | 2,796.3 | 807.5 |
| nova-lite-v1:0 | 2,601.6 | 488.6 |
| gemini-2.5-flash-preview-05-20 | 2,374.4 | 694 |
| seed-oss-36b-instruct-mlx | 2,231.8 | 821.6 |
| nova-micro-v1:0 | 1,564 | 233 |
| gemini-2.0-flash | 802.2 | 523.4 |
| nova-pro-v1:0 | 736.6 | 167.6 |
Failure analysis
Recorded failure counts by model and reason. Reasons use the archive's priority: usage limit, then MCP tool failure, then invalid diagram.
Chart loads as you approach. Table alternatives remain available.
View failure analysis data table
| Model | Invalid Diagram | MCP Tool Failure | Usage Limit Exceeded |
|---|---|---|---|
| gemini-2.5-pro-preview-06-05 | 35 | 0 | 0 |
| gemini-2.5-pro-preview-05-06 | 24 | 5 | 4 |
| gemini-2.5-pro-preview-03-25 | 27 | 1 | 9 |
| gemini-2.0-flash | 2 | 58 | 0 |
| gemini-2.5-flash-preview-04-17 | 21 | 21 | 1 |
| gemini-2.5-flash | 20 | 58 | 0 |
| gemini-2.5-flash-lite-preview-06-17 | 21 | 22 | 0 |
| gemini-2.5-flash-preview-05-20 | 8 | 35 | 0 |
| nova-premier-v1:0 | 16 | 18 | 10 |
| nova-pro-v1:0 | 4 | 1 | 40 |
| nova-lite-v1:0 | 10 | 12 | 23 |
| nova-micro-v1:0 | 8 | 2 | 35 |
| gemini-2.5-flash-lite | 60 | 27 | 0 |
| gpt-5-codex | 1 | 0 | 8 |
| gpt-oss-20b | 7 | 25 | 12 |
| qwen3-coder-30b-a3b-instruct-mlx | 45 | 0 | 0 |
| qwen3-30b-a3b-thinking-2507-mlx | 6 | 29 | 0 |
| qwen3-30b-a3b-2507 | 45 | 0 | 0 |
| gemma-3-27b | 45 | 0 | 0 |
| magistral-small-2509-mlx | 7 | 6 | 2 |
| seed-oss-36b-instruct-mlx | 3 | 4 | 8 |
| xlam-2-32b-fc-r | 13 | 0 | 2 |
| llama-xlam-2-70b-fc-r | 13 | 2 | 0 |
| gpt-4.1 | 26 | 0 | 0 |
| gpt-4.1-mini | 36 | 0 | 0 |
| gpt-4.1-nano | 45 | 0 | 0 |
| gemini-3-pro-preview | 1 | 0 | 0 |
| gpt-5.2 (none) | 6 | 0 | 0 |
| gpt-5.2 (medium) | 0 | 0 | 0 |
| o4-mini | 9 | 8 | 0 |
| gpt-5 (medium) | 3 | 0 | 0 |
| gemini-2.5-flash-preview-09-2025 | 25 | 0 | 0 |
| gemini-2.5-flash-lite-preview-09-2025 | 45 | 0 | 0 |
| gpt-5-mini | 14 | 15 | 0 |
| gpt-5-nano | 31 | 7 | 1 |
| gemini-2.5-pro | 29 | 0 | 0 |
| o3 | 2 | 1 | 0 |
| gpt-5.1-codex-max (medium) | 1 | 0 | 0 |
| gpt-5.1-codex (medium) | 1 | 2 | 0 |
| gpt-5.1-codex-mini (medium) | 0 | 30 | 0 |
| gpt-5.1 (none) | 20 | 0 | 0 |
| gpt-5.1 (medium) | 4 | 0 | 0 |
| gemini-3-flash-preview | 10 | 0 | 0 |
| claude-opus-4.5 | 0 | 0 | 0 |
| claude-sonnet-4.5 | 2 | 0 | 0 |
| gpt-5.2-codex (medium) | 0 | 0 | 0 |