Research / evaluation

Merbench

Can a language model write a diagram, check its work, and fix its mistakes?

Merbench evaluates how well language models write and debug Mermaid diagrams using an MCP validation tool. Models have up to five attempts per test case across three difficulty levels. Success reflects both valid diagram syntax and effective use of feedback.

Costs are the recorded US dollar values from these historical runs, not current provider prices. A recorded zero does not establish that a model is free to run. Missing costs are shown as N/A.

Evaluation runs
2,057
Models evaluated
46
Difficulty levels
3

Providers in this snapshot: Amazon, Anthropic, Google, OSS, OpenAI.

Evaluation source ↗
Difficulty
Provider

Showing all recorded runs.

How to read the results
Success rate
The percentage of runs that produced a valid Mermaid diagram.
Cost / run
Mean recorded cost of a complete run, including attempts and tool feedback.
Cost / success
Mean cost divided by success rate. Unavailable when there are no successful runs or no recorded cost.
Duration
Mean elapsed time per run in seconds.
Tokens
Mean total token usage per run.
Runs
The number of evaluation runs included after filtering.

Model leaderboard

Select a column heading to sort. Scroll the table horizontally to see all metrics.

Historical Merbench results. Costs are recorded US dollars per run.
Rank
1gpt-5.2-codex (medium)100.0%$0.1419$0.141954.06s64,10145OpenAI
2gpt-5.2 (medium)100.0%$0.1653$0.165343.19s69,76845OpenAI
3claude-opus-4.5100.0%$0.5252$0.525258.24s93,28045Anthropic
4gpt-5.1-codex-max (medium)97.8%$0.1247$0.127554.27s78,08545OpenAI
5gemini-3-pro-preview97.8%$0.2480$0.253679.49s80,74145Google
6claude-sonnet-4.595.6%$0.2596$0.271758.42s75,86045Anthropic
7gpt-5 (medium)93.3%$0.1381$0.148076.25s74,45845OpenAI
8gpt-5.1-codex (medium)93.3%$0.1399$0.149963.58s89,23045OpenAI
9o393.3%$0.1502$0.1609118.85s64,96845OpenAI
10gpt-5.1 (medium)91.1%$0.1061$0.116453.77s61,87645OpenAI
11gpt-5.2 (none)86.7%$0.1305$0.150621.41s57,19845OpenAI
12gpt-5-codex80.0%$0.0811$0.101359.10s47,45245OpenAI
13gemini-3-flash-preview77.8%$0.0488$0.062840.33s69,55445Google
14o4-mini62.2%$0.0833$0.133860.44s59,66745OpenAI
15gpt-5.1 (none)55.6%$0.0343$0.061733.31s17,78445OpenAI
16gemini-2.5-flash-preview-09-202545.7%$0.0332$0.072846.23s46,03546Google
17gpt-4.142.2%$0.0147$0.034816.41s4,30045OpenAI
18gpt-5-mini35.6%$0.0079$0.022351.49s8,57645OpenAI
19gemini-2.5-pro35.6%$0.1048$0.294852.74s31,30045Google
20gpt-5.1-codex-mini (medium)33.3%$0.0119$0.035631.13s32,39845OpenAI
21gemini-2.5-pro-preview-06-0527.1%$0.0387$0.142736.92s8,24648Google
22gemini-2.5-pro-preview-05-0626.7%$0.1308$0.490449.85s19,75445Google
23gemini-2.5-pro-preview-03-2522.9%$0.1133$0.494257.17s16,39448Google
24gpt-4.1-mini20.0%$0.0028$0.013923.43s3,73345OpenAI
25gpt-5-nano13.3%$0.0037$0.027588.50s11,96345OpenAI
26gemini-2.5-flash13.3%$0.0118$0.088710.15s6,99190Google
27qwen3-30b-a3b-thinking-2507-mlx10.3%$0.0017$0.016892.27s8,16739OSS
28seed-oss-36b-instruct-mlx6.3%$0.0009$0.0150396.96s3,05416OSS
29gemini-2.5-flash-lite-preview-06-174.4%$0.0008$0.01894.40s5,23445Google
30gemini-2.5-flash-preview-05-204.4%$0.0080$0.17899.26s5,12045Google
31gemini-2.5-flash-preview-04-174.4%$0.0233$0.523724.15s10,49345Google
32gemini-2.5-flash-lite3.3%$0.0013$0.03825.90s9,50790Google
33gpt-oss-20b2.2%$0.0002$0.011147.90s3,89745OSS
34nova-premier-v1:02.2%$0.0287$1.291658.14s7,52045Amazon
35nova-micro-v1:00.0%$0.0001N/A19.31s1,79845Amazon
36nova-lite-v1:00.0%$0.0003N/A24.21s3,09145Amazon
37gemini-2.0-flash0.0%$0.0003N/A4.21s1,32660Google
38xlam-2-32b-fc-r0.0%$0.0004N/A171.86s8,21115OSS
39qwen3-coder-30b-a3b-instruct-mlx0.0%$0.0005N/A21.61s4,38445OSS
40qwen3-30b-a3b-25070.0%$0.0006N/A23.04s4,27845OSS
41gpt-4.1-nano0.0%$0.0007N/A18.99s3,92045OpenAI
42gemma-3-27b0.0%$0.0007N/A120.44s6,95545OSS
43gemini-2.5-flash-lite-preview-09-20250.0%$0.0011N/A5.68s5,68845Google
44nova-pro-v1:00.0%$0.0011N/A49.14s90545Amazon
45llama-xlam-2-70b-fc-r0.0%$0.0027N/A238.06s8,59215OSS
46magistral-small-2509-mlx0.0%$0.0042N/A582.19s4,43915OSS

Performance & efficiency

Each point represents a model. The leaderboard lists success rate, cost, duration, and total tokens for the selected runs.

Chart loads as you approach. Table alternatives remain available.

Performance by difficulty

Compare model success rates across the selected difficulty levels. Exact percentages are available in the data table.

Chart loads as you approach. Table alternatives remain available.

View performance by difficulty data table
Performance by difficulty — selected historical runs
ModelEasy success rateHard success rateMedium success rate
gemini-2.0-flash0%0%0%
gemini-2.5-flash40%0%0%
gemini-2.5-flash-lite10%0%0%
gemini-2.5-flash-lite-preview-06-1713.3%0%0%
gemini-2.5-flash-lite-preview-09-20250%0%0%
gemini-2.5-flash-preview-04-1713.3%0%0%
gemini-2.5-flash-preview-05-2013.3%0%0%
gemini-2.5-flash-preview-09-202566.7%6.3%66.7%
gemini-2.5-pro66.7%6.7%33.3%
gemini-2.5-pro-preview-03-2568.8%0%0%
gemini-2.5-pro-preview-05-0673.3%0%6.7%
gemini-2.5-pro-preview-06-0568.8%0%12.5%
gemini-3-flash-preview100%33.3%100%
gemini-3-pro-preview100%93.3%100%
gemma-3-27b0%0%0%
gpt-4.1100%0%26.7%
gpt-4.1-mini53.3%0%6.7%
gpt-4.1-nano0%0%0%
gpt-5 (medium)93.3%86.7%100%
gpt-5-codex93.3%46.7%100%
gpt-5-mini53.3%6.7%46.7%
gpt-5-nano40%0%0%
gpt-5.1 (medium)86.7%86.7%100%
gpt-5.1 (none)100%6.7%60%
gpt-5.1-codex (medium)100%80%100%
gpt-5.1-codex-max (medium)100%93.3%100%
gpt-5.1-codex-mini (medium)26.7%33.3%40%
gpt-5.2 (medium)100%100%100%
gpt-5.2 (none)100%60%100%
gpt-oss-20b6.7%0%0%
llama-xlam-2-70b-fc-r0%0%0%
magistral-small-2509-mlx0%0%0%
nova-lite-v1:00%0%0%
nova-micro-v1:00%0%0%
nova-premier-v1:06.7%0%0%
nova-pro-v1:00%0%0%
o3100%93.3%86.7%
o4-mini86.7%33.3%66.7%
gpt-5.2-codex (medium)100%100%100%
claude-opus-4.5100%100%100%
claude-sonnet-4.586.7%100%100%
qwen3-30b-a3b-25070%0%0%
qwen3-30b-a3b-thinking-2507-mlx23.1%0%7.7%
qwen3-coder-30b-a3b-instruct-mlx0%0%0%
seed-oss-36b-instruct-mlx6.7%0%0%
xlam-2-32b-fc-r0%0%0%

Token usage

Average request and response tokens per model, shown separately in the data table.

Chart loads as you approach. Table alternatives remain available.

View token usage data table
Token usage — selected historical runs
ModelAverage request tokensAverage response tokens
claude-opus-4.590,339.52,939.6
gpt-5.1-codex (medium)85,990.73,239.2
gpt-5.1-codex-max (medium)74,987.23,097.7
gemini-3-pro-preview72,089.45,372.6
claude-sonnet-4.573,187.62,671.7
gpt-5 (medium)69,307.65,150.2
gpt-5.2 (medium)66,242.63,525.3
gemini-3-flash-preview63,931.43,746.2
o361,596.13,371.3
gpt-5.2-codex (medium)61,6712,429.2
gpt-5.1 (medium)58,592.33,283.2
o4-mini54,318.95,348.1
gpt-5.2 (none)54,716.22,480.9
gpt-5-codex44,9652,487
gemini-2.5-flash-preview-09-202534,330.67,158
gpt-5.1-codex-mini (medium)30,254.32,143.3
gemini-2.5-pro23,790.44,732.8
gpt-5.1 (none)16,4031,380
gpt-5-nano3,208.68,754
gemini-2.5-flash-lite8,427.31,079.4
gemini-2.5-pro-preview-05-067,631.81,165.7
llama-xlam-2-70b-fc-r7,792.8798.5
gpt-5-mini5,279.23,296.7
xlam-2-32b-fc-r7,465.5744.7
qwen3-30b-a3b-thinking-2507-mlx3,086.75,080.1
nova-premier-v1:06,529.6990.3
gemma-3-27b5,798.41,156.1
gemini-2.5-pro-preview-03-255,791.11,079
gemini-2.5-pro-preview-06-055,005.81,314.6
gemini-2.5-flash-lite-preview-09-20254,0651,622.9
gemini-2.5-flash-lite-preview-06-174,179.71,053.7
magistral-small-2509-mlx2,497.21,941.1
qwen3-coder-30b-a3b-instruct-mlx3,351.91,031.5
gpt-4.13,286.81,012.8
qwen3-30b-a3b-25073,242.41,035.5
gemini-2.5-flash-preview-04-173,260.2871.5
gpt-4.1-nano2,8841,035.8
gpt-oss-20b2,716.61,179.4
gpt-4.1-mini2,6591,073.7
gemini-2.5-flash2,796.3807.5
nova-lite-v1:02,601.6488.6
gemini-2.5-flash-preview-05-202,374.4694
seed-oss-36b-instruct-mlx2,231.8821.6
nova-micro-v1:01,564233
gemini-2.0-flash802.2523.4
nova-pro-v1:0736.6167.6

Failure analysis

Recorded failure counts by model and reason. Reasons use the archive's priority: usage limit, then MCP tool failure, then invalid diagram.

Chart loads as you approach. Table alternatives remain available.

View failure analysis data table
Failure analysis — selected historical runs
ModelInvalid DiagramMCP Tool FailureUsage Limit Exceeded
gemini-2.5-pro-preview-06-053500
gemini-2.5-pro-preview-05-062454
gemini-2.5-pro-preview-03-252719
gemini-2.0-flash2580
gemini-2.5-flash-preview-04-1721211
gemini-2.5-flash20580
gemini-2.5-flash-lite-preview-06-1721220
gemini-2.5-flash-preview-05-208350
nova-premier-v1:0161810
nova-pro-v1:04140
nova-lite-v1:0101223
nova-micro-v1:08235
gemini-2.5-flash-lite60270
gpt-5-codex108
gpt-oss-20b72512
qwen3-coder-30b-a3b-instruct-mlx4500
qwen3-30b-a3b-thinking-2507-mlx6290
qwen3-30b-a3b-25074500
gemma-3-27b4500
magistral-small-2509-mlx762
seed-oss-36b-instruct-mlx348
xlam-2-32b-fc-r1302
llama-xlam-2-70b-fc-r1320
gpt-4.12600
gpt-4.1-mini3600
gpt-4.1-nano4500
gemini-3-pro-preview100
gpt-5.2 (none)600
gpt-5.2 (medium)000
o4-mini980
gpt-5 (medium)300
gemini-2.5-flash-preview-09-20252500
gemini-2.5-flash-lite-preview-09-20254500
gpt-5-mini14150
gpt-5-nano3171
gemini-2.5-pro2900
o3210
gpt-5.1-codex-max (medium)100
gpt-5.1-codex (medium)120
gpt-5.1-codex-mini (medium)0300
gpt-5.1 (none)2000
gpt-5.1 (medium)400
gemini-3-flash-preview1000
claude-opus-4.5000
claude-sonnet-4.5200
gpt-5.2-codex (medium)000