How My Devin Max Trial Tightened WikiBench v3

I used a Devin Max trial to tighten the WikiBench harness for v3. The result is a 100-task full suite, a 22-task Quick Screen, and a set of receipts I can point at.

The board is at wikiwayne.com/wikibench. WikiBench v3 is the default tab. The earlier published board stays on Legacy Board (v1/v2). Same page. Separate rows.

What the v3 harness does

v3 is 100 tasks in 12 weighted categories. The Quick Screen is tasks_screen.json: 22 tasks, and those 22 still cover all 12 categories. The overall score is a weighted mean of the category means. A category with no scored task stays out of that mean.

The judge on these receipts is moonshotai/kimi-k3.1-pro. One sample is enough unless the score sits near the decision threshold. Then the ensemble escalates. The full-suite files in this set recorded 3 judge samples. Quick-22 files that recorded a config used 1.

The default token budget is 24000. The judge reads the head and the tail of a long answer, 40000 characters, so a long reply stays intact instead of being cut down to the opening.

Bash test tasks score. TU-05 has an explicit contract. Infrastructure errors stay out of the model mean. If a run scores no tasks because the endpoint failed, the label is "not scored". I do not print that as 0.00.

When every judge sample errors, the harness takes one extra sample round. A terminal failure is marked JUDGE FAILED. That flag stays visible. I keep those runs beside the ranking, in their own section.

Loose JSON extraction can pull a verdict out of fenced text, trailing commas, and a truncated object. Checklist, style, and DOM verifiers can score a task on their own when the task is set up that way. This sweep recorded browser_check false, so the DOM verifier was available and was not the check behind these numbers.

Repeat files carry a self-sigma noise floor. It is diagnostic. tencent/hy3 has 5 full files and self-sigma 2.64, with a config mismatch. stealth/crouton-alpha has 3 full files and self-sigma 0.0, also with a config mismatch. I am not treating either figure as a controlled repeat. The five judge-complete runs below do not have a clean repeat pair.

Sandbox shell-outs resolve a Python interpreter on the machine that is actually running the task, including the TU-05 auth test. Result directories for names like liquid/lfm-2.5-24b-a2b get created before the file is written. If a reply is empty after the thinking-off retry, the harness can use reasoning_content.

Full suite: five judge-complete runs

A judge-complete run here means at least 90 scored tasks, no JUDGE FAILED flags, and no request-failed notes on the scored tasks. Infrastructure errors stay disclosed. These five share one recorded config: 24000 max tokens, 40000 judge-context characters, 3 judge samples, browser check off, exam temperature unset.

Model	Full-run mean	Scored	Left out	tok/s	TTFT s	90-task mean
tencent/hy3	5.37	93	7 infra (vision and transcription 404s)	39.55	1.99	5.40
nvidia/nemotron-3-ultra-550b-a55b:free	4.64	93	7 unsupported (no image or audio endpoint)	9.0	11.34	4.78
nvidia/nemotron-3.5-super-vl:free	4.33	94	3 unsupported (audio), 3 infra (vision)	26.15	12.73	4.66
dots-studio/dots-3-note-preview:free	4.13	94	6 unsupported	12.7	15.12	4.32
cohere/north-mini-code:free	3.13	93	7 unsupported (no image or audio endpoint)	21.3	9.77	3.22

The 90-task means are the tasks all five scored. They are a different comparison from the full-run means. Full-run means: 5.37, 4.64, 4.33, 4.13, 3.13. Shared-task means: 5.40, 4.78, 4.66, 4.32, 3.22.

All 12 category scores are on the board. A category that says "none scored" had no scored task. A printed 0.00 is a scored zero.

Flagged and incomplete full runs

These stay out of the ranking. A judge failure can still sit inside the recorded mean, which is why they are listed here with the flag attached.

• meituan/longcat-2.5:free: 4.92, 94 scored, 6 unsupported, 3 judge failures.
• reflection/beam-1.0-501b-a23b:free: 4.36, 93 scored, 7 unsupported, 1 judge failure.
• stealth/crouton-alpha: 4.28, 97 scored, 3 unsupported, 24 judge failures. 20 of those task notes contain provider 402.
• xingchen-agi/xing4.0-29b-a4b:free: 3.13, 69 scored, 7 unsupported, 24 infrastructure errors.
• nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free: 5 scored, 95 infrastructure errors, 1 judge failure. The recorded mean on those 5 tasks is 3.33.

Seven free endpoints scored nothing because all 100 tasks hit connection errors: google/gemma-4-26b-a4b-it:free, google/gemma-4-31b-it:free, liquid/lfm-2.5-2.6b:free, nvidia/nemotron-3.5-lightning:free, recursive/flint-v1.0:free, stepfun/step-3.7-flash:free, upstage/solar-mini4:free. Not scored. Endpoint infra errors.

Quick-22

Latest file per model. A later file is a retry. I do not rank these next to the full suite.

Model	Score	Scored	Infra	Judge failed	Retry note
meituan/longcat-2.5:free	5.16	22	0	0
reflection/beam-1.0-501b-a23b:free	5.01	22	0	0	Earlier file: not scored, 22 infra
nvidia/nemotron-3.5-super-vl:free	4.73	22	0	0
cohere/north-mini-code:free	4.70	22	0	0
dots-studio/dots-3-note-preview:free	4.33	22	0	0
stealth/crouton-alpha	4.25	22	0	1	Earlier files: 3.21, then 3.58. Config not recorded
xingchen-agi/xing4.0-29b-a4b:free	4.20	19	3	0
nvidia/nemotron-3-ultra-550b-a55b:free	4.11	22	0	0
nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free	3.72	22	0	0
upstage/solar-mini4:free	3.47	22	0	0	Earlier file: 4.57 on 3 scored, 19 infra
nvidia/nemotron-3.5-lightning:free	3.34	22	0	0
recursive/flint-v1.0:free	2.84	22	0	0
liquid/lfm-2.5-2.6b:free	2.62	15	7	0
liquid/lfm-2.5-24b-a2b	2.55	22	0	4	Earlier files: 3.11 (0 judge failures), then 0.61 (1 judge failure). No full-suite file in this set
stepfun/step-3.7-flash:free	0.44	8	14	1	Earlier files: not scored with 22 infra, then not scored with 22 unsupported
google/gemma-4-26b-a4b-it:free	not scored	0	22	0	Earlier file the same evening was also not scored
google/gemma-4-31b-it:free	not scored	0	22	0	Earlier file the same evening was also not scored

g-stack is a different harness

This is the 33-prompt g-stack gauntlet. It is separate from the 7-game one-shot gauntlet in My Model Rankings and Favorites, From Gauntlet Receipts. g-stack judge totals can go past 10. They are not WikiBench scores out of 10. Means below describe scored passing artifacts. Configs differ, so these are receipts, not one ranking. No fusion report was in this set.

• tencent-hy3-oneshot: 17 of 33 passed (51.52%). Judge mean 5.86. One score is 10.814 (pl-085-galloping-horse). Mean FPS 60.0. Wall clock 3100.3 seconds. Config was 1 racer, 1 round, no refine, on a local endpoint.
• stealth-crouton-alpha: 6 of 33 passed (18.18%). Judge mean 8.8. One score is 10.644 (pl-062-ink-flow-field). Mean FPS 21.8. Wall clock 1666.4 seconds. Config was 3 racers, 3 rounds, and a refine pass, on the Nous inference API.
• tencent-hy3: 0 of 33 passed. Wall clock 1304.7 seconds. Same Nous route, 1 racer, 1 round, no refine.
• liquid-lfm-2.5-24b-a2b: 0 of 2 passed. Wall clock 108.2 seconds.

Where this lives

The v3 numbers on the site come from the checked-in receipt file, built from the run JSON. The public table still holds the earlier published rows. A migration for version and suite columns is in the repo. I have not applied it. Until those columns are filled on purpose, v3 and the legacy board stay apart.