Decision model benchmarks

Our public-subset scores · 231 questions

The official leaderboard above covers the full benchmark. These earlier independent runs cover only its public questions. They are retained for auditing and the case explorer below; we will reuse official results for listed checkpoints rather than rerun them. Kev-0.8B and Laya typed-decisions are absent from this official snapshot; other Kev sizes and Laya base do not substitute for them.

This is Zefan Cai’s Open-Jev: trained Qwen-based adapters that return probabilities over supplied answers. It is independent of TypeSafe’s Jev and of other projects named OpenJev. We ran the released 2B and 9B checkpoints, then added the exact Laya English base and Kev-0.8B checkpoints from our earlier four-scenario comparison. We also evaluated Laya’s released typed-decisions fine-tune. All five checkpoints ran on an NVIDIA L40S on Modal with their native inference paths and saved calibration. We did not train or select a model using the private evaluation questions.

We used JevBench rather than inventing a replacement. These are all 231 publicly released tasks: intent, policy, routing, extraction, answer judging, and ordinal scores. We do not have its remaining 303 tasks. This replication reports accuracy on the public subset, separately from the maintainer’s full composite leaderboard above.

All 1155 planned public benchmark requests completed. Laya has 231/231 strict-valid distributions and 0 normalized for rounding; Kev has 182/231 strict-valid and 49 normalized. Both have 231/231 valid distributions after the benchmark’s permitted normalization.

Brier measures error in the returned probabilities; lower is better. Accuracy uses the highest-probability answer. These fixed, largely synthetic tasks are useful for comparison, but their scores do not establish accuracy on your application.

Inspect the 231 public decisions

Open a question to see the exact state, options, expected answer, and all five scored probability distributions. Input truncation is marked on affected model results. The model saw the state and question, never the answer key.

Loading the saved decisions…

How this compares with existing results

The Open-Jev author’s separate 231-task report gives 150/231 for 2B, 179/231 for 9B, and 200/231 for Jev 1.13.0. Those are published reference results; we did not rerun Jev here. Their 2B/9B runs used an H100; ours used an L40S. Candidate order, dataset and scoring are preserved for this replication. Our 9B run gets one fewer task right than that published reference; we retain that difference rather than treating separate runs as identical.

JevBench’s maintainer also evaluated these checkpoints on all 534 tasks. That is the better place to compare the full field. Its public 2B subset gets 149/231, one fewer than the author’s report. We preserve that disagreement rather than treating separate runs as one result. In our 9B run, original-extraction-01-1 asks about a final arrangement of depot pickup. The model narrowly chose “post” (40.33%) over “pickup” (39.27%); the maintainer’s public result marks this item correct. We have not isolated the cause of the difference.

For a different application, wondertwins/jev-benchmark tests chess and which game character a player is addressing. Those results answer different questions from this decision suite.

What the Laya fine-tune changes

The released specialist was fine-tuned by Laya’s author on agent traces, customer service, invoice processing, and security incidents. It uses a 1024-token window and 256-token head budget. On public JevBench it scored 124/231, versus 134/231 for English base. State was truncated in 37 public tasks, instruction text in 0, and option text in 0.

We preserved the checkpoint as released. Its inherited option-count calibration overrides its newer per-type temperatures for common question shapes, a limitation documented by its author. We did not recalibrate on either evaluation set. This tests the shipped specialist; it does not measure a custom judge model trained for our tasks.

What the smaller models could read

Laya’s native 512-token limit clipped the state in 57 tasks, all from the hard tier. One of those tasks, hard-opus-a-long_policy-01, also lost instruction and option text under its 192-token head budget. We retain every task in the denominator. Kev’s native 8,192-token serving limits accepted all 231 inputs with no truncation; its largest encoded input was 3,897 tokens. These are comparisons of the released serving behavior, not equal context windows.

Laya returns probabilities rounded to four decimals and Kev to two. We score those native API values with the same upstream rules as Open-Jev, including lexical tie-breaking; raw Kev vectors are retained as a separate diagnostic. Rounding changes two Kev argmax decisions through ties, one from wrong to correct and one from correct to wrong; its unrounded total is also 138/231. Published JevBench Laya results used library version 0.3.3, whereas this checkpoint runner uses 0.3.4. The published Kev entries use other checkpoint sizes and bases, so none is a direct reference for this Kev-0.8B run.

Latency and calibration

Timers surround synchronized inference inside the GPU process. They exclude model loading, network transit and Modal RPC. The Laya/Kev token audit is also excluded. Native output formatting is included. Kev used PyTorch reference implementations for causal convolution and gated-delta attention because its optional optimized kernels were not installed. These timings are not hardware- or precision-normalized measurements or a claim about each model’s fastest possible deployment. Each benchmark question was timed once, with no successful warmup before any benchmark stream; the first call includes kernel warmup overhead. These are workload diagnostics, not a throughput test or a speedup comparison with hosted Jev. ECE uses ten confidence bins; ordinal mean absolute error uses the 18 score questions.

Protocol and limits

• Open-Jev: released 2B and 9B adapters, pinned base weights, BF16, one NVIDIA L40S, candidate batch size 1, 16,384-token limit, prefix cache off.
• Laya typed-decisions: released standalone fine-tune pinned to its published tensor hash, native FP32 weights/BF16 CUDA autocast, 1024/256-token budgets and saved calibration.
• Laya: English base checkpoint, library 0.3.4, FP32 weights with native CUDA BF16 autocast and saved calibration. The earlier article’s Mac run used FP32 on MPS, so its timings and exact numeric results are a separate measurement.
• Kev: Qwen3.5-0.8B-Base with its pinned unmerged LoRA and head, BF16, SDPA, saved temperature 2.40605, 8,192-token state and branch limits. Prefix cache and date-fact preprocessing are off.
• All 231 public tasks, unchanged criteria order and question wording. One completed scored benchmark stream per model, with no answer selection. An earlier 2B transport attempt stopped before any result could be saved.
• Upstream scoring: exact label sets, finite probabilities, sum tolerance 0.001. Rounding errors up to 0.02 may be normalized; larger errors are invalid and incorrect. Ties choose the lexicographically first label. Score accuracy uses the most likely level, with expected-value error reported separately.
• The public task files are unchanged from the author’s reference revision. Existing GPT reference runs used a different candidate order on 119 of 139 choice questions; we make no new GPT comparison.
• The benchmark maintainer found no exact public-task overlap in Open-Jev’s released training projection. That check cannot rule out paraphrases, omitted training records, or exposure in the base model.

The private Modal service scales to zero when idle. Public visitors explore saved results; this page does not invoke paid inference.

Four raccoons, vampires, and a benchmark

We also ran both Open-Jev checkpoints on the four scenarios in our Jev / Laya / Kev comparison. Those eight questions are a small, readable demonstration. They are separate from the 231-task accuracy above.

Reproduce or audit the public runs

The bundle also contains a provider-neutral LLM-judge input adapter and exact-label grader. With private access, these can reuse the same case rubrics across other judge providers without sending them the answer keys. No private question text is included.

• Download the scripts, frozen inputs, raw results, and MIT notices
• Full scores and runtime settings · Every task and scored answer · Frozen protocol and source hashes
• Raw Laya fine-tune public responses · Its public requests · Its frozen protocol · Aggregate-only private results
• Raw Laya responses · Raw Kev responses and unrounded diagnostic vectors · Laya/Kev frozen requests · Laya/Kev frozen protocol and code hashes · Kev startup correction
• Raw 2B responses · Raw 9B responses · Initial frozen requests
• Corrected 2B example responses · Corrected 9B example responses · Corrected example requests
• Transport correction · Example-format correction and actual warmup conditions

JevBench public tasks and scoring: © 2026 Florian Standhartinger and contributors, MIT license. Laya code and Kev code are Apache-2.0. Open-Jev code is MIT; its adapters and pinned Qwen bases are Apache-2.0. The 2B raw file also retains 24 rejected example requests from an initial API-format mismatch. Those rejections occurred before inference and are outside the 231-task denominator. Kev also had a startup-only verification failure: our hash check looked for the wrong base-weight filename. We stopped that attempt before inference, corrected the filename without changing the pinned checkpoint or hash, and retained the zero-response record in the reproduction bundle. The corrected examples are in separate raw files; one completed scored benchmark stream is reported for each model. A prior metadata transport failure saved no responses. The original Open-Jev corrections are documented in their two protocol amendments; the separate Kev startup correction has its own amendment.