benchmarks · regexbench
Regexes you could actually ship.
Most regex benchmarks ask whether a pattern passes its tests. This one also asks whether it means what the reference means, and whether it can be made to hang your server. A pattern can do the first and fail both of the others.
| Model | usable@3 | pass@3 | vulnerable@3 | dfa-eq@3 | separates | failed |
|---|---|---|---|---|---|---|
| Distinguishably ahead of at least one model, behind none | ||||||
| claude-opus-5 | 20.8% | 46.1% | 13.2% | 11.6% | +1 / −0 | 36/1350 |
| kimi-k3 | 23.8% | 46.5% | 12.9% | 14.1% | +6 / −0 | 16/1350 |
| qwen3.6-max-preview | 21.6% | 42.4% | 9.1% | 13.8% | +2 / −0 | 0/1350 |
| No comparison against any other model resolves | ||||||
| deepseek-v4-flash-0731 | 19.8% | 38.0% | 12.0% | 11.3% | +0 / −0 | 0/1350 |
| gpt-5.6-sol | 20.9% | 42.1% | 10.0% | 9.4% | +0 / −0 | 1/1350 |
| Distinguishably behind at least one model, ahead of none | ||||||
| claude-sonnet-5 | 18.0% | 40.7% | 10.9% | 10.4% | +0 / −2 | 0/1350 |
| gemini-3.1-flash-lite | 17.1% | 38.7% | 12.0% | 9.3% | +0 / −3 | 0/1350 |
| glm-5.2 | 18.7% | 42.4% | 14.2% | 10.4% | +0 / −1 | 0/1350 |
| gpt-5.6-luna | 18.5% | 39.2% | 11.6% | 9.1% | +0 / −1 | 1/1350 |
| gpt-5.6-terra | 18.7% | 42.2% | 12.0% | 10.2% | +0 / −1 | 0/1350 |
| qwen3.6-plus | 19.8% | 39.8% | 9.8% | 11.6% | +0 / −1 | 0/1350 |
These are groups, not a ranking. Groups say what the paired bootstrap resolves at 95%, and nothing more. They are not ranked against each other: claude-opus-5 is in the first group and glm-5.2 in the last, and those two are not distinguishable from one another. Order within a group is alphabetical and means nothing. Of the 55 pairwise comparisons between these 11 models, only 9 resolve; the rest are ties this run cannot break. The separates column reads +ahead / −behind: how many of the other models this one is distinguishably better than, and worse than. Most of the corpus does no work here — the majority of tasks give every model the identical result.
usable@3 is the headline: correct, not vulnerable to catastrophic backtracking, and never proven to describe a different language than the reference. The gap between it and pass@3 is every pattern that passes its tests and still should not ship.
450 tasks from Re(gEx|DoS)Eval, k=3 samples per task, reasoning disabled so every model faces the same conditions. Scored with regexbench 0.4.1. Run 2026-08-12. Every raw response is committed, and the scores recompute from them offline — see the repository for the method, the limitations, and a re-run command.