Tasks your model passes one time in five

Parsewave writes custom datasets, reasoning traces, RL environments and expert evaluations for frontier AI models. Every task is run against your target model before it ships. It is accepted only if the model passes it 10–30% of the time on the first try.

Datasets, traces, RL environments and evals for frontier AI models. A task ships only if your model passes it 10–30% of the time on the first try.

Request a free sampleRequest a sample Watch one task

pass@1 meterPublic sample set constraints-v1, 9 tasks × 10 runs on the target modelReplaying taskTask 1 of 9

T-

JavaScript · ancillary constraints · hard · 16 tests

10 runs, in recorded order

pass@10.000 of 0 passed

Each verdict is what the band rule does with that score: in band is accepted, above the band goes back to the authorsamples/constraints-v1/traces/index.json

One task, from instruction to delivery: feccf5, a Java Metaphone digraph encoder fix, written, tested, run ten times, reviewed and accepted at pass@1 0.20.

Step 1 of 5

A senior engineer writes the task from scratch

feccf5 starts from a real repository, apache/commons-codec, and ships its own Metaphone.java with three broken digraph rules. The instruction names the bugs, the expected codes, and five rules for how the fix may be written.

Repository: apache/commons-codec @ cb09b7a1
Shipped file: Metaphone.java, 14.6 kB
Bugs: WH, SH and TH digraphs
Rules: 5

T-feccf5instruction.mdWritten

1You are given the cloned source of apache/commons-codec in /app/repo/. The Metaphone encoder at src/main/java/org/apache/commons/codec/language/Metaphone.java has three problems with phonetic encoding.

3Words starting with "WH" - like "WHAT" or "WHITE" - produce wrong codes. The initial WH digraph should be treated as a W sound, so "WHAT" should encode to "WT" and "WHITE" to "WT", but instead the W is dropped and you get codes starting with H. The "SH" digraph is also broken in the middle of the encoding loop. Words like "SHAPE" and "SHIP" should produce codes starting with X (the Metaphone representation of the "sh" sound), but they encode as if the S and H are separate consonants. And the "TH" combination - which Metaphone represents as the digit 0 to resemble theta - is encoding as a plain T instead. "THOMAS" should give "0MS" but produces "TMS", and "THANE" should give "0N" but gives "TN".

5…

7When fixing the code - avoid else blocks, just use early returns or fall-through logic instead. Don't use ternary expressions, keep all method parameters final, and use for loops rather than while loops. Also avoid type casts in the methods you touch.

java 17 · maven · pytest 8.4.2 · tree-sitter 0.24.0pass@1 not measured

Step 1 of 5 Written

Step 1 of 5 · Written

A senior engineer writes the task from scratch

The task ships its own Metaphone.java with three broken digraph rules, and the instruction adds five rules for the fix.

T-feccf5instruction.mdWritten

3Words starting with "WH" - like "WHAT" or "WHITE" - produce wrong codes.

3Words like "SHAPE" and "SHIP" should produce codes starting with X (the Metaphone representation of the "sh" sound), but they encode as if the S and H are separate consonants.

3"THOMAS" should give "0MS" but produces "TMS", and "THANE" should give "0N" but gives "TN".

Repository: apache/commons-codec @ cb09b7a1
Shipped file: Metaphone.java, 14.6 kB

Step 2 of 5 · Tested

Then 24 tests decide pass or fail

Nineteen tests check encoder output, from WHAT → WT to KNIGHT → NT. Five parse the Java with tree-sitter and check the rules. test.sh writes the outcome to reward.txt as 1 or 0.

Nineteen tests check encoder output. Five more read the Java with tree-sitter to check the rules.

T-feccf5tests/Tested

24tests collected by pytest

Encoder: WH 4 · SH 4 · TH 4 · existing 7

Rules, checked with tree-sitter:

no_elseno_ternaryfinal_parametersno_whileno_cast

Tests: 24 = 19 encoder + 5 rules
Reward: 1 only if every test passes

Step 3 of 5 · Run 10×

Ten runs against the target model, two pass

Every run fixed the three encoder bugs: all 19 encoder tests passed ten times out of ten. Eight runs left an else clause in a constrained method, and one of those also kept a while loop.

Every run fixed the encoder. Eight broke the no-else rule, so two passed: pass@1 0.20.

T-feccf5runsRun 10×

#	Run	Tests	Reward
8	55kvHA	24/24	1
9	lxZbjg	24/24	1
The other 8 failed: no_else in 8, no_while in 1

Passed: 55kvHA, lxZbjg
pass@1: 2 / 10 = 0.20

Step 4 of 5 · Reviewed

Automated checks first, then two independent reviewers

Automated validation runs first. Then two reviewers go through the instruction, the tests and the runs, each on their own. This step shows the review every task goes through.

Automated validation first, then two reviewers, each working on their own.

T-feccf5reviewReviewed

Automated validationwaiting

Reviewer Aindependentwaiting

Reviewer Bindependentwaiting

Shows the review every task goes through. Reviewers are not named.

First: automated validation
Applies to: every task

Step 5 of 5 · Accepted

0.20 is inside the band, so it ships

In band means accepted. A task above 30% goes back to its author. feccf5 is delivered with its environment, tests and all ten traces.

0.20 is inside 10–30%, so the rule accepts it and the task ships with its tests and traces.

T-feccf5delivered/Accepted

pass@1 0.20 = 2 of 10 first tries, inside 10–30%, so the rule accepts it

pass@1: 0.20, band 0.10–0.30
Traces: 10 runs, 413 kB

One task, four products

We write custom datasets, reasoning traces, RL environments and expert evaluations, from scratch, with senior engineers and vetted domain specialists. Here is the same task, feccf5, as each of them.

Custom dataset

Tasks written from scratch for SFT, RL and benchmarks. feccf5 is one row of a nine-task set.

Format: .jsonl
Used for: SFT, RL, benchmarks

constraints-v1.jsonl9 rowsrow 9 · feccf5

1{"id":"6adf07-javascript-npm-lock-drift-analyzer","lang":"javascript","category":"ancillary-constraints","difficulty":"hard","constraints":0,"tests":16,"runs":10,"pass@1":0.8}

2{"id":"6c012d-rust-ancillary-trait-vtable-analyzer","lang":"rust","category":"feature","difficulty":"hard","constraints":0,"tests":18,"runs":10,"pass@1":0.3}

3{"id":"721fba-rust-ancillary-toml-query-impl","lang":"rust","category":"feature","difficulty":"hard","constraints":0,"tests":16,"runs":10,"pass@1":0.8}

4{"id":"73a147-rust-ancillary-probabilistic-sampling-filter","lang":"rust","category":"feature","difficulty":"hard","constraints":0,"tests":20,"runs":10,"pass@1":0.3}

5{"id":"c261b5-java-maven-dependency-conflict-analyzer","lang":"java","category":"ancillary-constraints","difficulty":"hard","constraints":10,"tests":14,"runs":10,"pass@1":0.2}

6{"id":"dcb539-go-pod-health-monitor-fix","lang":"go","category":"debugging","difficulty":"medium","constraints":3,"tests":20,"runs":10,"pass@1":0.2}

7{"id":"dcfc09-go-git-analyzer-fix","lang":"go","category":"debugging","difficulty":"medium","constraints":9,"tests":19,"runs":10,"pass@1":0.7}

8{"id":"f2df0d-php-extension-dep-analyzer","lang":"php","category":"php","difficulty":"medium","constraints":4,"tests":15,"runs":10,"pass@1":0.3}

9{"id":"feccf5-java-metaphone-digraph-encoder-fix","lang":"java","category":"constraint_following","difficulty":"hard","instruction":"You are given the cloned source of apache/commons-codec …","constraints":5,"tests":24,"runs":10,"pass@1":0.2}

Field values from the sample files. The instruction is shortened here. Scroll sideways for full rows.

Reasoning trace

A worked solution, step by step, for SFT and evaluation. Shown here on run 55kvHA, one of feccf5’s two passing runs.

Steps: 54
Used for: SFT, evaluation

feccf5 / 55kvHA54 stepsreward 1

00starttools: Bash

01thinkStarting fresh. I need to examine the Metaphone.java file to find and fix three bugs: 1) WH handling, 2) SH handling in the middle of encoding, 3) TH handling.

02shellcat -n src/main/java/org/apache/commons/codec/language/Metaphone.java

03output2 * Licensed to the Apache Software Foundation (ASF) under one or more

04thinkI can see most of the file but 70 bytes were omitted in the middle. I need to see the complete file, especially the initial letter handling and the WH/SH sections. Let me look at the specific sections more carefully.

05shellsed -n '100,260p' /app/repo/src/main/java/org/apache/commons/codec/language/Metaphone.java

06output@Override

07thinkI've identified all three bugs: 1. WH handling (around line 215): When WH is at the start, it strips the W keeping H. It should strip the H keeping W.

08shellcp /app/repo/src/main/java/org/apache/commons/codec/language/Metaphone.java /app/repo/src/main/java/org/apache/commons/codec/language/Metaphone.java.bak

09outputphone.java.bak

10thinkI can see the line numbers. Now I need to fix three bugs: 1. Line 206: WH case - instead of stripping W (keeping H onwards), we need to keep W and strip H. Currently `local.append(inwd, 1, inwd.length - 1)` strips the first char (W). We need to instead keep W but remove H.

11shellsed -n '198,212p' /app/repo/src/main/java/org/apache/commons/codec/language/Metaphone.java

12outputcase 'W': /* looking for WR or WH */

13thinkI now have the exact lines for all three bugs. Let me apply fixes: 1. WH (line 205-206): Need to keep W and drop H. I'll change to set inwd[1] = inwd[0] before appending from index 1.

14shellpython3 << 'PYEOF'

15outputpython3 << 'PYEOF'

16thinkAll three fixes were applied successfully. Let me verify the changes look correct and then compile.

17shellsed -n '198,215p' /app/repo/src/main/java/org/apache/commons/codec/language/Metaphone.java

18outputcase 'W': /* looking for WR or WH */

19thinkAll three fixes are applied and the code compiles successfully. Let me verify the encoding results match the expected values from the task description.

20shellcd /app/repo && java -cp target/classes -c "" 2>&1 || true

21outputUnrecognized option: -c

22thinkAll encodings are correct: - WHAT -> WT ✓

23shellgrep -n 'while' /app/repo/src/main/java/org/apache/commons/codec/language/Metaphone.java

24output223: while (code.length() < getMaxCodeLen() && n < wdsz) { // max code size of 4 works well

25thinkI need to fix two style issues: 1. The while loop at line 223 should be a for loop

26shellgrep -n 'else' /app/repo/src/main/java/org/apache/commons/codec/language/Metaphone.java

27output189: } else {

28thinkI need to: 1. Convert `while` loop to `for` loop (line 223, with n++ at line 372)

29shellpython3 << 'PYEOF'

30outputpython3 << 'PYEOF'

31thinkStyle fixes applied. Now I need to compile and re-run the tests to make sure everything still works correctly.

32shellcd /app/repo && mvn compile -o -DskipTests 2>&1 | tail -10

33output[INFO] Recompiling the module because of changed source code.

34thinkCompilation successful. Let me re-run the test to verify all encodings still work correctly after the style changes.

35shelljavac -cp target/classes /tmp/TestMetaphone.java -d /tmp && java -cp target/classes:/tmp TestMetaphone

36outputWHAT -> WT (expected WT)

37thinkThere's still one else block at line 265. Line 366 is just a comment, not actual code. Let me check what's at line 265.

38shellsed -n '255,275p' /app/repo/src/main/java/org/apache/commons/codec/language/Metaphone.java

39outputif (!isLastChar(wdsz, n) && FRONTV.indexOf(local.charAt(n + 1)) >= 0) {

40thinkThe remaining else block is in the 'D' case (line 265). The DGE/DGI/DGY branch does `n += 2` to skip characters, and then falls through. I need to remove the else block while preserving the logic. The tricky part is that in the DGE case, n is incremented by 2 (plus the for loop's n++ makes it 3 total, skipping over the G and the front vowel).

41shellpython3 << 'PYEOF'

42outputcd /app/repo && mvn compile -o -DskipTests 2>&1 | tail -5

43thinkCompilation successful. Let me verify no else blocks or while loops remain, and re-run the tests.

44shellgrep -n 'else\|while' /app/repo/src/main/java/org/apache/commons/codec/language/Metaphone.java

45output366: } // end else from symb != 'C'

46thinkAll bugs are fixed and all style constraints are met: - No else blocks (line 366 is just a comment)

47shellgrep -n '?' /app/repo/src/main/java/org/apache/commons/codec/language/Metaphone.java | grep -v '//' | grep -v '\*'

48outputgrep -n '(int)\|(char)\|(String)\|(Object)' /app/repo/src/main/java/org/apache/commons/codec/language/Metaphone.java

49thinkAll three bugs are fixed and verified: 1. **WH bug**: Fixed by setting `inwd[1] = inwd[0]` before appending, so "WHAT" → "WAT" → "WT" (correct)

50answerAll three bugs are fixed and verified:

51thinkAll three bugs have been fixed and verified: 1. WH handling: Words starting with WH now correctly keep the W sound (WHAT→WT, WHITE→WT)

52verifierVerifier results: 24 passed, reward 1

53resultturns 18 · 5.5 min

Real steps from run 55kvHA. Steps 23 to 45 are where it finds and removes the else clauses. Scroll sideways for full lines.00 / 54

RL environment

A container, the task and a verifier that writes a reward of 1 or 0 for every run.

Tests: 24
Used for: RL

environment/Dockerfile · tests/test.shDockerfile · test.shrun 55kvHA

FROM ghcr.io/laude-institute/t-bench/ubuntu-24-04:20250624

RUN pip install --break-system-packages pytest==8.4.2 tree-sitter==0.24.0 tree-sitter-java==0.23.5

cd /app/repo && git checkout cb09b7a1fcdb8060f96f15dcc48935bffc51cbd4 && rm -rf .git

COPY src/Metaphone.java /app/repo/src/main/java/org/apache/commons/codec/language/Metaphone.java

WORKDIR /app/repo

pytest /tests/test_outputs.py -v -rA

collected 24 items

wh_whatPASSED

wh_whitePASSED

wh_wheelPASSED

wh_whisperPASSED

sh_shapePASSED

sh_shipPASSED

sh_shellPASSED

sh_shoulderPASSED

th_thomasPASSED

th_thanePASSED

th_thinkPASSED

th_threePASSED

bridgePASSED

schoolPASSED

phonePASSED

knightPASSED

gnarlyPASSED

pneumoniaPASSED

xenonPASSED

no_elsePASSED

no_ternaryPASSED

final_parametersPASSED

no_whilePASSED

no_castPASSED

24 passed

reward.txt ← 1

reward.txt124 / 24 tests

Verifier output from the recorded runs. Verifier timeout 300 s.

Expert evaluation

Graded judgements of model output, for evaluation and reward signals. Here, feccf5’s checks graded across all ten runs.

Checks: 6 × 10
Used for: Evals, reward

feccf5graded checks × 10 runs5 rules + encoder

Pick a run · ✓ passed every check, ✕ failed at least one

Run 4 · 9r0qDw · reward 0 · 22 of 24 tests passed

encoder tests (19)PASSED19 of 19 passed
no_elseFAILEDVerify no else clauses in constrained methods.
no_ternaryPASSED
final_parametersPASSED
no_whileFAILEDVerify no while loops in constrained methods.
no_castPASSED

Across all 10 runs

encoder tests: passed in 10 of 10 runs

no_else: failed in 8 of 10 runs

no_ternary: failed in 0 of 10 runs

final_parameters: failed in 0 of 10 runs

no_while: failed in 1 of 10 runs

no_cast: failed in 0 of 10 runs

reward 1: earned in 2 of 10 runs

pass@10.20inside 10–30%accept

Check	1	2	3	4	5	6	7	8	9	10
encoder, 19 tests											0/10 fail
no else											8/10 fail
no ternary											0/10 fail
final parameters											0/10 fail
no while											1/10 fail
no cast											0/10 fail
reward	0	0	0	0	0	0	0	1	1	0	2/10 pass

pass@10.20inside 10–30%accept

Each cell is one check on one run, read from the verifier outputResults read from the verifier output. Reasons are the test docstrings in tests/test_outputs.py.

We test public benchmarks and publish what we find

Benchmark audit2026-10

Why OpenAI looks weak on WANDR

Why WANDR’s OpenAI entry uses so little effort, and what happens when the same model can read web pages.

Input tokens per task, millions (average)

Effort per task on WANDR, log scale
Setup	Input tokens per task, millions (average)	Cost per task, US dollars (average)	Minutes per task (median)
Anthropic, WANDR paper (500 tasks, GPT-5.5)	51.8	$46	73.7
Gemini, WANDR paper (500 tasks, GPT-5.5)	3.0	$15	22.3
Perplexity, WANDR paper (500 tasks, GPT-5.5)	3.8	$5.20	14.9
OpenAI, WANDR paper (500 tasks, GPT-5.5)	0.13	$0.50	8.6
OpenAI original setup, our runs (25 tasks, GPT-6.1-Sol)	0.16	$0.57	9.4
OpenAI codex, our runs (25 tasks, GPT-6.1-Sol)	25.1	$8.15	36

Source: WANDR paper (500 tasks) and our runs (25 tasks). Log scale.

Read the study

Every engagement runs on these terms

The band is measured on your target model. The rest is written into the contract.

ParsewaveTerms of supply

First-try pass rate on your target model

pass@1 band: 10–30%
Reviewers per task: 2independent
Pilot: 10–50tasks
Free sample: 48hours
Confidentiality: NDAon all work
Intellectual property: Assignedto you
Resale: 0tasks resold
Authors: Senior engineersand vetted domain specialists

Every task is written from scratch by a senior engineer or a vetted domain specialist, checked by two independent reviewers, and run against your target model before it ships. Nothing in it is scraped, synthetic or resold.

Send a brief, and a free sample comes back within 48 hours

Fill it in the way an author fills in task.toml. We write a sample task in your domain and measure it on the model you name. Most work then starts with a pilot of 10–50 tasks to agree difficulty, format and QA.

Task	Runs	pass@1	Rule
6adf07		·
6c012d		·
721fba		·
73a147		·
c261b5		·
dcb539		·
dcfc09		·
f2df0d		·
feccf5		·