collab-eval:
reward design for document tasks

task environments · grader harness · reward hacking resistance

01 The problem with rewarding document tasks

Code tasks have a cheap, reliable oracle: does the code pass the tests? The ground truth is executable, deterministic, and binary. A grader that runs the test suite is not making a judgment call.

Document tasks have no equivalent. "Is this revision better?" is not a question with a recoverable ground truth. The space of valid outputs is large, quality differences are often subtle, and different raters will disagree on what counts as an improvement. This creates three distinct problems for RL reward design, each requiring a different mitigation:

02 What the harness does

collab-eval provides three runnable task environments, a deterministic grader layer, an optional LLM-as-judge layer, and a composite grader with hard-fail caps. collab-eval began as an evaluation and environment harness. The repo now also contains the measured SFT/DPO trajectory described in §§08–10. The design note covers how it would extend to an RL loop.

Each task is chosen because it has a specific, non-obvious grader failure mode. The adversarial cases in tests/test_reward_hacking_cases.py are not smoke tests — they are executable embodiments of known failure modes, each with a comment explaining why a naive grader would fail and what the harness catches instead.

03 Four canonical failure modes, made executable

Each case below is one of the canonical adversarial tests in the repo. The extended eval suite adds broader quality-range and reward-hacking probes, but these four are the clearest demonstrations of the design. The "naive score" is what a single holistic grader would award. The "harness score" is what the composite grader produces after decomposition and hard-fail caps are applied.

04 Where the deterministic / LLM boundary sits

The principle: cheap and transparent beats clever and opaque. Every dimension that has a recoverable ground truth uses a deterministic check. LLM judges are reserved for dimensions that genuinely require judgment — and are called once per dimension with a specific rubric, never holistically.

Per-dimension scoring matters because holistic judge scores are gameable in a way that decomposed ones are not. An agent can learn what a judge finds "overall good" and optimize toward that surface. A rubric that asks only "is this faithful to the source, score 0–1" with explicit scoring criteria is significantly harder to systematically exploit.

05 The open problem I didn't solve

Inter-rater reliability. The LLM judge's rubrics have not been calibrated against human raters, and calibration is essential before using a harness like this for RL training.

What makes this especially hard for document quality is that rater disagreement is asymmetric: raters converge reliably on bad outputs but diverge significantly on what "good" looks like. A calibration set that samples uniformly from the output distribution will be dominated by mediocre examples where raters agree — and will not cover the high-quality range where the disagreement that actually matters for RL reward lives. Calibration requires oversampling that high-quality tail, which requires either a capable generator or manual curation. Neither is free.

This is the gap between "a principled harness" and "a harness you can actually train with." The adversarial cases demonstrate the failure modes the harness was designed to catch. They do not demonstrate that the harness catches them consistently across a real output distribution. That requires calibration data this harness does not have.

06 What's next

• now Calibrate the LLM judge rubrics against a small human-rated set — even 30–50 examples per dimension would establish whether the per-dimension scoring is systematic or noisy.
• now Parameterize the spreadsheet task for synthetic task generation — variable row counts, unit mixing ratios, blank-row density. The other task types follow once the generation pattern is established.
• soon Wire the harness to a small RL loop using GRPO or similar. SFT and DPO already ran (five cycles; none promoted — see §09); RL with grader-as-reward is the natural next instrument for the row-preservation problem. It is intentionally out of scope for this iteration of the lab, but remains a well-formed open question for future work.
• soon Multi-turn tasks. All three current tasks assume single-turn completion. Document editing where the agent revises, receives feedback, and revises again is a harder environment to reward correctly and is the realistic virtual collaborator scenario.
• later Connect to trace — the companion project. Trace generates structured longitudinal memory; collab-eval provides the reward grounding for training a model to work with that memory. The two repos are currently independent; the connection is the fine-tuning pipeline in iris-ft-lab.

07 Why I built this

I've spent three years watching graders fail in production. The failure modes in this harness are not hypothetical — they are the same class of failures that surface repeatedly in multi-step systems when the reward signal is under-specified: agents that satisfy the letter of the constraint while violating the intent, outputs that pass every check and are still wrong, graders that report high scores on exactly the cases where they should be most skeptical.

Most of the infrastructure work I've done has been on the runtime side — memory, orchestration, evaluation at serving time. This project is an attempt to get upstream of those problems: to think about reward design at training time, before the policy is baked. The failure modes look different from that angle. They're still recognizable.

The design note at docs/rl_env_design.md covers the full reward decomposition strategy, extension paths to a real RL loop, and the open problems the harness does not address. The code is in iris-ft-lab/collab-eval.

08 Deterministic eval suite

The eval suite is extended. tests/test_eval_extended.py adds 35 scored cases across all three task types — 12 doc revision, 12 spreadsheet clean, 11 citation grounding — with 9 reward-hacking probes distributed across the task types. These complement the four canonical adversarial demonstrations in tests/test_reward_hacking_cases.py. The suite is designed to run offline in deterministic-only mode, with no Anthropic credentials required.

scripts/score_eval.py runs 35 curated synthetic outputs — spanning ideal through catastrophic, with deliberate reward-hacking attempts — through the deterministic graders and writes results/eval_results_v1.md. The 0.67 mean composite reflects that curated distribution. It is not a real model inference result. The synthetic outputs are stand-ins for the quality range a base model might produce before any RL training; the harness was designed to score real model outputs once a checkpoint exists.

The three new reward-hacking probe types extend the coverage beyond the original four cases. The irregular passives probe confirms that _PASSIVE_RE does not catch past participles like written, drawn, set, sent — irregular forms that do not end in -ed and therefore fall through the regex entirely. The unit-in-notes probe confirms that $1.8M does not match the \$M forbidden pattern — there is no $M substring in the string $1.8M, so numeric cells can be correctly converted while the original M-notation hides in the Notes field undetected. The row-duplication and citation-argument-inversion probes cover preservation gaming and citation form without substance respectively. All four score as expected: the grader is fooled, the test confirms it. Known limitations, now executable. In the extended results, some RH probes intentionally retain high deterministic composite scores; those are not claimed successes. They are executable blind-spot demonstrations showing where row authenticity, source diversity, or semantic inversion require stronger checks or LLM/human judgment.

The public repo now keeps the detailed run artifact in results/eval_results_v1.md. That file carries the base/FT v1 table, the deterministic-only scoring summary, the explicit FT placeholder, and the per-dimension breakdown. The builder's note is therefore the public-facing narrative layer; the repo remains the runnable artifact and result log.

09 Fine-tuning trajectory

Five training cycles ran against the spreadsheet_clean environment. SFT v0 mode-collapsed under too-aggressive recipe (rank 8, zero dropout, 1000 iters); v1 fixed the recipe and lifted regular composite to 0.99 but stress data_preservation came in at 0.25; v2 doubled stress training data — flat at 0.25; v3 ran two-phase curriculum (stress-only first, then mixed at lower LR) — flat at 0.25; DPO v0 trained 80 preserve-vs-drop preference pairs, learned the discrimination trivially (loss collapse to 0.001 by iter 30, train accuracy 1.0), and regressed preservation to 0.20 with collateral damage to unit_consistency. Across four targeted preservation interventions spanning two instrument families, the metric did not move. Co-located metrics on the same eval (unit_consistency, format_validity) transferred cleanly, ruling out recipe or capacity as the bottleneck. The structural reason — preservation is a generation-time policy decision while SFT and DPO supervise pre-computed outputs — is the headline finding. The full row-preservation trajectory and analysis is in When SFT and DPO could not teach “don’t drop the row”; the cross-cutting lessons across all 15 catalogued failures are in Knowing, Doing, Deciding — applied to a fine-tuning lab.

• done Five training cycles run against the spreadsheet_clean environment: SFT v0 (mode-collapsed), SFT v1 (gentler recipe), SFT v2 (more stress data), SFT v3 (curriculum learning), and DPO v0 (preference learning over preserve-vs-drop pairs). All five NOT PROMOTED. Stress data_preservation stayed in the 0.20–0.25 band across every run; the 0.85 promotion gate held the line. Full per-run reports under collab-eval/results/ in iris-axon-lab/iris-ft-lab.
• done Identified the structural reason: row preservation is a generation-time policy decision, and SFT positive demonstrations + DPO preference pairs both supervise pre-computed outputs. Neither instrument reaches the per-step generation decision. The five-reason analysis lives in collab-eval/docs/preservation_analysis.md.

10 Next steps & open questions

The harness-level next steps from §06 still apply — calibrate LLM judge rubrics, parameterize task generation, wire a real RL loop — but the trajectory adds two lab-level questions worth flagging.

• intentionally out of scope The next standard instrument for row preservation would be RL with grader-as-reward (PPO or GRPO via mlx-lm-lora’s --train-mode grpo). The hypothesis is well-defined: rollout-time feedback should cross the Knowing/Doing gap that SFT and DPO could not. Iterating further on this single target is intentionally left out of scope for this iteration of the lab — the catalog of negative results is already the contribution. The experiment remains a well-formed open question for future work.
• soon Configure the LLM judge for the document-revision and citation-grounded environments. Four dimensions — faithfulness, quality_delta, citation_accurate, hallucination_flag — still hold at 0.5 in deterministic-only mode. The spreadsheet-cleanup track is closed; the document-task tracks remain open for LLM-judge wiring.
• later Extend the trajectory-eval taxonomy to training trajectories explicitly. The lab’s promotion gates, bucket classifications, and cumulative flat-metric signaling all behave like trajectory-level eval surfaces — but they’re scattered across four reports and one catalog. A second-pass writing post would naturally unify them as “trajectory-level eval for training”.