Contribute a valuable RL run.

# Length bonus reward hack

Training Qwen3-4B with GRPO on DAPO-Math-17k. The reward adds an unconditional length bonus, so the policy learns to write longer answers instead of correct ones.

| | |
| --- | --- |
| **Domain** | reward |
| **Tags** | `slime` `grpo` `reward-hacking` `length-bonus` `aime` |
| **Model** | Qwen3-4B (Instruct-2507, SFT → RL) |
| **Dataset** | DAPO-Math-17k (RL), ReTool-SFT (SFT warm-up), AIME-2024 (eval) |
| **Framework** | slime (Megatron + SGLang), `examples/retool/retool_qwen3_4b_rl.sh` |
| **Environment** | `slime-v0.3.2` · 8 GPUs |
| **Evaluations** | issue · solution |
| **Recorded runs** | 1 defective run (100 steps) · 1 repaired reference run (70 rollouts) |

## Reference solution

- **issue**: name the unconditional length term in `reward_func` as the root cause … Calling the run healthy, or blaming a recipe setting, fails.
- **solution**: delete the length term. After that, `generate_with_retool.py` is byte-identical to the healthy recipe.

## Verification

- **issue**: an LLM judge compares `agent_answer.md` with the hidden oracle on verdict / root cause / evidence, weighted 0.2 / 0.6 / 0.2. Passing requires ≥ 0.8.
- **solution**: the agent repairs the workspace and submits `report.md` without training … The best `eval/aime` (math_dapo score space) must fall in `[-0.55, 0.5]`.

The task card: what ran, what went wrong, the reference solution, and how each level is verified. A reviewer reads this first.

title = "Length bonus reward hack"
license = "Apache-2.0"
created_date = "2026-08-19"
domain = "reward"
tags = ["slime", "grpo", "reward-hacking", "length-bonus", "aime"]

[submitter]
name = "chenyinghao"

[model]
name = "Qwen3-4B"

[dataset]
name = "DAPO-Math-17k (RL) + ReTool-SFT (SFT)"

[training]
framework = "slime"
algo = "grpo"
script = "examples/retool/retool_qwen3_4b_rl.sh"

[environment]
image = "slime-v0.3.2"

# The recipe needs 8 GPUs: ray start --num-gpus 8 + --actor-num-gpus-per-node 8.
[environment.env]
CUDA_VISIBLE_DEVICES = "0,1,2,3,4,5,6,7"
NVIDIA_VISIBLE_DEVICES = "0,1,2,3,4,5,6,7"

[outcome]
failure_type = "reward_hacking"
failure_reason = "The ground-truth verdict is unhealthy because generate_with_retool.reward_func adds an unconditional length bonus (5 * min(response_length / rollout_max_response_len, 1.0)) to the math_dapo exact-match score for every sample; logged eval/aime rises to +3.84 by step 99 while decontaminated held-out accuracy falls to about 8.4%, so GRPO advantages reward longer generations instead of improving accuracy"

Metadata and the recorded outcome. The environment stanza fixes the image and GPUs every stage runs in.

#!/usr/bin/env bash
# slime_retool_length_hack: what to prepare before training.
#
# inside reward_func. The healthy/unhealthy pair must differ only by the injected defect: if the
# data, weights, or ref checkpoint differ anywhere, comparing the two curves no longer shows anything.
# Before changing this file, check whether the other copy needs the same change.
#
# Runs in a separate prepare-stage container: root, `slime-v0.3.2` image, **the only stage with
# network**, and $AUTORLBENCH_WORKSPACE, $AUTORLBENCH_MODELS, and $AUTORLBENCH_DATASETS are all writable.
#
# Four assets: three downloaded, one converted on the spot. Directory names are set by the
# DATA_DIR/MODEL_DIR guards in examples/retool/retool_qwen3_4b_rl.sh (both default to $PWD, which is
# /workspace in the container), and **must not change by a single character**: otherwise the guard
# misses, the script silently passes a nonexistent path to --hf-checkpoint, and the error surfaces in
# megatron, far from the cause.
set -euo pipefail

# --- Training set -----------------------------------------------------------
# --prompt-data + --input-key prompt + --label-key label read a jsonl.
autorlbench_fetch_hf \
    --repo zhuzilin/dapo-math-17k \
    --type dataset \
    --revision 2e65612930298bde4c5d58fd97b3f23a483aaff9 \
    --allow dapo-math-17k.jsonl \
    --expect dapo-math-17k.jsonl=cc9c39c2… \
    --expect-size dapo-math-17k.jsonl=10490834 \
    --dest "$AUTORLBENCH_WORKSPACE/zhuzilin_dapo-math-17k"

… two more pinned downloads (the AIME-2024 eval set and the SFT checkpoint,
  each with sha256 pins) and one on-the-spot HF → torch_dist conversion …

The only stage with network: four pinned assets, hash-checked.

…
async def reward_func(args, sample, **kwargs):
    """Tool call reward function using math_dapo as primary reward model"""
    if not isinstance(sample, Sample):
        raise TypeError("Sample must be an instance of Sample class.")

    # Build complete solution string
    solution_str = sample.prompt + sample.response

    # Get ground truth answer - label is a string, not a dict
    ground_truth = sample.label if sample.label is not None else ""

    # Get tool call count as num_turns
    num_turns = getattr(sample, "tool_call_count", 0)

    # use \\boxed{...} answer
    result = math_dapo_compute_score(solution_str, ground_truth, strict_box_verify=True)

    # encourage model to call tools
    if result["score"] < 0:
        tool_call_reward = (num_turns - 2) / 2 * 0.1
        result["score"] = min(-0.6, result["score"] + tool_call_reward)

    length_ratio = min(sample.response_length / args.rollout_max_response_len, 1.0)
    result["score"] = result["score"] + 5 * length_ratio

    if result["pred"] is None:
        result["pred"] = ""

    return result

The defect, highlighted: two lines add an unconditional length bonus to every sample's score, on top of exact-match. A long wrong answer can outscore a short correct one. The solution is to delete them.

# RL training run diagnosis

Review the staged post-training experiment and determine whether it is healthy
or unhealthy. Treat the workspace as a historical snapshot: inspect its source,
effective configuration, launch scripts, data processing, reward logic, logs,
tracked metrics, and frozen evaluation evidence. Do not assume that an unusual
setting or implementation is causal without connecting it to observed behavior.

Use held-out evaluation evidence together with rollout, truncation, runtime,
optimization, and training signals. Training reward alone does not establish
held-out health. A truncated log is not by itself proof of a defect.

Do not launch a full training run, use the network, or modify the historical
snapshot. The dataset, held-out set, model, training-step budget, and evaluation
interval are fixed.

## Deliverable

Write one concise UTF-8 diagnosis to:

```text
$AUTORLBENCH_RESULTS/agent_answer.md
```

State the verdict, identify the concrete root cause or causes, and explain the
causal chain using specific workspace evidence. Distinguish root causes from
symptoms. Do not propose or implement a fix, and do not write other deliverable
files.

Issue level: diagnose and write agent_answer.md. No fix, no training, no network. The hidden oracle grades the verdict, root cause, and evidence.

# Improve the staged experiment

Review the staged post-training experiment and determine whether it is healthy
or unhealthy. Treat the workspace as a snapshot you may repair: inspect its
source, effective configuration, launch scripts, data processing, reward logic,
logs, tracked metrics, and frozen evaluation evidence. …

## Your job

If you find a defect, fix it in the staged workspace at `/workspace`. **Do not
launch training.** Do not start the experiment, in the foreground or
background, and do not use `nohup`, `&`, `tmux`, `screen`, or any detach
mechanism. The system runs the experiment itself after you finish (see below);
anything you launch is wasted and scores nothing. Cheap, non-training checks
are fine.

When you are done, submit a report at `/workspace/report.md`. It must
summarize the root cause, every file you changed, and the expected effect.
A missing or empty `/workspace/report.md` scores 0.

## What the system will run

After you submit the report, the system deletes everything under
`$AUTORLBENCH_RESULTS/experiment_logs`, then re-runs the experiment in your
repaired workspace as a foreground process, using the launch script:

    examples/retool/retool_qwen3_4b_rl.sh

with LOG_DIR=$AUTORLBENCH_RESULTS/experiment_logs, WANDB_MODE=offline.

The effective run must use:

- HF checkpoint `/workspace/font-info_qwen3-4b-sft`
- train data `/workspace/zhuzilin_dapo-math-17k/dapo-math-17k.jsonl`
- validation data `/workspace/zhuzilin_aime-2024/aime-2024.jsonl`
- `--num-rollout 70`
- `--eval-interval 10`

**Make sure your repaired launch script actually produces that run.** The
staged `examples/retool/retool_qwen3_4b_rl.sh` bakes `--num-rollout 3000`
directly into the training invocation with no environment override, and the
graded rerun passes no budget override either — if you leave 3000 in place,
the rerun blows its time budget and scores 0. Edit the script to the required
70.

… two more traps: log placement is the launch script's own job (WANDB_DIR in
the environment does not reach the training process), and the offline W&B
key placeholder must not crash the script …

A task-specific grading script then reads **only this fresh run's** files
under `$AUTORLBENCH_RESULTS/experiment_logs` and scores the key metrics; logs
written anywhere else are ignored. Do not copy, edit, or fabricate log
metrics. Otherwise the run is scored 0.

Solution level: repair the workspace and submit report.md — the harness reruns training itself, then grades only that fresh run. Three traps, including a baked-in --num-rollout 3000.

{
  "schema_version": 1,
  "command": "mkdir -p \"$AUTORLBENCH_RESULTS/experiment_logs\" && bash examples/retool/retool_qwen3_4b_rl.sh",
  "env": {
    "LOG_DIR": "$AUTORLBENCH_RESULTS/experiment_logs",
    "WANDB_MODE": "offline",
    "WANDB_DIR": "$AUTORLBENCH_RESULTS/experiment_logs"
  },
  "timeout_s": 64800,
  "log_roots": ["experiment_logs"]
}

How the rerun is launched: offline W&B, logs collected under $AUTORLBENCH_RESULTS/experiment_logs, an 18-hour budget.

{
  "schema_version": 2,
  "task_id": "slime_retool_length_hack",
  "metric_key": "eval/aime",
  "final_step": 69,
  "historical_final_baseline": "3.580407918294271",
  "required_final_score": "-0.55",
  "required_final_score_max": "0.5",
  "score_mode": "best",
  "log_roots": [
    "experiment_logs"
  ]
}

How the fresh run is graded: the best eval/aime must fall in [-0.55, 0.5]. The defective run ends at +3.84, far outside; the repaired reference peaks at -0.42.

# Frozen historical eval/train trace reconstructed from W&B history.
# Cutoff: train step 100. Held-out eval interval is 10; last eval is step 99.
# Metric eval/aime is the mean of custom_rm reward_func scores on AIME 2024.

2026-08-20 10:47:13 step:0  - eval/aime:0.8218 eval/aime/response_len/mean:4057.9  eval/aime/truncated_ratio:0.0083
2026-08-20 11:44:30 step:9  - eval/aime:0.9175 eval/aime/response_len/mean:4275.2  eval/aime/truncated_ratio:0.0083
2026-08-20 12:55:17 step:19 - eval/aime:1.2919 eval/aime/response_len/mean:5439.0  eval/aime/truncated_ratio:0.0125
2026-08-20 13:53:57 step:29 - eval/aime:1.9868 eval/aime/response_len/mean:7654.4  eval/aime/truncated_ratio:0.0542
2026-08-20 14:51:14 step:39 - eval/aime:2.6410 eval/aime/response_len/mean:9833.1  eval/aime/truncated_ratio:0.2104
2026-08-20 15:51:28 step:49 - eval/aime:3.1166 eval/aime/response_len/mean:11603.1 eval/aime/truncated_ratio:0.3667
2026-08-20 17:04:08 step:59 - eval/aime:3.4148 eval/aime/response_len/mean:12862.4 eval/aime/truncated_ratio:0.5063
2026-08-20 18:21:44 step:69 - eval/aime:3.5804 eval/aime/response_len/mean:13863.1 eval/aime/truncated_ratio:0.6208
2026-08-20 19:46:32 step:79 - eval/aime:3.7282 eval/aime/response_len/mean:14749.7 eval/aime/truncated_ratio:0.7375
2026-08-20 21:11:55 step:89 - eval/aime:3.8174 eval/aime/response_len/mean:15215.8 eval/aime/truncated_ratio:0.8313
2026-08-20 22:27:46 step:99 - eval/aime:3.8411 eval/aime/response_len/mean:15310.3 eval/aime/truncated_ratio:0.8250

2026-08-20 10:50:14 step:0   - rollout/response_len/mean:1984.6  rollout/truncated_ratio:0.0039
2026-08-20 13:56:46 step:30  - rollout/response_len/mean:5340.1  rollout/truncated_ratio:0.0117
2026-08-20 15:56:34 step:50  - rollout/response_len/mean:9410.8  rollout/truncated_ratio:0.1797
2026-08-20 17:08:17 step:60  - rollout/response_len/mean:11134.1 rollout/truncated_ratio:0.3711
2026-08-20 18:29:39 step:70  - rollout/response_len/mean:13065.2 rollout/truncated_ratio:0.5195
2026-08-20 22:32:43 step:100 - rollout/response_len/mean:15159.2 rollout/truncated_ratio:0.7852

The smoking gun the agent starts from: eval/aime rises to +3.84 — outside the [-1, 1] base-score range — while answers balloon to 15.3k tokens and 82.5% get truncated.

{
  "task_id": "slime_retool",
  "scoring_note": "Historical AIME-2024 evaluation metrics frozen from the original run through train step 100 (last eval at step 99; eval_interval=10).",
  "metrics": {
    "eval_interval": 10,
    "metric_key": "eval/aime",
    "historical_cutoff_train_step": 100,
    "eval_points": [
      { "step": 0,  "eval_score": 0.8218, "response_len_mean": 4057.9,  "truncate_ratio": 0.0083 },
      …
      { "step": 49, "eval_score": 3.1166, "response_len_mean": 11603.1, "truncate_ratio": 0.3667 },
      { "step": 99, "eval_score": 3.8411, "response_len_mean": 15310.3, "truncate_ratio": 0.8250 }
    ],
    "train_points": [
      { "step": 0,   "response_len_mean": 1984.6,  "truncate_ratio": 0.0039 },
      …
      { "step": 100, "response_len_mean": 15159.2, "truncate_ratio": 0.7852 }
    ]
  }
}

The same trace as structured metrics, frozen at train step 100. Score climbs while held-out accuracy falls: reward hacking, not learning.