SDPO: learn from your own correct attempts

Self-Distillation Policy Optimization (arXiv:2601.20802) samples a question several times and turns the attempts that succeed into teachers for the others. The teacher is the model itself reading the question with a correct attempt appended. The student reads the question alone. The loss pulls the student's next-token distributions toward the teacher's on the student's own attempt. The paper reports that SDPO reaches GRPO's accuracy with fewer samples.

What it does #

The harness samples each question several times through Reef and scores every attempt. It reports each attempt with its score and its place in the grid. The recipe waits for the whole grid and trains on it in one step.

How Reef implements it #

SDPO has three pipeline stages (Training pipeline). SDPORecord2Record groups the reports of one grid. SDPOTasks turns each rollout's report into a task. SDPOBatches batches one complete grid.

The teacher's request is the question with the first correct attempt by another rollout appended, in the reference implementation's words. An attempt whose question no other rollout got right keeps the plain request and a sample weight of 0. It stays in the step's mean and has no target. A grid whose attempts came from different policy versions is dropped and listed in the pipeline's status. A teacher sequence over max_teacher_tokens fails the step.

The sdpo loss family is a thin family on the Slime backend's distillation base (Loss families) and its defaults are the reference's. The teacher is a copy of the weights that moves 5% toward the policy after every step. The loss is the Jensen-Shannon divergence over the student's top 100 tokens plus one bucket for the rest of the vocabulary. The student picks those tokens in one forward before the step, so the trainer needs zero dropout. Each token's loss is weighted by the capped ratio between the policy and the rollout engine's log-probs.

The report contract #

A report references one attempt and carries its score and its place in the grid. teacher_context is optional feedback for the teacher and the example leaves it empty.

{
  "references": ["<receipt of the attempt>"],
  "score": 1.0,
  "metadata": {"step": 3, "group": 12, "rollout": 5, "teacher_context": ""}
}

Configuration #

The deployment's top-level pipeline section configures the three stages. Each stage reads only its own config. A later stage reads the values it needs from an earlier stage:

• record2record (SDPORecord2Record) sets the grid. SDPOTasks reads the grid from it.

• record2task (SDPOTasks) sets the teacher's tokenizer and windows. SDPOBatches reads them from it.

• task2batch (SDPOBatches) sets what the teacher reads.

recipe:
  implementation: recipes.sdpo.recipe:SDPORecipe
  config:
    max-staleness: 0
pipeline:
  record2record:
    config:
      groups_per_step: 32
      rollouts_per_group: 8
  record2task:
    config:
      tokenizer_path: /models/Qwen3-8B
      max_teacher_tokens: 18944
  task2batch:
    config:
      include_environment_feedback: false
training:
  config:
    global_batch_size: 256           # groups_per_step x rollouts_per_group

Reef checks each stage's config when it loads the recipe. A stage key under recipe.config fails the deployment, and the error names the key's place in pipeline.

SDPOTasks also takes the sample assembly's keys (accept_multi_turn_policy_samples, realign_threshold and scaffold_tolerance). One batch is always one grid. The batch size, Slime's --global-batch-size, must not exceed the grid's rollouts (groups_per_step x rollouts_per_group); with a larger one the scenario's pipeline fails to build. Set it equal to the grid to train one optimizer step per grid, as the example does.

The Slime driver takes --loss-type custom_loss and --use-rollout-logprobs and --disable-compute-advantages-and-returns. The family adds its own flags:

The sampled reverse-KL mode also accepts score centering for training/inference mismatch. See Score centering for the required loss options and sampler top-K capture. The default JSD over top-K plus a tail bucket does not support it.

Run the example #

The example trains Qwen3-8B on the Chemistry split of SciKnowEval on four GPUs. Each step samples 32 questions eight times and the test split is scored every five steps. The example's README describes the protocol.

cd recipes/sdpo/examples/sciknoweval
hf download Qwen/Qwen3-8B --local-dir ~/models/Qwen3-8B
./run.sh

Results #

One run of 100 steps. Test accuracy goes from 41.2% to 74.4% at step 75 and ends at 71.7%. The rollouts stay diverse and a question seen a second time is answered no better than the rest of the grid.

Related guides #

• Inference and feedback quickstart: learn the request, receipt, and report workflow.

• Train model weights from agent feedback: set up the GPU stack and inspect published updates.

• Loss families: how a family such as sdpo plugs into the Slime backend, and the distillation base it is built on.