Researchers have developed a method to audit self-improvement in language models. They tested three rounds of LoRA self-training on the Qwen3-8B model against a frozen control and found seven measurement failures that inverted reported findings when the control was absent. These failures included standard practices like using a single greedy decode or expansion statistic. The researchers replaced these methods with a more robust approach, detecting nothing on held-out replicates and unchanged under multiple-testing rules. They found that external distillation improved problems the base model rarely reached while self-training did not, but had inconclusive evidence for problems the base model never reached. --- Why it matters: This matters to AI researchers because it highlights the importance of robust auditing methods in evaluating self-improvement in language models. The findings suggest that current practices may be flawed and can lead to incorrect conclusions about a model's capabilities. Source: https://arxiv.org/abs/2608.20290