When to Plan, When to Polish: Noise Level as a Granularity Axis for Diffusion Language Models
Researchers propose a new approach to training language models called Noise Dependent Granularity Control (NDGC). This method uses the level of noise in the model's training process as a cue to switch between coarse and fine-grained denoising. At high noise levels, NDGC focuses on forming coherent groups of tokens to support early meaning commitment, while at low noise levels it refines individual tokens. The authors claim that this approach allows for planning-like coarse-to
Researchers propose a new approach to training language models called Noise Dependent Granularity Control (NDGC). This method uses the level of noise in the model's training process as a cue to switch between coarse and fine-grained denoising. At high noise levels, NDGC focuses on forming coherent groups of tokens to support early meaning commitment, while at low noise levels it refines individual tokens. The authors claim that this approach allows for planning-like coarse-to-fine denoising without the need for explicit planners or hierarchical architectures.
---
Why it matters: This matters because it could lead to more efficient and effective training of language models, particularly those used in generation tasks. By allowing for earlier skeleton formation and better ordered recovery, NDGC has the potential to improve the quality and coherence of generated text.
Source: https://arxiv.org/abs/2606.21802
This article was originally published at: https://arxiv.org/abs/2606.21802