Post-Training Speech Enhancement Language Models with Perceptual Rewards

Interspeech 2026

F. Berdoz, L. A. Lanzendörfer, A. Asonitis, R. Wattenhofer

ETH Zurich, Switzerland

speech-enhancementreinforcement-learningperceptual-rewardslanguage-models

Abstract

Speech enhancement language models achieve strong results when trained on discrete audio tokens, but their optimization relies on token-level cross-entropy rather than the perceptual metrics used for evaluation. We introduce a post-training stage for autoregressive speech enhancement language models using Group Sequence Policy Optimization (GSPO) with multi-metric perceptual rewards. Our method directly optimizes non-differentiable quality metrics (DNSMOS, WER, and UTMOS) as reward signals, without learned surrogates or offline preference pairs. Applied to two autoregressive base models, UniSE and GenSE, our approach achieves state-of-the-art results on the DNS2020 benchmark. A human evaluation ablation further shows that the composite multi-metric reward is preferred over any single-metric variant, confirming that multi-reward optimization avoids the reward hacking observed with single-metric training.

Overview

Speech enhancement (SE) recovers clean speech from recordings degraded by noise, reverberation, or packet loss. The strongest recent systems treat SE as a sequence-to-sequence problem, tokenizing audio into discrete units and training an autoregressive language model to map noisy tokens to clean ones. These models inherit the strengths of large-scale pretraining, but they stop at supervised fine-tuning. They are optimized with token-level cross-entropy, even though they are judged at test time by perceptual metrics that cross-entropy does not capture.

This work closes that gap. We add a reinforcement-learning post-training stage that optimizes the perceptual metrics directly, completing the same pretrain, supervised fine-tuning, and RL pipeline that has become standard for text LLMs.

The train-eval gap

Lower cross-entropy on token sequences does not guarantee higher perceptual quality. A model can predict the most likely tokens on average and still produce audio that listeners dislike. Earlier attempts to optimize perceptual quality leaned on learned surrogates, such as a discriminator trained to approximate PESQ, or on offline preference pairs. Both add approximation error and a separate construction step. We instead optimize the real evaluation metrics online, with no surrogate model and no offline data.

Method: GSPO with perceptual rewards

We post-train two existing autoregressive SE models, UniSE and GenSE, with Group Sequence Policy Optimization (GSPO). For each noisy input, the policy samples several complete enhanced outputs, decodes each one to a waveform, scores it with a reward function, and updates the model with a sequence-level clipped policy gradient. GSPO needs no critic network and works on full-sequence likelihoods rather than per-token ratios, which keeps training stable.

The reward is the part that matters most. We combine three complementary metrics with equal weight.

  • DNSMOS for overall perceptual quality.
  • 1 − WER (computed with Whisper-Large-V3) for content preservation and intelligibility.
  • UTMOS for naturalness.

Composing these metrics is a deliberate defense against reward hacking. Optimizing a single metric lets the model inflate that one score while quietly degrading the others. Requiring simultaneous improvement across quality, intelligibility, and naturalness leaves far less room to cheat.

Training is lightweight. We use 20k paired noisy-clean clips of five seconds at 16 kHz, sample four outputs per input, and train each model on a single NVIDIA RTX 6000. At inference, the post-trained model emits one output at the same cost as the base model.

Results

GSPO post-training improves every metric for every base model on both benchmarks we evaluate.

Log-mel spectrogram comparison showing noisy input, base-model output, and post-trained output for UniSE and GenSE.
Figure 1: Qualitative comparison after composite-reward post-training applied to existing autoregressive SE models. Post-training sharpens harmonic structure and suppresses residual noise without introducing spectral artifacts.

DNS2020 blind test set

ModelReverb (OVRL)No reverb (OVRL)Real recordings (OVRL)
UniSE3.433.433.26
UniSE + GSPO3.493.483.37
GenSE3.163.412.60
GenSE + GSPO3.533.553.22

GenSE + GSPO gives the best quality on synthetic conditions and UniSE + GSPO the best on real recordings. Both post-trained models surpass every baseline, including recent LM-based systems such as AnyEnhance and LLaSE-G1.

DNS5 blind test set

The gains grow on the harder DNS5 set, measured with personalized DNSMOS (pDNSMOS). GenSE + GSPO lifts overall quality (pOVRL) from 3.41 to 4.45 on the headset track and from 2.96 to 4.36 on the speakerphone track. Crucially, pDNSMOS is not part of the reward function, so the improvement reflects genuine generalization rather than chasing the trained objective.

Human evaluation: does the composite reward actually help?

Objective metrics can be gamed, so we ran a pairwise listening test with 21 raters comparing the supervised baseline against four GSPO variants. We rank every variant with Bradley-Terry Elo ratings computed from all head-to-head matchups.

Reward variantElo rating
Composite (ours)1571
WER-only1541
UTMOS-only1494
SFT baseline1476
DNSMOS-only1335

The composite reward wins. It is preferred over the supervised baseline in 71% of comparisons and ranks first overall. The single-metric results are more revealing. DNSMOS-only optimization lands dead last, below the untouched baseline. This is a textbook case of reward hacking, where the model learns noise-suppression artifacts that inflate DNSMOS while sounding worse to people. UTMOS-only stays near parity because UTMOS is trained on human ratings and is harder to fool. Only the composite reward improves on every axis at once.

Key takeaway: Reinforcement learning is a third axis for improving speech enhancement language models, alongside architecture and data. Optimizing the metrics we actually evaluate, as long as we optimize several of them at once, closes the train-eval gap and produces audio that listeners genuinely prefer.

Citation

@inproceedings{berdoz2026post,
  author = {Berdoz, F. and Lanzend{\"o}rfer, L. A. and Asonitis, A. and Wattenhofer, R.},
  title = {{Post-Training Speech Enhancement Language Models with Perceptual Rewards}},
  booktitle = {{Annual Conference of the International Speech Communication Association (Interspeech)}},
  year = {2026}
}