Comprehensive Speech Prosody Modeling
A task-agnostic prosody model with explicit control over pitch, energy and phoneme duration. It transfers prosody better in TTS and its embeddings also improve emotion recognition and word prominence detection.
There is still no standalone speech prosody model that can be dropped into any downstream task and that gives explicit control over pitch, energy and phoneme duration. This work builds one. It extends ProMode with a Perceiver I/O backbone that handles variable-length speech, a hybrid regression and flow-matching duration decoder that balances deterministic and probabilistic duration estimates, and a soft alignment loss plus summary rotary positional embeddings that keep every cross-attention layer temporally aligned.
The model is pre-trained on GigaSpeech and benchmarked against StyleTTS2 and FluentSpeech on GigaSpeech and VoxPopuli. It beats them in pre-training, in a downstream TTS task, and in listening tests. Its prosody features also improve two tasks it was never trained for: emotion recognition and word prominence detection.
Matching the shape of natural prosody
Point-wise metrics such as MAE reward predictions that regress to the mean, so they cannot tell whether a model keeps the spread and heavy tails of real prosody. The paper therefore reports the two-sample Kolmogorov-Smirnov (KS) statistic over the full distribution. FluentSpeech and StyleTTS2 underestimate the variance; the proposed model lands much closer to the ground truth, with relative KS improvements of 68% for consonant durations, 61% for vowel durations and 58% for F0 over the strongest baseline in each case.
The hybrid duration head is what makes this possible. At the chosen operating point (10% regression, 90% flow matching) it reaches 1.6 to 2.4 times lower KS than the regression-only ablation on both corpora, and matches the flow-only ablation while keeping a deterministic component.
Listen: prosody in a downstream TTS system
Each test utterance is split in half. The first half is the audio prompt; the model predicts duration and F0 for the second half’s transcript, replacing FluentSpeech’s built-in predictors, and a FluentSpeech model retrained on GigaSpeech-XL (FluentSpeech-Up) plus a BigVGAN vocoder synthesizes the speech. Every baseline goes through the same pipeline. These are the exact stimuli from the ABX listening tests.
What to listen for: which system’s rhythm, phrasing and pitch movement sound closest to the Ground Truth continuation. Listeners were asked exactly that question.
VoxPopuli (European Parliament speeches)
| Utterance | Ground Truth | Proposed (Ours) | StyleTTS2 | FluentSpeech | FluentSpeech-Up 10x data, 3x params |
|---|---|---|---|---|---|
| 1 Is truly the assumption of their common humanity. | |||||
| 2 But we have to make sure that we are ready to take a single action in the next similar case. | |||||
| 3 I was talking about legislative proposals. | |||||
| 4 Sharing information about firearms trafficking and terrorist financing and learning from our previous mistakes. | |||||
| 5 Sometimes also very selfish behaviour of the member states. | |||||
| 6 Government that is able to tackle the immediate crisis. | |||||
| 7 I urge the European Council to hold firm in rejecting British provocations, in asserting international law, and in defending the interests of Ireland and the EU. | |||||
| 8 How deeply audit companies are also involved. |
GigaSpeech (podcasts and YouTube)
| Utterance | Ground Truth | Proposed (Ours) | StyleTTS2 | FluentSpeech | FluentSpeech-Up 10x data, 3x params |
|---|---|---|---|---|---|
| 1 Any game with dice and numbered squares, the better their basic math skills get. | |||||
| 2 They attempt at disguising the truth is so clumsy that it even presents another possibility. | |||||
| 3 To disentangle our various inputs. | |||||
| 4 Made in partnership with radiotopia. | |||||
| 5 Is a whole lot of different stages. | |||||
| 6 From the personal computers of that day. | |||||
| 7 You've got to find what you love and that is as true for work as it is for your lovers. | |||||
| 8 Basically income needs to grow faster than debt grows. | |||||
| 9 Information about the books that I'm looking at, information about where I'm going because I'm typing it into Google Maps. | |||||
| 10 One of the most important bastions of the church. |
Scroll the tables sideways to reach every system; the utterance column stays pinned. Starting one clip pauses the others.
What listeners and metrics said
In downstream TTS the model gives 18.5% higher AutoPCP than the best baseline, plus better speaker similarity, so it transfers prosody better. It does not improve WER, likely because the baselines train with full spectrogram losses while this model only learns prosody and is not fine-tuned for TTS.
Two ABX surveys on Prolific (GigaSpeech and VoxPopuli, 40 native English listeners each) asked which system was prosodically closer to the ground truth. The model is significantly preferred over FluentSpeech and StyleTTS2 on both sets. Against FluentSpeech-Up, trained on 10× the data with 3× the parameters, it is significantly preferred on VoxPopuli and on par on GigaSpeech.
Beyond TTS
- Emotion recognition (IEMOCAP). Without ever seeing emotional speech or emotion labels, the prosody embeddings reach 0.595 mean CCC, above raw prosody (0.501) and eGeMAPS (0.518). Fused with self-supervised backbones they raise mean CCC for every backbone tested, most of all on valence and dominance.
- Word prominence. A single linear layer on the frozen embeddings reaches 0.856 accuracy, above the raw prosody features the model takes as input (0.843) and above BERT-base (0.802). The best system overall is a fusion, Whisper-large-v3 + this model, at 0.881.
Alignment analysis: what the soft alignment loss and S-RoPE do
Perceiver-style encoders summarize variable-length input with a fixed set of learned latents that are not tied to time, so their cross-attention can end up weakly structured. The soft alignment loss (SAL) and summary rotary positional embeddings (S-RoPE) push it toward a monotonic alignment. In each figure below the panels are, left to right: the full model, without SAL, and without S-RoPE.
A masking probe tests which part of the predicted F0 each latent controls. The full model reaches ordering ρ = 0.96 and localization κ = 0.46. Without SAL the ordering survives (ρ = 0.89) but the influence spreads out (κ = 0.21); without S-RoPE the ordering itself breaks down (ρ = 0.61). S-RoPE ties each latent to a position, and SAL keeps its influence local.
