ProMode

A zero-shot prosody model that predicts pitch, duration and energy from text plus a short reference clip, then makes downstream TTS sound more natural.

INTERSPEECH 2025 Eray Eren, Q. Liu, H. Kim, P. Garrido, A. Alwan
arXiv PDF Original demo page
GigaSpeechtraining corpus
F0 + energypredicted features
7 systemscompared below

Prosody carries much of what makes speech sound human: emotion, emphasis, and a speaker’s individual habits. ProMode is a stand-alone model that maps text to prosodic features such as F0 and energy, so it can be dropped into a downstream system like TTS rather than being locked inside one.

The encoder takes partially masked acoustic features together with time-aligned text and produces a fixed-length prosodic embedding. The decoder then reconstructs the acoustics in the masked region from that embedding plus the unmasked text. Trained on GigaSpeech, it improves F0 and energy prediction over state-of-the-art style encoders at several levels of granularity, and listeners preferred its prosody when the predictions were fed into a TTS system.

ProMode model architecture: encoder maps masked acoustics and aligned text to a prosody embedding; decoder reconstructs the masked acoustics.
ProMode architecture. The encoder produces a fixed-length prosody embedding; the decoder predicts acoustics in the masked region.

Predicting F0

Comparison against state-of-the-art style encoders for F0 prediction on prosody continuation, given a reference prosody.

Predicted versus reference F0 contour, example 1.
Predicted versus reference F0 contour, example 2.

Predicting energy

The same comparison for energy. Values are on a log2 scale, shifted to start at 0.

Predicted versus reference energy contour, example 1.
Predicted versus reference energy contour, example 2.

Listen: prosody in a downstream TTS system

Each GigaSpeech test sentence is split in half. The first half is the audio prompt the prosody embedding is taken from; the transcript of the second half conditions the decoder, whose predicted pitch is passed to the TTS system. So every row asks the same question: given this speaker’s prosody, how should the rest of the sentence sound?

What to listen for: whether the rhythm and pitch movement continue naturally from the reference, rather than flattening out or drifting into a generic reading voice. ProMode also yields lower WER and higher UTMOS than the baselines.

Utterance Reference prompt audio the prosody embedding is taken from Ground Truth the real continuation ProMode (Ours) FluentSpeech Wav2Vec2-SER* StyleTTS2* Emotion2Vec*
1 Rorty suggests that we allow our political strategy to reflect the common values of the people that make up the culture.
2 She's always expected to hang out and be nice to industry people.
3 And that is so key for children to understand math concepts.
4 The world on its own unaided by the describing activities of human beings cannot, end quote.

Scroll the table sideways to reach every system; the utterance column stays pinned. Starting one clip pauses the others. Systems marked * were adapted for this comparison.

Citation

@inproceedings{eren2025promode,
  title     = {ProMode: A Speech Prosody Model Conditioned on Acoustic and Textual Inputs},
  author    = {Eren, Eray and Liu, Qingju and Kim, Hyeongwoo and Garrido, Pablo and Alwan, Abeer},
  booktitle = {Proc. INTERSPEECH},
  year      = {2025}
}