Descript(opens in a brand new window) is an AI-native video editor constructed round a easy thought: in case you can edit textual content, you need to have the ability to edit video. Since Descript’s early days, AI has powered each side of the product: transcription, enhancing, audio cleanup, and more and more complicated artistic workflows. They’ve constructed on OpenAI for years, utilizing Whisper for transcription and GPT collection fashions inside their co-editor Underlord.
Translation rapidly emerged as a high-impact use case. Historically, translating video has been sluggish and costly, requiring language specialists to handle tasks, produce rote translations, deal with high quality management, and generate corresponding audio. LLMs dramatically compress that workflow, making high-quality translation at scale potential.
Captions and dubbing each require semantic constancy: the interpretation should protect the unique which means. However length adherence performs a unique function in every. For captions, it is a nice-to-have. For dubbing, it is important, as a result of if translated speech runs too lengthy or too quick, it’s going to sound unnatural even when the which means is appropriate.
To handle this, Descript redesigned its translation pipeline utilizing OpenAI reasoning fashions to optimize for semantic constancy and length adherence throughout technology, not after. Within the first 30 days after rollout, exports of translated movies with dubbing elevated 15%, and length adherence improved by 13 to 43 proportion factors, relying on the language.
“Dubbing is an more and more widespread use case for Descript, so we’re constructing methods to do it in batch for firms that need to translate and lip-sync whole libraries,” mentioned Laura Burkhauser, CEO.
Translation was one in every of Descript’s earliest and most requested options. They began with captions-only translation, which labored effectively—however many customers wished to go additional and have spoken audio (dubbing) within the goal language.
Nonetheless, one subject stored surfacing: dubbed audio didn’t all the time sound correct. “Most likely the primary grievance we heard was that the tempo of the speech was unnatural within the translated language,” mentioned Aleks Mistratov, Head of AI Product at Descript.
The issue got here right down to the truth that totally different languages take totally different quantities of time to precise the identical thought. Descript noticed, as an illustration, that on common German is a “longer” language than English. To suit into mounted video segments, translated speech typically needed to be artificially sped up or slowed down. “You’d find yourself with one thing that gave the impression of chipmunks, or a sleepy big,” Mistratov defined.
On this case, the German audio would both need to be sped up unnaturally, or the interpretation would should be rewritten to suit the time price range.
Customers had been left with two choices: manually retime the audio section by section, or rewrite the interpretation itself to make it match. Each approaches required deep timeline edits and, typically, near-native fluency within the goal language. It was tedious for creators, and have become a blocker to scaling the characteristic to giant enterprise localization tasks.
The workforce had a transparent concept of what it might take to make dubbing work. The system would wish to not solely optimize for semantic which means, but in addition concentrate on timing constraints. When translating from English into German, for instance, the mannequin would wish to know methods to use fewer phrases or simplify the idea, so the dubbed audio would stay pure.
Earlier approaches optimized semantic constancy first and tried to appropriate timing afterward. The translations had been typically semantically appropriate, however they routinely missed the length constraints, and the general high quality nonetheless wasn’t ok.
“We ran incremental exams, not even producing something, simply asking the mannequin to output the variety of syllables in a bit of textual content,” Mistratov mentioned. “Earlier fashions merely weren’t good at that.”
Dependable syllable counting turned out to be crucial. If the mannequin couldn’t persistently calculate syllables, it couldn’t reliably goal a selected length window.
GPT‑5 collection fashions introduced a degree of reasoning consistency that earlier fashions lacked, particularly on duties like syllable counting and constraint monitoring. With that enchancment, Descript redesigned its translation and dubbing pipeline.
First, Descript’s system breaks the transcript into chunks, guided by sentence boundaries, pure pauses, and talking patterns within the authentic recording. Every chunk maintains semantic continuity, however is sufficiently small to cause about as a timing unit.
From there, the mannequin calculates the variety of syllables within the chunk. Utilizing language-specific speaking-rate assumptions, the system estimates what number of syllables the translated chunk ought to goal to protect pure pacing (“length adherence”). The immediate asks the mannequin to optimize for each length adherence and which means preservation. Surrounding chunks are handed in as context in order that the mannequin maintains semantic coherence throughout segments.
The workforce evaluated a number of configurations to stability length adherence, semantic constancy, latency, and price. The chosen setup delivered robust constraint-following at manufacturing pace, enabling high-volume translation with out handbook retiming. The result’s a translation pipeline the place pacing is handled as a first-class variable as an alternative of one thing corrected after the actual fact.
To develop the acceptance standards for evals, the workforce ran listening exams: they generated translated audio samples and adjusted the playback pace in small increments, asking customers to price when speech turned unnatural.
“Something that was slowed down by 10%, or sped up by 20%, typically nonetheless sounded pure,” Mistratov mentioned. Past this vary, speech turned too distorted.
Earlier methods carried out poorly by that measure. Relying on the language, solely 40% to 60% of segments fell inside the acceptable pacing window. With the redesigned pipeline, that quantity elevated from 40%–60% to between 73% and 83%, relying on language.
The workforce additionally evaluated semantic constancy utilizing a separate model-as-judge score on a scale starting from 1 (“fully totally different”) to five (“semantically equal”). For dubbing, they determined to just accept a decrease semantic threshold than for caption-only translation, the place length constraints are irrelevant. Even with that tradeoff, 85.5% of segments had been rated a 4 or 5 out of 5 for semantic adherence.
The consequence was a system that would stability two competing constraints—timing and which means—with measurable confidence. And since each metrics had been automated, Descript is ready to constantly consider new mannequin releases and immediate variations in opposition to the identical benchmarks.
As translation strikes from single movies to giant content material libraries, Descript is constructing extra management into how translations are tuned, together with the power to prioritize stricter semantic constancy when wanted.
Translation inside Descript is just one layer of a broader multimodal system. Translated textual content feeds into speech technology, which then drives lip sync and last video rendering.
Enhancements on the textual content layer make pure pacing potential, however the general expertise additionally will depend on how effectively the audio mannequin preserves tone, cadence, and nonverbal traits of speech. That’s the place the workforce sees the subsequent frontier.
“Lots of what is going on to enhance translation output is making the pipeline extra multimodal: incorporating audio, video, and textual content collectively when deciding methods to translate,” mentioned Mistratov. “That ought to higher preserve the nonverbal traits of speech, like tone and emphasis, and protect much more of the unique supply.”
For Descript, stronger reasoning fashions made the complexity of dubbing tractable. By crossing the edge the place fashions might reliably stability tradeoffs between pacing and which means, translation turned one thing the workforce might systematically enhance, and deploy at scale.