How might we help UGC authors confidently bring their stories to audio?
Dolby Labs X Purdue [2023] | Independent [2026]
(Spoiler: Not the way Dolby originally thought)
Overview
Problem
- 1.
Audiobook production is hard
Audiobooks are one of the fastest growing media formats, but their creation is time-consuming, expensive and inaccessible for independent novice authors.
- 2.
Authors don't know where to start
Independent book authors are gaining visibility through UGC platforms (BookTok/TikTok, Wattpad, Instagram), but face major challenges in converting their written works into accessible, high-quality audio formats.
Solution
AI voice generation tool
An AI-assisted audiobook creation tool that helps authors to:
- 1.Annotate scripts
- 2.Guide narration
- 3.Fix recording mistakes
- 4.Enhance audio
while preserving their creative control.
Users
Independent authors wanting to convert their written works into audiobooks.
Results
- 2023
User research: Users highlighted that the new workflow could reduce the time spent learning tools and setting up by at least 10 hours per project.
- 2026
AI evaluation: Built an AI audiobook narration tool with an 18-case test suite (three runs per case). Cases passing every run rose from 20% to 53% on the same 15 cases.
AI reliability: Fixed AI answers the app could not use. On six real book excerpts, answers that worked on the first try, without a retry, rose from 50% to 100% after a prompt fix.
Problem
01. Making an audiobook
Writer and content creator
She wants to turn her written stories into high-quality audiobooks and reach listeners on platforms like Audible.
But where does she start? Recording. Editing. Production. Too many tools to learn.
Why does it take so many apps just to make an audiobook?
02. So, should AI do the talking?
Dolby Labs’ initial brief was to explore AI voice generation for narrating audiobooks. But stakeholder conversations surfaced a different concern:
Why is AI chasing after our artistic careers?
The concerns: Voice ownership. Human expression. Artistic integrity.
03. Now, what if AI supported the creator instead?
AI shouldn’t replace the creator. It should support them.
An AI-assisted audiobook workflow that helps authors:
The goal: Make audiobook production easier while helping authors keep in control.
Product Strategy and Insights
02. Key Features
Drawing from the workflow of professional audiobook narrators, these are the features a UGC audiobook creator would need to easily produce high-quality audiobooks:
| JTBD | Feature |
|---|---|
| #1- When preparing to narrate a new audiobook, I want to create and organize detailed character and narration notes so I can stay consistent and confidently experiment with suggested interpretations before recording. | AI Annotation Guide [voice, expressions, nuances, pronunciation, tone, and pacing] |
| #2- When narrating the audiobook line by line, I want to view my character and performance notes alongside the script while recording and replaying takes. | Teleprompter-guided audiobook recording |
| #3- When recording narration, I want to quickly spot mistakes or weak performances and re-record only the affected lines, so I can improve quality efficiently. | Punch & roll narration |
| #4- When producing an audiobook without a professional studio, I want my recordings to sound studio-quality automatically, so I can meet professional standards without expensive equipment. | AI Vocal Studio |
03. Wireframes
JTBD #1AI Annotation Guide
Voice Recording
JTBD 2Teleprompter-Guided Audiobook Recording
JTBD 3 and 4A Punch and Roll Narration and AI Vocal Studio
SOLUTION: BUILDING THE AI PROTOTYPE
01. See it in action
01. Narration Studio & Features
Explore the narration workspace and its tools designed to help authors prepare and manage their audiobook recordings.
02. AI-Powered Narration Guidance
Review AI-generated narration recommendations, adjust emotional direction, and prepare to bring the manuscript to life.
03. Recording & Audio Enhancement
Record the narration using the guided teleprompter, refine mistakes with punch-and-roll, and compare the original recording with the studio-enhanced audio.
02. How the AI workflow works
Every step the author goes through, with what it is built with underneath. The AI suggests, the author decides.
- 01Upload
PDF, Word or text file, or the sample chapter
- 02Analyse
Claude suggests mood, characters and parts
- 03Decide
Author edits direction and parts, and locks parts
- 04Coach
Line tips on delivery, pauses, pronunciation, voice
- 05Hear
Example read aloud, one voice per character
- 06Record
Teleprompter, line notes, optional voice following
- 07Polish
Studio sound, punch-in re-records, undo
- pdfjs-dist, mammoth
Read PDF and Word files. Plain text works too.
- Own text parser
Cleans text, splits chapters and sentences.
- Uses an outside AI service. Anthropic API
claude-sonnet-5-5 reads the chapter.
- Validator and fallback
Up to 2 attempts, repairs, then a basic reading.
- Answer cache
In memory, with rate limits.
- Next.js 16, React 19
App screens and API routes.
- TypeScript, Tailwind 4
Code and styling.
- Uses an outside AI service. Anthropic API
Same model gives 2 to 3 tips per line.
- Refresh and update
Re-asks when the model skips parts.
- Uses an outside AI service. Anthropic API
Labels who speaks each line. Falls back to a guess.
- Uses an outside AI service. OpenAI TTS
gpt-4o-mini-tts, voice coral, American English.
- Web Speech API
Optional voice following in the browser.
- IndexedDB
Takes stay in the browser. Nothing uploaded.
- RNNoise (WASM)
Noise removal for Studio sound.
- Own audio chain
Gate, EQ, compression, loudness.
- Auto splice
Punch-in from line marks and pauses, with undo.
Product and engineering choices
- Author stays in control
Authors edit mood and parts, and lock a part so a refresh never overwrites it. 3 eval cases test this.
- Punch-in re-records
Re-record from line marks and pauses, with a lead-in, automatic splice and undo.
- Evaluation in four categories
Emotional ambiguity, beginner coaching, author override, output reliability. Includes real-text cases.
Data, guardrails and checks
- Where data goes
Scripts go to Anthropic and example text to OpenAI. Audio takes stay in the browser.
- Server guardrails
In-memory rate limits and an answer cache on the API routes.
- Tests and checks
57 automated tests with tsx, plus tsc and eslint.
- 01Upload
PDF, Word or text file, or the sample chapter
- pdfjs-dist, mammoth
Read PDF and Word files. Plain text works too.
- Own text parser
Cleans text, splits chapters and sentences.
- 02Analyse
Claude suggests mood, characters and parts
- Uses an outside AI service. Anthropic API
claude-sonnet-5-5 reads the chapter.
- Validator and fallback
Up to 2 attempts, repairs, then a basic reading.
- Answer cache
In memory, with rate limits.
- Uses an outside AI service.
- 03Decide
Author edits direction and parts, and locks parts
- Next.js 16, React 19
App screens and API routes.
- TypeScript, Tailwind 4
Code and styling.
- 04Coach
Line tips on delivery, pauses, pronunciation, voice
- Uses an outside AI service. Anthropic API
Same model gives 2 to 3 tips per line.
- Refresh and update
Re-asks when the model skips parts.
- Uses an outside AI service.
- 05Hear
Example read aloud, one voice per character
- Uses an outside AI service. Anthropic API
Labels who speaks each line. Falls back to a guess.
- Uses an outside AI service. OpenAI TTS
gpt-4o-mini-tts, voice coral, American English.
- Uses an outside AI service.
- 06Record
Teleprompter, line notes, optional voice following
- Web Speech API
Optional voice following in the browser.
- IndexedDB
Takes stay in the browser. Nothing uploaded.
- 07Polish
Studio sound, punch-in re-records, undo
- RNNoise (WASM)
Noise removal for Studio sound.
- Own audio chain
Gate, EQ, compression, loudness.
- Auto splice
Punch-in from line marks and pauses, with undo.
Product and engineering choices
- Author stays in control
Authors edit mood and parts, and lock a part so a refresh never overwrites it. 3 eval cases test this.
- Punch-in re-records
Re-record from line marks and pauses, with a lead-in, automatic splice and undo.
- Evaluation in four categories
Emotional ambiguity, beginner coaching, author override, output reliability. Includes real-text cases.
Data, guardrails and checks
- Where data goes
Scripts go to Anthropic and example text to OpenAI. Audio takes stay in the browser.
- Server guardrails
In-memory rate limits and an answer cache on the API routes.
- Tests and checks
57 automated tests with tsx, plus tsc and eslint.
03. One key engineering decision
Treat AI output as untrusted
Decision
Every analysis the model returns is checked before the app uses it. Small errors are repaired, failures are retried with the error fed back, and a basic reading is the last resort. The validated analysis is stored once, so coaching and Hear example build on the same output.
The challenge
On real manuscripts, Claude sometimes returned structured output with tags embedded inside fields. Downstream features could not read it.
What was built
- Validate and repair
A structured-output check, plus a repair step for minor errors.
- Retry with the error
Up to two attempts per chapter, with the specific validation error fed into the retry.
- Fail gracefully
A basic-reading fallback if both attempts fail.
- Prompt fix
A worked example of the expected structure, and an explicit ban on tags or XML inside fields.
- Reuse
Chapter analyses are stored so every feature builds on the same output.
What improved
Cases passing every run, Run 1 vs Run 4
Evaluation and Results
Cases passing every run went from 3 of 15 in Run 1 to 8 of the same 15 in Run 4 (9 of 18 including the 3 new real-excerpt cases).
01. How it is tested
18 test cases, each a small scenario with a defined good answer. Each case runs 3 times, because AI answers vary from run to run.
6 Reading the emotion of a line
5 Coaching a first-time narrator
3 Respecting the author's choices
4 Producing output the app can use
Rule checks (automatic)
Advice must be one or two sentences, a hard name must get a pronunciation tip, and so on.
Judge model (a second Claude call)
Decides whether the answer fits the scene and the author's chosen feeling. An answer the judge cannot decide on counts as a fail.
Three of the cases use real book excerpts, so the tests are not only invented examples.
03. Run history
Run 1
15 casesValid: 18/21 (86%)
Baseline
Run 2
15 casesValid: 18/21 (86%)
Fixed two scoring bugs. Shortened coach advice to two sentences and capped suggestions at three.
Run 3
15 casesValid: 20/21 (95%)
Told the model not to put code-like tags inside its answer fields, which had broken the app's reading of its output. Made the update step name the parts to return, and retry once if parts are skipped.
Run 4
18 casesValid: 39/39 (100%)
Added 3 real-excerpt cases. Gave the model a worked example of the exact answer format.
Run 4 has 18 cases, so compare it with runs 1 to 3 only on the 15 shared cases (8/15 passed every run).
| Case | Model calls per run |
|---|---|
| R1: full sample chapter analysis | 1 |
| R2: line coach, three selections | 3 |
| R3: update parts, then two advice refreshes | 3 |
| Total per run | 7 |
Run 1
15 casesValid: 18/21 (86%)
Baseline
Run 2
15 casesValid: 18/21 (86%)
Fixed two scoring bugs. Shortened coach advice to two sentences and capped suggestions at three.
Run 3
15 casesValid: 20/21 (95%)
Told the model not to put code-like tags inside its answer fields, which had broken the app's reading of its output. Made the update step name the parts to return, and retry once if parts are skipped.
Run 4
18 casesValid: 39/39 (100%)
Added 3 real-excerpt cases. Gave the model a worked example of the exact answer format.
| Run | Cases | Passed every run | Flaky | Failed every run | Valid first attempts | What changed |
|---|---|---|---|---|---|---|
| Run 1 | 15 | 3 | 7 | 5 | 18 of 21 | Baseline |
| Run 2 | 15 | 4 | 7 | 4 | 18 of 21 | Fixed two scoring bugs. Shortened coach advice to two sentences and capped suggestions at three. |
| Run 3 | 15 | 8 | 4 | 3 | 20 of 21 | Told the model not to put code-like tags inside its answer fields, which had broken the app's reading of its output. Made the update step name the parts to return, and retry once if parts are skipped. |
| Run 4 | 18 | 9 | 5 | 4 | 39 of 39 | Added 3 real-excerpt cases. Gave the model a worked example of the exact answer format. |
04. What still fails
Coaching a long, suspenseful passage
Tips keep breaking the length rule. Advice runs to three sentences where the limit is two.
Coaching a line with an invented fantasy name
The pronunciation tip for the name is always given. The judge rejects the other tips for adding details the text does not contain.
Refreshing advice after the author picks a tender, relieved feeling
The advice tells the narrator to sound like she is holding back tears. The judge reads that as sadness, not relief, so it counts as drifting from the author's choice.
Coaching a sharp line when the author chose a tender, relieved feeling
The same drift. Tips add held-back tears or keep a sharp edge.
The author's own text describes the moment as "a way of not crying", so the judge may be stricter than a human reader. That has not been checked by a person.
05. Limits
- The same model family judges its own output.
- No user testing has been done.
- Production monitoring: not built.
Learnings
- 01Human-in-the-loop evaluation
AI can misinterpret emotional nuance, making human review essential to ensure feedback aligns with the author’s intent.
- 02Author control is harder than it looks
Giving authors editing capabilities isn’t enough; AI coaching must consistently respect their corrections and preferences.
- 03Voice generation tools aren’t enough
Audiobook narration must capture the story’s emotional intensity and nuance, not just sound as human as possible.