Storyteller's Shane Friedman explains the CTC aligner behind immersive reading

Why it matters

Audible and Spotify offer features that connect reading with audiobook playback inside their services. Storyteller instead writes synchronization data into open EPUB files that readers can keep, copy and use with compatible software.

Shane Friedman has detailed the CTC-based forced-alignment system behind Storyteller, his self-hosted reading platform for synchronizing user-supplied ebooks and audiobooks.

Friedman published the explanation in an August 30 technical post. Storyteller is testing v3 releases, while the alignment algorithm remains the project's technical center.

The work grew from a problem Friedman had as a reader. He wanted to listen during a commute, read the same book later and keep his place across formats. His answer started as a Python script that accepted one audiobook and one ebook, then produced an EPUB containing synchronized audio metadata. Friedman writes that he entered the field of forced alignment without knowing it, armed with what he now describes as "clunky, nested while loops."

Storyteller has since expanded into a self-hosted web platform, Android and iOS apps and KOReader plugins. Storyteller lists macOS, watchOS and tvOS apps as upcoming. It keeps libraries on hardware controlled by the reader and produces EPUB 3 files using the standard's Media Overlays system, which lets compatible software highlight text as the narration plays.

Friedman maintains Storyteller alongside his work in software engineering. His professional background includes building content-management systems and dashboards for Amazon Music and YouTube Music and developer tooling at Google, according to his profile at Handle with Care Collective, a worker-owned product consultancy he co-founded.

Books make terrible clean-room datasets

A demo can align a clean transcript with clean audio. A commercial book pair is messier.

The acknowledgements may open the ebook and close the audiobook. An appendix may never be narrated. A narrator may skip a quotation, describe an illustration that has no matching prose or say "listening to this book" where the printed edition says "reading this book." Even chapter metadata can be missing from the audio files.

Those differences matter because a naive aligner expects both inputs to move through the same material in roughly the same order. Earlier tools such as syncabook required readers to identify corresponding ebook and audiobook chapters manually. Storyteller is designed to find those relationships itself.

Friedman's CTC pipeline uses Meta's Massively Multilingual Speech model to turn audio into Connectionist Temporal Classification emissions. Each emission represents probabilities for characters within a 20-millisecond audio frame. Storyteller greedily decodes those frames into an approximate stream of letters.

The resulting text is deliberately ugly. It does not need to serve as a readable transcript. Storyteller only needs enough matching material to locate each chapter in the audio.

Storyteller first uses CTC emissions to find approximate chapter boundaries, then applies forced alignment to locate the ebook text in the audiobook and write timing data into EPUB Media Overlays.

Once it has located a chapter, a CTC Viterbi algorithm finds the most likely path through the character probabilities. That path reveals when letters, words and sentences begin and end. Storyteller can then write those timings into the EPUB's Media Overlays data instead of storing the synchronization inside a proprietary player.

Friedman has also exposed the method through the --ctc flag in stalign, the standalone command-line tool and npm library he released on March 5. Developers can use the alignment pipeline without deploying the full library server.

The ownership trade

Storyteller's independence comes with requirements. Its alignment documentation says readers must provide DRM-free or otherwise accessible ebook and audiobook files. Storyteller cannot process or remove DRM from locked books.

Friedman has kept that complexity because local control is part of the product. In a 2024 post, he wrote that Storyteller was self-hosted so readers could retain control of their media. Storyteller later became available through PikaPods for people who wanted managed hosting without moving the software into a centralized Storyteller service.

Audible launched Read & Listen in February, while Spotify launched Page Match on February 5. Both keep the synchronized experience inside their own catalogs and apps. Read & Listen offers word-level highlighting for eligible audiobook and Kindle ebook pairs, while Page Match uses a phone camera to connect a printed or digital page to a position in Spotify's audiobook.

Storyteller uses a different distribution model. The reader brings the catalog, runs the software and receives a standards-compliant file that can move between compatible applications. Friedman does not control the bookstore, the audiobook service or the resulting library. The synchronization data lives in the EPUB rather than being tied to Storyteller's player.

The alignment algorithm is the technical center of that model. Updated interfaces and library tools may make Storyteller easier to use, but accurate automatic matching determines whether the platform can handle real ebook and audiobook pairs rather than a carefully prepared demo. Books disagree with their audiobooks in dozens of small ways. The Storyteller repository was created in 2021, and Friedman has spent the intervening years teaching software to accept that fact.