Support for `<img>` in SMIL

[HadrienGardeur]

Section

https://w3c.github.io/epub-specs/epub34/authoring/#sec-media-overlays

Describe the problem

EPUB 3.x uses a subset of SMIL 3.0 for Media Overlays.

A number of specialized libraries have adopted EPUB with Media Overlays as a replacement for Daisy with synchronized text and audio. In Northern Europe, some of these libraries have produced a number of accessible comics over the years using images and pre-recorded audio.
Due to the lack of support for <img> in SMIL for EPUB, most of these libraries are using proprietary formats right now and haven't been able to industrialize their processes.

Describe the fix or new feature you propose

In addition to <text> and <audio>, the EPUB 3.4 specification should also reference <img>.

These references to <img> would use a spatial fragment (#xywh) in their src rather than the region attribute available in SMIL.

<par>
  <img src="page1.avif#xywh=percent:7,7,60,17" />
  <audio src="page1.ogg" clipBegin="23s" clipEnd="28s" />
</par>

This would also allow three-way references between image, audio and text.

[sueneu]

This would be great for making comics accessible. I would love to see it.

Is there enough adoption of SMIL throughout reading systems to support making this change? I thought it is unevenly implemented at this time.

[iherman]

Can we reasonably assume that we will get two implementations for CR? "These libraries" may not bother changing and/or at least submitting an implementation report... But that is just my Staff Contact self talking :-)

[HadrienGardeur]

If not, this could be "incubated" as long as we don't outright reject EPUB files using <img> in SMIL.

In Readium we'll implement support for this in at least two of our codebases in Q1 (Swift and Go, they don't share a single line of code).

[sueneu]

Reaching out to a friend who might know of another implementation. I’ll post it here if he does.

[llemeurfr]

Adoption can only come from a good partnership between a significant set of accessible content producers, and innovative reading system developers.
Today, large producers are Daisy members, boosted by automatic alignment AI-tools. As for innovative reading system developers, we have names.

I agree with Hadrien that using spatial fragments rather than SMIL region (an id/idref indirection) is much simpler re. technical implementation.

[HadrienGardeur]

Here's an extended example based on the work that I've done for Readium and more specifically the second panel of the first page in this example:

<par epub:type="panel">
  <img src="page1.jpg#xywh=42,694,296,297" />
  <audio src="audio/page1-panel2-description.mp3" />
  <text src="page1.html#panel2-description" />
</par>

Where page1.html#panel2-description would be:

<p id="panel2-description">Cumin looks sadder than the rest of the witches and shakily addresses Pepper.</p>

[mattgarrish]

If not, this could be "incubated" as long as we don't outright reject EPUB files using <img> in SMIL.

Getting the schema epubcheck uses updated to allow img probably wouldn't be that hard. We've allowed metadata in the past prior to it being standardized in the specifications.

It'd be easy to put together a CG note making the case for img, plus what restrictions there might be on its use, and then see how it goes first before committing this to the spec.

[HadrienGardeur]

I've reached out to the maintainers of an automated alignment tool (Storyteller: https://storyteller-platform.gitlab.io/storyteller/) that also provides mobile (Android and iOS) and Web reading apps and they're willing to support this feature.

If we also count Thorium Mobile, we could have two implementations available for the usual rec track process.

[mattgarrish]

Maybe, but that's aiming for a low bar. We have a history of rushing changes that never pan out.

[HadrienGardeur]

Fair point. Let's discuss this in a future call (tomorrow?) but I'm not against the idea of a CG note if we can avoid the epubcheck trap.

[iherman]

This was discussed during the pmwg meeting on 15 January 2026.

View the transcript

<img> in SMIL - w3c/epub-specs#2883

@Hadrien: SMIL allows references to text, audio, and video
… currently in epub we only use text. It references IDs. We also use the audio element without a time fragment

<ivan> strictly speaking, 'img' and the others are defined as "alias" of 'ref'

@Hadrien: a number of specialized libraries have experimented with narrated comics using pre recorded audio
… the current spec doesn't provide a good way to do this, so the current experiments are in proprietary formats
… I have been investigating how much work it would be to support this in EPUB
… we would need a way to specify a fragment of an image
… we can either use regions or spacial fragments, which I think is much easier
… we already use this in a few other pages
… there is interest for this, there is no format to do this either in DAISY or EPUB
… it is a pretty small gap to allow this feature in EPUB

@ivan: for the record, the SMIL specification predates the specification for media fragments
… I agree that in 2025 we should use media fragments
… are there any prospects of real epub readers that would implement this
… we need two independent implementations for a new feature

duga: I think there were RS that were offering implementations

Hadrien: there is support in readium to support this, once it is in the Readium toolkit it is pretty easy to support
… Thorium will support this
… I have asked the maintainer of StoryTeller to support this also
… it is mostly used by people who do this on their own, who offer this to their library patrons
… I cannot commit on the timeline for Storyteller but Thorium will support it this year

@ivan: Thorium is clear, Storyteller, from the official point of view, the AC would ask for an EPUB Reader like Thorium
… to really implement and use this
… Colibrio would be fine, it doesn't need to a big player

Hadrien: I expect specialized libraries to use this

@ivan: if I have an EPUB that uses this, I should be able to read it in another system

duga: what are the implications for existing implementations for Read Aloud?
… could a publisher just reference portions of an image and not have the audio overlay? The image would then be the synced media?
… is that a potential use here? are we worried about it

Hadrien: I don't think it would be a good thing to overload text
… wrapping an image in text is a bad thing for accessibility
… the benefit of having image, is opening the door to image based content
… it also opens the door to something highly visual in nature, plus audio, and a textural element
… it opens the door to more things without negatively impacting the current system
… media support in EPUB remains more limited than we wish
… even with the current take on media overlays things could be better with RS support

GeorgeK: …read aloud means to me, taking the text and playing it into the system, different than audio overlays
… if we have a fixed image, and there are four audio clips in parallel with that image
… the media fragment shifts focus on the image, would you have something like
… two people in an image, a focus on one person, then a shift to the other person like a conversation

Hadrien: that is one way you could implement it
… currently they can zoom into a panel and play the audio from that panel

CharlesL: about implementation concerns, I don't think we need explicitly two reading system implementations
… just two implementations of the technology so a tool would be fine
… if you had a screen reader enabled, and you have the textural part overlayed on audio, I'm trying to understand

GeorgeK: I think the SMIL implementation would play and the screen reader would be silent
… if you pause the screen reader

duga: when I said Read Aloud I meant reading the included audio
… to Hadrien 's point, we have a flow for children's books
… the publisher adds audio to existing images/media
… the reading system then takes that and plays it
… is it expected now that Reading Systems must synchronize the audio?
… are we breaking children's books by enabling this?

Hadrien: I don't think this is very likely. Producing an audio layer is much more work than including text
… the kind of org that will produce this media overlay is one that cares deeply about accessibility
… the cost is much higher to produce them

duga: these are not text to speech books, but with media overlays

DaleRogers: I wonder if there would be a smooth transition from one image fragment to another?

Hadrien: this is not about controlling visual presentation. A lot of movement can be hard for people with certain sensitivities

<Zakim> wendyreid, you wanted to react to DaleRogers

wendreid: for example, a lot of RS have settings for page turn behavior. I think this would fall in that same class
… especially on mobile, where you can have multiple page turns actions

@ivan: we may be getting into areas that are not covered by this proposal
… the original SMIL spec refers only to the ref element
… the spec doesn't say anything about transitions
… it is always a trick discussion about the AC and how they accept implementation reports
… I think beyond the legal terminology for the process, they ask
… if I create a book with this feature, am I stuck with only one reading system?
… if we decide to put this in the spec, we mark it as an at risk feature
… so if we don't get enough implementation we can easily remove it at the end without triggering reviews

mgarrish: you want to see two similar implementations, like two Reading Systems and two authoring systems
… is this incubation material? Where are we going with this?
… is it at the point we want to implement this?
… what are the knock on implications? I'd like to be sure of this before we add this to the implementation at the end of the process
… maybe the at risk implementation or a cg note would be the way to do this

GeorgeK: what is the relationship of this technique to other ways to make comics?
… it seems like an alternative to other developments for comics

Hadrien: this is directly a response to a number of specialized libraries have implemented especially in Northern Europe

<gman> Hadrien the proprietary use cases in Scandinavia are comics only?

Hadrien: they have produced files that do this, and everyone does it slightly differently
… I look at what people are using, and what people need
… this is quite specialized, mostly about adaptive content, but it is missing
… this is different than adding something no one asked for

GeorgeK: I have seen the US Library of Congress dabbling with this too

Hadrien: this is one way we can support this

@ivan: I don't see this as an incubation matter because we don't have to create new vocabulary
… we are just allowing a tiny bit of what is already in SMIL in EPUB
… it is defined already, we just allow it here

CharlesL: I get your point about two implementations on one side or another, and the
… Readium implementation could go out to multiple readers
… in a comic, could you have it read each panel in the correct order
… if you click a Read button in the RS, would it do that? Is this a potential use case

Hadrien: yes, you could have a panel with a textural and/or an audio equivalent

gman: another potential use case is language acquisition/application. Having two audio setups in a Manga

gman: you could switch between the translations
… this kind of use case could be studied a bit further

Hadrien: I think this is different. You are talking about having two images in two languages?

gman: SMIL doesn't support multiple track?

Hadrien: not as we use it, no

wendyreid: that gets into something we are also discussing, parallelized content
… we may have an open Issue about that where we do want to talk about language acquisition

<gautierchomel> Parallels content is a discussion at #2829

wendyreid: this is where text to speech might be better than SMIL so they can choose their narration voice
… SMIL is fixed, referencing specific content
… it sounds to me like we want to keep exploring this
… it might not be a huge change to the spec
… the major question is if we have the right amount of implementations
… I'm concerned about industry uptake, if it will be supported by most major reading systems
… we need to make sure that there is more than one reading platform available
… just for the sake of user choice

duga: I'm not convinced, I'm still worried about breaking existing workflows.
… there is nothing tying this to specific content, and I would need more reassurance that people
… won't start making children's books that would break on current reading systems

wendreid: we have a greenlight to keep studying this. Can we get sample files to study?

Hadrien: Nota has public examples. We might get the files in JSON

@ivan: I'd like to explore what it means editiorialy, can Hadrien do a draft PR for the author and RS specs?

Hadrien: I can do this, but not immediately, maybe in February

<gman> Hadrien on one of these sleepless nights

wendyreid: we will carry over our last topic, selectors, next week

---