2025 HTML‐in‐Canvas: Proposal and Progress

WEH 2025 - HTML-in-Canvas: Proposal and Progress

• GitHub issue: https://github.com/Igalia/webengineshackfest/issues/56
• URL: https://meet.jit.si/WEH2025-Canvas

Stephen explains slides

• Why?
  • HTML has structured elements, but with canvas you can draw anywhere. How do you combine the two?
    • E.g., plots
  • WebGL (e.g., draw text in WebGL textures)

Current solutions:

1. take a library that is specialized in writing in canvases.
2. use SVG. If needed, put HTML inside SVG as an external object (or something like that, very convoluted), and transform to a base64 image

There was a WICG proposal, but it didn't get traction. FormattedTyext was implemented in Chrome, but then dropped because of its complexity.

Then two proposals from Google:

1. extended text metrics 2.live DOM content in canvas

The HTML-in-canvas is built upon placeElement.

Questions

1. Why not implement CSS element() function in other browsers?

Shane: This proposal is the next generation of canvas formatting text?

Stephen: yes. It is the only currently active thing to improve the text in canvas

Shane: CanvasFormatted text word, sentence and grapheme, but not line break. This looks like even more of a black box in terms of getting line break. Can we programmatically get the line breaks from a string?

Stephen: The goal would be to leverage any solution in html and use that to solve the canvas problem. (scribe missed something)

Shane: But what about solving problems in svg and other places.

Stephen: Yeah the problem still exists for svg. The svg thing is interesting, historically this is how you would do it. You could do text in svg, and the put that in the canvas.

Shane: I have a historical package that does svg formatted test. Anne: I was curious what the ideas around hit testing are

Stephen: There are no ideas yet. If you don't disable hit testing, you get them anyway -- the events call through the canvas to the element. The problem is if something is rendered on top of the text. To do this, in the dom structure... once the subtrees have layout, we can do layout. The content might be drawn in the canvas yet. If you draw stuff on top of the element. IF the user can't see it, then you can't let them click/interact with the element. WebGL, who knows what. The solution is to address the 2d use case, the google map use case. The map is webgl. If you click on something, it is html content on top of webgl content. They want it in the webgl content. Explict API call which says which rectangles -- you can mark regions of screen or texture as hittest regions. It forces develpers to choose. If its highly interactive, it will be harder.

(scribe missed something)

Anne: do events expose, this is the coordinate on the canvas?

Stephen: I have to look, I can't quite reply the API. But what should this API look like?

Anne: It does solve a problem, about inifite stack of commands.

Stephen: It is only a problem if you have to read back the canvas. If you render into the canvas, and nothing is going back to script. But in the webgl case, it does got to go back. I have a list of things you can't draw.

Anne: Even an allow list, say you have a list of css properties, and the next day the css working group introduces a new value.

??: Back to hit testing, we need coordinated attributes in events.

Stephen: The right solution is to keep track of the transforms and ideally you can detect you clicked on the dom and that it is visible and send event to element you hit in the canvas.

??: But you may need coordinates for virtual dom that is under the canvas. Seems like you need client x and y.

Stephen: WebGL... you need to tag the texture, and it all happens in . The way it used to be done, render 1 by 1 screen size, web GPL whatever was drawn first under the pixel.

Sergio: any interest in other venders from this new spec, just chrome?

Stephen: chrome and microsoft. In terms of the rebooted proposals, we aren't at the point of trying to get other venders involved. Web developers REALLY want this. It is a question of whether we can solve technical, privacy and security problems, I think the demand will move it forward.

Sergio: I was thinking for web XR

Stephen: So the google pm on this is the web XR pm.

??: Gecko would like to replace "draw window" that is how we do tab previews, it's gecko internal.