Support streaming generated UI surfaces widget-by-widget in breadth-first order

[jacobsimionato]

In order to reduce perceived latency, we should support streaming of UI responses.

Behavior

• User sends query as usual
• Developer integrates Gen UI as usual, with some minor modifications to hook up streamed responses or function calls
• Behind the scenes, Gen UI should progressively build the widget tree, so new elements appear one-by-one as they stream back.
• We should have this functionality behind some configuration flag that the developers sets initially. In the future, this may become the default approach.

Technical approach

• We need some permissive JSON parser for partial responses that can close brackets etc.
• We should consider how to handle IDs references, which must be able to be temporarily broken. Maybe we can have an optional 'Placeholder' widget which is rendered when IDs refer to widgets that haven't been loaded yet. Alternatively, we can have children refer to their parents by ID, but that makes it difficult to represent ordering in lists etc.
• The schema we already have should be close what is necessary to support streamed results because the widget list is flat
• We may need to switch from a function call integration to structured output integration due to Gemini limitations - see below
• Ideally, we'd still provide the JSON schema and have the LLM require conformance
• We may need to do some tweaks to make sure the fields are outputted in the right order, or make the schema not care. For example, we have a "root" field with the root widget ID which is adjacent to the actual widgets, so it is possible that we don't know the root widget initially and therefore may not be able to start rendering. We could somehow enforce field order (I think @gspencergoog had an idea on how to achieve this?) or else change the format, e.g. have the first element implicitly be the root, have root=true as a field of the root element etc.

Gemini streaming function calls limitation

Once complication here is that we currently integrate with Gemini API via function calls / tool use, but it seems that Gemini does not support streamed function calls - see google-gemini/cookbook#47.

Hopefully, we can work around this by giving the developer the option of using structured output instead.

It sounds like OpenAI and Anthropic models don't have this limitation, so we could try them too.

[gspencergoog]

We need some permissive JSON parser for partial responses that can close brackets etc.

I actually don't think we do. We just need to define the format in such a way that it can be expressed using small objects and a way to do incremental changes, and then use JSONL to stream them. Having a permissive parser is only one part of doing this: you have to end up with not only a valid object, but a fully specified layout. For instance, sending just a widget id:

{
	"id": "my_widget"
}

doesn't do anything useful if your object needs at least a type to be able to be rendered, and even then having only part of the parameters for an object often doesn't make it renderable, and will probably result in a bunch of jumping around even if it is, since later updates will include the missing settings that change the alignment, size, etc.

Better is to have it send whole objects in each line, and progressively build the UI:

{ "id": "my_widget", "type": "Column", "children": []}
{ "id": "my_title", "type": "Text", "value": "My Title"}
{ "id": "my_widget", "type": "Column", "children": ["my_title"]}

Or, as a patch protocol:

{ "id": "my_widget", "type": "Column", "children": []}
{ "id": "my_title", "type": "Text", "value": "My Title"}
{ "op": "add_children", "path": "/my_widget", "children": ["my_title"]}