Skip to content

Support incremental surface updates #283

@jacobsimionato

Description

@jacobsimionato

We want add the ability for the LLM to use a tool to efficiently make small changes to an existing surface by simply supplying a list of new widgets which will be "upserted" into the tree.

Current behavior

Currently, we support adding and updating surfaces via addOrUpdateSurface tool, and deleting surfaces via deleteSurface tool. Currently, the 'update' action actually replaces an entire surface, so if you want to add one more widget to a surface with many existing widgets, or else change the parameters of some widget, you need to make an 'update' call and reoutput the entire surface with the changes applied.

New behavior

We want to redesign the 'update' functionality to that instead of replacing an entire surface, it just adds and replaces widgets, preserving all existing widgets, then automatically cleans up orphaned widgets at the end.

This will be implemented by extending the existing addOrUpdateSurface tool so that it has three possible actions - add, update, replace. addOrUpdateSurface in genui_manager.dart will be updated to implement the third case. It should branch on the action field.

SurfaceUpdated class in Gen UI manager

At the Gen UI manager level, SurfaceUpdated which is used to notify other parts of the system about changed surfaces will be renamed to SurfaceChanged, and reported for both update and replace actions. Every part of the codebase that refers to SurfaceUpdated must be updated to reflect the rename.

replace action

The existing update action will be renamed to replace. The functionality will remain the same.

update action

This should be a newly implemented action that updates individual widgets in the tree. When the update action is used, the logic in genui_manager.dart will first get the UiDefinition for the current surface state and make a deep copy of it. It will then iterate over all the widgets in the new surface and replace the ones in the existing surface. There will be a TODO in the code to implement pruning of orphaned widgets in the tree after this, which is more complex to implement.

In the schema, in the description for addOrUpdateSurface, in the description for the action field, add a section which clearly defines the add, update and replace actions, including their contract and when they should be used. E.g. update should be preferred when some of the existing widget tree content should be preserved, while replace should be used when the entire surface content should be replaced.

Other changes

  • The root widget ID will remain a required field, and for update actions, it must be set to the same value as for the original surface. There should be an assertion that fails if this isn't the case. There should be a note in the schema description which explains this in a concise and clear manner.
  • Update the system prompts in main.dart in the travel_app and the simple_chat apps to reference both update and replace actions instead of just update.

Metadata

Metadata

Labels

P2An issue we'd like to fix eventually, but it's not being actively worked on.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions