Need a mechanism to provide instructions to the LLM on which widget to use in the catalogue and when

[abdallahshaban557]

Is your feature request related to a problem? Please describe.
I want to have a mechanism to provide the LLM with context on which widget to use for specific scenarios, so that I have confidence that the LLM always picks the widgets that work best for my users in the context of their interactions.

Describe the solution you'd like
Possibly the addition of a description field or a similar field when adding widgets to the catalogue that can then be provided as context for the LLM on which widget to use and when.

Describe alternatives you've considered
N/A

Additional context
N/A

[jacobsimionato]

There are two ways that the developer can do this, and based on our experience, we have a rough idea on the separation of concerns, though a lot of it depends on how reusable you want your catalog widget to be.

1. CatalogItem schema description field

Describes how this catalog item should be used with only reference to itself, and other catalog items or data that it is tightly coupled to and will always be used with e.g. "Should be used for displaying tabular data with 10-20 rows where there is a benefit to full text search of rows" or "should only be used within a TableSet widget", if the TableSet widget is intentionally tightly coupled.

Example - see the top-level description:

final _schema = S.object(
  description:
      'A chip used to choose from a set of mutually exclusive '
      'options. This *must* be placed inside an InputGroup.',
  properties: {
    'chipLabel': S.string(
      description:
          'The title of the filter chip e.g. "budget" or "activity type" '
          'etc',
    ),
    'options': S.list(
      description:
          '''The list of options that the user can choose from. There should be at least three of these.''',
      items: S.string(),
    ),
    'iconName': S.string(
      description: 'An icon to display on the left of the chip.',
    ),
  },
  required: ['chipLabel', 'options'],
);

2. System prompt

Describes the overall flow of the conversation, the interaction design of it. Links specific conversation steps to specific UI elements. Links specific data types and data tool results to UI treatments. Explains how to handle events etc. The system prompt is the glue that tells the LLM how Gen UI should be used in relation to the overall conversation and other parts of the system.

See the bottom of https://github.com/flutter/genui/blob/99891364cc812293890737ef60dbe3281625f272/examples/travel_app/lib/main.dart for an example of a "glue" system prompt

Formalizing this field

We can formalize this into some special-cased top level widget_description which would be a separate parameter on CatalogItem, but I personally think that right now, we should just keep using the 'description' field on the Schema that you pass to it, because they are functionally the same - if we had a purpose field, we'd just translate it to the schema description. At some point, we do want to revisit the whole CatalogItem API - perhaps do some neat codegen thing where you define a regular Flutter widget and then annotate it with some stuff (e.g. @GenUi(usage="something")) which we can then translate into a description etc. But until then, I'm not sure if the tweaks make a huge difference.