How to incrementally apply coalescing behavior when streaming chat

[ericstj]

The user problem is how to represent streaming updates without re-implementing the coalescing behavior we have in ToChatResponse

extensions/src/Libraries/Microsoft.Extensions.AI.Abstractions/ChatCompletion/ChatResponseExtensions.cs

Lines 126 to 136 in 192782e

	/// <summary>Combines <see cref="ChatResponseUpdate"/> instances into a single <see cref="ChatResponse"/>.</summary>
	/// <param name="updates">The updates to be combined.</param>
	/// <returns>The combined <see cref="ChatResponse"/>.</returns>
	/// <exception cref="ArgumentNullException"><paramref name="updates"/> is <see langword="null"/>.</exception>
	/// <remarks>
	/// As part of combining <paramref name="updates"/> into a single <see cref="ChatResponse"/>, the method will attempt to reconstruct
	/// <see cref="ChatMessage"/> instances. This includes using <see cref="ChatResponseUpdate.MessageId"/> to determine
	/// message boundaries, as well as coalescing contiguous <see cref="AIContent"/> items where applicable, e.g. multiple
	/// <see cref="TextContent"/> instances in a row may be combined into a single <see cref="TextContent"/>.
	/// </remarks>
	public static ChatResponse ToChatResponse(

Our template does a very basic thing of just concatenating all text

extensions/src/ProjectTemplates/Microsoft.Extensions.AI.Templates/src/ChatWithCustomData/ChatWithCustomData-CSharp.Web/Components/Pages/Chat/Chat.razor

Line 74 in 192782e

responseText.Text += update.Text;

Here's a sample of how I adapt that for also handling images -- https://github.com/ericstj/imageGeneratorSample/blob/bed6c588a84489a9b652613a8e8f2ac24a20c2f2/imageGeneratorSample.Web/Components/Pages/Chat/Chat.razor#L98-L169

Related discussion: #6749 (comment)
Here I proposed ApplyUpdate API that would allow folks to incrementally coalesce as content arrives.

While this satisfies the need, it does have the risk of excessive complexity - given N updates, each time iterating over all content to try to coalesce, and again to shift data over removed items - the existing implementation of ToChatResponse tries to avoid this complexity but incremental APIs would probably have it effectively when called for each update. Even though the result of incremental coalescing would be smaller - especially smaller in the case of contiguous items - there's still a risk in worst case.

It feels like a sorting problem. Perhaps that's how to solve it. If we had a data structure to store updates in that behaved like a sorted list with all the coalescing rules, and exposed ChatMessages.

@stephentoub