FR: Structured output

[svdgoor]

Operating System

NestJS backend run on Windows

Environment (if applicable)

Node v22.14.0

Firebase SDK Version

=> Found "firebase@12.3.0"

Firebase SDK Product(s)

AI

Project Tooling

Not important, but our backend is NestJS, which is where we run live API

Detailed Problem Description

What you were trying to achieve.

Our system emits various events, some of which we would like to relay to our end users. To be able to triage this, we need the live API to emit structured output so we can distinguish between when it decides to and decides not to take action on a certain event it receives.

What actually happened.

The bidirectional API does not support Generating Structured Output, where the regular AI suite does.

Any error messages or unexpected behavior you observed.

N/a

Relevant log snippets or console output (if available).

N/a / already well defined by users of the GA structured output

Steps and code to reproduce issue

import { initializeApp } from "firebase/app";
import { 
  getAI, 
  getGenerativeModel, 
  getLiveGenerativeModel, 
  GoogleAIBackend, 
  Schema, 
  ResponseModality 
} from "firebase/ai";

// Replace with your Firebase config
const firebaseConfig = { /* ... */ };

// Initialize Firebase app + AI backend
const firebaseApp = initializeApp(firebaseConfig);
const ai = getAI(firebaseApp, { backend: new GoogleAIBackend() });

/**
 * --- Structured output (works in non-live APIs) ---
 */
const enumSchema = Schema.enumString({
  enum: ["drama", "comedy", "documentary"],
});

const structuredModel = getGenerativeModel(ai, {
  model: "gemini-2.5-flash",
  generationConfig: {
    responseMimeType: "text/x.enum",
    responseSchema: enumSchema,
  },
});

async function runStructuredExample() {
  let prompt = `The film aims to educate and inform viewers about real-life
  subjects, events, or people. It offers a factual record of a particular topic
  by combining interviews, historical footage, and narration.`;

  let result = await structuredModel.generateContent(prompt);
  console.log("Structured output:", result.response.text()); // → "documentary"
}

/**
 * --- Live API (works for streaming/audio, but no structured schemas yet) ---
 */
const liveModel = getLiveGenerativeModel(ai, {
  model: "gemini-2.0-flash-live-preview-04-09",
  generationConfig: {
    responseModalities: [ResponseModality.AUDIO],
    // ⚠ No responseSchema support here (yet)
  },
});

async function runLiveExample() {
  const session = await liveModel.connect();

  // Example: start an audio conversation
  const audioConversationController = await startAudioConversation(session);

  // Later you can stop it:
  // await audioConversationController.stop();
}

// Run them
await runStructuredExample();
await runLiveExample();

[svdgoor]

Currently, a workaround is for the AI to return a specific keyword in the response by prompt engineering it, but this is less sustainable and expandable than structured output

[svdgoor]

On a more philosophical note, I believe this to be key in the development of AI in the future, since it will become more and more important for the AI to be able to decide when to take action on certain contextual input; similar to how humans do. This applies especially much to the live API AI, since because of it being innately temporal, more unimportant context can occur.

[dlarocque]

@svdgoor Thanks for the feature request! Unfortunately structured output is not supported for Live models, so we can't offer this feature from the SDK.

For your use case, have you considered using Function Calling? If you describe your functions to the model, it will decide when it is appropriate to call those functions, and populate fields that you define in a schema. In your example, perhaps you could have the following function declarations for each type of movie, then the model would decide when it is appropriate to call each function, and it will populate a toolCall with the parameters you've defined:

const documentary: FunctionDeclaration = {
  name: "documentary",
  description: "A documentary given the users' description of an informative story.",
  parameters: Schema.object({
    properties: {
      title: Schema.string({
        description: "The title of the documentary."
      }),
      summary: Schema.string({
        description: "The summary of the documentary."
      }),
    }
  })
};

const drama: FunctionDeclaration = {
  name: "drama",
  description: "A drama given the users' description of a dramatic story.",
  parameters: Schema.object({
    properties: {
      title: Schema.string({
        description: "The title of the drama film."
      }),
      summary: Schema.string({
        description: "The summary of the drama film."
      }),
      characters: Schema.array({
        description: 'List of characters in the drama film',
        items: Schema.string()
      })
    }
  })
};

const comedy: FunctionDeclaration = {
  name: "comedy",
  description: "A comedy film given the users' description of a funny story",
  parameters: Schema.object({
    properties: {
      title: Schema.string({
        description: "The title of the comedy film."
      }),
      summary: Schema.string({
        description: "The summary of the comedy film."
      }),
      characters: Schema.array({
        description: 'List of characters in the comedy film',
        items: Schema.string()
      })
    }
  })
};

const filmCreationTool: FunctionDeclarationsTool = {
  functionDeclarations: [documentary, drama, comedy]
};

const ai = getAI(app);
const liveGenerativeModel = getLiveGenerativeModel(
  ai,
  {
    model: 'model-name',
    tools: [filmCreationTool]
  }
);

const liveSession = await liveGenerativeModel.connect();
await liveSession.sendTextRealtime('Can you generate a funny movie about a dog exploring the cosmos?');

const messages = liveSession.receive();
for await (const message of messages) {
  switch (message.type) {
    case "serverContent":
      // ...
      break;
    case "toolCall":
      const firstFunctionCall = message.functionCalls[0];
      switch (firstFunctionCall.name) {
        case 'comedy':
          const comedyMovieArgs = firstFunctionCall.args;
          if ('title' in comedyMovieArgs) {
            const title = comedyMovieArgs.title as string;
            console.log(`Comedy movie title: ${title}`); // "Space Lab"
          }
          if ('summary' in comedyMovieArgs) {
            const summary = comedyMovieArgs.summary as string;
            console.log(`Comedy movie summary: ${summary}`); // "A labrador sets out on a journey through the cosmos..."
          }
        case 'drama':
          // ...
        case 'documentary': 
          // ...
        default:
          // ...
      }
    case "toolCallCancellation":
      // ...
    default:
      // ...
  }
};

(This is just an example- I haven't tested this snippet live)