Add app testing agent tools and prompts #9409

tagboola · 2025-10-30T14:05:00Z

Description

Add prompt and corresponding tools for running mobile automated tests with the Firebase App Testing agent behind the mcalpha experiment.

Scenarios Tested

Tested locally with the Gemini CLI.

Sample Commands

/apptesting:run_test --testDescription="Test the add to cart feature on pixel and samsung devices"

* Scaffolding for new appdistribution MCP tool. * Refactor business logic out of the appdistribution CLI so that it can be used by an MCP tool. * Wire new appdistribution tool up to the business logic. * Fix linting errors. * Update src/appdistribution/distribution.ts Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --------- Co-authored-by: Jamie Rothfeder <[email protected]> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Rename appdistribution directory to apptesting * Make variables consistent with directory rename. --------- Co-authored-by: Jamie Rothfeder <[email protected]>

…9280) * Use a datastructure to represent test devices rather than a string. * Update src/mcp/tools/apptesting/tests.ts Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update src/mcp/tools/apptesting/tests.ts Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Pretty --------- Co-authored-by: Jamie Rothfeder <[email protected]> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Add initial MCP prompt for running automated tests * Fix typos

* Create a on-off test and execute. * Can now create a on-off test. * Update src/appdistribution/client.ts Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update src/mcp/tools/apptesting/tests.ts Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * PR feedback * Separate test check in to a different tool so that gemini can orchestrate running and checking for completion. * Set the devices field to optional --------- Co-authored-by: Jamie Rothfeder <[email protected]> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

… description (#9322) * Use the same default device that's used in the Console * Update prompt to support generating a test case when there is no test description passed

* Add custom auto-enablement for app testing * Address gemini code assist comments * Fix intersection bug * Fix issues with test

gemini-code-assist · 2025-10-30T14:05:26Z

Summary of Changes

Hello @tagboola, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces comprehensive support for the Firebase App Testing agent within the Firebase CLI, specifically for AI-guided mobile app testing. It adds new client-side capabilities to define and execute AI-driven test cases, along with corresponding prompts and tools integrated into the Managed Code Platform (MCP). This feature, enabled under an experiment flag, allows developers to define detailed test steps with goals, hints, and success criteria, and then run these tests on specified devices, with the CLI providing status updates and links to results.

Highlights

New Feature: App Testing Agent Tools and Prompts: This PR introduces new tools and prompts to integrate the Firebase App Testing agent, enabling automated mobile tests, specifically AI-guided tests, through the Firebase CLI.
AIInstruction Type Definition: A new AIInstruction type and AIStep interface have been added to define the structure for AI-guided test cases, including fields for goals, hints, and success criteria.
App Distribution Client and Utility Enhancements: The AppDistributionClient now supports passing AIInstruction to createReleaseTest. New utility functions for uploading releases and awaiting test results have been extracted and added to src/appdistribution/distribution.ts.
Managed Code Platform (MCP) Integration: The new App Testing prompts and tools are integrated into the MCP framework, enabling their use under the mcalpha experiment flag. This includes new apptesting prompts (run_test) and tools (get_devices, run_tests, check_test).
App Testing Availability Checks: A new availability check (isAppTestingAvailable) has been added to determine if App Testing should be enabled based on the project type (Android, iOS, Flutter) and the status of the App Distribution API.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces the App Testing agent tools and prompts, enabling automated mobile app testing within the Firebase ecosystem. The changes include new commands, types, and utilities to facilitate test execution and result analysis. The review focuses on correctness and maintainability, with an emphasis on error handling and code organization.

gemini-code-assist · 2025-10-30T14:06:59Z

src/appdistribution/client.ts

+          aiInstructions: aiInstruction,
        },


The aiInstructions parameter is being passed directly from the request to the backend. Ensure that this data is properly validated and sanitized to prevent potential injection attacks or unexpected behavior on the backend. Consider adding validation logic to the client to enforce a specific structure or allowed values for the AI instructions.

src/appdistribution/distribution.ts

src/commands/appdistribution-distribute.ts

gemini-code-assist · 2025-10-30T14:06:59Z

src/mcp/prompts/apptesting/run_test.ts

+  Only include specific details in the test case if you are certain they will be available to the agent, otherwise the
+  agent will likely fail if it tries to follow specific guidance that doesn't work (e.g. click the 'Play' button but the
+  button isn't visible to the app testing agent). Do not include Android resource ids in the test case. Include


The text mentions that the agent will likely fail if it tries to follow specific guidance that doesn't work. It would be better to proactively validate the availability of the elements or actions before including them in the test case. This could involve using the get_devices tool to query the available devices and their capabilities, and then tailoring the test case accordingly.

src/mcp/tools/apptesting/tests.ts

src/appdistribution/distribution.ts

src/mcp/tools/apptesting/tests.ts

samedson

Added some questions - this is looking awesome!

samedson · 2025-11-03T19:13:31Z

src/commands/appdistribution-distribute.ts

-            );
-        }
-      }
+    const releaseTests = await Promise.all(releaseTestPromises);


Suggestion: It looks like this'll make a request in parallel for each promise. Could this cause problems if you have say 100 requests in flight at once?

There are some good libraries for this, eg https://www.npmjs.com/package/p-limit. It looks like src/database/import.ts already uses it.

It might, however, it's also unchanged for the original code. The change here was to refactor this code so that it could be used by both the command line tool and the MCP server.

Maybe we can create an issue to track separately?

samedson · 2025-11-03T19:16:57Z

src/mcp/prompts/apptesting/run_test.ts

+   this is not an Android app, instruct the user that this command can't be used with this app.
+
+2. **Make sure the user is logged in. No App Testing tools will work if the user is not logged in.**
+  a. Use the \`firebase_get_environment\` tool to verify that the user is logged in.


Not blocking: I think Joe is working on a way to unite how we do the "setup" part of prompts. I think an iteration of "sign in, get / create the user's project and app" is used in Crashlytics and other prompts

samedson · 2025-11-03T19:21:37Z

src/mcp/prompts/apptesting/run_test.ts

+Here are a list of prerequisite steps that must be completed before running a test.
+
+1. **Make sure this is an Android app**. The App Testing agent only works with Android apps. If 
+   this is not an Android app, instruct the user that this command can't be used with this app.


Small thing / not blocking at all: You don't need to include newlines, which might make future edits easier for you all

samedson · 2025-11-03T19:31:12Z

src/mcp/tools/apptesting/devices.ts

+export const get_devices = tool(
+  "apptesting",
+  {
+    name: "get_devices",


Is there a use case for this in the run_test prompt?

Separately, I'm wondering if there should be a combined apptesting_get_status tool that shows available devices and the current in-progress tests, or if this tool should be part of the CLI params instead of an MCP tool. Thinking about this in the spirit of reducing the number of MCP tools

samedson · 2025-11-03T19:33:19Z

src/mcp/tools/apptesting/tests.ts

+export const check_test = tool(
+  "apptesting",
+  {
+    name: "check_test",


Similar question here to get_devices - should the prompt be advising on using this tool to check the status of tests?

samedson · 2025-11-03T19:33:33Z

src/mcp/tools/apptesting/devices.ts

+    description:
+      "Get available devices that can be used for automated tests using the app testing agent",
+    inputSchema: z.object({
+      type: z.enum(["ANDROID"]).describe("The type of device"),


Should this be platform, or is type more idiomatic?

Should we omit this at this point? We're not supporting anything else here yet are we?

Jamie Rothfeder and others added 21 commits October 6, 2025 12:10

creation of feature branch

11b53b8

Merge branch 'master' into feature-branch/mcp/mobile-testing

7d8e857

Rename appdistribution directory to apptesting (#9268)

455d6d3

* Rename appdistribution directory to apptesting * Make variables consistent with directory rename. --------- Co-authored-by: Jamie Rothfeder <[email protected]>

Merge branch 'master' into feature-branch/mcp/mobile-testing

b26c98b

Merge branch 'master' into feature-branch/mcp/mobile-testing

62e3171

Merge branch 'master' into feature-branch/mcp/mobile-testing

26dd4f3

Merge branch 'master' into feature-branch/mcp/mobile-testing

c318b06

Add run_test prompt (#9292)

81c1a61

* Add initial MCP prompt for running automated tests * Fix typos

Merge branch 'master' into feature-branch/mcp/mobile-testing

75d279f

Merge branch 'master' into feature-branch/mcp/mobile-testing

9bbf17a

Use the same default device that's used in the Console (#9320)

973e8d7

Update prompt to support generating a test case when there is no test…

82e1724

… description (#9322) * Use the same default device that's used in the Console * Update prompt to support generating a test case when there is no test description passed

Merge branch 'master' into feature-branch/mcp/mobile-testing

c081d6d

Add custom auto-enablement for app testing (#9373)

380b001

* Add custom auto-enablement for app testing * Address gemini code assist comments * Fix intersection bug * Fix issues with test

Add get devices tool (#9387)

8a95353

Display link to results in the Firebase Console (#9406)

7356c41

Merge branch 'master' into feature-branch/mcp/mobile-testing

32c5a93

Place app testing tools behind an experiment

7f94828

github-project-automation bot added this to [Cloud] Extensions + Functions Oct 30, 2025

gemini-code-assist bot reviewed Oct 30, 2025

View reviewed changes

jrothfeder approved these changes Oct 30, 2025

View reviewed changes

src/appdistribution/distribution.ts Show resolved Hide resolved

src/mcp/tools/apptesting/tests.ts Outdated Show resolved Hide resolved

github-project-automation bot moved this to Approved [PR] in [Cloud] Extensions + Functions Oct 30, 2025

Address GCA comments

cb9c42b

tagboola requested review from fredzqm, joehan and samedson October 30, 2025 14:17

Explicitly set default devices

81a2734

samedson reviewed Nov 3, 2025

View reviewed changes

Add app testing agent tools and prompts #9409

Are you sure you want to change the base?

Add app testing agent tools and prompts #9409

Conversation

tagboola commented Oct 30, 2025

Description

Scenarios Tested

Sample Commands

Uh oh!

gemini-code-assist bot commented Oct 30, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

gemini-code-assist bot Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

samedson left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

samedson Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

samedson Nov 3, 2025 •

edited

Loading