feat(telemetry): add provider_type to computer initialization events#921
Merged
sarinali merged 6 commits intotrycua:mainfrom Mar 2, 2026
Merged
feat(telemetry): add provider_type to computer initialization events#921sarinali merged 6 commits intotrycua:mainfrom
sarinali merged 6 commits intotrycua:mainfrom
Conversation
Add provider_type property to telemetry events to enable tracking of computer provider usage breakdown (cloud, docker, lume, winsandbox, etc.). Python SDK: - Add provider_type to computer_initialized event TypeScript SDK: - Add vmProvider field to BaseComputerConfig interface - Add provider_type to computer_initialized and ts_computer_init events Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Contributor
|
@sarinali is attempting to deploy a commit to the Cua Team on Vercel. A member of the Team first needs to authorize it. |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Fix formatting issues flagged by CI: - prettier: 7 files (json, tsx, md, yml, ts) - isort: 4 Python files with incorrect import sorting - black: 1 Python file reformatted Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
asyncio.coroutine was removed in Python 3.11. Use AsyncMock from unittest.mock instead. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
f-trycua
approved these changes
Jan 28, 2026
- sequoia.yml: accept upstream's delay improvements and streamlined Terminal automation - telemetry/__init__.py: accept upstream's removal of Sentry imports - resources/unattended-sequoia.yml: accept deletion (moved to src/Resources/) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughAdds comprehensive telemetry instrumentation across Python and TypeScript implementations to track provider_type during ComputerAgent initialization and computer setup. Includes new test suite validating telemetry event emission with various configuration scenarios and mocked telemetry recording. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
libs/typescript/computer/src/computer/providers/base.ts (1)
38-56: Normalizeprovider_typeonce and reuse it across events.Line 42 and Line 55 duplicate provider normalization logic. Centralizing it avoids drift and makes fallback behavior consistent.
♻️ Proposed refactor
constructor(config: BaseComputerConfig) { this.name = config.name; this.osType = config.osType; this.vmProvider = config.vmProvider; + const providerType = + typeof config.vmProvider === 'string' && config.vmProvider.trim().length > 0 + ? config.vmProvider + : 'unknown'; this.telemetry = new Telemetry(); @@ this.telemetry.recordEvent('computer_initialized', { os: os.platform(), os_version: os.version(), node_version: process.version, - provider_type: config.vmProvider ?? 'unknown', + provider_type: providerType, }); @@ - if (config.vmProvider) argsProvided.push('vmProvider'); + if (providerType !== 'unknown') argsProvided.push('vmProvider'); @@ const initEventData: Record<string, unknown> = { os_type: config.osType, args_provided: argsProvided, - provider_type: config.vmProvider ?? 'unknown', + provider_type: providerType, };🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@libs/typescript/computer/src/computer/providers/base.ts` around lines 38 - 56, The code duplicates provider normalization (config.vmProvider ?? 'unknown') in telemetry.recordEvent and initEventData; create a single normalized variable (e.g., const providerType = config.vmProvider ?? 'unknown') near the start of the method and use providerType in telemetry.recordEvent, in argsProvided logic if needed, and in initEventData to ensure consistent fallback behavior and avoid drift (update references: telemetry.recordEvent, argsProvided pushes, and initEventData).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@libs/python/agent/tests/test_telemetry_events.py`:
- Around line 126-142: test_event_structure currently only asserts local
hardcoded dicts (expected_computer_tool_event / expected_function_tool_event)
which is tautological; update the test to exercise ComputerAgent (or the
telemetry emitter) to produce an actual agent_tool_executed event, capture that
emitted telemetry (e.g., via telemetry stub/mocked TelemetryClient or event
collector used in other tests), and assert the event payload contains the keys
"tool_type" and "tool_name" and that "tool_type" value is one of
["computer","function"] and "tool_name" matches the invoked tool; remove the
current assertions that only check the local expected_* dicts and instead
validate the real emitted event from ComputerAgent/TelemetryClient.
- Around line 84-121: The test creates a ComputerAgent and a computer_call item
but never exercises the agent code that emits telemetry; update
test_computer_action_executed_event to invoke the agent method that processes
items (e.g., ComputerAgent._handle_item or ComputerAgent.process_item —
whichever method implements the computer_call path) with the prepared item,
ensure agent.computer is set to the mock_computer and telemetry is enabled, then
assert mock_record_event was called and that one of its call_args contains the
expected_event payload (verify call includes {"action_type": "click"}); use
mock_record_event.assert_called() and inspect mock_record_event.call_args to
validate the emitted event.
---
Nitpick comments:
In `@libs/typescript/computer/src/computer/providers/base.ts`:
- Around line 38-56: The code duplicates provider normalization
(config.vmProvider ?? 'unknown') in telemetry.recordEvent and initEventData;
create a single normalized variable (e.g., const providerType =
config.vmProvider ?? 'unknown') near the start of the method and use
providerType in telemetry.recordEvent, in argsProvided logic if needed, and in
initEventData to ensure consistent fallback behavior and avoid drift (update
references: telemetry.recordEvent, argsProvided pushes, and initEventData).
ℹ️ Review info
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
libs/python/agent/tests/test_telemetry_events.pylibs/python/computer/computer/computer.pylibs/typescript/computer/src/computer/providers/base.tslibs/typescript/computer/src/computer/types.ts
Comment on lines
+84
to
+121
| @pytest.mark.asyncio | ||
| @patch("agent.agent.record_event") | ||
| @patch("agent.agent.is_telemetry_enabled", return_value=True) | ||
| async def test_computer_action_executed_event(self, mock_telemetry_enabled, mock_record_event): | ||
| """Test that computer_action_executed is emitted for computer calls.""" | ||
| from agent.agent import ComputerAgent | ||
|
|
||
| agent = ComputerAgent(model="anthropic/claude-sonnet-4-5-20250929") | ||
| agent.telemetry_enabled = True | ||
|
|
||
| # Mock computer handler | ||
| mock_computer = MagicMock() | ||
| mock_computer.click = AsyncMock(return_value=None) | ||
| mock_computer.screenshot = AsyncMock(return_value="base64screenshot") | ||
|
|
||
| # Create a mock computer_call item | ||
| item = { | ||
| "type": "computer_call", | ||
| "call_id": "test-call-id", | ||
| "action": { | ||
| "type": "click", | ||
| "x": 100, | ||
| "y": 200, | ||
| }, | ||
| } | ||
|
|
||
| # Process the item (this would normally happen in the agent loop) | ||
| # Note: We can't easily test this without running the full agent loop | ||
| # This is more of an integration test | ||
|
|
||
| # For unit testing, we verify the event structure | ||
| expected_event = { | ||
| "action_type": "click", | ||
| } | ||
|
|
||
| # Verify event structure is correct | ||
| assert "action_type" in expected_event | ||
|
|
Contributor
There was a problem hiding this comment.
This test doesn’t verify telemetry emission.
test_computer_action_executed_event never executes a production action path and never asserts mock_record_event calls. It currently only checks a local dict, so it can pass while the actual telemetry is broken.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@libs/python/agent/tests/test_telemetry_events.py` around lines 84 - 121, The
test creates a ComputerAgent and a computer_call item but never exercises the
agent code that emits telemetry; update test_computer_action_executed_event to
invoke the agent method that processes items (e.g., ComputerAgent._handle_item
or ComputerAgent.process_item — whichever method implements the computer_call
path) with the prepared item, ensure agent.computer is set to the mock_computer
and telemetry is enabled, then assert mock_record_event was called and that one
of its call_args contains the expected_event payload (verify call includes
{"action_type": "click"}); use mock_record_event.assert_called() and inspect
mock_record_event.call_args to validate the emitted event.
Comment on lines
+126
to
+142
| def test_event_structure(self): | ||
| """Test that agent_tool_executed event has correct structure.""" | ||
| expected_computer_tool_event = { | ||
| "tool_type": "computer", | ||
| "tool_name": "click", | ||
| } | ||
|
|
||
| expected_function_tool_event = { | ||
| "tool_type": "function", | ||
| "tool_name": "my_custom_function", | ||
| } | ||
|
|
||
| # Verify expected structure | ||
| assert "tool_type" in expected_computer_tool_event | ||
| assert "tool_name" in expected_computer_tool_event | ||
| assert expected_computer_tool_event["tool_type"] in ["computer", "function"] | ||
|
|
Contributor
There was a problem hiding this comment.
test_event_structure is tautological and non-protective.
The assertions only validate hardcoded local literals, not emitted events from ComputerAgent. This gives false confidence and won’t catch regressions in agent_tool_executed telemetry behavior.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@libs/python/agent/tests/test_telemetry_events.py` around lines 126 - 142,
test_event_structure currently only asserts local hardcoded dicts
(expected_computer_tool_event / expected_function_tool_event) which is
tautological; update the test to exercise ComputerAgent (or the telemetry
emitter) to produce an actual agent_tool_executed event, capture that emitted
telemetry (e.g., via telemetry stub/mocked TelemetryClient or event collector
used in other tests), and assert the event payload contains the keys "tool_type"
and "tool_name" and that "tool_type" value is one of ["computer","function"] and
"tool_name" matches the invoked tool; remove the current assertions that only
check the local expected_* dicts and instead validate the real emitted event
from ComputerAgent/TelemetryClient.
- Regenerate agent-sdk and cli docs to sync with source code - Fix Prettier formatting in README.md, python-sdk.ts, SKILL.md Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
provider_typetocomputer_initializedevent in Python SDKvmProviderfield toBaseComputerConfiginterface in TypeScript SDKprovider_typetocomputer_initializedandts_computer_initevents in TypeScript SDKThis enables tracking of computer provider usage breakdown (cloud, docker, lume, winsandbox, etc.) in PostHog dashboards.
Test plan
provider_typein telemetry eventsprovider_typein telemetry events🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Tests