flowchart TD
subgraph "Browser (Next.js 14)"
UI["Canvas UI\n(React + Tailwind)"]
VC["VoiceContext\n(WebSocket + WebAudio)"]
GC["GraphContext\n(SSE streaming)"]
AC["AssistantContext\n(CRUD)"]
end
subgraph "Next.js API Layer"
PROXY["Catch-all proxy\n/api/[..._path]"]
SSE_ROUTE["POST /api/run_sse"]
end
subgraph "Ice Backend (FastAPI)"
WS["WebSocket /ws/{user}/{session}\nrun_live() voice_runner"]
SSE["POST /run_sse\nrun_async() text_runner"]
REST["Session / Assistant / Store REST"]
OAUTH["GitHub OAuth\n/connect/github"]
HEALTH["GET /health"]
end
subgraph "Agent Tree (ADK)"
ROOT["ice_agent (root)\nGemini 2.5 Flash Native Audio\nor Gemini 2.5 Flash"]
RESEARCHER["researcher\nGemini 2.5 Flash\nVoice: Kore"]
COMPUTE["compute\nGemini 2.5 Flash\nVoice: Charon"]
end
subgraph "Tools"
ART["Artifact Tools\ngenerate / rewrite / highlight"]
SEARCH["Google Search\nAgent Tool"]
TU["ToolUniverse MCP\n2000+ remote tools"]
COGNEE["Cognee\nKnowledge Graph"]
GITHUB["GitHub MCP\n70+ tools"]
VISION["monitor_camera_stream\nVision Streaming"]
SKILLS["151 Scientific Skills\n(SkillToolset)"]
end
subgraph "Storage"
SESSION_DB["Session Service\nInMemory or SQLite/Postgres"]
ARTIFACT_SVC["Artifact Service\nInMemory or GCS"]
FIRESTORE["Firestore\nassistants / store / shares"]
end
UI --> VC
UI --> GC
UI --> AC
VC -->|"binary PCM + JSON"| WS
GC --> SSE_ROUTE --> PROXY --> SSE
AC --> PROXY --> REST
PROXY --> REST
WS --> ROOT
SSE --> ROOT
ROOT --> RESEARCHER
ROOT --> COMPUTE
ROOT --> ART
ROOT --> SEARCH
ROOT --> TU
ROOT --> COGNEE
ROOT --> GITHUB
ROOT --> VISION
COMPUTE --> SKILLS
ROOT --> SESSION_DB
ROOT --> ARTIFACT_SVC
REST --> FIRESTORE
OAUTH --> FIRESTORE
| Tool | Version |
|---|---|
| Python | 3.13+ |
| Node.js | 18+ |
| npm / yarn / pnpm | Latest |
uvicorn |
via pip |
git clone <repo-url>
cd ice/backend python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -r requirements.txt Optional extras for full capability:
pip install tooluniverse mcp # ToolUniverse MCP pip install cognee cognee-integration-google-adk # Knowledge graph
Create a .env file in backend/ (or export them in your shell):
# Required [header-1](#header-1)
GOOGLE_API_KEY=your_google_api_key_here
# Optional — Google Cloud [header-2](#header-2)
GOOGLE_CLOUD_PROJECT=your-gcp-project-id
GOOGLE_CLOUD_LOCATION=us-central1
# Optional — Model overrides (all default to gemini-2.5-flash) [header-3](#header-3)
ICE_LIVE_MODEL=gemini-2.5-flash-native-audio-latest
ICE_TEXT_MODEL=gemini-2.5-flash
ICE_META_MODEL=gemini-2.5-flash
ICE_REFLECTION_MODEL=gemini-2.5-flash
ICE_VISION_MODEL=gemini-2.5-flash
# Optional — Session persistence (default: in-memory) [header-4](#header-4)
ICE_SESSION_DB_URI=sqlite+aiosqlite:///sessions.db
# Optional — Artifact persistence (default: in-memory) [header-5](#header-5)
ICE_ARTIFACT_BUCKET=your-gcs-bucket-name
# Optional — Firestore persistence (default: false) [header-6](#header-6)
FIRESTORE_ENABLED=false
GCP_PROJECT=your-gcp-project-id
FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 # For local Firestore emulator
# Optional — GitHub OAuth [header-7](#header-7)
GITHUB_CLIENT_ID=your_github_client_id
GITHUB_CLIENT_SECRET=your_github_client_secret
GITHUB_REDIRECT_URI=http://localhost:8000/connect/github/callback
ICE_FRONTEND_URL=http://localhost:3000
# Optional — Production code execution sandbox [header-8](#header-8)
ICE_SANDBOX_RESOURCE_NAME=projects/.../sandboxEnvironments/...
# Optional — CORS (default: "*") [header-9](#header-9)
ICE_CORS_ORIGINS=http://localhost:3000 make dev
# or directly: [header-10](#header-10)
uvicorn main:app --host 0.0.0.0 --port 8000 --reload The backend will be available at http://localhost:8000.
Verify with:
make health
# or: [header-11](#header-11)
curl http://localhost:8000/health make test # All tests
make test-quick # Skip slow integration tests cd ice/frontend npm install
# or: yarn install / pnpm install [header-12](#header-12)Create a .env.local file in frontend/:
# Required — Ice backend URL [header-13](#header-13)
NEXT_PUBLIC_ICE_BACKEND_URL=http://localhost:8000
# Optional — API proxy base (default: /api) [header-14](#header-14)
NEXT_PUBLIC_API_URL=/api npm run dev
# or: yarn dev [header-15](#header-15)The frontend will be available at http://localhost:3000.
| Method | Path | Description |
|---|---|---|
GET |
/health |
Health check / readiness probe |
WS |
/ws/{user_id}/{session_id}?modality=audio|both |
Bidi voice streaming (run_live) |
POST |
/run_sse |
Text-mode SSE streaming (run_async) |
POST |
/apps/{app}/users/{uid}/sessions |
Create session (with optional assistant_id) |
GET |
/apps/{app}/users/{uid}/sessions |
List sessions with state snapshot |
GET |
/apps/{app}/users/{uid}/sessions/{sid} |
Get session with events |
PATCH |
/apps/{app}/users/{uid}/sessions/{sid} |
Update session state (EventActions) |
DELETE |
/apps/{app}/users/{uid}/sessions/{sid} |
Delete session |
GET |
/apps/{app}/users/{uid}/sessions/{sid}/messages |
Reconstruct messages from events |
POST |
/apps/{app}/users/{uid}/sessions/{sid}/share |
Generate share link (short slug) |
GET |
/s/{slug} |
Get shared session info |
POST |
/apps/{app}/users/{uid}/sessions/{sid}/fork |
Fork session to a new user |
POST/GET/DELETE |
/apps/{app}/users/{uid}/sessions/{sid}/artifacts[/{name}] |
Artifact CRUD |
POST |
/assistants |
Create assistant |
POST |
/assistants/search |
Search assistants |
PATCH |
/assistants/{id} |
Update assistant |
DELETE |
/assistants/{id} |
Delete assistant |
POST/DELETE |
/store/get, /store/put, /store/delete |
KV store (reflections, context docs) |
GET |
/connect/github |
Start GitHub OAuth flow (PKCE) |
GET |
/connect/github/callback |
GitHub OAuth callback |
GET |
/integrations/github/status |
Check GitHub connection status |
DELETE |
/integrations/github/disconnect |
Disconnect GitHub, shut down MCP |
ice is built on these open-source projects:
- Agent Development Kit (ADK) — multi-agent orchestration, live streaming, session management (Apache 2.0)
- ToolUniverse — 2,064 API tools via MCP server (Apache 2.0)
- K-Dense Scientific Skills — 151 skills for rdkit, biopython, scipy (MIT)
- RDKit — cheminformatics and molecule rendering (BSD)
- 3Dmol.js — WebGL molecular visualization (BSD)
- BlockNote — rich text editor (MPL 2.0)
- assistant-ui — chat interface components
- Open Canvas — artifact editor and reflection patterns (MIT)
- TalkingHead — 3D avatar with lip-sync and morph targets (MIT)
- Cognee — knowledge graph with Gemini embeddings (MIT)
- GitHub MCP Server — GitHub tools via Model Context Protocol (MIT)
This project is licensed under the MIT - see the LICENSE file for details.