Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Git
.git
.gitignore

# GitHub
.github

# Documentation
*.md
LICENSE

# Compiled binaries
advanced_server
examples/advanced_server/advanced_server

# IDE / editor
.vscode
.idea
*.swp
*.swo
*~

# OS files
.DS_Store
Thumbs.db
73 changes: 73 additions & 0 deletions .github/workflows/docker-publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
name: Build and Push Docker Images

on:
push:
branches: [main]
tags: ["v*"]
pull_request:
branches: [main]

env:
REGISTRY: ghcr.io
IMAGE_PREFIX: ${{ github.repository }}

jobs:
build-and-push:
runs-on: ubuntu-latest

permissions:
contents: read
packages: write

strategy:
fail-fast: false
matrix:
example:
- ab_testing
- advanced_server
- basic_rules
- content_filter
- pii_redactor
- user_blocking

steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Set up QEMU
uses: docker/setup-qemu-action@v3

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Log in to GHCR
if: github.event_name != 'pull_request'
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Extract metadata (tags, labels)
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_PREFIX }}/${{ matrix.example }}
tags: |
type=ref,event=branch
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
type=semver,pattern={{major}}
type=sha

- name: Build and push
uses: docker/build-push-action@v6
with:
context: .
file: examples/${{ matrix.example }}/Dockerfile
platforms: linux/amd64,linux/arm64
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
80 changes: 80 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# Hook Server Examples

Example webhook servers implementing the CATE (Contextual Access for Tool Execution) hook system. These servers handle access control, pre-execution validation, and post-execution filtering for tool calls.

## Examples

### Advanced Server (Full-Featured with Web UI)

A comprehensive server combining all features with a browser-based dashboard for configuration.

- **[examples/advanced_server/](examples/advanced_server/)** - Access rules, PII redaction, A/B testing, and a web UI

### Focused Examples (Single-Purpose, No UI)

Minimal servers demonstrating individual hook capabilities:

| Example | Description | Hook Points Used |
| ------- | ----------- | ---------------- |
| **[user_blocking](examples/user_blocking/)** | Block specific users from tools | Access, Pre |
| **[content_filter](examples/content_filter/)** | Filter/block based on content | Access, Pre, Post |
| **[pii_redactor](examples/pii_redactor/)** | Detect and redact PII in outputs | Post |
| **[ab_testing](examples/ab_testing/)** | A/B and canary test tool versions | Pre |
| **[basic_rules](examples/basic_rules/)** | Configurable rules for all hooks | Access, Pre, Post |

## Quick Start

```bash
# Run the advanced server with the web dashboard
go run ./examples/advanced_server -config ./examples/advanced_server/example-config.yaml

# Or run a focused example
go run ./examples/pii_redactor -types "email,ssn,credit_card"
go run ./examples/user_blocking -block "user1,user2"
go run ./examples/content_filter -config ./examples/content_filter/example-config.yaml
go run ./examples/ab_testing -config ./examples/ab_testing/example-config.yaml
```

## Hook Points

These servers implement webhook endpoints that integrate with an engine's hook system:

| Endpoint | Purpose |
| -------- | ------- |
| `GET /health` | Health check |
| `POST /access` | Control which tools users can see |
| `POST /pre` | Validate/modify tool inputs before execution |
| `POST /post` | Validate/modify tool outputs after execution |

## Architecture

```
Engine Request Flow
────────────────────────────────────────────────►

↓ ↓ ↓
┌─────────┐ ┌─────────┐ ┌─────────┐
│ ACCESS │ │ PRE │ │ POST │
│ Hook │ │ Hook │ │ Hook │
└────┬────┘ └────┬────┘ └────┬────┘
│ │ │
▼ ▼ ▼
Can user see Can user run Filter/modify
this tool? this tool? the output?
```

## Project Structure

```
├── examples/
│ ├── advanced_server/ # Full-featured server with web UI
│ ├── basic_rules/ # Configurable rules (original example)
│ ├── user_blocking/ # Block specific users
│ ├── content_filter/ # Content-based filtering
│ ├── pii_redactor/ # PII detection and redaction
│ └── ab_testing/ # A/B and canary testing
├── pkg/
│ └── server/ # Generated types from OpenAPI schema
├── go.mod
└── go.sum
```
Binary file added advanced_server
Binary file not shown.
25 changes: 25 additions & 0 deletions examples/ab_testing/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
FROM --platform=$BUILDPLATFORM golang:1.25-alpine AS builder

ARG TARGETOS
ARG TARGETARCH

WORKDIR /src

# Cache dependency downloads
COPY go.mod go.sum ./
RUN go mod download

# Copy shared package and example source
COPY pkg/ pkg/
COPY examples/ab_testing/ examples/ab_testing/

RUN CGO_ENABLED=0 GOOS=${TARGETOS} GOARCH=${TARGETARCH} \
go build -ldflags="-s -w" -trimpath -o /bin/server ./examples/ab_testing

FROM gcr.io/distroless/static-debian12

COPY --from=builder /bin/server /bin/server

EXPOSE 8080

ENTRYPOINT ["/bin/server"]
110 changes: 110 additions & 0 deletions examples/ab_testing/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
# A/B Testing Example

A minimal hook server that demonstrates how to **A/B test and canary-deploy tool versions** by routing tool calls to different servers.

## What It Shows

- **Pre-execution hook**: Route tool calls to different servers/versions based on experiment config
- **Consistent hashing**: Same user always gets the same variant (sticky assignment)
- **Weighted traffic splitting**: Control what percentage of traffic goes to each variant
- **Tool registry integration**: Fetch available tools from an external API (e.g., [Arcade](https://docs.arcade.dev/en/references/api))
- **Statistics tracking**: Monitor how many requests each variant receives

## Quick Start

```bash
# Run with experiment config
go run ./examples/ab_testing -config experiments.yaml
```

## Config File Format

```yaml
# Optional: external tool registry for discovering tools
registry_url: "https://api.example.com"
registry_key: "your-api-key"

experiments:
# Canary deployment: 10% traffic to new version
- name: "search-v2-canary"
enabled: true
toolkit: "Search"
tool: "WebSearch"
mode: canary
variants:
- name: "stable"
weight: 90
version: "1.0.0"
- name: "canary"
weight: 10
version: "2.0.0"
server_name: "search-v2"
server_uri: "http://search-v2.internal:8080"
server_type: "arcade"

# 50/50 A/B test
- name: "email-provider-compare"
enabled: true
toolkit: "Email"
tool: "*"
mode: ab
variants:
- name: "provider-a"
weight: 50
- name: "provider-b"
weight: 50
server_name: "email-alt"
server_uri: "http://email-alt.internal:8080"
server_type: "arcade"
```

## How It Works

### Variant Selection
1. When a tool call matches an active experiment (by toolkit and tool patterns), a variant is selected
2. Selection uses consistent hashing: `SHA256(user_id + ":" + experiment_name)`
3. The hash is mapped to a variant based on configured weights
4. The same user always gets the same variant for a given experiment

### Server Routing
- If the selected variant has a `server_uri`, the pre-hook overrides the server routing
- This allows routing to different backend servers, different tool versions, etc.
- If no server override is specified, the tool executes normally (useful for tracking only)

### Statistics
- GET `/stats` returns per-experiment, per-variant request counts
- This shows the actual traffic distribution across variants

## Testing

```bash
# Start with example config
go run ./examples/ab_testing -config experiments.yaml &

# Send pre-hook requests for different users
for i in $(seq 1 20); do
curl -s -X POST http://localhost:8888/pre \
-H "Content-Type: application/json" \
-d "{
\"execution_id\": \"exec-$i\",
\"tool\": {\"name\": \"WebSearch\", \"toolkit\": \"Search\", \"version\": \"1.0.0\"},
\"context\": {\"user_id\": \"user-$i\"},
\"inputs\": {\"query\": \"test\"}
}" | python3 -m json.tool
echo
done

# Check statistics
curl -s http://localhost:8888/stats | python3 -m json.tool
```

## Tool Registry Integration

The server can fetch available tools from an external tool registry API:

```bash
# Configure registry URL in config, then fetch
curl -s -X POST http://localhost:8888/registry/fetch | python3 -m json.tool
```

This is useful for discovering what tools and versions are available before setting up experiments.
42 changes: 42 additions & 0 deletions examples/ab_testing/example-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# A/B Testing Configuration
#
# Defines experiments for routing tool calls to different versions/servers.

# Optional: external tool registry for discovering available tools
registry_url: ""
registry_key: ""

experiments:
# Canary deployment: route 10% of traffic to a new search tool version
- name: "search-v2-canary"
enabled: true
toolkit: "Search"
tool: "WebSearch"
mode: canary
variants:
- name: "stable"
weight: 90
version: "1.0.0"
- name: "canary"
weight: 10
version: "2.0.0"
server_name: "search-v2"
server_uri: "http://search-v2.internal:8080"
server_type: "arcade"

# 50/50 A/B test comparing two email tool implementations
- name: "email-provider-compare"
enabled: true
toolkit: "Email"
tool: "*"
mode: ab
variants:
- name: "provider-a"
weight: 50
version: "1.0.0"
- name: "provider-b"
weight: 50
version: "1.0.0"
server_name: "email-alt"
server_uri: "http://email-alt.internal:8080"
server_type: "arcade"
Loading