Hippodrome

Hippodrome is the development environment orchestrator for running the entire Cloud Control Plane stack locally. The name comes from the ancient Greek horse-racing stadium, where all the racing action happened in one unified location.

Quick Start

# Start core services
PANTS_CONCURRENT=True pants run //components/hippodrome/hippodrome/cli.py -- start

# Start with e-commerce services (admin_server, search_proxy)
PANTS_CONCURRENT=True pants run //components/hippodrome/hippodrome/cli.py -- start --profile ecom

This starts: - Dashboard at http://localhost:9000 - Status page showing all services - fake_cell at http://localhost:9001 - Data plane mock - controller at http://localhost:9002 - Control plane API - console at http://localhost:9008 - Web dashboard UI

With --profile ecom, additional services are started: - admin_server at http://localhost:9004 - E-commerce backend - search_proxy at http://localhost:9005 - Search API gateway

The dashboard provides real-time status updates and aggregated logs from all services.

Service Ports

Service	Port	Profile
Dashboard	9000	All
fake_cell	9001	core (when cell=local)
controller	9002	core
console	9008	core
admin_server	9004	ecom
search_proxy	9005	ecom
ecom_settings_exporter	9010	ecom
merchandising_exporter	9011	ecom

Profiles

Use the --profile flag to select which services to start:

Profile	Services	Use Case
core (default)	fake_cell, controller, console	Basic control plane development
ecom	core + admin_server, search_proxy	E-commerce development
full	All services	Full stack testing

Cell Connection

Use --cell to connect to deployed cells instead of fake_cell:

# Connect to staging cell (skips fake_cell)
PANTS_CONCURRENT=True pants run //components/hippodrome/hippodrome/cli.py -- start --profile ecom --cell staging

# Connect to production cell (WARNING: real data!)
PANTS_CONCURRENT=True pants run //components/hippodrome/hippodrome/cli.py -- start --profile ecom --cell prod

Table Prefix

Use --table-prefix to customize DynamoDB table names. Default: dev-{git-branch}-

# Use custom table prefix
PANTS_CONCURRENT=True pants run //components/hippodrome/hippodrome/cli.py -- start --profile ecom --table-prefix my-feature-

Service Communication

Core Profile:

┌─────────────┐     ┌─────────────┐
│ controller  │────▶│  fake_cell  │
│   :9002     │     │   :9001     │
└─────────────┘     └─────────────┘
       ▲
       │
┌──────┴──────┐
│   console   │
│   :9008     │
└─────────────┘

E-commerce Profile (--profile ecom):

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│search_proxy │────▶│admin_server │────▶│  fake_cell  │
│   :9005     │     │   :9004     │     │   :9001     │
└─────────────┘     └─────────────┘     └─────────────┘
                           │
                    ┌──────┴──────┐
                    │ controller  │
                    │   :9002     │
                    └─────────────┘

The controller communicates with fake_cell via the CONTROL_PLANE_URL_OVERRIDE environment variable, which is set automatically by the orchestrator based on the --cell flag.

First-Time Setup

The orchestrator handles most setup automatically: - Creates controller's Python venv if missing - Installs controller requirements if needed - Runs npm ci for console if node_modules/ is missing

No manual setup is required beyond having Python 3.11+, Node.js, and npm installed.

Hot Reload

All services run with hot reload enabled: - Python services: Use uvicorn's --reload or Django's auto-reloader - Console: Uses React's Hot Module Replacement

Changes to source files are detected and services restart automatically.

Graceful Shutdown

Press Ctrl+C to stop all services. The orchestrator sends SIGTERM, waits up to 5 seconds, then SIGKILL if needed.

global_worker Setup (External Repository)

The global_worker is a Cloudflare Worker that handles search query routing, merchandising rules, and caching. It lives in a separate repository and must be set up manually for full e-commerce search functionality.

Why global_worker is Needed

When running with --profile ecom, the search flow is:

search_proxy (:9005) → global_worker (:9012) → fake_cell (:9001)

• search_proxy receives incoming search API requests
• global_worker applies merchandising rules and caching
• global_worker proxies the final request to the cell

Without global_worker, search requests will fail at the search_proxy level.

Quick Setup

1. Clone the repository (outside this repo):

   cd ~/dev
   git clone [email protected]:marqo-ai/global-worker.git
   cd global-worker
   npm install
   

2. Create local configuration (wrangler.local.toml):

   name = "local-global-worker"
   main = "src/index.ts"
   compatibility_date = "2024-09-23"
   compatibility_flags = ["nodejs_compat"]
   
   [vars]
   ENV = "dev"
   FULL_ENV = "dev-local"
   CELL_URL = "http://localhost:9001"
   
   [dev]
   port = 9012
   local_protocol = "http"
   

3. Start the worker:

   npx wrangler dev --config wrangler.local.toml --port 9012
   

Running Without global_worker

If you only need admin operations (not search), you can skip global_worker: - admin_server API calls will work - Controller operations will work - Only search requests through search_proxy will fail

EventBridge Integration

The hippodrome orchestrator includes a webhook endpoint for receiving EventBridge events and routing them to local services. This enables local testing of event-driven services like ecom_settings_exporter and merchandising_exporter.

Architecture

In production, DynamoDB Streams trigger EventBridge Pipes which publish events to EventBridge. Local services subscribe to EventBridge rules. For local development, the orchestrator provides a webhook that simulates this flow:

Production:
DynamoDB Table → DynamoDB Stream → EventBridge Pipe → EventBridge Bus → Service

Local Development:
AWS EventBridge → Webhook → Orchestrator (:9000) → Local Service (:9010/:9011)
                    │
                    └── POST /webhook/eventbridge

Webhook Endpoint

The orchestrator exposes a webhook at POST http://localhost:9000/webhook/eventbridge that accepts EventBridge events and routes them to local services.

Event Format:

{
  "source": "marqo.dynamodb",
  "detail-type": "EcomIndexSettings.MODIFY",
  "detail": {
    "eventName": "MODIFY",
    "tableName": "dev-main-EcomIndexSettingsTable",
    "keys": {
      "pk": {"S": "INDEX#abc123"},
      "sk": {"S": "SETTINGS"}
    },
    "newImage": { ... },
    "oldImage": { ... }
  }
}

Event Routing: | Detail-Type Prefix | Target Service | Port | Endpoint | |--------------------|----------------|------|----------| | EcomIndexSettings.* | ecom_settings_exporter | 9010 | /events | | Merchandising.* | merchandising_exporter | 9011 | /events |

Manual Testing

Test the webhook endpoint directly with curl:

# Test routing to ecom_settings_exporter (port 9010)
curl -X POST http://localhost:9000/webhook/eventbridge \
  -H "Content-Type: application/json" \
  -d '{
    "source": "marqo.dynamodb",
    "detail-type": "EcomIndexSettings.MODIFY",
    "detail": {
      "eventName": "MODIFY",
      "tableName": "dev-main-EcomIndexSettingsTable",
      "keys": {"pk": {"S": "INDEX#test"}, "sk": {"S": "SETTINGS"}},
      "newImage": {"pk": {"S": "INDEX#test"}}
    }
  }'

# Test routing to merchandising_exporter (port 9011)
curl -X POST http://localhost:9000/webhook/eventbridge \
  -H "Content-Type: application/json" \
  -d '{
    "source": "marqo.dynamodb",
    "detail-type": "Merchandising.INSERT",
    "detail": {
      "eventName": "INSERT",
      "tableName": "dev-main-MerchandisingTable",
      "keys": {"pk": {"S": "RULE#123"}},
      "newImage": {"pk": {"S": "RULE#123"}}
    }
  }'

Response format (success):

{
  "webhook_status": "forwarded",
  "target": "localhost:9010",
  "service_response": {"status": "success", "result": ...}
}

Response format (unrouted):

{
  "status": "ignored",
  "reason": "No route configured for detail-type: Unknown.Event"
}

Automatic Tunnel Setup

The --enable-events flag starts a tunnel to expose the local webhook for EventBridge events:

PANTS_CONCURRENT=True pants run //components/hippodrome/hippodrome/cli.py -- start --profile ecom --enable-events

This flag will: 1. Start a tunnel (prefers cloudflared, falls back to ngrok) to expose the local webhook 2. Create a temporary EventBridge rule that forwards events for your table prefix to the tunnel 3. Print the public webhook URL and rule name at startup 4. Clean up the rule and tunnel when the orchestrator stops

Requirements: - Install cloudflared (recommended, free, no account required) - Or install ngrok (requires free account) - AWS credentials configured with EventBridge and IAM permissions

What gets created: - An EventBridge rule named local-stack-webhook-{id} that filters events for your --table-prefix - An API destination and connection to forward events to your tunnel - An IAM role local-stack-eventbridge-api-dest-role (created once, reused)

All resources except the IAM role are automatically cleaned up on shutdown. The rule description indicates it's safe to delete manually if needed.

Validating EventBridge Infrastructure

After deploying the CDK infrastructure (ecom and controller stacks), validate that events are flowing correctly:

Option 1: Use the validation script

# Validate infrastructure for your table prefix
python components/hippodrome/scripts/validate_eventbridge.py --table-prefix dev-main-

# With verbose output
python components/hippodrome/scripts/validate_eventbridge.py --table-prefix dev-main- --verbose

The script checks: - Event buses exist (EcomEventBus, MerchandisingEventBus) - EventBridge Pipes exist and are RUNNING - DynamoDB tables have streaming enabled

Option 2: Validate manually in AWS Console

1. Check Event Buses:
2. Go to EventBridge Console
3. Navigate to "Event buses" → "Custom event buses"

4. Verify {table-prefix}EcomEventBus and {table-prefix}MerchandisingEventBus exist

5. Check EventBridge Pipes:

6. Go to EventBridge Pipes
7. Verify {table-prefix}IndexSettingsEventPipe and {table-prefix}MerchandisingEventPipe exist

8. Confirm status is "Running"

9. Check DynamoDB Streams:

10. Go to DynamoDB Console
11. Open {table-prefix}EcomIndexSettingsTable → "Exports and streams"
12. Confirm DynamoDB Stream is enabled (NEW_IMAGE or NEW_AND_OLD_IMAGES)
13. Repeat for {table-prefix}MerchandisingTable

Option 3: Test event flow end-to-end

1. Start hippodrome with --enable-events
2. In AWS Console, make a change to a DynamoDB table:

   aws dynamodb put-item \
     --table-name dev-main-EcomIndexSettingsTable \
     --item '{"pk": {"S": "test-account"}, "sk": {"S": "INDEX#test"}}'
   

3. Watch the hippodrome logs for incoming events
4. Clean up:

   aws dynamodb delete-item \
     --table-name dev-main-EcomIndexSettingsTable \
     --key '{"pk": {"S": "test-account"}, "sk": {"S": "INDEX#test"}}'
   

Troubleshooting

See troubleshooting.md for common issues and solutions.

Common issues: - Port already in use: Kill the process using lsof -ti :PORT | xargs kill -9 - Services stuck waiting: Ensure you're using PANTS_CONCURRENT=True - Django module errors: The venv is set up automatically; check controller's .venv directory

CLI Reference

pants run //components/hippodrome/hippodrome/cli.py -- start [OPTIONS]

Options:

• --profile [core|ecom|full] - Service profile to run (default: core)
• core: fake_cell, controller, console
• ecom: core + admin_server, search_proxy, e-commerce services

• full: All available services

• --cell [local|staging|prod] - Cell to connect to (default: local)

• local: Uses fake_cell for local development
• staging: Connects to deployed staging cell (skips fake_cell)

• prod: Connects to production cell (WARNING: real data!)

• --table-prefix PREFIX - DynamoDB table name prefix (default: dev-{git-branch}-)

• --enable-events - Enable EventBridge webhook integration with automatic tunnel setup

• --project-root PATH - Project root directory (auto-detected if not specified)

Requirements

• Python 3.11+
• Node.js and npm (for console)
• Pants build system
• AWS credentials (when using --cell staging or --cell prod)

Architecture

For detailed architecture, service configuration, and development guidelines, see AGENTS.md.