Stargate LLM Gateway

OpenAI-compatible LLM Gateway for AWS Bedrock with user management, usage tracking, and cost monitoring. Powered by LiteLLM.

Table of Contents

• Overview
• Architecture
• Quick Start
• User Management
• Security
• Monitoring & Observability
• Documentation Index
• API Reference

Overview

Features

Architecture

Request Flow

Deployment Architecture

Security Model

Quick Start

For Claude Code Users

Add these environment variables to your shell:

export ANTHROPIC_BASE_URL="https://d18l8nt8fin3hz.cloudfront.net"
export ANTHROPIC_API_KEY="<your-api-key>"

Then use Claude Code normally - requests will route through the gateway.

Test API Access

# Test with curl
curl -X POST "$ANTHROPIC_BASE_URL/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $ANTHROPIC_API_KEY" \
  -d '{
    "model": "claude-haiku-4-5",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 50
  }'

View Dashboards

User Management

Using the Management Script

cd scripts/

# Show help
./litellm-users.sh help

# Create a user with budget
./litellm-users.sh create-user --email [email protected] --budget 50 --duration monthly

# Create API key with model restrictions
./litellm-users.sh create-key --alias "haiku-only" --models '["claude-haiku-4-5"]' --budget 10

# List users and keys
./litellm-users.sh list-users
./litellm-users.sh list-keys

Available Models

Model-User Restrictions

See User Management Guide for full documentation.

Security

Admin Endpoint Protection

Admin endpoints are blocked from CloudFront and only accessible via ALB direct (from VPC):

Managing Users from VPC

# Via ALB direct (from VPC/VPN)
curl -X POST "http://kong-llm-gateway-poc-xxx.us-west-1.elb.amazonaws.com/user/new" \
  -H "Authorization: Bearer $MASTER_KEY" \
  -d '{"user_email":"[email protected]","max_budget":10}'

# Via CloudFront (blocked)
curl -X POST "https://d18l8nt8fin3hz.cloudfront.net/user/new"
# Returns: 403 Forbidden - Admin endpoints not accessible via CloudFront

Monitoring & Observability

Observability Stack (Verified ✅)

Dashboards

Grafana - Infrastructure Metrics

Grafana - Token Usage & Costs

Grafana - Architecture Flow

Grafana - PostgreSQL Metrics

Langfuse - LLM Traces

Metrics & Traces Integration

Grafana dashboards correlate Prometheus metrics with Langfuse traces:

Key Metrics

# Total requests
sum(litellm_proxy_total_requests_metric_total)

# Total tokens by model
sum(litellm_total_tokens_metric_total) by (model)

# Total spend by user
sum(litellm_spend_metric_total) by (user)

# Latency P95
histogram_quantile(0.95, sum(rate(litellm_llm_api_latency_metric_bucket[5m])) by (le))

Cost Tracking

Documentation Index

Architecture & Design

Observability

Operations

Configuration

Infrastructure

GitHub Actions

API Reference

Endpoints

Request Format

{
  "model": "claude-haiku-4-5",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Hello!"}
  ],
  "max_tokens": 1024,
  "temperature": 0.7,
  "stream": false
}

Authentication

Include API key in Authorization header:

Authorization: Bearer <your-api-key>

Error Responses

Development

Local Development

Run the full stack locally with Docker Compose:

cd infra
cp .env.example .env
# Edit .env with your AWS credentials

docker-compose up -d

Local services:

• LiteLLM: http://localhost:4000
• Grafana: http://localhost:3000 (admin/admin)
• Victoria Metrics: http://localhost:8428

CI/CD Pipeline

Operational Workflows (Manual)

Note: Operational workflows require poc environment approval for security.

Estimated Costs

License

MIT License - See LICENSE for details.