The gateway that shows you what cheap actually costs.

Why this exists

What it is

   opencode              relay                    OpenRouter
   your scripts   ──▶    127.0.0.1:8000   ──▶    openrouter.ai
   openai SDK            /v1/chat/completions
                             │
                             ├─▶ route rules   → cheapest tier that fits
                             ├─▶ exact cache   → free repeat answers
                             ├─▶ failover      → sibling model on 5xx
                             ├─▶ spend cap     → 402 when the key is spent
                             └─▶ request_log   → tokens, cost, counterfactual
                                    │
                                    └─▶ dashboard · weekly digest · benchmark

• [+] Tier routing — rules evaluated top to bottom, each request lands on cheap, smart, or baseline; the reason ships back in X-Relay-Route-Reason.
• [+] Exact-match cache — in-memory by default, Redis if you point it at one. A hit is a full save with real token accounting.
• [+] Failover — timeout, connect error, or 5xx retries once, then cycles to the sibling model inside the tier.
• [+] Spend caps — per gateway key. Over the cap is a 402, not a surprise invoice.
• [+] Full request log — model, tokens, latency, cost, route reason, trace id. One row per request, streaming included.
• [+] Dashboard + weekly digest — server-rendered at /, one-line summary at /digest, optional webhook POST.
• [+] Benchmark harness — replays real logged traffic through both the routed path and the baseline, scores them with an LLM judge, and refuses to publish a quality number the judge can't be trusted on.
• [+] OpenTelemetry — spans on every dispatch, no-op when no collector is configured.
• [-] SQLite by default — no Postgres, no Docker, no cloud. Tables auto-create on first start. Postgres is there if you want it.

How the savings number is built

1. 1 · Route Each request is checked against your rules — message length, presence of tools, model alias — and dispatched to the first tier that matches. Nothing is inferred or learned; the decision is deterministic and logged with the rule that produced it.
2. 2 · Price both sides Relay records what the request actually cost, and what it would have cost had your stated baseline model served it instead. That second number is the counterfactual, and it is the only thing that makes "saved" mean anything.
3. 3 · Subtract savings = counterfactual − actual. Positive means the routing paid. Negative means a failover or an escalation cost you more than doing nothing would have — it stays in the total. Unknown baseline price means NULL, rendered as —.
4. 4 · Calibrate The naive version assumes the baseline would have written the same number of output tokens. That's an estimate, and it's labelled as one. Run the benchmark and Relay learns real per-category output-length ratios from measured traffic, at which point the number is labelled a measurement instead.
5. 5 · Gate the quality claim A dual-path benchmark run answers every sampled case twice — routed and baseline — and an LLM judge scores the pairs blind, with path identity hidden and side order randomised. The judge is then checked against your own hand-labels with Cohen's κ. Below the threshold, the run is blocked and the dashboard shows no quality delta at all.

Honest accounting

• [+] Positive — routing beat the baseline. Counted.
• [-] Negative — a failover or escalation cost more than the baseline would have. Shown in red, counted in the same aggregate. It is not quietly floored at zero.
• [+] Zero-cost hit — the cache answered it. Full save, with tokens read from the cached body so the accounting is real.
• [x] Unknown — the baseline model isn't in the price cache, so the counterfactual can't be computed. Written as NULL, displayed as —. Writing $0.00 here would be a lie about a number we don't have.

Install

uv tool install git+https://github.com/pulkyeet/relay

pipx install git+https://github.com/pulkyeet/relay

git clone https://github.com/pulkyeet/relay && cd relay
uv sync

# 1. Write ~/.relay/relay.yaml, generate a gateway key, create the SQLite DB
relay init

# 2. Paste your OpenRouter key into ~/.relay/relay.yaml under openrouter.api_key
#    (or export OPENROUTER_API_KEY instead)

# 3. Start it
relay run

# → http://127.0.0.1:8000/health  {"status":"ok","version":"0.1.0"}
# → http://127.0.0.1:8000/         dashboard

Wiring it into opencode

{
  "provider": {
    "relay": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Relay Gateway",
      "options": {
        "baseURL": "http://127.0.0.1:8000/v1"
      },
      "models": {
        "cheap":    { "name": "Cheap (alias → ling-2.6-flash)" },
        "smart":    { "name": "Smart (alias → deepseek-v4-pro)" },
        "baseline": { "name": "Baseline (alias → deepseek-v4-flash)" }
      }
    }
  }
}

1. 1 Run /connect inside opencode.
2. 2 Search for relay — it's the provider you just added.
3. 3 Paste the gateway key from ~/.relay/relay.yaml.
4. 4 /models now lists Cheap, Smart, and Baseline. Pick one and work normally.

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="sk-relay-...",          # your gateway key, not the OpenRouter one
)

client.chat.completions.create(
    model="cheap",                   # alias → resolved by the router
    messages=[{"role": "user", "content": "..."}],
)

Configuring the router

tiers:
  cheap:
    models: ["inclusionai/ling-2.6-flash", "nex-agi/nex-n2-mini"]   # primary, sibling
  smart:
    models: ["deepseek/deepseek-v4-pro", "xiaomi/mimo-v2.5-pro"]
  baseline:
    models: ["deepseek/deepseek-v4-flash", "xiaomi/mimo-v2.5"]

routes:
  - name: "long-content"
    conditions: [{ type: message_length, min: 4000 }]
    route_to: "smart"
  - name: "tool-use"
    conditions: [{ type: has_tools, value: true }]
    route_to: "smart"
  - name: "default"
    route_to: "cheap"

savings:
  baseline_model: "deepseek/deepseek-v4-flash"   # the counterfactual reference
  methodology: naive                              # "calibrated" after a benchmark run

Proving it

# 1. Dogfood for a few days with log_payloads: true
#    (mode: passthrough if you want unrouted reference traffic)

relay-benchmark sample --seed 0        # stratified draw from request_log
relay-benchmark run --dry-run          # preview the cost first — this spends real money
relay-benchmark run --concurrency 4    # answer every case twice: routed + baseline

relay-benchmark annotate --run-id 1    # hand-label a slice yourself
relay-benchmark judge --run-id 1       # LLM judge, blind; computes Cohen's κ vs your labels
relay-benchmark ratios --run-id 1      # per-category output-length calibration

# then flip methodology: calibrated + calibration_run_id in relay.yaml

Shipped & upcoming

FAQ