Circuit breakers

How Forven's exchange-health circuit breakers (hl_price, hl_trade, hl_account) trip, recover, and tell rate-limits apart from real outages.

Circuit breakers are Forven's defence against a sick exchange. When HyperLiquid stops answering cleanly, the breakers stop sending it requests instead of hammering a failing endpoint and corrupting your order state. They are distinct from the kill-switch, which reacts to your drawdown, not the exchange's health.

This page is for operators running live or testnet execution. If you are only paper trading, the breakers never engage — paper fills come from local candle data, not the exchange.

Forven is a research tool. Nothing here is financial advice, and breaker behaviour is a safety mechanism, not a promise that orders will fill. Always confirm position state manually after a breaker event.

What a circuit breaker is

A circuit breaker is a small state machine that sits in front of a category of HyperLiquid calls. It counts failures. When failures cross a threshold it "opens" and short-circuits further calls for a timeout window, so Forven stops issuing requests that are likely to fail anyway. After the timeout it cautiously tests the connection again before fully resuming.

Forven runs three independent breakers, one per request category:

Breaker	Guards	Failure threshold	Open timeout
`hl_price_breaker`	Market-data / price reads	5 failures	20s
`hl_trade_breaker`	Order submit / cancel / close	3 failures	30s
`hl_account_breaker`	Account state, balances, positions	4 failures	25s

The thresholds above are the real defaults from the execution layer. They are tuned so that order placement (hl_trade_breaker) is the most conservative — it opens after just three failures — while price reads tolerate more transient noise before tripping.

The state machine: CLOSED, OPEN, HALF_OPEN

Each breaker moves through three states.

CLOSED — Normal. Requests pass through. The breaker counts consecutive failures.
OPEN — Tripped. Requests in this category are rejected immediately without touching the exchange. The breaker stays open for its timeout window (20s / 30s / 25s).
HALF_OPEN — Probationary. After the timeout the breaker allows a single test request through. Success closes the breaker; failure re-opens it immediately.

        threshold failures
CLOSED ───────────────────► OPEN
   ▲                          │
   │ test request             │ open timeout
   │ succeeds                 ▼
   └───────────── HALF_OPEN ◄─┘
                     │
                     │ test request fails
                     └──────────► OPEN (re-open)

By default the half-open state allows only one probe request (half_open_max_calls=1). If that single probe fails, the breaker re-opens at once and waits another full timeout. This is deliberately aggressive: it prevents a slow-recovery cascade where a barely-alive exchange gets flooded the instant the window expires.

What trips a breaker — and what does not

The most important distinction in this subsystem is rate-limit versus outage.

A 429 (rate-limit) does NOT trip the breaker. A rate-limit means the exchange is healthy but busy. Signed order submissions retry with bounded exponential backoff (roughly 0.5s up to a 4s cap) and read-side requests treat 429 as transient too. Crucially, this keeps the emergency-close path alive during a rate-limit burst — the kill-switch can still close positions even while you are being throttled.
True failures DO trip the breaker. 5xx responses (502 / 503 / 504), connection errors, and timeouts count toward the failure threshold and open the breaker.

In short: throttling is survivable and never freezes order routing; a genuine outage stops Forven from sending more doomed requests.

Because 429 is treated as transient, you will not see a circuit-breaker entry in status when you are merely rate-limited. Look for bounded 429 backoff messages in the logs instead — that is normal throttling, not a fault.

Where to see breaker state

Breaker state is read-only and surfaced in two places:

GET /api/risk → the circuit_breakers block reports each breaker's current state. The recovery_status field reflects any breaker-triggered recovery in progress. This is what the dedicated /risk page renders.
GET /api/dashboard → circuit_breakers gives the same live states for the quick-overview panel on the home dashboard.

On the /risk page you will see the three breakers (hl_price, hl_trade, hl_account) alongside execution mode, the kill-switch, daily-loss halt, and recovery status. A healthy system shows all three CLOSED.

What you'll see

When everything is well, all three breakers read CLOSED and there is no recovery activity. During an exchange incident you will see one or more breakers flip to OPEN, then HALF_OPEN as the timeout expires, then back to CLOSED once a probe succeeds. If a breaker keeps cycling OPEN → HALF_OPEN → OPEN, the exchange is not recovering and the incident needs human attention.

Recovery and interaction with the kill-switch

Breakers recover automatically. There is no operator reset for a circuit breaker — once the exchange answers a half-open probe cleanly, the breaker closes and routing resumes on its own. (The manual reset endpoint, POST /api/ops/reset-kill-switch, exists for the drawdown kill-switch, not for the breakers.)

The two systems are designed to cooperate during a crisis:

The drawdown kill-switch needs to emergency-close positions even when the exchange is degraded.
Rate-limits (429) are excluded from tripping hl_trade_breaker, so the close path keeps retrying through throttling.
Emergency closes escalate their slippage cap across retries (illustrative defaults: 300 bps, then 600 bps, then 1000 bps) so a fill still lands during a volatile, throttled moment.

This is why a rate-limit storm cannot lock you out of closing a losing position.

Caveats

The breakers protect order routing, not order outcome. An OPEN hl_trade_breaker means new submissions are short-circuited — it does not unwind positions already on the exchange. Reconcile state via phantom recovery and the /risk recovery fields after any incident.
Half-open allows only a single probe by default. A flaky exchange that fails the probe will keep the breaker effectively open for repeated full timeout windows.
Drawdown calculation depends on the high-water mark being synced from the exchange account value at startup. If account reads are failing (an open hl_account_breaker), risk checks may run against stale equity until the breaker closes.
Breakers are a live/testnet concern only. In paper mode, trades fill against local OHLCV prices and never reach HyperLiquid, so no breaker can trip.

Forven is early-access. Treat breaker state as a signal to slow down and verify, not as a guarantee that the platform has fully contained an exchange fault.

Risk controls — the kill-switch, drawdown, daily-loss, and per-trade limits.
HyperLiquid integration — credentials, testnet vs mainnet, and the account connection the breakers guard.
Health monitoring — green/amber/red component states and data-stream SLAs.

Circuit breakers

On this page