Operations & system controls

The Operations console (/ops) is the operator's day-to-day home. It is where you stop and start the whole system, switch the execution mode, set how much the pipeline is allowed to do on its own, and watch the scheduler and health of every background loop. If the dashboard is where you read the lab's state, /ops is where you change it.

This page covers the controls in the order you reach for them: the master stop, autonomy and generation, execution mode, scheduler job status, system health, the soak report, and notification management. Several of these actions are operator-gated — they touch trading and autonomy — so read what each one does before you click it.

Forven is a research tool. Nothing on this page is a prediction, a recommendation, or financial advice. The controls here govern a research process; they do not create an edge, and no setting guarantees against loss.

Two independent switches: pause vs. autonomy

The single most important thing to understand on /ops is that system pause and pipeline autonomy mode are orthogonal — separate switches that answer different questions.

• System pause answers "should anything run at all?" It is a binary halt. When paused, all trading and every autonomous scheduler job is skipped on the next tick.
• Autonomy mode answers "when the system is running, how much may it decide for itself?" It is a three-way setting — manual, semi_auto, or auto — that gates which kinds of jobs are allowed to act without you.

You can be paused in any autonomy mode, and you can be running in the safest autonomy mode. They do not override each other. Treat the pause as the emergency brake and the autonomy mode as the speed limit.

System pause (stop / start)

The Stop / Pause System control is the master brake. It calls POST /api/system/stop, which sets a single paused=true flag in the forven:system_state key-value store along with a paused_at timestamp. On its next tick the scheduler sees is_system_paused() is true and skips every pending and due job.

Starting again is the inverse: Start / Resume System calls POST /api/system/start, clears the flag, and the scheduler resumes on the following tick.

# Pause everything (operator-gated; requires FORVEN_OPERATOR_KEY if set)
Invoke-RestMethod -Method Post -Uri "http://127.0.0.1:8003/api/system/stop"

# Resume
Invoke-RestMethod -Method Post -Uri "http://127.0.0.1:8003/api/system/start"

A pause does not close positions. Stopping the system halts new work and new orders, but it does not auto-exit anything already open. To flatten live exposure you must close positions manually or use the kill-switch on the risk page. In a beta build execution is paper-locked, so there is no live exposure to flatten — but build the habit now.

Pipeline autonomy modes

The autonomy selector sets how far the pipeline may run on its own. It calls POST /api/system/mode with {"mode": "manual"}, {"mode": "semi_auto"}, or {"mode": "auto"}, which persists system_mode and derives the generation-pause flag from it.

In manual, is_autonomy_paused() is true and all autonomous jobs are blocked until you change mode. In semi_auto, generation stays paused but testing, scanning, and execution jobs run. In auto, generation is unfrozen and everything runs. The full behaviour of each mode — and how transitions thaw and freeze jobs — lives on autonomy modes; this page is just where you flip the switch.

Leaving manual is explicit. While the mode is manual, asking to resume generation alone is a no-op — the request is rejected and the mode stays manual. To re-enable autonomous work you must set the mode to semi_auto or auto with POST /api/system/mode. There is no back door.

Generation pause (backlog throttle)

Independently of autonomy mode, you can freeze only the strategy-creation jobs — the crucible planner and ideation — while leaving testing, scanning, and execution running. This is the safe way to let a large backlog drain without inventing more work on top of it.

# Freeze new-strategy generation; keep testing & scanning alive
Invoke-RestMethod -Method Post -Uri "http://127.0.0.1:8003/api/system/generation/pause"

# Resume generation (only takes effect outside manual mode)
Invoke-RestMethod -Method Post -Uri "http://127.0.0.1:8003/api/system/generation/resume"

Resuming generation has no effect while the mode is manual — change the mode instead.

Execution mode (paper ↔ live)

The execution-mode toggle switches whether orders are simulated or sent to the exchange. It calls POST /api/execution-mode with {"mode": "paper"} or {"mode": "live"}.

• paper — trades are recorded in SQLite only. Nothing reaches an exchange. This is the default (FORVEN_EXECUTION_MODE=paper).
• live — orders route to HyperLiquid. Gated, and additionally guarded by the FORVEN_ALLOW_MAINNET env flag for real-funds (mainnet) routing.

Beta builds hard-lock to paper. When FORVEN_ENV=beta (set by the packaged installer), a request to switch to live is rejected with a loud error regardless of any config.json value or environment variable. The lock is enforced where the mode is read, so even a stale live setting can never take effect. See execution modes for the full paper-vs-live distinction and why paper PnL never reconciles against the exchange.

Scheduler job status

The Scheduler section lists the 35+ built-in background jobs — the promotion loop, verdict loop, revisit, paper graduation, the hourly scanner, daily learning, database maintenance, and more. Each row shows the job's name, enabled state, last_run_at, last_error, and next_run_at.

From here you can:

• Enable or disable a job, or adjust its cadence, with PATCH /api/scheduler/{job_id}. Changes persist in the scheduler_jobs table; the next tick uses the updated schedule to recompute next_run_at.
• Read each job's schedule — schedule_type (cron or interval) and schedule_expr.

Some cadences are managed for you. When throughput_auto_scheduler_control=true in settings, a runtime override recalculates cadences from the throughput level (normal / high / aggressive) so you do not have to tune each job by hand. The full job catalogue, throughput control, and tuning advice live on the scheduler page.

Stale locks recover themselves — usually. If a job hangs, its running_since lock is held in the DB. On the next tick the scheduler force-recovers locks older than the stale threshold (capped at an illustrative 3900s absolute). It will not force-recover while a background task or worker thread is still alive (zombie-thread tracking, ticket B-30) — the lock is held until that thread exits, to prevent duplicate execution. A job stuck "running" with a live thread usually means a slow external call (LLM or exchange). See troubleshooting if a lock never clears.

System health

/ops surfaces the system health state: a rolled-up green / amber / red signal aggregated from the scheduler, the brain workers, the bot, and the data collector, plus per-stream data-freshness checks.

• green — healthy.
• amber — a component or data stream is overdue past its SLA.
• red — critical; a component needs attention.

Data-stream SLAs are per-stream and independent — for example OHLCV is considered stale after 60 minutes, funding after 12 hours, open interest after 3 hours. A stream missing past its SLA emits an amber alert; past roughly twice the staleness threshold it goes red. There is no single overall data-health rollup — each stream is judged on its own. The thresholds and the alert-routing detail live on health monitoring.

The soak report

The soak report is a longer-horizon health readout: it summarizes how cleanly the system has been running over a sustained window rather than at this instant. Use it to answer "has the lab been stable lately?" as opposed to the live health signal's "is it healthy right now?" A clean soak with a green live signal is the state you want before trusting the lab to run unattended in auto.

Notification management

The notifications area is where the alerts the system raises are routed and reviewed. Every alert flows through one emit_notification() path, which applies a routing policy per event type:

• app_only — shown in the in-app notification center only.
• discord_immediate — pushed straight to Discord.
• discord_thread — delivered into a Discord thread.
• drop — silenced.

Routing is deduplicated by key with a cooldown, so a flapping condition does not spam you — with one deliberate exception: deduplication is severity-aware, and a critical alert is never suppressed by a matching non-critical row. Critical noise always reaches you.

From a notification you can acknowledge it (POST /api/notifications/{id}/acknowledge), resend it through the current policy (POST /api/notifications/{id}/resend), or hand it to the full-stack-engineer agent as a repair task (POST /api/notifications/{id}/repair). Per-event delivery preferences (which events go to Discord) are configured in Settings; the detail lives on notifications.

Steps: a daily ops sweep

A short, repeatable pass to confirm the lab is in the state you intend:

1. Open /ops. Confirm the system is running (not paused) unless you paused it on purpose.
2. Check the autonomy mode is what you expect — manual for a hands-on session, auto only when you mean to leave the lab unattended.
3. Confirm the execution mode. In a beta build this is always paper; if you see anything else, the lock has been bypassed and you should stop and investigate.
4. Scan the scheduler list for any job with a recent last_error or a next_run_at that never advances — that is your first sign of a stuck loop.
5. Read the health signal. Green, move on; amber or red, follow it to health monitoring before doing anything else.
6. Glance at the soak report for slow-building instability the instantaneous signal would not show.
7. Clear or acknowledge any notifications that need it; route a genuine fault to a repair task.

What you'll see

On a healthy lab: the system shows running, the autonomy mode and execution mode read what you set them to (paper in beta), every scheduler job has a future next_run_at and an empty last_error, the health signal is green, the soak report is clean, and the notification center is quiet or holds only acknowledged informational notices. When something is wrong, it surfaces here first — a job's last_error fills in, a data stream goes amber, or a critical notification refuses to be deduped away.

Caveats

• Beta is paper-locked. Execution mode cannot be switched to live in a packaged beta build; the toggle will reject the change. Live-only readings will be empty until live trading is unlocked.
• Pause is not a flatten. Stopping the system never closes open positions — use the kill-switch or close manually if you need exposure gone.
• Leaving manual requires a mode change. Resuming generation alone is rejected in manual; set semi_auto or auto explicitly.
• A held scheduler lock can be legitimate. Recovery deliberately keeps a lock while a worker thread is still alive, so a "running" job may simply be waiting on a slow external call rather than being stuck.
• Numbers quoted here (the 3900s stale-lock cap; OHLCV 60m, funding 12h, OI 3h SLAs) are illustrative defaults drawn from the current build, not guarantees — they can change between releases.

Related

• Scheduler & jobs — the full job catalogue, throughput control, and cadence tuning.
• Health monitoring — how the green / amber / red states and data-stream SLAs are computed.
• Autonomy modes — what manual, semi_auto, and auto each enable in depth.
• Execution modes — paper vs. live, the beta paper-lock, and how to switch safely.