Database & maintenance

Forven keeps everything on your machine, in one place. The cost of that local-first design is that the on-disk store grows as the research daemon works: backtests, activity rows, notifications, and heartbeats accumulate. Maintenance is how you keep that store fast and bounded without losing the data you care about.

This page is for operators. It covers the automatic maintenance job, the retention windows you can tune, when to run a manual VACUUM, the factory-reset categories, and the one-time legacy home migration that runs for returning users.

Where your data lives

All state lives under FORVEN_HOME, which defaults to ~/.forven. See first run & setup for how this directory is created. The pieces that maintenance touches:

• forven.db — the single source of truth (SQLite, schema version 28). Strategies, trades, agent tasks, scheduler jobs, notifications, the activity log, and the key-value settings store all live here.
• forven.db-wal and forven.db-shm — the write-ahead log and shared-memory index. SQLite runs in WAL mode for concurrency, so the database is really three files, not one.
• forven_lab.db — the lab/optimization database.
• workspace/, chromadb/ — markdown workspace files and the vector memory store.

Because WAL mode spans three files, a stray copy of just forven.db is not a complete backup. Always pair the .db, .db-wal, and .db-shm files when you copy them, or checkpoint first (below) so the WAL is flushed.

The maintenance job

A scheduled job, forven-db-maintenance, runs on an interval and does three things in order:

1. Prune aged rows past their retention window, in bounded batches.
2. Checkpoint the WAL to flush pending writes back into the main database file.
3. Optionally VACUUM to reclaim file size (off by default — see below).

Pruning happens in small batches (roughly 500 rows per transaction) so the job never holds a long write lock. That keeps lock contention with live trading and scanning low while the store is being trimmed.

Why it matters: without pruning, a few tables grow without bound. In observed installs the backtest results table reached hundreds of megabytes and the activity log passed half a million rows. Illustrative figures, not targets — your numbers depend on how hard you run the daemon — but the direction is the point: an untrimmed store gets slow.

Default retention windows

These are the defaults the prune step uses. Each can be overridden in settings (next section).

If you are troubleshooting an event from more than 60 days ago, the notification rows for it may already be pruned, and heartbeats older than two days are almost certainly gone. Keep periodic backups if you need a longer audit trail than the defaults provide.

Tuning retention windows

Retention windows are settings-driven, stored in the key-value store under forven:pipeline:settings. Maintenance reads keys such as retention_activity_log_days and retention_scanner_results_days. Like all Forven configuration, settings follow the precedence chain env > kv-store > config.json > defaults — see the configuration reference for the full picture.

You can read and update these settings through the pipeline settings endpoint:

# Read the current pipeline settings (includes retention_* windows)
curl http://127.0.0.1:8003/api/settings/pipeline

# Override a retention window (operator key required if FORVEN_OPERATOR_KEY is set)
curl -X PUT http://127.0.0.1:8003/api/settings/pipeline `
  -H "Content-Type: application/json" `
  -d '{ "retention_activity_log_days": 120 }'

The next time forven-db-maintenance runs, it uses the new windows. Longer windows keep more history at the cost of disk and query speed; shorter windows keep the store lean. Pick based on how much audit history you actually consult.

Manual WAL checkpoint and VACUUM

The maintenance job checkpoints the WAL on every run, so you rarely need to do it by hand. Two cases where a manual step helps:

• Before a backup, checkpoint so the main forven.db file holds the latest writes and the WAL is small.
• To reclaim disk after a large prune, run VACUUM. Deleting rows frees space inside the file but does not shrink the file on disk; VACUUM rewrites the database compactly.

VACUUM takes an exclusive lock and rewrites the whole file, so it can take a while and blocks other writers while it runs.

Steps: reclaim disk with a manual VACUUM

1. Open the operations dashboard at /ops and pause the system so trading and autonomous jobs stop. A paused system does not auto-close open positions — see the caveat below.
2. Confirm the scheduler shows no jobs currently running (the WAL and locks should be quiet).
3. Trigger the maintenance routine so the prune and WAL checkpoint complete first.
4. Run VACUUM while the app is quiet. This holds an exclusive lock until it finishes.
5. Resume the system from /ops when VACUUM completes.

What you'll see: the forven.db file size drops to roughly the size of the live data, the -wal file shrinks after the checkpoint, and the scheduler resumes on its next tick once you start the system again.

Pausing the system from /ops halts trading and scheduler jobs but does not close open positions. If you are running paper trading this is harmless. If real positions are open, exit them deliberately before any disruptive maintenance.

Factory reset (selective cleanup)

When you want to wipe categories of data rather than prune them, Forven exposes an operator-gated factory reset. It is selective: you choose which categories to clear, and credentials are preserved by default so you do not have to re-link your account or re-enter exchange keys.

Reset categories cover, broadly:

• Pipeline data — strategies and trades
• Agent tasks and the task queue
• The activity log
• AI memory (the workspace/vector store)
• Settings

Credentials and your encrypted secrets are left untouched unless you explicitly opt to clear them. Factory reset is destructive and operator-gated; if FORVEN_OPERATOR_KEY is set, you must supply it. Treat it as a last resort and take a backup of FORVEN_HOME first.

Legacy home migration

Forven was previously named, and earlier installs stored state under ~/.juddex or ~/.judex. On first startup the app detects those directories and auto-merges them into ~/.forven, renaming database files in the process (juddex.db → forven.db). Returning users keep their history without any manual steps.

Two things to know:

• Workspace writes are mirrored back to the legacy ~/.juddex/workspace for compatibility. If a legacy copy has diverged, the merge uses a longest-text-wins rule, which can drop one version. Check a backup if a workspace file looks wrong after migration.
• The encryption key lives in a non-synced location (%LOCALAPPDATA%\Forven\.forven_key on Windows), while auth.json lives under FORVEN_HOME. If your backups separate these, encrypted values can stop decrypting. Always back up the key and FORVEN_HOME together.

Caveats

• WAL means three files. A backup of forven.db alone is incomplete. Pair .db, .db-wal, and .db-shm, or checkpoint first.
• Notifications and heartbeats are short-lived. 60-day and 2-day defaults mean old audit rows disappear; keep backups if you need them.
• VACUUM is exclusive. Run it only when the app is quiet; it blocks writers until done.
• System pause does not close positions. Maintenance never touches open trades, and pausing for a VACUUM will not flatten them.
• Encryption key is non-synced. It will not follow OneDrive or Dropbox; back it up with FORVEN_HOME so encrypted secrets stay readable.

Forven is a research tool. Nothing here is financial advice, results are not predictive, and maintenance routines operate only on local records — they do not affect any real exchange position.

Related

• Configuration reference — settings precedence and the keys that drive retention
• Health monitoring — component states and data-stream SLAs
• First run & setup — where FORVEN_HOME lives and how it's created
• Operations & system controls — pause the system and trigger routines