CLI Internals

Companion to hermes_cli/AGENTS.md (the rules) — this page holds the longer explanations.

Update pipeline

User-facing behaviour (receipts, --plan, snapshot modes) is in Updating; hermes_cli/AGENTS.md carries the short form of the rules.

Fleet-update campaign #91277 (Aug 2026). A PR that weakens a stage must answer for the failure class it guards. plan → snapshot → apply → restart-per-kind → verify → report

• Plan (update_inventory.py, nedura update --plan): read-only inventory — install kind, all profiles, every live gateway with supervisor + running code version. Deployment kinds are first-class: git updates in place; docker/nix/apt are NOT in-place-updatable and the updater reports the correct external command instead of fighting the deployment model.
• Snapshot (backup.py): pre-update quick snapshot for EVERY profile (the code swap + fleet restart touch all of them), each into its own state-snapshots/, identical file set, 1 GiB per-file cap, keep=1. Never add a partial/tiered snapshot set — mixed coverage creates torn-restore states across schema generations. Quick snapshots are FILE-LOSS RECOVERY (the per-profile cron-jobs safety net restores from them), NOT code-rollback insurance; --backup full mode owns rollback.
• Post-swap, the Desktop rebuild decision also trusts the build stamp under NEDURA_HOME, so an install that already lost its artifacts in an earlier update is rebuilt instead of "forgotten" (#90495).
• Restart-per-kind: systemd and launchd restarts are FLEET-WIDE within the updating install (every hermes-gateway* unit / ai.hermes.gateway* LaunchAgent whose home is the updating root or one of its profiles/<name>), drain-first (SIGUSR1), with per-unit/per-label failure isolation. Restarting only the invoking profile's service leaves siblings on stale sys.modules until they crash — the largest dupe-PR cluster in the repo's history came from that bug. The fleet is bounded by HOME, not by namespace: hermes_cli/update_fleet_scope.py judges every unit/label/process by the home it actually runs on (live environ, unit Environment=, plist NEDURA_HOME), and a runtime of another NEDURA_HOME on the same account — a sibling install, the real hermes-gateway.service seen from a scratch home — is named and left alone, never restarted (#93349).
• Verify: gateways stamp code_sha/code_version into gateway_state.json on every runtime-status write (gateway/status.py); the updater compares each live gateway against the fresh checkout and prints a fleet version matrix. After the commit point (the tree moved) nothing fails nedura update (exit 0 unless the code was rolled back): a provably-stale or stopped gateway, a failed build/launcher/config migration/bytecode sweep/channel adoption/Windows resume each print ⚠, land in the receipt as outcome: "success" with followups: [{step, reason}], and keep their own obligation armed (source-completion-pending for the tail, the fleet-restart obligation for gateways) so the next launch or nedura update — including "Already up to date" — retries them. The owed-fleet-restart CLI startup warning is what keeps a mixed-version fleet from ever looking healthy. A completion process that could not be started (temporary directory, request file, spawn) or died without a correlated result (OOM, SIGKILL) is the same kind of debt: an owed completion follow-up with the source-update tail re-armed, exit 0. Exit 2 (refused/concurrent) and exit 1 (nothing committed / rolled back) keep their meaning; the one committed run that exits 1 is an autostash whose restore conflicted and stays parked: the receipt is partial with a user_action, since nothing retries what only the user can re-apply (#122557).
• Report: every run writes a machine-readable receipt to the ROOT home's logs/update_receipts/ (never a sticky profile's; update.log likewise) — latest.json pointer; steps, skips WITH reasons, restart outcome, plan, fleet snapshot, followups. The receipt is on disk as outcome: "running" from the start and refreshed at each stage boundary; the next update marks a running record whose processes are gone interrupted and says so. Readers (nedura logs update, debug bundle, dashboard status, pm receipts) use the root home too. A dependency-sync failure in the completion bootstrap is a dependencies follow-up (A6), and Ctrl-C after the commit point finalizes interrupted (exit 130), never failed. Before a source swap, the parent captures plan/snapshots/receipt and its Windows pause token. update_completion.py runs new-code PM preparation with site initialization disabled, then selected-Python builds, maintenance, scans/restarts and verification. Git/current/ZIP share this owner; never reload or purge modules to continue in the old interpreter. The parent keeps the lock, waits, and accepts only a correlated terminal result. cmd_update still finalizes early failures and missing/killed-child outcomes; PM refusal data survives the handoff. See website/docs/developer-guide/source-update-completion.md. A begun-but-unwritten receipt is a bug.
• Nothing runs pulled code in the pre-pull interpreter. This tree finishes updates through update_completion.run_completion / _update_takeover.py. hermes_cli/update_handoff.py and hermes_cli/update_serve_obligations.py are the FROZEN COMPAT SURFACE for releases that lazily import those module names from the NEW tree after the checkout swap. Keep their public names importable and behavior-preserving. They call into _old_updater.stop_for_relaunch → _run_child. Removing a name bricks every release mid-update. see tests/compat/old_updater_surface.json.

Process-scan coordination between updater, serve/dashboard, and gateway is being replaced by a gateway-owned control socket (#92091); scans are the fallback layer for old/crashed processes — read #92091 before adding any heuristic. Process identity rules (never argv substrings; canonical matchers; parser-derived flag sets; never blanket-exclude gateway ancestors, #87594): root AGENTS.md and website/docs/developer-guide/cli-internals.md.

The systemd blunt-restart fallback waits for the unit's TimeoutStopUSec plus TimeoutStartUSec, with 15 seconds of client-side slack. It reads the target unit in the same manager scope as the restart; both the initial attempt and retry use this budget, including the catch-up restart after an interrupted update. A start after a graceful drain uses only the start budget plus slack. A missing, unparseable, or infinite phase limit falls back to 90 seconds for that phase, keeping unattended updates bounded. Timing out the systemctl client does not cancel the manager's transaction. Custom multi-command stop chains or EXTEND_TIMEOUT_USEC can still outlast this estimate; a real timeout remains an incomplete restart, and successful commands still require the existing service-health and fleet-version verification. Raw numeric *USec values are microseconds, while formatted values use systemd's fixed units, including days, weeks, months and years. The combined timeout is capped below the native signed 32-bit millisecond poll limit (with rounding headroom), so exceptionally long unit limits cannot overflow subprocess polling. Zero/unknown/infinite phase limits use the bounded fallback. This does not change active-turn drain settings.

Nous Portal sign-in

Every Nous Portal sign-in is the plain device-code login. Sign-in completion is one function, settle_after_upgrade, called once per completion: it moves a config on the welcome route to the account's host and the tier's recommended default (models.recommended_nous_default_model, shared with GET /api/model/recommended-default).

The shared flow, states, and copy live in anon_sign_in.py; CLI rendering lives in anon_sign_in_cli.py. anon_auth.py keeps identity and settlement, and re-exports the sign-in API. The flow resolves identity and persistence collaborators through anon_auth at call time to preserve module-attribute monkeypatch seams.

The sign-in itself is one composition: anon_auth.run_sign_in() yields SignInStates (Code, Waiting, Completed, Cancelled, TimedOut, Failed, AlreadySignedIn). It requests a device code, polls the token endpoint on one deadline (an expired code is TimedOut), saves the account, runs settle_after_upgrade exactly once, and never lets a persist or settle failure escape as an exception: it becomes Failed. It works with or without a free tier on disk. Completed.email comes from the access token's email claim; Completed.handoff carries the guest handoff's result lines (what moved, what to reconnect) and Completed.handoff_status the latest record as fields (guest_handoff.wire_status: the public view plus those lines), for a surface that renders the result itself. Every state carries its own .copy (the chat form, which never contains a raw exception, a URL or a nedura verb) and .copy_terminal, so no caller maps a reason to a string. cancelled() is checked at least once a second inside the token poll and once more before the save; a cancel always wins and nothing is saved. TLS verification comes from _resolve_verify, so NEDURA_CA_BUNDLE and friends apply. scope is entered only around the precondition and persist blocks, never across a yield or a network wait, because run_in_executor does not carry contextvars. upgrade_guest (nedura auth upgrade), the CLI and TUI /login, the gateway /login and the desktop start route are renderers over it; a surface that needs the cancel check and the save to be atomic passes persist_guard. nedura model and nedura auth add nous keep _login_nous (it carries --portal-url, --ca-bundle, --insecure and the shared-store import); that is the same plain device login, and it prints the handoff result the same way. Every sign-in that saves an account (run_sign_in, _login_nous, nedura auth add nous, the billing step-up, the tool-setup login) calls settle_after_upgrade itself once its locks are released; the handoff runner never settles.

Guest handoff after a sign-in

When any writer saves a Nous Portal account over a free-tier identity, hermes_cli/guest_handoff.py hands that identity's connected apps to the account: POST /api/anonymous/claim {"token": "<anon_...>"} with the account's own Bearer token. Nothing asks the user first; the result is shown afterwards (nedura auth status nous; the desktop reads it from the sign-in poll's handoff_status and, while a move is still retrying, from the GET /api/providers/oauth/nous/handoff?since=&wait= long-poll, which answers once the record's updated_at changes). Two halves:

• note_replacement runs at the two lowest write seams (auth._store_provider_state and auth_nous._write_shared_nous_state) under the caller's auth-store locks and only writes a pending record.
• Only a sign-in changes which identity the shared store holds. Every _write_shared_nous_state caller declares renewal: a token refresh (renewal=True) never replaces another identity there (identity = the guest's anon_ credential, or the account's JWT sub), so a refresh of one profile's account never replaces a guest another profile minted. That refresh is kept beside the held identity under renewal, where sibling profiles on the same account read their next single-use refresh token.
• drain makes one attempt per due record of this profile and returns what each did. It sends a request, so it runs only outside the auth-store locks: every sign-in surface calls it right after its save (run_sign_in after its persist_guard is released), and so do a free-tier reconcile, nedura auth status and nedura auth upgrade. It only moves apps; config completion (settle_after_upgrade) stays with the sign-in that saved the account.
• A move a transient error left retrying is finished by a retry worker that every long-lived Nous Portal process starts (start_nous_auth_keepalive starts it, guest or not); each tick drains the process's own profile.
• Each attempt is counted and the record leased (not due for about 30 s) in one locked write before the request leaves, so a lost answer still counts and two runs never send the same record.

Answers: claimed records what moved (and refreshes the account token only when cohort_carried; an answer without connectors reads as unknown, never "nothing moved"); unknown_token is Gone, read as "probably moved" after an earlier attempt (a lost success); already_retired is Gone too. Only codes no later attempt can change are final (client_not_allowed, account_locked, invalid_request). Every other answer, 401 and unknown codes included, and every transport failure retries after 1 minute, 10 minutes, 1 hour and 6 hours (Retry-After honoured), then the record is Failed. A Claimed record is never reopened, and user copy never shows a server code; app slugs show as display names (tools.tool_labels.app_title).

Process identity: never infer it from argv substrings

The bug class behind ~10 fleet-update issues (#90778, #87594, #78089, #76129, #91964, ...): classifying a process by "serve" in cmdline or similar. kanban --preserve-cache contains "serve"; a flag VALUE can equal a subcommand (-m dashboard serve); truncated cmdlines hide the real subcommand. Rules:

• Use the canonical matchers: gateway.status.looks_like_gateway_command_line (gateway run), hermes_cli.update_cmd._hermes_holder_subcommand (top-level subcommand of any Nedura argv). Never hand-roll token scans.
• Flag sets must be DERIVED from the parser (_holder_value_flags() introspects build_top_level_parser()), never hand-written lists — they drift.
• Never blanket-exclude ancestors from process scans: when /update runs as the gateway's child, a gateway ancestor must stay visible to the pause machinery (#87594). Exclude interactive ancestry, carve out gateway-shaped ancestors.
• Match on FULL cmdlines; truncate only at display time (#78089).
• Before adding any new scan heuristic, read #92091 — the gateway control socket replaces scans as the primary coordination mechanism; scans are the fallback layer for old/crashed processes.

Skin engine — what skins customize

Element	Skin key	Used by
Banner panel border / title / section headers / dim / body	colors.banner_border, banner_title, banner_accent, banner_dim, banner_text	banner.py
Response box border	colors.response_border	cli.py
Spinner faces (waiting / thinking)	spinner.waiting_faces, spinner.thinking_faces	display.py
Spinner verbs / wings (optional)	spinner.thinking_verbs, spinner.wings	display.py
Tool output prefix / per-tool emojis	tool_prefix, tool_emojis	display.py → get_tool_emoji()
Agent name / welcome / response label / prompt symbol	branding.agent_name, welcome, response_label, prompt_symbol	banner.py, cli.py

Built-in skins (_BUILTIN_SKINS in hermes_cli/skin_engine.py): default (classic gold/kawaii), ares (crimson/bronze with custom spinner wings), mono (grayscale), slate (cool blue). Add a built-in as a dict entry {"name", "description", "colors", "spinner", "branding", "tool_prefix"}. User skins are ~/.nedura/skins/<name>.yaml with the same keys, activated with /skin <name> or display.skin: <name>; the full YAML template is in the Skins & Themes user guide.

Profiles: multi-instance support

Nedura supports profiles — fully isolated instances, each with its own NEDURA_HOME (config, API keys, memory, sessions, skills, gateway). For single-profile commands (nedura -p x <cmd>), _apply_profile_override() in hermes_cli/main.py sets NEDURA_HOME before any module imports, so every get_hermes_home() reference scopes to the active profile. The multiplex gateway and the Desktop/dashboard serve backend serve several profiles from one process instead: the active profile is a contextvar override bound per activity, os.environ["NEDURA_HOME"] stays the launch profile's, and a module-level constant derived from the home freezes to that launch profile (see Gateway Internals § Multiplexed profiles). Profile operations are HOME-anchored (_get_profiles_root() returns Path.home() / ".hermes" / "profiles", not get_hermes_home() / "profiles") so nedura -p coder profile list sees all profiles regardless of which one is active — intentional. Profile-safe coding rules are in the root AGENTS.md; multiplex secret-scope rules in gateway/AGENTS.md.