Operate

Operations

Monitor cluster state, replication, resource pressure, snapshots, cleanup, and TLS from the first day of a Nexir deployment.

Start and stop

Start nexir in the foreground under your process supervisor. Configuration is loaded once at startup. Send SIGINT with Ctrl+C or SIGTERM for graceful shutdown; Nexir stops listeners and Raft, then flushes storage, within a bounded teardown. The exit status is a verdict: 0 clean, 1 a fatal condition was recorded (during operation or during the teardown itself), 2 a usage or configuration error before the server started. A request that received no reply while a node stopped has an unknown outcome — it may or may not have committed — and must be reconciled or retried idempotently, never assumed lost.

Restart an intact node with its original node ID, addresses, and storage path. For planned leader maintenance, pause writes first: CLUSTER STEPDOWN needs a voter that has matched the leader's last log entry and is normally refused under sustained writes.

redis-cli --raw -p 16379 CLUSTER STEPDOWN
# wait for OK and verify the new leader
redis-cli --raw -p 16379 CLUSTER INFO

Health and readiness

redis-cli --raw -p 16379 PING
redis-cli --raw -p 16379 CLUSTER INFO
redis-cli --raw -p 16379 CLUSTER MEMBERS
redis-cli --raw -p 16379 CLUSTER IDENTITY

PING proves the RESP listener can reply. For readiness, also require a healthy initialized identity, known leader, expected membership, and acceptable lag. A local PING alone does not prove write quorum.

Metrics

curl http://127.0.0.1:9090/metrics

The endpoint defaults to 127.0.0.1:9090 and has no TLS or authentication. Key signals:

Area	Metrics
Leadership	`nexir_raft_current_leader_id`, `raft_state`, `raft_term`, `nexir_raft_leader_changes_total`
Replication	`raft_last_log_index`, `raft_last_applied`, `nexir_raft_replication_lag_entries{peer_id}`, `nexir_raft_membership_joint`
Clients	`nexir_active_client_connections`, `nexir_client_connection_rejected_total`, `nexir_client_backpressure_total`
Writes/reads	`nexir_pending_raft_write_tasks`, `nexir_raft_client_write_seconds`, `nexir_linearizable_read_barrier_seconds`, `nexir_linearizable_read_barrier_errors_total`
Response budget	`nexir_client_response_budget_bytes`, `nexir_client_response_reserved_bytes`, `nexir_client_response_refused_total{path}`, `nexir_client_buffered_response_bytes`
Memory	`nexir_process_resident_memory_bytes`, `nexir_memory_admission_open`, `nexir_memory_pressure_cause{cause}`, `nexir_memory_admission_refusals_total{stage,reason}`, `nexir_allocator_resident_bytes` (Linux)
Follower catch-up	`nexir_raft_retention_state{peer_id}`, `nexir_raft_retention_catchup_eta_seconds{peer_id}`, `nexir_raft_retention_releases_total{reason}`, `nexir_raft_retention_disk_pressure`, `nexir_snapshot_build_deferred`
Storage	`nexir_storage_io_queue_wait_seconds{pool}`, `nexir_mvcc_safe_point_lag`
Snapshots	`nexir_active_snapshot_receives`, `nexir_snapshot_disk_preflight_total`, `nexir_retained_snapshot_bytes`
TLS	`nexir_client_tls_handshake_total`, `nexir_cluster_tls_handshake_total`, `nexir_tls_cert_not_after_timestamp_seconds{surface}`
Disk footprint	`rocksdb_directory_size_bytes{db="state"}`, `rocksdb_directory_size_bytes{db="raft_log"}`, `rocksdb_l0_files_by_family`, `rocksdb_pending_compaction_bytes_by_family`, `rocksdb_write_stopped`
Snapshot installs	`nexir_snapshot_receive_bytes_total`, `nexir_snapshot_receive_timeouts_total{phase}`, `nexir_snapshot_install_disposals_total`, `nexir_snapshot_install_stage_failures_total`
Garbage collection	`nexir_mvcc_compaction_filter_decisions_total{decision}`, `nexir_mvcc_gc_published_cutoff_index`, `nexir_mvcc_gc_eligible_candidate_backlog`, `nexir_mvcc_gc_candidates_drained_total`, `nexir_mvcc_gc_versions_removed_total`, `nexir_mvcc_gc_oldest_candidate_timestamp`
Diagnostics	`nexir_diagnostics_lost_records_total`, `nexir_diagnostics_truncated_records_total`

What healthy looks like: one node with raft_state 3 and the rest 1 (or 0 for learners); raft_term equal on every node and flat — a rising term means elections; nexir_raft_replication_lag_entries near zero on the leader; rocksdb_write_stopped at zero; the raft_log directory small at rest — a log that keeps growing is waiting on snapshots.

How failures look to clients

Nexir never degrades a write silently; it answers with a named error or not at all.

Client sees	What happened	What to do
`-MOVED <slot> <addr>`	The command needs the leader and this node is not it.	Reconnect to `<addr>`; cluster-aware clients do this themselves.
`-CLUSTERDOWN no known leader`	An election is in progress (a few seconds by default).	Retry with backoff.
`-CLUSTERDOWN write quorum unavailable`	Fewer than a majority of voters are reachable.	Restore a voter; until then no write commits.
`-ERR Server overloaded`	The node's write admission bound (`max_pending_write_tasks` minus the system reserve) is full.	Back off; reduce connections × pipeline depth.
`-TRYAGAIN memory pressure; request not submitted`	The leader is under memory pressure and refused a new write before it reached the log.	The write had no effect; retry with backoff and check the memory metrics.
`-ERR read refused: response memory not available`	The node-wide response budget had no room for the reply.	Nothing changed; retry later.
`-ERR server shutting down`	The node is stopping.	Reconnect elsewhere.
no reply	The node stopped or the connection broke before answering.	Outcome unknown; reconcile or retry idempotently (a retried `SET … EX` restarts its deadline).
`-ERR cluster administration commands are disabled`	An admin command reached a node with `enable_admin_commands` off.	Expected on a running cluster; enable only for a change.

Logging

NEXIR_ACCEPT_TERMS=1 RUST_LOG=info nexir
# warn for quieter operation; debug temporarily for diagnostics

Logs go to stdout/stderr. Container deployments should collect them through the runtime. Mutating cluster commands emit structured events with operation, target, outcome, membership index, and error fields.

Disk and data directories

Treat storage.path as opaque and monitor filesystem free space.
Never share a path between nodes or edit files inside storage.path.
Large reads and slow clients can hold older MVCC versions and delay reclamation.
Overwritten values are removed when the storage engine compacts their files; only deletes and TTL expiries create cleanup work of their own. Temporary disk growth is normal under churn.
A leader keeps extra replication log for a follower that is catching up, up to raft.log_retention_budget_bytes (32 GiB by default), and releases it when the next snapshot would not fit on disk. Size disks for it, or lower the budget on small disks.
A snapshot build that would not fit beside storage.min_free_disk_bytes is deferred and retried every 10 seconds instead of stopping the node; nexir_snapshot_build_deferred reads 1 while it waits. Free space to let it continue.
A snapshot install needs room for the incoming state, a checkpoint of it and the node's existing state at the same time, plus the configured reserve; the replication log can grow large while it waits for a snapshot. The four sizing checks and the measured peaks are in the configuration reference.

Unavailable followers

With a voting majority alive, restart the failed node using intact state. It catches up through log replication or internal snapshot. While it is behind, the leader keeps a bounded window of log for it so that it can replay rather than receive a second snapshot; nexir_raft_retention_state and nexir_raft_retention_catchup_eta_seconds on the leader show its progress, and -1 for the estimate means it is not catching up. No recovery time is guaranteed. If one node's storage is lost, preserve it for investigation, remove that member through the leader, restart it with an empty path, add as learner, and promote after catch-up.

Quorum loss

Writes and linearizable reads fail closed without a majority. Restore a voter when possible. For permanent majority loss, use the offline survivor procedure in Cluster operations; online membership changes cannot repair missing quorum.

Backups

There is no supported external backup/restore or point-in-time recovery interface. Internal snapshots are not backups. A stopped whole-directory cold copy is only a public-beta, same-version node recovery workaround and must never be taken from a live process.

Rolling maintenance

Verify quorum and zero lag.
Restart non-leader voters one at a time and wait for catch-up.
Transfer or step down the leader.
Restart the former leader.
Verify cluster state, quorum writes, metrics, and logs.

This is appropriate for restart-based certificate rotation. There is no migration between storage-format generations and no rolling upgrade across them; 0.1.0-beta.4 changed the layout (9 → 10), so a beta.3 directory is refused. 0.1.0-beta.5 changes no storage format, but an in-place upgrade from beta.4 was not validated. Review the release notes before changing Nexir versions.

Recommended alerts

No leader or repeated leader changes.
Replication lag, joint membership, or last-applied index stuck; a follower whose catch-up estimate stays at -1.
Memory admission closed (nexir_memory_admission_open at 0) or a deferred snapshot build that does not clear.
Connection rejection, backpressure, queue wait, or overload errors.
Snapshot validation/preflight failure or retained snapshot budget pressure.
Certificate expiry within your rotation window.
Free disk low, live data growing unexpectedly, cleanup errors, or safe-point lag.

Previous
Redis compatibility Next
Beta limitations →