Operate

Operations

Monitor cluster state, replication, resource pressure, snapshots, cleanup, and TLS from the first day of a Nexir deployment.

Start and stop

Start nexir in the foreground under your process supervisor. Configuration is loaded once at startup. Send SIGINT with Ctrl+C or SIGTERM for graceful shutdown; Nexir stops listeners and Raft, then flushes storage.

Restart an intact node with its original node ID, addresses, and storage path. For planned leader maintenance:

redis-cli --raw -p 16379 CLUSTER STEPDOWN
# wait for OK and verify the new leader
redis-cli --raw -p 16379 CLUSTER INFO

Health and readiness

redis-cli --raw -p 16379 PING
redis-cli --raw -p 16379 CLUSTER INFO
redis-cli --raw -p 16379 CLUSTER MEMBERS
redis-cli --raw -p 16379 CLUSTER IDENTITY

PING proves the RESP listener can reply. For readiness, also require a healthy initialized identity, known leader, expected membership, and acceptable lag. A local PING alone does not prove write quorum.

Metrics

curl http://127.0.0.1:9090/metrics

The endpoint defaults to 127.0.0.1:9090 and has no TLS or authentication. Key signals:

AreaMetrics
Leadershipnexir_raft_current_leader_id, raft_state, raft_term, nexir_raft_leader_changes_total
Replicationraft_last_log_index, raft_last_applied, nexir_raft_replication_lag_entries{peer_id}, nexir_raft_membership_joint
Clientsnexir_active_client_connections, nexir_client_connection_rejected_total, nexir_client_backpressure_total
Writes/readsnexir_pending_raft_write_tasks, nexir_raft_client_write_seconds, nexir_linearizable_read_barrier_seconds, nexir_linearizable_read_barrier_errors_total
Response buffersnexir_client_buffered_response_bytes, nexir_client_buffered_response_bytes_max, nexir_client_buffered_response_bytes_worst_case, nexir_client_response_permit_wait_seconds
Storagenexir_storage_io_queue_wait_seconds{pool}, nexir_mvcc_safe_point_lag
Snapshotsnexir_active_snapshot_receives, nexir_snapshot_disk_preflight_total, nexir_retained_snapshot_bytes
TLSnexir_client_tls_handshake_total, nexir_cluster_tls_handshake_total, nexir_tls_cert_not_after_timestamp_seconds{surface}

Logging

NEXIR_ACCEPT_TERMS=1 RUST_LOG=info nexir
# warn for quieter operation; debug temporarily for diagnostics

Logs go to stdout/stderr. Container deployments should collect them through the runtime. Mutating cluster commands emit structured events with operation, target, outcome, membership index, and error fields.

Disk and data directories

  • Treat storage.path as opaque and monitor filesystem free space.
  • Never share a path between nodes or edit files inside storage.path.
  • Large collection reads and slow clients can hold older MVCC versions and delay reclamation.
  • TTL logical removal and physical cleanup are asynchronous; temporary disk growth is normal under churn.
  • Snapshot install requires room for archive, extracted state, retained files, and the configured reserve.

Unavailable followers

With a voting majority alive, restart the failed node using intact state. It catches up through log replication or internal snapshot. If one node's storage is lost, preserve it for investigation, remove that member through the leader, restart it with an empty path, add as learner, and promote after catch-up.

Quorum loss

Writes and linearizable reads fail closed without a majority. Restore a voter when possible. For permanent majority loss, use the offline survivor procedure in Cluster operations; online membership changes cannot repair missing quorum.

Backups

There is no supported external backup/restore or point-in-time recovery interface. Internal snapshots are not backups. A stopped whole-directory cold copy is only a public-beta, same-version node recovery workaround and must never be taken from a live process.

Rolling maintenance

  1. Verify quorum and zero lag.
  2. Restart non-leader voters one at a time and wait for catch-up.
  3. Transfer or step down the leader.
  4. Restart the former leader.
  5. Verify cluster state, quorum writes, metrics, and logs.

This is appropriate for restart-based certificate rotation. Storage-format upgrade compatibility is not guaranteed; review release notes before changing Nexir versions.

Recommended alerts

  • No leader or repeated leader changes.
  • Replication lag, joint membership, or last-applied index stuck.
  • Connection rejection, backpressure, queue wait, or overload errors.
  • Snapshot validation/preflight failure or retained snapshot budget pressure.
  • Certificate expiry within your rotation window.
  • Free disk low, live data growing unexpectedly, cleanup errors, or safe-point lag.
Previous
Redis compatibilityNext
Beta limitations →