ADR 0019: One Openraft Consensus Engine

The SDK historically grew two distributed-persistence implementations: a custom config-store Raft-style engine in opc-persist and a custom majority-visible session coordinator. Splitting elections, voting, log matching, commitment, membership, read barriers, snapshots, and repair across SDK-owned algorithms multiplied failure modes and made qualification ambiguous. Combining Openraft with a custom majority writer in one authority path would be worse: either side could select different durable truth.

The SDK must still own domain state machines, persistence schemas, authenticated transport composition, bounded codecs, payload protection, metrics, and operator policy. Those are adapters around consensus, not reasons to implement consensus again.

Decision

Openraft is the only consensus engine permitted for SDK-owned distributed persistence authority.

opc-consensus exact-pins and re-exports the approved Openraft version. No domain crate imports Openraft directly.
Openraft exclusively owns election, term/vote state, leader authority, log matching, quorum commitment, membership transitions, linearizable read barriers, compaction, and snapshot lineage/install authority.
Domain adapters may implement deterministic commands and state machines, Openraft storage traits, bounded RPC encoding, authenticated peer routing, application journals/watch cursors, idempotent request outcomes, and redaction-safe status. They must not count votes, select a majority value, allocate an authoritative sequence outside client_write, or repair a distributed log through an independent algorithm.
Raw append, truncate, rebuild, term/vote mutation, membership mutation, and snapshot-install APIs are not production service surfaces. An offline migration may replace legacy state only under explicit bounded operator approval; it cannot run as a second live authority.
ConsensusSessionStore is the session adapter delivered by #127. QuorumSessionStore may remain only as a type alias to it.
ConsensusConfigStore is the config adapter migrated under #177. The custom config Raft modules, QuorumConfigStore, config TCP peer/server, and standalone consensus-node binary are removed rather than retained as a compatibility engine.

Both store adapters distinguish planned retirement from unplanned drain. Ordinary shutdown() retains the automatic bounded leadership handoff. shutdown_unplanned() is the explicit route for an embedder with a failed replication listener: revoke admission, retain the failure diagnostic, join transports, then drain without starting a handoff. It cancels outstanding preparation without retracting already dispatched RPCs. Every clone observes one retained engine/storage drain despite caller cancellation or timeout. Session native close proofs retain their existing active-incarnation, normal engine-stop and durable-cut checks; reopen never grants traffic authority by itself. No election, emergency-session, storage-format or operator-cleanup policy changes. Voluntary retirement remains subject to the consumer's emergency hold.

The shared engine also has one runtime and complete-call profile. opc-consensus owns the 2,000 ms heartbeat/AppendEntries/read-index ceiling, 5,000 ms Vote ceiling, [5,000 ms, 8,000 ms) election range, 10,000 ms snapshot/forward/read-barrier and operation ceilings, 30,000 ms listener ceilings, and the contained 1,500 ms cold-connect sub-bound. It also owns the replication payload, snapshot trigger/chunk, retained-log, and Tokio runtime choices. Session and configuration adapters select only their non-secret cluster label; they cannot silently drift to separate timing or runtime behavior.

Configuration admission may count a majority of fixed voters whose existing compatibility profile is mutually proved on authenticated connections, including the local voter. This is a compatibility gate, not another election or commit algorithm: Openraft still decides every vote, leader and committed index. New-to-new engine calls require proof on their actual connection before dispatch. The optional negotiated transport extension preserves the old handshake; legacy peers pass explicit probes and retain all-peer admission, without contributing verification credit to the new admission rule. Pristine formation also retains all-peer verification. Stored formats and profile values are unchanged. See the configuration admission contract for mixed-version operation and the legacy proof limitation.

Interim engine-source and release gate

The accepted one-engine rule applies to source selection as well as APIs. Until an official stable Openraft release contains per-campaign election-timeout resampling, the original source-build candidate exact-pinned https://github.com/openpacketcore/openraft revision f607e636406b16bd0ad7925dbb631da1b7a4cd96 (signed tag opc-v0.9.24-election-resampling-1). The dependency is a full immutable rev, not a mutable branch or tag, and locked metadata must resolve only openraft/openraft-macros 0.9.24 from that revision.

SDK-741 advanced the immutable pin to signed commit 702ff263d2122820e09d35abd53518b01c85823a. This descendant preserves the election repair and fixes snapshot-to-log handoff and failed-target purge retention. It also joins former-leader replication readers before conflict truncation can remove their uncommitted suffix. Independent engine and SDK regressions retain the original one-second negative guard, strict storage errors and configured operation deadlines. The exact package versions, features, source-build closure and consensus authority remain unchanged. Historical HA profiles still bind the original revision above; current manifest/lock/metadata checks independently bind the new candidate. This source change does not confer qualification.

The shared dependency update advances the candidate to 2e1fb7455752455a3505b354d003d6effa55985d, which incorporates release-0.9 through v0.9.25 and the fork's bounded apply, joined replication retirement and fatal completion repairs. It also retires obsolete leadership and mismatched campaigns on an accepted newer self-vote, and paces legal append responses that acknowledge no new entries with a cancellable interval. The original range, request identity and deadline remain unchanged. The exact openraft/openraft-macros versions are now 0.9.25. Frozen qualification profiles retain their original versions and revisions; the current manifest, lock and metadata checks bind this candidate separately. Dependency tests do not establish SDK qualification.

Registry 0.9.24 SDK one-shot leader-loss runs happened to pass. They do not invalidate the deterministic scripted engine regression or the historical observed-leader split-vote in the multi-process qualification harness: the registry implementation reused one sampled timeout across campaigns. The forked engine resamples every campaign without adding SDK election, vote, leader-lease, or quorum logic.

Pre-Vote retains rolling-election compatibility in the fork. An eligible real campaign with a shorter log can allow an upgraded voter to start ordinary voting after a bounded inquiry round, using the observed actual candidate term. This does not grant a speculative vote or change the SDK transport contract. The self-vote must be durable, actual joint majorities remain necessary, and lease, context and peer-generation fences discard obsolete evidence. Silence alone does not cause fallback. The preceding fork is a test-only dependency of the fork's mixed-version regression; it is not another production engine.

A published crates.io manifest cannot preserve this git revision. Therefore the exact 26-crate transitive normal reverse-dependency closure rooted at opc-consensus, opc-session-store, and opc-persist is source-build-only and publish = false. Metadata/profile tests and scripts/publish-order.py derive and check the closure; the other 51 workspace crates keep their existing publication status. Remove the gate only when all three conditions hold:

an official stable Openraft release contains the fix;
the workspace uses an exact registry version and checksum; and
the complete #143 profile is requalified against that revision.

This gate does not graduate the HA profile. Its machine-readable maturity stays experimental, qualification_complete stays false, and #143 remains an unresolved dependency.

Kubernetes controller leader election, gNMI master arbitration, local single-node SQLite transactions, session fencing leases, caches, and test fakes do not become Openraft concerns unless they start deciding distributed durable state authority.

Encryption and HKMS boundary

For configuration persistence the production composition is:

application -> HKMS-backed encryption -> ConsensusConfigStore
            -> Openraft -> SQLite and Openraft snapshots

The session composition follows the same outer-protection rule through its encryption or remote-sealing wrapper. Consensus commands contain already sealed envelopes. The config adapter additionally masks audit values and finalizes the audit chain before proposal. Openraft therefore persists and replicates sealed ciphertext and redacted finalized audit content, never plaintext, an HKMS/KMS provider, a provider or key handle, or raw key material. Follower apply, replay, catch-up, request outcomes, and snapshot installation do not call a provider. Reads decrypt only after crossing back through the outer protection adapter.

Provider unavailability blocks a new plaintext protection operation before client_write and can block decryption, but it does not prevent Openraft from replicating or recovering already sealed state.

The envelope marker alone is insufficient. Each adapter validates its canonical envelope/AAD representation and record-visible binding before proposal and again when persisted state is decoded. A durable authority marker fences public standalone config mutations after Openraft claims a database; each domain may impose a stricter raw-storage fence.

This is payload-envelope encryption. Unless a separate storage layer says otherwise, consensus metadata, routing fields, terms/indexes, timestamps, ownership/fence metadata, request IDs, and envelope key IDs are not full-database encrypted.

Shared transport boundary

opc-consensus owns the bounded, transport-neutral ConsensusPeer and ConsensusRpcHandler contracts. Domain crates provide handlers and consume peers; they do not provide competing sockets. The production mTLS listener and peer, live certificate authentication, framing, and connection lifecycle are owned by opc-session-net and the CNF composition. A real three-node ConsensusConfigStore integration forms Openraft and commits/linearizably reads through RemoteSessionConsensusPeer/SessionConsensusServer over mTLS, proving that config uses this shared boundary in process.

The #177 migration deliberately deletes the private opc-persist TCP/mTLS stack. It does not create another endpoint or another credential-rotation API. Certificate and trust-bundle rotation remains the shared transport's existing responsibility, including trust overlap, fresh authentication, connection drain, readiness gating, and old-trust retirement. #163 real-mTLS tests cover finite retained-connection retirement, overlapping trust, complete replacement handshakes, and old/wrong-scope trust rejection. #164/#143 retain the broader fleet production qualification gates.

Migration rule

An adapter must never reinterpret a nonempty legacy consensus log as Openraft metadata or use startup heuristics to choose a legacy tail. Pristine state may be claimed directly. Nonempty legacy authority fails closed unless the fleet is offline and an operator explicitly approves one coherent applied snapshot.

Config recovery must bind the complete source file's exact SHA-256 checksum, the exact latest applied transaction ID and config version, and an explicit DiscardUnknownAppendedSuffix disposition. The source must be checkpointed with no nonempty WAL. Integrity, required tables, audit chains, config envelopes, checksum, and chain head are verified before the target is replaced and the Openraft marker is created in one immediate SQLite transaction. Every unprovable target suffix is discarded; it is never merged or promoted.

For config authority, recovery binds the complete source file's exact SHA-256 checksum, latest applied transaction ID and config version, and an explicit DiscardUnknownAppendedSuffix disposition. The source must be checkpointed with no nonempty WAL. Integrity, required tables, audit chains, config envelopes, checksum, and chain head are verified before the target is replaced and the Openraft marker is created in one immediate SQLite transaction. Every unprovable target suffix is discarded; it is never merged or promoted.

For session authority, the operator-safe procedure is the offline, full-fleet campaign in the legacy recovery runbook. It may copy an operator-selected immutable checkpoint, but Openraft alone commits the recovery epoch and returns the fleet to service.

A local conversion transaction is not a fleet transaction. Operators must drain every old authority, preserve one coherent authority decision, convert members under a coordinated rollout, and keep untouched pre-migration backups.

Migration is one-way. Rollback to a removed engine is only a stopped-fleet restore of those pre-migration backups. Removing Openraft tables or markers, or attempting to reconstruct legacy logs from Openraft state, is prohibited.

Consequences

The SDK accepts the dependency and integration cost of Openraft once in opc-consensus and no longer maintains competing distributed-safety algorithms. Config and session tests focus on deterministic state-machine and adapter behavior; shared engine and transport qualification can exercise one set of election, replication, membership, read, and snapshot semantics.

The interim git source also makes the affected release closure intentionally source-build-only. This is a bounded distribution cost, not permission to mix registry and forked Openraft consumers or to reintroduce SDK-owned consensus logic.

#127 and #177 close the single-engine implementation transition. They do not by themselves declare either domain carrier-production ready. Recovery, restore, credential lifecycle, real-network compatibility, restart/rejoin, resource, soak, and candidate release evidence retain their domain-specific gates.

Evidence

crates/opc-consensus/
crates/opc-session-store/src/consensus/
crates/opc-session-store/tests/consensus_openraft.rs
crates/opc-persist/src/consensus/
crates/opc-persist/tests/consensus_openraft.rs
crates/opc-amf-lite/tests/config_consensus_encryption.rs
crates/opc-session-net/src/consensus.rs
crates/opc-session-net/tests/consensus_transport.rs
crates/opc-session-testkit/tests/qualification_multiprocess.rs
crates/opc-session-testkit/qualification/v2/session-ha-profile.json
scripts/publish-order.py
docs/adr/0002-config-store-consensus-ha.md
docs/adr/0003-session-store-quorum-replication.md
docs/consensus-operator-runbook.md

OpenPacketCore SDK Documentation