OPC-SDK-RFC-004: High-Performance Session Store

Status: Draft; commit authority implemented, production qualification pending
Version: 2.1.0
Date: 2026-07-14
Audience: SDK implementers, NF owners, data-plane engineers, reliability engineers

1. Abstract

This RFC defines opc-session-store, the SDK substrate for high-rate network function state such as PDU sessions, PFCP associations, TEID mappings, QoS flow state, handover coordination metadata, and data-plane derived counters that need controlled persistence.

The initial draft correctly identified the need for partitioning, local-first operation, and distributed leases. It was not strict enough for 5G continuity: last-writer-wins based on synchronized clocks is not safe for authoritative session state. This version requires monotonic fencing tokens, compare-and-set updates, owner epochs, explicit handover state transitions, and a documented consistency model per data class.

The #127 implementation uses one shared Openraft engine for intra-cluster election, voting, log matching, commitment, membership, snapshots, and linearizable-read authority. ConsensusSessionStore is the operational store; QuorumSessionStore is a compatibility alias, not a second quorum algorithm. This RFC does not claim production qualification: #128 supplies current-format recovery and #129 supplies the audited offline legacy-fork campaign, while #133 supplies bounded restore from the Openraft-applied state without becoming readiness evidence by itself. Connection reauthentication and retained-connection continuity are implemented under #163. Distributed fleet qualification (#164/#143) remains a gate. Single-host three- and five-process tests cover trust overlap/removal and one bounded synthetic admission-loss/malformed-last-good plus short-lived-SVID-expiry recovery slice under mixed lease/CAS mutation, linearizable-read, watch, complete-restore, readiness, and connection-recycling traffic. Only typed backend-unavailable or operation-outcome-unavailable terminal results may enter qualification recovery. Mutation or lease outcomes that can make authority ambiguous discard the prior guard, reacquire same-owner authority at a strictly higher fence, and validate the exact scheduled record. Read-only get, restore-scan, and readiness outcomes retain the already-proven guard and validate that same exact record without minting unnecessary fencing authority. Evidence binds this routing as stage-aware-known-authority/v1. The fixed schedule drops one successful release response per mutator, allows eight outcomes per node, uses the fixed 26-second two-election-plus-operation transition envelope per recovery episode, and applies a 50 ms retry delay; phase completion requires every interruption to be reconciled. Lease loss, unexpected state, and invariant failures fail closed. The admission-loss exact-address restart is watcher-only before exit and joins the mutator set only after bounded journal reconciliation. Recovering a committed generation does not rearm the once-per-logical-mutator injection. One additional schedule-v4 phase kills a stable follower uncleanly while its mutation and watch tasks are active. Survivors advance committed canary and mixed traffic; the same-disk, exact-address restart must reconcile a bounded gap-free journal, prove the exact generation/owner/fence/payload, and resume at a strictly higher same-owner fence under the versioned same-disk-exact-address-active-mutator/v3 profile. That profile independently bounds termination/reaping at 5 seconds, outage/survivor progress at 26 seconds, replacement-child startup at 45 seconds, Openraft recovery/readiness observation at 37 seconds (a 26-second recovery envelope plus one reserved 11-second final all-voter readiness round comprising a 10-second backend operation and 1 second of bounded local result delivery), journal reconciliation at 25 seconds, and higher-fence mutation resume at 26 seconds. The sequential stages compose to a 164-second crash-to-resume ceiling, but each stage fails at its own deadline. This retains the v1 deadline-composition fix and corrects v2's stranded readiness-observation tail; it does not qualify a broader restart matrix or deployed production readiness. The tests do not cover deployed partitions, a broader restart/fault matrix, resource/soak, remote HKMS, deployed CNFs, or signed release evidence. Generic CRL/OCSP/denylist revocation is not implemented.

2. Scope

2.1 In Scope

  • Per-session control-plane state needed by AMF, SMF, UPF, and related NFs.
  • Data-plane lookup state that can be safely snapshotted or reconstructed.
  • Lease and fencing mechanisms for single-owner session mutation.
  • Local cache and distributed backend abstraction.
  • Geo-redundant replication for disaster recovery and warm standby.
  • Serialization, encryption, integrity, TTL, metrics, and fault injection.

2.2 Out of Scope

  • Configuration management. See RFC 001.
  • Packet parsing and protocol codecs. See RFC 005.
  • Full 3GPP procedure implementation. This RFC provides storage primitives and state-machine support used by NF-specific procedure logic.
  • Hard real-time packet forwarding in the remote store. Packet fast paths must use local data-plane structures.

3. Design Goals

3.1 Security

  • Encrypt session state before it leaves process memory unless the backend is explicitly trusted by profile.
  • Bind encrypted records to tenant, NF kind, session key, generation, and state type through AEAD AAD.
  • Prevent stale owners from overwriting newer session state.
  • Prevent cross-tenant key collision or data exposure.
  • Redact SUPI/GPSI and other subscriber identifiers in logs by default.

3.2 Performance

  • Keep packet forwarding off the remote store path.
  • Support 100,000+ session updates/second per NF replica for local in-memory or batched backend profiles.
  • Keep hot read p99 below 1 ms for local-cluster operations where the selected backend can meet it.
  • Provide bounded allocation and zero-copy or low-copy decode for common session reads.
  • Support batching, pipelining, and async replication without sacrificing fencing correctness.

3.3 Maintainability

  • Separate storage API, lease API, serialization, encryption, and replication.
  • Require backend capability declarations so NF code does not assume semantics a backend cannot provide.
  • Use typed session records instead of arbitrary blobs at module boundaries.
  • Provide a deterministic testkit for split-brain, failover, and handover races.

3.4 Functionality

  • Support create, get, update, delete, compare-and-set, TTL refresh, lease, renew, release, snapshot, and replication.
  • Support session handover prepare/activate/abort flows.
  • Support backend implementations for in-memory, Redis, Aerospike, and optional strongly consistent stores.
  • Support region-aware replication and recovery.

4. State Classes

The SDK distinguishes state by consistency need:

ClassExamplesConsistency Requirement
authoritative-sessionPDU session owner, AMF/SMF ownership, handover phaseSingle writer with fencing
dataplane-lookupTEID to session mapping, FAR/QER/PDR snapshotsLocal atomic snapshot, rebuildable
replicated-drWarm standby copy of session recordsAsync, ordered by generation
telemetry-derivedCounters, rates, last seen timestampsMergeable or lossy
ephemeral-procedureTemporary handover transaction stateTTL, fenced owner

Only telemetry-derived state may use last-writer-wins based on timestamps. Authoritative session state MUST NOT use wall-clock LWW.

5. Session Identity

Session keys MUST be tenant-scoped and type-scoped:

#![allow(unused)]
fn main() {
pub struct SessionKey {
    pub tenant: TenantId,
    pub nf_kind: NetworkFunctionKind,
    pub key_type: SessionKeyType,
    pub stable_id: StableId,
}
}

Examples:

  • SUPI-derived subscriber context key.
  • PDU session ID plus SUPI hash.
  • TEID mapping key.
  • PFCP session SEID key.
  • Handover transaction key.

StableId has a structural 1..=64-byte invariant shared by every local, SQLite, cache, quorum, restore, replication, watch, and session-net boundary. Valid pre-existing bytes retain their exact wire and SQLite representation. Empty or wider legacy values are not silently truncated or hashed; they fail the mandatory pre-upgrade audit and hydration.

Raw SUPI/GPSI MUST NOT be used directly as a backend key in production. Subscriber-derived keys MUST use StableId::derive_hmac_sha256 with a 16-through-64-byte tenant-specific KMS/HSM privacy key and one product-defined 1-through-256-byte canonical subject representation. The canonical profile is full-width 32-byte HMAC-SHA256 over the SDK domain followed by unsigned 64-bit big-endian length-prefixed tenant and subject bytes. Truncated keyed digests are not supported.

See docs/session-store-stable-id-migration.md for the required count-only audit, coordinated remediation, snapshot handling, and rollback procedure.

5.1 Owner and Session-Key Type Invariants

An OwnerId and the name of a deployment-specific SessionKeyType MUST each contain 1 through 128 UTF-8 encoded bytes. The limit applies to encoded bytes, not characters. Empty and oversized values MUST be rejected at construction and decode boundaries without including the raw value in an error.

SessionKeyType::Other MUST contain a structurally validated CustomSessionKeyType; callers MUST use the fallible SessionKeyType::other for runtime custom names. These canonical persisted strings are reserved for the corresponding well-known variants and MUST NOT be constructible as custom values:

  • subscriber-context
  • pdu-session
  • teid-mapping
  • pfcp-seid
  • handover-transaction

Parsing a reserved string MUST produce the well-known variant. Display, serialization, SQLite identity, key-digest input, and ordering MUST use the same canonical string; ordering MUST therefore be string ordering across known and custom values, not enum declaration order.

The invariant MUST be applied by Serde, SQLite record and restore hydration, active-lease reads before acquire/renew/release/fenced mutation, replication-log hydration including nested operations, and session-net request and response decode. Invalid persisted or remote data MUST fail closed before mutation or caller exposure. Diagnostics MUST be fieldless or fixed and MUST NOT expose the owner, key type, stable ID, row, transaction, or raw entry.

Valid protocol-v4 values retain their JSON byte-array shape. This does not make the change rolling-compatible: replacing SessionKey::stable_id: Bytes with StableId and replacing Other(String) with Other(CustomSessionKeyType) and making SessionKeyType::other fallible is a Rust source break. Both HandoverEnvelope::unpack_raw and HandoverSessionRecord::unpack_raw now return a typed Result; both public unpack_json methods change their error type, and HandoverError adds an InvalidEnvelope variant. Packers now write the versioned OPCH form while readers retain a bounded original/bare migration path. Rejecting values an older v3 peer could emit is also a semantic-admission break. Protocol v4 now binds that rule in its exact fixed-width DTO and handshake profile. Operators MUST stop, upgrade, and restart every session-net client, server, and protection wrapper plus every NF/product handover reader or writer as one coordinated unit. The v4 handshake does not make persisted OPCH bytes readable by old code.

5.2 Bounded Legacy SQLite Audit

Before a new binary opens persisted SQLite state written by an older SDK, the operator MUST drain all writers and run:

opc-session-store-audit identity-invariants \
  --database PATH \
  --max-rows N \
  --max-entry-json-bytes N \
  --max-total-json-bytes N \
  --expiry-reference 2026-07-13T18:00:00Z

All limits MUST be explicit and non-zero, and the per-entry JSON-byte limit MUST NOT exceed the total JSON-byte limit or SQLite's signed i64 length range. The command opens an existing database read-only/query-only, reads one consistent snapshot in fixed 256-row pages, applies the row budget across session_records, leases, key_fences, and session_replication_log, and bounds individual and cumulative replication JSON before strict typed decode and domain validation.

Report schema version 4 is count-only. It contains the supplied limits, the expiry reference, per-table scanned counts, counts for invalid owner fields, invalid session-key type fields, invalid stable-ID fields, invalid replication transaction-ID fields, invalid replication entries, and invalid relational record-expiry fields, and at most one bounded incomplete reason. Relational expiry MUST be classified against the reported reference; each nested legacy CAS expiry MUST be classified against its immutable replication-entry timestamp. Relational stable-ID checks read only SQLite type and length. It MUST NOT contain the database path, row identity, tenant, owner, key type, stable ID, payload, transaction, rejected row timestamp, or raw JSON. Omitting --expiry-reference selects current UTC, but a migration campaign MUST record and pass one explicit RFC 3339 reference so repeated audits are reproducible. The stable command outcomes are:

  • compliant on stdout with exit 0;
  • violations_found on stdout with exit 1;
  • incomplete on stdout with exit 2; or
  • redacted error on stderr with exit 2.

Only compliant after a complete snapshot inspection permits the identity portion of the upgrade to continue. violations_found, incomplete, and error MUST block startup. An incomplete audit reports one of row_budget_exceeded, entry_json_budget_exceeded, total_json_budget_exceeded, unsupported_schema, database_read_failed, or counter_overflow. The operator MAY increase budgets and re-audit, but the SDK and audit MUST NOT truncate, rename, normalize, delete, repair, or rewrite invalid identity or replication state automatically. A violation requires a separately reviewed product migration that preserves authoritative identity and history, or audited store replacement, followed by another complete audit. Every retained SQLite snapshot or restore/rebuild image that can become authoritative MUST pass the same audit. The identity procedure is docs/session-store-stable-id-migration.md; absolute-expiry re-authoring, OpenRaft recovery, and rollback are defined by docs/session-store-record-expiry-migration.md.

The identity audit MUST NOT be treated as a handover-payload preflight. It does not classify live payloads or payload bytes inside nested CAS log operations, so compliant says nothing about legacy envelope/bare compatibility. Every product using HandoverEnvelope MUST separately preflight the complete drained and decrypted replay population: live records, recursively nested replication log and snapshot records, restore/rebuild sources, and every retained copy that can become authoritative. It MUST use unpack_raw_with_format or typed unpack_json_with_format and verify the syntactic result against snapshot provenance and product payload semantics; decoder success alone is insufficient. A rejected or unprovable value MUST be resolved by a reviewed product migration or store replacement before startup; automatic guessing/truncation is forbidden.

This bounded identity admission closes #135's scoped model/persistence boundary. Protocol-v4 fixed-width wire admission is implemented under #134, and #127 now supplies Openraft durable commit authority. #128 supplies current-format divergence recovery and #129 supplies explicit offline legacy-fork recovery. #133 supplies bounded snapshot-bound applied-state restore; production qualification (#143) remains open. #161 atomic reload, #162 coherent material epochs, and #163 connection reauthentication are implemented; #164 fleet qualification remains under umbrella #158; payload-protection-key rotation and distributed production evidence remain #143.

6. Backend Capability Model

The initial get/set/delete trait is too weak. Backends MUST declare capabilities:

#![allow(unused)]
fn main() {
pub struct BackendCapabilities {
    pub atomic_compare_and_set: bool,
    pub monotonic_fencing_token: bool,
    pub per_key_ttl: bool,
    pub server_side_lease_expiry: bool,
    pub ordered_replication_log: bool,
    pub batch_write: bool,
    pub watch: bool,
    pub max_value_bytes: usize,
}
}

Carrier profiles MUST reject a backend for authoritative-session state unless it supports atomic compare-and-set and monotonic fencing tokens or an adapter can provide equivalent semantics.

7. Storage API

#![allow(unused)]
fn main() {
#[async_trait::async_trait]
pub trait SessionBackend: Send + Sync {
    async fn capabilities(&self) -> BackendCapabilities;

    async fn get(&self, key: &SessionKey)
        -> Result<Option<StoredSessionRecord>, StoreError>;

    async fn compare_and_set(&self, op: CompareAndSet)
        -> Result<CompareAndSetResult, StoreError>;

    async fn delete_fenced(&self, key: &SessionKey, fence: FenceToken)
        -> Result<(), StoreError>;

    async fn refresh_ttl(&self, key: &SessionKey, fence: FenceToken, ttl: Duration)
        -> Result<(), StoreError>;

    async fn batch(&self, ops: Vec<SessionOp>)
        -> Result<Vec<SessionOpResult>, StoreError>;
}
}

set without fencing is allowed only for state classes that explicitly do not require authoritative ownership.

7.1 TTL Admission and Deadline Arithmetic

The SDK-wide maximum for a session or lease TTL is the public MAX_SESSION_TTL, exactly 365 days. Duration::ZERO MUST be accepted and means immediate expiry. The exact maximum MUST be accepted. Any larger value MUST be rejected with StoreError::InvalidSessionTtl for store operations or LeaseError::InvalidSessionTtl for lease operations.

A zero-duration acquire MAY consume a fence, credential, and replication-log position before the lease is observed expired. Callers MUST use explicit release for revocation and MUST NOT treat a zero TTL as transaction rollback.

The ceiling accommodates long-lived packet-core sessions and planned maintenance or disaster-recovery windows while preventing a malformed value from creating an effectively permanent lease. A product profile MAY enforce a smaller operational limit.

This section bounds Duration inputs. Caller-authored absolute record expiry has the related but distinct authority contract in §7.2.

validate_session_ttl defines the duration check and checked_session_deadline defines conversion and deadline calculation. Implementations MUST convert seconds and subsecond nanoseconds using checked integer arithmetic and MUST use checked timestamp addition. Floating-point duration conversion, saturating/clamping an invalid input, and panicking timestamp arithmetic are forbidden.

Validation MUST occur before any application/backend effect for direct acquire, renew, or TTL-refresh calls; each TTL-bearing operation nested in a batch or replication entry; forwarding, encryption, cache, or quorum adapters; local Fake/SQLite backends; and session-net client/server admission. A client MUST reject before resolver or network work. A server necessarily receives and decodes the request, but MUST reject before backend dispatch and MAY return the typed error on that authenticated connection. Repeating the check at each public or trust boundary is intentional: direct callers and older peers must fail closed even if an outer layer omitted validation.

The new errors are public enum variants, so external exhaustive matches MUST be updated. Protocol v4 introduced their private fixed-width DTOs in error revision 1; current v5 error revision 9 retains those encodings and adds bounded expiry-preflight and topology-authority outcomes. The exact direct v5 profile is wire-schema revision 7/error-set revision 9; every non-current direct profile combination (including error revision 8 or older) MUST be rejected during the exact handshake. Deployments MUST use the coordinated v5 rollout in §12.3.

7.2 Absolute Record Expiry Authority

For a mutation with authority reference T, a finite StoredSessionRecord::expires_at MUST be accepted when it is in the past, equal to T, or no later than T + MAX_SESSION_TTL. The exact upper bound MUST be accepted and one nanosecond later MUST be rejected. Addition near the timestamp range maximum MUST saturate for comparison and MUST NOT unwind. MAX_RECORD_EXPIRY_CLOCK_SKEW is zero; an implementation MUST NOT silently extend retention to accommodate caller/coordinator clock skew. Deployments MUST synchronize coordinator clocks and a product MAY impose a smaller horizon.

expires_at = None is intentional non-expiring state. It MUST be accepted for AuthoritativeSession, DataplaneLookup, ReplicatedDr, and TelemetryDerived. It MUST be rejected for EphemeralProcedure, because that profile requires per-key expiry to collect abandoned procedure state. A violation MUST return the fieldless StoreError::InvalidRecordExpiry without the record, timestamp, key, or peer-controlled detail.

A direct Fake or SQLite backend MUST capture its injected clock once before a CAS or whole-batch preflight. All CAS slots MUST be checked before any slot can mutate. A forwarding, cache, or crypto wrapper MAY delegate an inner backend's explicit reference; it MUST NOT invent one from its own process clock. A remote or consensus client without coordinator authority MUST perform the time-independent None/state-profile check and leave the finite verdict to the authenticated mutation coordinator.

A legacy ReplicationEntry MUST validate every nested CAS against the entry's immutable timestamp, including replay and rebuild before mutation. This is a reproducible compatibility reference, not production consensus authority. The production OpenRaft leader MUST capture the command logical time, validate the CAS before proposal, and commit that same time with the command. Admission, state-machine apply, replay, follower apply, and journal publication MUST repeat the deterministic check against committed command metadata. A follower's wall clock MUST NOT alter the verdict. Authenticated cluster/configuration identity, leader term/membership, commitment, and applied index supply the coordinator authority described in §12.4.

Profile rejection MUST precede cache invalidation, provider work, or backend dispatch. A wrapper above remote/consensus authority MUST obtain a bounded, payload-free authenticated authority preflight before cache invalidation, provider/HKMS work, sealing, or backend dispatch. The authenticated CAS/batch dispatcher MUST repeat the preflight before idempotency admission. Invalid input and preflight timeout/unavailability perform no provider work or requested mutation; retry is safe because only a consensus logical-time floor may have committed. This rule does not change payload encoding, AAD, key selection, HKMS/KMS placement, or encryption at rest.

Existing valid row and JSON representations are unchanged. Legacy admission, product-aware re-authoring, OpenRaft recovery, and rollback MUST follow docs/session-store-record-expiry-migration.md. The audit MUST be run over a drained snapshot with one recorded --expiry-reference; runtime and audit MUST NOT guess intent, clamp a far-future value, or edit OpenRaft history in place.

8. Record Format

#![allow(unused)]
fn main() {
pub struct StoredSessionRecord {
    pub key: SessionKey,
    pub generation: Generation,
    pub owner: OwnerId,
    pub fence: FenceToken,
    pub state_class: StateClass,
    pub state_type: StateType,
    pub expires_at: Option<Timestamp>,
    pub payload: EncryptedSessionPayload,
}
}

generation is a monotonic per-session version. Every authoritative update MUST increment it atomically.

9. Lease and Fencing

9.1 Lease API

#![allow(unused)]
fn main() {
#[async_trait::async_trait]
pub trait SessionLeaseManager: Send + Sync {
    async fn acquire(&self, key: &SessionKey, owner: OwnerId, ttl: Duration)
        -> Result<LeaseGuard, LeaseError>;

    async fn renew(&self, lease: &LeaseGuard, ttl: Duration)
        -> Result<LeaseGuard, LeaseError>;

    async fn release(&self, lease: LeaseGuard)
        -> Result<(), LeaseError>;
}

pub struct LeaseGuard {
    pub key: SessionKey,
    pub owner: OwnerId,
    pub fence: FenceToken,
    pub acquired_at: Timestamp,
    pub expires_at: Timestamp,
}
}

9.2 Fencing Rules

Every successful lease acquisition MUST produce a monotonic fencing token for that session key. Backends MUST reject any write with a token lower than the current recorded token.

This prevents an old owner whose lease expired during a pause or partition from overwriting a newer owner after it resumes.

9.3 Lease Expiry

Lease expiry alone is not correctness. It is only a liveness mechanism. Safety comes from fencing.

Rules:

  • Lease TTLs MUST satisfy the 365-day bound in §7.1; zero is API-valid and creates a guard whose deadline is immediate, but does not satisfy the operational sizing rule below for an active owner.
  • Lease TTL MUST be longer than worst-case expected procedure pause plus backend failover detection time.
  • Renewals MUST happen before 50 percent of TTL elapsed by default.
  • A failed renewal MUST stop authoritative writes immediately.
  • Owners MUST treat unknown lease state as lost.
  • Stale writes MUST fail with a distinct StaleFence error.

9.4 Backend Notes

  • Redis implementations MUST use atomic Lua scripts or equivalent server-side transactions for acquire, renew, and fenced CAS. Redis deployments that can lose acknowledged writes during failover MUST NOT be used for strict authoritative state without an external consensus/fencing source.
  • Aerospike implementations SHOULD use generation checks and record UDF or transaction mechanisms where available.
  • In-memory backend is for single-process tests or single-replica development unless paired with a consensus lease manager.
  • Strongly consistent stores may be used for leases even when bulk state is in a faster backend.

10. 3GPP Session Continuity and Handover

10.1 Storage Guarantees Needed by Handover

5G handover procedures require avoiding duplicate authoritative writers while preserving continuity of PDU session and bearer/QoS state. The store must support:

  • Idempotent procedure steps.
  • Prepared-but-not-active state.
  • Activation with a fencing token.
  • Abort/rollback of prepared handover.
  • Recovery after source or target NF restart.
  • Detection of stale source updates after target activation.

A lease mechanism without fencing is not sufficient.

10.2 Handover State Machine

The SDK provides generic storage states:

#![allow(unused)]
fn main() {
pub enum HandoverPhase {
    Stable,
    Preparing { tx: HandoverTxId, target: OwnerId },
    Prepared { tx: HandoverTxId, target: OwnerId },
    Activating { tx: HandoverTxId, target: OwnerId },
    Active { owner: OwnerId },
    Aborting { tx: HandoverTxId },
}
}

NF-specific AMF/SMF/UPF logic maps 3GPP procedure messages to these states.

10.3 Procedure Rules

The session store MUST support these generic steps:

  1. Source owner holds a valid lease.
  2. Source creates Preparing record with current generation.
  3. Target acquires or is assigned a higher fence for activation.
  4. Target writes Prepared with expected generation.
  5. Activation performs a fenced CAS to Active { owner: target }.
  6. Source updates with old fence are rejected.
  7. Abort performs a fenced CAS back to Stable if activation did not complete.

All steps MUST be idempotent by HandoverTxId.

New handover envelopes MUST start with the OPCH magic, an exact format version, a bounded phase length, and the typed JSON phase. Every versioned header and phase is decoded strictly. For non-OPCH input, readers MUST apply this exact migration classifier:

  1. Fewer than four bytes are an unframed Stable payload.
  2. The first four bytes are a big-endian potential phase length. Zero, or a value from 1 through HANDOVER_PHASE_HEADER_MAX_BYTES (1,024) whose phase slice is truncated, is InvalidHeader.
  3. A complete phase slice within that bound is an original envelope only when it decodes as the current HandoverPhase. A JSON-looking invalid slice is InvalidPhase; a non-JSON-looking slice falls back to unframed Stable.
  4. A length above 1,024 is InvalidHeader when the bytes after the first word begin, after ASCII whitespace, like JSON. Otherwise it falls back to unframed Stable.

This bounded rule intentionally rejects ambiguous historical bare bytes and original envelopes whose phase is oversized or invalid under the current model. Syntax can also produce false positives: in a checkpoint known to predate OPCH, VersionedV1 is a bare-prefix collision, and an OriginalLengthPrefixed result MUST be confirmed from product provenance and payload meaning. Products MUST run the complete live/replay payload preflight in §5.2 and explicitly wrap the complete bytes of an authoritatively identified bare Stable value, or perform a reviewed semantic migration/store replacement. A successful transition writes the versioned form.

Writing the first OPCH record is a one-way migration barrier. A pre-OPCH reader silently interprets that record as opaque bare Stable data. Operators MUST NOT roll back binaries after the barrier unless the fleet remains drained and either one coherent fleet-wide pre-upgrade checkpoint is restored (with post-checkpoint mutations explicitly lost or reconciled) or every affected live and replayable payload—including nested logs, snapshots, and restore/rebuild sources—is reverse-migrated under a reviewed procedure. Every NF/product handover reader and writer MUST cross the barrier together; protocol negotiation alone cannot make the persisted payload backward-readable.

10.4 Packet Continuity

The session store does not itself guarantee zero packet loss. It provides the state consistency needed by NFs to implement make-before-break, buffering, or tunnel switching. NF-specific procedures MUST state their packet continuity behavior and evidence in RFC 006 reports.

11. Geo-Redundancy

11.1 Corrected Consistency Model

Asynchronous geo-replication is suitable for disaster recovery and warm standby. It is not sufficient for strict active/active mutation of the same authoritative session unless a higher-level single-owner protocol is used.

Authoritative state MUST use one of:

  • Home-region ownership per session.
  • Explicit ownership transfer with fencing.
  • A strongly consistent multi-region backend, if the deployment accepts the latency cost.

Wall-clock last-writer-wins is forbidden for authoritative session state.

11.2 Replication Log

Backends SHOULD expose an ordered replication log:

#![allow(unused)]
fn main() {
pub struct ReplicationEvent {
    pub key: SessionKey,
    pub generation: Generation,
    pub fence: FenceToken,
    pub state_class: StateClass,
    pub payload_digest: Sha256Digest,
    pub encrypted_payload: EncryptedSessionPayload,
}
}

Replication positions are 1-based and gap-free. Sequence zero is reserved for the empty-log head and MUST be rejected as an entry before mutation, external provider work, persistence, or transport dispatch. Rebuild input MUST be validated as one complete contiguous prefix before existing state is replaced. Sequence arithmetic and persistence-width conversions MUST be checked and fail closed without exposing entry contents in diagnostics.

Application of one replication entry is all-or-nothing across its complete operation tree. A failure in any later child MUST leave records, leases, fence/credential high-water marks, the log head and retained log, compaction state, and watcher-visible state exactly as they were before the entry. A successful compound entry MUST preserve child order, append the submitted outer entry once, and publish that outer entry to each eligible watcher only after the local backend transaction or atomic swap succeeds.

Whole-state rebuild MUST replay into an isolated stage or database transaction and replace prior state only after every supplied entry succeeds. Replay failure MUST preserve the complete prior state and established watch subscriptions. A successful rebuild MUST preserve those subscriptions but MUST NOT publish replayed history as new live append events; later locally successful appends remain observable normally. These are backend-local atomicity requirements. In the production HA profile, a caller MUST NOT invoke rebuild or append as an alternative authority path; only an Openraft-committed command or snapshot installation may replace authoritative state. #127 supplies that commit gate, #128 owns current-format reconciliation, and #129 provides the offline operator-directed legacy campaign documented in the recovery runbook.

An operator upgrading persisted state from an older SDK MUST audit every TTL-bearing replication entry before rollout. A legacy entry above 365 days fails closed during replay or rebuild under this contract; implementations MUST NOT silently clamp, discard, or rewrite it. Recovery or migration must follow a product-owned, audited procedure that preserves the authoritative-history contract.

For migration compatibility only, replicated absolute-deadline cross-field validation MAY admit at most one microsecond above the exact entry.timestamp + ttl result produced by an older seconds_f64 conversion. New deadline construction MUST remain exact. This tolerance does not increase MAX_SESSION_TTL; a larger mismatch MUST fail closed.

11.2.1 Bounded Protected Operation Trees

Each ReplicationEntry MUST contain at most MAX_REPLICATION_OPERATIONS_PER_ENTRY (256) operation nodes and MUST NOT exceed MAX_REPLICATION_OPERATION_DEPTH (16). The root operation is depth 1, and each child increases depth by one. Every node counts once toward the total, including each Batch container and every leaf operation. These rules apply to all variants, not only Batch and CompareAndSet.

Validation of an outbound entry or complete rebuild prefix MUST be iterative and MUST finish before payload transformation or backend dispatch. Validation of a complete returned page or item MUST finish before read-side transformation or caller exposure; the backend has necessarily already produced that read. A limit violation MUST return the fieldless StoreError::ReplicationOperationLimitExceeded; diagnostics MUST NOT reveal the observed count, depth, record, key, payload, provider detail, or tree shape. By-value public/wire boundaries MUST also dismantle rejected trees iteratively so the error path cannot recurse while dropping hostile nesting.

An encryption or remote-sealing wrapper MUST transform every CompareAndSet.new_record.payload at every permitted depth. Replicate and rebuild paths MUST stage the complete transformed entry or prefix before delegating to the backend. Replication-log and watch paths MUST decrypt or unseal each complete entry before exposing it. Traversal and reconstruction MUST be iterative and MUST preserve operation order and every non-payload field exactly.

Provider calls MUST run sequentially. If a late write-side provider call fails, earlier provider calls MAY already have occurred, but the wrapper MUST NOT delegate any part of the entry/prefix to its backend. If a read-side provider call fails, the wrapper MUST return an error without exposing a partially transformed entry or page; earlier provider calls, and earlier independent watch items already yielded, MAY have occurred.

This contract changed confidentiality semantics before the v4 boundary. A v3 peer built before this rule cannot decode the new error and its wrapper may forward a deeply nested CAS in plaintext/unsealed form. Protocol v4 rejects the older wire participant and pins both tree limits and error revision, but the handshake cannot attest that the product actually installed a protection wrapper. Operators MUST drain and upgrade every client, server, and protection-wrapper participant as one coordinated fleet and MUST verify wrapper composition before restoring traffic.

Persisted historical nested plaintext/unsealed payloads are not detected or scrubbed automatically. Before upgrade, an operator MUST audit operation-tree shape and payload encoding offline without emitting payloads into diagnostics. An affected entry already within the 16/256 limits MAY be explicitly rewritten or rebuilt through the configured encryption/sealing wrapper. An over-limit historical entry MUST fail before wrapper transformation and MUST NOT be fed to the new SDK unchanged, silently clamped, discarded, or split. It requires a separately reviewed offline migration that preserves the original atomic semantics, or store replacement under an audited product recovery procedure, before the new SDK reads the log. Rebuilding through the raw inner backend does not satisfy this requirement.

11.2.2 Intra-Cluster Consensus Authority

ConsensusSessionStore MUST be the only session-store implementation allowed to claim the quorum platform profile. QuorumSessionStore MAY remain as a source-compatibility type alias to that exact implementation, but MUST NOT own a parallel coordinator. Openraft, imported through the shared opc-consensus crate, owns election, voting, log matching, commit, membership, snapshot coordination, and linearizable-read authority. The SDK state machine owns only deterministic session semantics.

Public dynamic consensus construction, including membership-candidate construction, MUST be supported only on Linux. On every other platform it MUST return DynamicConsensusUnsupportedPlatform before topology or durable-state inspection and before creating a snapshot directory, consensus schema, or other consensus-owned filesystem state. The core initializer MUST independently return its typed UnsupportedPlatform storage error before those effects. Internal path-based snapshot helpers MUST NOT be treated as a portable consensus fallback. Standalone SqliteSessionBackend use remains cross-platform.

HA topology admission MUST start from the complete descriptor set and one explicit logical self ReplicaId. It MUST bind a cluster ID, the exact order-independent configuration digest over the cluster, epoch, and complete descriptor-fingerprint set, and a positive monotonic configuration epoch. Stable non-zero node IDs MUST be derived from cluster identity and the logical ReplicaId, and derived collisions MUST fail admission. Endpoints are routing data: a short logical ID such as epdg-app-0 can select a member whose endpoint is the FQDN epdg-app-0.epdg-app-quorum.epdg-gateway.svc.cluster.local:7443. No code may identify self, derive a vote, or rewrite a logical ID by comparing or shortening those endpoint strings.

The durable storage adapter MUST persist the Openraft vote and log, committed/applied/purged positions, membership, deterministic state-machine chain and logical time, and idempotent request outcomes. Application journal and watch events MUST become visible only after committed apply. A request ID MUST bind the semantic mutation intent; retry after ambiguous response delivery MUST return the original durable outcome, while reuse with different intent MUST fail closed. Caller-selected raw replication entries, whole-state rebuild, and lease sequencing MUST be rejected by this production adapter.

Snapshots MUST be bounded, checksummed, tied to the exact consensus identity, and installed atomically as one coherent state-machine image. They MUST contain only payloads already admitted by the protection wrapper described in §14.1. Automatic current-format reconciliation is supplied by #128. Pre-#127 persisted forks use #129's full-fleet, backup-before-mutation procedure and remain readiness-fenced until Openraft commits the recovery epoch.

The Linux snapshot namespace is a cooperative-service-UID trust boundary. The adapter MUST retain an O_RDONLY|O_DIRECTORY|O_NOFOLLOW|O_CLOEXEC|O_NONBLOCK directory descriptor at admission, require fstat ownership by the effective UID and mode & 022 == 0, and reject an insecure pre-existing directory. A missing namespace MUST be created 0700; SDK snapshot files MUST be created 0600. Post-admission namespace operations MUST be directory-FD-relative so a parent-path replacement cannot redirect accepted work. Durable snapshot rows name logical basenames, not a mutable parent path.

Supported writers are cooperative SDK processes under one dedicated service UID, serialized by the snapshot/database leases. Operators MUST use a private parent directory, must not share that UID with untrusted workloads, and MUST fail closed on owner or mode mismatch. POSIX ACL group-class masks MUST not restore group-class write authority. This contract excludes root, CAP_DAC_OVERRIDE, CAP_FOWNER, non-cooperating same-eUID actors, and writable aliases of the retained directory. It does not claim universal unlink-by-FD semantics or privileged-attacker resistance.

probe_durable_readiness MUST use the same bounded authority path as an authoritative read: discover or follow the current leader, execute Openraft's linearizable-read barrier against the admitted voting configuration, and wait until the local state machine has applied through the returned log index. A bound listener, completed TLS handshake, cached capability set, local SQLite availability, or successful single-node restore scan MUST NOT produce Ready. That method is engine/lab evidence only: it does not authenticate observed physical node, failure-domain, or durable-backing facts, and its Ready result MUST NOT authorize production traffic.

Attested-HA production session traffic MUST use topology admitted through ValidatedQuorumTopology::try_from_attested, require Quorum from the store's time-aware production profile, then require DurableReadinessScope::ProductionTopologyAttested and is_production_traffic_ready() from probe_production_durable_readiness or its refreshed-attestation form. The evidence MUST have AuthenticatedPlatform provenance and bind every exact member, service identity, observed physical node, failure domain, durable backing, descriptor digest, collector, cluster/configuration/epoch, observation time, and expiry. Verification MUST anchor a monotonic expiry. Each open store MUST retain a bounded nondecreasing wall-clock high-water and MUST recheck both authorities after asynchronous quorum work, so clock rollback, exact expiry, and an older delayed probe racing a newer evaluation fail closed. Identity and production provenance MUST be rejected before a supplied time can advance that high-water. Explicit-time calls on one store MUST use one trusted nondecreasing clock source. The high-water and monotonic anchor are process-local. Restart MUST NOT deserialize or reuse a prior VerifiedQuorumTopologyAttestation; it MUST authenticate evidence again against current time and establish a new monotonic anchor. The adapter-owned proof/replay policy decides whether a still-unexpired underlying proof may be re-presented or replacement evidence is required. Token non-serializability alone is not proof anti-replay.

The attested-HA readiness report MUST carry the bounded DurableReadinessScope. Attested-HA callers MUST require ProductionTopologyAttested and is_production_traffic_ready() and MUST NOT route an EngineOnly report into the traffic gate. Fixed durable quorums use the separate typed authority and placement results in §11.2.2.1; an attested report cannot be substituted for their immutable-voter authority. Every readiness result is point-in-time evidence, never an ownership lease. Products MUST continuously gate ownership publication, VIP/service advertisement, and traffic on fresh attested-HA readiness or, for a fixed durable quorum, a fresh fixed-quorum authority observation. Restore scans MUST execute only after the Openraft barrier and local apply. One absolute deadline MUST begin at the public restore entry and cover the barrier/apply path, blocking-worker and asynchronous connection admission, SQLite progress, and blocking-task join. Each page MUST examine no more than 4,096 live candidates plus one non-decoded lookahead, return no more than 1,024 records, an aggregate local 4 MiB + 64 KiB of stored-envelope payload, or 8 MiB of retained record/key/metadata/payload/cursor bytes, examine no more than 8 MiB of key/filter metadata, and obey the SQLite VM-step, wall-time, and cancellation budgets. Candidate/lookahead SQL MUST NOT select payload blobs; admitted records are fetched by exact primary key inside the same transaction. Scope filtering occurs inside the backend over that bounded candidate window, so an empty page is valid only with a different durable cursor and nonzero excluded/examined count. Pagination MUST seek the existing composite primary key; it MUST NOT use OFFSET or add a digest-order authority. The cursor MUST confidentially and authentically bind that seek key, backend epoch, record revision, logical-time snapshot, scope, and examined progress. Any edit or mismatch MUST return RestoreScanCursorStale before the record query rather than skip, merge, or guess. Restore method availability alone is not readiness evidence.

11.2.2.1 Fixed Durable Quorum Authority and Placement Resilience

ValidatedQuorumTopology::try_from_fixed_durable_quorum admits only an exact three- or five-voter immutable Openraft configuration. It requires distinct logical replica IDs, network endpoints, authenticated TLS identities, and declared backing identities. It does not promote a caller-declared failure domain into physical-placement evidence. The default rejects correlated failure-domain descriptors; only an explicit reduced-resilience policy admits them so the deployment can report their resilience disposition truthfully. The fixed authority-profile marker and explicit placement policy are part of a domain-separated fixed-quorum authority identity: otherwise-identical strict and reduced-resilience fixed profiles MUST derive different authenticated peer, durable-store, and snapshot scopes. A fixed profile and a dynamic profile with the same descriptor set MUST also derive different scopes. Mixed-policy or mixed-profile peers MUST fail authenticated admission before Openraft or durable Raft initialization. Dynamic-profile identities remain descriptor- and epoch-derived and do not include the fixed-profile or placement-policy binding. ConsensusSessionStore::open_fixed_durable_quorum is supported only on Linux, where descriptor-pinned SQLite snapshots are available; other platforms MUST return FixedQuorumUnsupportedPlatform before durable initialization. Linux alone is insufficient for the fixed profile: the snapshot filesystem MUST support the exact fs-verity v1 profile (SHA-256, 4 KiB block size, no salt, and no signature). Build, installation, startup, and recovery MUST reject an unsupported filesystem or an unsealed fixed-profile artifact before it can be accepted as durable state. The dynamic profile remains available without this fixed artifact requirement, but MUST retain its bounded corruption detection and fail closed on invalid snapshot evidence. This is not an online migration: an existing pre-fixed or unsealed metadata-referenced artifact is not auto-sealed, auto-repaired, or accepted on open. Operators MUST preserve it and use the reviewed offline reseed/recovery/migration procedure before reopening; a startup retry, metadata edit, or byte-identical replacement does not cross this boundary. Fixed membership does not authorize dynamic membership transitions, a second consensus engine, a controller feed, or a new packet-core protocol path. try_from_fixed_durable_quorum_with_authenticated_placement may additionally verify a fresh exact-member AuthenticatedPlatform placement evidence set. Replacement evidence is verified through verify_fixed_durable_quorum_placement_evidence; neither constructor nor replacement proof changes the immutable voter configuration. A verified replacement proof is consumed only by the fixed probe's explicit placement-attestation form; it cannot refresh traffic authority.

probe_fixed_durable_quorum_readiness MUST return separate typed results for traffic authority and placement resilience. Traffic authority requires the exact persisted consensus identity and admitted 3/5 voter set, distinct authenticated voter identities and declared backing bindings, a clear recovery latch, and a fresh linearizable Openraft majority barrier. Lost membership, unavailable majority, or recovery state revokes traffic authority immediately. Placement uses the strict RequireIndependentFailureDomains policy by default: only fresh AuthenticatedPlatform evidence may report independent placement. The explicit AllowReducedResilience policy may report correlated or unknown placement as reduced resilience, never as independent. Expiring placement evidence may only downgrade that placement result; it MUST NOT alter fixed-quorum authority, Openraft sequencing, fencing, leases, or mutation admission. Concrete backing-instance and voter-incarnation hardening remain a separate concern; this contract does not manufacture those facts from paths or caller descriptors.

11.2.3 Replication-Log Range Cursors

get_replication_log(start, limit) MUST define one checked inclusive range. Sequence zero is a read-side empty-log sentinel and MUST normalize to inclusive sequence one; it remains invalid as an entry. A zero limit MUST return an empty page before backend I/O, provider work, an Openraft barrier, resolution, or network dispatch. A non-zero range begins at max(start, 1) and ends at that value plus limit - 1. The SDK-wide page limit MUST be 65,536 entries. A larger limit MUST return ReplicationLogPageTooLarge; interval overflow MUST return InvalidReplicationLogRange. start = u64::MAX, limit = 1 is valid, while any larger non-zero interval from that start MUST fail overflow. An empty log, the terminal cursor immediately after the head, or a future cursor MUST return an empty page.

A non-empty page MUST begin at the normalized first sequence, remain internally contiguous, and end no later than the checked last sequence. It MAY be shorter only at the current head or an outer response-frame boundary. Frame shaping MUST emit only the largest complete exact prefix and MUST leave the first unsent sequence as the next cursor. A backend or peer page wholly before or after the requested interval MUST fail with InvalidReplicationSequence before caller exposure. An authenticated compatibility client observing such a wire violation MUST discard both the connection and its cached capabilities before a later request re-handshakes.

If compaction has removed the requested first sequence, the backend MUST return ReplicationLogCursorCompacted { resume_from }, where resume_from is the first sequence after the compacted floor. It MUST NOT silently substitute the first retained entry. The caller MUST install a coherent snapshot or rebuild through its existing authority before using that resume point; the error is not permission to discard missing history. A zero-limit request MUST NOT consult the compaction floor.

After its linearizable barrier, ConsensusSessionStore MUST read one local applied state. It MUST NOT collect or union replication-log pages or compaction floors across replicas. Differing replica floors therefore yield typed local outcomes and cannot synthesize a page that skips committed history. This range contract does not create sequencing, commit, snapshot, restore, or watch authority and does not change payload envelopes, AAD, HKMS/provider placement, or encryption-at-rest boundaries.

11.2.4 Replication-Watch Cursors and Atomic Handoff

watch(start_sequence) MUST use one inclusive 1-based cursor contract. Sequence zero MUST normalize to one. Existing, future, and terminal u64::MAX positions are valid and MUST NOT receive a lower sequence. A watch that delivers u64::MAX MUST deliver it once and close because a reconnect successor cannot be represented. Otherwise a reconnect MUST use the checked successor of the last processed entry.

Backlog capture and live registration MUST be atomic with append/apply notification, or use an equivalent checked handoff that cannot lose or duplicate an entry. Every registration MUST retain its next eligible sequence. A notification below that sequence MAY be ignored when it is either below a requested future cursor or the atomic handoff proves it is already present in that watch's backlog. A position above the next eligible sequence is an integrity gap and MUST close the watch. Backlog and live state MUST each have fixed finite bounds. This SDK admits at most 64 captured backlog entries and 64 queued live entries per watch. More retained backlog MUST return ReplicationWatchCatchUpRequired without a skip cursor. The caller MUST invalidate dependent state, perform a coherent snapshot or full-cache catch-up, and reconnect from the position that procedure proves. Blind retry is forbidden. A compacted cursor remains the distinct ReplicationLogCursorCompacted { resume_from } result and MUST NOT use its resume point until a coherent snapshot covers the missing interval. Live channel overflow MUST evict the slow consumer; cancellation and stream close MUST NOT permit registrations to accumulate without bound.

The production Openraft adapter MUST complete its linearizable barrier before the atomic local handoff and MUST publish only application-journal entries emitted by state-machine apply. An uncommitted or merely log-appended local entry MUST NOT be observable. Raw append/rebuild beside Openraft remains forbidden. The legacy session-net client MUST complete watch setup within its absolute deadline before returning: an initial typed store rejection is returned exactly, not converted into disconnect/retry ambiguity. After acceptance it MUST require every authenticated-peer item to equal the next inclusive sequence and MUST terminate the dedicated connection on duplicate, gap, invalid, or otherwise corrupt metadata before an outer encryption wrapper performs provider work. Errors and diagnostics MUST be redaction-safe, and a subsequent independent request MUST use a usable freshly authenticated connection.

ReplicationWatchCatchUpRequired advances the quarantined protocol-v4 error set from revision 5 to revision 6. The wire schema remains revision 4. All legacy compatibility peers MUST be drained and upgraded together; this is not a rolling mixed-profile transition. The Openraft consensus profile, persisted SQLite/journal/snapshot format, payload envelopes, AAD, and HKMS/provider placement are unchanged.

Replicas MUST apply events only if generation and fence are newer according to the state class rules.

11.3 RPO and RTO

Every deployment profile MUST publish:

  • Recovery point objective for session state.
  • Recovery time objective for session service.
  • Maximum tolerated replication lag.
  • Which state classes are replicated.
  • Which state classes are rebuildable.

12. Serialization

Rust has no garbage collector, so the goal is allocation, CPU, and cache efficiency rather than "GC pressure" reduction.

12.1 Formats

Allowed formats:

  • FlatBuffers for read-mostly zero-copy records.
  • Prost/Protobuf for compatibility, with careful allocation profiling.
  • Postcard or bincode-like formats only for internal state with stable version policy.

Each state type MUST define:

  • schema version
  • compatibility policy
  • max encoded size
  • fuzz target
  • migration path

12.2 Decode Rules

Decoders MUST:

  • Validate length prefixes and offsets.
  • Reject trailing garbage unless explicitly allowed.
  • Avoid borrowing data beyond the lifetime of the source buffer.
  • Avoid panics on corrupt data.
  • Support partial decode for lookup keys where useful.

12.3 Legacy Direct-Backend Session-Net Protocol v5

The direct SessionBackend protocol is retained only behind the non-default legacy-session-net-compat feature for controlled migration and compatibility testing. It MUST NOT be enabled on a production consensus node or served on the consensus endpoint. When used for migration, it MUST use the exact opc-session-net/5 ALPN, contract version, and contract profile. It MUST NOT negotiate down or select a highest-common version. A mismatch MUST fail before backend dispatch, close the connection, and be non-retryable for that request.

The public semantic Request and Response types remain available, but their Serde boundary MUST delegate to private fixed-width v5 DTOs. Hello and HelloAck add an optional contract_profile; HelloAck also carries the server's optional cas_idempotency_epoch, and direct CAS carries an optional idempotency_epoch. Exhaustive Rust construction and matching MUST account for the new fields. The profile pins wire-schema revision 7 and error-set revision 9; max_restore_scan_page_payload_bytes = 2096128; owner, custom-key, and state-type bounds of 128 UTF-8 bytes; min_frame_size = 8192; max_frame_size = 16777216; stable_id_max_bytes = 64; replication_tx_id_max_bytes = 128; cas_request_id_bytes = 36; the 31,536,000-second session TTL maximum; restore-page maximum 1,024; and the depth-16/256-node replication-tree rules. Every transported stable ID MUST contain 1 through 64 bytes. Every transported replication transaction ID MUST contain 1 through 128 UTF-8 bytes and MUST be represented by the bounded ReplicationTxId domain type before mutation or durable sequence allocation. New committed coordinator writes MUST encode the 16-byte consensus request ID as exactly 32 lowercase hexadecimal bytes. A reader MUST preserve any valid legacy representation byte-for-byte and MUST NOT trim, case-fold, parse, or normalize it; exact equality remains idempotent redelivery and any distinct representation remains divergent. Every CAS request ID that is present MUST use the canonical lowercase hyphenated UUID representation and therefore contain exactly 36 bytes. The public profile's max_frame_size addition is a Rust source break for external struct literals and exhaustive destructuring and MUST be deployed in the same coordinated revision-2 fleet transition.

The fixed-width mapping is:

  • Hello requested_response_frame_size, HelloAck accepted_response_frame_size, and HelloAck server_request_frame_size: u32;
  • restore/log request limits and the client restore-response budget: u32;
  • restore request/response cursors and restore excluded count: u64;
  • backend max_value_bytes: u64; and
  • PayloadTooLarge.actual/max, RestoreScanPageTooLarge.requested/max, ReplicationLogPageTooLarge.requested/max, and RestoreScanResponseTooLarge.max_bytes: u64, including errors nested in batch results.

The restore wire page MUST omit loaded_count and complete; the receiver MUST derive them from the record vector and next_cursor. Conversion to or from a domain usize MUST be checked, and a non-representable value MUST fail before backend dispatch or caller exposure. Collection work MUST be bounded independently from encoded frame size: at most 256 batch operations, 1,024 restore records, 65,536 replication-log entries, and 65,536 rebuild entries. The configured frame limit remains a separate encoded-byte bound. Log requests and returned pages MUST also satisfy the exact range contract in §11.2.3 before dispatch or caller exposure.

Wire-schema revision 2 MUST negotiate directional frame budgets during the frozen bootstrap. The client's requested response size, the server's accepted response size, and the server's request size MUST each be at least MIN_NEGOTIATED_FRAME_SIZE (8 KiB, or 8,192 bytes), at most MAX_NEGOTIATED_FRAME_SIZE (16 MiB, or 16,777,216 bytes), and representable as u32. Their public bootstrap fields are Option<u32> so a revision-2 decoder can classify an otherwise decodable legacy minimal bootstrap. This MUST NOT be treated as bidirectional mismatch negotiation: a revision-1 decoder MAY reject unknown revision-2 fields by closing without a typed response. Revision-2 admission MUST require all three as Some. The accepted response size MUST be no greater than either the client's receive limit or the server's configured frame limit. The server request size independently states the maximum operation frame the server will accept. Peers MUST use these values for the lifetime of that connection and MUST NOT infer equal limits in both directions. MIN_RESTORE_SCAN_RESPONSE_FRAME_SIZE MUST alias MIN_NEGOTIATED_FRAME_SIZE; it is not a second negotiable minimum. The restore request's existing max_response_frame_size MUST remain an additional per-call cap and MUST NOT enlarge the negotiated response budget. Before binding or spawning, a server MUST reject a configured frame size below 8 KiB or above 16 MiB, a zero/runtime-unrepresentable connection-slot count, or an unrepresentable idle/restore timeout with InvalidInput. A zero timeout MAY remain an intentional immediate-fail policy. Before DNS, socket allocation, or watch-task spawning, a client MUST reject a configured frame size outside the same range. Bootstrap output MUST use the separate 8 KiB MAX_HANDSHAKE_FRAME_SIZE cap.

Every post-bootstrap response and watch item MUST be fully bounded-encoded into retained byte storage capped at the accepted response size before a length prefix is written. The common non-pageable and complete-page success path MUST perform one bounded encode without a separate sizing serialization. For a replication-log page, if the complete pageable response is oversized, that direct encode MUST emit no prefix; prefix selection MAY then perform bounded logarithmic sizing probes followed by one final bounded encode. Restore pages MUST be validated as whole backend results and MUST NOT be transport-shaped. No retained encoded-JSON byte storage may exceed the negotiated cap. The retained/requested encoded-JSON byte storage MUST remain no greater than the cap, including for non-power-of-two budgets. An implementation MUST NOT coalesce or create a temporary payload buffer when doing so would exceed that bound. This SDK satisfies the contract with lazy exact-length boxed chunks and no coalescing copy. Chunk-pointer metadata and allocator slab/RSS overhead are not encoded JSON bytes and MUST be accounted for separately by runtime resource qualification. One absolute deadline MUST be established before the first direct encode or sizing probe and reused through every probe, the final encode, length prefix, complete payload, and flush. Implementations MUST NOT restart the deadline per probe, phase, write, or chunk. Deadline expiry MUST terminate the connection and release its task and connection permits. Synchronous storage and sizing sinks MUST check the deadline and the server's abort cancellation signal cooperatively between serializer writes and retained chunks. Task abortion cannot preempt one synchronous serializer callback; therefore every wire field and collection processed between checks MUST remain bounded, and shutdown claims MUST include that finite callback interval.

Outbound behavior is family-specific:

  • The fixed-width Capabilities envelope MUST fit within the 8 KiB protocol minimum; an encoding failure MUST close without emitting an oversized frame. Scalar mutation results, replication/rebuild acknowledgements, and lease results MUST use an SDK-owned fixed, redaction-safe fallback when a backend-provided result cannot fit. If the fallback cannot fit, the connection MUST close without emitting an oversized frame.
  • Get results and CAS conflicts MUST NOT truncate a record. They MUST replace the complete record-bearing result with the fixed fallback or close.
  • Batch results MUST preserve exact request cardinality and order. They MUST NOT be truncated; an oversized complete result becomes one fixed batch error or a connection close.
  • Restore backends MAY independently return a shorter cursor-correct page under their count, payload, or work budgets. The transport MUST validate the entire returned page against the fixed 2,096,128-byte wire-payload cap and the negotiated frame; it MUST NOT trim or rewrite records or cursors. If the whole page is oversized, the server MUST return typed RestoreScanResponseTooLarge if representable or close.
  • Replication-log results MAY return only the largest complete contiguous entry prefix that fits. An entry MUST NOT be split, reordered, or skipped. If no requested entry fits, the server MUST use its fixed fallback or close.
  • Watch acknowledgement and each watch item MUST be bounded independently. The server MUST NOT skip an oversized entry because that would conceal a sequence gap. It MUST send a fixed error item when representable and then terminate the stream/connection, or close immediately when the fallback cannot fit.

Fallback text MUST be static SDK-owned text. It MUST NOT contain a key, owner, payload, transaction/request ID, peer identity, backend error string, or other peer-controlled text. Consuming rejection of nested replication operations MUST retain iterative disposal and the existing depth/node work bounds.

BackendCapabilities::max_value_bytes transported over session-net MUST be no greater than the backend limit, conservative_payload_budget(accepted_response_frame_size), or conservative_payload_budget(server_request_frame_size). That function MUST compute frame_size.saturating_sub(8192) / 8: the 8 KiB block reserves the record/key/error envelope, while the factor of eight covers four-byte worst-case JSON byte-array expansion plus equal escaping/metadata headroom. The advertised maximum MUST complete a real write/read round trip under unequal limits. At exactly 8 KiB, the conservative payload budget is zero: that minimum MUST fit maximum-profile metadata/envelopes but does not promise a non-zero application payload. Capability evidence remains descriptive and MUST NOT authorize quorum or traffic readiness. The 1 MiB default yields 130,048 payload bytes and the 16 MiB ceiling yields 2,096,128. The wire ceiling is intentionally below standalone SQLite's local 4 MiB + 64 KiB stored-envelope restore capacity, which is not a session-net wire capability. This is a per-frame limit, not aggregate admission: at the server's default 128 connection slots, simultaneous ceiling-sized encoded stores can retain about 2 GiB before chunk metadata, TLS, and runtime overhead. The aggregate scales with the configured connection limit. #143 owns aggregate byte permits and distributed resource/soak qualification.

Backend mutation and response delivery are not one transaction. A mutation MAY commit before response encoding, write, or flush fails. Direct CAS idempotency MUST be keyed by the authenticated logical peer plus canonical request UUID and MUST bind a redaction-safe digest of the complete operation, cluster and configuration identity, monotonic configuration epoch, and server-issued process epoch. Same-scope exact duplicates MUST share one in-flight execution and replay the exact success or conflict. Reuse by another peer or operation MUST return CasIdempotencyConflict before backend dispatch. Cancellation MUST leave a tracked ambiguous tombstone, never an untracked in-flight entry.

The compatibility cache MUST bound total and per-peer entries and bytes, retention age, and cleanup work. One peer MUST NOT evict another peer's active retry window. Restart or retention cleanup MUST rotate the process epoch, and an old epoch MUST return CasIdempotencyOutcomeUnavailable before mutation. Pressure that cannot retain a result MUST return the same typed unavailable outcome rather than evicting an active result and treating its UUID as new. The public client MUST NOT automatically resubmit a CAS after any ambiguous transport boundary. A caller that receives no valid response or the typed unavailable outcome MUST perform an authoritative re-read and derive a new mutation; it MUST NOT infer rollback or replay the historical operation under either the old or a fresh UUID.

Every authenticated request MUST have three bounded phases: one inbound idle-timeout to receive and decode a complete frame, one backend admission/work deadline started after decode, and one reserved bounded response interval. The checked sum of the latter two is the post-decode dispatch/response lifetime; full connection-slot occupancy includes the inbound phase as well. Reads, mutations, lease mutations, and watch setup MUST have independent fixed-size admission pools; restore MAY retain a stricter dedicated pool. Queue expiry before backend polling is known not applied. Read execution MUST be cancellable and release resources. Once a non-CAS or lease mutation has been polled, deadline, disconnect, cancellation, or response loss MUST either recover a durable operation-bound outcome or return the non-retryable BackendOperationOutcomeUnavailable / LeaseError::OperationOutcomeUnavailable class. The public legacy client MUST NOT reconnect and resubmit such a mutation. A transport failure proven to have occurred before the first request write remains known not applied and MAY be retried. CAS continues to use its stronger operation-bound idempotency outcome. Code that itself drops a polled mutation future receives no result and MUST treat that cancellation as the same unknown-outcome class.

The production Openraft adapter MUST create one durable request identity before leader selection and retain it across internal forwarding retries. A local failure before proposal submission MAY remain retryable. Once Openraft accepts the proposal into its client-write channel, loss of the result receiver, deadline expiry, or an unvalidated forwarded result MUST return CasIdempotencyOutcomeUnavailable for direct CAS and BackendOperationOutcomeUnavailable for every other mutation (mapped to LeaseError::OperationOutcomeUnavailable at the lease API). It MUST NOT return a generic retryable availability error. Durable state-machine request outcomes MUST make retry of the same internal identity idempotent.

The production adapter MUST use the shared fixed eight-slot proposal-admission pool for both normal mutations and finite-expiry logical-time-floor proposals. It MUST acquire a slot within the operation's original absolute deadline. Once client_write_ff returns the accepted proposal's result receiver, a detached supervisor MUST retain that slot until the receiver resolves; caller drop, peer EOF, and response timeout MUST NOT release it early. Saturation MUST fail closed before another proposal is submitted. A finite-expiry preflight MUST validate its descriptors against the logical time returned by its committed floor command before reporting success.

Every fresh read-index and mutation preflight MUST use one shared linearizability supervisor per Openraft node. Exactly one supervisor-owned ensure_linearizable call may be in flight, with at most 64 total admitted callers across the active and waiting cohorts. Only callers collected before dispatch may share that exact Openraft result; later callers require a later check. Caller cancellation or deadline expiry MUST NOT cancel a dispatched check, release its admission early, or start an overlapping check. The supervisor is a resource bound only: Openraft remains the sole source of leadership, quorum, read-index, and applied-state authority.

After a legacy request is transmitted, a malformed or wrong-family response, or a same-family response that violates request-bound key, owner, fence, credential, ordering, or cardinality semantics, MUST use the same typed ambiguous-outcome classification. Direct-CAS retry caches MUST NOT retain a backend availability result as a completed retryable outcome; they MUST retain an ambiguous tombstone instead.

Server cancellation and peer EOF MUST race pending backend work and idle watch streams. Backend adapters MUST treat future drop as a cancellation signal and retain bounded admission/supervision until underlying work exits. Read resources are released after bounded cancellation completes. A mutation may finish after its caller drops, but that caller MUST treat the outcome as unknown and re-read authoritative state. Durable operation-bound replay is Openraft/direct-CAS-specific, not a generic adapter promise. No timeout or shutdown path may create unbounded detached work. Static capabilities MUST fail closed when backend admission cannot be obtained and MUST NOT substitute for fresh readiness.

Outbound diagnostics SHOULD expose only bounded response_family categories and fixed reasons such as frame_too_large, page_shortened, write_timeout, transport, and encoding. They MUST NOT label or log session keys, payloads, transaction IDs, owners, SPIFFE IDs, backend/peer-controlled error text, or other high-cardinality identifiers. The fixed metric family opc_session_net_backend_lifetime_events_total MAY expose only queue_timeout, execution_timeout, cancellation, peer_disconnect, and ambiguous_outcome; it MUST NOT contain dynamic labels.

A fresh version/profile/authentication or malformed-handshake failure MUST clear the cached capabilities and report all capability booleans false with max_value_bytes = 0. A cache retained after transient transport loss is descriptive only and MUST NOT authorize a store operation, durable readiness, or traffic admission. A cache MUST be keyed by the exact profile and negotiated directional limits and cleared when a successful reconnect changes either limit. Callers MUST use fresh bounded quorum evidence.

The transition to v5 wire-schema revision 7 and error-set revision 9 is a coordinated stop/upgrade/start boundary, not a rolling deployment. Operators MUST drain traffic and writers; run the #135 identity audit; inventory every retained record, replication log, snapshot, restore source, and replay source for the stable-ID and transaction-ID bounds; and complete product-aware handover/nested-payload preflights. A retained-value migration MUST be decoder-first: while writers remain quiesced, every migration reader MUST be able to decode the legacy representation before any rewrite or replacement occurs. Stable IDs MUST follow the product-aware #167 model/persistence/privacy/audit policy. Durable transaction IDs MUST follow the #168 bounded-type and migration policy, including current report version 4 and coordinated cutover with #127/#128/#143. The migration MUST NOT silently truncate, hash, or rename a key or idempotency identity. Operators MUST verify that the strict revision-3 decoder accepts the result; then they MUST stop every session-net client, server, and protection wrapper plus every handover reader/writer; upgrade them together; verify exact-v5 authenticated restore/log traffic, rejection of modified/legacy restore cursors, sparse empty-page progress, and fresh quorum evidence; and only then restore traffic. Once an OPCH value has been written, v3 rollback additionally requires a coherent drained checkpoint restore or reviewed reverse migration of every live and replayable record, log, snapshot, and restore source. Revision 3 adds only an O(1) per-store cursor key to local restore metadata; it does not rewrite session records or create another authority. A pre-revision-3 consensus snapshot lacks that metadata and MUST NOT be installed after upgrade; operators MUST take and validate a coherent post-upgrade snapshot before claiming repair or rollback coverage. In-profile data needs no format conversion, but out-of-profile retained values MUST be migrated or replaced before strict transport starts. Binary rollback MUST restore one exact drained fleet profile and install a rollback-side decoder that can read the retained target representation before old writers restart; otherwise it MUST restore a coherent checkpoint or run a reviewed reverse migration. Mixed revision-3 and older participants fail closed. Rollback across the independent OPCH/#135 boundary retains its checkpoint/reverse- migration requirement.

That compatibility cutover is distinct from credential rotation. Only after every participant is admitted on the same revision-7 profile MAY operators use material-epoch or explicit reauthentication to recycle connections without draining application traffic. The lifecycle MUST NOT be used to mix protocol profiles, negotiate a downgrade, or turn a binary rollback into a rolling operation.

The cursor encoding is variable-length but strictly bounded by the consensus RPC/key ceiling. HMAC-derived AEAD and synthetic-nonce keys are separated; identical semantic positions encode identically. The seek key and snapshot metadata remain confidential, while a clear cumulative examined-row position is bound into cursor authentication. A receiver can reject a structurally inconsistent claimed step and the issuer authenticates the position when the cursor returns, but neither fact proves peer-page completeness or server honesty. Production completeness comes from the local Openraft-applied state after its linearizable barrier. Cursors are backend-incarnation/node-bound: same-PVC restart can resume, but another node or installed snapshot MUST return typed stale-cursor state and the caller MUST discard partial pagination and restart at the first page.

#159 establishes only session-net response/write and wire-containment bounds. It does not close #167 or #168 and does not provide #143's payload-key/distributed production qualification. #163 real-mTLS transport tests cover local/peer leaf-expiry retirement, overlapping trust, complete replacement negotiation, old-trust rejection, and request/watch continuity. TLS material tests separately prove effective configured/presented-chain expiry through a real mutual-TLS handshake, while lifecycle unit tests prove the corresponding local/peer retirement deadlines and fixed metric reasons. Additional non-ignored single-host three- and five-process tests now exercise one bounded multi-process slice: a test-only consensus-RPC admission loss on one stable follower while a different member retains last-good material after malformed trust; exact-address restart, catch-up, and repair; and a same-issuer leaf with a 75-second remaining-validity/expiry budget through the expiry - 30 seconds soft boundary, hard drain, source/controller LastGoodExpired, survivor durable readiness and encrypted-canary progress, and same-process valid replacement while bounded mixed lease/CAS mutation, linearizable-read, watch, complete-restore, readiness, and connection-recycling traffic remains active. After repair, one stable follower is also killed uncleanly with active mutation/watch tasks; survivors commit during the outage and its same-disk, exact-address restart must reconcile the exact record/watch state and resume at a higher fence under the v3 stage bounds described above. The six sequential stage bounds compose to a 164-second crash-to-resume ceiling; the total does not replace any individual stage deadline. Other active-mutator restart patterns, a real/deployed partition, a broader restart/fault matrix, resource/soak, remote-HKMS, deployed-CNF, signed release, and evidence-schema/production-profile claims remain unqualified. Generic CRL/OCSP/certificate-or-identity-denylist revocation is not implemented. #177 removes opc-persist's separate config TCP path and reuses the shared consensus peer/handler boundary instead of defining another timeout or credential lifecycle. An in-process real-mTLS integration forms a three-node config Openraft cluster and commits/linearizably reads through the existing peer/server types. Any compatibility transport work must preserve the single Openraft authority rather than reopen direct mutation as a quorum path. #161 atomic reload, #162 coherent material epochs, and #163 finite connection reauthentication are implemented. Fleet SVID/trust-bundle qualification remains #164 under umbrella #158.

12.4 Consensus-Only Session Transport

The production session HA transport MUST use SessionConsensusServer and RemoteSessionConsensusPeer on the exact opc-session-consensus/2 ALPN. The server MUST own only a SessionConsensusRpcHandler; it MUST NOT accept a SessionBackend, lease manager, direct mutation request, caller-authored replication append, or rebuild request. The consensus ALPN and legacy opc-session-net/5 ALPN MUST NOT be multiplexed as equivalent authority on one production listener.

The exact consensus contract profile MUST use transport/wire-schema revision 5, application revision 4, and error-set revision 6. The revision-5 transport profile retains the explicit forwarded consumer scope, so a peer cannot silently downgrade a consumer-scoped operation to an internal call; application revision 4 fences the former 728bc5 application-revision-3 Postcard tag-27 FinalizeOperatorRecoveryV2 encoding, which conflicts with the merged roster profile's tags 27 through 30. Error revision 6 binds that semantic boundary into the exact profile. Any other transport, application, or error-set revision MUST fail before engine dispatch. Operators MUST drain traffic and writers, stop every consensus member, upgrade the full membership together, verify exact-profile handshakes, and only then restore traffic. Mixed-profile rolling operation is unsupported.

Each connection MUST perform mutual TLS and bind all of the following before engine dispatch:

  • the live certificate's one canonical SPIFFE URI;
  • the logical ReplicaId and derived stable node ID of each side;
  • the expected opposite peer and authenticated request sender;
  • the cluster ID, exact configuration digest, and positive configuration epoch;
  • the engine RPC family, peer role, exact transport revision/profile, and a fresh challenge.

The sender authenticated by the outer transport MUST equal the sender carried inside the bounded engine request. DNS names, FQDNs, IP addresses, resolver aliases, and Kubernetes pod hostnames MUST affect only connection routing and MUST NOT be accepted as substitutes for any logical, stable, or certificate identity.

One absolute family deadline MUST begin before lane acquisition and cover bounded encode, write, and response read. The outer hard/direct complete ceiling for AppendEntries/Openraft read-index MUST be 2,000 ms, Vote 5,000 ms, and InstallSnapshot, forwarded mutation, and consumer ReadBarrier 10,000 ms. An Openraft network call uses the smaller soft TTL described below. If no valid cached connection exists, resolution, TCP connect, mutual TLS, identity admission, and bootstrap MUST use the lesser of two thirds of the remaining family budget and a 1,500 ms cold sub-bound. The reserved final third MUST remain available for the first negotiated RPC; cold time MUST NOT be added to the family deadline. Each directed peer MAY cache a fixed primary/overflow pool of at most two authenticated connections, with at most one in-flight RPC per lane. Sequential calls MUST prefer primary; a concurrent call MAY use overflow; when both lanes are busy, further calls MUST wait for either lane under the same absolute family deadline. It MUST recache a selected lane only after a complete, correctly correlated, authenticated, validated successful response or typed semantic Unavailable response. The Unavailable exception preserves a known stream position but grants no success or authority. A cached lane MUST NOT clear shared reconnect cooldown until such a reusable response has proved that lane usable. Cancellation, timeout, EOF, malformed/cross-correlated response, protocol, authentication, scope mismatch, rejection, lifecycle evidence mismatch, or any uncertain stream position MUST evict that lane. A late connection or response after cancellation MUST NOT be reused. The transport MUST carry only the shared bounded consensus envelope; the session-store adapter compact-encodes Openraft RPCs, and the network layer MUST NOT interpret commands or decide leadership, voting, log matching, commit, or repair. An identity, authentication, schema, payload-bound, or sender mismatch MUST fail before Openraft dispatch with redaction-safe diagnostics.

For an Openraft engine RPC, the adapter MUST pass RPCOption::soft_ttl() to a deadline-aware network peer and MUST NOT install another hard timeout around that peer future. The peer MUST apply the lesser of the supplied soft TTL and its configured family ceiling, classify deadline expiry before returning, and evict any socket with an uncertain stream position. Openraft's outer hard_ttl() remains the sole hard cancellation authority. A compatibility or in-process peer that does not own transport deadlines MAY retain its existing call behavior and rely on that outer hard deadline; the deadline-aware default MUST NOT introduce a new soft cancellation boundary for such a peer.

Every authenticated client, peer, and listener MUST apply one finite ConnectionLifecyclePolicy. Its hard deadline MUST be the earliest of the configured maximum authentication age, the expiry of every certificate in the local configured/presented SVID chain, and the expiry of every certificate actually presented by the peer. A redundantly presented root contributes to that bound. A certificate appearing only in a configured trust bundle is not independently scanned for the deadline, and the time an anchor is removed is not an expiry deadline. Production SVID chains SHOULD omit the trust anchor. Soft retirement MUST begin early enough to leave at most one configured drain window before the hard deadline. A coherent TLS material-epoch change or an explicit process-local reauthentication generation MUST also schedule retirement, using deterministic directed-peer jitter no greater than the configured bound.

After soft retirement the client MUST NOT assign a new operation to the connection and the server MUST NOT read or dispatch another request. An operation admitted before retirement MAY return once within its existing operation deadline, but transport ownership MUST stop waiting by the lifecycle hard deadline and MUST release every connection/task slot. Dropping the backend future requests cancellation but MUST NOT be interpreted as rollback: bounded supervised mutation work MAY finish later. Such an outcome MUST remain typed ambiguous, MUST NOT be automatically replayed, and requires authoritative readback or the operation's existing idempotency/fencing contract. A replacement MUST repeat DNS/route resolution, mutual TLS, live certificate identity, nonce/challenge, ALPN, version, and exact contract-profile checks. TLS resumption, cached peer authority, plaintext fallback, and task-abort reauthentication MUST NOT replace that handshake.

After authentication and bootstrap acknowledgement, if no byte of the next request arrives before the listener idle deadline, the server MUST record the fixed idle_timeout lifecycle-retirement reason, enter and complete the normal drain/slot-release path, dispatch no request, and finish the connection handler as a successful bounded lifecycle outcome rather than a timeout failure. This rule applies to both the consensus listener and the legacy direct listener. Silence during TLS/application bootstrap MUST remain a timeout failure. Once any byte of an authenticated frame arrives, the remaining length prefix and payload MUST complete within the original absolute idle deadline; an incomplete active frame MUST remain a timeout failure and MUST NOT be relabeled as idle retirement.

If a lifecycle retirement boundary (maximum authentication age, local or peer certificate expiry, material epoch, or explicit reauthentication) is observed after mutual TLS but before any bootstrap acknowledgement bytes are written, the generic transport MUST return one complete authenticated BootstrapResponse::ConnectionRetiring result. The consensus bootstrap context MUST use SessionConsensusBootstrapResponse::Rejected(SessionConsensusPeerError::Rejected) as the corresponding no-dispatch control. That nested value is reserved in this context only: ordinary authentication, identity or scope, contract, and protocol failures MUST retain their existing classifications and MUST NOT be emitted or interpreted as this control; a post-bootstrap engine Rejected result MUST remain an ordinary call response. The sequential client MUST NOT send application or Openraft request bytes before bootstrap succeeds. After it decodes the complete retirement control, it MUST discard that connection and retry the pending operation only through the existing bounded deadline and backoff path.

EOF, an incomplete control, or an acknowledgement whose write has partially completed MUST fail closed. Once an acknowledgement write may have emitted a byte, the server MUST close rather than append a bootstrap retirement frame. The client MUST NOT infer no-dispatch from that incomplete stream. The server MUST count a connection-attempt success only after completely writing the bootstrap retirement control; the client MUST count its own success only after decoding the complete control. In both cases success means authenticated transport/control completion, not application admission. That decode MUST initiate the client's existing bounded deadline/backoff path and count a reconnect attempt; the client MUST NOT count a reconnect failure or a connection-failure outcome for that complete control.

The legacy direct profile MAY automatically retry a mutation after retirement only when it has decoded the complete fixed ConnectionRetiring response, which proves server dispatch did not occur. EOF, a partial retirement frame, write failure without a complete buffered proof, or a generic transport error MUST remain an ambiguous mutation outcome. A legacy watch MUST keep the caller stream alive across planned retirement, pin any partially read item to its old connection, advance the resume cursor only after caller delivery, and resume from checked last_delivered_sequence + 1. Cursor overflow, compaction or another permanent backend error, cancellation, and bounded slow-consumer failure MUST terminate explicitly rather than reconnect forever.

This bootstrap behavior was introduced for the then-frozen direct revision-6 variant and is retained by the current revision-7 profile, alongside the existing consensus error value in their restricted bootstrap contexts. It changes no public API, direct or consensus profile revision, persisted SQLite/journal/snapshot format, Openraft commit authority, payload envelope, encryption-at-rest boundary, or HKMS/provider placement. Older same-profile decoders fail closed on the control rather than negotiating a downgrade, so mixed-patch rolling rotation is not seamless. This closes only the narrow authenticated post-TLS/pre-acknowledgement race; it does not satisfy the remaining #164 fleet qualification.

This authenticated transport plus #127 commit authority is still not a production qualification. #128 supplies current-format divergence recovery, #129 supplies the audited offline legacy-fork campaign without reopening a runtime consensus path, and #133 provides bounded applied-state restore without reopening a direct backend/rebuild port. #143 remains the distributed partition/restart/resource/soak and payload-key gate. #161 atomic reload, #162 material epochs, and #163 bounded reauthentication are implemented, including scoped retained-connection, request, and watch continuity evidence. Production rotation already has single-host multi-process trust transitions and the exact synthetic fault/expiry slice described above. It MUST additionally qualify deployed trust/root cutover, real network/storage faults and a broader restart matrix, deployed mixed traffic/watch/restore under those real faults, reconnect-storm bounds, resources, soak, remote HKMS, deployed CNFs, and signed release evidence under #164/#143. The lack of immediate generic CRL, OCSP, or certificate/identity-denylist revocation MUST remain explicit. #158 remains the umbrella until that fleet evidence passes.

12.5 Typed Session-Quorum Consumer Transport

StatelessSessionConsumerClient, PersistentSessionConsumerClient, and SessionQuorumConsumerServer provide the typed least-authority application-consumer boundary. They MUST use mutual TLS and three independent exact lanes: the general opc-session-consumer/1 ALPN at transport revision 6, the epoch-fenced V2 opc-session-consumer/2 ALPN at transport revision 5, and the protected-roster opc-session-consumer/3 ALPN at transport revision 5. These are separate exact protocols from both opc-session-consensus/2 and the quarantined opc-session-net/5 compatibility protocol. /2 MUST NOT fall back to /1, and a lane authenticated for one ALPN MUST NOT carry or be reused for another lane's Hello or request envelope. When protected-roster ingress is enabled, the listener advertises /3 before /2 and /1; each client offers exactly one ALPN. /3 admits only its protected roster operation set under its tenant/scope/fence authority, and /2 MUST NOT count or reclaim /3 lanes. The server MUST reject every /2 Hello revision other than 5 before dispatch.

General revision 6 does not interoperate with older general revisions. Because this SDK is unreleased, a general-lane revision cutover MUST drain consumer clients and listeners; fallback, general-lane dual mode, and mixed-revision operation are unsupported. Deployment MUST provision revision-5 /2 listener support, V2-capable store authority, and the required protected-wrapper V2 journal before enabling an explicit V2 client call. /1, /2, and /3 coexist only as independently authenticated and decoded lanes, never as common or dual-revision authority. Removing either additive lane requires draining its callers first. General revision-6 and V2 revision-5 private JSON DTO bytes are canonical; reordered or otherwise noncanonical encodings, aliases, omissions, and unknown fields MUST fail closed.

StatelessSessionConsumerClient remains a public, source-compatible production/compatibility fresh-authentication typed least-authority surface required by #649, #688, and #691; it is neither hidden, deprecated, nor test-only. PersistentSessionConsumerClient remains the required warm fixed-pool primitive for #695/ePDG latency, so production deployments requiring warm reuse should use it. This distinction is not an API-removal or feature-gating claim.

The consumer listener authenticates the peer from the live mTLS connection and authorizes it only through the store-issued current-member manifest and the configured consumer allow-list. Consensus-member identities are excluded from the consumer role. Every bootstrap and request carries the exact cluster/configuration/epoch scope, which the listener and quorum-side service MUST verify before backend work. Consumer identity and scope values are security-sensitive: diagnostics, profile inventories, and observability MUST record only redaction-safe status/count information, never their concrete values.

The V1 API exposes typed session reads, bounded mutation/lease operations, bounded restore scans, capability discovery, a coarse committed-change watch, and #696's generic one-record atomic fenced transition. The latter includes an exact-key observation, one lease-acquire or lease-renew action plus one bounded record mutation, and exact retained-status readback. V2 exposes only typed V2 capability, V2 history state, one epoch-fenced transition, SessionConsumerV2Operation::FencedTransitionV2Batch, and exact V2 transition status. The batch is an ordered, same-epoch 1..=256 transition batch and is not all-or-nothing: each item has its own outcome and earlier items may have effects when a later item does not. V2 does not add raw consensus or replication operations, membership/voting authority, or product-specific roster/policy semantics. Before V1 activation for the exact current voter scope, capability, observation, status, and first transition admission require fresh authenticated replies from every exact voter; an unavailable or incompatible voter fails closed. A quorum is not a mixed-version proof. It does not expose membership, voting, peer discovery, replication-log read/append, raw replication operation trees, snapshots, rebuild/recovery, product composition, or any topology/consensus authority. It also excludes every legacy RemoteSessionBackend API and all consensus/replication/snapshot/rebuild/membership/admin APIs. The server constructor accepts only the existing least-authority SessionQuorumConsumer port, and all accepted mutations route through the durable quorum leader path.

The raw physical store and authenticated-consumer transport advertise exactly AtomicFencedTransitionCapability::V1. A protected EncryptingSessionBackend or RemoteSealingSessionBackend advertises the separate FencedTransitionV2Capability::V2 only when it owns the separately scoped SDK caller-side FencedTransitionV2PreparedJournal and its exact inner physical boundary supports V2 history. That journal binds the stable backend authority, protection mode, and payload namespace; it is distinct from the permanent capped V1 prepared-transition journal. The session-store documentation defines its provision/open/recovery rules. V2 transport neither creates nor substitutes for this durable recovery boundary, and MUST NOT fall back to V1. A legacy protection wrapper without that journal, an older binary, and raw V1 transport MUST fail closed for the V2 history path.

The first authorized transition after that unanimous proof carries the scope identity and canonical voter-set commitment inside its same single user command and application position. Apply atomically installs its receipt/effects, the one-way persistent schema-version downgrade fence, and an optional single-row exact-current-scope activation certificate; it creates no separate user mutation or log position. Those internal proof fields are not consumer-wire semantics. After this command commits, normal linearizable Raft quorum availability suffices for capability, observation, execution, and status; a leader change or minority loss does not require another all-voter probe. A topology cutover deletes only the old scope certificate, retaining the schema fence and receipt bindings. Its successor scope must repeat the every-voter proof and first activating or recovery transition, while stable request-ID/body receipt recovery survives the rollover.

The exact #684 layout has no ledger. A current writable open may add the exact empty Prepared ledger and zero marker without changing the predecessor schema version, so no V1 authority exists and predecessor readers remain safe until activation. Activated main databases, snapshots, and recovery preserve the schema fence and receipt bindings plus any exact-current certificate; snapshot install never regresses Activated to Prepared or erases/substitutes a same-scope certificate. A legitimately unactivated successor scope may have no certificate pending its new proof. Exact predecessor binaries reject the higher schema fence. An offline pre-V1 minority is not safe to catch up merely because it did not acknowledge activation: an old reader could silently omit new snapshot state, which the persistent fence prevents by rejecting the activated image.

Each V2 request connection carries a connection-local, nonzero u32 sequence that increases monotonically and never wraps, paired with a fresh full-width 128-bit OS-CSPRNG nonce and the fixed 32-byte request commitment. The commitment is SHA-256 of opc-session-consumer-v2-call-phase, the big-endian /2 revision 5, and the exact serialized V2 request bytes. The server admits only the exact next sequence and the client accepts only the exact composite response tuple. A V2 lane retires after at most 4,096 sequential calls. There is exactly one in-flight call per connection, with no multiplexing. This is structural: it isolates cancellation and pre-staged/late responses and removes write-position ambiguity.

The /2 HelloAck request-frame ceiling is independent from the client's response-frame capacity: any nonzero request ceiling through the fixed 16 MiB maximum is valid, and the exact bounded encoder MUST reject an oversized Call before writing its length prefix. The response-frame capacity retains the larger fixed minimum required by bounded batch responses.

The client uses a fixed, fair pool of four request connections by default (at most 16 when configured), with 64 pending calls by default and a hard maximum of 256. A pending call may wait or age for at most 250 ms. The retained Watch transport uses two reserved slots by default (at most 16 when configured), but typed tenant/NF consumer Watch does not acquire them while its cursor is global.

PersistentSessionConsumerClient has an ALPN-specific V2 idle pool while V1 and V2 share one configured request width, pending queue, and prewarm gate. prewarm_v2 establishes revision-5 lanes within that aggregate width without dispatching an operation, execute_v2 dispatches only on a V2 lane, and v2_diagnostics reports only V2 redaction-safe pool state. V1 and V2 idle sockets and authenticated Hello exchanges are never cross-reused; their logical lane and pending-call admission is aggregate. Their aggregate persistent width is the configured request width (at most 16), and may rebalance only by retiring an opposite-protocol idle lane (never active work or V2 poison debt) before the existing setup deadline; it does not allocate a connection, task, channel, or pool per subscriber. DNS resolution, TCP, TLS, and Hello are performed only on establishment or re-establishment of the relevant lane, not on reuse.

Stateless-client clones share bounded physical admission per clone lineage. The V1 family, including protected-roster /3, has exactly 16 request connections, and ordinary V2 /2 has an independent 16, for at most 32 request connections in total. Watch admission remains separately capped at 16. The respective permit MUST be acquired before resolve/TCP and held for the complete physical connection lifetime, including by a persistent client derived from that stateless lineage. Independently constructed stateless clients define independent logical clients, as independently constructed persistent clients do. The typed persistent watch surface MUST preserve exhaustion of either bound as Overloaded and record that bounded outcome; it MUST NOT relabel intentional load shedding as endpoint unavailability.

The hard listener limit is 256 live connections and its retained connection-task set is bounded by that limit; each watch owns one delivery task. Consumer frames are at most 16 MiB and a configured listener frame limit cannot be lower than the 8 MiB batch-response limit plus 4 KiB framing allowance. The bootstrap and active-frame idle bound remains 5 seconds and one complete request/response operation remains bounded to 10 seconds. A watch has a 64-item, 512 KiB transport queue, rechecks cancellation at least every 50 ms, and is also bounded by the 256 KiB store-side projection buffer. The fixed V1 request identity is 16 bytes; a V2 singleton or V2 batch item identity is 56 bytes; consumer identity input is capped at 253 UTF-8 bytes. One V1 generic batch has at most 256 operations and retains at most 8 MiB of serialized response data. A V2 batch has at most 256 operations and separately limits its fully Postcard-encoded request vector and outcome vector to 1 MiB each; the outer authenticated-consumer JSON frame remains subject to its negotiated frame bound.

The complete operation timeout MUST validate strictly greater than zero and no greater than 10 seconds. The configured idle timeout is at most 5 seconds and caps every active partial frame on all client bootstrap and unary reads; partial bytes do not reset that bound. Each discarded checked-out request lane MUST have exactly one reconnect/replacement accounting outcome. Under concurrent shutdown callers, phase progression is monotonic from running to draining to forced and MUST NOT regress.

Every client and listener retains the existing finite TLS lifecycle bounds: by default authentication age is at most 15 minutes, connection-retirement drain is at most 30 seconds, and material-rotation jitter is at most 30 seconds. A consumer shutdown drains for at most 5 seconds. An establishment attempt has a 1,500 ms setup limit and a call makes at most two pre-write establishment attempts. Resolution occurs only for establishment or re-establishment, never for a reused connection. Every cold request and rolling-prewarm setup MUST enter one pool-wide recovery lane after bounded physical admission. One failed setup or proven cached-lane loss publishes the shared exponential lifecycle backoff floor (50 ms by default) plus at most 25 ms jitter, clipped to each logical deadline; concurrent waiters MUST NOT start independent resolver/TCP/TLS/Hello attempts. Reauthentication, material changes, certificate expiry, idle retirement, cancellation, malformed frames, EOF, or an uncertain stream position terminate the connection and release its transport task slot; they do not create another request on that connection. "Material changes" here means an accepted material-epoch change. It MUST schedule each already-admitted lane at a stable directed authenticated-edge deadline no later than the configured rotation-jitter maximum; admitted work and reuse remain permitted before that deadline and retire at it. An explicit generation change MUST invalidate cached admission immediately. A fresh client or server handshake MUST exactly match the current generation and material epoch at its final pre-publication sample and MUST NOT use the cooperative jitter exception. TLS MUST expose only the single fixed-domain, fixed-range consumer jitter duration needed by session-net. It MUST NOT expose a comparable edge-key object, caller-selected digest range, identity, or digest bytes to callers or session-net diagnostics. A rejected publication that retains the admitted epoch MUST NOT interrupt an active frame. Each logical request pool MUST use exactly one maintenance task to remove cached lanes autonomously at the earliest idle/lifecycle deadline. Maintenance task/table cardinality MUST NOT scale with lanes, subscribers, or records.

The caller owns the request ID for every mutation or lease operation. Only a failure classified as NotTransmitted may be automatically retried, and then only with the identical request ID and body. Positive ciphertext acceptance MUST be observed below TLS as well as at the framed plaintext writer, so a later outer TLS error cannot relabel an accepted prefix as NotTransmitted. Anything possibly written is OutcomeUnknown: the client evicts that lane and MUST NOT replay the request. The SDK MUST NOT mint a new request ID. Recovery may retry only the identical request body under the retained ID through the durable request binding; reuse of that ID for a different request is a closed conflict. Applications otherwise must perform authoritative readback and apply the existing fencing/idempotency contract.

For persistent V2 calls, NotTransmitted is a pre-write result. A ReadUnavailable result is a post-write read loss for a non-effectful V2 capability, history, or status operation and may be retried as that read. OutcomeUnknown { request_id } is returned when an epoch-fenced V2 transition may have been delivered; the exact caller-owned FencedTransitionV2RequestId and complete body remain the recovery identity. The client MUST discard that lane and MUST NOT mint a successor ID or automatically replay the transition; recovery uses exact V2 status under the same ID/body and the durable V2 journal where protection is present. OutcomeUnknownBatch { request_ids } is returned when a V2 batch may have been delivered; request_ids preserves input order, and the client MUST use each matching ID for exact V2 status recovery rather than blindly replaying any mutation.

FencedTransitionStorageExhausted is a retained, complete-body-bound, deterministic no-effect receipt, returned only after ordinary stale-fence, CAS, and lease admission for an otherwise successful transition. Its SQLite representability check covers requested generation/fence, exact acquire fence and global successors, credential allocation, application/watch sequences, and restore-scan revision. While retained, exact ID/body replay and status return Recorded(Err(StorageExhausted)); another body conflicts. No lease, record, watch, restore, or ordinary mutation effect occurs. Existing fenced receipts, generic-ID conflicts, HistoryFull, and RetentionExhausted precede this decision, and revoked authority masks storage state. At maximum application sequence it binds with the current nonzero sequence/digest and advances only logical time, the applied pointer, and the receipt; later blank or membership entries may still apply, without promising normal mutations remain available.

For a fenced transition the public consumer request ID MUST be byte-identical to its nested FencedTransitionRequestId. The quorum adapter namespaces the internal receipt ID by authenticated consumer identity and stable cluster identity, enforces the exact current cluster/configuration/epoch scope under the activation lifecycle above, and submits no separate BindConsumerRequest or binding log entry. The transition receipt therefore binds the complete canonical body in the same single consensus entry as the lease and record effect. A changing authorized configuration scope does not change that internal receipt ID, so an authorized successor can recover the same retained result; a revoked predecessor cannot observe it. Status is read-only, and NotFound does not prove that an earlier delayed proposal cannot still commit, does not permit deletion, and does not permit reuse of the stable transition ID.

Each explicit prewarm MUST perform a rolling resolver/TCP/TLS/Hello refresh of every configured request lane and preserve refreshed plus unprocessed healthy capacity after a partial failure. Prewarm and readiness may prove authenticated consumer transport capacity only; they never prove quorum or product readiness. Readiness deliberately becomes false while a request lane is leased; reserved Watch transport slots are non-gating. Diagnostics are fixed and nonidentifying: setup phase, pool wait, active/maximum/idle counts, reuse/reconnect, queue/in-flight/oldest age, and bounded outcome classes. They MUST NOT include endpoints, identities, scope values, credentials, keys, payloads, request or correlation IDs, owners, or fences. Any performance evidence for this transport is synthetic only and makes no ePDG production-SLO claim.

Reauthentication, accepted material rotation, idle/lifecycle retirement, and the bounded public shutdown apply to both ALPN-isolated persistent pools. Shutdown begins both drain paths together so V2 cannot extend the bounded V1 shutdown window. Per-pool maintenance remains constant; no lane-, subscriber-, or record-scaled maintenance state is introduced.

The v7 qualification profile remains the revision-2 persistent-transport inventory and records its connection, frame, request/response, watch, task, and lifecycle limits beside the consensus profile. The published v6 profile remains the unchanged revision-1 contract. General /1 revision 6 retains the bounded transport properties, the revision-3 generic #696 family, and revision-4 exact retained status recovery for ordinary leases. V2 /2 revision 5 retains its fixed full-width attempt tuple, exact request commitment, exact below-TLS write observation, rolling fresh prewarm, and pool-wide cold-setup serialization. Protected-roster /3 remains an isolated revision-5 roster-only lane. The live v8 exact-head schema remains experimental and fixes qualification_complete=false. No profile or evidence records consumer identity or scope material. Synthetic warm accept/reuse checks gate only their transport method; elapsed samples are non-gating and are not an SLO.

V2 is an additive protocol family; it neither changes the V1 revision-6 wire enum nor upgrades a V1 request. Its exact capability, history, execution, and status outcomes remain on the separate ALPN and V2 identity described above.

Revision 5 retains revision 4's StorageExhausted only inside the closed fenced-transition Recorded status result. Frozen session-net v5 maps this outcome fail-closed as an unknown capability; its wire enum and revision remain unchanged. Product and ePDG composition remain outside this generic API.

13. Local Cache

The SDK SHOULD provide a two-level model:

  1. Local in-process cache for hot reads.
  2. Distributed backend for ownership, recovery, and replication.

Cache entries MUST include generation and fence. Stale cache entries MUST NOT be used for authoritative writes. Data-plane lookup snapshots SHOULD be updated through atomic swap or RCU-like mechanisms.

Cache invalidation options:

  • backend watch stream
  • polling by generation
  • explicit publish from owner
  • TTL expiry

NF owners must choose a cache mode per state class.

14. Security

14.1 Encryption

Session payloads MUST be encrypted before storage unless the profile explicitly marks the backend as inside the same cryptographic boundary.

The production HA composition MUST place encryption or remote sealing above consensus:

application -> EncryptingSessionBackend / RemoteSealingSessionBackend
            -> ConsensusSessionStore -> Openraft -> SQLite/snapshots

Protection MUST finish before client_write. Openraft replication, follower apply, replay, durable request-outcome storage, and snapshot build/install MUST therefore receive only opaque RFC 003 envelopes. The consensus engine, network adapter, and deterministic state machine MUST NOT receive plaintext payloads, an HKMS/KMS provider, key material, or a provider key handle. Read-side decryption/unsealing MUST happen only after the consensus read returns through the wrapper, using the envelope key ID for historical-key selection. Provider unavailability MAY block new protection or plaintext reads, but MUST NOT cause provider I/O during deterministic apply or make already sealed Raft replay and quorum formation depend on provider availability.

The following protected-transition rules apply only to #701's protected V2 composition. Its raw physical store and authenticated-consumer transport execute only the V1 fenced_transition protocol. This is distinct from #702's epoch-fenced V2 protocol, which has its separate FencedTransitionV2Capability::V2, FencedTransitionV2PreparedJournal, 56-byte identity, and /2 consumer lane.

For #701 protected transitions, the outer EncryptingSessionBackend or RemoteSealingSessionBackend advertises AtomicFencedTransitionCapability::V2 only when it owns an SDK caller-side durable PreparedFencedTransitionJournal and composes over an exact V1 physical boundary that explicitly witnesses unchanged protected bytes. This #701 journal, rather than a legacy prepared token or application-persisted request state, is the durable recovery authority. The application retains only the caller-stable FencedTransitionRequestId, which it MUST reuse for that same logical operation after restart.

Protected preparation validates the request, capability-gates the exact inner V1 boundary, and checks that the journal has no binding for the ID before effects. It obtains record-expiry preflight before provider work, seals each create/update body exactly once, leaves delete and refresh provider-free, obtains the inner physical token, and durably inserts the complete outer token with create-only semantics before returning success. Execution, status, and recovery reload and authenticate the exact journal token; execution and status may dispatch only its retained physical bytes. They MUST NOT reseal, unseal, read back, reconstruct, or consult the current active key/provider. Missing, incompatible, corrupt, or byte-mismatched journal state fails closed with no provider or transport I/O; execute reports the known-local case as NotTransmitted, while status and recovery return their typed local fail-closed result. Any may-have-sent result remains OutcomeUnknown under the expected request ID. NotFound is non-exclusion: it never permits deletion or reuse of the ID and does not prove a delayed proposal cannot commit.

The consumer physical bridge is only for use underneath such a protected journaled wrapper. It is atomic-subset-only, fails every unrelated SessionBackend operation locally without I/O, and does not implement SessionLeaseManager. Its opaque local marker commits only the authenticated consumer SPIFFE identity and stable cluster ID; it excludes endpoint/address, TLS server identity, leader, configuration ID/epoch, certificate/key, and material epoch. Authorized endpoint, leader, topology, TLS-leaf, server, and provider/key rotation can therefore use the same durable journal path, key, and volume. This does not claim host failover, host/volume-loss recovery, journal replication, or a second consensus transition.

SessionConsumerPreparedFencedTransitionBackend is the public protected V1 prepared fenced-transition facade for that wrapper. Its persistent_exact_voter_prewarm_roster constructor consumes the complete set of persistent clients for one scope, activates and prewarms every V1 physical voter internally, and returns an opaque roster value. Only the facade's local-AEAD and remote-sealing constructors can consume that value; neither a raw physical backend nor a dispatchable voter value leaves the net crate. A partial roster or any non-V1 voter fails activation. The facade rejects an incomplete, duplicate, or differently authenticated/bound roster and canonicalizes admitted voters by node ordinal; caller input order is not authority. For every caller-stable FencedTransitionRequestId, the router derives the same origin from the authenticated scope, canonical roster, and ID. It uses the existing V1 fenced_transition operation only. It is not #702's V2 receipt-history protocol or FencedTransitionV2PreparedJournal, and grants neither activation or readiness-probe authority nor a new quorum, replication, membership, or consensus authority.

ProtectedFencedTransitionBackend is a sealed, methodless marker; it has no SessionBackend supertrait. EncryptingSessionBackend and RemoteSealingSessionBackend implement it around inner types that separately implement SessionBackend. Therefore the router composes directly over the real SessionConsumerFencedTransitionBackend and MUST NOT grant that physical adapter synthetic lease authority. The existing SessionConsumerPreparedCheckpointBackend remains the distinct complete protected-session path for prepared CAS and lease APIs, which require ProtectedSessionBackend and SessionLeaseManager.

Preparation retains the exact outer protected journal token privately in a move-only affine handle. The sealed boundary MUST NOT expose a dispatchable physical prepared token. The router revalidates that exact outer token before every physical mutation and receipt-status attempt and derives only its authenticated-consumer request view; it MUST NOT reconstruct the request or replace the retained token with current provider/key state. Mutation starts at the deterministic origin and visits the canonical voter roster. It MAY advance to another voter only after a proven pre-dispatch NotTransmitted result. Cancellation during the pre-dispatch setup is safe and retryable because no application bytes can cross the transport boundary. After dispatch begins, a possible send, including OutcomeUnknown, or cancellation is ambiguous and MUST permanently make the handle receipt-only; it MUST NOT regain mutation authority. BeforeCallWrite(SessionConsumerClientError::Scope) is a terminal topology authority revocation and MUST be returned as rejected StoreError::TopologyAuthorityRevoked; it MUST NOT be downgraded to a generic NotTransmitted result or trigger successor mutation dispatch.

Receipt status is read-only. A single status call uses the next deterministic canonical successor voter. A status-until-terminal call repeats bounded canonical-roster passes under the single immutable caller absolute deadline, with every physical attempt additionally capped by the prepared physical-attempt budget. It can end with a deadline; NotFound, unavailable, or a per-attempt deadline remains nonterminal while outer budget remains. None is proof that a delayed mutation cannot commit. A terminal receipt is cached locally. Restart recovery returns only the status-only SessionConsumerRecoveredFencedTransitionStatus handle; it deliberately has no execute authority, even if the recovered token is otherwise valid.

Journal provisioning and reopening are distinct. A deployment MUST call PreparedFencedTransitionJournal::create_new exactly once for a missing path and open_existing on every restart. Reopening never creates or initializes a missing, pristine, truncated, reset, or partial database; the deprecated open alias is reopen-only. The independent stable integrity key is unique to that exact journal path/storage boundary, MUST NOT be reused for another journal, and MUST NOT be logged. On Unix the SDK descriptor-walks and retains the full path chain, requires a private effective-user-owned 0700 parent and regular single-link 0600 file, and revalidates the path, file, and SQLite main-file movement state around each operation. The deployment MUST give that effective user exclusive writer authority over the durable path; an actor with equivalent same-user replacement authority is part of the trusted storage boundary. The containing directory MUST reserve the database leaf and its SQLite sidecar names exclusively for this journal, on a local filesystem with truthful POSIX locking, fsync, directory-sync, and storage-barrier semantics; NFS-like mounts are unsupported. Within one process, callers MUST clone the admitted SDK journal handle instead of reopening the same inode or opening it directly through SQLite. The SDK enforces one live SQLite connection per admitted inode so its pre-open header check cannot release another connection's process-scoped POSIX locks. Platforms without those checks fail closed for the #701 protected V2 composition.

The #701 journal uses zeroize-on-drop HMAC-SHA-256 state, SQLite WAL, a fixed pre-open page/cache-header profile, synchronous=EXTRA, bounded SQLite limits and catalog/membership scans, and bounded opaque rows containing no plaintext. A full authenticated journal rejects a new ID before expiry, provider, or inner-prepare work. Payloads, identities, request IDs, paths, keys, provider material, token bytes, and journal contents MUST NOT appear in examples, fixtures, logs, diagnostics, or evidence. The #701 prepared-token and PreparedFencedTransitionJournal schemas are downgrade fences: unknown versions, raw V1, and older binaries MUST NOT operate this protected journaled path. #702's separate FencedTransitionV2Capability::V2 and FencedTransitionV2PreparedJournal do not upgrade, replace, or share the #701 protocol or journal. The #701 journal layer makes no journal GC, retention, ledger-lifetime, or capacity-lifecycle claim.

The schema-3 journal also commits a fresh per-journal incarnation, bounded membership count, and root over the exact retained request-ID/tag set with a separate HMAC. Health, lookup/recovery, and insertion verify that complete small bounded set; lookup authenticates the selected token, and insertion verifies its new row and updates the membership commitment in its single transaction before commit. A fixed covering index keeps the proof independent of token size. That authenticated index is the presence authority; the bounded proof cross-validates every index rowid, request ID, and fixed tag against the table, and compares independently bounded table and primary-index scans. The schema stores the fixed tag before the potentially overflowing body, so the global proof never reads retained bodies. A selected row then validates its body against the authenticated tag. Divergent table, primary, or secondary-index state therefore fails closed instead of becoming absence. A finite VDBE-work budget applies before schema initialization and to every journal operation, in addition to bounded SQL, catalog, and membership limits. The SQLite catalog is an exact whitelist of the SDK tables, generated primary-key autoindex, and membership index, rejecting every other object, including reserved-prefix catalog entries, before setup. This detects offline row deletion, addition, primary-key replacement, index divergence, and tag corruption inside the same durable file. A corrupt selected body fails its exact row authentication and cannot be treated as absent or rebound. It cannot detect restoration of an older complete valid database snapshot: that rollback is outside the same-durable-file guarantee unless deployment provides an external monotonic anti-rollback anchor.

The token wire form begins with a fixed magic, schema version, and body length; version dispatch precedes decoding the frozen V1 body. A golden compatibility corpus pins both lease forms, every mutation, no-expiry and finite-expiry record shapes, and every supported local/remote/consensus protection-stack shape and order. SDK-owned canonical and complete-body scratch allocations wipe on drop. External serialization and persistence buffers remain the caller's responsibility and inherit the same prohibition on diagnostic or metric emission.

Remote unseal MUST pass the canonical envelope key ID through the provider boundary after validating envelope shape and record AAD. The active remote key is an atomic process-local material epoch used only by future seals; a seal already in flight keeps its selected key ID, but may still fail because of timeout, provider outage, or revocation. Mixed-epoch fleet writes remain readable while KMS retains each envelope's exact key. Missing, revoked, malformed, cross-tenant, and wrong-AAD inputs fail with coarse, redacted crypto errors. Provider, endpoint, tenant, key ID, and payload text MUST NOT be included in those errors.

KMS/HKMS owns remote historical-key retention and retirement. The SDK keeps no local historical material or authorization cache and exposes no retirement API or enforcement gate. It supplies exact key selection and bounded live-state scan inputs, not a rewrap campaign or complete dependency proof. Operators MUST combine a snapshot-bound, write-fenced live-state scan after rewrap with separate compaction, expiry, or inspection of logs and snapshots, plus inspection and rewrap/deletion/retention decisions for backups, restore sources, and rollback checkpoints. A restore scan alone does not prove those retained artifacts. Incomplete/stale evidence or an unbounded record blocks retirement.

EnvelopeV1 MUST be validated rather than trusted as a marker. Construction, wire decode, durable-row decode, log append, replay, and snapshot validation MUST reject a malformed or non-canonical RFC 003 envelope, mismatched embedded key ID, invalid algorithm nonce/tag shape, non-session AAD, or mismatch between the AAD's visible tenant/NF/state/generation/fence fields and the record. Consensus admission of a SQLite file MUST atomically fence all standalone backend operations through retained or newly opened handles; only internal state-machine apply and barrier-gated committed reads may bypass that fence.

AAD MUST include:

  • tenant
  • NF kind
  • session key digest
  • state type
  • generation
  • fence
  • backend namespace

The bounded iterative transformation in §11.2.1 is mandatory for replication wrappers. Protecting only the root or one Batch level is not conformant. The envelope protects payload bytes, not the complete SQLite database. Raft and SQLite metadata—including membership, terms/indexes, tenant and key routing, owners, generations, fences, timestamps, request identities, and envelope key IDs—remains visible to the host storage boundary. A deployment requiring metadata or full-file encryption MUST add and qualify an approved database or volume layer without moving provider access below the wrapper. Scoped three-node in-process Openraft evidence uses actual file-backed nodes and controllable RPC, explicitly forces snapshot installation, shuts down/restarts, and restores both key epochs while counters prove replication, replay, quorum formation, and snapshots perform no provider calls. Production KMS framing is tested separately. This does not qualify multi-process or deployed-network behavior; distributed protection/failure/soak evidence (#143) remains a separate production-profile gate.

14.2 Integrity

AEAD integrity is required. Additional MAC fields MAY be used for backends that need independent integrity checks, but they do not replace AEAD.

14.3 Privacy

Logs and metrics MUST NOT expose raw subscriber identifiers. The SDK SHOULD use stable keyed digests for correlation when needed.

14.4 Transport Credential Rotation

Session TTL is application-state lifetime and MUST NOT be used as a certificate lifetime, trust-bundle lifetime, or maximum-authentication-age policy. A production networked session-store profile MUST rotate workload certificates and trust bundles without interrupting service, use short-lived SVID expiry as the bounded same-issuer credential-compromise/revocation response, and document a maximum authentication age on long-lived connections. #161 atomic reload, #162 coherent material epochs, and #163 finite connection retirement/reauthentication are implemented. On epoch change or an explicit orchestration request, both sides MUST stop new admission, end the transport wait and release connection slots within the finite hard deadline, and repeat the full mutual-TLS and application handshake on replacements. Already-admitted supervised mutations retain the ambiguity/readback contract above if their bounded backend work finishes later.

Rotation and reauthentication move cooperative participants but do not revoke the old certificate/key. Its holder can establish a fresh connection until the earliest expiry across every certificate in that presented chain while its issuer remains trusted. Immediate generic CRL, OCSP, certificate/identity-denylist, and other selective same-issuer revocation are not implemented. Root removal is a trust-anchor cutover for all chains that depend on it, not an expiry deadline or selective revocation.

The projected source's ongoing expiry monitor clears retained source material at leaf expiry. It is not the authority for an earlier intermediate expiry. TlsMaterialController MUST pre-scan every configured SVID-chain certificate, mark material unavailable at the earliest effective chain expiry, and provide the TLS readiness status. A production projected source MUST be paired through TlsMaterialController::new_from_projected_source or new_pinned_from_projected_source; direct subscription through a generic controller constructor does not bind source failures and controller gauges to one recorder. Source Ready alone MUST NOT satisfy this section.

An operator MUST publish overlapping old/new trust before new leaves, preserve the exact stable SPIFFE and consensus scope, trigger reauthentication, and verify that every directed peer path has authenticated on current material before removing old trust. Removing that anchor cuts over later handshakes; trigger reauthentication and prove that every chain depending on it is rejected. The negative proof MUST remain visible to authentication/trust alerting and MUST be accepted only when an immediate checkpoint proves the exact qualified per-member delta with no concurrent increase, process reset, or alert silence. Rollback before old-trust removal restores the prior leaf/material publication and triggers another monotonic reauthentication generation; rollback after removal MUST first restore overlapping trust and prove the controller status is Ready, then restore the old leaf and trigger reauthentication. A rollback MUST NOT reuse an old authenticated connection as evidence. Its deadline MUST be derived from the exact fleet size and all bounded two-pass operations, and the selected complete rollback material MUST be revalidated against the deadline remaining immediately before every publication. Evidence MUST bind one invocation, live-lease binding, monotonic operation/nonce, exact member/checkpoint, phase/step, and fresh timestamp; it MUST NOT contain the lease token. Serving withdrawal MUST remain executable when evidence storage is unavailable. Reconnect-storm, deployed root cutover, real partition/restart including active-mutator crash/restart, broader fault behavior, deployed mixed traffic/watch/restore, resource/soak, remote-HKMS, deployed-CNF, signed release, and wider distributed production evidence remain #164/#143 under umbrella #158. The single-host tests described in §12.3 cover bounded mixed traffic only through their exact synthetic fault/expiry slice and make no evidence-schema/profile claim. They do not change Openraft's sole commit authority, payload encryption, AAD, key-provider/HKMS placement, durable formats, or encryption-at-rest responsibilities.

15. Observability

Required metrics:

  • opc_session_store_ops_total{op,state_class,outcome}
  • opc_session_store_latency_seconds{op,state_class}
  • opc_session_store_cas_conflicts_total{state_class}
  • opc_session_store_stale_fence_total{state_class}
  • opc_session_lease_acquire_total{outcome}
  • opc_session_lease_renew_total{outcome}
  • opc_session_lease_lost_total{reason}
  • opc_session_replication_lag_seconds{region}
  • opc_session_cache_hit_ratio{state_class}
  • opc_session_record_bytes{state_type}
  • opc_session_restore_pages_total{outcome,cursor_profile,complete}
  • opc_session_restore_page_records{cursor_profile}
  • opc_session_restore_page_examined{cursor_profile}
  • opc_session_restore_page_payload_bytes{cursor_profile}
  • opc_session_net_connection_retirements_total{reason}
  • opc_session_net_connection_lifecycle{state}
  • opc_session_net_connection_drain_events_total{event}
  • opc_session_net_connection_attempts_total{outcome}
  • opc_session_net_reconnect_events_total{outcome}
  • opc_session_net_watch_slow_consumers_total

The lifecycle metric labels MUST come only from their closed SDK-owned reason, state, event, and outcome enums. They MUST NOT contain endpoints, DNS names, SPIFFE IDs, certificates, key material, transaction IDs, record keys, or payload/backend text. The closed connection-retirement reason set includes idle_timeout; it MUST NOT be counted as the timeout connection-attempt failure. Resolver, TCP, TLS, bootstrap, and partial active-frame deadlines remain timeout failures. When the transport observes a newer material or explicit-reauthentication epoch, it MUST terminate the old attempt as superseded. When an attempt guard is dropped before any explicit terminal classification, it MUST use abandoned rather than infer timeout or supersession. Both outcomes participate in the quiescent started = terminal + outstanding accounting invariant but MUST NOT be treated as peer timeouts. Exporters MAY expose transient skew between the separate relaxed counters while connection handlers are changing state.

  • opc_session_restore_page_latency_seconds{cursor_profile}
  • opc_session_restore_restarts_total{reason} where reason is one of stale_cursor, work_budget, response_too_large, or cancelled

Restore metric labels MUST NOT include cursor bytes, key fields, tenant, owner, payload, peer-controlled text, paths, or certificate identity. A product MAY expose these metrics through its existing metrics facade; #133 does not add a second registry or metrics authority.

Required logs for state transitions:

  • session_key_digest
  • tenant
  • state_class
  • generation
  • fence
  • owner
  • handover_tx_id, when applicable
  • outcome

Raw subscriber identifiers MUST be redacted.

16. Module Ownership

ModuleResponsibility
opc-session-modelKeys, record headers, generations, state classes
opc-session-backendBackend trait and capability model
opc-session-leaseLease manager and fencing rules
opc-session-cacheLocal cache and snapshot publication
opc-session-codecSession serialization and migrations
opc-session-cryptoPayload envelope integration with RFC 003
opc-session-replicationRegion log and apply rules
opc-handoverGeneric handover storage state machine
opc-session-testkitFake backend, split-brain tests, stale fence tests
opc-consensusThe workspace's single Openraft import, identity, bounded codec, and consensus transport contracts
opc-session-store::consensusOpenraft adapter, deterministic session state machine, SQLite log/state/snapshot storage, and linearizable readiness
opc-session-net consensus profileMutual-TLS consensus-only peer transport; no direct backend mutation or rebuild authority

Agents implementing backends must not modify NF-specific handover logic. Agents implementing handover logic must use the public lease/CAS APIs and not bypass fencing.

17. Testing Requirements

17.1 Unit Tests

  • Session key tenant separation.
  • CAS success and conflict.
  • Stale fence rejection.
  • Lease acquire/renew/release.
  • TTL refresh with valid and stale fences.
  • TTL zero, the exact 365-day maximum, maximum plus one, and Duration::MAX across direct, batch, replicated, persisted, and authenticated-wire paths; rejected values must have no partial effect.
  • Serialization corrupt input rejection.
  • Protocol-v5 golden frames with no target-width integer fields; checked fixed-width maximum/overflow conversion; exact collection limits; omitted restore fields recomputed; and size errors nested in batch results.
  • Revision-2 negotiation with equal and unequal client/server limits, rejection below MIN_NEGOTIATED_FRAME_SIZE (8,192 bytes), the restore-minimum alias, executable conservative maximum-payload round trips, and fail-closed revision-1/revision-2 profile mismatch.
  • Exact-limit and one-byte-over outbound encoding for every response/watch family; no oversized allocation or emitted prefix on rejection; non-truncated record/batch behavior; contiguous log and cursor-correct restore pages; fixed fallback redaction; and iterative consuming rejection of nested trees.
  • One absolute write deadline for prefix/payload/flush; authenticated slow-reader reaping; handler/connection-slot return to baseline; repeated reconnect bounds on memory/tasks/file descriptors/CPU; and deterministic shutdown/abort while response serialization or socket writes are blocked.
  • Exact-v5 handshake success plus older ALPN/version, profile, authentication, malformed acknowledgement, and replay rejection before backend dispatch; incompatible peers clear cached capabilities to all false/zero.
  • Exact 1-byte and 128-byte owner/custom-key acceptance, empty and 129-byte rejection, canonical reserved-name handling, string ordering, and hostile Serde/session-net decode rejection without raw-value disclosure.
  • Exact stable-ID 1/64-byte and replication-transaction-ID 1/128-UTF-8-byte acceptance/rejection, plus canonical lowercase hyphenated 36-byte CAS UUID admission across requests, responses, batches, nested replication carriers, log pages, and watch items.
  • Valid legacy SQLite hydration; hostile owner/key types in records, active leases, key fences, and nested replication logs; no-effect rejection; and the bounded count-only audit's budgets, status/exit codes, and redaction.
  • Versioned and bounded/current-valid original handover-envelope round trips; exact non-OPCH classifier cases (including ambiguous bare rejection); and malformed, zero-length, truncated, oversized, and typed-invalid rejection before mutation.
  • AEAD AAD mismatch rejection.
  • Nested replicated CAS protection at depths 1 through 16, rejection at depth 17, exact 256-node acceptance and 257-node rejection, and fieldless errors.
  • Replicate/rebuild/log/watch round trips through encryption and remote-sealing wrappers, including late-provider failure with no backend delegation or partial entry/page exposure.
  • Cache generation checks.

17.2 Integration Tests

  • Two owners racing for the same session.
  • Owner pause beyond TTL, new owner writes, old owner resumes and is rejected.
  • Handover prepare/activate/abort idempotency.
  • Backend restart with leases recovered or invalidated according to profile.
  • Geo-replication applies newer generation and rejects older generation.
  • Cache invalidation after remote update.
  • Coordinated v5 multi-replica admission and fresh-readiness behavior, including fail-closed mixed-profile peers and non-authoritative cached capabilities.
  • Ambiguous mutation outcomes under response rejection/write timeout, proving callers recover through idempotency, fencing, and authoritative re-read rather than assuming rollback or blindly replaying the operation.
  • A real Openraft proposal that commits before its forwarded result is delayed beyond the caller deadline returns typed ambiguity and produces exactly one durable application-journal event.
  • Real SQLite external write-lock contention and async-future cancellation are bounded, retain at most one supervised worker, release it after interruption, and never classify a started mutation as safely retryable.
  • Concurrent pristine three-node formation and mutation submission with one gap-free committed application journal on every replica.
  • A one-node partition produces bounded readiness/write failure, then heals and rejoins without admitting a second authority path.
  • Cross-node lease/CAS visibility and follower linearizable reads use the same Openraft barrier as probe_durable_readiness.
  • Plaintext canaries written through the encryption wrapper are absent from SQLite database/WAL/SHM files, Raft logs and outcomes, captured consensus frames, and snapshots; restart and active-key rotation retain decryptability.
  • Non-ignored three- and five-process single-host projected-mTLS cases combine one stable follower's test-only consensus-RPC admission loss with a different member's malformed-trust retained-last-good state, then prove survivor readiness/encrypted-canary progress, exact-address restart/catch-up, and repair. They separately drive a same-issuer leaf with a 75-second remaining-validity/expiry budget through its fixed 30-second soft-retirement window, hard drain, LastGoodExpired, survivor progress, and same-process valid replacement. Replacement advances only the recovered member's explicit reauthentication generation, proves fresh bidirectional mTLS/bootstrap paths on every incident edge, leaves unrelated survivor explicit/material-epoch retirement counters unchanged, and settles all lifecycle drains plus survivor availability episodes before the next traffic baseline. The schedule-bound member-scoped-reauth-settled-baseline/v4 checkpoint starts its 86-second absolute bound and 60-second two-stage server tail at the atomic projected-data rename, then requires a final 2.5-second outbound-ledger quiet tail. A prepublication common-key pulse and conservative 13-second observations require one active key to advance on every survivor observer and bound that pulse's worst-case actual event gap to 26 seconds. An independent 26-second checkpoint requires every active key on every observer and cannot be reset by a faster key. Each survivor may record at most one availability episode while the expired member rejoins. Consecutive typed retry outcomes inside that episode remain separately bounded by the unchanged eight-outcome ceiling; all must settle inside the 26-second SLO, and a second or late episode fails closed. Fault-era new-attempt and reconnect deltas retain a fixed 85/161 per-node bound: ordinary 24/40, fifteen five-second refresh rounds over four/eight incident paths, and one scheduled post-hard-expiry survivor-to-expired network-negative attempt per involved node. The reverse probe fails local material preflight without dialing. Terminal outcomes may additionally include only the exact attempts already outstanding at the baseline and must satisfy interval conservation; Schedule v6 binds new-attempts-plus-baseline-outstanding/v1 and common-key-pulse-all-active-key-coverage/v1. Cancellation-classified abandoned outcomes, protocol/backend outcomes, and drain overruns remain forbidden; the clean scoped-reauthentication interval retains a zero-failure budget. Continuity polling is a non-intrusive workload snapshot; authoritative final watch-head settlement still performs the fail-closed replication-head read.

Run those two exact cases serially:

cargo test --locked -p opc-session-testkit --test qualification_mtls_multiprocess --no-default-features three_process_projected_mtls_unavailable_malformed_and_expiry_recovery -- --exact --test-threads=1
cargo test --locked -p opc-session-testkit --test qualification_mtls_multiprocess --no-default-features five_process_projected_mtls_unavailable_malformed_and_expiry_recovery -- --exact --test-threads=1

They are synthetic regression evidence, not a deployed network partition or a production qualification. The v2 stage correction does not relax Openraft's sole commit authority or change HKMS/provider placement, payload encryption, AAD, SQLite/Openraft durable formats, or encryption-at-rest responsibilities. The bounded lease/CAS/read, watch, restore-scan, readiness, and connection-recycling workload remains active throughout both cases, and a restarted watcher reconciles the exact committed journal prefix before resubscription.

17.3 Fault Injection

  • Backend timeout.
  • Partial batch failure.
  • Redis/Aerospike failover.
  • Clock skew.
  • Network partition between owners and backend.
  • Replication lag spike.
  • Corrupt encrypted payload.
  • Missing session key decryption key.

17.4 Performance Gates

Profiles must state which backend they apply to. Minimum SDK reference gates:

  • Local cache read p99 under 50 microseconds.
  • In-memory fenced CAS p99 under 100 microseconds.
  • Backend adapter exposes measured p50/p99 for get, CAS, lease acquire, and renew.
  • 100,000 updates/second per replica for in-memory or batched local profile.
  • No packet fast-path benchmark depends on remote backend availability.

18. Acceptance Criteria

This RFC is implemented when:

  1. Authoritative session writes require monotonic fencing and CAS.
  2. Stale owners cannot overwrite newer session state after lease expiry.
  3. Handover state transitions are idempotent and recoverable.
  4. Geo-replication does not use wall-clock LWW for authoritative state.
  5. Backend capabilities are declared and enforced by profile.
  6. Session payloads are encrypted and tenant-bound.
  7. Local cache supports fast reads without compromising write correctness.
  8. Fault injection covers split-brain, failover, replication lag, and stale fences.
  9. Every Duration-based TTL boundary accepts zero and the exact 365-day maximum, rejects larger values with the appropriate typed error before application/backend effects, and performs exact checked deadline arithmetic without unwinding.
  10. Every replication operation tree is iteratively bounded to depth 16 and 256 total nodes; every nested CAS is protected on write and unprotected on read; and transformation failure cannot delegate or expose a partial entry/prefix/page.
  11. Owner IDs and custom session-key types have structural 1-through-128-byte invariants at every model, persistence, and transport decode boundary; legacy SQLite admission is bounded, count-only, read-only, and fail-closed; and invalid state is never silently rewritten.
  12. ConsensusSessionStore is the only quorum-profile authority, all election, voting, log matching, commitment, membership, snapshots, and linearizable reads use the shared Openraft engine, and raw append/rebuild/lease sequencing cannot bypass it.
  13. Durable readiness executes an Openraft linearizable barrier and waits for local committed apply; listener bind, TLS success, capabilities, local SQLite availability, and restore method availability cannot report ready.
  14. The encryption/remote-sealing wrapper runs above consensus, plaintext and provider/key handles never enter Raft apply/log/snapshot transport, and the documented payload-envelope versus full-database boundary is qualified.
  15. Divergence recovery (#128), operator-safe legacy-fork recovery (#129), bounded applied-state restore (#133), and finite connection reauthentication (#163) are implemented; distributed production qualification (#143) and fleet credential-rotation evidence (#164) under umbrella #158 have passed their own acceptance gates before a production claim (#161/#162 are implemented prerequisites).