ZeroLock Benchmarking: Current State, Gaps, and a Plan to Get to "Real-World Qualified"

Status: Draft — working document for the engineering plan below Scope: What a defensible benchmark suite for ZeroLock needs to contain, what already exists in this repo, what's missing, and the concrete steps to close the gap.

1. Why this document exists

"Benchmark" gets used loosely. For a concurrency-correctness substrate like ZeroLock, a credible benchmark suite actually has to answer two separate questions, and most of the value is in not conflating them:

1. Does it stay correct under adversity? — races, partial failures, malicious/broken agents, cross-shard staleness. This is the question a systems engineer evaluating ZeroLock for production will ask first, and it's a correctness question, not a speed question.
2. How fast is it, and how does it degrade under load? — throughput, latency percentiles, coordination overhead as agent count scales. This is the question that determines whether ZeroLock is usable at the concurrency levels a real deployment needs.

The industry has established, load-bearing methodologies for both. Question 1 is the domain Jepsen defined and that the wider distributed-systems community (CockroachDB, Cassandra, etcd, etc.) is now held to. Question 2 is the domain YCSB defined for NoSQL/distributed stores and that's still the reference point people expect (throughput-vs-latency curves, reported until the system saturates).

This document inventories what ZeroLock already has toward each, what's missing, and what to build next — in priority order, using the harness scaffolding that already exists rather than starting over.

2. What already exists (verified, not aspirational)

This is not a from-scratch situation. packages/harness is a real, working adversarial test harness that executes against a live gateway over the actual wire protocol — no mocks. It was independently verified during this engagement (each scenario run and its pass/fail confirmed live), and the scorecard aggregator (runHarness() + formatScorecard() in packages/harness/src/runner.ts, invoked via packages/harness/src/cli.ts) already runs all scenarios end-to-end and reports pass rates, failure reasons, and summary statistics per scenario.

2.1 The nine harness scenarios

Run any/all of these today with:

cd packages/harness
node dist/cli.js --target http://localhost:8080 --runs 50 --scenario ZL-001 --verbose
node dist/cli.js --target http://localhost:8080 --runs 20        # all scenarios, human-readable scorecard
node dist/cli.js --target http://localhost:8080 --runs 20 --json # CI-friendly

2.2 Applied, real-world-shaped verification (this engagement)

Beyond the harness scenarios, this engagement built and ran a full applied multi-agent scenario (the "Research Desk": researchers claiming tasks, appending findings, editors racing on a thesis, a publisher gated by Layer-4 approval) three separate ways, each producing real measured results, not simulated ones:

A. Against the ZeroLock SDK directly (examples/research-desk-demo/demo.py) — verified at multiple scales:

• 5 researchers / 8 tasks: correct convergence, zero duplicate claims, zero lost findings, exactly one thesis winner, approval gate held.
• Scale sweep on the task-claim protocol specifically: task-claim attempt counts scaled from 18 (at 5 concurrent agents) to ~2,000 (at 50 agents) to ~6,700 (at 80 agents) — a real, measured, non-linear coordination-overhead curve on a single contended decision shard. This is the concrete data point behind ZL-004/ZL-005's "coupling sweep" and "thrash point" concepts; it's real, not modeled.

B. Against a naive, ZeroLock-free baseline (examples/research-desk-demo/naive-baseline/) — identical scenario logic, raw HTTP against an unversioned key-value store, zero coordination:

• 5 researchers / 8 tasks: 20 claim-results for 8 unique tasks (12 wasted/duplicate claims), 16 findings recorded for only 7 unique questions (1 task's result silently overwritten).
• 50 researchers / 100 tasks: 996 claim-results issued, but only 74/100 tasks ever marked completed; 195 findings recorded but only 84 unique questions actually covered; no thesis exclusivity enforced at all; no publish gate — publish just happens, unconditionally.

This is the first-party "before/after" evidence: not a claim about what naive coordination would do, but a measured run of the same scenario, same scale, same machine, with the safety layer removed.

C. Against a real LangGraph StateGraph (examples/langgraph-demo/research_desk_langgraph.py) — each researcher agent is a compiled LangGraph graph using ZeroLockSaver as its checkpointer and create_zerolock_tools() for its tool calls (not the raw SDK called directly):

• 4 researchers / 8 tasks and 6 researchers / 8 tasks: both OVERALL: PASS, and — the part specific to this test — the LangGraph checkpoint was independently read back from ZeroLock after each run to confirm it was genuinely persisted (4/4 and 6/6), not just that the graph didn't crash.
• This also surfaced and fixed three real integration bugs in zerolock_langgraph/checkpointer.py that had never been caught before, because the project's original "LangGraph demo" never actually imported langgraph.

None of A/B/C are academic-literature numbers or projections. They're measured runs against the real gateway and real client SDKs, captured in this repo (examples/research-desk-demo/README.md, examples/research-desk-demo/naive-baseline/README.md, examples/langgraph-demo/README.md).

2.3 What this adds up to

For a project at this stage, having (a) a working adversarial harness with 9 scenario families matching most of the categories a rigorous reviewer would ask for, (b) a scorecard aggregator already wired, and (c) real measured before/after and framework-integration evidence is unusually strong. The gap to "real-world qualified" isn't starting from zero — it's four specific, well-scoped additions.

3. The gap: what's missing before this is "real-world qualified"

3.1 No real infrastructure-fault nemesis (the biggest gap)

Every fault condition tested so far — in the harness scenarios and in the applied demos — is injected at the application layer: timing races between concurrent HTTP calls, malformed payloads from a "Byzantine" agent, degraded judge-quality simulation. None of it kills the actual gateway process, drops a WebSocket connection mid-sync, or restarts the gateway and checks recovery.

This is the core of what Jepsen actually tests (via its "nemesis" component): network partitions, process pauses/kills, clock skew — real infrastructure faults, not just concurrent well-behaved-except-for-timing clients. It's also the part of the CLAUDE.md architecture (Layer 4 irreversibility, the audit log, the SQLite/Postgres/Redis storage adapters) that currently has zero test coverage against an actual crash. "The gateway is the single stateful authority" is a design claim; nothing today proves what happens to in-flight commits when that single authority dies mid-write.

What's needed: a new scenario (proposed ZL-010) that, mid-load, SIGKILLs the gateway process, restarts it, and verifies: (a) no shard is left at an impossible/partial version, (b) every commit that returned success to a client is durably present after restart, (c) no commit that returned an error or timeout is silently present (no phantom writes), and (d) the audit log is consistent with the recovered state. This should run against each storage adapter (SQLite at minimum; Postgres if time allows) since durability behavior can differ by adapter.

3.2 No generic, scenario-agnostic correctness checker

Every current scenario checks its own hand-written invariant ("findings count equals task count," "exactly one thesis winner"). That's fine for demonstrating the scenario works, but it means the "proof" of correctness is only as trustworthy as the scenario author's own bookkeeping — a skeptical reviewer has to trust the assertions, not just the data.

Jepsen's answer to this (via its Elle checker) is a generic history analyzer: record every operation and its result as a plain log, independent of what the scenario was "about," then verify the resulting history against a formal model (e.g., "is this history linearizable," "did every append survive," "is there a cycle implying a non-serializable anomaly") using only the raw log, not scenario-specific assertions.

What's needed: a lightweight op-log recorder wired into the harness's ScenarioContext (every getShard/commitShard/appendItem call logged with timestamp, agent, request, and response) plus a standalone checker that, given only that log, verifies (a) OCC/decision-shard linearizability — no two commits both "succeeded" against the same version, and every successful commit's version strictly increases — and (b) CRDT/additive-shard convergence — every append that got a success response is present in the final state, with nothing extra. This turns "trust the scenario's assertions" into "verify from raw data independently," which is the actual credibility upgrade.

3.3 No throughput/latency benchmark under sustained mixed load

Everything measured so far answers "did it stay correct" and, incidentally, "how long did the whole scenario take." Nothing answers the YCSB-style question: at a given number of concurrent agents issuing a realistic mix of operations, what's the sustained throughput (ops/sec) and what does the latency distribution (p50/p95/p99) look like — and where does that curve bend as concurrency increases?

ZL-005 (scale sweep) is the closest existing piece — it already sweeps agent count (2/4/8/16 in the harness version; manually verified to 80 in the applied demo) — but it currently reports success/conflict counts on a single burst of simultaneous commits, not sustained-load latency percentiles.

What's needed: extend ZL-005 (or add ZL-011) to run a fixed-duration, mixed-workload benchmark — a configurable read/decision-commit/additive-append ratio (a reasonable default mirroring the Research Desk scenario: majority reads, a minority of contended decision writes, a minority of CRDT appends), ramping concurrency (5 → 25 → 50 → 80 → 150), reporting ops/sec and p50/p95/p99 latency at each level until throughput stops increasing (the saturation point YCSB reports). Feed this into the scorecard as a table + a simple degradation-curve plot.

3.4 Only one baseline, and it's a low bar

The naive-baseline comparison (Section 2.2.B) proves ZeroLock beats no coordination at all. That's real evidence, but it's the easiest possible bar to clear — any engineer already knows raw unversioned writes under concurrency are unsafe. The comparison that actually moves a skeptical evaluator is against what a competent team would build instead of ZeroLock: optimistic locking with Redis WATCH/MULTI, or Postgres SELECT ... FOR UPDATE / advisory locks. If ZeroLock doesn't clearly win against that — on safety, on coordination overhead at scale, or on lines-of-application-code needed to get the same guarantee — that's important to know now, not after a customer builds the comparison themselves.

What's needed: a second baseline implementation of the identical Research Desk scenario using Redis optimistic locking (or Postgres row-level locking) as the coordination layer, run at the same scales as the ZeroLock and naive versions, reporting the same metrics (duplicate claims, lost writes, coordination overhead curve, and — importantly — lines of coordination-specific code in the agent logic).

3.5 No reproducibility packaging

Numbers without a rerunnable methodology don't carry much weight with a technical audience. Every result in Section 2.2 currently lives in prose in various READMEs, generated from ad hoc runs during this engagement, without pinned seeds, documented hardware/environment, or a single command that reproduces the full scorecard.

What's needed: once 3.1–3.4 are in place, a single packages/harness command (or thin wrapper script) that runs the full suite — all 9+ scenarios, the throughput sweep, both baselines — against a fresh gateway instance and emits one scorecard document (numbers + methodology + environment description) that's independently rerunnable. This is what makes the whole exercise "real-world qualified" rather than "numbers we generated once."

4. Action plan, in priority order

Items 1 and 2 are the recommended starting point: both build directly on code that already exists and passed live verification during this engagement, and together they close the two gaps (real infra faults, real latency data) most likely to be the first questions a technically serious evaluator asks.

5. Proposed final scorecard shape

Once items 1–5 are in place, the target output is a single scorecard with three sections, mirroring the structure the project's own harness design (see CLAUDE.md) already anticipates:

• Correctness scorecard — per-scenario pass rate across ZL-001–ZL-003, ZL-006, ZL-007, ZL-009, ZL-010 (the new infra-fault scenario), with the generic op-log checker's independent verdict alongside each scenario's own assertions; structural-corruption rate and staleness-bound-violation count, both target zero.
• Performance scorecard — ZL-004 (coupling sweep) and ZL-005/ZL-011 (scale sweep + mixed-workload throughput) as ops/sec-vs-latency curves at each concurrency level, with the coordination-overhead crossover point called out explicitly (already have real data suggesting this exists between 50 and 80 concurrent writers on a single contended decision shard).
• Comparative scorecard — ZeroLock vs. naive-no-coordination vs. Redis/Postgres-optimistic-lock, same scenario, same scales, same metrics (duplicate/lost-write rates, coordination overhead, coordination-specific LOC).

6. References

• Jepsen — Distributed Systems Safety Research — the methodology (generators, nemeses, model-checkers) behind Section 3.1/3.2.
• Jepsen Analyses — examples of the report format Section 5 is modeled on.
• Testing Distributed Systems (curated resource list)
• Benchmarking Consistency Levels of Cloud-Distributed NoSQL Databases Using YCSB — throughput-vs-latency methodology behind Section 3.3.
• How to Evaluate Distributed Coordination Systems? — A Survey and Analysis

This document should be updated as items in Section 4 are completed — treat Section 2 as the "verified" ledger and move items out of Section 3 into it only once there's a real, rerun-and-confirmed result to cite, following the same standard applied to everything already in Section 2.