Example — prod-demo-suite

Production Demo Suite

Four production-shaped scenarios — code refactoring, code review, trade execution, data enrichment — each run twice with real Gemini agents: once uncoordinated, once with ZeroLock.

llm: Gemini 3.6 Flash
agents: 5 concurrent
gateway: localhost:8080

Executive summary

14 → 0

Data loss events

across all 4 scenarios

3 → 0

Duplicate operations

task/batch claim races

85–100% → 100%

Completion rate

unique work actually done

100%

Approval gates enforced

irreversible actions blocked without one

Across the 4 scenarios, the uncoordinated runs lost 14 items and duplicated 3 operations — with every run still reporting success. The coordinated runs, using the same agents and the same model, lost nothing and enforced every approval gate. Full numbers per scenario are below; this is a single run, not a statistical claim — see limitations.

The problem

Three ways coordination fails — silently. No exception, no error log. The system reports success while the data is wrong.

Lost updates

write-write conflicts

Agent A reads file (version 1)
Agent B reads file (version 1)
Agent A writes changes -> file now version 2
Agent B writes changes -> file now version 3
  (Agent A's changes LOST)

Duplicate work

task-claiming races

Agent A checks task queue -> sees "task-1" available
Agent B checks task queue -> sees "task-1" available
Agent A claims "task-1" and starts work
Agent B claims "task-1" and starts work
  (DUPLICATE)

Unauthorized mutations

missing approval gates

Agent executes irreversible action (payment, deployment)
No approval required -> action proceeds
Error discovered -> CANNOT ROLLBACK

The Four Scenarios

Code Refactoring Swarm

Task: Add 8 new functions across 2 Python files
Agents: 5 concurrent refactoring agents
Coordination needed: File locking + task claiming

without zerolock

[15:29:14] [naive-agent-1] Taking task: add-sqrt
[15:29:14] [naive-agent-2] Taking task: add-sqrt
[15:29:14] [naive-agent-3] Taking task: add-sqrt
[15:29:14] [naive-agent-4] Taking task: add-sqrt
[15:29:14] [naive-agent-5] Taking task: add-sqrt

All 5 agents grabbed the same task simultaneously. Each read the file, added its function, and wrote back — only the last writer's changes survived.

with zerolock

[15:29:33] [agent-2] Lost race for 'add-sqrt' (owner: agent-3)
[15:29:33] [agent-3] Claimed task: add-sqrt
[15:29:33] [agent-3] Acquired lock on math_utils.py
[15:29:33] [agent-2] Claimed task: add-absolute
[15:29:33] [agent-2] Waiting for lock on math_utils.py...

Task claiming picks one winner per task (write-wait-reread). File locks serialize edits. All 8 functions land.

agents.py — OCC file lock with retry

for attempt in range(60):
    locks = (await client.get_shard(locks_shard)).value or {}
    holder = locks.get(file_name)

    if holder and holder != agent_id:
        await asyncio.sleep(0.5)
        continue

    locks[file_name] = agent_id
    try:
        await client.commit_shard(locks_shard, locks, version)
        return  # lock acquired
    except StaleCommitError:
        await asyncio.sleep(0.1)

Metric	Naive	ZeroLock
Tasks completed	8	8
Functions lost	5	0
Duplicate tasks	2	0
Overwrites	11	0
LLM calls	13	8

Why this matters — The naive run reported 100% task completion while silently losing 5 of 8 functions. Completion rate is not a correctness signal.

AI Code Review Platform

Task: 5 specialist reviewers analyze a PR in parallel
Agents: Security, Performance, Style, Test Coverage, Docs + 1 synthesizer
Coordination needed: Concurrent append to a shared findings list

without zerolock

[naive] reviewer-performance: found 2 issues, appending
[naive] reviewer-style: found 2 issues, appending
[naive] LOST: performance/race_condition_list_corruption
[naive] LOST: style/race_condition_list_corruption

Classic append race: two reviewers read the same list, each appends locally, the second write wins.

with zerolock

[15:33:17] [reviewer-performance] Found 2 performance issues
[15:33:17] [reviewer-performance] Submitted 2 findings
[15:33:24] [reviewer-docs] Found 4 docs issues
[15:33:24] [reviewer-docs] Submitted 4 findings

Findings live on an additive (CRDT array) shard. Concurrent appends merge unconditionally.

agents.py — CRDT append

async def submit_findings_node(state: ReviewerState) -> ReviewerState:
    """Submit findings using CRDT append."""
    for finding in state["findings"]:
        await client.append_item(findings_shard, finding)

    return {**state, "submitted": True}

Metric	Naive	ZeroLock
Categories covered	5/5	5/5
Total findings	13	11
Findings lost	2	0
LLM calls	6	6

Why this matters — A lost finding means a security issue slips through with no trace. The naive run lost 15% of findings silently.

Financial Order Processing

Task: Process trade orders with mandatory risk review
Agents: 3 market analysts + 1 risk agent + 1 executor
Coordination needed: Layer 4 approval gate on an irreversible shard

without zerolock

[executor] Executing order: BUY 150 AAPL @ $178.50
[executor] Order executed successfully

No approval step exists. The executor commits trades with no risk review.

with zerolock

[15:35:03] [executor] Requesting approval for order execution...
[15:35:03] [risk-officer] Reviewing: Execute BUY 150 AAPL @ $182.45
[15:35:03] [risk-officer] Approved order execution
[15:35:03] [executor] Order approved and executed

The execution shard is marked reversible: false. The gateway rejects direct commits without an approval token.

agents.py — executor blocked without approval

if not risk or not risk.get("approved"):
    return {**state, "approval_granted": False}

approved = await approval_callback(
    order_id=order["id"], order=order,
    risk_score=risk.get("risk_score", 0.5),
)

Metric	Naive	ZeroLock
Orders processed	3	3
Approval gates enforced	0	3/3
Unauthorized executions	possible	blocked

Why this matters — For irreversible actions, the gate must live in the protocol. This tests that the gateway itself refuses the write.

Customer Data Enrichment

Task: Enrich 10 customer records across 5 batches
Agents: 5 parallel enrichment agents
Coordination needed: Batch claiming + result aggregation

without zerolock

[naive-enricher-1] Processing batch: batch-1
[naive-enricher-2] Processing batch: batch-1  ← DUPLICATE

Two enrichers picked up the same batch. One agent's write overwrote the other's.

with zerolock

[15:38:08] [enricher-1] Lost race for batch-1 (owner: enricher-3)
[15:38:08] [enricher-1] Claimed batch: batch-2
[15:38:08] [enricher-3] Claimed batch: batch-1

Batch claims go through write-wait-reread. Losers move to the next batch.

agents.py — claim-or-move-on

pending = await get_pending_tasks(client, batches_shard)
batch_task = pending[0]

result = await claim_task(client, batches_shard, batch_task.id, state["agent_id"])

if not result.claimed:
    return {**state, "current_batch": None, "lost_races": state["lost_races"] + 1}

return {**state, "current_batch": batch_task, "batch_claimed": True}

Metric	Naive	ZeroLock
Customers enriched	9	10
Completion rate	90%	100%
Records lost	2	0
Duplicate batches	1	0
LLM calls	12	11

Why this matters — Duplicate processing means paying for the same LLM call twice plus an inconsistency question.

Aggregate analysis

Data loss by scenario

Scenario	At risk	Naive
Code refactoring	8 functions	5
Code review	15 findings	2
Financial orders	3 orders	0*
Data enrichment	10 records	2

*Financial orders scenario measures approval-gate enforcement, not data loss.

Root causes addressed

Failure mode	Occurrences (naive)	ZeroLock mechanism
Write-write conflicts	11 overwrites	OCC + file locking
Task-claiming races	5 duplicates	Write-wait-reread
List append races	2 lost items	CRDT arrays
Missing approval gates	3 orders	Layer 4 policy

Total LLM calls across all 4 scenarios: 37 (naive) vs. 33 (ZeroLock) — an 11% reduction, because ZeroLock eliminates the wasted calls spent on duplicate work.

Limitations of this benchmark

What this proves

— ZeroLock eliminates coordination failures at small scale (5 agents)
— All four coordination mechanisms work correctly
— Real LLM integration functions properly, not just mocked calls

What this doesn't prove

— Performance at scale (50–100+ agents)
— Latency characteristics under load
— Comparison with alternative solutions (Redis, Postgres locks)
— Statistical significance — this is a single run per scenario

Recommended follow-up (not yet done): a scale sweep at 10/20/50/100 agents, 30 runs per configuration for statistical significance, p50/p95/p99 latency profiling, and a baseline comparison against Redis distributed locks. The separate adversarial harness (9 scenario families, verified to 80 concurrent agents) covers scale and adversarial conditions this suite doesn't — see the benchmarking docs.

How to reproduce

terminal

# start the gateway
cd packages/gateway && pnpm dev

# install demo deps
cd examples/prod-demo-suite
pip install -r requirements.txt

# deterministic, no API key
python run_demo.py --llm fake

# real Gemini
GOOGLE_API_KEY="your-key" python run_demo.py --llm real

# single scenario
python run_demo.py --llm fake --scenario code_refactoring

Run this demo yourself or explore the full benchmarking methodology:

View source on GitHub Full benchmarking docs