Skip to main content
YCSB (Yahoo! Cloud Serving Benchmark, Apache-2.0) is the first suite on the key-value domain. It also establishes the SUT-connection abstraction: a suite runs against a local reference OR an already-running service selected purely by config — the same seam a cloud-managed backend attaches to later. The suite wraps the recognized upstream YCSB tool (load phase + run phase), like the SWE-bench suite wraps the Docker harness — it does not reimplement the load generator. The real path needs the YCSB launcher (Java ≥ 11) on PATH or $YCSB_HOME; the offline mock path needs nothing.

​
The connection abstraction

The system under test is YCSB’s own binding + endpoint, resolved from the run Target by the key-value adapters:
params.workload is a YCSB core workload (workloada..workloadf, default workloada); recordcount / operationcount default to 10000.

​
What it measures

The official-ycsb-evaluator parses YCSB’s run-phase output into ycsb.* measurements, all reproducibility_class="environmental" (YCSB is a throughput/latency benchmark — there is no answer-correctness dimension): A metric absent from the output (e.g. no UPDATE line for a read-only workload) is omitted. All carry official=True — a provenance flag (emitted by the suite’s recognized evaluator), not an audit claim. The last three are reliability evidence and remain tool-evidence-only: the error counts exist only when the tool printed Return= lines, and the disruption claim only when the harness recorded that it fired — nothing is invented.

​
Reliability (disruption) runs

params.reliability: {action: reset|stall, at_s, stall_ms?} arms a driver-side disruption of the measured run phase (the load phase stays clean). The plan is validated at resolve() and folded into the dataset digest — a different disruption is a different benchmark. The measured phase routes through a driver-side DisruptionProxy TCP relay (core/disruption.py) that fires the action at at_s; it requires the redis binding plus a target.endpoint and fails loudly otherwise — never a clean benchmark under a disruption-labeled dataset. A disruption span lands in the trajectory. Full runbook: Evaluate a managed KV service §5.

​
Deliberate scope (slice 1)

  • Real numbers need the real YCSB tool + a real backend; offline is the mock path. ycsb-local (binding=basic) exercises YCSB’s own machinery without a datastore.
  • ycsb-endpoint connects to an already-running service (Redis-protocol first) — full cloud runbook: Evaluate a managed KV service.
  • Cloud-provisioned / managed KV backends attach later on the same seam.