Agent
Agent Workflow: Question to Answer
Overview
The end-to-end loop an agent follows to turn a benchmarking question into an
evidence-backed answer. Two files are the actual input/output contract —
contracts/contract_catalog.yml (what
can be asked for) and
contracts/contract_result.yml (what
comes back, and how to answer) — self-contained enough that nothing else in
the repository is required to know the shape of a valid experiment.yml or
to interpret a finished run, by design (see each contract’s own header
comment). One more file, environment.yml,
is genuinely a third input — see step 4 — but it’s a live cluster snapshot,
not a contract, and is only needed for the placement/resource-fit half of
validation.
1. Question in
2. Read contracts/contract_catalog.yml + contracts/contract_result.yml
3. Build experiment.yml
4. python validate_experiment.py experiment.yml ← dry run, no cluster touched
5. python experiment.py experiment.yml ← actual run
6. Read {resultfolder}/{code}/report/index.md, answer per contract_result.yml's answer_contract
Step 2 — read the two contracts (and know about the third file)
contracts/contract_catalog.yml is not an abstract JSON-Schema-style
description that then needs a separately instantiated catalog.yaml to
become usable — it is the concrete catalog data. Its systems and
workloads keys already list what’s actually offered:
>>> import yaml
>>> d = yaml.safe_load(open('contracts/contract_catalog.yml'))
>>> list(d['systems'])
['PostgreSQL', 'PgDuckDB']
>>> list(d['workloads'])
['tpch']
The name catalog.yaml shows up elsewhere in the codebase
(experiment.py’s default sibling-file lookup, Design-Yaml-Experiment-Entry-Script.md’s
catalog:/environment: provenance-pointer fields) purely as a generic name
for wherever a working copy of this same file happens to live — nobody in
this repo has ever created a second, differently-populated one. An agent that
has read contracts/contract_catalog.yml already knows everything the
catalog knows; there is no hidden third contract file to go find.
contracts/contract_result.yml is the matching output-side contract: tiers,
provenance globs, validity checks, verdict_shape, and (since v1.4.0)
answer_contract — the shape the final answer in step 6 should take.
environment.yml is a genuinely different, third file — not a contract.
Unlike catalog.yaml, it isn’t just an alias for a file already read in step
2. contracts/contract_catalog.yml answers “what can be asked for” as a
static, hand-curated, repo-committed fact; environment.yml answers “what
does this specific cluster actually have, right now” — live node
allocatable/free capacity, storage classes, namespace resource limits — and
can only be produced by connecting to a real cluster:
python -m bexhoma.environment [-cx context] [-o dev/catalog/environment.yml] [-xhw]
# or, in-process: bexhoma environment create
Generated by bexhoma/environment.py
(build_environment()), which is why it has its own embedded
environment_contract_version (currently 1.0.0) instead of a
contracts/contract_environment.yml doc — the file documents its own shape
at generation time, per that module’s own header comment, so there is
nothing separate to keep in sync. It carries a cluster.collected_at
timestamp and goes stale the moment cluster capacity changes after that —
the checked-in dev/catalog/environment.yml
example is a snapshot from a specific point in time, not a live view;
regenerate it before trusting it for a real run. spec.validate_environment()
(the check step 4 runs against it) is deliberately independent of the
catalog check — an agent that only has one of contract_catalog.yml /
environment.yml can still validate what it has.
Step 3 — build experiment.yml
Conforms to contract_catalog.yml’s experiment_schema. Three fields are
required — title, hypothesis, discriminates — plus workload/system
selection; a fourth, follow_up_of, is optional and names a prior run’s
experiment_code when this run continues it. See
AgentCatalogContract.md for the condensed,
agent-facing shape of everything contract_catalog.yml currently offers
(the one workload and two systems in scope, their params/knobs/profiles) and
dev/catalog/experiment.yml for the
maintained worked example. experiment*.yml is gitignored at the repo
root — a scratch ./experiment.yml there is a normal, disposable working
copy, not a file that needs to be committed.
Step 4 — validate before running
validate_experiment.py is a dry run: it
resolves experiment.yml against contract_catalog.yml (same resolution
path experiment.py/bexhoma/spec.py::translate() use to build the real CLI
argv) and, by default, also checks it against
dev/catalog/environment.yml — placement
node existence/taints, CPU/memory ceilings, storage-class existence. No
subprocess, no live cluster connection.
python validate_experiment.py experiment.yml
# or, spelling out the defaults explicitly:
python validate_experiment.py experiment.yml -c contracts/contract_catalog.yml -e dev/catalog/environment.yml
# pass -e "" to skip the environment/placement check
A failing check prints FAIL with the specific reason (unsupported
workload/system pairing, illegal parameter value, unmet profile precondition,
placement/resource ceiling exceeded, …) and exits non-zero — fix
experiment.yml and re-run step 4 before touching step 5.
Step 5 — run it
python experiment.py experiment.yml
# or: python experiment.py experiment.yml -c path/to/catalog.yaml (only if not using the repo's own contract_catalog.yml)
Dispatches internally on experiment.yml’s workload: shape (catalog-driven
vs. self-specified — see Design-Yaml-Experiment-Entry-Script.md) and, for a
catalog-driven file, through the same build_argv() resolution
validate_experiment.py already exercised in step 4 — so a run that passed
validation resolves identically here, it just also actually submits to the
cluster this time. Always writes the tiered Markdown report (-rp is forced
on for a YAML-driven run). At run start, experiment.yml itself plus the
contract_catalog.yml/contract_result.yml pair that governed it are copied
verbatim into the result folder — see contract_result.yml’s
provenance.workflow and answer_contract.hypothesis.
Current limit: bexhoma/spec.py::build_argv() has argv builders for the
tpch and ycsb workloads today — a catalog-driven experiment.yml naming
hammerdb, benchbase, or tpcds fails resolution at step 4, before step 5
is ever reached.
Step 6 — read the result, answer per contract
Entry point: {resultfolder}/{code}/report/index.md (only if -rp was used
— always true for a YAML-driven run). Follow contract_result.yml’s
answer_contract.steps in order:
hypothesis — restate the question, quoting
experiment.yml’shypothesisfield verbatim from{resultfolder}/{code}/experiment.yml. If that file isn’t there (a hand-typedpython tpch.py run ...invocation never copies one in), say plainly that no hypothesis was recorded — don’t invent one from workload params.verdict — pass/fail/skip counts from
report/index.md’s frontmatteroverall_status; a FAILED row scopes/invalidates metrics reported below it, a SKIPPED row never does.evidence — cite the specific tier-1/tier-2 file and value behind every claim (
report/workflow.md,loading.md,benchmarking.md,monitoring.md,connections.md, each written only if that phase was active).follow_up — if the verdict doesn’t fully resolve the hypothesis, or
discriminatesnames a factor not yet varied, propose a newexperiment.ymlwithfollow_up_ofset to this run’sexperiment_code.
See also
contracts/contract_catalog.yml/contracts/contract_catalog_comments.md— the input-side contract and its rationale.contracts/contract_result.yml/contracts/contract_result_comments.md— the output-side contract and its rationale.AgentCatalogContract.md— condensed, agent-facing walkthrough ofcontract_catalog.yml: what a validexperiment.ymlmay contain.AgentResultContract.md— prose walkthrough ofcontract_result.yml, with worked examples.AgentReport.md— design rationale for the tiered report read in step 6.validate_experiment.py/experiment.py— the two entry points used in steps 4–5.bexhoma/environment.py— generatesenvironment.yml(python -m bexhoma.environment/bexhoma environment create); the live-cluster snapshot used in step 4’s placement/resource check.
Catalog Input Contract
A pre-build reference for an agent (or human) that needs to know, before
writing an experiment.yml, exactly what it may legally contain — which
workloads and systems exist, which parameters/knobs each takes, and which
three header fields are required — without reading any source code.
Mirrors AgentResultContract.md’s role on the
output side: that file tells an agent what it’ll get back, this one tells it
what it’s allowed to ask for. Everything below is read directly from
contracts/contract_catalog.yml itself — no other file is needed to know
the current shape of a valid experiment.yml.
catalog_contract_version: "1.1.0" # == bexhoma.spec.CATALOG_CONTRACT_VERSION
catalog_concepts: # vocabulary used throughout this file's own fields
workloads: {semantics: "what to run: params, loading behavior, physical-design semantics"}
systems: {semantics: "what to run it on: server knobs, physical-design support, profiles"}
physical_design: {semantics: "a system's CAPABILITY only (indexes/constraints/statistics/storage_format) --
whether it's actually applied is a separate per-experiment SELECTION,
via loading.post_load / systems[].post_load"}
derive: {semantics: "arithmetic expression scaling a profile knob with the experiment's own
resource limits -- only +,-,*,/ over memory_limit/cpu_limit/storage_class/scaling_factor,
no functions/conditionals/string ops"}
extends: {semantics: "systems.<name>.extends: BaseName merges BaseName's knobs:/profiles: in
(this system's own entries of the same name win); every OTHER top-level
key (physical_design, deployment, image, arg_style, ...) is NOT merged --
e.g. PgDuckDB.extends: PostgreSQL still needs its own physical_design: block"}
profile_ref: {semantics: "systems.<name>.profiles.<p>.ref: 'System.profiles.name' resolves to that
OTHER system's profile object directly, for profile parity"}
requires: {semantics: "a profile's requires: {storage_class: [...]} lists which resources.storage_class
values an experiment.yml using it may set; null means 'unset' is also legal;
a profile without requires: imposes no constraint"}
arg_style: {semantics: "how a resolved knob is applied -- pg-guc (default): a --set ...GUC patch;
env-var: via the knob's own env_var name instead; a knob may override its
system's default"}
knob_status: {semantics: "a knob with status: reference-only exists in the DBMS but is commented out
in the shipped k8s template -- still legal to set via profile/override,
just not active by default. fixed: true (separate) marks a knob that isn't
mechanically blocked but has no other legal value in practice for this pairing"}
sut_isolation: {semantics: "the default situation is one system-under-test at a time, not several at once:
every systems[] entry (crossed with any resources: sweep) is benchmarked on
its own, next SUT started only after the previous is torn down. Co-located
SUTs interfere (shared node CPU/memory-bandwidth/disk/network), so a
side-by-side run would measure interference, not the discriminates: factor.
Enforced by two independent caps, both default 1: top-level max_sut (-ms,
cluster-wide) and max_sut_experiment (-mse, this experiment only). Set
either to 0 (no limit) or N>1 for parallel SUTs -- only for SUTs on
separate nodes. Omitting them keeps the serial default.
Parallel loader pods / benchmarker clients run within one SUT and are exempt"}
experiment_schema:
required_header_fields: [title, hypothesis, discriminates] # validate_experiment() rejects a missing/empty one before anything else resolves
optional_header_fields:
follow_up_of: "experiment_code (contract_result.yml's structure.experiment_code) of a prior run this
one follows up on -- structured lineage instead of prose in hypothesis; bookkeeping
only today, see known_gaps below"
top_level_shape: # every field below is a SIBLING at the top of experiment.yml
mode: {type: enum, values: [run, profiling, start, load, empty, summary], default: run}
title: {type: str, required: true}
hypothesis: {type: str, required: true}
discriminates: {type: "list[str]", required: true, example: "[system, concurrency, memory]"}
follow_up_of: {type: str, required: false}
max_sut: {type: int, default: 1, semantics: "max SUTs running at once CLUSTER-WIDE (-ms);
1 = one system at a time, 0 = no limit, N>1 = up to N -- see catalog_concepts.sut_isolation"}
max_sut_experiment: {type: int, default: 1, semantics: "same, scoped to this experiment only (-mse);
independent of max_sut, both enforced together; 0 = no limit"}
workload: {type: object, fields: [name, params, rounds, repetitions]}
loading: {type: object, fields: [pods, threads, split, post_load],
pitfall: "must be a TOP-LEVEL sibling of workload:, NOT nested under it -- a
workload.loading block silently resolves to {} instead of erroring"}
systems: {type: list, item_fields: [name, profile, override, post_load],
semantics: "one resolved configuration per entry; benchmarked one at a time,
never concurrently -- see catalog_concepts.sut_isolation"}
observe: {type: object, fields: [monitoring_sut, monitoring_cluster, monitoring_app]}
placement: {type: object, fields: [sut, loading, benchmarking],
semantics: "each node named must exist, and not be tainted out, in environment.yml's nodes:"}
resources: {type: object, fields: [cpu, memory, storage, storage_class],
semantics: "cpu/memory: a single {request,limit} dict shared by every system, OR a list
to sweep every systems: entry against every list entry (one resolved
config per system*cell pair); cpu and memory sweep lists must share one length"}
quantity_format:
memory_and_storage: {binary: [Ki, Mi, Gi, Ti], decimal: [K, M, G, T], out_of_scope: [KB, MB, GB, TB], examples: ["32Gi", "512Mi"]}
cpu: {semantics: "cores, or millicores with trailing m", examples: ["8", "0.5", "500m"]}
workloads:
tpch:
supports: [PostgreSQL, PgDuckDB]
modes: [profiling, run, start, load, empty, summary]
resource_profile: {cpu: high, memory: high, why: "multi-way hash joins + aggregation are CPU/RAM-bound; storage bandwidth matters less"}
params: # workload.params keys
scaling_factor: {type: int, unit: GB}
timeout: {type: int, unit: seconds}
query_repeats: {type: int, default: 1, min: 1}
measure_datatransfer: {type: bool, default: false}
active_queries: {type: "list[int]", default: all, example: "[5,7,8,9,21] = multi-way joins"}
recreate_parameter: {type: bool, default: false}
shuffle_queries: {type: bool, default: false}
refresh_streams: {type: int, default: 0}
refresh_stream_offset: {type: int, default: 0}
store_explain: {type: bool, default: false, when: "requires an 'explain' key in the DBMS connection's JDBC config"}
loading:
pods: {type: int, min: 1, support: "works for every DBMS"}
threads: {type: int, min: 1, support: "only honored by some loaders (e.g. MySQL); prefer pods"}
split: {type: int, default: 1}
post_load: # indexes/constraints/statistics are mutually independent -- all 8 combinations legal per system
indexes: {type: bool, default: false}
constraints: {type: bool, default: false}
statistics: {type: bool, default: false}
storage_format: {type: enum, values: [heap], default: heap}
rounds: {type: "list[int]", rule_of_thumb: "official sizing: floor(log(scaling_factor, 3)) + 2, e.g. SF=100 -> 6"}
repetitions: {type: int, default: 1}
produces:
per_query: {metric: latency, unit: ms}
summary: {metrics: [Power@Size, Throughput@Size, Geo Times], unit: [Q/h, Q/h, s]}
quality: {metric: sql_errors_warnings}
out_of_scope: {time_series: "no per-second signal like YCSB/Benchbase -- per-query/per-phase aggregates only"}
systems:
PostgreSQL:
image: postgres:18.3
arg_style: pg-guc
physical_design: {indexes: true, constraints: true, statistics: true, storage_format: [heap]}
knobs_active_by_default: [max_connections, max_worker_processes, max_parallel_workers,
max_parallel_workers_per_gather, max_parallel_maintenance_workers, shared_buffers,
effective_cache_size, work_mem, maintenance_work_mem, autovacuum, wal_level,
max_wal_senders, max_wal_size, checkpoint_timeout, checkpoint_completion_target,
lock_timeout, idle_in_transaction_session_timeout]
knobs_reference_only: [effective_io_concurrency, io_method, random_page_cost, seq_page_cost,
default_statistics_target, fsync, synchronous_commit, wal_compression]
profiles:
analytical-ssd:
requires: {storage_class: [ssd, null]}
why: "OLAP on node-local NVMe, sized from the experiment's memory limit"
knobs: {random_page_cost: 1.1, effective_io_concurrency: 200, io_method: io_uring,
max_parallel_workers_per_gather: 2, max_parallel_workers: 4, max_worker_processes: 6}
derive: {shared_buffers: "0.3125 * memory_limit", effective_cache_size: "0.75 * memory_limit",
work_mem: "0.015625 * memory_limit", maintenance_work_mem: "0.03125 * memory_limit"}
PgDuckDB:
why: "PostgreSQL + pg_duckdb extension, vectorized DuckDB execution engine alongside Postgres' own planner"
extends: PostgreSQL # every knob not listed below inherits unchanged
image: pgduckdb/pgduckdb:18-v1.1.1
arg_style: pg-guc
physical_design: {indexes: true, constraints: true, statistics: true, storage_format: [heap],
out_of_scope: "columnar (native `USING duckdb` tables) blocked upstream, github.com/duckdb/pg_duckdb#385"}
knobs_own:
shared_preload_libraries: {default: pg_duckdb, fixed: true}
duckdb_force_execution: {type: bool, default: false, arg_style: env-var, env_var: DUCKDB_FORCE_EXECUTION}
profiles:
analytical-ssd: {ref: "PostgreSQL.profiles.analytical-ssd"} # identical knob values from the same memory/cpu limits
catalog.yaml is not a separate file you need to build
contracts/contract_catalog.yml above is not an abstract schema requiring a
separately instantiated catalog.yaml — it already is the concrete
catalog data (systems: [PostgreSQL, PgDuckDB], workloads: [tpch, ycsb]).
validate_experiment.py’s own -c default points straight at it. The name
catalog.yaml elsewhere in the codebase (experiment.py’s sibling-file
lookup, Design-Yaml-Experiment-Entry-Script.md’s catalog: provenance
pointer) just names wherever a working copy of this same file happens to
live — nobody in this repo has ever populated a different one. See
AgentWorkflow.md for the full build → validate → run →
answer loop this contract is step 2–3 of, including why environment.yml (a
live cluster snapshot, not a contract) is a genuinely separate third input.
Minimal example
python validate_experiment.py experiment.yml
# OK resolves against catalog
# command: python tpch.py run -dbms PostgreSQL PgDuckDB -sf 10 -t 300 -xqr 3 ...
See dev/catalog/experiment.yml for the
maintained, real, runnable experiment.yml this resolves — a two-system
(PostgreSQL vs. PgDuckDB) analytical-ssd-profile sweep across
concurrency (1→16) and memory (64Gi→32Gi), with discriminates: [system, concurrency, memory]. Both systems (× every swept cell) resolve into one
command, but bexhoma benchmarks them one SUT at a time — max_sut and
max_sut_experiment both default to 1 — so the two never contend for the
same node. Set either to 0 (no limit) or N in the experiment.yml to
allow parallel SUTs; see catalog_concepts.sut_isolation.
Known gaps versus an idealized contract
Two workloads are translatable today:
tpchandycsb.bexhoma/spec.py::build_argv()dispatches byexperiment.workload.nameto that workload’s own argv builder —bexhoma/experiments/tpch_catalog.py::build_tpch_argv()andbexhoma/experiments/ycsb_catalog.py::build_ycsb_argv(). A catalog-drivenexperiment.ymlnaminghammerdb,benchbase, ortpcdsfails resolution withSpecError: no argv builder implemented yet for workload '<name>'— those workloads still run fine via their own entry scripts directly, just not through this catalog contract yet.System scope is trimmed per workload.
tpchsupportsPostgreSQL/PgDuckDB(the full workload outside this contract also supportsMonetDB,MySQL,MariaDB,DatabaseService,Citus,CedarDB).ycsbsupportsPostgreSQLonly (the full workload also supportsMySQL,MariaDB,YugabyteDB,CockroachDB,TiDB,DatabaseService,PGBouncer,Redis,Citus,CedarDB,Dragonfly). See each workload’sout_of_scope.systems.ycsbhas no resource sweep and no post_load.resources.cpu/resources.memorymust each be a single{request, limit}dict (a list is rejected), and YCSB manages its own schema so there is no indexes/constraints/statistics selection.No comparative/historical validity check reads
follow_up_of. It records intended lineage structurally, but nothing enforces or verifies that the namedexperiment_codeexists or ran a comparable workload — seecontract_result.yml’sknown_gaps.cross_experiment_comparison.extends/profile_refdon’t imply capability parity. A system thatextends:another still needs its own explicitphysical_design:block (that key is never merged) — seecatalog_concepts.extends.out_of_scope.
See also
AgentWorkflow.md— the end-to-end loop this contract is the input half of: question → contracts →experiment.yml→ validate → run → answer.AgentResultContract.md— the output-side counterpart: what a completed run’s result folder contains.contracts/contract_catalog.yml/contracts/contract_catalog_comments.md— the actual contract and its human-only rationale doc.validate_experiment.py— dry-run validates anexperiment.ymlagainst this contract without touching a cluster.
Result Folder Output Contract
A pre-run reference for an agent (or human) that needs to know, before submitting an experiment, exactly what will exist in the result folder afterwards, what it will be named, and which files are safe to treat as ground truth versus which are only a rendered convenience.
This is the contract version of AgentReport.md (which
explains the tiered-report design) and of the naming/validity/interpretation
blocks embedded verbatim in every report/index.md
(bexhoma/report_writer.py) — read this file to plan; read the generated
report/index.md to interpret an actual run.
result_contract_version: "1.4.0" # == bexhoma.report_writer.SCHEMA_VERSION;
# bump tracks report_writer.py's own frontmatter/tier/layout changes
entry_point:
with_report: report/index.md # only exists if the run passed -rp/--report
without_report: connections.config, queries.config # always exist once benchmarking started; read these directly
structure:
result_dir: "{resultfolder}/{code}/"
experiment_code: unix-timestamp # `code`: seconds, generated at experiment start;
# unique + monotonically increasing, but NOT evidence
# two codes ran under comparable conditions
configuration: "<system>-<n>" # e.g. postgresql-1 (lowercased when
# embedded in phase/job/connection below;
# original case, e.g. "PostgreSQL-1", when
# shown standalone, e.g. connections.config)
phase: "<configuration>-<experiment_run>-<client>" # drops benchmark_run, pod
job: "<configuration>-<experiment_run>-<client>-<benchmark_run>" # drops pod
connection: "<configuration>-<experiment_run>-<client>-<benchmark_run>-<pod>"
# decode any identifier by counting dash-separated segments from the right
tiers: # only "with_report" tiers 1-2 are new files; tier 3 is always the raw folder
1_answers: {glob: "report/index.md"}
2_evidence: {glob: "report/{workflow,loading,benchmarking,monitoring,connections}.md"}
# each written only if that phase was active
3_diagnosis: {result_dir: "*"} # see provenance: below; linked from every tier-2 "### Provenance" footer
provenance: # pre-existing files, never written or modified by the report
connections: {"connections.config": "repr() list of every connection dict (identity, params, timings)",
"{connection}.config": "durable single-connection backup, survives dashboard rewrites",
"queries.config": "SF/type/duration/defaultParameters/benchmark_sequence/workflow_planned"}
workflow: {"*.yml / *.yaml (minus the input-provenance filenames below)":
"rendered K8s Job/Deployment/Service manifests actually submitted;
every image: field is a concrete tag, BEXHOMA_PACKAGE_VERSION
already substituted — the authoritative source for image versions",
"experiment.yml / experiment.yaml, contract_catalog.yml, contract_result.yml,
catalog.yaml, environment.yml (whichever exist)":
"the input(s) this run was actually built from, not a K8s manifest —
a YAML-driven run (experiment.py) copies the experiment.yml/.yaml it
was given, plus (catalog-driven only) the contract_catalog.yml/
contract_result.yml pair that governed it, or (self-specified only)
any catalog:/environment: pointer files the spec named; a hand-typed
python tpch.py ... invocation writes none of these",
"{pod-name}.describe.log": "kubectl describe pod: scheduling/image-pull/restart/OOMKill events
for that specific Pod object",
"{job-name}.describe.job.log": "kubectl describe job (loading/generator jobs only; .job.log,
not .describe.log, so it globs apart from per-pod describes):
the Job's own Events list every Pod it ever spawned over
its full lifetime, including a failed one replaced under
backoffLimit — evidence that survives even after the failed Pod
object itself has been garbage-collected and dropped out of the
per-pod *.describe.log set above"}
loading: {"*-loading-*.sql.log / *-loading-*.sh.log": "rendered script SOURCE despite the .log suffix",
"*-loading-*.stdout.log": "stdout of that script",
"*-loading-*.stderr.log": "stderr — check first on a silent loading failure"}
benchmarking: {"bexhoma-benchmarker-*.log": "raw per-pod benchmarker/driver stdout",
"bexhoma-benchmarker.*.all.df.pickle": "cached parsed+aggregated DataFrame",
"queries.config": "literal SQL text — DBMSBenchmarker-family (TPC-H/TPC-DS) only"}
monitoring: {"query_{component}_metric_{key}.csv": "wide format: one column per connection, one row per Prometheus scrape;
{component} (e.g. loading/benchmarking/loader/benchmarker/datagenerator)
is a fixed vocabulary owned by the vendored dbmsbenchmarker dependency,
not bexhoma's to rename freely — see monitoring.md's component_title
column for the human-readable pairing"}
restarts: {"bexhoma-sut-{configuration}-{experiment_run}-restarts.json": "per-pod SUT container restart counts, one snapshot per experiment_run; aggregate by max per pod (restartCount is cumulative across runs, same pod, not recreated), not by summing every file"}
sut_logs: {"bexhoma-sut-{configuration}-{code}-{experiment_run}.yml": "SUT Deployment manifest — one archived copy per experiment_run, even when identical to the previous run's, since the live Deployment itself is restarted in place rather than recreated",
"bexhoma-sut-{configuration}-{code}-{experiment_run}-{pod-hash}-{pod-suffix}.{container}.log": "SUT container stdout, one capture per experiment_run",
"bexhoma-sut-{configuration}-{code}-{experiment_run}-{pod-hash}-{pod-suffix}.describe.log": "kubectl describe pod, one capture per experiment_run"}
# see "Result-folder filenames vs. report identifiers" below for the one remaining
# asymmetry: the live k8s object's own name has no experiment_run segment, even
# though every filename on disk (including its own archived manifest) does
versions: # see Known gaps below for what's genuinely still missing
images: recorded_as_tag_not_digest # every submitted manifest's image: field is a concrete tag
# (provenance.workflow *.yml); connections.config's own
# `dockerimage` field mirrors the SUT's resolved tag too;
# no sha256 digest either way, so a re-pushed tag is invisible
bexhoma: recorded_directly_and_via_image_tag
# report/index.md frontmatter's bexhoma_version field records
# the installed bexhoma.__version__ at report-generation time
# (can differ from the version that actually ran the experiment
# if -rp/--report is applied later, e.g. `bexhoma summary -e
# <code> -rp`, after an upgrade); for the submission-time version,
# BEXHOMA_PACKAGE_VERSION in every bexhoma/* image tag is
# substituted with the real installed version before the
# manifest is written to the result folder
# (clusters.py::create_object_from_file()) and can't drift later
dbmsbenchmarker: not_recorded # baked into the bexhoma/benchmarker_dbmsbenchmarker image,
# whose tag tracks bexhoma's own version, not
# dbmsbenchmarker's — genuinely not recoverable
validity: # from experiment._test_results -> report/index.md "### Tests" table
- id: workflow_as_planned # planned (queries.config's workflow_planned) == actual submitted jobs/pods
kind: absolute
- id: no_sut_container_restarts # bexhoma-sut-*-restarts.json sums to 0
kind: absolute
- id: key_metric_present # benchmark-type headline column(s) contain no 0/NaN — see table below
kind: absolute
- id: no_sql_errors # DBMSBenchmarker-family (TPC-H/TPC-DS) only
kind: absolute
- id: no_sql_warnings # = no result-set mismatch across systems; DBMSBenchmarker-family only
kind: absolute
- id: monitoring_component_cpu_nonzero # per monitored component; SKIPPED (not failed) when phase < 1 scrape interval
kind: absolute
- id: cross_experiment_comparison # NOT IMPLEMENTED — see Known gaps
kind: comparative
verdict: {passed: int, failed: int, skipped: int} # index.md frontmatter overall_status;
# only a FAILED row scopes/invalidates metrics below it — skipped never does
answer_contract: # how to structure the final written answer, once the above has been read
hypothesis:
source_file: experiment.yml # <result_dir>/experiment.yml; fields: title, hypothesis, discriminates, follow_up_of
present_when: catalog-driven run # `python experiment.py run <file>.yml` copies it in at run start
absent_when: direct entry-script run # e.g. `python tpch.py run -dbms ...` never had a catalog file to copy —
# state "no hypothesis recorded", don't reconstruct one from workload params
also_copied: [contract_catalog.yml, contract_result.yml] # frozen at run time, may differ from the repo's current copies
steps:
- {id: hypothesis, instruction: "restate the question, quoting experiment.yml's hypothesis verbatim when present"}
- {id: verdict, instruction: "pass/fail/skip counts; note any FAILED row scoping a metric below it", source: verdict_shape}
- {id: evidence, instruction: "cite the specific tier-1/tier-2 file and value behind every claim", source: tiers}
- {id: follow_up, instruction: "if unresolved, propose a new experiment.yml with follow_up_of set to this run's experiment_code", source: "experiment.yml discriminates/follow_up_of, else known_gaps.cross_experiment_comparison"}
Naming legend, as a decoding rule
Every identifier is a positional dash-concatenation — decode by counting segments from the right, no lookup table needed:
Term |
Meaning |
Example |
|---|---|---|
|
SUT instance name (original case when shown standalone, e.g. |
|
|
Repeat counter for the whole experiment ( |
|
|
1-based index of the benchmark phase/round within a run ( |
|
|
|
|
|
1-based index of a parallel benchmark job within a round (query stream vs. refresh stream, etc.) |
|
|
|
|
|
1-based index of a driver pod within a job |
|
|
|
|
code (the result folder’s own directory name) is a Unix epoch timestamp
in seconds, assigned once at experiment start — unique and monotonically
increasing, but comparing two different codes as “the same conditions”
requires independently verifying that (see Interpretation Rules in every
index.md).
Result-folder filenames vs. report identifiers
The table above decodes identifiers inside the report (table indexes,
connections.md anchors). Filenames actually on disk — manifests, logs,
.describe.log — follow a related but distinct convention:
<app>-<component>-<configuration>-<code>[-<experiment_run>[-<client>[-<benchmark_run>]]],
optionally followed by Kubernetes’ own pod-hash/random suffix on files tied
to one specific pod, e.g.
bexhoma-benchmarker-postgresql-1-1784910886-1-1-1-qp9nt.dbmsbenchmarker.log.
Decode these the same way — count from code, not from the right — since a
trailing Kubernetes pod suffix (a hash plus 5 random characters) isn’t part
of the schema and can’t be told apart from it by position alone.
The SUT Deployment’s own k8s identity is the one asymmetric case — but its
filenames are not. The live Deployment object is restarted in place across
every -nc repeat rather than recreated, so its metadata.name (and the
service/pod names derived from it) stay identical run after run, with no
experiment_run segment. Every filename related to it, however, is
experiment_run-scoped like everything else: its manifest is archived as
bexhoma-sut-{configuration}-{code}-{experiment_run}.yml — a fresh copy
written every run (even when byte-identical to the previous run’s, since the
Deployment spec didn’t change), not just its .log/.describe.log captures
(bexhoma-sut-postgresql-1-1784910886-3-7bd45c7b95-pwzkz.dbms.log). So an
agent decoding filenames never needs a special case for the SUT — only code
that resolves a filename back to which live k8s object it came from needs
to know that several manifest files can point at the same, still-running
Deployment.
Whether report/ exists at all
report/*.md is not written by default. It only exists when the run
passed -rp/--report (any entry script, or bexhoma summary -e <code> -rp
run later against the same result folder — no live cluster connection
needed either way). An agent that has not confirmed -rp was used must plan
to read the raw files listed under provenance: above directly — start from
connections.config (what ran) and queries.config (workload identity +
workflow_planned), rather than assuming report/index.md exists.
Key metric per benchmark type
The key_metric_present validity check and report/index.md’s Key Metrics
block both test the same column(s), one set per benchmark type — this is the
column an agent should treat as “the” headline number:
Benchmark type |
Entry script |
Key metric column(s) |
|---|---|---|
DBMSBenchmarker (TPC-H/TPC-DS) |
|
|
YCSB |
|
|
HammerDB TPC-C |
|
|
Benchbase |
|
|
Hardware (fio/sysbench/sockperf/netperf) |
|
IOPS / CPU events-per-sec / message rate / transaction rate, per active probe |
Known gaps versus an idealized contract
Image tags are recorded; digests are not, and
dbmsbenchmarker’s own version isn’t either. Every manifest actually submitted to the cluster — SUT deployment, loader/generator/benchmarker jobs, monitoring sidecars — is written into the result folder byclusters.py::create_object_from_file()after itsBEXHOMA_PACKAGE_VERSIONplaceholder is substituted with the real installed bexhoma version, so everyimage:field inprovenance.workflow’s*.ymlfiles is a concrete tag (e.g.postgres:18.3,bexhoma/benchmarker_dbmsbenchmarker:0.9.8,gcr.io/cadvisor/cadvisor:v0.47.0) — this is the source an agent should read for versions, notconnections.config’sdockerimagefield alone (which does carry the SUT’s own resolved tag once the SUT has started, viaconfigurations/benchmarking.py:127, but is a narrower single-image view). What’s still genuinely missing: a sha256 digest (a tag can be re-pushed to point at different bytes) and thedbmsbenchmarkerpackage version specifically — it’s baked inside thebexhoma/benchmarker_dbmsbenchmarkerimage, whose own tag tracks bexhoma’s version, not dbmsbenchmarker’s.No comparative/historical validity check. Every validity test is absolute (pass/fail/skip against this run’s own data); there is no archived-corridor or cross-run regression check. “Compare only within this experiment code” (see every
index.md’s Interpretation Rules) is a rule an agent must apply itself — nothing in the result folder does it automatically.contract_catalog.yml’sexperiment_schema.fields.follow_up_oflets an experiment.yml record which prior experiment_code it follows up on, but that’s bookkeeping only — nothing reads or validates it yet.Per-system post_load selection isn’t a queryable field. A catalog-driven experiment can choose, per named system, whether indexes/constraints/ statistics were applied after loading (
contract_catalog.yml’ssystems[].post_load— seeAgentCatalogContract.md’s catalog concepts).connections.config/queries.configrecord which SUT ran, not which post-load steps it received — an agent has to fall back to tier-3’s*-loading-*.sql.log(the rendered DDL source, perprovenance.loadingabove) and check forCREATE INDEX/constraint/ANALYZEstatements itself.
See also
AgentWorkflow.md— the end-to-end loop this contract is one half of: question → contracts →experiment.yml→ validate → run → answer.AgentCatalogContract.md— the input-side counterpart: what a validexperiment.ymlmay contain.AgentReport.md— design rationale for the tiered report,index.md’s eleven sections, the Full Metric Catalog.bexhoma/report_writer.pymodule docstring — the same output contract, embedded next to the code that implements it.bexhoma/experiments/README.md§9 — fullshow_summary()call graph, per-benchmark-type evaluator/column details, result-folder file naming for every benchmarker type (§7).
Concept: Agent Report
Overview
The agent report is a tiered Markdown summary written alongside a
bexhoma experiment’s result folder, designed to be read by an LLM agent (or a
human skimming quickly) without needing the repository, a live cluster
connection, or prior context. It is generated by
bexhoma/report_writer.py and is a second view
of the exact same data show_summary() already prints to stdout — not a
separate analysis pipeline.
{resultfolder}/{code}/report/
index.md ← Tier 1: Answers
workflow.md ← Tier 2: Evidence
loading.md ← Tier 2: Evidence (only when loading was active)
benchmarking.md ← Tier 2: Evidence
monitoring.md ← Tier 2: Evidence (only when monitoring was active)
connections.md ← Tier 2: Evidence
# Tier 3: Diagnosis is not new files — it's the pre-existing result-folder
# files (connections.config, per-pod logs, rendered K8s manifests, loading
# scripts + stdout/stderr, SUT container logs, Prometheus CSVs), linked from
# every tier-2 file's Provenance footer.
Enable it with -rp/--report on any entry script (python tpch.py run ... -rp)
or on bexhoma summary -e <code> -rp. It never requires a live cluster
connection — like a plain bexhoma summary -e <code>, it only reads local
result-folder files (add -fe first only if the result folder itself needs
re-evaluating).
Why a report, not just richer stdout?
show_summary()’s stdout output is tuned for a human watching a live run: it
is also intentionally capped in a few places (four hardcoded hardware
metrics, the first five active application metrics) that a written report
should not inherit. The report reuses every DataFrame show_summary() already
computes, but formats it independently and, in monitoring.md, adds the
metrics show_summary() never shows at all — the Full Metric Catalog (see
below).
The three tiers
Tier |
Files |
Read when |
|---|---|---|
1 — Answers |
|
Always, first. Often the only file needed. |
2 — Evidence |
|
An actual metric value is needed, or a Tests-table failure needs tracing to its connection/phase. |
3 — Diagnosis |
linked raw result-folder files |
Tier 2’s aggregated tables don’t resolve the question. |
index.md
Eleven pieces, in order — items 3–5 and 10 are static boilerplate (identical on every report); the rest vary per experiment:
YAML frontmatter (
schema_version,experiment_code,workload_type,generated_at, active-phase flags,overall_statuscounts,sections).Workload identity (name, type, duration, description).
Entry-point / stop-early instruction.
Naming Conventions (positional decoding rule + the naming table + the experiment-code-is-a-Unix-timestamp fact).
Validity-First Rules (which failed test invalidates which metric).
### Tests— the full pass/failed/skipped table.Key Metrics — the benchmark type’s own headline performance metric(s) (e.g. Geo Times/Power@Size/Throughput@Size for DBMSBenchmarker, NOPM for HammerDB), the same columns its evaluator already tests via
record_tests(). Report-only — never printed to stdout, sinceindex.mdhas no stdout equivalent. Omitted when the benchmark type defines none.Monitoring — one peak-CPU/peak-RAM bullet per curated hardware component table in
monitoring.md, aggregated from the same DataFrames (no re-fetch), plus a link tomonitoring.mdfor per-phase detail and the full metric catalog. Omitted when monitoring was not active or collected no data.Health Summary — terse restart/error/warning counts; “none” in the clean case, a link to the tier-2 file with the full detail otherwise. Never the full detail itself.
Interpretation Rules (compare only within one experiment code, report variance across repetitions, cite file paths).
Section links, one per tier-2 file actually written.
Tier-2 files
workflow.md, loading.md, benchmarking.md, and monitoring.md carry the
same content show_summary() prints for those sections — only written when
the corresponding phase was active. benchmarking.md additionally holds any
secondary (co-running) benchmark’s section (e.g. a TPC-H refresh stream, or a
YCSB benchmark co-running with TPC-H — see bexhoma/experiments/README.md §9 for how
that dispatch works) and, for DBMSBenchmarker-family benchmarks, per-query
Latency/Errors/Warnings.
connections.md is new relative to show_summary(): one subsection per row
of evaluator.get_connections_of_experiment(), each with that connection’s
own parameter columns plus links to its own benchmarker log, its SUT’s
container log, its kubectl describe pod output, and the monitoring CSV
covering it. Deliberately one file with many anchors rather than one file per
connection — a parameter sweep can produce hundreds of connections.
Cross-referencing
Any table whose index holds connection names (Per Connection tables,
Application Metrics, the Full Metric Catalog’s value tables) has that index
rewritten into links to connections.md’s anchors before rendering — a
direct edit of a copied DataFrame, not text search-and-replace, so it can’t
silently miss a match. The stdout renderer never sees this rewrite.
Full Metric Catalog (monitoring.md)
Enumerates every metric key configured for the experiment (not the four
hardcoded hardware metrics or the first five active application metrics
show_summary() caps at), for every monitoring component, with its key (for
tracing back to the raw query_{component}_metric_{key}.csv), human-readable
title, category (type), and aggregation kind (metric: counter → delta,
ratio → max, other → mean).
Component-key naming convention: component is an internal routing key
(e.g. benchmarking for the SUT during the benchmarking phase, loader
for the loading-phase loader pods, datagenerator, benchmarker, …) — not
self-explanatory on its own, and not the same string as the curated section
titles used elsewhere in monitoring.md ("Benchmarking phase: SUT deployment",
etc.). loading/benchmarking/loader/benchmarker/datagenerator are a
fixed vocabulary owned by the vendored dbmsbenchmarker dependency’s own
monitor.py/evaluator.py (which read/write these exact filenames
independently of bexhoma) — not bexhoma’s naming choice to change freely,
which is exactly why component_title exists as a separate, renamable
human-readable layer instead of the raw key itself. So every catalog row also
carries a component_title column with that matching human-readable title,
and every per-metric subsection heading is
{metric title} (`{metric_key}`, {component} — {component_title}) — e.g.
“CPU Throttle (total_cpu_throttled, benchmarking — Benchmarking phase: SUT
deployment)”. A metric for a specific phase/component is therefore findable
by searching either the raw key or its title, without needing to trace the
key back through source code.
Provenance and consistency
Every ### Provenance link is built by globbing the real result folder at
generation time (pathlib.Path.glob()), never a hand-written filename
pattern — a link can never point at a file that doesn’t exist. Each group of
links carries a one-line italic description above it — why look, what’s in
there — so an agent doesn’t have to open a file just to find out what kind of
evidence it holds (e.g. “the exact rendered SQL/bash script that ran … despite
the .log suffix, this is the script source itself, not output”). Every relative
path is os.path.relpath()-computed rather than a hand-typed ../, so links
stay correct regardless of future changes to the report’s own directory
depth. index.md’s sections list is built by recording each tier-2 file as
it is actually written, not maintained as a separate constant — it cannot
list a file that was never produced, or omit one that was.
Files linked from tier 2, all pre-existing and unmodified by the report:
Artifact |
Written by |
|---|---|
|
existing bexhoma pipeline |
Benchmarker/loader pod logs, pickled DataFrames, DBMSBenchmarker cube |
existing bexhoma pipeline |
Prometheus metric CSVs ( |
|
Rendered Kubernetes Job/Deployment/Service manifests |
|
Loading DDL/bash scripts + their stdout/stderr (3 files per script) |
|
SUT container log + |
|
Architecture: structured-return sections, two independent renderers
show_summary()’s hooks (_show_loading_sections, _show_extra_sections,
show_summary_section) no longer print() their content — they return a
tree of bexhoma.benchmarks.base.Section objects (heading, optional
DataFrame, optional freeform lines, children). Two renderer functions consume
the same tree:
render_stdout()— reproducesshow_summary()’s exact historical output.report_writer.write_markdown_report()— builds the tiered report, free to format, tier, and cross-reference the same data independently.
No evaluator call or hook override is duplicated between the two — only
rendering is. This lets the report’s format evolve without constraining, or
being constrained by, the terminal output, while every DataFrame is still
fetched exactly once. See bexhoma/experiments/README.md §9 for the full
show_summary() call-graph this fits into.
A fourth, report-only method follows the same pattern for benchmark-specific
knowledge: Benchmark._build_key_metrics_section(df_aggregated_reduced)
(default None) is overridden per benchmark type — DBMSBenchmarkerBenchmark,
YCSB, TPCC, Benchbase — to name the exact column(s) that benchmark’s own
evaluator already tests via record_tests(). Deliberately not a lookup
table inside report_writer.py: which column is the headline metric is
benchmark-specific knowledge, so it lives on the benchmark class, the same
place every other benchmark-specific override already lives — report_writer.py
stays generic, rendering whatever Section it is handed without knowing what
kind of benchmark produced it.
Minimal example
python tpch.py run -dbms PostgreSQL -sf 1 -ne 1 -rp
# ...
# writes /path/to/results/<code>/report/{index,workflow,benchmarking,monitoring,connections}.md
bexhoma summary -e <code> -rp
# regenerates the report from local files only, no cluster connection needed
See also
AgentWorkflow.md— the end-to-end loop this report is read in step 6 of: question → contracts →experiment.yml→ validate → run → answer.AgentResultContract.md— the machine-readable contract version of this design doc.