Agent Workflow: Question to Answer

Overview

The end-to-end loop an agent follows to turn a benchmarking question into an evidence-backed answer. Two files are the actual input/output contract — contracts/contract_catalog.yml (what can be asked for) and contracts/contract_result.yml (what comes back, and how to answer) — self-contained enough that nothing else in the repository is required to know the shape of a valid experiment.yml or to interpret a finished run, by design (see each contract’s own header comment). One more file, environment.yml, is genuinely a third input — see step 4 — but it’s a live cluster snapshot, not a contract, and is only needed for the placement/resource-fit half of validation.

Question in
Read contracts/contract_catalog.yml + contracts/contract_result.yml
Build experiment.yml
python validate_experiment.py experiment.yml   ← dry run, no cluster touched
python experiment.py experiment.yml            ← actual run
Read {resultfolder}/{code}/report/index.md, answer per contract_result.yml's answer_contract

Step 2 — read the two contracts (and know about the third file)

contracts/contract_catalog.yml is not an abstract JSON-Schema-style description that then needs a separately instantiated catalog.yaml to become usable — it is the concrete catalog data. Its systems and workloads keys already list what’s actually offered:

>>> import yaml
>>> d = yaml.safe_load(open('contracts/contract_catalog.yml'))
>>> list(d['systems'])
['PostgreSQL', 'PgDuckDB']
>>> list(d['workloads'])
['tpch']

The name catalog.yaml shows up elsewhere in the codebase (experiment.py’s default sibling-file lookup, Design-Yaml-Experiment-Entry-Script.md’s catalog:/environment: provenance-pointer fields) purely as a generic name for wherever a working copy of this same file happens to live — nobody in this repo has ever created a second, differently-populated one. An agent that has read contracts/contract_catalog.yml already knows everything the catalog knows; there is no hidden third contract file to go find.

contracts/contract_result.yml is the matching output-side contract: tiers, provenance globs, validity checks, verdict_shape, and (since v1.4.0) answer_contract — the shape the final answer in step 6 should take.

environment.yml is a genuinely different, third file — not a contract. Unlike catalog.yaml, it isn’t just an alias for a file already read in step 2. contracts/contract_catalog.yml answers “what can be asked for” as a static, hand-curated, repo-committed fact; environment.yml answers “what does this specific cluster actually have, right now” — live node allocatable/free capacity, storage classes, namespace resource limits — and can only be produced by connecting to a real cluster:

python -m bexhoma.environment [-cx context] [-o dev/catalog/environment.yml] [-xhw]
# or, in-process: bexhoma environment create

Generated by bexhoma/environment.py (build_environment()), which is why it has its own embedded environment_contract_version (currently 1.0.0) instead of a contracts/contract_environment.yml doc — the file documents its own shape at generation time, per that module’s own header comment, so there is nothing separate to keep in sync. It carries a cluster.collected_at timestamp and goes stale the moment cluster capacity changes after that — the checked-in dev/catalog/environment.yml example is a snapshot from a specific point in time, not a live view; regenerate it before trusting it for a real run. spec.validate_environment() (the check step 4 runs against it) is deliberately independent of the catalog check — an agent that only has one of contract_catalog.yml / environment.yml can still validate what it has.

Step 3 — build `experiment.yml`

Conforms to contract_catalog.yml’s experiment_schema. Three fields are required — title, hypothesis, discriminates — plus workload/system selection; a fourth, follow_up_of, is optional and names a prior run’s experiment_code when this run continues it. See AgentCatalogContract.md for the condensed, agent-facing shape of everything contract_catalog.yml currently offers (the one workload and two systems in scope, their params/knobs/profiles) and dev/catalog/experiment.yml for the maintained worked example. experiment*.yml is gitignored at the repo root — a scratch ./experiment.yml there is a normal, disposable working copy, not a file that needs to be committed.

Step 4 — validate before running

validate_experiment.py is a dry run: it resolves experiment.yml against contract_catalog.yml (same resolution path experiment.py/bexhoma/spec.py::translate() use to build the real CLI argv) and, by default, also checks it against dev/catalog/environment.yml — placement node existence/taints, CPU/memory ceilings, storage-class existence. No subprocess, no live cluster connection.

python validate_experiment.py experiment.yml
# or, spelling out the defaults explicitly:
python validate_experiment.py experiment.yml -c contracts/contract_catalog.yml -e dev/catalog/environment.yml
# pass -e "" to skip the environment/placement check

A failing check prints FAIL with the specific reason (unsupported workload/system pairing, illegal parameter value, unmet profile precondition, placement/resource ceiling exceeded, …) and exits non-zero — fix experiment.yml and re-run step 4 before touching step 5.

Step 5 — run it

python experiment.py experiment.yml
# or: python experiment.py experiment.yml -c path/to/catalog.yaml   (only if not using the repo's own contract_catalog.yml)

Dispatches internally on experiment.yml’s workload: shape (catalog-driven vs. self-specified — see Design-Yaml-Experiment-Entry-Script.md) and, for a catalog-driven file, through the same build_argv() resolution validate_experiment.py already exercised in step 4 — so a run that passed validation resolves identically here, it just also actually submits to the cluster this time. Always writes the tiered Markdown report (-rp is forced on for a YAML-driven run). At run start, experiment.yml itself plus the contract_catalog.yml/contract_result.yml pair that governed it are copied verbatim into the result folder — see contract_result.yml’s provenance.workflow and answer_contract.hypothesis.

Current limit: bexhoma/spec.py::build_argv() has argv builders for the tpch and ycsb workloads today — a catalog-driven experiment.yml naming hammerdb, benchbase, or tpcds fails resolution at step 4, before step 5 is ever reached.

Step 6 — read the result, answer per contract

Entry point: {resultfolder}/{code}/report/index.md (only if -rp was used — always true for a YAML-driven run). Follow contract_result.yml’s answer_contract.steps in order:

hypothesis — restate the question, quoting experiment.yml’s hypothesis field verbatim from {resultfolder}/{code}/experiment.yml. If that file isn’t there (a hand-typed python tpch.py run ... invocation never copies one in), say plainly that no hypothesis was recorded — don’t invent one from workload params.
verdict — pass/fail/skip counts from report/index.md’s frontmatter overall_status; a FAILED row scopes/invalidates metrics reported below it, a SKIPPED row never does.
evidence — cite the specific tier-1/tier-2 file and value behind every claim (report/workflow.md, loading.md, benchmarking.md, monitoring.md, connections.md, each written only if that phase was active).
follow_up — if the verdict doesn’t fully resolve the hypothesis, or discriminates names a factor not yet varied, propose a new experiment.yml with follow_up_of set to this run’s experiment_code.