Concept: Agent Report

Overview

The agent report is a tiered Markdown summary written alongside a bexhoma experiment’s result folder, designed to be read by an LLM agent (or a human skimming quickly) without needing the repository, a live cluster connection, or prior context. It is generated by bexhoma/report_writer.py and is a second view of the exact same data show_summary() already prints to stdout — not a separate analysis pipeline.

{resultfolder}/{code}/report/
    index.md          ← Tier 1: Answers
    workflow.md        ← Tier 2: Evidence
    loading.md          ← Tier 2: Evidence  (only when loading was active)
    benchmarking.md      ← Tier 2: Evidence
    monitoring.md         ← Tier 2: Evidence  (only when monitoring was active)
    connections.md         ← Tier 2: Evidence
# Tier 3: Diagnosis is not new files — it's the pre-existing result-folder
# files (connections.config, per-pod logs, rendered K8s manifests, loading
# scripts + stdout/stderr, SUT container logs, Prometheus CSVs), linked from
# every tier-2 file's Provenance footer.

Enable it with -rp/--report on any entry script (python tpch.py run ... -rp) or on bexhoma summary -e <code> -rp. It never requires a live cluster connection — like a plain bexhoma summary -e <code>, it only reads local result-folder files (add -fe first only if the result folder itself needs re-evaluating).

Why a report, not just richer stdout?

show_summary()’s stdout output is tuned for a human watching a live run: it is also intentionally capped in a few places (four hardcoded hardware metrics, the first five active application metrics) that a written report should not inherit. The report reuses every DataFrame show_summary() already computes, but formats it independently and, in monitoring.md, adds the metrics show_summary() never shows at all — the Full Metric Catalog (see below).

The three tiers

Tier	Files	Read when
1 — Answers	`index.md`	Always, first. Often the only file needed.
2 — Evidence	`workflow.md`, `loading.md`, `benchmarking.md`, `monitoring.md`, `connections.md`	An actual metric value is needed, or a Tests-table failure needs tracing to its connection/phase.
3 — Diagnosis	linked raw result-folder files	Tier 2’s aggregated tables don’t resolve the question.

`index.md`

Eleven pieces, in order — items 3–5 and 10 are static boilerplate (identical on every report); the rest vary per experiment:

YAML frontmatter (schema_version, experiment_code, workload_type, generated_at, active-phase flags, overall_status counts, sections).
Workload identity (name, type, duration, description).
Entry-point / stop-early instruction.
Naming Conventions (positional decoding rule + the naming table + the experiment-code-is-a-Unix-timestamp fact).
Validity-First Rules (which failed test invalidates which metric).
### Tests — the full pass/failed/skipped table.
Key Metrics — the benchmark type’s own headline performance metric(s) (e.g. Geo Times/Power@Size/Throughput@Size for DBMSBenchmarker, NOPM for HammerDB), the same columns its evaluator already tests via record_tests(). Report-only — never printed to stdout, since index.md has no stdout equivalent. Omitted when the benchmark type defines none.
Monitoring — one peak-CPU/peak-RAM bullet per curated hardware component table in monitoring.md, aggregated from the same DataFrames (no re-fetch), plus a link to monitoring.md for per-phase detail and the full metric catalog. Omitted when monitoring was not active or collected no data.
Health Summary — terse restart/error/warning counts; “none” in the clean case, a link to the tier-2 file with the full detail otherwise. Never the full detail itself.
Interpretation Rules (compare only within one experiment code, report variance across repetitions, cite file paths).
Section links, one per tier-2 file actually written.

Tier-2 files

workflow.md, loading.md, benchmarking.md, and monitoring.md carry the same content show_summary() prints for those sections — only written when the corresponding phase was active. benchmarking.md additionally holds any secondary (co-running) benchmark’s section (e.g. a TPC-H refresh stream, or a YCSB benchmark co-running with TPC-H — see bexhoma/experiments/README.md §9 for how that dispatch works) and, for DBMSBenchmarker-family benchmarks, per-query Latency/Errors/Warnings.

connections.md is new relative to show_summary(): one subsection per row of evaluator.get_connections_of_experiment(), each with that connection’s own parameter columns plus links to its own benchmarker log, its SUT’s container log, its kubectl describe pod output, and the monitoring CSV covering it. Deliberately one file with many anchors rather than one file per connection — a parameter sweep can produce hundreds of connections.

Cross-referencing

Any table whose index holds connection names (Per Connection tables, Application Metrics, the Full Metric Catalog’s value tables) has that index rewritten into links to connections.md’s anchors before rendering — a direct edit of a copied DataFrame, not text search-and-replace, so it can’t silently miss a match. The stdout renderer never sees this rewrite.

Full Metric Catalog (`monitoring.md`)

Enumerates every metric key configured for the experiment (not the four hardcoded hardware metrics or the first five active application metrics show_summary() caps at), for every monitoring component, with its key (for tracing back to the raw query_{component}_metric_{key}.csv), human-readable title, category (type), and aggregation kind (metric: counter → delta, ratio → max, other → mean).

Component-key naming convention: component is an internal routing key (e.g. benchmarking for the SUT during the benchmarking phase, loader for the loading-phase loader pods, datagenerator, benchmarker, …) — not self-explanatory on its own, and not the same string as the curated section titles used elsewhere in monitoring.md ("Benchmarking phase: SUT deployment", etc.). loading/benchmarking/loader/benchmarker/datagenerator are a fixed vocabulary owned by the vendored dbmsbenchmarker dependency’s own monitor.py/evaluator.py (which read/write these exact filenames independently of bexhoma) — not bexhoma’s naming choice to change freely, which is exactly why component_title exists as a separate, renamable human-readable layer instead of the raw key itself. So every catalog row also carries a component_title column with that matching human-readable title, and every per-metric subsection heading is {metric title} (`{metric_key}`, {component} — {component_title}) — e.g. “CPU Throttle (total_cpu_throttled, benchmarking — Benchmarking phase: SUT deployment)”. A metric for a specific phase/component is therefore findable by searching either the raw key or its title, without needing to trace the key back through source code.

Provenance and consistency

Every ### Provenance link is built by globbing the real result folder at generation time (pathlib.Path.glob()), never a hand-written filename pattern — a link can never point at a file that doesn’t exist. Each group of links carries a one-line italic description above it — why look, what’s in there — so an agent doesn’t have to open a file just to find out what kind of evidence it holds (e.g. “the exact rendered SQL/bash script that ran … despite the .log suffix, this is the script source itself, not output”). Every relative path is os.path.relpath()-computed rather than a hand-typed ../, so links stay correct regardless of future changes to the report’s own directory depth. index.md’s sections list is built by recording each tier-2 file as it is actually written, not maintained as a separate constant — it cannot list a file that was never produced, or omit one that was.

Files linked from tier 2, all pre-existing and unmodified by the report:

Artifact	Written by
`connections.config`, `{connection}.config`, `queries.config`	existing bexhoma pipeline
Benchmarker/loader pod logs, pickled DataFrames, DBMSBenchmarker cube	existing bexhoma pipeline
Prometheus metric CSVs (`query_{component}_metric_{key}.csv`)	`transform_monitoring_results()`
Rendered Kubernetes Job/Deployment/Service manifests	`configurations/manifest.py`, `configurations/lifecycle.py`
Loading DDL/bash scripts + their stdout/stderr (3 files per script)	`configurations/loading.py`
SUT container log + `kubectl describe pod` output	`clusters.py::store_pod_log()`/`store_pod_description()`

Architecture: structured-return sections, two independent renderers

show_summary()’s hooks (_show_loading_sections, _show_extra_sections, show_summary_section) no longer print() their content — they return a tree of bexhoma.benchmarks.base.Section objects (heading, optional DataFrame, optional freeform lines, children). Two renderer functions consume the same tree:

render_stdout() — reproduces show_summary()’s exact historical output.
report_writer.write_markdown_report() — builds the tiered report, free to format, tier, and cross-reference the same data independently.

No evaluator call or hook override is duplicated between the two — only rendering is. This lets the report’s format evolve without constraining, or being constrained by, the terminal output, while every DataFrame is still fetched exactly once. See bexhoma/experiments/README.md §9 for the full show_summary() call-graph this fits into.

A fourth, report-only method follows the same pattern for benchmark-specific knowledge: Benchmark._build_key_metrics_section(df_aggregated_reduced) (default None) is overridden per benchmark type — DBMSBenchmarkerBenchmark, YCSB, TPCC, Benchbase — to name the exact column(s) that benchmark’s own evaluator already tests via record_tests(). Deliberately not a lookup table inside report_writer.py: which column is the headline metric is benchmark-specific knowledge, so it lives on the benchmark class, the same place every other benchmark-specific override already lives — report_writer.py stays generic, rendering whatever Section it is handed without knowing what kind of benchmark produced it.

Minimal example

python tpch.py run -dbms PostgreSQL -sf 1 -ne 1 -rp
# ...
# writes /path/to/results/<code>/report/{index,workflow,benchmarking,monitoring,connections}.md

bexhoma summary -e <code> -rp
# regenerates the report from local files only, no cluster connection needed