bexhoma.experiments.dbmsbenchmarker module

Experiment class for DBMSBenchmarker benchmarks.

Provides DbmsBenchmarkerExperiment, which extends MixedExperiment to orchestrate the DBMSBenchmarker tool inside a Kubernetes cluster, including loading, benchmarking, and result collection phases.

Authors: Patrick K. Erdelt Copyright (C) 2020 Patrick K. Erdelt SPDX-License-Identifier: AGPL-3.0-or-later See LICENSE for details.

class bexhoma.experiments.dbmsbenchmarker.DbmsBenchmarkerExperiment(cluster, code=None, SF='1', num_experiment_to_apply=1, timeout=7200)

Bases: MixedExperiment

Experiment class for DBMSBenchmarker-based benchmarks.

Provides evaluate_results (which joins per-connection results on a dashboard pod and builds the evaluation cube) and show_summary (which presents per-query latencies, errors, and warnings).

TPC-H and TPC-DS subclass this to add their own benchmark registration and experiment dict template.

Subclasses: TpchExperiment, TpcdsExperiment.

evaluate_results(pod_dashboard: str = '') → None

Let the dashboard pod build the evaluations. This is specific to dbmsbenchmarker.

  1. All local logs are copied to the pod.

  2. Benchmarker in the dashboard pod is updated (dev channel)

  3. All results of all DBMS are joined (merge.py of benchmarker) in dashboard pod

  4. Evaluation cube is built (python benchmark.py read -e yes -scm) in dashboard pod; its stdout/stderr is persisted to evaluate_results.log in the result folder so the cube-building step can be inspected/timed after the fact. -scm skips dbmsbenchmarker’s per-component (loading/streaming/loader/ benchmarker/datagenerator) hardware metric aggregates, since bexhoma’s own evaluators never read them (they are only consumed by dbmsbenchmarker’s interactive evaluation notebooks).

set_active_queries(active_queries: list[int] | None) → None

Restrict the query workload to a set of query numbers.

Only affects the per-experiment query config copy written to the result folder and uploaded to the cluster; the query config file in the corresponding experiments/ folder is never modified. Query numbers are 1-based positions within the query config’s queries list. Queries not listed are marked inactive ('active': False) for this experiment.

Parameters:

active_queries (list[int] | None) – 1-based query numbers to keep active, or None to leave every query’s active flag as defined in the query config file.

show_summary(write_report: bool = False) → None

Print the experiment summary by delegating to the primary benchmark.

Finds the benchmark with benchmark_index == 1 and calls its show_summary() template method, passing this experiment as the context object.

Parameters:

write_report – When True, also write a tiered Markdown report (report/index.md + detail files) to the result folder.