bexhoma.experiments package

Submodules

Module contents

Public API of the bexhoma.experiments package.

Exports ExperimentBase, MixedExperiment, DbmsBenchmarkerExperiment, TpchExperiment, TpcdsExperiment, BenchbaseExperiment, TpccExperiment, and YcsbExperiment experiment classes.

class bexhoma.experiments.BenchbaseExperiment(cluster, code=None, SF='1', num_experiment_to_apply=1, timeout=7200)

Bases: MixedExperiment

Benchbase experiment: orchestrates Benchbase loading and benchmarking inside a Kubernetes cluster.

Registers a Benchbase benchmark object and pre-populates the experiment dict template with Benchbase-specific job templates.

OLD_evaluate_results(pod_dashboard=''): Build a DataFrame locally that contains all benchmarking results. This is specific to Benchbase.

set_benchmark_type(benchmark: str = 'tpcc') → None

Select the Benchbase workload to run and update the storage label and config folder accordingly.

Parameters:: benchmark – Benchbase benchmark name, e.g. ‘tpcc’, ‘ycsb’, ‘seats’.

class bexhoma.experiments.DbmsBenchmarkerExperiment(cluster, code=None, SF='1', num_experiment_to_apply=1, timeout=7200)

Bases: MixedExperiment

Experiment class for DBMSBenchmarker-based benchmarks.

Provides evaluate_results (which joins per-connection results on a dashboard pod and builds the evaluation cube) and show_summary (which presents per-query latencies, errors, and warnings).

TPC-H and TPC-DS subclass this to add their own benchmark registration and experiment dict template.

Subclasses: TpchExperiment, TpcdsExperiment.

evaluate_results(pod_dashboard: str = '') → None

Let the dashboard pod build the evaluations. This is specific to dbmsbenchmarker.

All local logs are copied to the pod.
Benchmarker in the dashboard pod is updated (dev channel)
All results of all DBMS are joined (merge.py of benchmarker) in dashboard pod
Evaluation cube is built (python benchmark.py read -e yes) in dashboard pod

show_summary() → None

Print the experiment summary by delegating to the primary benchmark.

Finds the benchmark with benchmark_index == 1 and calls its show_summary() template method, passing this experiment as the context object.

class bexhoma.experiments.ExperimentBase(cluster, code=None, num_experiment_to_apply=1, timeout=7200, detached=True)

Bases: object

Class for defining an experiment. Settings are set generally. This class should be overloaded to define specific experiments.

OLD_show_summary_monitoring()

add_benchmark(benchmark) → None

Register a benchmark and create its evaluator.

Assigns a 1-based benchmark_index, instantiates the evaluator, wires evaluator.experiment back to this experiment, and sets self.evaluator to the first registered benchmark’s evaluator for backward-compat.

Parameters:: benchmark – Benchmark instance.

add_benchmark_list(list_clients: list) → None

Add a list of (number of) benchmarker instances, that are to benchmark the current SUT. Example [1,2,1] means sequentially we will have 1, then 2 and then 1 benchmarker instances. This is applied to all dbms configurations of the experiment.

Parameters:: list_clients – List of (number of) benchmarker instances

add_configuration(configuration: object) → None

Adds a configuration object to the list of configurations of this experiment. When a new configuration object is instanciated, an experiment object has to be provided. This method is then called automatically.

Parameters:: configuration – Configuration object

benchmark_list(list_clients)

DEPRECATED? Is not used anymore. Runs a given list of benchmarker applied to all running SUTs of experiment.

Parameters:: list_clients – List of (number of) benchmarker instances

benchmarking_is_active() → bool

Returns True, when this is a benchmarking experiment. Returns False in case of mode=start or mode=load. In that case, benchmarking phase is skipped and no results are expected.

Returns:: Iff there is a benchmarking phase

download_experiment_file(filename: str) → str

Download a result file from the dashboard pod.

Parameters:: filename – Name of the file relative to the experiment result folder.
Returns:: Output of the download command, or empty string when no dashboard pod is available.

end_benchmarking(jobname: str, config: object | None = None) → None

Ends a benchmarker job. This is for storing or cleaning measures.

Parameters:

jobname – Name of the job to clean
config – Configuration object

end_loading(jobname: str) → None

Ends a loading job. This is for storing or cleaning measures.

Parameters:: jobname – Name of the job to clean

evaluate_results(pod_dashboard: str = '') → None: Let the dashboard pod build the evaluations.

extract_job_timing(jobname: str, container: str) → list

Extracts start and end times from a job. Looks for all container logs of the job. Start and end are expected to be noted as BEXHOMA_START and BEXHOMA_END in the logs.

Parameters:

jobname – Name of the job
container – Name of the container in the pods of the job

Returns:

List of pairs (start,end) per pod

generate_port_forward(service: str) → str

Generate a kubectl port-forward command string for a given service.

Parameters:: service – Name of the Kubernetes service to forward.
Returns:: Shell command string that forwards the cluster port to localhost.

get_dashboard_pod(pod_dashboard: str = '') → str

Get name of dashboard pod. This also checks the status. Waits until available. If name of dashboard pod is known: do nothing

Parameters:: pod_dashboard – Optional name of dashboard pod

get_job_timing_benchmarking(jobname: str) → list

Extracts start and end times from a benchmarking job. All benchmarker containers (including refresh-stream loaders) must be named dbmsbenchmarker so the log file ends with .dbmsbenchmarker.log.

Parameters:: jobname – Name of the job
Returns:: List of pairs (start,end) per pod

get_job_timing_loading(jobname: str) → tuple

Extracts start and end times from a loading job. This uses extract_job_timing() and sets container to ‘datagenerator’ and ‘sensor’.

Parameters:: jobname – Name of the job
Returns:: List of pairs (start,end) per pod, triple of datagenerator, sensor and total (i.e., sum)

get_parameter_as_list(parameter: str) → list

Transform comma separated CLI parameters to list. This is for example used to transform num_loading_pods.

Parameters:: parameter – Comma separated list of values
Returns:: Python list of values

get_workflow_list()

Returns the planned benchmarking workflow extracted from experiment_dict.

The workflow is a dict keyed by configuration name. Each value is a list of experiment runs (length num_experiment_to_apply), where each run is a list of client rounds, and each client round is a list of job descriptors:

{
    'PostgreSQL-1-1-1': [
        [  # experiment run 1
            [  # client round 1
                {'type': 'tpch',         'pods': 3},
                {'type': 'tpch_refresh', 'pods': 1},
            ],
        ],
    ],
}

When workflow_planned is already present in self.workload (e.g. because store_workflow_results() was already called), that stored value is returned unchanged to preserve idempotency.

Returns:: Dict mapping configuration name to the nested workflow structure.
Return type:: dict

loading_is_active() → bool

Returns True, when this is an experiment including loading. Returns False in case of mode=start. In that case, loading and benchmarking phases are skipped and no results are expected.

Returns:: Iff there is a benchmarking phase

patch_benchmarking(patch: str) → None

Patches YAML of loading components. Can be set by experiment before creation of configuration.

Parameters:: patch – String in YAML format, overwrites basic YAML file content

patch_loading(patch: str) → None

Patches YAML of loading components. Can be overwritten by configuration.

Parameters:: patch – String in YAML format, overwrites basic YAML file content

prepare_testbed(parameter: dict) → None

Prepares a testbed. It takes the CLI arguments as a dict. This method for example sets workload[‘info’]. It also sets monitoring, storage, components settings, SUT resources and nodes for components. It makes sure, dashboard, messagequeue, data dir and result dir are ready.

Parameters:: parameter – Comma separated list of values

process() → None

Run the experiment according to self.args.mode.

Dispatches to the appropriate workflow branch (start, load, summary, or the default full benchmark run) and prints timing information.

result_filename(filename: str, posix: bool = True) → str

Returns filename including path in a local result folder.

Parameters:

filename – name of a file in a result folder
posix – iff posix format should be used

Returns:

self.path / filename

result_filename_local(filename: str) → str

Returns filename including path in result folder.

Parameters:

filename – name of a file in a result folder
posix – iff posix format should be used

Returns:

self.path / filename

result_filename_remote(filename: str, posix: bool = True) → str

Returns filename including path in result folder.

Parameters:

filename – name of a file in a result folder
posix – iff posix format should be used

Returns:

self.path / filename

set_additional_labels(**kwargs) → None

Sets additional labels, that will be put to K8s objects (and ignored otherwise). This is for the SUT component. Can be overwritten by configuration.

Parameters:: kwargs – Dict of labels, example ‘SF’ => 100

set_benchmarking_parameters(**kwargs) → None

Sets ENV for benchmarking components. Can be overwritten by configuration.

Parameters:: kwargs – Dict of meta data, example ‘PARALLEL’ => ‘64’

set_connection_management(**kwargs) → None

Sets connection management data for the experiment. This is for the benchmarker component (dbmsbenchmarker). Can be overwritten by configuration.

Parameters:: kwargs – Dict of meta data, example ‘timout’ => 60

set_ddl_parameters(**kwargs) → None

Sets DDL parameters for the experiments. This substitutes placeholders in DDL script. Can be overwritten by configuration.

Parameters:: kwargs – Dict of meta data, example ‘index’ => ‘btree’

set_default_benchmarking_parameters(**kwargs) → None

Sets experiment-wide default ENV for benchmarking components.

These defaults are merged into every per-configuration benchmarking parameter dict before per-configuration kwargs are applied (per-configuration values win).

Parameters:: kwargs – Default ENV vars shared across all configurations of this experiment.

set_default_loading_parameters(**kwargs) → None

Sets experiment-wide default ENV for loading components.

These defaults are merged into every per-configuration loading parameter dict before per-configuration kwargs are applied (per-configuration values win).

Parameters:: kwargs – Default ENV vars shared across all configurations of this experiment.

set_eval_parameters(**kwargs) → None

Sets some arbitrary parameters that are supposed to be handed over to the benchmarker component. These are for evaluation purposes only. Can be overwritten by configuration.

Parameters:: kwargs – Dict of meta data, example ‘type’ => ‘noindex’

set_experiment(instance=None, volume=None, docker=None, script=None, indexing=None)

Read experiment details from cluster config

Parameters:

instance
volume
docker
script

set_experiments_configfolder(experiments_configfolder: str) → None

Set the relative config folder for the experiment type.

Bexhoma expects subfolders per experiment type (e.g. experiments/tpch). Each subfolder must contain a queries.config and per-DBMS schema subdirectories.

Parameters:: experiments_configfolder – Relative path to the experiment config folder.

set_loading(parallel: int, num_pods: int | None = None) → None

Sets job parameters for loading components: Number of parallel pods and optionally (if different) total number of pods. By default total number of pods is set to number of parallel pods. Can be overwritten by configuration.

Parameters:

parallel – Number of parallel pods
num_pods – Optionally (if different) total number of pods

set_loading_parameters(**kwargs) → None

Sets ENV for loading components. Can be overwritten by configuration.

Parameters:: kwargs – Dict of meta data, example ‘PARALLEL’ => ‘64’

set_maintaining(parallel: int, num_pods: int | None = None) → None

Sets job parameters for maintaining components: Number of parallel pods and optionally (if different) total number of pods. By default total number of pods is set to number of parallel pods. Can be overwritten by configuration.

Parameters:

parallel – Number of parallel pods
num_pods – Optionally (if different) total number of pods

set_maintaining_parameters(**kwargs) → None

Sets ENV for maintaining components. Can be overwritten by configuration.

Parameters:: kwargs – Dict of meta data, example ‘PARALLEL’ => ‘64’

set_nodes(**kwargs) → None

Sets parameters for nodes for the components of an experiment. Will be used for nodeSelector. Example:

sut = ‘sut’, loading = ‘auxiliary’, monitoring = ‘auxiliary’, benchmarking = ‘auxiliary’,

Can be overwritten by configuration.

Parameters:: kwargs – Dict of node infos, example ‘sut’ => ‘sut’,

set_query_management(**kwargs) → None

Sets query management data for the experiment. This is for the benchmarker component (dbmsbenchmarker).

Parameters:: kwargs – Dict of meta data, example ‘numRun’ => 3

set_queryfile(queryfile: str) → None

Set the name of the query config file used by the benchmarker component.

Parameters:: queryfile – Filename of the dbmsbenchmarker query config (e.g. queries.config).

set_querymanagement_monitoring(numRun: int = 256, delay: int = 10, datatransfer: bool = False) → None

Sets some parameters that are supposed to be suitable for a monitoring test:

high number of runs
optional delay
optional data transfer
monitoring active

Parameters:

numRun – Number of runs per query (this is for the benchmarker component)
delay – Number of seconds to wait between queries (this is for the benchmarker component)
datatransfer – If data should we retrieved and compared

set_querymanagement_quicktest(numRun: int = 1, datatransfer: bool = False) → None

Sets some parameters that are supposed to be suitable for a quick functional test:

small number of runs
no delay
optional data transfer
no monitoring

Parameters:

numRun – Number of runs per query (this is for the benchmarker component)
datatransfer – If data should we retrieved and compared

set_resources(**kwargs) → None

Sets resources for the experiment. This is for the SUT component. Can be overwritten by experiment and configuration.

Parameters:: kwargs – Dict of meta data, example ‘requests’ => {‘cpu’ => 4}

set_storage(**kwargs) → None

Sets parameters for the storage that might be attached to components. This is in particular for the database of dbms under test. Example:

storageClassName = ‘ssd’, storageSize = ‘100Gi’, keep = False

Can be overwritten by configuration.

Parameters:: kwargs – Dict of meta data, example ‘storageSize’ => ‘100Gi’

set_sut_parameters(**kwargs) → None

Sets ENV for sut and worker components. Can be overwritten by configuration.

Parameters:: kwargs – Dict of meta data, example ‘PARALLEL’ => ‘64’

set_workload(**kwargs) → None

Sets mata data about the experiment, for example name and description.

Parameters:: kwargs – Dict of meta data, example ‘name’ => ‘TPC-H’

show_summary(): Print a basic experiment summary using the DBMSBenchmarker inspector directly.

Deprecated since version Probably: unused in practice. All named experiment types (tpch, tpcds, benchbase, ycsb, tpcc) override this via show_summary(), which delegates to the bexhoma.benchmarks pipeline. This implementation is only reachable for legacy result folders whose queries.config still contains type='dbmsbenchmarker' — a value written by experiments.base before the named experiment subclasses existed.

show_summary_header() → tuple

Print workload metadata and per-connection details, then return sorted connection list and monitoring data.

Returns:: Tuple of (connections_sorted, monitoring_applications) where connections_sorted is a list of connection dicts ordered by name and monitoring_applications maps component title to a DataFrame of application metrics.

show_summary_monitoring() → None

Print monitoring tables for all registered monitoring components and record test results.

Results are appended to self._test_results via _record_test().

show_summary_monitoring_table(evaluate: object, component: str) → list

Build a list of DataFrames containing CPU and RAM monitoring metrics for one component.

Parameters:

evaluate – Evaluator object exposing get_monitoring_metric().
component – Component name used as the metric scope (e.g. ‘loading’, ‘stream’).

Returns:

List of single-column DataFrames, one per collected metric.

start_loading(): Tells all dbms configurations of this experiment to start loading data.

start_monitoring(): Start monitoring for all dbms configurations of this experiment.

start_sut(): Start all dbms configurations of this experiment.

stop_benchmarker(configuration: str = '') → None: Stop all benchmarker jobs of this experiment. If a dbms configurations is given, use it. Otherwise tell the cluster to stop all benchmarker jobs belonging to this experiment code.

stop_loading(): Stop all loading jobs of this experiment. If a list of dbms configurations is set, use it. Otherwise tell the cluster to stop all loading jobs belonging to this experiment code.

stop_maintaining(): Stop all maintaining jobs of this experiment. If a list of dbms configurations is set, use it. Otherwise tell the cluster to stop all maintaining jobs belonging to this experiment code.

stop_monitoring(): Stop all monitoring deployments of this experiment. If a list of dbms configurations is set, use it. Otherwise tell the cluster to stop all monitoring deployments belonging to this experiment code.

stop_sut(): Stop all SUT deployments of this experiment. If a list of dbms configurations is set, use it. Otherwise tell the cluster to stop all monitoring deployments belonging to this experiment code.

store_workflow_results()

Constructs a list of runs for the planned workflow. Stores this information in self.workload[‘workflow_planned’]. Updates query.config locally and remotely via update_workload().

When the experiment has registered Benchmark objects (via add_benchmark()), the ordered benchmark type sequence is written to self.workload['benchmark_sequence'] so that collectors can later reconstruct the benchmark-run → tool-type mapping from queries.config.

test_results() → None

Run test script locally. Extract exit code.

Returns:: exit code of test script

test_results_in_dashboard() → int

DEPRECATED? Not used currently - depends on good test script for dbmsbenchmarker Run test script in dashboard pod. Extract exit code.

Returns:: exit code of test script

test_workflow(workflow_1: dict, workflow_2: dict) → bool

Compares two workflow dicts for equality.

Each workflow maps a configuration name to a list of experiment runs, where each run is a list of client rounds, and each round is a list of job descriptors {'type': str, 'pods': int}.

The comparison is order-sensitive across experiment runs and client rounds (their sequence matters) but order-insensitive within a single client round (parallel jobs may be scheduled in any order).

Parameters:

workflow_1 (dict) – First workflow dict.
workflow_2 (dict) – Second workflow dict.

Returns:

True if the two workflows are equivalent.

Return type:

bool

update_workload(): Updates query.config locally and remotely via dashboard pod.

upload_experiment_file(filename: str) → str

Upload a result file to the dashboard pod.

Parameters:: filename – Name of the file relative to the experiment result folder.
Returns:: Output of the upload command, or empty string when no dashboard pod is available.

wait(sec: int, silent: bool = False) → None

Wait for a given number of seconds, optionally without printing.

Parameters:

sec – Number of seconds to wait.
silent – If True, suppress output during the wait.

work_benchmark_list(intervals: int = 30, stop_after_starting: bool = False, stop_after_loading: bool = False, stop_after_benchmarking: bool = False) → None

Run typical workflow:

start SUT
start monitoring
start loading (at first scripts (schema or loading via pull), then optionally parallel loading pods)
optionally start maintaining pods
at the same time as 4. run benchmarker jobs corresponding to list given via add_benchmark_list()
remove everything when done

Parameters:

intervals – Seconds to wait before checking change of status
stop_after_starting – stops after phase 2)
stop_after_loading – stops after phase 3)
stop_after_benchmarking – stops after phase 5) This tells if SUT should not be removed when all benchmarking has finished. Set to True if we want to have loaded SUTs for inspection.

zip(): Zip the result folder in the dashboard pod.

class bexhoma.experiments.MixedExperiment(cluster, code=None, num_experiment_to_apply: int = 1, timeout: int = 7200)

Bases: ExperimentBase

General experiment class supporting multiple parallel or sequential benchmarks.

Holds an experiment_dict_template that is deep-copied to every new configuration on add_configuration(), and a list of registered Benchmark objects that govern result interpretation.

Named single-benchmark experiments (YCSB, TPC-C, TPC-H, TPC-DS, Benchbase) subclass MixedExperiment and pre-populate the template in __init__. Direct use of MixedExperiment is intended for fully custom experiment dicts.

Parameters:

cluster – Cluster object, typically referring to a Kubernetes cluster.
code – Unique experiment identifier; generated from current time if None.
num_experiment_to_apply – How many times to repeat the full experiment.
timeout – Maximum timeout per query in seconds.

add_benchmark(benchmark) → None

Register a benchmark and create its evaluator.

Assigns a 1-based benchmark_index to the benchmark, instantiates its evaluator, wires evaluator.experiment back to this experiment, and sets self.evaluator to the first registered benchmark’s evaluator for backward-compatibility with single-benchmark call sites.

Parameters:: benchmark – Benchmark instance to register.

add_configuration(configuration) → None

Add a configuration and copy the experiment dict template to it.

If the configuration already has an experiment_dict attribute with empty "loader" and "benchmarker" lists, the template is deep-copied into it. Configurations that do not carry experiment_dict (pre-migration configurations) are left unchanged.

Parameters:: configuration – Configuration object to register.

prepare_testbed(parameter: dict) → None

Configure all registered benchmarks and then delegate to ExperimentBase.prepare_testbed().

Calls configure_workload() on every registered benchmark in registration order before forwarding to the parent.

Parameters:: parameter – Dict of CLI arguments as produced by argparse.

show_summary() → None

Print a Markdown-formatted summary for every registered benchmark.

Delegates to show_summary() on each benchmark in registration order.

test_results() → None

Validate results for every registered benchmark.

Delegates to test_results() on each benchmark in registration order.

class bexhoma.experiments.TpccExperiment(cluster, code=None, SF='1', num_experiment_to_apply=1, timeout=7200)

Bases: MixedExperiment

TPC-C experiment: orchestrates HammerDB TPC-C loading and benchmarking inside a Kubernetes cluster.

Registers a TPCC benchmark object and pre-populates the experiment dict template with HammerDB-specific job templates.

OLD_evaluate_results(pod_dashboard=''): Build a DataFrame locally that contains all benchmarking results. This is specific to HammerDB.

class bexhoma.experiments.TpcdsExperiment(cluster, code=None, queryfile='queries-tpcds.config', SF='100', num_experiment_to_apply=1, timeout=7200, script=None)

Bases: DbmsBenchmarkerExperiment

TPC-DS experiment: orchestrates data generation, loading, and DBMSBenchmarker query execution inside a Kubernetes cluster.

Registers a TPCDS benchmark object and pre-populates the experiment dict template. Workload configuration (modes, info strings, indexing strategies) is delegated to configure_workload().

Extends DbmsBenchmarkerExperiment.

set_queries_full() → None: Switch to the full TPC-DS query file covering all 99 queries.

set_queries_profiling() → None: Switch to the abbreviated profiling query file for import validation.

class bexhoma.experiments.TpchExperiment(cluster, code=None, queryfile='queries-tpch.config', SF='100', num_experiment_to_apply=1, timeout=7200, script=None)

Bases: DbmsBenchmarkerExperiment

TPC-H experiment: orchestrates data generation, loading, and DBMSBenchmarker query execution inside a Kubernetes cluster.

Registers a TPCH benchmark object and pre-populates the experiment dict template. Workload configuration (modes, info strings, indexing strategies) is delegated to configure_workload().

Extends DbmsBenchmarkerExperiment.

enable_refresh_stream(template: str = 'jobtemplate-benchmarking-tpch-refresh-PostgreSQL.yml') → None

Add a TPC-H RF1/RF2 refresh stream that runs in parallel with the query stream.

The refresh stream becomes benchmark_run=2 within each client round. Call set_default_benchmarking_parameters() with TPCH_REFRESH_STREAMS and TPCH_REFRESH_STREAM_OFFSET before calling this method so those values reach both the generator initContainer and the loader main container.

Parameters:: template – k8s job-template file for the refresh benchmarker job. Choose the variant matching the target DBMS (jobtemplate-benchmarking-tpch-refresh-PostgreSQL.yml or jobtemplate-benchmarking-tpch-refresh-MySQL.yml).

set_queries_full() → None: Switch to the full TPC-H query file covering all 22 queries.

set_queries_profiling() → None: Switch to the abbreviated profiling query file for import validation.

show_summary() → None

Print the TPC-H experiment summary, including the refresh stream section.

When enable_refresh_stream() was called during the live run, RefreshStreamBenchmark is already in self.benchmarks and the generic loop inside dbmsbenchmarker.show_summary() places the section right after ### Execution → Per Phase.

When called post-hoc via bexperiments summary (no enable_refresh_stream()), a temporary RefreshStreamBenchmark is appended to self.benchmarks before delegating to super(), so the same loop positions it identically. The temporary entry is removed afterwards.

class bexhoma.experiments.YcsbExperiment(cluster, code=None, SF='1', num_experiment_to_apply=1, timeout=7200)

Bases: MixedExperiment

YCSB experiment: orchestrates loading and benchmarking via the Yahoo Cloud Serving Benchmark tool inside a Kubernetes cluster.

Registers a YCSB benchmark object and pre-populates the experiment dict template with YCSB-specific job templates.

Extends MixedExperiment.

OLD_evaluate_results(pod_dashboard=''): Build a DataFrame locally that contains all benchmarking results. This is specific to YCSB.

bexhoma.experiments.base: alias of ExperimentBase

bexhoma.experiments.benchbase: alias of BenchbaseExperiment

bexhoma.experiments.dbmsbenchmarker: alias of DbmsBenchmarkerExperiment

bexhoma.experiments.mixed: alias of MixedExperiment

bexhoma.experiments.tpcc: alias of TpccExperiment

bexhoma.experiments.tpcds: alias of TpcdsExperiment

bexhoma.experiments.tpch: alias of TpchExperiment

bexhoma.experiments.ycsb: alias of YcsbExperiment