bexhoma.experiments.base module

Base experiment class and general-purpose experiment types.

Provides DictToObject, ExperimentBase (the core experiment class plugged into a cluster object to manage the full lifecycle of a bexhoma experiment), and derived types IotExperiment, TsbsExperiment, ExampleExperiment, and TpcxaiExperiment. Each experiment folder must contain a query file and per-DBMS schema subfolders.

Authors: Patrick K. Erdelt Copyright (C) 2020 Patrick K. Erdelt SPDX-License-Identifier: AGPL-3.0-or-later See LICENSE for details.

class bexhoma.experiments.base.DictToObject(dictionary)

Bases: object

Recursively convert a nested dict into an object with attribute access.

Nested dicts become nested DictToObject instances; all other values are stored as-is.

class bexhoma.experiments.base.ExampleExperiment(cluster, code=None, queryfile='queries.config', num_experiment_to_apply=1, timeout=7200, script=None)

Bases: ExperimentBase

Class for defining a custom example experiment. This sets

  • the folder to the experiment - including query file and schema informations per dbms

  • name and information about the experiment

class bexhoma.experiments.base.ExperimentBase(cluster, code=None, num_experiment_to_apply=1, timeout=7200, detached=True)

Bases: object

Class for defining an experiment. Settings are set generally. This class should be overloaded to define specific experiments.

OLD_show_summary_monitoring()
add_benchmark(benchmark) → None

Register a benchmark and create its evaluator.

Assigns a 1-based benchmark_index, instantiates the evaluator, wires evaluator.experiment back to this experiment, and sets self.evaluator to the first registered benchmark’s evaluator for backward-compat.

Parameters:

benchmark – Benchmark instance.

add_benchmark_list(list_clients: list) → None

Add a list of (number of) benchmarker instances, that are to benchmark the current SUT. Example [1,2,1] means sequentially we will have 1, then 2 and then 1 benchmarker instances. This is applied to all dbms configurations of the experiment.

Parameters:

list_clients – List of (number of) benchmarker instances

add_configuration(configuration: object) → None

Adds a configuration object to the list of configurations of this experiment. When a new configuration object is instanciated, an experiment object has to be provided. This method is then called automatically.

Parameters:

configuration – Configuration object

benchmark_list(list_clients)

DEPRECATED? Is not used anymore. Runs a given list of benchmarker applied to all running SUTs of experiment.

Parameters:

list_clients – List of (number of) benchmarker instances

benchmarking_is_active() → bool

Returns True, when this is a benchmarking experiment. Returns False in case of mode=start or mode=load. In that case, benchmarking phase is skipped and no results are expected.

Returns:

Iff there is a benchmarking phase

download_experiment_file(filename: str) → str

Download a result file from the dashboard pod.

Parameters:

filename – Name of the file relative to the experiment result folder.

Returns:

Output of the download command, or empty string when no dashboard pod is available.

end_benchmarking(jobname: str, config: object | None = None) → None

Ends a benchmarker job. This is for storing or cleaning measures.

Parameters:

  • jobname – Name of the job to clean

  • config – Configuration object

end_loading(jobname: str) → None

Ends a loading job. This is for storing or cleaning measures.

Parameters:

jobname – Name of the job to clean

evaluate_results(pod_dashboard: str = '') → None

Let the dashboard pod build the evaluations.

extract_job_timing(jobname: str, container: str) → list

Extracts start and end times from a job. Looks for all container logs of the job. Start and end are expected to be noted as BEXHOMA_START and BEXHOMA_END in the logs.

Parameters:

  • jobname – Name of the job

  • container – Name of the container in the pods of the job

Returns:

List of pairs (start,end) per pod

generate_port_forward(service: str) → str

Generate a kubectl port-forward command string for a given service.

Parameters:

service – Name of the Kubernetes service to forward.

Returns:

Shell command string that forwards the cluster port to localhost.

get_dashboard_pod(pod_dashboard: str = '') → str

Get name of dashboard pod. This also checks the status. Waits until available. If name of dashboard pod is known: do nothing

Parameters:

pod_dashboard – Optional name of dashboard pod

get_job_timing_benchmarking(jobname: str) → list

Extracts start and end times from a benchmarking job. All benchmarker containers (including refresh-stream loaders) must be named dbmsbenchmarker so the log file ends with .dbmsbenchmarker.log.

Parameters:

jobname – Name of the job

Returns:

List of pairs (start,end) per pod

get_job_timing_loading(jobname: str) → tuple

Extracts start and end times from a loading job. This uses extract_job_timing() and sets container to ‘datagenerator’ and ‘sensor’.

Parameters:

jobname – Name of the job

Returns:

List of pairs (start,end) per pod, triple of datagenerator, sensor and total (i.e., sum)

get_parameter_as_list(parameter: str) → list

Transform comma separated CLI parameters to list. This is for example used to transform num_loading_pods.

Parameters:

parameter – Comma separated list of values

Returns:

Python list of values

get_workflow_list()

Returns the planned benchmarking workflow extracted from experiment_dict.

The workflow is a dict keyed by configuration name. Each value is a list of experiment runs (length num_experiment_to_apply), where each run is a list of client rounds, and each client round is a list of job descriptors:

{
    'PostgreSQL-1-1-1': [
        [  # experiment run 1
            [  # client round 1
                {'type': 'tpch',         'pods': 3},
                {'type': 'tpch_refresh', 'pods': 1},
            ],
        ],
    ],
}

When workflow_planned is already present in self.workload (e.g. because store_workflow_results() was already called), that stored value is returned unchanged to preserve idempotency.

Returns:

Dict mapping configuration name to the nested workflow structure.

Return type:

dict

loading_is_active() → bool

Returns True, when this is an experiment including loading. Returns False in case of mode=start. In that case, loading and benchmarking phases are skipped and no results are expected.

Returns:

Iff there is a benchmarking phase

patch_benchmarking(patch: str) → None

Patches YAML of loading components. Can be set by experiment before creation of configuration.

Parameters:

patch – String in YAML format, overwrites basic YAML file content

patch_loading(patch: str) → None

Patches YAML of loading components. Can be overwritten by configuration.

Parameters:

patch – String in YAML format, overwrites basic YAML file content

prepare_testbed(parameter: dict) → None

Prepares a testbed. It takes the CLI arguments as a dict. This method for example sets workload[‘info’]. It also sets monitoring, storage, components settings, SUT resources and nodes for components. It makes sure, dashboard, messagequeue, data dir and result dir are ready.

Parameters:

parameter – Comma separated list of values

process() → None

Run the experiment according to self.args.mode.

Dispatches to the appropriate workflow branch (start, load, summary, or the default full benchmark run) and prints timing information.

result_filename(filename: str, posix: bool = True) → str

Returns filename including path in a local result folder.

Parameters:

  • filename – name of a file in a result folder

  • posix – iff posix format should be used

Returns:

self.path / filename

result_filename_local(filename: str) → str

Returns filename including path in result folder.

Parameters:

  • filename – name of a file in a result folder

  • posix – iff posix format should be used

Returns:

self.path / filename

result_filename_remote(filename: str, posix: bool = True) → str

Returns filename including path in result folder.

Parameters:

  • filename – name of a file in a result folder

  • posix – iff posix format should be used

Returns:

self.path / filename

set_additional_labels(**kwargs) → None

Sets additional labels, that will be put to K8s objects (and ignored otherwise). This is for the SUT component. Can be overwritten by configuration.

Parameters:

kwargs – Dict of labels, example ‘SF’ => 100

set_benchmarking_parameters(**kwargs) → None

Sets ENV for benchmarking components. Can be overwritten by configuration.

Parameters:

kwargs – Dict of meta data, example ‘PARALLEL’ => ‘64’

set_connection_management(**kwargs) → None

Sets connection management data for the experiment. This is for the benchmarker component (dbmsbenchmarker). Can be overwritten by configuration.

Parameters:

kwargs – Dict of meta data, example ‘timout’ => 60

set_ddl_parameters(**kwargs) → None

Sets DDL parameters for the experiments. This substitutes placeholders in DDL script. Can be overwritten by configuration.

Parameters:

kwargs – Dict of meta data, example ‘index’ => ‘btree’

set_default_benchmarking_parameters(**kwargs) → None

Sets experiment-wide default ENV for benchmarking components.

These defaults are merged into every per-configuration benchmarking parameter dict before per-configuration kwargs are applied (per-configuration values win).

Parameters:

kwargs – Default ENV vars shared across all configurations of this experiment.

set_default_loading_parameters(**kwargs) → None

Sets experiment-wide default ENV for loading components.

These defaults are merged into every per-configuration loading parameter dict before per-configuration kwargs are applied (per-configuration values win).

Parameters:

kwargs – Default ENV vars shared across all configurations of this experiment.

set_eval_parameters(**kwargs) → None

Sets some arbitrary parameters that are supposed to be handed over to the benchmarker component. These are for evaluation purposes only. Can be overwritten by configuration.

Parameters:

kwargs – Dict of meta data, example ‘type’ => ‘noindex’

set_experiment(instance=None, volume=None, docker=None, script=None, indexing=None)

Read experiment details from cluster config

Parameters:

  • instance

  • volume

  • docker

  • script

set_experiments_configfolder(experiments_configfolder: str) → None

Set the relative config folder for the experiment type.

Bexhoma expects subfolders per experiment type (e.g. experiments/tpch). Each subfolder must contain a queries.config and per-DBMS schema subdirectories.

Parameters:

experiments_configfolder – Relative path to the experiment config folder.

set_loading(parallel: int, num_pods: int | None = None) → None

Sets job parameters for loading components: Number of parallel pods and optionally (if different) total number of pods. By default total number of pods is set to number of parallel pods. Can be overwritten by configuration.

Parameters:

  • parallel – Number of parallel pods

  • num_pods – Optionally (if different) total number of pods

set_loading_parameters(**kwargs) → None

Sets ENV for loading components. Can be overwritten by configuration.

Parameters:

kwargs – Dict of meta data, example ‘PARALLEL’ => ‘64’

set_maintaining(parallel: int, num_pods: int | None = None) → None

Sets job parameters for maintaining components: Number of parallel pods and optionally (if different) total number of pods. By default total number of pods is set to number of parallel pods. Can be overwritten by configuration.

Parameters:

  • parallel – Number of parallel pods

  • num_pods – Optionally (if different) total number of pods

set_maintaining_parameters(**kwargs) → None

Sets ENV for maintaining components. Can be overwritten by configuration.

Parameters:

kwargs – Dict of meta data, example ‘PARALLEL’ => ‘64’

set_nodes(**kwargs) → None

Sets parameters for nodes for the components of an experiment. Will be used for nodeSelector. Example:

sut = ‘sut’, loading = ‘auxiliary’, monitoring = ‘auxiliary’, benchmarking = ‘auxiliary’,

Can be overwritten by configuration.

Parameters:

kwargs – Dict of node infos, example ‘sut’ => ‘sut’,

set_query_management(**kwargs) → None

Sets query management data for the experiment. This is for the benchmarker component (dbmsbenchmarker).

Parameters:

kwargs – Dict of meta data, example ‘numRun’ => 3

set_queryfile(queryfile: str) → None

Set the name of the query config file used by the benchmarker component.

Parameters:

queryfile – Filename of the dbmsbenchmarker query config (e.g. queries.config).

set_querymanagement_monitoring(numRun: int = 256, delay: int = 10, datatransfer: bool = False) → None

Sets some parameters that are supposed to be suitable for a monitoring test:

  • high number of runs

  • optional delay

  • optional data transfer

  • monitoring active

Parameters:

  • numRun – Number of runs per query (this is for the benchmarker component)

  • delay – Number of seconds to wait between queries (this is for the benchmarker component)

  • datatransfer – If data should we retrieved and compared

set_querymanagement_quicktest(numRun: int = 1, datatransfer: bool = False) → None

Sets some parameters that are supposed to be suitable for a quick functional test:

  • small number of runs

  • no delay

  • optional data transfer

  • no monitoring

Parameters:

  • numRun – Number of runs per query (this is for the benchmarker component)

  • datatransfer – If data should we retrieved and compared

set_resources(**kwargs) → None

Sets resources for the experiment. This is for the SUT component. Can be overwritten by experiment and configuration.

Parameters:

kwargs – Dict of meta data, example ‘requests’ => {‘cpu’ => 4}

set_storage(**kwargs) → None

Sets parameters for the storage that might be attached to components. This is in particular for the database of dbms under test. Example:

storageClassName = ‘ssd’, storageSize = ‘100Gi’, keep = False

Can be overwritten by configuration.

Parameters:

kwargs – Dict of meta data, example ‘storageSize’ => ‘100Gi’

set_sut_parameters(**kwargs) → None

Sets ENV for sut and worker components. Can be overwritten by configuration.

Parameters:

kwargs – Dict of meta data, example ‘PARALLEL’ => ‘64’

set_workload(**kwargs) → None

Sets mata data about the experiment, for example name and description.

Parameters:

kwargs – Dict of meta data, example ‘name’ => ‘TPC-H’

show_summary()

Print a basic experiment summary using the DBMSBenchmarker inspector directly.

Deprecated since version Probably: unused in practice. All named experiment types (tpch, tpcds, benchbase, ycsb, tpcc) override this via show_summary(), which delegates to the bexhoma.benchmarks pipeline. This implementation is only reachable for legacy result folders whose queries.config still contains type='dbmsbenchmarker' — a value written by experiments.base before the named experiment subclasses existed.

show_summary_header() → tuple

Print workload metadata and per-connection details, then return sorted connection list and monitoring data.

Returns:

Tuple of (connections_sorted, monitoring_applications) where connections_sorted is a list of connection dicts ordered by name and monitoring_applications maps component title to a DataFrame of application metrics.

show_summary_monitoring() → None

Print monitoring tables for all registered monitoring components and record test results.

Results are appended to self._test_results via _record_test().

show_summary_monitoring_table(evaluate: object, component: str) → list

Build a list of DataFrames containing CPU and RAM monitoring metrics for one component.

Parameters:

  • evaluate – Evaluator object exposing get_monitoring_metric().

  • component – Component name used as the metric scope (e.g. ‘loading’, ‘stream’).

Returns:

List of single-column DataFrames, one per collected metric.

start_loading()

Tells all dbms configurations of this experiment to start loading data.

start_monitoring()

Start monitoring for all dbms configurations of this experiment.

start_sut()

Start all dbms configurations of this experiment.

stop_benchmarker(configuration: str = '') → None

Stop all benchmarker jobs of this experiment. If a dbms configurations is given, use it. Otherwise tell the cluster to stop all benchmarker jobs belonging to this experiment code.

stop_loading()

Stop all loading jobs of this experiment. If a list of dbms configurations is set, use it. Otherwise tell the cluster to stop all loading jobs belonging to this experiment code.

stop_maintaining()

Stop all maintaining jobs of this experiment. If a list of dbms configurations is set, use it. Otherwise tell the cluster to stop all maintaining jobs belonging to this experiment code.

stop_monitoring()

Stop all monitoring deployments of this experiment. If a list of dbms configurations is set, use it. Otherwise tell the cluster to stop all monitoring deployments belonging to this experiment code.

stop_sut()

Stop all SUT deployments of this experiment. If a list of dbms configurations is set, use it. Otherwise tell the cluster to stop all monitoring deployments belonging to this experiment code.

store_workflow_results()

Constructs a list of runs for the planned workflow. Stores this information in self.workload[‘workflow_planned’]. Updates query.config locally and remotely via update_workload().

When the experiment has registered Benchmark objects (via add_benchmark()), the ordered benchmark type sequence is written to self.workload['benchmark_sequence'] so that collectors can later reconstruct the benchmark-run → tool-type mapping from queries.config.

test_results() → None

Run test script locally. Extract exit code.

Returns:

exit code of test script

test_results_in_dashboard() → int

DEPRECATED? Not used currently - depends on good test script for dbmsbenchmarker Run test script in dashboard pod. Extract exit code.

Returns:

exit code of test script

test_workflow(workflow_1: dict, workflow_2: dict) → bool

Compares two workflow dicts for equality.

Each workflow maps a configuration name to a list of experiment runs, where each run is a list of client rounds, and each round is a list of job descriptors {'type': str, 'pods': int}.

The comparison is order-sensitive across experiment runs and client rounds (their sequence matters) but order-insensitive within a single client round (parallel jobs may be scheduled in any order).

Parameters:

  • workflow_1 (dict) – First workflow dict.

  • workflow_2 (dict) – Second workflow dict.

Returns:

True if the two workflows are equivalent.

Return type:

bool

update_workload()

Updates query.config locally and remotely via dashboard pod.

upload_experiment_file(filename: str) → str

Upload a result file to the dashboard pod.

Parameters:

filename – Name of the file relative to the experiment result folder.

Returns:

Output of the upload command, or empty string when no dashboard pod is available.

wait(sec: int, silent: bool = False) → None

Wait for a given number of seconds, optionally without printing.

Parameters:

  • sec – Number of seconds to wait.

  • silent – If True, suppress output during the wait.

work_benchmark_list(intervals: int = 30, stop_after_starting: bool = False, stop_after_loading: bool = False, stop_after_benchmarking: bool = False) → None

Run typical workflow:

  1. start SUT

  2. start monitoring

  3. start loading (at first scripts (schema or loading via pull), then optionally parallel loading pods)

  4. optionally start maintaining pods

  5. at the same time as 4. run benchmarker jobs corresponding to list given via add_benchmark_list()

  6. remove everything when done

Parameters:

  • intervals – Seconds to wait before checking change of status

  • stop_after_starting – stops after phase 2)

  • stop_after_loading – stops after phase 3)

  • stop_after_benchmarking – stops after phase 5) This tells if SUT should not be removed when all benchmarking has finished. Set to True if we want to have loaded SUTs for inspection.

zip()

Zip the result folder in the dashboard pod.

class bexhoma.experiments.base.IotExperiment(cluster, code=None, queryfile='queries-iot.config', SF='1', num_experiment_to_apply=1, timeout=7200)

Bases: ExperimentBase

Class for defining an TSBS experiment. This sets

  • the folder to the experiment - including query file and schema informations per dbms

  • name and information about the experiment

  • additional parameters - here SF (the scaling factor)

set_queries_full()
set_queries_profiling()
set_querymanagement_maintaining(numRun=128, delay=5, datatransfer=False)
class bexhoma.experiments.base.TpcxaiExperiment(cluster, code=None, queryfile='queries-tpcxai.config', SF='100', num_experiment_to_apply=1, timeout=7200, script=None)

Bases: ExperimentBase

Class for defining an TPCx-AI experiment. This sets

  • the folder to the experiment - including query file and schema informations per dbms

  • name and information about the experiment

  • additional parameters - here SF (the scaling factor)

set_queries_full()
set_queries_profiling()
class bexhoma.experiments.base.TsbsExperiment(cluster, code=None, queryfile='queries-tsbs.config', SF='1', num_experiment_to_apply=1, timeout=7200)

Bases: ExperimentBase

Class for defining an TSBS experiment. This sets

  • the folder to the experiment - including query file and schema informations per dbms

  • name and information about the experiment

  • additional parameters - here SF (the scaling factor)

set_queries_full()
set_queries_profiling()
set_querymanagement_maintaining(numRun=128, delay=5, datatransfer=False)
bexhoma.experiments.base.parse_set_arg(s: str) → Tuple[dict, str]

Parse a single --set argument of the form <selector>=<value>.

The selector must match one of:

  • deployment[NAME].container[CONTAINER].PARAM

  • statefulset[NAME].container[CONTAINER].PARAM

Parameters:

s – Raw --set string from the CLI.

Returns:

A tuple of (selector_dict, value_str) where selector_dict contains keys kind, workload, container, and param.

Return type:

tuple[dict, str]

Raises:

ValueError – When the string has no = or the selector does not match.