bexhoma.evaluators.base module

Base evaluator class for bexhoma experiments.

Provides natural_sort() and EvaluatorBase, which loads an experiment result folder, parses workflow state and connection configuration, and exposes monitoring data. All other evaluators inherit from EvaluatorBase via LogEvaluator.

class bexhoma.evaluators.base.EvaluatorBase(code, path, include_loading=False, include_benchmarking=True, benchmark_run: int = 0, name: str = '')

Bases: object

Base evaluator for a single bexhoma experiment.

Loads the experiment result folder identified by code inside path, provides helpers for scanning log files and reconstructing the workflow, and exposes connection/loading metadata. All benchmark-specific evaluators inherit from this class (via logger).

Parameters:

code – Experiment identifier — also the name of the result sub-folder.
path – Root path that contains the result folders.
include_loading – Whether loading-phase results are expected.
include_benchmarking – Whether benchmarking-phase results are expected.
benchmark_run – 1-based position in the benchmark sequence; 0 means unset.
name – Short identifier matching the "name" field of this benchmark’s experiment dict entries; empty means unset.

add_connection_to_result(c, connection_id, result)

Appends a flattened connection entry to result keyed by connection_id.

Extracts scalar fields from the connection config dict c — including host-system, loading-parameter, benchmarking-parameter, SUT-parameter, and args sections — and stores them with prefixed keys (host_*, loading_parameters_*, benchmarking_parameters_*, sut_parameters_*, arg_*). Non-scalar values (lists and dicts) are skipped.

Parameters:

c (dict) – Single connection entry from connections.config.
connection_id (str) – Key to use when inserting into result.
result (dict) – Accumulator dict that maps connection IDs to metadata rows.

cleanup_connection_subfolders() → None: No-op base implementation; overridden by DbmsBenchmarkerEvaluator to delete each connection’s private DBMSBenchmarker working-directory subfolder (only that evaluator’s benchmarker tool creates one).

end_benchmarking(jobname)

Processes all benchmarker log files for a given job name.

Scans the result folder for files matching bexhoma-benchmarker-<jobname>*.dbmsbenchmarker.log and calls log_to_df() on each one.

Parameters:: jobname (str) – Job name used to filter matching log files.

end_loading(jobname)

Processes all loader sensor log files for a given job name.

Scans the result folder for files matching bexhoma-loading-<jobname>*.sensor.log and calls log_to_df() on each one.

Parameters:: jobname (str) – Job name used to filter matching log files.

evaluate_results(pod_dashboard=''): Scans all log files and grabs some information. In this class basically it scans for errors.

get_benchmark_name_at(configuration: str, client: int, benchmark_run: int)

Resolve the name field of the experiment dict entry at (configuration, client, benchmark_run).

Parameters:

configuration (str) – SUT configuration name.
client (int) – 1-based client round number.
benchmark_run (int) – 1-based position within that round.

Returns:

The entry’s name, or None if the experiment dict is unavailable or the indices are out of range.

Return type:

str or None

get_connections_of_experiment()

Returns connection metadata for a single experiment.

Reads connections.config and builds a row per pod/client with the following key columns: phase (code-prefixed phase identifier, <code>-<configuration>-<experiment_run>-<client>), job (code-prefixed job identifier, <code>-<configuration>-<experiment_run>-<client>-<benchmark_run>), code, connection, configuration, experiment_run, client, type_tenants, num_tenants, vol_tenants, plus flattened host-system, loading-parameter, benchmarking-parameter, and SUT-parameter fields.

When a connection entry carries orig_name, the entry represents an individual pod; otherwise a synthetic row is generated for each parallel client.

Returns:: DataFrame of connection metadata, one row per pod/client.
Return type:: pandas.DataFrame

get_df_benchmarking()

Returns the DataFrame containing all benchmarking-phase results.

Returns:: Empty DataFrame; overridden by subclasses.
Return type:: pandas.DataFrame

get_df_loading()

Returns the DataFrame containing all loading-phase results.

Returns:: Empty DataFrame; overridden by subclasses.
Return type:: pandas.DataFrame

get_loading_per_connection()

Returns loading metrics for each individual connection (pod/client), enriched with the scale factor and a 'Throughput [SF/h]' derived column.

The scale factor is read per connection from that connection’s own loading_parameters_SF (set via set_default_loading_parameters(SF=...) when its loader job was submitted), falling back to the experiment-wide queries.config defaultParameters.SF only for connections that carry no loading_parameters_SF of their own (e.g. benchmark types with no loading phase). This keeps Throughput [SF/h] correct even in a mixed experiment where two co-running benchmarks (e.g. TPC-H and YCSB, both registered via add_benchmark()) use different scale factors – the single experiment-wide default can only ever reflect one of them.

Returns:: DataFrame with one row per connection.
Return type:: pandas.DataFrame

get_loading_per_run()

Returns loading metrics aggregated per (code, configuration, experiment_run).

Takes the per-connection DataFrame from get_loading_per_connection() and reduces it to one row per experiment run by taking the max across connections, then recomputes 'Throughput [SF/h]' from the aggregated load time.

Returns:: DataFrame with one row per experiment run.
Return type:: pandas.DataFrame

get_loading_per_run_multitenant()

Returns loading metrics aggregated per (code, experiment_run, type_tenants, vol_tenants, num_tenants, tenant_id) for multi-tenant experiments.

For container tenancy the tenant_id key distinguishes individual tenant loading times.

For schema/database tenancy, reads per-pod sensor log files via _get_tenant_loading_from_logs() to expand the single shared connection row into one row per tenant. Each row carries the tenant’s own loading duration (time_ingest / time_load) and a matching tenant_id. If the sensor logs are absent the result collapses to one row with tenant_id = '' (same as before).

Returns:: DataFrame with one row per tenant per experiment run.
Return type:: pandas.DataFrame

get_summary_loading_per_run_multitenant()

Returns loading metrics per tenant per experiment run, with housekeeping columns removed.

Wraps get_loading_per_run_multitenant() and drops code and configuration so the result is ready to display in show_summary().

Returns:: DataFrame with one row per (tenant_id, experiment_run) combination.
Return type:: pandas.DataFrame

get_terminals(loading_parameters)

Returns the number of client terminals/virtual users a benchmarking tool ran with, read from its own loading-parameter env var.

The base implementation knows no tool-specific env var and always returns 0; subclasses for tools that report a terminal/VU count (e.g. Benchbase, HammerDB) override this.

Parameters:: loading_parameters (dict) – The connection’s loading_parameters dict.
Returns:: Terminal/VU count, or 0 when the tool does not report one.
Return type:: int

get_workload()

Returns the workload configuration of an experiment.

Reads the queries.config file from the experiment result folder and returns its contents as a Python dictionary.

Returns:: Workload properties dictionary.
Return type:: dict

is_own_benchmark(configuration: str, client: int, benchmark_run: int) → bool

Decide whether a connection identified by (configuration, client, benchmark_run) belongs to this evaluator’s own benchmark.

Looks up experiment_dict['benchmarker'][client - 1][benchmark_run - 1]['name'] from the persisted per-configuration experiment dict (see _load_experiment_dict()) and compares it to name. This is robust to benchmark_run denoting a different benchmark type in different rounds (e.g. a benchmark that runs alone in one round and alongside another benchmark in a different round) — unlike a raw benchmark_run == self.benchmark_run comparison, which assumes every round has the same entries in the same order.

Falls back to benchmark_run == self.benchmark_run when name is unset or the experiment dict lookup fails (e.g. offline results predating this mechanism), preserving the prior behaviour.

Parameters:

configuration (str) – SUT configuration name.
client (int) – 1-based client round number.
benchmark_run (int) – 1-based position within that round.

Returns:

Whether this connection belongs to this evaluator’s benchmark.

Return type:

bool

is_own_loading(configuration: str, data_job: int) → bool

Decide whether a loader connection identified by (configuration, data_job) belongs to this evaluator’s own benchmark.

Looks up experiment_dict['loader'][data_job - 1] from the persisted per-configuration experiment dict (see _load_experiment_dict()) and matches it against name. Unlike benchmarker entries (whose name directly matches the owning benchmark’s own name, e.g. 'ycsb'), loader entries name themselves after the experiment type with a conventional '-loader' suffix (e.g. 'tpcc-loader'), which does not always equal the benchmark’s name (e.g. TPC-C’s own benchmark is named 'hammerdb', see TPCC). So a match is accepted either via the entry’s name with that suffix stripped, or via its benchmarker field (the tool identifier, e.g. 'hammerdb').

Falls back to data_job == self.benchmark_run when name is unset or the experiment dict lookup fails, preserving the prior behaviour.

Parameters:

configuration (str) – SUT configuration name.
data_job (int) – 1-based position of the loader entry (BEXHOMA_DATA_JOB).

Returns:

Whether this connection belongs to this evaluator’s benchmark.

Return type:

bool

log_to_df(filename)

Scans a pod log file for known errors and records them in self.workflow_errors.

Returns an empty DataFrame; subclasses override this method to also parse benchmark results out of the log.

Parameters:: filename (str) – Absolute path to the log file.
Returns:: Empty DataFrame (subclasses return populated DataFrames).
Return type:: pandas.DataFrame

log_to_df_loading(filename: str) → DataFrame

Parse a loading pod log file and return the result as a DataFrame.

Default implementation delegates to log_to_df(), which is correct for benchmarks whose loading and benchmarking log formats are identical (e.g. YCSB). Subclasses where the formats differ must override this method.

Parameters:: filename (str) – Absolute path to the loading log file.
Returns:: DataFrame of loading results, or empty DataFrame on failure.
Return type:: pandas.DataFrame

reconstruct_workflow(df: DataFrame) → dict

Reconstructs the actual experiment workflow from connection metadata.

Resolves each (configuration, client, benchmark_run) to its benchmarker type via get_benchmark_name_at() (falling back to benchmark_sequence from queries.config when the persisted experiment dict is unavailable), then groups the DataFrame by (configuration, experiment_run, client, benchmark_run) to produce a structure that mirrors the planned workflow format:

{
    'MySQL-24-4-1024': [
        [  # experiment run 1
            [  # client round 1
                {'type': 'dbmsbenchmarker', 'pods': 4},
                {'type': 'tpch_refresh',    'pods': 1},
            ],
            [  # client round 2
                {'type': 'dbmsbenchmarker', 'pods': 8},
                {'type': 'tpch_refresh',    'pods': 1},
            ],
        ],
    ],
}

Parameters:: df (pandas.DataFrame) – Connection metadata DataFrame returned by get_connections_of_experiment(), with at least the columns configuration, experiment_run, client, benchmark_run, and pods.
Returns:: Workflow dict mapping configuration name to the nested structure.
Return type:: dict

test_results()

Validates results locally and returns an exit code.

Returns:: 0 on success; subclasses return 1 on failure.
Return type:: int

test_results_column(df, test_column: str) → bool

Check whether a column in a DataFrame contains any zero or NaN values.

Parameters:

df (pandas.DataFrame) – DataFrame to check.
test_column (str) – Column name to inspect.

Returns:

True if the column is fully populated with non-zero values, False otherwise.

Return type:

bool

transform_all_logs_benchmarking()

Iterates over all benchmarker log files and calls end_benchmarking() for each.

Only processes log files that is_own_benchmark() identifies as belonging to this evaluator’s own benchmark (resolved via the persisted experiment dict, or by raw benchmark_run position when that lookup is unavailable), so that each evaluator in a multi-benchmark experiment only ingests its own logs.

transform_all_logs_loading()

Iterates over all loader sensor log files and calls end_loading() for each.

Only processes log files that is_own_loading() identifies as belonging to this evaluator’s own benchmark (resolved via the persisted experiment dict, or by raw benchmark_run position when that lookup is unavailable), so that each evaluator only ingests its own loading logs.

bexhoma.evaluators.base.natural_sort(items)

Sorts a list in natural (human) order so that embedded digit runs are compared numerically rather than lexicographically. Works for lists of strings, integers, or any mix whose elements have a meaningful str() representation.

Parameters:: items (list) – List to sort.
Returns:: Sorted list.
Return type:: list

bexhoma.evaluators.base.resolve_within_result_folder(root: str, name: str) → str | None

Resolve name as a path inside root, refusing anything that escapes it.

Call this before deleting a file or folder whose name is sourced from experiment metadata (e.g. a connection name read from connections.config) rather than hardcoded, so a crafted or unexpected value (.. segments, an absolute path) can never resolve to a location outside the experiment result folder.

Parameters:

root (str) – Directory the resolved path must stay inside (normally self.path, the experiment result folder).
name (str) – Untrusted relative file or folder name to resolve under root.

Returns:

The resolved absolute path, or None if it would escape root or equal root itself.

Return type:

str | None