bexhoma.collectors package

Submodules

Module contents

Public API of the bexhoma.collectors package.

Exports the collector classes (CollectorBase, BenchbaseCollector, DbmsBenchmarkerCollector, MixedCollector, TpccCollector, YcsbCollector) and the utility functions get_non_constant() and map_index_to_queryname().

Backward-compatible aliases (base, benchbase, dbmsbenchmarker, mixed, tpcc, ycsb) are provided so that existing code that imports the old lowercase class names continues to work without modification.

class bexhoma.collectors.BenchbaseCollector(path, codes, benchmark_run: int = 0)

Bases: CollectorBase

Collector for Benchbase experiments.

Overrides get_evaluator() to return a evaluators.benchbase instance. All data collection and aggregation methods are inherited from CollectorBase.

get_benchmark_timeseries_all(metric='throughput')

Collects long-format Benchbase time-series data for a given metric across all experiment codes.

For each code and each unique (configuration, client, experiment_run) combination, calls evaluators.benchbase.get_benchmark_logs_timeseries_df_aggregated(), reshapes the result to long format, and annotates each row with its identifying fields. Connection metadata (e.g. type_tenants, num_tenants, vol_tenants) is joined in from get_connections() for each code.

Parameters:: metric (str) – Benchbase metric to retrieve (default 'throughput').
Returns:: Long-format DataFrame with columns second, code, configuration, client, experiment_run, metric, value, plus connection metadata columns.
Return type:: pandas.DataFrame

get_benchmark_timeseries_per_phase(metric='throughput')

Combines aggregated Benchbase time-series per phase from all experiment codes into a wide-format DataFrame.

For each code and each unique (configuration, client, experiment_run, benchmark_run) combination, calls evaluators.benchbase.get_benchmark_logs_timeseries_df_aggregated() and places the metric column as one column in the result. Each column is labelled {code}-{configuration}-{client}-{experiment_run}-{benchmark_run}.

Parameters:: metric (str) – Benchbase metric to retrieve (default 'throughput').
Returns:: Wide-format DataFrame indexed by second with one column per phase, or an empty DataFrame when no data is available.
Return type:: pandas.DataFrame

get_evaluator(code='')

Returns a evaluators.benchbase instance for the given experiment code.

Parameters:: code (str) – Experiment identifier. Defaults to the first code in self.codes.
Returns:: Benchbase evaluator for the specified experiment.
Return type:: evaluators.benchbase

class bexhoma.collectors.CollectorBase(path, codes)

Bases: object

Base class for collecting and aggregating results from several experiments.

Subclasses override get_evaluator() to return a benchmark-specific evaluator. All data retrieval and aggregation methods are defined here.

Date:: 2025-07-22
Version:: 0.8.10

TEST_get_loading_per_run()

Experimental: computes per-run loading throughput in SF/h from per-connection data.

Groups the per-connection loading result by (code, configuration, experiment_run), takes the column-wise maximum, then derives Throughput [SF/h] as SF * 3600 / time_load. The connection, phase, and client columns are dropped from the result.

Returns:: DataFrame indexed by code-configuration-experiment_run with a Throughput [SF/h] column added.
Return type:: pandas.DataFrame

add_metadata(df)

Joins connection metadata from get_connections() onto a monitoring DataFrame.

Attempts to merge df with the connection metadata using one of several strategies, tried in order:

Index × job column — when df’s index intersects df_connections['job'] (job-level monitoring DataFrames indexed by the code-prefixed 4-part job identifier).
Index × phase column — when df’s index intersects df_connections['phase'] (phase-level timeseries DataFrames indexed by the code-prefixed 3-part phase identifier).
Shared index — when both DataFrames share common index values.
Phase column join — when both DataFrames have a phase column (phase-aggregated DataFrames that carry phase as a regular column).
Multi-tenant key join — when both DataFrames have (code, experiment_run, client, type_tenants, num_tenants) columns.

If none of the strategies match, a warning is printed and an empty DataFrame is returned.

Parameters:: df (pandas.DataFrame) – Monitoring DataFrame to enrich with connection metadata.
Returns:: Enriched DataFrame with connection metadata columns added, or empty on failure.
Return type:: pandas.DataFrame

get_connections(evaluation=None)

Returns connection metadata for all experiments in the collection.

When called without arguments, iterates over self.codes and concatenates the result of get_connections_of_experiment() for each code. When an evaluation is provided, returns only the connections for that experiment.

Key columns in the returned DataFrame: phase (code-prefixed job identifier: <code>-<configuration>-<experiment_run>-<client>-<benchmark_run>), code, connection, configuration, experiment_run, benchmark_run, client, type_tenants, num_tenants, vol_tenants.

Parameters:: evaluation (object, optional) – Evaluator instance. If provided, only that experiment is queried.
Returns:: DataFrame of connection metadata.
Return type:: pandas.DataFrame

get_evaluator(code='')

Returns a benchmarking evaluator for the given experiment code.

Subclasses override this method to return the appropriate evaluator type. The base implementation returns a evaluators.dbmsbenchmarker instance.

Parameters:: code (str) – Experiment identifier. Defaults to the first code in self.codes.
Returns:: Evaluator instance for the specified experiment.
Return type:: evaluators.dbmsbenchmarker

get_loading_per_connection()

Combines loading metrics per connection from all experiment codes into one DataFrame.

Concatenates the per-connection loading DataFrames returned by each evaluator.

Returns:: Combined DataFrame of per-connection loading metrics for all experiments.
Return type:: pandas.DataFrame

get_loading_per_pod()

Combines loading metrics per pod from all experiment codes into one DataFrame.

Concatenates the per-pod loading DataFrames returned by each evaluator. The connection, phase, and client columns are dropped from the result because they are not meaningful across experiments.

Returns:: Combined DataFrame of per-pod loading metrics for all experiments.
Return type:: pandas.DataFrame

get_loading_per_run()

Combines loading metrics per run from all experiment codes into one DataFrame.

Concatenates the per-run loading DataFrames returned by each evaluator.

Returns:: Combined DataFrame of per-run loading metrics for all experiments.
Return type:: pandas.DataFrame

get_loading_per_run_multitenant()

Combines multi-tenant loading metrics per run from all experiment codes into one DataFrame.

Concatenates the per-run multi-tenant loading DataFrames returned by each evaluator.

Returns:: Combined DataFrame of per-run multi-tenant loading metrics for all experiments.
Return type:: pandas.DataFrame

get_metrics(evaluation=None)

Returns metadata for the hardware metrics collected during the experiment.

Reads connections.config for the given evaluation and extracts the monitoring metric definitions. The returned DataFrame has one row per metric and the following columns:

title: human-readable metric name,
active: whether the metric was active (defaults to 'True'),
type: metric category, e.g. 'cluster' (default),
metric: raw metric identifier.

Sets self.with_monitoring = False and returns an empty DataFrame when no 'metrics' key is found in the first connection’s monitoring block.

Parameters:: evaluation (object) – Evaluator instance. Defaults to the first code’s evaluator.
Returns:: DataFrame of metric metadata indexed by metric key.
Return type:: pandas.DataFrame

get_metrics_metadata()

Returns the metrics metadata DataFrame built during initialisation.

Returns:: DataFrame listing monitored hardware metrics and their metadata.
Return type:: pandas.DataFrame

get_monitored_components(code='')

Returns a DataFrame of monitored components defined in the workload configuration.

Parameters:: code (str) – Experiment identifier. Defaults to the first code in self.codes.
Returns:: DataFrame indexed by component key with a description column, or an empty DataFrame when monitoring is disabled.
Return type:: pandas.DataFrame

get_monitoring_aggregated_per_job(type='stream')

Combines aggregated monitoring metrics from all experiment codes into one DataFrame.

For each code, calls show_summary_monitoring_table() and concatenates the results. The connection column is dropped because it is no longer meaningful across experiments.

The result index is the code-prefixed job identifier (<code>-<configuration>-<experiment_run>-<client>-<benchmark_run>), producing one row per benchmark job.

Parameters:: type (str) – Component type to filter monitoring metrics (default 'stream').
Returns:: Combined DataFrame of aggregated monitoring metrics, one row per job, or an empty DataFrame when monitoring is disabled.
Return type:: pandas.DataFrame

get_monitoring_aggregated_per_phase(type='stream')

Combines aggregated monitoring metrics per phase from all experiment codes.

Builds on get_monitoring_aggregated_per_job() and further aggregates by phase (configuration-experiment_run-client), collapsing parallel benchmark jobs within the same phase into a single row.

Aggregation per metric type: ratio → max, counter → sum, others → mean. 'Total I/O Wait Time [s]' is always reduced with max.

The result index is the code-prefixed phase identifier (<code>-<configuration>-<experiment_run>-<client>).

Parameters:: type (str) – Component type to filter monitoring metrics (default 'stream').
Returns:: Combined DataFrame of aggregated monitoring metrics, one row per phase, or an empty DataFrame when monitoring is disabled.
Return type:: pandas.DataFrame

get_monitoring_aggregated_per_phase_multitenant(type='stream')

Combines aggregated multi-tenant monitoring metrics from all experiment codes into one DataFrame.

Calls get_monitoring_aggregated_per_job() to collect the per-job monitoring data, enriches it with connection metadata via add_metadata(), then groups by (code, experiment_run, client, type_tenants, num_tenants) and reduces each metric column using 'max' for ratio metrics and 'sum' for counter metrics. 'Total I/O Wait Time [s]' is always reduced with 'max'.

Parameters:: type (str) – Component type forwarded to get_monitoring_aggregated_per_job().
Returns:: DataFrame of grouped multi-tenant monitoring metrics indexed by the underscore-joined group key.
Return type:: pandas.DataFrame

get_monitoring_timeseries_all(metric='pg_locks_count', component='stream')

Collects long-format time-series data for a given metric across all experiment codes.

For each code, fetches the wide-format time series, melts it to long format, merges connection metadata, and concatenates the results. The final DataFrame is grouped by (timestamp, code, phase, experiment_run, client, type_tenants, vol_tenants, num_tenants, metric, component) and summed.

Parameters:

metric (str) – Monitoring metric name (default 'pg_locks_count').
component (str) – Component name used as a label column (default 'stream').

Returns:

Grouped long-format time-series DataFrame, or empty if monitoring is disabled.

Return type:

pandas.DataFrame

get_monitoring_timeseries_all_multitenant(metric='pg_locks_count', component='stream')

Collects long-format multi-tenant time-series data for a given metric across all experiment codes.

Behaves like get_monitoring_timeseries_all() but additionally annotates each row with tenant metadata (type_tenants, num_tenants, vol_tenants) from the workload configuration. For non-container tenancy, the tenant_id column is set to "0" to indicate a single shared DBMS.

The final DataFrame is grouped by (timestamp, code, experiment_run, client, type_tenants, vol_tenants, num_tenants, metric, component) and summed.

Parameters:

metric (str) – Monitoring metric name (default 'pg_locks_count').
component (str) – Component name used as a label column (default 'stream').

Returns:

Grouped long-format time-series DataFrame, or empty if monitoring is disabled.

Return type:

pandas.DataFrame

get_monitoring_timeseries_per_phase(code, metric='pg_locks_count', component='stream')

Returns a single monitoring metric as a transposed time-series DataFrame.

Compared to get_monitoring_timeseries_single(), the result is transposed so that rows are monitored component instances and columns are timestamps.

Parameters:

code (str) – Experiment identifier.
metric (str) – Monitoring metric name (default 'pg_locks_count').
component (str) – Component name to filter metrics (default 'stream').

Returns:

Transposed time-series DataFrame, or empty if monitoring is disabled.

Return type:

pandas.DataFrame

get_monitoring_timeseries_single(code, metric='pg_locks_count', component='stream')

Returns a single monitoring metric as a wide-format time-series DataFrame.

Rows are timestamps; columns are monitored component instances (e.g. pods).

Parameters:

code (str) – Experiment identifier.
metric (str) – Monitoring metric name (default 'pg_locks_count').
component (str) – Component name to filter metrics (default 'stream').

Returns:

Wide-format time-series DataFrame, or empty if monitoring is disabled.

Return type:

pandas.DataFrame

get_performance_aggregated_per_job(type='stream')

Combines aggregated performance results per job from all experiment codes.

Groups by the job column (configuration-experiment_run-client-benchmark_run), keeping parallel benchmark jobs within the same phase as separate rows. Both phase (code-prefixed phase identifier) and job (code-prefixed job identifier) are included in the output.

The result index is the code-prefixed job identifier (<code>-<configuration>-<experiment_run>-<client>-<benchmark_run>).

Parameters:: type (str) – Component type passed to the aggregation call (currently unused).
Returns:: Combined DataFrame of aggregated performance metrics, one row per job.
Return type:: pandas.DataFrame

get_performance_aggregated_per_phase(type='stream')

Combines aggregated performance results per phase from all experiment codes.

Groups by the phase column (configuration-experiment_run-client), aggregating all parallel benchmark jobs within the same phase into a single row. The job column is dropped from the output. The result index is the code-prefixed phase identifier (<code>-<configuration>-<experiment_run>-<client>).

Parameters:: type (str) – Component type passed to the aggregation call (currently unused).
Returns:: Combined DataFrame of aggregated performance metrics, one row per phase.
Return type:: pandas.DataFrame

get_performance_aggregated_per_phase_multitenant(type='stream')

Combines aggregated multi-tenant performance results from all experiment codes into one DataFrame.

Extends get_performance_aggregated_per_phase() by annotating each row with tenant metadata (type_tenants, num_tenants, vol_tenants) read from the workload configuration before aggregation. Groups by (code, experiment_run, client, benchmark_run, type_tenants, num_tenants, tenant_id), producing one row per tenant per job.

tenant_id is added to the per-connection DataFrame by benchmarking_set_datatypes() (default -1 for experiments without explicit tenant assignment).

Parameters:: type (str) – Component type passed to the aggregation call (currently unused in this method).
Returns:: Combined DataFrame of aggregated multi-tenant performance metrics.
Return type:: pandas.DataFrame

get_performance_per_connection()

Combines unaggregated performance results per connection from all experiment codes.

For each code, retrieves the per-client performance data and prefixes the phase, connection, configuration, and DataFrame index with the experiment code before concatenating.

Returns:: Combined DataFrame of unaggregated performance metrics for all experiments.
Return type:: pandas.DataFrame

get_performance_single(evaluation=None)

Returns unaggregated benchmarking performance metrics per client.

Retrieves the benchmarking DataFrame from the evaluator, sorts it by experiment run and client, and returns the result without further aggregation.

Parameters:: evaluation (object) – Evaluator instance. Defaults to the first code’s evaluator.
Returns:: DataFrame of per-client performance metrics sorted by code, experiment_run, and client.
Return type:: pandas.DataFrame

get_workload(code='')

Returns the workload configuration of an experiment.

Reads the queries.config file for the given experiment code and returns its contents as a dictionary. The tenant_per key is normalised to the string 'None' when absent or empty.

Parameters:: code (str) – Experiment identifier. Defaults to the first code in self.codes.
Returns:: Workload properties dictionary.
Return type:: dict

show_summary_monitoring_table(evaluation, type='stream')

Collects all active monitoring metrics for a given component without aggregation.

Iterates over self.df_metrics, skipping inactive metrics, and fetches the time-series data for each metric via the evaluator. Counter metrics are reduced by max - min; ratio metrics by max; all others by mean. Results are combined column-wise into a summary DataFrame.

Parameters:

evaluation (object) – Evaluator instance providing the monitoring data.
type (str) – Component name to retrieve metrics for (e.g. 'stream', 'database').

Returns:

Summary DataFrame of monitoring metrics rounded to two decimal places, or an empty DataFrame when monitoring is disabled.

Return type:

pandas.DataFrame

class bexhoma.collectors.DbmsBenchmarkerCollector(path, codes, benchmark_run: int = 0)

Bases: CollectorBase

Collector for DBMSBenchmarker experiments.

Extends CollectorBase with query-level aggregation methods for warnings, errors, and latencies. Overrides get_evaluator() to return a evaluators.dbmsbenchmarker instance.

get_evaluator(code='')

Returns a evaluators.dbmsbenchmarker instance for the given experiment code.

Parameters:: code (str) – Experiment identifier. Defaults to the first code in self.codes.
Returns:: DBMSBenchmarker evaluator for the specified experiment.
Return type:: evaluators.dbmsbenchmarker

get_query_latencies(query_titles=False)

Aggregates query latency metrics across all experiment codes.

For each code, retrieves the per-query latency DataFrame from the evaluator and prefixes its index with the experiment code before concatenating.

Parameters:: query_titles (bool) – If True, use human-readable query titles as index labels.
Returns:: A combined DataFrame of query latencies for all experiments.
Return type:: pandas.DataFrame

get_total_errors(query_titles=False)

Aggregates error counts (failed queries) across all experiment codes.

For each code, retrieves the per-query error DataFrame from the evaluator and prefixes its index with the experiment code before concatenating.

Note

The current implementation delegates to evaluation.get_total_warnings.

Parameters:: query_titles (bool) – If True, use human-readable query titles as index labels.
Returns:: A combined DataFrame of error counts for all experiments.
Return type:: pandas.DataFrame

get_total_warnings(query_titles=False)

Aggregates warning counts (result mismatches) across all experiment codes.

For each code, retrieves the per-query warning DataFrame from the evaluator and prefixes its index with the experiment code before concatenating.

Parameters:: query_titles (bool) – If True, use human-readable query titles as index labels.
Returns:: A combined DataFrame of warning counts for all experiments.
Return type:: pandas.DataFrame

class bexhoma.collectors.MixedCollector(path: str, codes: list[str])

Bases: CollectorBase

Collector for experiments that contain multiple benchmark types.

Reads the benchmark_sequence key from queries.config — written by store_workflow_results() — to determine which benchmark tool ran at each benchmark_run index. Exposes one typed sub-collector per benchmark-run index via get_typed_collector(), and a convenience method get_performance_per_benchmark() that returns a dict of aggregated-performance DataFrames keyed by benchmark type name.

Parameters:

path – Base filesystem path that contains the experiment directories.
codes – List of experiment codes to collect results for.

get_performance_per_benchmark() → dict

Returns aggregated performance results separated by benchmark type.

Iterates over benchmark_sequence, creates a typed sub-collector for each entry via get_typed_collector(), and calls get_performance_aggregated_per_phase() on it.

Returns:: Dict mapping benchmark type name to its aggregated-performance DataFrame. When two entries share the same type (parallel runs of the same tool), both DataFrames are concatenated under that key.
Return type:: dict[str, pandas.DataFrame]

get_typed_collector(benchmark_run: int) → CollectorBase

Returns a typed sub-collector scoped to a single benchmark-run index.

The collector class is resolved from _COLLECTOR_FOR_TYPE using the 'type' field of the matching entry in benchmark_sequence.

Parameters:

benchmark_run (int) – 1-based benchmark-run index.

Returns:

Typed collector instance filtered to benchmark_run.

Return type:

CollectorBase

Raises:

StopIteration – If benchmark_run is not found in benchmark_sequence.
KeyError – If the benchmark type is not registered in _COLLECTOR_FOR_TYPE.

class bexhoma.collectors.TpccCollector(path, codes, benchmark_run: int = 0)

Bases: CollectorBase

Collector for HammerDB TPC-C experiments.

Overrides get_evaluator() to return a evaluators.tpcc instance. All data collection and aggregation methods are inherited from CollectorBase.

get_evaluator(code='')

Returns a evaluators.tpcc instance for the given experiment code.

Parameters:: code (str) – Experiment identifier. Defaults to the first code in self.codes.
Returns:: TPC-C evaluator for the specified experiment.
Return type:: evaluators.tpcc

class bexhoma.collectors.YcsbCollector(path, codes, benchmark_run: int = 0)

Bases: CollectorBase

Collector for YCSB experiments.

Overrides get_evaluator() to return a evaluators.ycsb instance. All data collection and aggregation methods are inherited from CollectorBase.

get_benchmark_timeseries_all(metric='current_ops_per_sec')

Collects long-format YCSB time-series data for a given metric across all experiment codes.

For each code and each unique (configuration, client, experiment_run) combination, calls evaluators.ycsb.get_benchmark_logs_timeseries_df_aggregated(), reshapes the result to long format, and annotates each row with its identifying fields. Connection metadata (e.g. type_tenants, num_tenants, vol_tenants) is joined in from get_connections() for each code.

The YCSB evaluator uses 'sec' as the index name; this method normalises it to 'second' for consistency with other time-series methods.

Parameters:: metric (str) – YCSB metric to retrieve (default 'current_ops_per_sec').
Returns:: Long-format DataFrame with columns second, code, configuration, client, experiment_run, metric, value, plus connection metadata columns.
Return type:: pandas.DataFrame

get_benchmark_timeseries_per_phase(metric='current_ops_per_sec')

Combines aggregated YCSB time-series per phase from all experiment codes into a wide-format DataFrame.

For each code and each unique (configuration, client, experiment_run, benchmark_run) combination, calls evaluators.ycsb.get_benchmark_logs_timeseries_df_aggregated() and places the metric column as one column in the result. Each column is labelled {code}-{configuration}-{client}-{experiment_run}-{benchmark_run}.

Parameters:: metric (str) – YCSB metric to retrieve (default 'current_ops_per_sec').
Returns:: Wide-format DataFrame indexed by second with one column per phase, or an empty DataFrame when no data is available.
Return type:: pandas.DataFrame

get_evaluator(code='')

Returns a evaluators.ycsb instance for the given experiment code.

Parameters:: code (str) – Experiment identifier. Defaults to the first code in self.codes.
Returns:: YCSB evaluator for the specified experiment.
Return type:: evaluators.ycsb

get_loading_timeseries_all(metric='current_ops_per_sec')

Collects long-format YCSB loading time-series data for a given metric across all experiment codes.

For each code and each unique (configuration, experiment_run) combination, calls evaluators.ycsb.get_loading_logs_timeseries_df_aggregated(), reshapes the result to long format, and annotates each row with its identifying fields. Connection metadata (e.g. type_tenants, num_tenants, vol_tenants) is joined in from get_connections() for each code.

Unlike the benchmarking variant, the loading phase has no client dimension.

The YCSB evaluator uses 'sec' as the index name; this method normalises it to 'second' for consistency with other time-series methods.

Parameters:: metric (str) – YCSB metric to retrieve (default 'current_ops_per_sec').
Returns:: Long-format DataFrame with columns second, code, configuration, experiment_run, metric, value, plus connection metadata columns.
Return type:: pandas.DataFrame

get_loading_timeseries_per_phase(metric='current_ops_per_sec')

Combines aggregated YCSB loading time-series per phase from all experiment codes into a wide-format DataFrame.

For each code and each unique (configuration, experiment_run) combination, calls evaluators.ycsb.get_loading_logs_timeseries_df_aggregated() and places the metric column as one column in the result. Each column is labelled {code}-{configuration}-{experiment_run}.

Unlike the benchmarking variant, the loading phase has no client dimension.

Parameters:: metric (str) – YCSB metric to retrieve (default 'current_ops_per_sec').
Returns:: Wide-format DataFrame indexed by second with one column per phase, or an empty DataFrame when no data is available.
Return type:: pandas.DataFrame

bexhoma.collectors.base: alias of CollectorBase

bexhoma.collectors.benchbase: alias of BenchbaseCollector

bexhoma.collectors.dbmsbenchmarker: alias of DbmsBenchmarkerCollector

bexhoma.collectors.get_non_constant(df)

Filters a DataFrame to keep only columns whose values vary across rows.

Parameters:: df (pandas.DataFrame) – Input DataFrame.
Returns:: DataFrame containing only non-constant columns.
Return type:: pandas.DataFrame

bexhoma.collectors.map_index_to_queryname(numQuery)

Maps a query index string (e.g., 'q1') to a human-readable title from the global query_properties dictionary.

If the title cannot be resolved, the original input string is returned unchanged.

Parameters:: numQuery (str) – A query index string, typically a letter followed by a number (e.g., 'q1').
Returns:: The query title from query_properties, or numQuery if not found.
Return type:: str

bexhoma.collectors.mixed: alias of MixedCollector

bexhoma.collectors.tpcc: alias of TpccCollector

bexhoma.collectors.ycsb: alias of YcsbCollector