bexhoma.evaluators.logger module

Logger evaluator: reads benchmark log files into DataFrames.

Provides LogEvaluator, which extends EvaluatorBase by parsing bexhoma benchmarker log files produced by Kubernetes pods and transforming them into structured pandas DataFrames. All benchmark-specific evaluators inherit from LogEvaluator.

class bexhoma.evaluators.logger.LogEvaluator(code, path, include_loading=False, include_benchmarking=True, benchmark_run: int = 0, name: str = '')

Bases: EvaluatorBase

Evaluator base that reads benchmark log files into DataFrames.

Extends base by implementing end_benchmarking() and end_loading() to parse pod log files, pickle the resulting DataFrames, and collect them into a single combined pickle per phase. All benchmark-specific evaluators (benchbase, ycsb, tpcc, dbmsbenchmarker) inherit from this class.

end_benchmarking(jobname)

Parses all benchmarker log files for a job and caches results as pickle files.

Scans the result folder for files matching bexhoma-benchmarker-<jobname>*.dbmsbenchmarker.log, calls log_to_df() on each, and writes non-empty results to a <filename>.df.pickle side-car file.

Parameters:: jobname (str) – Job name used to filter matching log files.

end_loading(jobname)

Parses all loader sensor log files for a job and caches results as pickle files.

Scans the result folder for files matching bexhoma-loading-<jobname>*.sensor.log, calls log_to_df() on each, prints a message when errors are detected, and writes non-empty results to a <filename>.df.pickle side-car file.

Parameters:: jobname (str) – Job name used to filter matching log files.

evaluate_results(pod_dashboard='')

Parses all pod log files and persists the results as pickled DataFrames.

Calls transform_all_logs_benchmarking() and _collect_dfs() for the benchmarking phase when include_benchmarking is set, and analogously for the loading phase. When benchmark_run > 0, each phase writes a per-benchmark pickle file rather than the shared *.all.df.pickle.

get_connection_config()

Returns the parsed connections.config as a list of connection dicts, sorted by connection name.

Returns:: List of connection configuration dicts.
Return type:: list[dict]

get_df_benchmarking()

Returns the DataFrame containing all benchmarking-phase results.

Reads from the combined pickle file, triggering evaluate_results() to generate it on first access if it does not yet exist.

Ensures a pods column is present: when the pickle was written before this column was added (older runs), it is derived from pod_count.

Returns:: DataFrame of benchmarking results, or empty DataFrame when unavailable.
Return type:: pandas.DataFrame

get_df_loading()

Returns the DataFrame containing all loading-phase results.

Reads from the combined pickle file if it exists.

Returns:: DataFrame of loading results, or empty DataFrame when unavailable.
Return type:: pandas.DataFrame

get_monitoring_metric(metric, component='loading')

Returns a wide-format DataFrame of a single monitoring metric for a component.

Reads the pre-combined CSV produced by transform_monitoring_results() and returns it transposed so that rows are timestamps and columns are connections.

Parameters:

metric (str) – Metric key (e.g. 'cpu_throttled_seconds_total').
component (str) – Component label used in the metric filename prefix.

Returns:

Wide-format DataFrame, or empty DataFrame if the file does not exist.

Return type:

pandas.DataFrame

get_monitoring_metrics()

Returns the list of metric keys defined in the first connection’s monitoring block.

Returns:: List of metric key strings, or empty list when no metrics are configured.
Return type:: list[str]

plot(df, column, x, y, plot_by=None, kind='line', dict_colors=None, figsize=(12, 8))

Plots one or more line (or other) charts from a DataFrame.

When plot_by is None, a single chart is produced with one line per value in column. When plot_by is given, a grid of sub-plots is created — one per group defined by plot_by — with lines split by column within each sub-plot.

Parameters:

df (pandas.DataFrame) – DataFrame containing the data to plot.
column (str) – Column whose unique values define individual lines.
x (str) – Column to use as the x-axis.
y (str) – Column to use as the y-axis.
plot_by (str or None) – Optional column whose values define separate sub-plots.
kind (str) – Plot kind passed to DataFrame.plot (e.g. 'line', 'bar').
dict_colors (dict or None) – Optional colour mapping for the kind keyword.
figsize (tuple) – Figure size as (width, height) in inches.

Returns:

Matplotlib axes object (single axes when plot_by is None).

record_tests(experiment, df_loading: DataFrame, df_reduced: DataFrame, workflow_actual: dict, workflow_planned: dict, **extra) → None

Record pass/fail test results for this benchmark.

Default: tests only that the workflow matches the plan. Override in benchmark-specific evaluator subclasses to test metric columns.

Parameters:

experiment – The owning experiment object.
df_loading – Per-run loading DataFrame.
df_reduced – Per-phase execution DataFrame.
workflow_actual – Reconstructed actual workflow dict.
workflow_planned – Planned workflow dict from workload config.

test_results()

Validates results by loading and reconstructing the workflow.

Reconstructs the workflow from get_connections_of_experiment() rather than get_df_benchmarking(): the latter is filtered to only this evaluator’s own benchmark (via is_own_benchmark()), so for a benchmark co-running alongside another one in the same client round, it would never match the full planned workflow even when everything succeeded. get_connections_of_experiment() is unfiltered, matching what get_workflow_list() plans for the whole round.

Returns:: 0 on success, 1 if an exception is raised.
Return type:: int

transform_monitoring_results(component='loading')

Combines per-connection monitoring CSV files into a single wide-format CSV, one column per SUT.

For example, per-connection files like:

query_datagenerator_metric_total_cpu_util_MonetDB-NIL-1-1.csv
query_datagenerator_metric_total_cpu_util_MonetDB-NIL-1-2.csv

are merged into:

query_datagenerator_metric_total_cpu_util.csv

A connection’s identity is its orig_name when it has one, its own name otherwise. What matters here: once an identity has been merged in, any other connection resolving to the same identity is skipped, instead of being merged in again. Without that guard, several connections sharing one orig_name (how DBMSBenchmarker represents parallel loaders/clients against a single physical SUT) each load and merge the same file under the same column name, and pandas’ merge silently renames the second one to _x/_y instead of erroring, corrupting the combined metric. If the job-level file named after the identity doesn’t exist (some components, e.g. loading, only ever write that one, before benchmark.py read has expanded connections.config into per-pod entries; others only write per-pod files), a connection’s own per-pod file is tried as a fallback and stands in for the whole identity.

Every per-connection file read along the way is deleted afterward, since the combined file is the only one downstream code (get_monitoring_metric()) reads. Each filename is resolved via resolve_within_result_folder() before deletion, since connection names originate from connections.config rather than a hardcoded value.

Idempotent across repeated calls: an existing combined file is loaded first and used as the starting point, so an identity resolved (and thus deleted) by an earlier call is not lost from the combined file just because its per-connection source is already gone.

Parameters:: component (str) – Component label used in the metric filename prefix (e.g. 'loading', 'benchmarking').