bexhoma.clusters module

Kubernetes cluster management for bexhoma experiments.

Provides Kubernetes for managing experiment deployments on Kubernetes, and AWS for AWS-specific Kubernetes clusters.

Authors: Patrick K. Erdelt Copyright (C) 2020 Patrick K. Erdelt SPDX-License-Identifier: AGPL-3.0-or-later See LICENSE for details.

class bexhoma.clusters.AWS(clusterconfig='cluster.config', experiments_configfolder='experiments/', yamlfolder='k8s/', context=None, code=None, instance=None, volume=None, docker=None, script=None, queryfile=None)

Bases: Kubernetes

AWS EKS extension of the Kubernetes cluster manager.

Adds eksctl-based nodegroup scaling and AWS-specific node label handling.

check_nodegroup(nodegroup_type='', nodegroup_name='', num_nodes_aux_planned=0)

Return whether a nodegroup is at the expected size.

Parameters:

  • nodegroup_type – type label value.

  • nodegroup_name – EKS nodegroup name.

  • num_nodes_aux_planned – Expected node count.

Returns:

True if actual count equals num_nodes_aux_planned.

eksctl(command)

Run an eksctl command and return its stdout.

Parameters:

command – eksctl subcommand string (without the eksctl prefix).

Returns:

stdout of the eksctl command.

get_nodegroup_size(nodegroup_type='', nodegroup_name='')

Return the current number of ready nodes in an EKS nodegroup.

Parameters:

  • nodegroup_type – type label value.

  • nodegroup_name – EKS nodegroup name.

Returns:

Number of nodes currently in the nodegroup.

get_nodes(app='', nodegroup_type='', nodegroup_name='')

Return node objects matching the given selectors.

Overrides Kubernetes.get_nodes() to use the EKS-specific alpha.eksctl.io/nodegroup-name label instead of the generic name label.

Parameters:

  • app – app label value. Defaults to self.appname.

  • nodegroup_type – type label value.

  • nodegroup_name – EKS nodegroup name (alpha.eksctl.io/nodegroup-name).

Returns:

List of Kubernetes node objects.

scale_nodegroup(nodegroup_name, size)

Scale a single EKS nodegroup to the requested number of nodes.

No-ops if the nodegroup is already at the target size.

Parameters:

  • nodegroup_name – EKS nodegroup name.

  • size – Desired number of nodes.

Returns:

eksctl output, or None if already at the target size.

scale_nodegroups(nodegroup_names, size=None)

Scale multiple EKS nodegroups.

Parameters:

  • nodegroup_names – Dict mapping nodegroup name to default target size.

  • size – If given, overrides the default size for every nodegroup.

wait_for_nodegroup(nodegroup_type='', nodegroup_name='', num_nodes_aux_planned=0)

Block until a single EKS nodegroup reaches the target size, polling every 30 s.

Parameters:

  • nodegroup_type – type label value.

  • nodegroup_name – EKS nodegroup name.

  • num_nodes_aux_planned – Desired node count.

Returns:

True once the nodegroup is at the target size.

wait_for_nodegroups(nodegroup_names, size=None)

Block until all listed EKS nodegroups reach their target sizes.

Parameters:

  • nodegroup_names – Dict mapping nodegroup name to default target size.

  • size – If given, overrides the default size for every nodegroup.

class bexhoma.clusters.Kubernetes(clusterconfig='cluster.config', experiments_configfolder='experiments/', yamlfolder='k8s/', context=None, code=None, instance=None, volume=None, docker=None, script=None, queryfile=None)

Bases: object

Manages bexhoma experiments on a Kubernetes cluster.

Provides Kubernetes API wrappers (pod/job/service/PVC queries and deletions), cluster component lifecycle methods (dashboard, message queue, monitoring, SUT), experiment bookkeeping (code, result folder, experiment list), and pod/job log persistence.

Subclass: AWS (K8s on AWS with EKS nodegroup scaling).

add_experiment(experiment)

Append an experiment object to this cluster’s experiment list.

Parameters:

experiment – Experiment object to add.

add_to_messagequeue(queue, data)

Push data onto the tail of a Redis list (message queue).

Parameters:

  • queue – Redis key (queue name).

  • data – Value to push onto the queue.

check_dbms_connection(ip, port)

Test whether a TCP connection to ip:port can be established.

Used to probe DBMS readiness before starting a benchmark.

Parameters:

  • ip – Hostname or IP address to connect to.

  • port – TCP port number.

Returns:

True if the connection succeeded, False otherwise.

cluster_access()

Initialise Kubernetes API clients using the configured context.

Sets self.v1core, self.v1apps, and self.v1batches. Prints a warning if the cluster cannot be reached.

create_object_from_file(filename_source)

Apply a Kubernetes manifest file via kubectl create.

The manifest is copied to the experiment result folder with the BEXHOMA_PACKAGE_VERSION placeholder substituted, then applied.

Parameters:

filename_source – Path to the source manifest template file.

delete_deployment(deployment)

Delete a Kubernetes Deployment by name.

Parameters:

deployment – Name of the Deployment to delete.

delete_job(jobname='', app='', component='', experiment='', configuration='', client='')

Delete a Job by name or by matching label selectors.

If jobname is empty, the first Job matching the given selectors is deleted.

Parameters:

  • jobname – Name of the Job to delete (optional).

  • app – app label value. Defaults to self.appname.

  • component – component label value.

  • experiment – experiment label value.

  • configuration – configuration label value.

  • client – client label value (legacy).

Returns:

True on success, or None if the Job was not found.

delete_job_pods(jobname='', app='', component='', experiment='', configuration='', client='')

Delete all Pods of a Job identified by name or by matching label selectors.

If jobname is empty, all Pods matching the given selectors are deleted individually by recursing with their names.

Parameters:

  • jobname – Pod or Job name (optional).

  • app – app label value. Defaults to self.appname.

  • component – component label value.

  • experiment – experiment label value.

  • configuration – configuration label value.

  • client – client label value (legacy).

delete_pod(name)

Delete a Pod by name. A 404 (already gone) is silently ignored.

Parameters:

name – Name of the Pod to delete.

delete_pvc(name)

Delete a PersistentVolumeClaim by name.

Parameters:

name – Name of the PVC to delete.

Returns:

True if deleted successfully, False on error.

delete_service(name)

Delete a Service by name.

Parameters:

name – Name of the Service to delete.

delete_stateful_set(name)

Delete a StatefulSet by name.

Parameters:

name – Name of the StatefulSet to delete.

download_file(filename_remote, filename_local, pod, container='dashboard')

Download a file from a Pod container to the local machine using kubectl cp.

On Windows the local destination path is converted to a UNC path first.

Parameters:

  • filename_remote – Source path inside the container.

  • filename_local – Destination path on the local machine.

  • pod – Source Pod name.

  • container – Source container name. Defaults to dashboard.

Returns:

Output of the kubectl command.

execute_command_in_pod(command, pod='', container='', params='')

Execute a shell command inside a container of a running Pod.

Retries automatically on transient error dialing backend failures.

Parameters:

  • command – Shell command string.

  • pod – Name of the target Pod.

  • container – Container name within the Pod (optional for single-container pods).

  • params – Unused; reserved for future use.

Returns:

Tuple ("", stdout_str, stderr_str).

forward_dashboard_ports()

Forward the dashboard Pod’s ports (8050 and 8888) to localhost.

Port 8050 is the DBMSBenchmarker result dashboard; port 8888 is Jupyter.

forward_sut_port(experiment='', configuration='')

Forward the SUT master service port to localhost.

Parameters:

  • experiment – Filter by experiment code (optional).

  • configuration – Filter by DBMS configuration name (optional).

get_dashboard_name(app='', component='dashboard')

Build a canonical name string for the dashboard component.

Parameters:

  • app – App name. Defaults to self.appname.

  • component – Component label. Defaults to dashboard.

Returns:

Name string in the format <app>_<component>.

get_dashboard_pod_name(app='', component='dashboard')

Return the name of the dashboard Pod, or "" if none exists.

Parameters:

  • app – App name (passed to get_pods(); unused if empty).

  • component – Component label. Defaults to dashboard.

Returns:

Pod name string, or "" if no dashboard Pod was found.

get_deployments(app='', component='', experiment='', configuration='')

Return names of Deployments matching the given label selectors.

Parameters:

  • app – app label value. Defaults to self.appname.

  • component – component label value (e.g. sut, monitoring).

  • experiment – experiment label value (experiment code).

  • configuration – configuration label value (DBMS config name).

Returns:

List of Deployment names.

get_job_pods(app='', component='', experiment='', configuration='', client='')

Return names of Pods belonging to Jobs matching the given label selectors.

Parameters:

  • app – app label value. Defaults to self.appname.

  • component – component label value.

  • experiment – experiment label value.

  • configuration – configuration label value.

  • client – client label value (legacy).

Returns:

List of Pod names, or None on a 404 error.

get_job_status(jobname='', app='', component='', experiment='', configuration='', client='')

Return the completion status of a Job.

If jobname is empty, the first Job matching the given selectors is used. Returns True if the completion count has been reached, 0 if in-progress or on error, "no job" if no matching Job was found.

Parameters:

  • jobname – Name of the Job (optional).

  • app – app label value. Defaults to self.appname.

  • component – component label value.

  • experiment – experiment label value.

  • configuration – configuration label value.

  • client – client label value (legacy).

Returns:

True, 0, or "no job".

get_jobs(app='', component='', experiment='', configuration='', client='')

Return names of Jobs matching the given label selectors.

Parameters:

  • app – app label value. Defaults to self.appname.

  • component – component label value.

  • experiment – experiment label value.

  • configuration – configuration label value.

  • client – client label value (legacy, may be unused).

Returns:

List of Job names, or None on a 404 error.

get_jobs_labels(app='', component='', experiment='', configuration='', client='')

Return a dict mapping Job name to label dict for Jobs matching the given selectors.

Parameters:

  • app – app label value. Defaults to self.appname.

  • component – component label value.

  • experiment – experiment label value.

  • configuration – configuration label value.

  • client – client label value (legacy).

Returns:

Dict {job_name: labels_dict}, or [] if no Jobs found.

get_messagequeue_name(app='', component='messagequeue')

Build a canonical name string for the message-queue component.

Parameters:

  • app – App name. Defaults to self.appname.

  • component – Component label. Defaults to messagequeue.

Returns:

Name string in the format <app>_<component>.

get_nodes(app='', nodegroup_type='', nodegroup_name='')

Return node objects matching the given label selectors.

Parameters:

  • app – app label value. Defaults to self.appname.

  • nodegroup_type – type label value (e.g. sut).

  • nodegroup_name – name label value (e.g. sut_high_memory).

Returns:

List of Kubernetes node objects.

get_pod_containers(pod)

Return the names of all containers and init-containers in a Pod.

Parameters:

pod – Name of the Pod.

Returns:

List of container name strings (regular containers + init containers).

get_pod_status(pod, app='')

Return the phase of a named Pod.

Parameters:

  • pod – Pod name to look up.

  • app – app label value. Defaults to self.appname.

Returns:

Phase string (e.g. Running, Succeeded) or "" if not found.

get_pods(app='', component='', experiment='', configuration='', dbms='', status='')

Return names of Pods matching the given label selectors and optional phase.

Parameters:

  • app – app label value. Defaults to self.appname.

  • component – component label value.

  • experiment – experiment label value.

  • configuration – configuration label value.

  • dbms – dbms label value (DBMS type, e.g. MonetDB).

  • status – Pod phase to filter by (e.g. Running, Succeeded).

Returns:

List of Pod names.

get_pods_labels(app='', component='', experiment='', configuration='')

Return a dict mapping Pod name to label dict for Pods matching the given selectors.

Parameters:

  • app – app label value. Defaults to self.appname.

  • component – component label value.

  • experiment – experiment label value.

  • configuration – configuration label value.

Returns:

Dict {pod_name: labels_dict}.

get_ports_of_service(app='', component='', experiment='', configuration='')

Return the port numbers exposed by the first Service matching the given selectors.

Parameters:

  • app – app label value. Defaults to self.appname.

  • component – component label value.

  • experiment – experiment label value.

  • configuration – configuration label value.

Returns:

List of port number strings from the first matched Service.

get_pvc(app='', component='', experiment='', configuration='', pvc='')

Return names of PersistentVolumeClaims matching the given selectors.

If pvc is provided, only the entry with that exact name is returned.

Parameters:

  • app – app label value. Defaults to self.appname.

  • component – component label value.

  • experiment – experiment label value.

  • configuration – configuration label value.

  • pvc – Optional PVC name to filter by.

Returns:

List of PVC names.

get_pvc_labels(app='', component='', experiment='', configuration='', pvc='')

Return label dicts of PVCs matching the given selectors.

Parameters:

  • app – app label value. Defaults to self.appname.

  • component – component label value.

  • experiment – experiment label value.

  • configuration – configuration label value.

  • pvc – Optional PVC name to filter by.

Returns:

List of label dicts.

get_pvc_specs(app='', component='', experiment='', configuration='', pvc='')

Return spec objects of PVCs matching the given selectors.

Parameters:

  • app – app label value. Defaults to self.appname.

  • component – component label value.

  • experiment – experiment label value.

  • configuration – configuration label value.

  • pvc – Optional PVC name to filter by.

Returns:

List of PVC spec objects.

get_pvc_status(app='', component='', experiment='', configuration='', pvc='')

Return status objects of PVCs matching the given selectors.

When pvc is given, returns the status of the matching PVC; otherwise returns the spec of all matched PVCs.

Parameters:

  • app – app label value. Defaults to self.appname.

  • component – component label value.

  • experiment – experiment label value.

  • configuration – configuration label value.

  • pvc – Optional PVC name to filter by (returns status for match).

Returns:

List of PVC status or spec objects.

get_service_endpoints(service_name='bexhoma-service-monitoring-default')

Return IP addresses of all endpoints for a named Service.

Particularly useful for headless Services (e.g. to enumerate monitoring nodes).

Parameters:

service_name – Name of the Service to query.

Returns:

List of endpoint IP strings, or [] on error.

get_services(app='', component='', experiment='', configuration='')

Return names of Services matching the given label selectors.

Parameters:

  • app – app label value. Defaults to self.appname.

  • component – component label value.

  • experiment – experiment label value.

  • configuration – configuration label value.

Returns:

List of Service names.

get_stateful_set_pods(stateful_set='')

Return names of Pods belonging to a given StatefulSet.

Parameters:

stateful_set – Name of the StatefulSet.

Returns:

List of Pod names.

get_stateful_sets(app='', component='', experiment='', configuration='')

Return names of StatefulSets matching the given label selectors.

Parameters:

  • app – app label value. Defaults to self.appname.

  • component – component label value.

  • experiment – experiment label value.

  • configuration – configuration label value.

Returns:

List of StatefulSet names.

is_dashboard_running()

Return whether the dashboard Pod exists and is in the Running phase.

Returns:

True if the dashboard is running.

is_messagequeue_running(component='messagequeue')

Return whether the message-queue Pod exists and is in the Running phase.

Parameters:

component – Component label to query. Defaults to messagequeue.

Returns:

True if the message queue is running.

is_monitoring_healthy()

Probe Prometheus by issuing a query_range request from inside the dashboard pod.

Queries sum(node_memory_MemTotal_bytes) over a 5-minute window ending 4 minutes ago. Returns True if Prometheus responds with HTTP 200.

Returns:

True if Prometheus is reachable and healthy.

is_pod_ready(pod)

Return whether a Pod’s Ready condition is True.

Parameters:

pod – Name of the Pod to check.

Returns:

True if the Pod is ready, False otherwise.

kubectl(command)

Run a kubectl command in the configured context.

Decodes output using UTF-8, Latin-1, or CP-1252 (in that order). On an Unauthorized response the access token is refreshed and the command is retried once.

Parameters:

command – kubectl subcommand string (without kubectl --context ... prefix).

Returns:

Decoded stdout string, or None on failure.

log_experiment(experiment)

Append a step record to the experiment log and persist it to disk.

The record is enriched with cluster-level metadata and appended to self.experiments. The list is written to experiments.config in the benchmark result folder.

Note

This method should be updated to produce YAML output and to handle detached parallel-loader workflows.

Parameters:

experiment – Dict describing the current experiment step.

pod_description(pod, container='')

Return the kubectl describe pod output for a given Pod.

The container parameter is accepted for API compatibility but is not forwarded to kubectl (describe is not container-sensitive).

Parameters:

  • pod – Name of the Pod to describe.

  • container – Ignored; kept for API compatibility.

Returns:

kubectl output string.

pod_description_exists(pod_name, container='')

Return whether a cached describe file exists in the result folder.

Parameters:

  • pod_name – Name of the Pod.

  • container – Accepted for API compatibility but ignored — kubectl describe is not container-sensitive.

Returns:

True if the .describe file exists on disk.

pod_log(pod, container='')

Return the kubectl logs --tail=-1 output for a given Pod or container.

Parameters:

  • pod – Name of the Pod.

  • container – Container name within the Pod (optional).

Returns:

kubectl output string.

pod_log_exists(pod_name, container='')

Return whether a cached log file exists in the result folder.

Parameters:

  • pod_name – Name of the Pod.

  • container – Container name within the Pod (optional).

Returns:

True if the .log file exists on disk.

pvc_exists(name)

Return whether a PVC with the given name exists in the namespace.

Parameters:

name – Name of the PVC to check.

Returns:

True if the PVC exists, False if not found (HTTP 404).

restart_dashboard(app='', component='dashboard')

Force-restart the dashboard by deleting its Pod (Kubernetes will recreate it).

Parameters:
set_code(code)

Set the unique identifier of the current experiment.

If code is not None and an experiments.config file exists in the result folder, the stored experiment list is loaded from it.

Parameters:

code – Unique experiment identifier string.

set_connection_management(**kwargs)

Set DBMSBenchmarker connection-management parameters.

Can be overridden per experiment or per DBMS configuration.

Parameters:

kwargs – Key/value pairs (e.g. timeout=60, numProcesses=4).

set_ddl_parameters(**kwargs)

Set DDL template substitution parameters.

Values replace placeholders in DDL scripts executed during loading. Can be overridden per experiment or per DBMS configuration.

Parameters:

kwargs – Key/value pairs (e.g. index='btree').

set_experiment(instance=None, volume=None, docker=None, script=None)

Select a specific instance/volume/docker/script combination for the experiment.

Parameters:

  • instance – Instance key within self.instances (legacy IaaS).

  • volume – Volume key within self.volumes.

  • docker – Docker image key within self.dockers.

  • script – Init-script key within the selected volume’s initscripts.

set_experiments(instances=None, volumes=None, dockers=None)

Store the top-level experiment catalog from the cluster configuration.

Parameters:

  • instances – Dict of IaaS instance specs (legacy, was for IaaS scaling).

  • volumes – Dict of volume definitions carrying dataset metadata.

  • dockers – Dict of Docker image descriptors and usage metadata.

set_experiments_configfolder(experiments_configfolder)

Set the folder that contains experiment configuration sub-folders.

Bexhoma expects sub-folders named by experiment type (e.g. tpch), each containing a queries.config file and per-DBMS DDL schema folders.

Parameters:

experiments_configfolder – Relative path to the experiments folder.

set_pod_config(key: str, config: dict) → None

Store per-pod configuration as a JSON string in Redis.

The JSON object is stored at key and can be retrieved by shell scripts running inside pods using redis-cli get <key>. The value is then expanded into BEXHOMA_POD_-prefixed environment variables by bexhoma-pod-env.sh.

Parameters:

  • key (str) – Redis key to store the configuration under.

  • config (dict) – Dict mapping parameter names to their values.

set_pod_counter(queue, value=0)

Set a Redis key to an integer value (used as a pod-count synchronisation counter).

Parameters:

  • queue – Redis key to set.

  • value – Integer value to assign. Defaults to 0.

set_query_management(**kwargs)

Set DBMSBenchmarker query-management parameters.

Parameters:

kwargs – Key/value pairs forwarded to DBMSBenchmarker (e.g. numRun=3).

set_queryfile(queryfile)

Set the query config file for the DBMSBenchmarker benchmarker component.

Parameters:

queryfile – Path to the DBMSBenchmarker query configuration file.

set_resources(**kwargs)

Set Kubernetes resource requests/limits for the SUT component.

Can be overridden per experiment or per DBMS configuration.

Parameters:

kwargs – Key/value pairs (e.g. requests={'cpu': 4, 'memory': '16Gi'}).

set_workload(**kwargs)

Set workload metadata for the experiment (e.g. name, description).

Parameters:

kwargs – Arbitrary key/value pairs stored in self.workload.

start_dashboard(app='', component='dashboard')

Start the dashboard Deployment and its Service if not already running.

Manifest template: deploymenttemplate-bexhoma-dashboard.yml.

Parameters:

  • app – App name passed to get_dashboard_name().

  • component – Component label. Defaults to dashboard.

start_datadir()

Provision the shared data-source PVC if it does not exist.

The PVC is used by data-generator pods for writing and by loader pods for reading. Manifest template: pvc-bexhoma-data.yml.

start_messagequeue(app='', component='messagequeue')

Start the message-queue Deployment if not already running.

Manifest template: deploymenttemplate-bexhoma-messagequeue.yml.

Parameters:
start_monitoring_cluster(app='', component='monitoring')

Start cluster-level monitoring (Prometheus + node exporters) if not healthy.

Waits up to 100 seconds for an existing Prometheus to become healthy before deploying the daemonset. Manifest template: daemonsettemplate-monitoring.yml.

Parameters:

  • app – App name (unused; kept for API consistency).

  • component – Component label. Defaults to monitoring.

start_resultdir()

Provision the shared results PVC if it does not exist.

Benchmarker pods write results here; the evaluator pod reads from here. Collected metrics are also stored here. Manifest template: pvc-bexhoma-results.yml.

stop_benchmarker(experiment='', configuration='')

Stop all benchmarker Jobs and their Pods in the cluster.

Parameters:

  • experiment – Filter by experiment code (optional).

  • configuration – Filter by DBMS configuration name (optional).

stop_dashboard(app='', component='dashboard')

Stop the dashboard Deployment and its Service.

Parameters:

  • app – app label value. Defaults to self.appname via sub-calls.

  • component – component label value. Defaults to dashboard.

stop_loading(experiment='', configuration='')

Stop all loading Jobs and their Pods in the cluster.

Parameters:

  • experiment – Filter by experiment code (optional).

  • configuration – Filter by DBMS configuration name (optional).

stop_maintaining(experiment='', configuration='')

Stop all maintaining Jobs and their Pods in the cluster.

Parameters:

  • experiment – Filter by experiment code (optional).

  • configuration – Filter by DBMS configuration name (optional).

stop_monitoring(app='', component='monitoring', experiment='', configuration='')

Stop all monitoring Deployments and their Services in the cluster.

Parameters:

  • app – app label value.

  • component – Component label. Defaults to monitoring.

  • experiment – Filter by experiment code (optional).

  • configuration – Filter by DBMS configuration name (optional).

stop_sut(app='', component='sut', experiment='', configuration='')

Stop all SUT Deployments, StatefulSets, and Services in the cluster.

When component='sut', worker and pool sub-components are recursively stopped.

Parameters:

  • app – app label value.

  • component – Component label. Defaults to sut.

  • experiment – Filter by experiment code (optional).

  • configuration – Filter by DBMS configuration name (optional).

store_pod_description(pod_name, container='', number=None)

Fetch and persist kubectl describe pod output to the result folder.

The file is not overwritten if it already exists. Up to 10 retries are attempted in case of transient kubectl failures.

Parameters:

  • pod_name – Name of the Pod to describe.

  • container – Accepted for API compatibility but ignored — kubectl describe is not container-sensitive.

  • number – Optional index suffix appended to the filename.

store_pod_log(pod_name, container='', number=None)

Fetch and persist kubectl logs output to the result folder.

The file is not overwritten if it already exists. Up to 10 retries are attempted in case of transient kubectl failures.

Parameters:

  • pod_name – Name of the Pod.

  • container – Container name within the Pod (optional).

  • number – Optional index suffix appended to the filename.

upload_file(filename_remote, filename_local, pod, container='dashboard')

Upload a local file into a Pod container using kubectl cp.

On Windows the local path is converted to a UNC path first.

Parameters:

  • filename_remote – Destination path inside the container.

  • filename_local – Source path on the local machine.

  • pod – Target Pod name.

  • container – Target container name. Defaults to dashboard.

Returns:

Output of the kubectl command.

wait(sec, silent=False)

Sleep for sec seconds, optionally printing progress messages.

Parameters:

  • sec – Number of seconds to wait.

  • silent – If True, suppress all output.

bexhoma.clusters.to_unc(path: str) → str

Convert a local Windows drive path to a UNC administrative share path.

D:/foo and D:\foo become \\localhost\D$\foo. On Linux/macOS the normalized path is returned unchanged. A path that is already UNC is returned unchanged.