Benchmark: YCSB
YCSB does allow scaling data generation and ingestion, and scaling the benchmarking driver. Scale-out can simulate distributed clients. It is not self-evident and sure, that scale-out and scale-up yield the same performance results [3].
The goal of the YCSB project is to develop a framework and common set of workloads for evaluating the performance of different key-value and cloud-serving stores. […] The workloads in the core package are a variation of the same basic application type. In this application, there is a table of records, each with F fields. Each record is identif ied by a primary key, which is a string like user234123. Each field is named field0, field1 and so on. The values of each field are a random string of ASCII characters of length L. Each operation against the data store is randomly chosen to be one of:
Insert: Insert a new record.
Update: Update a record by replacing the value of one f ield.
Read: Read a record, either one randomly chosen field or all fields.
Scan: Scan records in order, starting at a randomly chosen record key. The number of records to scan is randomly chosen. [1,2]
The results are not official benchmark results. Exact performance depends on a number of parameters. You may get different results. These examples are solely to illustrate how to use bexhoma and show the result evaluation.
References:
YCSB Repository: https://github.com/brianfrankcooper/YCSB/wiki/Running-a-Workload
Benchmarking cloud serving systems with YCSB: https://dl.acm.org/doi/10.1145/1807128.1807152
A Cloud-Native Adoption of Classical DBMS Performance Benchmarks and Tools: https://doi.org/10.1007/978-3-031-68031-1_9
Perform Benchmark
You will have to change the node selectors there (to names of nodes, that exist in your cluster - or to leave out the corresponding parameters):
BEXHOMA_NODE_SUT="cl-worker11"
BEXHOMA_NODE_LOAD="cl-worker19"
BEXHOMA_NODE_BENCHMARK="cl-worker19"
LOG_DIR="./logs_tests"
BEXHOMA_MS=1
BEXHOMA_STORAGE_CLASS="shared"
mkdir -p $LOG_DIR
For performing the experiment we can run the ycsb file.
Example:
bexhoma ycsb \
-dbms PostgreSQL \
-sf 1 \
-xwl a \
-xtb 16384 \
-xnbf 2 \
-xnlf 1,4 \
-nc 1 \
-ne 1 \
-nlp 1,8 \
-nlt 64 \
-nbp 1 \
-nbt 64 \
-ms $BEXHOMA_MS \
-tr \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/doc_ycsb_testcase_loading.log
This
loops over
nin [1,8] andtin [1,4]starts a clean instance of PostgreSQL (
-dbms)data directory inside a Docker container
creates YCSB schema in each database
starts
nloader pods per DBMSwith a loading container each
threads = 64/
n(-nlt)target throughput is
t* 16384generates YCSB data = 1.000.000 rows (i.e., SF=1,
-sf)imports it into the DBMS
loops over
min [1] andsin [2]runs
mparallel streams of YCSB queries per DBMS1.000.000 operations
workload A = 50% read / 50% write (
--workload)target throughput is
s* 16384threads = 64/
m(-nbt)
with a maximum of 1 DBMS per time (
-ms)
tests if results match workflow (
-tr)shows a summary
Status
You can watch the status while benchmark is running via bexperiments status
Dashboard: Running
Cluster Prometheus: Running
Message Queue: Running
Data directory: Running
Result directory: Running
+-----------------------+--------------+--------------+------------+-------------+
| 1726160982 | sut | loaded [s] | use case | loading |
+=======================+==============+==============+============+=============+
| PostgreSQL-64-1-16384 | (1. Running) | 1 | ycsb | (1 Running) |
+-----------------------+--------------+--------------+------------+-------------+
The code 1726160982 is the unique identifier of the experiment.
You can find the number also in the output of ycsb.py.
Cleanup
The script is supposed to clean up and remove everything from the cluster that is related to the experiment after finishing.
If something goes wrong, you can also clean up manually with bexperiment stop (removes everything) or bexperiment stop -e 1726160982 (removes everything that is related to experiment 1726160982).
Evaluate Results
At the end of a benchmark you will see a summary like
doc_ycsb_testcase_loading.log
## Show Summary
### Workload
YCSB SF=1
* Type: ycsb
* Duration: 3975s
* Code: 1775546985
* YCSB driver runs the experiment.
* This experiment compares run time and resource consumption of YCSB queries.
* Workload is 'A'.
* Number of rows to insert is 1000000.
* Ordering of inserts is hashed.
* Number of operations is 1000000.
* Batch size is ''.
* Target is based on multiples of '16384'.
* Factors for loading are [1, 4].
* Factors for benchmarking are [2].
* Experiment uses bexhoma version 0.9.5.
* Experiment is limited to DBMS ['PostgreSQL'].
* Import is handled by 1 and 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker14.
* Loading is tested with [64] threads, split into [1, 8] pods.
* Benchmarking is tested with [64] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* PostgreSQL-64-1-16384-1 uses docker image postgres:18.3
* RAM:541008474112
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:5.15.0-164-generic
* node:cl-worker14
* disk:152713
* datadisk:2392
* cpu_list:0-63
* args:['-c', 'max_connections=3000', '-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'effective_io_concurrency=64', '-c', 'io_method=worker', '-c', 'shared_buffers=256GB', '-c', 'effective_cache_size=256GB', '-c', 'work_mem=32GB', '-c', 'maintenance_work_mem=4GB', '-c', 'temp_buffers=4GB', '-c', 'wal_buffers=1GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'fsync=on', '-c', 'wal_compression=on', '-c', 'synchronous_commit=on', '-c', 'max_wal_size=32GB', '-c', 'min_wal_size=32GB', '-c', 'checkpoint_timeout=12h', '-c', 'checkpoint_completion_target=1.0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1775546985
* PostgreSQL-64-1-65536-1 uses docker image postgres:18.3
* RAM:541008474112
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:5.15.0-164-generic
* node:cl-worker14
* disk:152713
* datadisk:2392
* cpu_list:0-63
* args:['-c', 'max_connections=3000', '-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'effective_io_concurrency=64', '-c', 'io_method=worker', '-c', 'shared_buffers=256GB', '-c', 'effective_cache_size=256GB', '-c', 'work_mem=32GB', '-c', 'maintenance_work_mem=4GB', '-c', 'temp_buffers=4GB', '-c', 'wal_buffers=1GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'fsync=on', '-c', 'wal_compression=on', '-c', 'synchronous_commit=on', '-c', 'max_wal_size=32GB', '-c', 'min_wal_size=32GB', '-c', 'checkpoint_timeout=12h', '-c', 'checkpoint_completion_target=1.0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1775546985
* PostgreSQL-64-8-16384-1 uses docker image postgres:18.3
* RAM:541008474112
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:5.15.0-164-generic
* node:cl-worker14
* disk:152711
* datadisk:2391
* cpu_list:0-63
* args:['-c', 'max_connections=3000', '-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'effective_io_concurrency=64', '-c', 'io_method=worker', '-c', 'shared_buffers=256GB', '-c', 'effective_cache_size=256GB', '-c', 'work_mem=32GB', '-c', 'maintenance_work_mem=4GB', '-c', 'temp_buffers=4GB', '-c', 'wal_buffers=1GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'fsync=on', '-c', 'wal_compression=on', '-c', 'synchronous_commit=on', '-c', 'max_wal_size=32GB', '-c', 'min_wal_size=32GB', '-c', 'checkpoint_timeout=12h', '-c', 'checkpoint_completion_target=1.0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1775546985
* PostgreSQL-64-8-65536-1 uses docker image postgres:18.3
* RAM:541008474112
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:5.15.0-164-generic
* node:cl-worker14
* disk:152711
* datadisk:2391
* cpu_list:0-63
* args:['-c', 'max_connections=3000', '-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'effective_io_concurrency=64', '-c', 'io_method=worker', '-c', 'shared_buffers=256GB', '-c', 'effective_cache_size=256GB', '-c', 'work_mem=32GB', '-c', 'maintenance_work_mem=4GB', '-c', 'temp_buffers=4GB', '-c', 'wal_buffers=1GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'fsync=on', '-c', 'wal_compression=on', '-c', 'synchronous_commit=on', '-c', 'max_wal_size=32GB', '-c', 'min_wal_size=32GB', '-c', 'checkpoint_timeout=12h', '-c', 'checkpoint_completion_target=1.0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1775546985
### Loading
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) |
|:----------------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|
| PostgreSQL-64-1-16384 | 1.00 | 64.00 | 16384.00 | 1.00 | 0.00 | 3302.10 | 302838.00 | 1000000.00 | 83839.00 |
| PostgreSQL-64-8-16384 | 1.00 | 64.00 | 16384.00 | 8.00 | 0.00 | 3347.94 | 298733.00 | 1000000.00 | 75583.00 |
| PostgreSQL-64-1-65536 | 1.00 | 64.00 | 65536.00 | 1.00 | 0.00 | 3391.84 | 294825.00 | 1000000.00 | 67199.00 |
| PostgreSQL-64-8-65536 | 1.00 | 64.00 | 65536.00 | 8.00 | 0.00 | 3392.04 | 294831.00 | 1000000.00 | 50595.00 |
### Execution
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:------------------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| PostgreSQL-64-1-16384-1 | 1.00 | 64.00 | 32768.00 | 1.00 | 0.00 | 2853.12 | 350494.00 | 500507.00 | 994.00 | 499493.00 | 871423.00 |
| PostgreSQL-64-1-65536-1 | 1.00 | 64.00 | 32768.00 | 1.00 | 0.00 | 2852.27 | 350598.00 | 500530.00 | 855.00 | 499470.00 | 871423.00 |
| PostgreSQL-64-8-16384-1 | 1.00 | 64.00 | 32768.00 | 1.00 | 0.00 | 2840.35 | 352069.00 | 500033.00 | 853.00 | 499967.00 | 856063.00 |
| PostgreSQL-64-8-65536-1 | 1.00 | 64.00 | 32768.00 | 1.00 | 0.00 | 2795.10 | 357769.00 | 499687.00 | 864.00 | 500313.00 | 886783.00 |
### Workflow
#### Actual
* DBMS PostgreSQL-64-1-16384 - Pods [[1]]
* DBMS PostgreSQL-64-1-65536 - Pods [[1]]
* DBMS PostgreSQL-64-8-65536 - Pods [[1]]
* DBMS PostgreSQL-64-8-16384 - Pods [[1]]
#### Planned
* DBMS PostgreSQL-64-1-16384 - Pods [[1]]
* DBMS PostgreSQL-64-1-65536 - Pods [[1]]
* DBMS PostgreSQL-64-8-16384 - Pods [[1]]
* DBMS PostgreSQL-64-8-65536 - Pods [[1]]
### Tests
* TEST passed: Loading Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Execution Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: contains no FAILED column
We can see that the overall throughput is very close to the target and that scaled-out drivers (8 pods with 8 threads each) have similar results as a monolithic driver (1 pod with 64 thread). The runtime is between 8 seconds and 1 minute.
To see the summary again you can simply call bexperiments summary -e 1708411664 with the experiment code.
Detailed Evaluation
Results are transformed into pandas DataFrames and can be inspected in detail. See for example
You can connect to an evaluation server in the cluster by bexperiments dashboard.
This forwards ports, so you have
a Jupyter notebook server at http://localhost:8888
You can connect to an evaluation server locally by bexperiments jupyter.
This forwards ports, so you have
a Jupyter notebook server at http://localhost:8888
Time Series of Metrics
See an example notebook about how to analyze results in detail.
Adjust Parameters
The script supports
exact repetitions for statistical confidence
variations to scan a large parameters space
combine results for easy evaluation
There are various ways to change parameters.
Manifests
The YAML manifests for the components can be found in https://github.com/Beuth-Erdelt/Benchmark-Experiment-Host-Manager/tree/master/k8s
SQL Scrips
The SQL scripts for pre and post ingestion can be found in https://github.com/Beuth-Erdelt/Benchmark-Experiment-Host-Manager/tree/master/experiments/ycsb
There are per DBMS
initschema-files, that are invoked before loading of datacheckschema-files, that are invoked after loading of data
You can find the output of the files in the result folder.
Dockerfiles
The Dockerfiles for the components can be found in https://github.com/Beuth-Erdelt/Benchmark-Experiment-Host-Manager/tree/master/images/ycsb
Command line
You maybe want to adjust some of the parameters that are set in the file: python ycsb.py -h
usage: ycsb.py [-h] [-aws] [-dbms {PostgreSQL,MySQL}] [-db] [-cx CONTEXT] [-e EXPERIMENT] [-m] [-mc] [-ms MAX_SUT] [-nc NUM_CONFIG] [-ne NUM_QUERY_EXECUTORS] [-nl NUM_LOADING] [-nlp NUM_LOADING_PODS] [-xwl {a,b,c,e,f}] [-sf SCALING_FACTOR] [-xop SCALING_FACTOR_OPERATIONS] [-su SCALING_USERS]
[-xsbs SCALING_BATCHSIZE] [-ltf LIST_TARGET_FACTORS] [-xtb TARGET_BASE] [-t TIMEOUT] [-rr REQUEST_RAM] [-rc REQUEST_CPU] [-rct REQUEST_CPU_TYPE] [-rg REQUEST_GPU] [-rgt REQUEST_GPU_TYPE] [-rst {None,,local-hdd,shared}] [-rss REQUEST_STORAGE_SIZE] [-rnn REQUEST_NODE_NAME] [-rnl REQUEST_NODE_LOADING]
[-rnb REQUEST_NODE_BENCHMARKING] [-tr]
{run,start,load,summary}
Perform YCSB benchmarks in a Kubernetes cluster. Number of rows and operations is SF*1,000,000. This installs a clean copy for each target and split of the driver. Optionally monitoring is activated.
positional arguments:
{run,start,load,summary}
import YCSB data or run YCSB queries
options:
-h, --help show this help message and exit
-aws, --aws fix components to node groups at AWS
-dbms {PostgreSQL,MySQL}, --dbms {PostgreSQL,MySQL}
DBMS to load the data
-db, --debug dump debug informations
-cx CONTEXT, --context CONTEXT
context of Kubernetes (for a multi cluster environment), default is current context
-e EXPERIMENT, --experiment EXPERIMENT
sets experiment code for continuing started experiment
-m, --monitoring activates monitoring for sut
-mc, --monitoring-cluster
activates monitoring for all nodes of cluster
-ms MAX_SUT, --max-sut MAX_SUT
maximum number of parallel DBMS configurations, default is no limit
-nc NUM_CONFIG, --num-config NUM_CONFIG
number of runs per configuration
-ne NUM_QUERY_EXECUTORS, --num-query-executors NUM_QUERY_EXECUTORS
comma separated list of number of parallel clients
-nl NUM_LOADING, --num-loading NUM_LOADING
number of parallel loaders per configuration
-nlp NUM_LOADING_PODS, --num-loading-pods NUM_LOADING_PODS
total number of loaders per configuration
-xwl {a,b,c,e,f}, --workload {a,b,c,e,f}
YCSB default workload
-sf SCALING_FACTOR, --scaling-factor SCALING_FACTOR
scaling factor (SF) = number of rows in millions
-xop SCALING_FACTOR_OPERATIONS, --scaling-factor-operations SCALING_FACTOR_OPERATIONS
scaling factor = number of operations in millions (=SF if not set)
-su SCALING_USERS, --scaling-users SCALING_USERS
scaling factor = number of total threads
-xsbs SCALING_BATCHSIZE, --scaling-batchsize SCALING_BATCHSIZE
batch size
-ltf LIST_TARGET_FACTORS, --list-target-factors LIST_TARGET_FACTORS
comma separated list of factors of 16384 ops as target - default range(1,9)
-xtb TARGET_BASE, --target-base TARGET_BASE
ops as target, base for factors - default 16384 = 2**14
-t TIMEOUT, --timeout TIMEOUT
timeout for a run of a query
-rr REQUEST_RAM, --request-ram REQUEST_RAM
request ram for sut, default 16Gi
-rc REQUEST_CPU, --request-cpu REQUEST_CPU
request cpus for sut, default 4
-rct REQUEST_CPU_TYPE, --request-cpu-type REQUEST_CPU_TYPE
request node for sut to have node label cpu=
-rg REQUEST_GPU, --request-gpu REQUEST_GPU
request number of gpus for sut
-rgt REQUEST_GPU_TYPE, --request-gpu-type REQUEST_GPU_TYPE
request node for sut to have node label gpu=
-rst {None,,local-hdd,shared}, --request-storage-type {None,,local-hdd,shared}
request persistent storage of certain type
-rss REQUEST_STORAGE_SIZE, --request-storage-size REQUEST_STORAGE_SIZE
request persistent storage of certain size
-rnn REQUEST_NODE_NAME, --request-node-name REQUEST_NODE_NAME
request a specific node for sut
-rnl REQUEST_NODE_LOADING, --request-node-loading REQUEST_NODE_LOADING
request a specific node for loading pods
-rnb REQUEST_NODE_BENCHMARKING, --request-node-benchmarking REQUEST_NODE_BENCHMARKING
request a specific node for benchmarking pods
-tr, --test-result test if result fulfills some basic requirements
Perform Execution Benchmark
The default behaviour of bexhoma is that several different settings of the loading component are compared. We might only want to benchmark the workloads of YCSB in different configurations and have a fixed loading phase.
For performing the experiment we can run the ycsb file.
Example:
bexhoma ycsb \
-dbms PostgreSQL \
-sf 1 \
-xwl a \
-xtb 16384 \
-xnbf 2,3 \
-xnlf 4 \
-nc 1 \
-ne 1 \
-nlp 8 \
-nlt 64 \
-nbp 1,8 \
-nbt 64 \
-ms $BEXHOMA_MS \
-tr \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/doc_ycsb_testcase_benchmarking.log
This loads a YCSB data set with 8 pods (-lnp) of 64 threads in total.
Each of the drivers has 64 threads and a target of twice or three times (-ltf) the base, that is 16384.
doc_ycsb_testcase_benchmarking.log
## Show Summary
### Workload
YCSB SF=1
* Type: ycsb
* Duration: 2253s
* Code: 1775551000
* YCSB driver runs the experiment.
* This experiment compares run time and resource consumption of YCSB queries.
* Workload is 'A'.
* Number of rows to insert is 1000000.
* Ordering of inserts is hashed.
* Number of operations is 1000000.
* Batch size is ''.
* Target is based on multiples of '16384'.
* Factors for loading are [4].
* Factors for benchmarking are [2, 3].
* Experiment uses bexhoma version 0.9.5.
* Experiment is limited to DBMS ['PostgreSQL'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker14.
* Loading is tested with [64] threads, split into [8] pods.
* Benchmarking is tested with [64] threads, split into [1, 8] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* PostgreSQL-64-8-65536-1 uses docker image postgres:18.3
* RAM:541008474112
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:5.15.0-164-generic
* node:cl-worker14
* disk:152712
* datadisk:2391
* cpu_list:0-63
* args:['-c', 'max_connections=3000', '-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'effective_io_concurrency=64', '-c', 'io_method=worker', '-c', 'shared_buffers=256GB', '-c', 'effective_cache_size=256GB', '-c', 'work_mem=32GB', '-c', 'maintenance_work_mem=4GB', '-c', 'temp_buffers=4GB', '-c', 'wal_buffers=1GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'fsync=on', '-c', 'wal_compression=on', '-c', 'synchronous_commit=on', '-c', 'max_wal_size=32GB', '-c', 'min_wal_size=32GB', '-c', 'checkpoint_timeout=12h', '-c', 'checkpoint_completion_target=1.0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1775551000
* PostgreSQL-64-8-65536-2 uses docker image postgres:18.3
* RAM:541008474112
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:5.15.0-164-generic
* node:cl-worker14
* disk:153221
* datadisk:2900
* cpu_list:0-63
* args:['-c', 'max_connections=3000', '-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'effective_io_concurrency=64', '-c', 'io_method=worker', '-c', 'shared_buffers=256GB', '-c', 'effective_cache_size=256GB', '-c', 'work_mem=32GB', '-c', 'maintenance_work_mem=4GB', '-c', 'temp_buffers=4GB', '-c', 'wal_buffers=1GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'fsync=on', '-c', 'wal_compression=on', '-c', 'synchronous_commit=on', '-c', 'max_wal_size=32GB', '-c', 'min_wal_size=32GB', '-c', 'checkpoint_timeout=12h', '-c', 'checkpoint_completion_target=1.0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1775551000
* PostgreSQL-64-8-65536-3 uses docker image postgres:18.3
* RAM:541008474112
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:5.15.0-164-generic
* node:cl-worker14
* disk:153555
* datadisk:3234
* cpu_list:0-63
* args:['-c', 'max_connections=3000', '-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'effective_io_concurrency=64', '-c', 'io_method=worker', '-c', 'shared_buffers=256GB', '-c', 'effective_cache_size=256GB', '-c', 'work_mem=32GB', '-c', 'maintenance_work_mem=4GB', '-c', 'temp_buffers=4GB', '-c', 'wal_buffers=1GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'fsync=on', '-c', 'wal_compression=on', '-c', 'synchronous_commit=on', '-c', 'max_wal_size=32GB', '-c', 'min_wal_size=32GB', '-c', 'checkpoint_timeout=12h', '-c', 'checkpoint_completion_target=1.0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1775551000
* PostgreSQL-64-8-65536-4 uses docker image postgres:18.3
* RAM:541008474112
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:5.15.0-164-generic
* node:cl-worker14
* disk:153855
* datadisk:3534
* cpu_list:0-63
* args:['-c', 'max_connections=3000', '-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'effective_io_concurrency=64', '-c', 'io_method=worker', '-c', 'shared_buffers=256GB', '-c', 'effective_cache_size=256GB', '-c', 'work_mem=32GB', '-c', 'maintenance_work_mem=4GB', '-c', 'temp_buffers=4GB', '-c', 'wal_buffers=1GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'fsync=on', '-c', 'wal_compression=on', '-c', 'synchronous_commit=on', '-c', 'max_wal_size=32GB', '-c', 'min_wal_size=32GB', '-c', 'checkpoint_timeout=12h', '-c', 'checkpoint_completion_target=1.0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1775551000
### Loading
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) |
|:----------------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|
| PostgreSQL-64-8-65536 | 1.00 | 64.00 | 65536.00 | 8.00 | 0.00 | 3311.41 | 302011.00 | 1000000.00 | 69367.00 |
### Execution
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:------------------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| PostgreSQL-64-8-65536-1 | 1.00 | 64.00 | 32768.00 | 1.00 | 0.00 | 2848.56 | 351055.00 | 500222.00 | 930.00 | 499778.00 | 848383.00 |
| PostgreSQL-64-8-65536-2 | 1.00 | 64.00 | 32768.00 | 8.00 | 0.00 | 2835.37 | 357592.00 | 501761.00 | 755.00 | 498239.00 | 920575.00 |
| PostgreSQL-64-8-65536-3 | 1.00 | 64.00 | 49152.00 | 1.00 | 0.00 | 2744.35 | 364385.00 | 500197.00 | 745.00 | 499803.00 | 896511.00 |
| PostgreSQL-64-8-65536-4 | 1.00 | 64.00 | 49152.00 | 8.00 | 0.00 | 2839.71 | 357722.00 | 499657.00 | 741.00 | 500343.00 | 920575.00 |
### Workflow
#### Actual
* DBMS PostgreSQL-64-8-65536 - Pods [[8, 1, 8, 1]]
#### Planned
* DBMS PostgreSQL-64-8-65536 - Pods [[1, 8, 1, 8]]
### Tests
* TEST passed: Loading Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Execution Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: contains no FAILED column
Monitoring
Monitoring can be activated for DBMS only (-m) or for all components (-mc).
Example:
bexhoma ycsb \
-dbms PostgreSQL \
-sf 3 \
-xwl a \
-xtb 16384 \
-xnbf 2,3 \
-xnlf 4 \
-nc 1 \
-ne 1 \
-nlp 8 \
-nlt 64 \
-nbp 1,8 \
-nbt 64 \
-m \
-mc \
-ms $BEXHOMA_MS \
-tr \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/doc_ycsb_testcase_monitoring.log
If monitoring is activated, the summary also contains a section like this:
doc_ycsb_testcase_monitoring.log
## Show Summary
### Workload
YCSB SF=3
* Type: ycsb
* Duration: 5782s
* Code: 1775553281
* YCSB driver runs the experiment.
* This experiment compares run time and resource consumption of YCSB queries.
* Workload is 'A'.
* Number of rows to insert is 3000000.
* Ordering of inserts is hashed.
* Number of operations is 3000000.
* Batch size is ''.
* Target is based on multiples of '16384'.
* Factors for loading are [4].
* Factors for benchmarking are [2, 3].
* Experiment uses bexhoma version 0.9.5.
* System metrics are monitored by a cluster-wide installation.
* Experiment is limited to DBMS ['PostgreSQL'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker14.
* Loading is tested with [64] threads, split into [8] pods.
* Benchmarking is tested with [64] threads, split into [1, 8] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* PostgreSQL-64-8-65536-1 uses docker image postgres:18.3
* RAM:541008474112
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:5.15.0-164-generic
* node:cl-worker14
* disk:157412
* datadisk:7092
* cpu_list:0-63
* args:['-c', 'max_connections=3000', '-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'effective_io_concurrency=64', '-c', 'io_method=worker', '-c', 'shared_buffers=256GB', '-c', 'effective_cache_size=256GB', '-c', 'work_mem=32GB', '-c', 'maintenance_work_mem=4GB', '-c', 'temp_buffers=4GB', '-c', 'wal_buffers=1GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'fsync=on', '-c', 'wal_compression=on', '-c', 'synchronous_commit=on', '-c', 'max_wal_size=32GB', '-c', 'min_wal_size=32GB', '-c', 'checkpoint_timeout=12h', '-c', 'checkpoint_completion_target=1.0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1775553281
* PostgreSQL-64-8-65536-2 uses docker image postgres:18.3
* RAM:541008474112
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:5.15.0-164-generic
* node:cl-worker14
* disk:158867
* datadisk:8546
* cpu_list:0-63
* args:['-c', 'max_connections=3000', '-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'effective_io_concurrency=64', '-c', 'io_method=worker', '-c', 'shared_buffers=256GB', '-c', 'effective_cache_size=256GB', '-c', 'work_mem=32GB', '-c', 'maintenance_work_mem=4GB', '-c', 'temp_buffers=4GB', '-c', 'wal_buffers=1GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'fsync=on', '-c', 'wal_compression=on', '-c', 'synchronous_commit=on', '-c', 'max_wal_size=32GB', '-c', 'min_wal_size=32GB', '-c', 'checkpoint_timeout=12h', '-c', 'checkpoint_completion_target=1.0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1775553281
* PostgreSQL-64-8-65536-3 uses docker image postgres:18.3
* RAM:541008474112
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:5.15.0-164-generic
* node:cl-worker14
* disk:159849
* datadisk:9527
* cpu_list:0-63
* args:['-c', 'max_connections=3000', '-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'effective_io_concurrency=64', '-c', 'io_method=worker', '-c', 'shared_buffers=256GB', '-c', 'effective_cache_size=256GB', '-c', 'work_mem=32GB', '-c', 'maintenance_work_mem=4GB', '-c', 'temp_buffers=4GB', '-c', 'wal_buffers=1GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'fsync=on', '-c', 'wal_compression=on', '-c', 'synchronous_commit=on', '-c', 'max_wal_size=32GB', '-c', 'min_wal_size=32GB', '-c', 'checkpoint_timeout=12h', '-c', 'checkpoint_completion_target=1.0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1775553281
* PostgreSQL-64-8-65536-4 uses docker image postgres:18.3
* RAM:541008474112
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:5.15.0-164-generic
* node:cl-worker14
* disk:160723
* datadisk:10402
* cpu_list:0-63
* args:['-c', 'max_connections=3000', '-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'effective_io_concurrency=64', '-c', 'io_method=worker', '-c', 'shared_buffers=256GB', '-c', 'effective_cache_size=256GB', '-c', 'work_mem=32GB', '-c', 'maintenance_work_mem=4GB', '-c', 'temp_buffers=4GB', '-c', 'wal_buffers=1GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'fsync=on', '-c', 'wal_compression=on', '-c', 'synchronous_commit=on', '-c', 'max_wal_size=32GB', '-c', 'min_wal_size=32GB', '-c', 'checkpoint_timeout=12h', '-c', 'checkpoint_completion_target=1.0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1775553281
### Loading
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) |
|:----------------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|
| PostgreSQL-64-8-65536 | 1.00 | 64.00 | 65536.00 | 8.00 | 0.00 | 3328.67 | 901390.00 | 3000000.00 | 75407.00 |
### Execution
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:------------------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| PostgreSQL-64-8-65536-1 | 1.00 | 64.00 | 32768.00 | 1.00 | 0.00 | 2769.53 | 1083217.00 | 1500413.00 | 815.00 | 1499587.00 | 911359.00 |
| PostgreSQL-64-8-65536-2 | 1.00 | 64.00 | 32768.00 | 8.00 | 0.00 | 2818.71 | 1071538.00 | 1499635.00 | 750.00 | 1500365.00 | 920063.00 |
| PostgreSQL-64-8-65536-3 | 1.00 | 64.00 | 49152.00 | 1.00 | 0.00 | 2801.92 | 1070694.00 | 1500977.00 | 753.00 | 1499023.00 | 901119.00 |
| PostgreSQL-64-8-65536-4 | 1.00 | 64.00 | 49152.00 | 8.00 | 0.00 | 2813.43 | 1076227.00 | 1499333.00 | 745.00 | 1500667.00 | 920575.00 |
### Workflow
#### Actual
* DBMS PostgreSQL-64-8-65536 - Pods [[8, 8, 1, 1]]
#### Planned
* DBMS PostgreSQL-64-8-65536 - Pods [[1, 8, 1, 8]]
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-64-8-65536-1 | 1086.22 | 1.15 | 9.82 | 12.75 |
| PostgreSQL-64-8-65536-2 | 1086.22 | 1.15 | 9.82 | 12.75 |
| PostgreSQL-64-8-65536-3 | 1086.22 | 1.15 | 9.82 | 12.75 |
| PostgreSQL-64-8-65536-4 | 1086.22 | 1.15 | 9.82 | 12.75 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-64-8-65536-1 | 423.45 | 0.50 | 0.11 | 0.11 |
| PostgreSQL-64-8-65536-2 | 423.45 | 0.50 | 0.11 | 0.11 |
| PostgreSQL-64-8-65536-3 | 423.45 | 0.50 | 0.11 | 0.11 |
| PostgreSQL-64-8-65536-4 | 423.45 | 0.50 | 0.11 | 0.11 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-64-8-65536-1 | 964.71 | 1.11 | 10.92 | 15.12 |
| PostgreSQL-64-8-65536-2 | 900.78 | 0.95 | 11.37 | 15.99 |
| PostgreSQL-64-8-65536-3 | 908.55 | 0.91 | 11.74 | 16.00 |
| PostgreSQL-64-8-65536-4 | 920.38 | 0.89 | 12.11 | 16.00 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-64-8-65536-1 | 259.97 | 0.34 | 0.13 | 0.13 |
| PostgreSQL-64-8-65536-2 | 259.97 | 0.29 | 0.13 | 0.13 |
| PostgreSQL-64-8-65536-3 | 260.50 | 0.60 | 0.13 | 0.13 |
| PostgreSQL-64-8-65536-4 | 262.33 | 0.69 | 0.13 | 0.13 |
### Tests
* TEST passed: Loading Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Execution Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: contains no FAILED column
This gives a survey about CPU (in CPU seconds) and RAM usage (in Gb) during loading and execution of the benchmark.
In this example, metrics are very instable. Metrics are fetched every 30 seconds. This is too coarse for such a quick example.
Use Persistent Storage
The default behaviour of bexhoma is that the database is stored inside the ephemeral storage of the Docker container. If your cluster allows dynamic provisioning of volumes, you might request a persistent storage of a certain type (storageClass) and size.
Example:
bexhoma ycsb \
-dbms PostgreSQL \
-sf 1 \
-xwl a \
-xtb 16384 \
-xnbf 2,3 \
-xnlf 4 \
-nc 2 \
-ne 1 \
-nlp 8 \
-nlt 64 \
-nbp 1,8 \
-nbt 64 \
-ms $BEXHOMA_MS \
-tr \
-rss 30Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/doc_ycsb_testcase_storage.log
The following status shows we have one volume of type shared.
Every PostgreSQL experiment running YCSB of SF=1 will take the databases from these volumes and skip loading.
In this example -nc is set to two, that is the complete experiment is repeated twice for statistical confidence.
The first instance of PostgreSQL mounts the volume and generates the data.
All other instances just use the database without generating and loading data.
+------------------------------------------+-----------------+---------------+--------------+-------------------+------------+----------------------+-----------+----------+--------+--------+
| Volumes | configuration | experiment | loaded [s] | timeLoading [s] | dbms | storage_class_name | storage | status | size | used |
+==========================================+=================+===============+==============+===================+============+======================+===========+==========+========+========+
| bexhoma-storage-postgresql-ycsb-1 | postgresql | ycsb-1 | True | 16 | PostgreSQL | shared | 100Gi | Bound | 100G | 3.7G |
+------------------------------------------+-----------------+---------------+--------------+-------------------+------------+----------------------+-----------+----------+--------+--------+
| bexhoma-storage-postgresql-ycsb-10 | postgresql | ycsb-10 | True | 217 | PostgreSQL | shared | 100Gi | Bound | 100G | 33G |
+------------------------------------------+-----------------+---------------+--------------+-------------------+------------+----------------------+-----------+----------+--------+--------+
The result looks something like
doc_ycsb_testcase_storage.log
## Show Summary
### Workload
YCSB SF=1
* Type: ycsb
* Duration: 11020s
* Code: 1775559134
* YCSB driver runs the experiment.
* This experiment compares run time and resource consumption of YCSB queries.
* Workload is 'A'.
* Number of rows to insert is 1000000.
* Ordering of inserts is hashed.
* Number of operations is 1000000.
* Batch size is ''.
* Target is based on multiples of '16384'.
* Factors for loading are [4].
* Factors for benchmarking are [2, 3].
* Experiment uses bexhoma version 0.9.5.
* Experiment is limited to DBMS ['PostgreSQL'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker14.
* Database is persisted to disk of type shared and size 30Gi.
* Loading is tested with [64] threads, split into [8] pods.
* Benchmarking is tested with [64] threads, split into [1, 8] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run 2 times.
### Connections
* PostgreSQL-64-8-65536-1-1 uses docker image postgres:18.3
* RAM:541008474112
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:5.15.0-164-generic
* node:cl-worker14
* disk:150322
* datadisk:2390
* volume_size:30G
* volume_used:2.4G
* cpu_list:0-63
* args:['-c', 'max_connections=3000', '-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'effective_io_concurrency=64', '-c', 'io_method=worker', '-c', 'shared_buffers=256GB', '-c', 'effective_cache_size=256GB', '-c', 'work_mem=32GB', '-c', 'maintenance_work_mem=4GB', '-c', 'temp_buffers=4GB', '-c', 'wal_buffers=1GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'fsync=on', '-c', 'wal_compression=on', '-c', 'synchronous_commit=on', '-c', 'max_wal_size=32GB', '-c', 'min_wal_size=32GB', '-c', 'checkpoint_timeout=12h', '-c', 'checkpoint_completion_target=1.0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1775559134
* PostgreSQL-64-8-65536-1-2 uses docker image postgres:18.3
* RAM:541008474112
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:5.15.0-164-generic
* node:cl-worker14
* disk:150322
* datadisk:2902
* volume_size:30G
* volume_used:2.4G
* cpu_list:0-63
* args:['-c', 'max_connections=3000', '-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'effective_io_concurrency=64', '-c', 'io_method=worker', '-c', 'shared_buffers=256GB', '-c', 'effective_cache_size=256GB', '-c', 'work_mem=32GB', '-c', 'maintenance_work_mem=4GB', '-c', 'temp_buffers=4GB', '-c', 'wal_buffers=1GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'fsync=on', '-c', 'wal_compression=on', '-c', 'synchronous_commit=on', '-c', 'max_wal_size=32GB', '-c', 'min_wal_size=32GB', '-c', 'checkpoint_timeout=12h', '-c', 'checkpoint_completion_target=1.0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1775559134
* PostgreSQL-64-8-65536-1-3 uses docker image postgres:18.3
* RAM:541008474112
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:5.15.0-164-generic
* node:cl-worker14
* disk:150322
* datadisk:3237
* volume_size:30G
* volume_used:2.4G
* cpu_list:0-63
* args:['-c', 'max_connections=3000', '-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'effective_io_concurrency=64', '-c', 'io_method=worker', '-c', 'shared_buffers=256GB', '-c', 'effective_cache_size=256GB', '-c', 'work_mem=32GB', '-c', 'maintenance_work_mem=4GB', '-c', 'temp_buffers=4GB', '-c', 'wal_buffers=1GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'fsync=on', '-c', 'wal_compression=on', '-c', 'synchronous_commit=on', '-c', 'max_wal_size=32GB', '-c', 'min_wal_size=32GB', '-c', 'checkpoint_timeout=12h', '-c', 'checkpoint_completion_target=1.0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1775559134
* PostgreSQL-64-8-65536-1-4 uses docker image postgres:18.3
* RAM:541008474112
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:5.15.0-164-generic
* node:cl-worker14
* disk:150950
* datadisk:3536
* volume_size:30G
* volume_used:2.4G
* cpu_list:0-63
* args:['-c', 'max_connections=3000', '-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'effective_io_concurrency=64', '-c', 'io_method=worker', '-c', 'shared_buffers=256GB', '-c', 'effective_cache_size=256GB', '-c', 'work_mem=32GB', '-c', 'maintenance_work_mem=4GB', '-c', 'temp_buffers=4GB', '-c', 'wal_buffers=1GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'fsync=on', '-c', 'wal_compression=on', '-c', 'synchronous_commit=on', '-c', 'max_wal_size=32GB', '-c', 'min_wal_size=32GB', '-c', 'checkpoint_timeout=12h', '-c', 'checkpoint_completion_target=1.0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1775559134
* PostgreSQL-64-8-65536-2-1 uses docker image postgres:18.3
* RAM:541008474112
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:5.15.0-164-generic
* node:cl-worker14
* disk:150951
* datadisk:3817
* volume_size:30G
* volume_used:3.8G
* cpu_list:0-63
* args:['-c', 'max_connections=3000', '-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'effective_io_concurrency=64', '-c', 'io_method=worker', '-c', 'shared_buffers=256GB', '-c', 'effective_cache_size=256GB', '-c', 'work_mem=32GB', '-c', 'maintenance_work_mem=4GB', '-c', 'temp_buffers=4GB', '-c', 'wal_buffers=1GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'fsync=on', '-c', 'wal_compression=on', '-c', 'synchronous_commit=on', '-c', 'max_wal_size=32GB', '-c', 'min_wal_size=32GB', '-c', 'checkpoint_timeout=12h', '-c', 'checkpoint_completion_target=1.0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1775559134
* PostgreSQL-64-8-65536-2-2 uses docker image postgres:18.3
* RAM:541008474112
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:5.15.0-164-generic
* node:cl-worker14
* disk:150951
* datadisk:3935
* volume_size:30G
* volume_used:3.8G
* cpu_list:0-63
* args:['-c', 'max_connections=3000', '-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'effective_io_concurrency=64', '-c', 'io_method=worker', '-c', 'shared_buffers=256GB', '-c', 'effective_cache_size=256GB', '-c', 'work_mem=32GB', '-c', 'maintenance_work_mem=4GB', '-c', 'temp_buffers=4GB', '-c', 'wal_buffers=1GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'fsync=on', '-c', 'wal_compression=on', '-c', 'synchronous_commit=on', '-c', 'max_wal_size=32GB', '-c', 'min_wal_size=32GB', '-c', 'checkpoint_timeout=12h', '-c', 'checkpoint_completion_target=1.0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1775559134
* PostgreSQL-64-8-65536-2-3 uses docker image postgres:18.3
* RAM:541008474112
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:5.15.0-164-generic
* node:cl-worker14
* disk:150951
* datadisk:4053
* volume_size:30G
* volume_used:3.8G
* cpu_list:0-63
* args:['-c', 'max_connections=3000', '-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'effective_io_concurrency=64', '-c', 'io_method=worker', '-c', 'shared_buffers=256GB', '-c', 'effective_cache_size=256GB', '-c', 'work_mem=32GB', '-c', 'maintenance_work_mem=4GB', '-c', 'temp_buffers=4GB', '-c', 'wal_buffers=1GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'fsync=on', '-c', 'wal_compression=on', '-c', 'synchronous_commit=on', '-c', 'max_wal_size=32GB', '-c', 'min_wal_size=32GB', '-c', 'checkpoint_timeout=12h', '-c', 'checkpoint_completion_target=1.0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1775559134
* PostgreSQL-64-8-65536-2-4 uses docker image postgres:18.3
* RAM:541008474112
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:5.15.0-164-generic
* node:cl-worker14
* disk:150951
* datadisk:4170
* volume_size:30G
* volume_used:3.8G
* cpu_list:0-63
* args:['-c', 'max_connections=3000', '-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'effective_io_concurrency=64', '-c', 'io_method=worker', '-c', 'shared_buffers=256GB', '-c', 'effective_cache_size=256GB', '-c', 'work_mem=32GB', '-c', 'maintenance_work_mem=4GB', '-c', 'temp_buffers=4GB', '-c', 'wal_buffers=1GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'fsync=on', '-c', 'wal_compression=on', '-c', 'synchronous_commit=on', '-c', 'max_wal_size=32GB', '-c', 'min_wal_size=32GB', '-c', 'checkpoint_timeout=12h', '-c', 'checkpoint_completion_target=1.0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1775559134
### Loading
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) |
|:----------------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|
| PostgreSQL-64-8-65536 | 1.00 | 64.00 | 65536.00 | 8.00 | 0.00 | 707.42 | 1414197.00 | 1000000.00 | 773695.00 |
### Execution
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:--------------------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| PostgreSQL-64-8-65536-1-1 | 1.00 | 64.00 | 32768.00 | 1.00 | 0.00 | 761.34 | 1313467.00 | 500459.00 | 861.00 | 499541.00 | 1860607.00 |
| PostgreSQL-64-8-65536-1-2 | 1.00 | 64.00 | 32768.00 | 8.00 | 0.00 | 1105.79 | 906733.00 | 499911.00 | 801.00 | 500089.00 | 1743871.00 |
| PostgreSQL-64-8-65536-1-3 | 1.00 | 64.00 | 49152.00 | 1.00 | 0.00 | 1146.05 | 872566.00 | 500357.00 | 773.00 | 499643.00 | 1692671.00 |
| PostgreSQL-64-8-65536-1-4 | 1.00 | 64.00 | 49152.00 | 8.00 | 0.00 | 781.97 | 1280511.00 | 499619.00 | 815.00 | 500381.00 | 2185215.00 |
| PostgreSQL-64-8-65536-2-1 | 2.00 | 64.00 | 32768.00 | 1.00 | 0.00 | 784.10 | 1275342.00 | 499561.00 | 1005.00 | 500439.00 | 2617343.00 |
| PostgreSQL-64-8-65536-2-2 | 2.00 | 64.00 | 32768.00 | 8.00 | 0.00 | 821.22 | 1220212.00 | 499681.00 | 824.00 | 500319.00 | 2357247.00 |
| PostgreSQL-64-8-65536-2-3 | 2.00 | 64.00 | 49152.00 | 1.00 | 0.00 | 1133.55 | 882184.00 | 500194.00 | 789.00 | 499806.00 | 1519615.00 |
| PostgreSQL-64-8-65536-2-4 | 2.00 | 64.00 | 49152.00 | 8.00 | 0.00 | 1141.26 | 877835.00 | 499894.00 | 775.00 | 500106.00 | 1960959.00 |
### Workflow
#### Actual
* DBMS PostgreSQL-64-8-65536 - Pods [[8, 8, 1, 1], [8, 8, 1, 1]]
#### Planned
* DBMS PostgreSQL-64-8-65536 - Pods [[1, 8, 1, 8], [1, 8, 1, 8]]
### Tests
* TEST passed: Loading Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Execution Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: contains no FAILED column
Note the added section about volume_size and volume_used in the connections section.
Set DBMS Args
Some DBMS allow to set parameter via args during startup.
In general, the Kubernetes manifest should be used to set all arguments.
In special cases (for example a batch of benchmarks) you may want to set a parameter via CLI.
Here is an example for how to use --set for PostgreSQL 18. This sets effective_io_concurrency to 64.
bexhoma ycsb \
-dbms PostgreSQL \
-sf 1 \
-xwl a \
-xtb 16384 \
-xnbf 2 \
-xnlf 1 \
-nc 1 \
-ne 1 \
-nlp 1 \
-nlt 64 \
-nbp 1 \
-nbt 64 \
-ms $BEXHOMA_MS \
-tr \
--set deployment[bexhoma-deployment-postgres].container[dbms].effective_io_concurrency=64 \
run &>$LOG_DIR/doc_ycsb_testcase_loading_patch.log
The result looks something like
doc_ycsb_testcase_loading_patch.log
## Show Summary
### Workload
YCSB SF=1
* Type: ycsb
* Duration: 369s
* Code: 1775570239
* YCSB driver runs the experiment.
* This experiment compares run time and resource consumption of YCSB queries.
* Workload is 'A'.
* Number of rows to insert is 1000000.
* Ordering of inserts is hashed.
* Number of operations is 1000000.
* Batch size is ''.
* Target is based on multiples of '16384'.
* Factors for loading are [1].
* Factors for benchmarking are [2].
* Experiment uses bexhoma version 0.9.5.
* Experiment is limited to DBMS ['PostgreSQL'].
* Import is handled by 1 processes (pods).
* Loading is tested with [64] threads, split into [1] pods.
* Benchmarking is tested with [64] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* PostgreSQL-64-1-16384-1 uses docker image postgres:18.3
* RAM:2164173209600
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-90-generic
* node:cl-worker36
* disk:1019058
* datadisk:2392
* cpu_list:0-223
* args:['-c', 'max_connections=3000', '-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'effective_io_concurrency=64', '-c', 'io_method=worker', '-c', 'shared_buffers=256GB', '-c', 'effective_cache_size=256GB', '-c', 'work_mem=32GB', '-c', 'maintenance_work_mem=4GB', '-c', 'temp_buffers=4GB', '-c', 'wal_buffers=1GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'fsync=on', '-c', 'wal_compression=on', '-c', 'synchronous_commit=on', '-c', 'max_wal_size=32GB', '-c', 'min_wal_size=32GB', '-c', 'checkpoint_timeout=12h', '-c', 'checkpoint_completion_target=1.0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1775570239
### Loading
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) |
|:----------------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|
| PostgreSQL-64-1-16384 | 1.00 | 64.00 | 16384.00 | 1.00 | 0.00 | 16298.06 | 61357.00 | 1000000.00 | 1928.00 |
### Execution
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:------------------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| PostgreSQL-64-1-16384-1 | 1.00 | 64.00 | 32768.00 | 1.00 | 0.00 | 32412.81 | 30852.00 | 499695.00 | 554.00 | 500305.00 | 1359.00 |
### Workflow
#### Actual
* DBMS PostgreSQL-64-1-16384 - Pods [[1]]
#### Planned
* DBMS PostgreSQL-64-1-16384 - Pods [[1]]
### Tests
* TEST passed: Loading Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Execution Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: contains no FAILED column
All Workloads
Workload A
Workload A is 50% READ and 50% UPDATE.
Example:
bexhoma ycsb \
-dbms PostgreSQL \
-sf 10 \
-xwl a \
-xtb 16384 \
-xnbf 4 \
-xnlf 4 \
-nc 1 \
-ne 1 \
-nlp 8 \
-nlt 64 \
-nbp 1,8 \
-nbt 64 \
-m \
-mc \
-ms $BEXHOMA_MS \
-tr \
-lr 64Gi \
-rr 64Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/doc_ycsb_testcase_a.log
Evaluate Results
doc_ycsb_testcase_a.log
## Show Summary
### Workload
YCSB SF=10
* Type: ycsb
* Duration: 3091s
* Code: 1775576195
* YCSB driver runs the experiment.
* This experiment compares run time and resource consumption of YCSB queries.
* Workload is 'A'.
* Number of rows to insert is 10000000.
* Ordering of inserts is hashed.
* Number of operations is 10000000.
* Batch size is ''.
* Target is based on multiples of '16384'.
* Factors for loading are [4].
* Factors for benchmarking are [4].
* Experiment uses bexhoma version 0.9.5.
* System metrics are monitored by a cluster-wide installation.
* Experiment is limited to DBMS ['PostgreSQL'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker14.
* Loading is tested with [64] threads, split into [8] pods.
* Benchmarking is tested with [64] threads, split into [1, 8] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* PostgreSQL-64-8-65536-1 uses docker image postgres:18.3
* RAM:541008474112
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:5.15.0-164-generic
* node:cl-worker14
* disk:169211
* datadisk:39
* cpu_list:0-63
* args:['-c', 'max_connections=3000', '-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'effective_io_concurrency=64', '-c', 'io_method=worker', '-c', 'shared_buffers=256GB', '-c', 'effective_cache_size=256GB', '-c', 'work_mem=32GB', '-c', 'maintenance_work_mem=4GB', '-c', 'temp_buffers=4GB', '-c', 'wal_buffers=1GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'fsync=on', '-c', 'wal_compression=on', '-c', 'synchronous_commit=on', '-c', 'max_wal_size=32GB', '-c', 'min_wal_size=32GB', '-c', 'checkpoint_timeout=12h', '-c', 'checkpoint_completion_target=1.0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1775576195
* PostgreSQL-64-8-65536-2 uses docker image postgres:18.3
* RAM:541008474112
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:5.15.0-164-generic
* node:cl-worker14
* disk:169211
* datadisk:39
* cpu_list:0-63
* args:['-c', 'max_connections=3000', '-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'effective_io_concurrency=64', '-c', 'io_method=worker', '-c', 'shared_buffers=256GB', '-c', 'effective_cache_size=256GB', '-c', 'work_mem=32GB', '-c', 'maintenance_work_mem=4GB', '-c', 'temp_buffers=4GB', '-c', 'wal_buffers=1GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'fsync=on', '-c', 'wal_compression=on', '-c', 'synchronous_commit=on', '-c', 'max_wal_size=32GB', '-c', 'min_wal_size=32GB', '-c', 'checkpoint_timeout=12h', '-c', 'checkpoint_completion_target=1.0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1775576195
### Loading
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) |
|:----------------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|
| PostgreSQL-64-8-65536 | 1.00 | 64.00 | 65536.00 | 8.00 | 0.00 | 3322.70 | 2323566.00 | 7718809.00 | 70119.00 |
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
list index out of range
### Workflow
#### Planned
* DBMS PostgreSQL-64-8-65536 - Pods [[1, 8]]
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-64-8-65536-1 | 2499.67 | 1.17 | 15.94 | 16.00 |
| PostgreSQL-64-8-65536-2 | 2499.67 | 1.17 | 15.94 | 16.00 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-64-8-65536-1 | 954.04 | 0.47 | 0.11 | 0.11 |
| PostgreSQL-64-8-65536-2 | 954.04 | 0.47 | 0.11 | 0.11 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-64-8-65536-1 | 2501.21 | 7.69 | 15.94 | 16.00 |
| PostgreSQL-64-8-65536-2 | 1230.65 | 8.18 | 5.84 | 5.88 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-64-8-65536-1 | 1286.36 | 8.39 | 0.14 | 0.14 |
| PostgreSQL-64-8-65536-2 | 1506.52 | 19.43 | 0.14 | 0.14 |
### Tests
* TEST passed: Loading Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST failed: Workflow not as planned
* TEST passed: Execution Phase: contains no FAILED column
Workload B
Workload B is 95% READ and 5% UPDATE.
Example:
bexhoma ycsb \
-dbms PostgreSQL \
-sf 10 \
-xwl b \
-xtb 16384 \
-xnbf 4 \
-xnlf 4 \
-nc 1 \
-ne 1 \
-nlp 8 \
-nlt 64 \
-nbp 1,8 \
-nbt 64 \
-m \
-mc \
-ms $BEXHOMA_MS \
-tr \
-lr 64Gi \
-rr 64Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/doc_ycsb_testcase_b.log
Evaluate Results
doc_ycsb_testcase_b.log
## Show Summary
### Workload
YCSB SF=10
* Type: ycsb
* Duration: 4473s
* Code: 1775633754
* YCSB driver runs the experiment.
* This experiment compares run time and resource consumption of YCSB queries.
* Workload is 'B'.
* Number of rows to insert is 10000000.
* Ordering of inserts is hashed.
* Number of operations is 10000000.
* Batch size is ''.
* Target is based on multiples of '16384'.
* Factors for loading are [4].
* Factors for benchmarking are [4].
* Experiment uses bexhoma version 0.9.5.
* System metrics are monitored by a cluster-wide installation.
* Experiment is limited to DBMS ['PostgreSQL'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker14.
* Loading is tested with [64] threads, split into [8] pods.
* Benchmarking is tested with [64] threads, split into [1, 8] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* PostgreSQL-64-8-65536-1 uses docker image postgres:18.3
* RAM:541008474112
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:5.15.0-164-generic
* node:cl-worker14
* disk:174522
* datadisk:23566
* cpu_list:0-63
* args:['-c', 'max_connections=3000', '-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'effective_io_concurrency=64', '-c', 'io_method=worker', '-c', 'shared_buffers=256GB', '-c', 'effective_cache_size=256GB', '-c', 'work_mem=32GB', '-c', 'maintenance_work_mem=4GB', '-c', 'temp_buffers=4GB', '-c', 'wal_buffers=1GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'fsync=on', '-c', 'wal_compression=on', '-c', 'synchronous_commit=on', '-c', 'max_wal_size=32GB', '-c', 'min_wal_size=32GB', '-c', 'checkpoint_timeout=12h', '-c', 'checkpoint_completion_target=1.0']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1775633754
* PostgreSQL-64-8-65536-2 uses docker image postgres:18.3
* RAM:541008474112
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:5.15.0-164-generic
* node:cl-worker14
* disk:174967
* datadisk:24012
* cpu_list:0-63
* args:['-c', 'max_connections=3000', '-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'effective_io_concurrency=64', '-c', 'io_method=worker', '-c', 'shared_buffers=256GB', '-c', 'effective_cache_size=256GB', '-c', 'work_mem=32GB', '-c', 'maintenance_work_mem=4GB', '-c', 'temp_buffers=4GB', '-c', 'wal_buffers=1GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'fsync=on', '-c', 'wal_compression=on', '-c', 'synchronous_commit=on', '-c', 'max_wal_size=32GB', '-c', 'min_wal_size=32GB', '-c', 'checkpoint_timeout=12h', '-c', 'checkpoint_completion_target=1.0']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1775633754
### Loading
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) |
|:----------------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|
| PostgreSQL-64-8-65536 | 1.00 | 64.00 | 65536.00 | 8.00 | 0.00 | 3320.45 | 3011729.00 | 10000000.00 | 75519.00 |
### Execution
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:------------------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| PostgreSQL-64-8-65536-1 | 1.00 | 64.00 | 65536.00 | 1.00 | 0.00 | 27019.51 | 370103.00 | 9501110.00 | 934.00 | 498890.00 | 792575.00 |
| PostgreSQL-64-8-65536-2 | 1.00 | 64.00 | 65536.00 | 8.00 | 0.00 | 29099.71 | 346523.00 | 9499557.00 | 917.00 | 500443.00 | 783871.00 |
### Workflow
#### Actual
* DBMS PostgreSQL-64-8-65536 - Pods [[8, 1]]
#### Planned
* DBMS PostgreSQL-64-8-65536 - Pods [[1, 8]]
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-64-8-65536-1 | 3873.05 | 7.75 | 19.12 | 29.01 |
| PostgreSQL-64-8-65536-2 | 3873.05 | 7.75 | 19.12 | 29.01 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-64-8-65536-1 | 1261.37 | 0.48 | 0.11 | 0.11 |
| PostgreSQL-64-8-65536-2 | 1261.37 | 0.48 | 0.11 | 0.11 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-64-8-65536-1 | 1748.97 | 5.40 | 19.71 | 31.50 |
| PostgreSQL-64-8-65536-2 | 1609.45 | 5.44 | 20.14 | 32.31 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-64-8-65536-1 | 475.93 | 1.52 | 0.14 | 0.14 |
| PostgreSQL-64-8-65536-2 | 475.93 | 2.60 | 0.13 | 0.13 |
### Tests
* TEST passed: Loading Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Execution Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: contains no FAILED column
Workload C
Workload C is 100% READ.
Example:
bexhoma ycsb \
-dbms PostgreSQL \
-sf 10 \
-xwl c \
-xtb 16384 \
-xnbf 4 \
-xnlf 4 \
-nc 1 \
-ne 1 \
-nlp 8 \
-nlt 64 \
-nbp 1,8 \
-nbt 64 \
-m \
-mc \
-ms $BEXHOMA_MS \
-tr \
-lr 64Gi \
-rr 64Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/doc_ycsb_testcase_c.log
Evaluate Results
doc_ycsb_testcase_c.log
## Show Summary
### Workload
YCSB SF=10
* Type: ycsb
* Duration: 4194s
* Code: 1775638256
* YCSB driver runs the experiment.
* This experiment compares run time and resource consumption of YCSB queries.
* Workload is 'C'.
* Number of rows to insert is 10000000.
* Ordering of inserts is hashed.
* Number of operations is 10000000.
* Batch size is ''.
* Target is based on multiples of '16384'.
* Factors for loading are [4].
* Factors for benchmarking are [4].
* Experiment uses bexhoma version 0.9.5.
* System metrics are monitored by a cluster-wide installation.
* Experiment is limited to DBMS ['PostgreSQL'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker14.
* Loading is tested with [64] threads, split into [8] pods.
* Benchmarking is tested with [64] threads, split into [1, 8] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* PostgreSQL-64-8-65536-1 uses docker image postgres:18.3
* RAM:541008474112
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:5.15.0-164-generic
* node:cl-worker14
* disk:174521
* datadisk:23566
* cpu_list:0-63
* args:['-c', 'max_connections=3000', '-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'effective_io_concurrency=64', '-c', 'io_method=worker', '-c', 'shared_buffers=256GB', '-c', 'effective_cache_size=256GB', '-c', 'work_mem=32GB', '-c', 'maintenance_work_mem=4GB', '-c', 'temp_buffers=4GB', '-c', 'wal_buffers=1GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'fsync=on', '-c', 'wal_compression=on', '-c', 'synchronous_commit=on', '-c', 'max_wal_size=32GB', '-c', 'min_wal_size=32GB', '-c', 'checkpoint_timeout=12h', '-c', 'checkpoint_completion_target=1.0']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1775638256
* PostgreSQL-64-8-65536-2 uses docker image postgres:18.3
* RAM:541008474112
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:5.15.0-164-generic
* node:cl-worker14
* disk:174521
* datadisk:23566
* cpu_list:0-63
* args:['-c', 'max_connections=3000', '-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'effective_io_concurrency=64', '-c', 'io_method=worker', '-c', 'shared_buffers=256GB', '-c', 'effective_cache_size=256GB', '-c', 'work_mem=32GB', '-c', 'maintenance_work_mem=4GB', '-c', 'temp_buffers=4GB', '-c', 'wal_buffers=1GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'fsync=on', '-c', 'wal_compression=on', '-c', 'synchronous_commit=on', '-c', 'max_wal_size=32GB', '-c', 'min_wal_size=32GB', '-c', 'checkpoint_timeout=12h', '-c', 'checkpoint_completion_target=1.0']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1775638256
### Loading
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) |
|:----------------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|
| PostgreSQL-64-8-65536 | 1.00 | 64.00 | 65536.00 | 8.00 | 0.00 | 3318.21 | 3013718.00 | 10000000.00 | 75647.00 |
### Execution
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) |
|:------------------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|
| PostgreSQL-64-8-65536-1 | 1.00 | 64.00 | 65536.00 | 1.00 | 0.00 | 65404.36 | 152895.00 | 10000000.00 | 1169.00 |
| PostgreSQL-64-8-65536-2 | 1.00 | 64.00 | 65536.00 | 8.00 | 0.00 | 65435.23 | 152842.00 | 10000000.00 | 1228.00 |
### Workflow
#### Actual
* DBMS PostgreSQL-64-8-65536 - Pods [[8, 1]]
#### Planned
* DBMS PostgreSQL-64-8-65536 - Pods [[1, 8]]
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-64-8-65536-1 | 3835.27 | 7.33 | 19.08 | 29.01 |
| PostgreSQL-64-8-65536-2 | 3835.27 | 7.33 | 19.08 | 29.01 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-64-8-65536-1 | 1249.78 | 0.45 | 0.11 | 0.11 |
| PostgreSQL-64-8-65536-2 | 1249.78 | 0.45 | 0.11 | 0.11 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-64-8-65536-1 | 1061.62 | 10.94 | 19.20 | 30.61 |
| PostgreSQL-64-8-65536-2 | 1211.77 | 10.67 | 19.20 | 30.61 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-64-8-65536-1 | 476.62 | 4.44 | 0.13 | 0.13 |
| PostgreSQL-64-8-65536-2 | 717.54 | 9.70 | 0.13 | 0.13 |
### Tests
* TEST passed: Loading Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Execution Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: contains no FAILED column
Workload D
Workload D is 95% READ and 5% INSERT.
This means there are more rows in the database after the benchmark than before the benchmark.
The range of key that can be read or inserted changes.
Repetition is only fully sensible after a clean creation of the database.
It is not possible to scale out the driver.
This is because of the latest distribution.
It assumes the keyspace is filled completely up to some recordcount and new data is added straight after this number [1].
[1] https://github.com/brianfrankcooper/YCSB/blob/1e62880552fc95fa4b012846c0f3887420e676a8/core/src/main/java/site/ycsb/workloads/CoreWorkload.java#L491
Example:
bexhoma ycsb \
-dbms PostgreSQL \
-sf 10 \
-xwl d \
-xtb 16384 \
-xnbf 4 \
-xnlf 4 \
-nc 1 \
-ne 1 \
-nlp 8 \
-nlt 64 \
-nbp 1 \
-nbt 64 \
-xio hashed \
-m \
-mc \
-ms $BEXHOMA_MS \
-tr \
-lr 64Gi \
-rr 64Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/doc_ycsb_testcase_d.log
Evaluate Results
doc_ycsb_testcase_d.log
## Show Summary
### Workload
YCSB SF=10
* Type: ycsb
* Duration: 4009s
* Code: 1775642518
* YCSB driver runs the experiment.
* This experiment compares run time and resource consumption of YCSB queries.
* Workload is 'D'.
* Number of rows to insert is 10000000.
* Ordering of inserts is hashed.
* Number of operations is 10000000.
* Batch size is ''.
* Target is based on multiples of '16384'.
* Factors for loading are [4].
* Factors for benchmarking are [4].
* Experiment uses bexhoma version 0.9.5.
* System metrics are monitored by a cluster-wide installation.
* Experiment is limited to DBMS ['PostgreSQL'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker14.
* Loading is tested with [64] threads, split into [8] pods.
* Benchmarking is tested with [64] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* PostgreSQL-64-8-65536-1 uses docker image postgres:18.3
* RAM:541008474112
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:5.15.0-164-generic
* node:cl-worker14
* disk:174520
* datadisk:23564
* cpu_list:0-63
* args:['-c', 'max_connections=3000', '-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'effective_io_concurrency=64', '-c', 'io_method=worker', '-c', 'shared_buffers=256GB', '-c', 'effective_cache_size=256GB', '-c', 'work_mem=32GB', '-c', 'maintenance_work_mem=4GB', '-c', 'temp_buffers=4GB', '-c', 'wal_buffers=1GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'fsync=on', '-c', 'wal_compression=on', '-c', 'synchronous_commit=on', '-c', 'max_wal_size=32GB', '-c', 'min_wal_size=32GB', '-c', 'checkpoint_timeout=12h', '-c', 'checkpoint_completion_target=1.0']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1775642518
### Loading
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) |
|:----------------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|
| PostgreSQL-64-8-65536 | 1.00 | 64.00 | 65536.00 | 8.00 | 0.00 | 3315.55 | 3016248.00 | 10000000.00 | 75519.00 |
### Execution
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) | [READ].Return=OK | [READ].99thPercentileLatency(us) |
|:------------------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|-------------------:|-----------------------------------:|
| PostgreSQL-64-8-65536-1 | 1.00 | 64.00 | 65536.00 | 1.00 | 0.00 | 54056.68 | 184991.00 | 499934.00 | 43807.00 | 9500066.00 | 1509.00 |
### Workflow
#### Actual
* DBMS PostgreSQL-64-8-65536 - Pods [[1]]
#### Planned
* DBMS PostgreSQL-64-8-65536 - Pods [[1]]
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-64-8-65536-1 | 3826.85 | 6.05 | 18.93 | 29.01 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-64-8-65536-1 | 1263.28 | 0.72 | 0.11 | 0.11 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-64-8-65536-1 | 1268.97 | 9.87 | 19.76 | 31.59 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-64-8-65536-1 | 422.82 | 2.99 | 0.13 | 0.13 |
### Tests
* TEST passed: Loading Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Execution Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: contains no FAILED column
Workload E
Workload E is 95% SCAN and 5% INSERT. This means there are more rows in the database after the benchmark than before the benchmark. The range of key that can be read or inserted changes. Repetition is only fully sensible after a clean creation of the database. This workload can be scaled out. Currently this does not seem to work properly. Scans might be done over empty keyspaces.
Example:
bexhoma ycsb \
-dbms PostgreSQL \
-sf 10 \
-xwl e \
-xtb 16384 \
-xnbf 4 \
-xnlf 4 \
-nc 1 \
-ne 1 \
-nlp 8 \
-nlt 64 \
-nbp 1 \
-nbt 64 \
-xio ordered \
-m \
-mc \
-ms $BEXHOMA_MS \
-tr \
-lr 64Gi \
-rr 64Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/doc_ycsb_testcase_e.log
Evaluate Results
doc_ycsb_testcase_e.log
## Show Summary
### Workload
YCSB SF=10
* Type: ycsb
* Duration: 4393s
* Code: 1775661847
* YCSB driver runs the experiment.
* This experiment compares run time and resource consumption of YCSB queries.
* Workload is 'E'.
* Number of rows to insert is 10000000.
* Ordering of inserts is ordered.
* Number of operations is 10000000.
* Batch size is ''.
* Target is based on multiples of '16384'.
* Factors for loading are [4].
* Factors for benchmarking are [4].
* Experiment uses bexhoma version 0.9.5.
* System metrics are monitored by a cluster-wide installation.
* Experiment is limited to DBMS ['PostgreSQL'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker14.
* Loading is tested with [64] threads, split into [8] pods.
* Benchmarking is tested with [64] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* PostgreSQL-64-8-65536-1 uses docker image postgres:18.3
* RAM:541008474112
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:5.15.0-164-generic
* node:cl-worker14
* disk:174302
* datadisk:23344
* cpu_list:0-63
* args:['-c', 'max_connections=3000', '-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'effective_io_concurrency=64', '-c', 'io_method=worker', '-c', 'shared_buffers=256GB', '-c', 'effective_cache_size=256GB', '-c', 'work_mem=32GB', '-c', 'maintenance_work_mem=4GB', '-c', 'temp_buffers=4GB', '-c', 'wal_buffers=1GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'fsync=on', '-c', 'wal_compression=on', '-c', 'synchronous_commit=on', '-c', 'max_wal_size=32GB', '-c', 'min_wal_size=32GB', '-c', 'checkpoint_timeout=12h', '-c', 'checkpoint_completion_target=1.0']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1775661847
### Loading
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) |
|:----------------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|
| PostgreSQL-64-8-65536 | 1.00 | 64.00 | 65536.00 | 8.00 | 0.00 | 3271.91 | 3056400.00 | 10000000.00 | 83847.00 |
### Execution
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) | [SCAN].Return=OK | [SCAN].99thPercentileLatency(us) |
|:------------------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|-------------------:|-----------------------------------:|
| PostgreSQL-64-8-65536-1 | 1.00 | 64.00 | 65536.00 | 1.00 | 0.00 | 19903.43 | 502426.00 | 499865.00 | 68735.00 | 9500135.00 | 7419.00 |
### Workflow
#### Actual
* DBMS PostgreSQL-64-8-65536 - Pods [[1]]
#### Planned
* DBMS PostgreSQL-64-8-65536 - Pods [[1]]
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-64-8-65536-1 | 3718.57 | 6.05 | 18.93 | 29.03 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-64-8-65536-1 | 1222.32 | 0.47 | 0.11 | 0.11 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-64-8-65536-1 | 7545.53 | 18.32 | 18.43 | 30.34 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-64-8-65536-1 | 4325.21 | 10.95 | 0.14 | 0.14 |
### Tests
* TEST passed: Loading Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Execution Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: contains no FAILED column
Workload F
Workload F is 50% READ and 50% READ-MODIFY-WRITE.
Example:
bexhoma ycsb \
-dbms PostgreSQL \
-sf 10 \
-xwl f \
-xtb 16384 \
-xnbf 4 \
-xnlf 4 \
-nc 1 \
-ne 1 \
-nlp 8 \
-nlt 64 \
-nbp 1,8 \
-nbt 64 \
-m \
-mc \
-ms $BEXHOMA_MS \
-tr \
-lr 64Gi \
-rr 64Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/doc_ycsb_testcase_f.log
Evaluate Results
doc_ycsb_testcase_f.log
## Show Summary
### Workload
YCSB SF=10
* Type: ycsb
* Duration: 10914s
* Code: 1775650862
* YCSB driver runs the experiment.
* This experiment compares run time and resource consumption of YCSB queries.
* Workload is 'F'.
* Number of rows to insert is 10000000.
* Ordering of inserts is hashed.
* Number of operations is 10000000.
* Batch size is ''.
* Target is based on multiples of '16384'.
* Factors for loading are [4].
* Factors for benchmarking are [4].
* Experiment uses bexhoma version 0.9.5.
* System metrics are monitored by a cluster-wide installation.
* Experiment is limited to DBMS ['PostgreSQL'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker14.
* Loading is tested with [64] threads, split into [8] pods.
* Benchmarking is tested with [64] threads, split into [1, 8] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* PostgreSQL-64-8-65536-1 uses docker image postgres:18.3
* RAM:541008474112
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:5.15.0-164-generic
* node:cl-worker14
* disk:174522
* datadisk:23565
* cpu_list:0-63
* args:['-c', 'max_connections=3000', '-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'effective_io_concurrency=64', '-c', 'io_method=worker', '-c', 'shared_buffers=256GB', '-c', 'effective_cache_size=256GB', '-c', 'work_mem=32GB', '-c', 'maintenance_work_mem=4GB', '-c', 'temp_buffers=4GB', '-c', 'wal_buffers=1GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'fsync=on', '-c', 'wal_compression=on', '-c', 'synchronous_commit=on', '-c', 'max_wal_size=32GB', '-c', 'min_wal_size=32GB', '-c', 'checkpoint_timeout=12h', '-c', 'checkpoint_completion_target=1.0']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1775650862
* PostgreSQL-64-8-65536-2 uses docker image postgres:18.3
* RAM:541008474112
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:5.15.0-164-generic
* node:cl-worker14
* disk:179162
* datadisk:28206
* cpu_list:0-63
* args:['-c', 'max_connections=3000', '-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'effective_io_concurrency=64', '-c', 'io_method=worker', '-c', 'shared_buffers=256GB', '-c', 'effective_cache_size=256GB', '-c', 'work_mem=32GB', '-c', 'maintenance_work_mem=4GB', '-c', 'temp_buffers=4GB', '-c', 'wal_buffers=1GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'fsync=on', '-c', 'wal_compression=on', '-c', 'synchronous_commit=on', '-c', 'max_wal_size=32GB', '-c', 'min_wal_size=32GB', '-c', 'checkpoint_timeout=12h', '-c', 'checkpoint_completion_target=1.0']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1775650862
### Loading
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) |
|:----------------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|
| PostgreSQL-64-8-65536 | 1.00 | 64.00 | 65536.00 | 8.00 | 0.00 | 3342.01 | 2992350.00 | 10000000.00 | 68767.00 |
### Execution
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) | [READ-MODIFY-WRITE].Operations | [READ-MODIFY-WRITE].99thPercentileLatency(us) |
|:------------------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|---------------------------------:|------------------------------------------------:|
| PostgreSQL-64-8-65536-1 | 1.00 | 64.00 | 65536.00 | 1.00 | 0.00 | 2843.39 | 3516923.00 | 10000000.00 | 895.00 | 4998327.00 | 905215.00 | 4998327.00 | 905727.00 |
| PostgreSQL-64-8-65536-2 | 1.00 | 64.00 | 65536.00 | 8.00 | 0.00 | 2847.82 | 3536556.00 | 10000000.00 | 854.00 | 4997726.00 | 911871.00 | 4997726.00 | 912383.00 |
### Workflow
#### Actual
* DBMS PostgreSQL-64-8-65536 - Pods [[8, 1]]
#### Planned
* DBMS PostgreSQL-64-8-65536 - Pods [[1, 8]]
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-64-8-65536-1 | 3945.26 | 6.51 | 19.07 | 29.02 |
| PostgreSQL-64-8-65536-2 | 3945.26 | 6.51 | 19.07 | 29.02 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-64-8-65536-1 | 1232.69 | 0.44 | 0.11 | 0.11 |
| PostgreSQL-64-8-65536-2 | 1232.69 | 0.44 | 0.11 | 0.11 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-64-8-65536-1 | 4117.48 | 1.50 | 22.29 | 36.25 |
| PostgreSQL-64-8-65536-2 | 3945.47 | 1.19 | 23.86 | 39.20 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-64-8-65536-1 | 1113.46 | 0.34 | 0.13 | 0.13 |
| PostgreSQL-64-8-65536-2 | 1113.46 | 0.60 | 0.12 | 0.13 |
### Tests
* TEST passed: Loading Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Execution Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: contains no FAILED column