Test Cases
There is a variety of combinations of options that can be tested.
We here list some more basic use cases to test the functionality of bexhoma.
See repository for implementations. You will have to change the node selectors there (to names of nodes, that exist in your cluster - or to leave out the corresponding parameters):
BEXHOMA_NODE_SUT="cl-worker11"
BEXHOMA_NODE_LOAD="cl-worker19"
BEXHOMA_NODE_BENCHMARK="cl-worker19"
LOG_DIR="./logs_tests"
BEXHOMA_MS=1
BEXHOMA_STORAGE_CLASS="shared"
mkdir -p $LOG_DIR
See also more test cases for more and longer running test cases and other DBMS.
See the log folder for some demo test logs.
The folder also contains \*_summary.txt files containing only the result summary.
TPC-H
Compare
TPC-H Compare
bexhoma tpch \
-sf 1 \
-nlp 8 \
-nlt 8 \
-xii -xic -xis \
-xdt \
-ms $BEXHOMA_MS \
-tr \
-lr 128Gi \
-rr 128Gi \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_tpch_compare.log
yields (after ca. 120 minutes) something like
testcase_tpch_compare.log
## Show Summary
### Workload
TPC-H Queries SF=1
* Type: tpch
* Duration: 1432s
* Code: 1782328952
* This includes the reading queries of TPC-H.
* This experiment compares run time and resource consumption of TPC-H queries in different DBMS.
* TPC-H (SF=1) data is loaded and benchmark is executed.
* Query ordering is Q1 - Q22.
* All instances use the same query parameters.
* Timeout per query is 600.
* Import sets indexes and constraints after loading and recomputes statistics.
* Experiment uses bexhoma version 0.10.0.
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker38.
* Loading is tested with [8] threads, split into [8] pods.
* Benchmarking is tested with [1] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* MariaDB-1-1-1-1-1 uses docker image mariadb:11.4.7
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:261589
* cpu_list:0-127
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1782328952
* MonetDB-1-1-1-1-1 uses docker image monetdb/monetdb:Dec2025-SP1
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:231054
* cpu_list:0-127
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1782328952
* MySQL-1-1-1-1-1 uses docker image mysql:8.4.0
* RAM:540492877824
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:261592
* cpu_list:0-127
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1782328952
* PostgreSQL-1-1-1-1-1 uses docker image postgres:18.3
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:232015
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1782328952
### Workflow
#### Actual
* DBMS MariaDB-1 - Experiment 1 Client 1: tpch (1 pods)
* DBMS MonetDB-1 - Experiment 1 Client 1: tpch (1 pods)
* DBMS MySQL-1 - Experiment 1 Client 1: tpch (1 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 1: tpch (1 pods)
#### Planned
* DBMS PostgreSQL-1 - Experiment 1 Client 1: tpch (1 pods)
* DBMS MonetDB-1 - Experiment 1 Client 1: tpch (1 pods)
* DBMS MariaDB-1 - Experiment 1 Client 1: tpch (1 pods)
* DBMS MySQL-1 - Experiment 1 Client 1: tpch (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:---------------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| MariaDB-1-1 | 1 | 1 | 443.00 | 1.00 | 23.00 | 83.00 | 333.00 | 8 | 0 | | None | 0 | False | 8.13 |
| MonetDB-1-1 | 1 | 1 | 234.00 | 2.00 | 18.00 | 31.00 | 181.00 | 8 | 0 | | None | 0 | False | 15.38 |
| MySQL-1-1 | 1 | 1 | 301.00 | 2.00 | 21.00 | 28.00 | 246.00 | 8 | 0 | | None | 0 | False | 11.96 |
| PostgreSQL-1-1 | 1 | 1 | 341.00 | 1.00 | 21.00 | 44.00 | 269.00 | 8 | 0 | | | 0 | False | 10.56 |
### Execution
#### Per Connection
| | configuration | phase | job | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id | pod |
|:---------------------|:----------------|:-----------------|:-------------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|:---------------------|
| MariaDB-1-1-1-1-1 | MariaDB-1 | MariaDB-1-1-1 | MariaDB-1-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 22 | 112 | 1.12 | 3430.02 | 707.14 | -1 | MariaDB-1-1-1-1-1 |
| MonetDB-1-1-1-1-1 | MonetDB-1 | MonetDB-1-1-1 | MonetDB-1-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 22 | 84 | 0.11 | 39418.11 | 942.86 | -1 | MonetDB-1-1-1-1-1 |
| MySQL-1-1-1-1-1 | MySQL-1 | MySQL-1-1-1 | MySQL-1-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 22 | 67 | 1.02 | 3685.84 | 1182.09 | -1 | MySQL-1-1-1-1-1 |
| PostgreSQL-1-1-1-1-1 | PostgreSQL-1 | PostgreSQL-1-1-1 | PostgreSQL-1-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 22 | 14 | 0.32 | 12056.34 | 5657.14 | 0 | PostgreSQL-1-1-1-1-1 |
#### Per Phase
| | phase | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id |
|:-----------------|:-----------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|
| MariaDB-1-1-1 | MariaDB-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 22 | 112 | 1.12 | 3430.02 | 707.14 | -1 |
| MonetDB-1-1-1 | MonetDB-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 22 | 84 | 0.11 | 39418.11 | 942.86 | -1 |
| MySQL-1-1-1 | MySQL-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 22 | 67 | 1.02 | 3685.84 | 1182.09 | -1 |
| PostgreSQL-1-1-1 | PostgreSQL-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 22 | 14 | 0.32 | 12056.34 | 5657.14 | 0 |
### Latency of Timer Execution [ms]
| Queries | MariaDB-1-1-1-1-1 | MonetDB-1-1-1-1-1 | MySQL-1-1-1-1-1 | PostgreSQL-1-1-1-1-1 |
|:----------------------------------------------------|--------------------:|--------------------:|------------------:|-----------------------:|
| Pricing Summary Report (TPC-H Q1) | 8820.34 | 614.68 | 7905.24 | 1097.89 |
| Minimum Cost Supplier Query (TPC-H Q2) | 522.07 | 38.23 | 102.52 | 249.21 |
| Shipping Priority (TPC-H Q3) | 1629.75 | 89.61 | 1370.22 | 314.21 |
| Order Priority Checking Query (TPC-H Q4) | 357.93 | 165.66 | 474.27 | 153.04 |
| Local Supplier Volume (TPC-H Q5) | 1101.91 | 73.61 | 1014.53 | 302.16 |
| Forecasting Revenue Change (TPC-H Q6) | 1040.46 | 23.98 | 1167.47 | 192.42 |
| Volume Shipping Query (TPC-H Q7) | 1132.46 | 51.69 | 814.53 | 326.12 |
| National Market Share (TPC-H Q8) | 1935.03 | 348.10 | 2870.65 | 191.40 |
| Product Type Profit Measure (TPC-H Q9) | 2095.02 | 81.75 | 2291.24 | 494.87 |
| Returned Item Reporting Query (TPC-H Q10) | 896.21 | 128.73 | 1132.40 | 592.82 |
| Important Stock Identification (TPC-H Q11) | 167.13 | 24.36 | 158.33 | 81.88 |
| Shipping Modes and Order Priority (TPC-H Q12) | 3055.45 | 45.58 | 2003.53 | 314.06 |
| Customer Distribution (TPC-H Q13) | 3098.81 | 252.96 | 3612.46 | 1170.01 |
| Promotion Effect Query (TPC-H Q14) | 12437.81 | 41.74 | 1487.79 | 231.00 |
| Top Supplier Query (TPC-H Q15) | 1799.15 | 38.74 | 13742.99 | 234.21 |
| Parts/Supplier Relationship (TPC-H Q16) | 259.05 | 85.78 | 468.76 | 281.65 |
| Small-Quantity-Order Revenue (TPC-H Q17) | 58.79 | 54.63 | 302.97 | 773.02 |
| Large Volume Customer (TPC-H Q18) | 2990.93 | 164.98 | 1727.31 | 2777.70 |
| Discounted Revenue (TPC-H Q19) | 94.40 | 99.96 | 141.71 | 52.78 |
| Potential Part Promotion (TPC-H Q20) | 208.96 | 44.33 | 288.36 | 120.95 |
| Suppliers Who Kept Orders Waiting Query (TPC-H Q21) | 56730.98 | 1276.72 | 4682.39 | 319.37 |
| Global Sales Opportunity Query (TPC-H Q22) | 126.72 | 58.68 | 143.07 | 100.69 |
### Errors (failed queries)
No errors
### Warnings (result mismatch)
No warnings
### Tests
* TEST passed: Geo Times [s] contains no 0 or NaN
* TEST passed: Power@Size [~Q/h] contains no 0 or NaN
* TEST passed: Throughput@Size contains no 0 or NaN
* TEST passed: No SQL errors
* TEST passed: No SQL warnings
* TEST passed: Workflow as planned
PostgreSQL
TPC-H Simple
bexhoma tpch \
-dbms PostgreSQL \
-sf 1 \
-nlp 8 \
-nlt 8 \
-xii -xic -xis \
-xdt \
-nc 1 \
-ne 1 \
-nbp 1 \
-ms $BEXHOMA_MS \
-t 1200 \
-tr \
-lr 128Gi \
-rr 128Gi \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_tpch_postgresql_1.log
yields (after ca. 10 minutes) something like
testcase_tpch_postgresql_1.log
## Show Summary
### Workload
TPC-H Queries SF=1
* Type: tpch
* Duration: 555s
* Code: 1782332834
* This includes the reading queries of TPC-H.
* This experiment compares run time and resource consumption of TPC-H queries in different DBMS.
* TPC-H (SF=1) data is loaded and benchmark is executed.
* Query ordering is Q1 - Q22.
* All instances use the same query parameters.
* Timeout per query is 1200.
* Import sets indexes and constraints after loading and recomputes statistics.
* Experiment uses bexhoma version 0.10.0.
* Experiment is limited to DBMS ['PostgreSQL'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker38.
* Loading is tested with [8] threads, split into [8] pods.
* Benchmarking is tested with [1] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* PostgreSQL-1-1-1-1-1 uses docker image postgres:18.3
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:265619
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782332834
### Workflow
#### Actual
* DBMS PostgreSQL-1 - Experiment 1 Client 1: tpch (1 pods)
#### Planned
* DBMS PostgreSQL-1 - Experiment 1 Client 1: tpch (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:---------------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| PostgreSQL-1-1 | 1 | 1 | 241.00 | 1.00 | 20.00 | 7.00 | 209.00 | 8 | 0 | | | 0 | False | 14.94 |
### Execution
#### Per Connection
| | configuration | phase | job | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id | pod |
|:---------------------|:----------------|:-----------------|:-------------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|:---------------------|
| PostgreSQL-1-1-1-1-1 | PostgreSQL-1 | PostgreSQL-1-1-1 | PostgreSQL-1-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 22 | 15 | 0.33 | 11626.62 | 5280.00 | 0 | PostgreSQL-1-1-1-1-1 |
#### Per Phase
| | phase | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id |
|:-----------------|:-----------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|
| PostgreSQL-1-1-1 | PostgreSQL-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 22 | 15 | 0.33 | 11626.62 | 5280.00 | 0 |
### Latency of Timer Execution [ms]
| Queries | PostgreSQL-1-1-1-1-1 |
|:----------------------------------------------------|-----------------------:|
| Pricing Summary Report (TPC-H Q1) | 1147.95 |
| Minimum Cost Supplier Query (TPC-H Q2) | 275.59 |
| Shipping Priority (TPC-H Q3) | 337.22 |
| Order Priority Checking Query (TPC-H Q4) | 148.22 |
| Local Supplier Volume (TPC-H Q5) | 287.51 |
| Forecasting Revenue Change (TPC-H Q6) | 186.53 |
| Volume Shipping Query (TPC-H Q7) | 339.46 |
| National Market Share (TPC-H Q8) | 180.79 |
| Product Type Profit Measure (TPC-H Q9) | 476.76 |
| Returned Item Reporting Query (TPC-H Q10) | 535.35 |
| Important Stock Identification (TPC-H Q11) | 93.17 |
| Shipping Modes and Order Priority (TPC-H Q12) | 312.49 |
| Customer Distribution (TPC-H Q13) | 954.68 |
| Promotion Effect Query (TPC-H Q14) | 368.14 |
| Top Supplier Query (TPC-H Q15) | 250.94 |
| Parts/Supplier Relationship (TPC-H Q16) | 291.70 |
| Small-Quantity-Order Revenue (TPC-H Q17) | 809.24 |
| Large Volume Customer (TPC-H Q18) | 3603.30 |
| Discounted Revenue (TPC-H Q19) | 53.53 |
| Potential Part Promotion (TPC-H Q20) | 147.53 |
| Suppliers Who Kept Orders Waiting Query (TPC-H Q21) | 299.18 |
| Global Sales Opportunity Query (TPC-H Q22) | 91.77 |
### Errors (failed queries)
No errors
### Warnings (result mismatch)
No warnings
### Tests
* TEST passed: Geo Times [s] contains no 0 or NaN
* TEST passed: Power@Size [~Q/h] contains no 0 or NaN
* TEST passed: Throughput@Size contains no 0 or NaN
* TEST passed: No SQL errors
* TEST passed: No SQL warnings
* TEST passed: Workflow as planned
TPC-H Monitoring
bexhoma tpch \
-dbms PostgreSQL \
-sf 10 \
-nlp 8 \
-nlt 8 \
-xii -xic -xis \
-xdt \
-nc 1 \
-ne 1 \
-nbp 1 \
-m \
-mc \
-ms $BEXHOMA_MS \
-t 1200 \
-tr \
-lr 128Gi \
-rr 128Gi \
-rss 150Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_tpch_postgresql_2.log
yields (after ca. 15 minutes) something like
testcase_tpch_postgresql_2.log
## Show Summary
### Workload
TPC-H Queries SF=10
* Type: tpch
* Duration: 954s
* Code: 1782333413
* This includes the reading queries of TPC-H.
* This experiment compares run time and resource consumption of TPC-H queries in different DBMS.
* TPC-H (SF=10) data is loaded and benchmark is executed.
* Query ordering is Q1 - Q22.
* All instances use the same query parameters.
* Timeout per query is 1200.
* Import sets indexes and constraints after loading and recomputes statistics.
* Experiment uses bexhoma version 0.10.0.
* System metrics are monitored by a cluster-wide installation.
* Experiment is limited to DBMS ['PostgreSQL'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker38.
* Loading is tested with [8] threads, split into [8] pods.
* Benchmarking is tested with [1] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* PostgreSQL-1-1-1-1-1 uses docker image postgres:18.3
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:254316
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:128Gi
* limits_memory:128Gi
* eval_parameters
* code:1782333413
### Workflow
#### Actual
* DBMS PostgreSQL-1 - Experiment 1 Client 1: tpch (1 pods)
#### Planned
* DBMS PostgreSQL-1 - Experiment 1 Client 1: tpch (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:---------------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| PostgreSQL-1-1 | 1 | 10 | 619.00 | 1.00 | 22.00 | 75.00 | 518.00 | 8 | 0 | | | 0 | False | 58.16 |
### Execution
#### Per Connection
| | configuration | phase | job | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id | pod |
|:---------------------|:----------------|:-----------------|:-------------------|-----------------:|---------:|----------------:|------------:|------:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|:---------------------|
| PostgreSQL-1-1-1-1-1 | PostgreSQL-1 | PostgreSQL-1-1-1 | PostgreSQL-1-1-1-1 | 1 | 1 | 1 | 1 | 10.00 | 22 | 94 | 2.49 | 14860.42 | 8425.53 | 0 | PostgreSQL-1-1-1-1-1 |
#### Per Phase
| | phase | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id |
|:-----------------|:-----------------|-----------------:|---------:|----------------:|------------:|------:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|
| PostgreSQL-1-1-1 | PostgreSQL-1-1-1 | 1 | 1 | 1 | 1 | 10.00 | 22 | 94 | 2.49 | 14860.42 | 8425.53 | 0 |
### Latency of Timer Execution [ms]
| Queries | PostgreSQL-1-1-1-1-1 |
|:----------------------------------------------------|-----------------------:|
| Pricing Summary Report (TPC-H Q1) | 5885.46 |
| Minimum Cost Supplier Query (TPC-H Q2) | 2817.11 |
| Shipping Priority (TPC-H Q3) | 3476.09 |
| Order Priority Checking Query (TPC-H Q4) | 1092.58 |
| Local Supplier Volume (TPC-H Q5) | 2667.90 |
| Forecasting Revenue Change (TPC-H Q6) | 1433.63 |
| Volume Shipping Query (TPC-H Q7) | 2433.04 |
| National Market Share (TPC-H Q8) | 1421.55 |
| Product Type Profit Measure (TPC-H Q9) | 6231.35 |
| Returned Item Reporting Query (TPC-H Q10) | 2514.16 |
| Important Stock Identification (TPC-H Q11) | 869.66 |
| Shipping Modes and Order Priority (TPC-H Q12) | 1972.85 |
| Customer Distribution (TPC-H Q13) | 11093.08 |
| Promotion Effect Query (TPC-H Q14) | 1839.57 |
| Top Supplier Query (TPC-H Q15) | 1805.42 |
| Parts/Supplier Relationship (TPC-H Q16) | 1387.64 |
| Small-Quantity-Order Revenue (TPC-H Q17) | 7635.49 |
| Large Volume Customer (TPC-H Q18) | 24063.03 |
| Discounted Revenue (TPC-H Q19) | 319.89 |
| Potential Part Promotion (TPC-H Q20) | 3827.27 |
| Suppliers Who Kept Orders Waiting Query (TPC-H Q21) | 2487.12 |
| Global Sales Opportunity Query (TPC-H Q22) | 444.12 |
### Errors (failed queries)
No errors
### Warnings (result mismatch)
No warnings
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 703.73 | 4.45 | 5.76 | 20.47 |
### Loading phase: component data generator
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 0.00 | 0.00 | 0.00 | 0.00 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 82.82 | 1.50 | 0.00 | 0.93 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 301.30 | 3.52 | 16.21 | 31.50 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 16.67 | 0.11 | 0.33 | 0.33 |
### Tests
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST skipped: Loading phase: component data generator contains 0 or NaN in CPU [CPUs] (data pre-existing)
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Geo Times [s] contains no 0 or NaN
* TEST passed: Power@Size [~Q/h] contains no 0 or NaN
* TEST passed: Throughput@Size contains no 0 or NaN
* TEST passed: No SQL errors
* TEST passed: No SQL warnings
* TEST passed: Workflow as planned
TPC-H Throughput Test
bexhoma tpch \
-dbms PostgreSQL \
-sf 10 \
-nlp 8 \
-nlt 8 \
-xii -xic -xis \
-xdt \
-nc 2 \
-ne 1,2 \
-nbp 1 \
-m \
-mc \
-ms $BEXHOMA_MS \
-t 1200 \
-tr \
-lr 128Gi \
-rr 128Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rss 150Gi \
-rsr \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_tpch_postgresql_3.log
yields (after ca. 15 minutes) something like
testcase_tpch_postgresql_3.log
## Show Summary
### Workload
TPC-H Queries SF=10
* Type: tpch
* Duration: 2173s
* Code: 1782334451
* This includes the reading queries of TPC-H.
* This experiment compares run time and resource consumption of TPC-H queries in different DBMS.
* TPC-H (SF=10) data is loaded and benchmark is executed.
* Query ordering is Q1 - Q22.
* All instances use the same query parameters.
* Timeout per query is 1200.
* Import sets indexes and constraints after loading and recomputes statistics.
* Experiment uses bexhoma version 0.10.0.
* System metrics are monitored by a cluster-wide installation.
* Experiment is limited to DBMS ['PostgreSQL'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker38.
* Database is persisted to disk of type shared and size 50Gi. Persistent storage is removed at experiment start.
* Loading is tested with [8] threads, split into [8] pods.
* Benchmarking is tested with [1] threads, split into [1] pods.
* Benchmarking is run as [1, 2] times the number of benchmarking pods.
* Experiment is run 2 times.
### Connections
* PostgreSQL-1-1-1-1-1 uses docker image postgres:18.3
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220467
* volume_size:50G
* volume_used:27G
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:128Gi
* limits_memory:128Gi
* eval_parameters
* code:1782334451
* PostgreSQL-1-1-2-1-1 uses docker image postgres:18.3
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220467
* volume_size:50G
* volume_used:27G
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:128Gi
* limits_memory:128Gi
* eval_parameters
* code:1782334451
* PostgreSQL-1-1-2-1-2 uses docker image postgres:18.3
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220467
* volume_size:50G
* volume_used:27G
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:128Gi
* limits_memory:128Gi
* eval_parameters
* code:1782334451
* PostgreSQL-1-2-1-1-1 uses docker image postgres:18.3
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220467
* volume_size:50G
* volume_used:27G
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:128Gi
* limits_memory:128Gi
* eval_parameters
* code:1782334451
* PostgreSQL-1-2-2-1-1 uses docker image postgres:18.3
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220467
* volume_size:50G
* volume_used:27G
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:128Gi
* limits_memory:128Gi
* eval_parameters
* code:1782334451
* PostgreSQL-1-2-2-1-2 uses docker image postgres:18.3
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220467
* volume_size:50G
* volume_used:27G
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:128Gi
* limits_memory:128Gi
* eval_parameters
* code:1782334451
### Workflow
#### Actual
* DBMS PostgreSQL-1 - Experiment 1 Client 1: tpch (1 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 2: tpch (2 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 1: tpch (1 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 2: tpch (2 pods)
#### Planned
* DBMS PostgreSQL-1 - Experiment 1 Client 1: tpch (1 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 2: tpch (2 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 1: tpch (1 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 2: tpch (2 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:---------------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| PostgreSQL-1-1 | 1 | 10 | 1058.00 | 3.00 | 31.00 | 278.00 | 741.00 | 8 | 0 | | | 0 | False | 34.03 |
| PostgreSQL-1-2 | 2 | 10 | 1058.00 | 3.00 | 31.00 | 278.00 | 741.00 | 8 | 0 | | | 0 | False | 34.03 |
### Execution
#### Per Connection
| | configuration | phase | job | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id | pod |
|:---------------------|:----------------|:-----------------|:-------------------|-----------------:|---------:|----------------:|------------:|------:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|:---------------------|
| PostgreSQL-1-1-1-1-1 | PostgreSQL-1 | PostgreSQL-1-1-1 | PostgreSQL-1-1-1-1 | 1 | 1 | 1 | 1 | 10.00 | 22 | 97 | 2.61 | 14166.28 | 8164.95 | 0 | PostgreSQL-1-1-1-1-1 |
| PostgreSQL-1-1-2-1-1 | PostgreSQL-1 | PostgreSQL-1-1-2 | PostgreSQL-1-1-2-1 | 1 | 2 | 1 | 1 | 10.00 | 22 | 97 | 2.51 | 14818.73 | 8164.95 | 0 | PostgreSQL-1-1-2-1-1 |
| PostgreSQL-1-1-2-1-2 | PostgreSQL-1 | PostgreSQL-1-1-2 | PostgreSQL-1-1-2-1 | 1 | 2 | 1 | 1 | 10.00 | 22 | 96 | 2.51 | 14834.72 | 8250.00 | 0 | PostgreSQL-1-1-2-1-2 |
| PostgreSQL-1-2-1-1-1 | PostgreSQL-1 | PostgreSQL-1-2-1 | PostgreSQL-1-2-1-1 | 2 | 1 | 1 | 1 | 10.00 | 22 | 237 | 4.11 | 9042.36 | 3341.77 | 0 | PostgreSQL-1-2-1-1-1 |
| PostgreSQL-1-2-2-1-1 | PostgreSQL-1 | PostgreSQL-1-2-2 | PostgreSQL-1-2-2-1 | 2 | 2 | 1 | 1 | 10.00 | 22 | 97 | 2.36 | 15746.08 | 8164.95 | 0 | PostgreSQL-1-2-2-1-1 |
| PostgreSQL-1-2-2-1-2 | PostgreSQL-1 | PostgreSQL-1-2-2 | PostgreSQL-1-2-2-1 | 2 | 2 | 1 | 1 | 10.00 | 22 | 91 | 2.33 | 15921.19 | 8703.30 | 0 | PostgreSQL-1-2-2-1-2 |
#### Per Phase
| | phase | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id |
|:-----------------|:-----------------|-----------------:|---------:|----------------:|------------:|------:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|
| PostgreSQL-1-1-1 | PostgreSQL-1-1-1 | 1 | 1 | 1 | 1 | 10.00 | 22 | 97 | 2.61 | 14166.28 | 8164.95 | 0 |
| PostgreSQL-1-1-2 | PostgreSQL-1-1-2 | 1 | 2 | 1 | 2 | 10.00 | 44 | 97 | 2.51 | 14826.72 | 16329.90 | 0 |
| PostgreSQL-1-2-1 | PostgreSQL-1-2-1 | 2 | 1 | 1 | 1 | 10.00 | 22 | 237 | 4.11 | 9042.36 | 3341.77 | 0 |
| PostgreSQL-1-2-2 | PostgreSQL-1-2-2 | 2 | 2 | 1 | 2 | 10.00 | 44 | 97 | 2.35 | 15833.39 | 16329.90 | 0 |
### Latency of Timer Execution [ms]
| Queries | PostgreSQL-1-1-1-1-1 | PostgreSQL-1-1-2-1-1 | PostgreSQL-1-1-2-1-2 | PostgreSQL-1-2-1-1-1 | PostgreSQL-1-2-2-1-1 | PostgreSQL-1-2-2-1-2 |
|:----------------------------------------------------|-----------------------:|-----------------------:|-----------------------:|-----------------------:|-----------------------:|-----------------------:|
| Pricing Summary Report (TPC-H Q1) | 5926.36 | 6531.76 | 6127.09 | 54526.81 | 5925.93 | 6040.47 |
| Minimum Cost Supplier Query (TPC-H Q2) | 2988.69 | 3379.46 | 2938.73 | 51102.35 | 2657.01 | 3039.43 |
| Shipping Priority (TPC-H Q3) | 3174.99 | 2675.75 | 2567.53 | 32686.61 | 2002.60 | 2181.31 |
| Order Priority Checking Query (TPC-H Q4) | 1034.29 | 1014.10 | 1014.10 | 1178.37 | 893.73 | 802.55 |
| Local Supplier Volume (TPC-H Q5) | 2781.10 | 2396.17 | 2556.43 | 4513.11 | 2554.06 | 2157.31 |
| Forecasting Revenue Change (TPC-H Q6) | 1392.03 | 1315.23 | 1405.32 | 1501.46 | 1221.00 | 1251.16 |
| Volume Shipping Query (TPC-H Q7) | 2324.60 | 2391.55 | 2388.02 | 2585.10 | 2232.32 | 2198.49 |
| National Market Share (TPC-H Q8) | 1525.71 | 1202.62 | 1240.08 | 12725.63 | 1136.64 | 1217.53 |
| Product Type Profit Measure (TPC-H Q9) | 5772.25 | 5463.60 | 5423.10 | 5986.04 | 5517.99 | 5437.40 |
| Returned Item Reporting Query (TPC-H Q10) | 2624.55 | 2460.89 | 2195.44 | 2487.83 | 2336.53 | 2386.97 |
| Important Stock Identification (TPC-H Q11) | 1105.06 | 883.79 | 913.12 | 992.60 | 976.82 | 819.32 |
| Shipping Modes and Order Priority (TPC-H Q12) | 2239.08 | 2236.89 | 1872.40 | 1962.66 | 1880.95 | 1843.78 |
| Customer Distribution (TPC-H Q13) | 11070.89 | 12422.30 | 12461.49 | 10733.74 | 10857.93 | 11053.31 |
| Promotion Effect Query (TPC-H Q14) | 2067.61 | 1988.61 | 2078.07 | 2006.54 | 1814.19 | 1950.73 |
| Top Supplier Query (TPC-H Q15) | 1982.54 | 1819.13 | 2147.15 | 1844.39 | 1888.51 | 1853.00 |
| Parts/Supplier Relationship (TPC-H Q16) | 1516.21 | 1309.24 | 1607.68 | 1475.63 | 1422.98 | 1456.61 |
| Small-Quantity-Order Revenue (TPC-H Q17) | 7925.07 | 7907.19 | 7454.49 | 7783.33 | 7338.59 | 6920.17 |
| Large Volume Customer (TPC-H Q18) | 25308.74 | 25380.95 | 25227.39 | 25541.17 | 29604.33 | 24955.05 |
| Discounted Revenue (TPC-H Q19) | 335.50 | 316.29 | 321.96 | 342.94 | 316.74 | 313.72 |
| Potential Part Promotion (TPC-H Q20) | 4378.97 | 4356.45 | 3903.93 | 4257.82 | 3729.18 | 3515.04 |
| Suppliers Who Kept Orders Waiting Query (TPC-H Q21) | 2616.71 | 2911.91 | 2844.06 | 2628.06 | 2321.61 | 2447.35 |
| Global Sales Opportunity Query (TPC-H Q22) | 512.99 | 442.54 | 484.28 | 511.15 | 477.77 | 505.32 |
### Errors (failed queries)
No errors
### Warnings (result mismatch)
No warnings
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 777.15 | 2.60 | 5.52 | 25.13 |
| PostgreSQL-1-1-2-1 | 777.15 | 2.60 | 5.52 | 25.13 |
### Loading phase: component data generator
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 0.02 | 0.00 | 0.00 | 0.00 |
| PostgreSQL-1-1-2-1 | 0.02 | 0.00 | 0.00 | 0.00 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 112.16 | 1.65 | 0.01 | 1.28 |
| PostgreSQL-1-1-2-1 | 112.16 | 1.65 | 0.01 | 1.28 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 291.38 | 4.35 | 16.91 | 33.82 |
| PostgreSQL-1-1-2-1 | 591.61 | 10.29 | 16.46 | 33.17 |
| PostgreSQL-1-2-1-1 | 1782.80 | 3.60 | 21.37 | 35.90 |
| PostgreSQL-1-2-2-1 | 567.27 | 11.08 | 15.53 | 31.77 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 17.12 | 0.38 | 0.33 | 0.34 |
| PostgreSQL-1-1-2-1 | 35.56 | 0.42 | 0.34 | 0.34 |
| PostgreSQL-1-2-1-1 | 18.63 | 0.13 | 0.33 | 0.33 |
| PostgreSQL-1-2-2-1 | 34.67 | 0.81 | 0.34 | 0.34 |
### Tests
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component data generator contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Geo Times [s] contains no 0 or NaN
* TEST passed: Power@Size [~Q/h] contains no 0 or NaN
* TEST passed: Throughput@Size contains no 0 or NaN
* TEST passed: No SQL errors
* TEST passed: No SQL warnings
* TEST passed: Workflow as planned
TPC-H RAM Disk Test
This loads TPC-H data (SF=3) into a database that is stored on a RAM disk. The disk has size 50GB. Make sure you have enough RAM.
bexhoma tpch \
-dbms PostgreSQL \
-sf 3 \
-nlp 8 \
-nlt 8 \
-xii -xic -xis \
-xdt \
-m \
-mc \
-ma \
-ms $BEXHOMA_MS \
-t 1200 \
-tr \
-lr 64Gi \
-rr 64Gi \
-rst ramdisk \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_tpch_postgresql_ramdisk.log
yields (after ca. 15 minutes) something like
testcase_tpch_postgresql_ramdisk.log
## Show Summary
### Workload
TPC-H Queries SF=3
* Type: tpch
* Duration: 526s
* Code: 1782336718
* This includes the reading queries of TPC-H.
* This experiment compares run time and resource consumption of TPC-H queries in different DBMS.
* TPC-H (SF=3) data is loaded and benchmark is executed.
* Query ordering is Q1 - Q22.
* All instances use the same query parameters.
* Timeout per query is 1200.
* Import sets indexes and constraints after loading and recomputes statistics.
* Experiment uses bexhoma version 0.10.0.
* System metrics are monitored by a cluster-wide installation.
* Application metrics are monitored by sidecar containers.
* Experiment is limited to DBMS ['PostgreSQL'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker38.
* Database is persisted to disk of type ramdisk and size 50Gi.
* Loading is tested with [8] threads, split into [8] pods.
* Benchmarking is tested with [1] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* PostgreSQL-1-1-1-1-1 uses docker image postgres:18.3
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:262129
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:16Gi
* limits_memory:64Gi
* eval_parameters
* code:1782336718
### Workflow
#### Actual
* DBMS PostgreSQL-1 - Experiment 1 Client 1: tpch (1 pods)
#### Planned
* DBMS PostgreSQL-1 - Experiment 1 Client 1: tpch (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:---------------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| PostgreSQL-1-1 | 1 | 3 | 214.00 | 1.00 | 20.00 | 17.00 | 173.00 | 8 | 0 | | | 0 | False | 50.47 |
### Execution
#### Per Connection
| | configuration | phase | job | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id | pod |
|:---------------------|:----------------|:-----------------|:-------------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|:---------------------|
| PostgreSQL-1-1-1-1-1 | PostgreSQL-1 | PostgreSQL-1-1-1 | PostgreSQL-1-1-1-1 | 1 | 1 | 1 | 1 | 3.00 | 22 | 37 | 0.83 | 13810.53 | 6421.62 | 0 | PostgreSQL-1-1-1-1-1 |
#### Per Phase
| | phase | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id |
|:-----------------|:-----------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|
| PostgreSQL-1-1-1 | PostgreSQL-1-1-1 | 1 | 1 | 1 | 1 | 3.00 | 22 | 37 | 0.83 | 13810.53 | 6421.62 | 0 |
### Latency of Timer Execution [ms]
| Queries | PostgreSQL-1-1-1-1-1 |
|:----------------------------------------------------|-----------------------:|
| Pricing Summary Report (TPC-H Q1) | 2605.43 |
| Minimum Cost Supplier Query (TPC-H Q2) | 711.95 |
| Shipping Priority (TPC-H Q3) | 1143.19 |
| Order Priority Checking Query (TPC-H Q4) | 344.25 |
| Local Supplier Volume (TPC-H Q5) | 762.01 |
| Forecasting Revenue Change (TPC-H Q6) | 423.97 |
| Volume Shipping Query (TPC-H Q7) | 938.71 |
| National Market Share (TPC-H Q8) | 480.93 |
| Product Type Profit Measure (TPC-H Q9) | 1338.47 |
| Returned Item Reporting Query (TPC-H Q10) | 1430.88 |
| Important Stock Identification (TPC-H Q11) | 312.82 |
| Shipping Modes and Order Priority (TPC-H Q12) | 690.23 |
| Customer Distribution (TPC-H Q13) | 3185.49 |
| Promotion Effect Query (TPC-H Q14) | 719.95 |
| Top Supplier Query (TPC-H Q15) | 572.81 |
| Parts/Supplier Relationship (TPC-H Q16) | 569.23 |
| Small-Quantity-Order Revenue (TPC-H Q17) | 2096.00 |
| Large Volume Customer (TPC-H Q18) | 10342.04 |
| Discounted Revenue (TPC-H Q19) | 105.74 |
| Potential Part Promotion (TPC-H Q20) | 424.53 |
| Suppliers Who Kept Orders Waiting Query (TPC-H Q21) | 729.71 |
| Global Sales Opportunity Query (TPC-H Q22) | 191.38 |
### Errors (failed queries)
No errors
### Warnings (result mismatch)
No warnings
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 135.60 | 3.84 | 13.15 | 13.15 |
### Loading phase: component data generator
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 0.00 | 0.00 | 0.00 | 0.00 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 0.25 | 0.00 | 0.00 | 0.00 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 53.22 | 1.47 | 12.79 | 12.79 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 13.23 | 0.00 | 0.25 | 0.25 |
### Application Metrics
#### Loading phase: SUT deployment
| DBMS | Number of Idle Sessions | Number of Idle-in-transaction Sessions | Number of Idle-in-transaction Aborted Sessions | Number of Active Sessions | Number of Active Application Sessions |
|:-------------------|--------------------------:|-----------------------------------------:|-------------------------------------------------:|----------------------------:|----------------------------------------:|
| PostgreSQL-1-1-1-1 | 1.00 | 0.00 | 0.00 | 9.00 | 8.00 |
#### Execution phase: SUT deployment
| DBMS | Number of Idle Sessions | Number of Idle-in-transaction Sessions | Number of Idle-in-transaction Aborted Sessions | Number of Active Sessions | Number of Active Application Sessions |
|:-------------------|--------------------------:|-----------------------------------------:|-------------------------------------------------:|----------------------------:|----------------------------------------:|
| PostgreSQL-1-1-1-1 | 1.00 | 0.00 | 0.00 | 5.00 | 5.00 |
### Tests
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST skipped: Loading phase: component data generator contains 0 or NaN in CPU [CPUs] (data pre-existing)
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Geo Times [s] contains no 0 or NaN
* TEST passed: Power@Size [~Q/h] contains no 0 or NaN
* TEST passed: Throughput@Size contains no 0 or NaN
* TEST passed: No SQL errors
* TEST passed: No SQL warnings
* TEST passed: Workflow as planned
MySQL
TPC-H Simple
bexhoma tpch \
-dbms MySQL \
-sf 1 \
-nlp 8 \
-nlt 8 \
-xii -xic -xis \
-xdt \
-nc 1 \
-ne 1 \
-nbp 1 \
-ms $BEXHOMA_MS \
-t 1200 \
-tr \
-lr 128Gi \
-rr 128Gi \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_tpch_mysql_1.log
yields (after ca. 10 minutes) something like
testcase_tpch_mysql_1.log
## Show Summary
### Workload
TPC-H Queries SF=1
* Type: tpch
* Duration: 678s
* Code: 1782332824
* This includes the reading queries of TPC-H.
* This experiment compares run time and resource consumption of TPC-H queries in different DBMS.
* TPC-H (SF=1) data is loaded and benchmark is executed.
* Query ordering is Q1 - Q22.
* All instances use the same query parameters.
* Timeout per query is 1200.
* Import sets indexes and constraints after loading and recomputes statistics.
* Experiment uses bexhoma version 0.10.0.
* Experiment is limited to DBMS ['MySQL'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker38.
* Loading is tested with [8] threads, split into [8] pods.
* Benchmarking is tested with [1] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* MySQL-1-1-1-1-1 uses docker image mysql:8.4.0
* RAM:540492877824
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:265547
* cpu_list:0-127
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782332824
### Workflow
#### Actual
* DBMS MySQL-1 - Experiment 1 Client 1: tpch (1 pods)
#### Planned
* DBMS MySQL-1 - Experiment 1 Client 1: tpch (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:----------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| MySQL-1-1 | 1 | 1 | 235.00 | 2.00 | 21.00 | 18.00 | 190.00 | 8 | 0 | | None | 0 | False | 15.32 |
### Execution
#### Per Connection
| | configuration | phase | job | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id | pod |
|:----------------|:----------------|:------------|:--------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|:----------------|
| MySQL-1-1-1-1-1 | MySQL-1 | MySQL-1-1-1 | MySQL-1-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 22 | 52 | 1.01 | 3713.89 | 1523.08 | -1 | MySQL-1-1-1-1-1 |
#### Per Phase
| | phase | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id |
|:------------|:------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|
| MySQL-1-1-1 | MySQL-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 22 | 52 | 1.01 | 3713.89 | 1523.08 | -1 |
### Latency of Timer Execution [ms]
| Queries | MySQL-1-1-1-1-1 |
|:----------------------------------------------------|------------------:|
| Pricing Summary Report (TPC-H Q1) | 8498.18 |
| Minimum Cost Supplier Query (TPC-H Q2) | 125.34 |
| Shipping Priority (TPC-H Q3) | 1411.44 |
| Order Priority Checking Query (TPC-H Q4) | 469.69 |
| Local Supplier Volume (TPC-H Q5) | 1073.29 |
| Forecasting Revenue Change (TPC-H Q6) | 1202.28 |
| Volume Shipping Query (TPC-H Q7) | 763.35 |
| National Market Share (TPC-H Q8) | 2860.23 |
| Product Type Profit Measure (TPC-H Q9) | 2027.72 |
| Returned Item Reporting Query (TPC-H Q10) | 1140.93 |
| Important Stock Identification (TPC-H Q11) | 167.67 |
| Shipping Modes and Order Priority (TPC-H Q12) | 1912.12 |
| Customer Distribution (TPC-H Q13) | 3513.30 |
| Promotion Effect Query (TPC-H Q14) | 1378.28 |
| Top Supplier Query (TPC-H Q15) | 13242.41 |
| Parts/Supplier Relationship (TPC-H Q16) | 390.39 |
| Small-Quantity-Order Revenue (TPC-H Q17) | 325.20 |
| Large Volume Customer (TPC-H Q18) | 1705.66 |
| Discounted Revenue (TPC-H Q19) | 139.85 |
| Potential Part Promotion (TPC-H Q20) | 261.68 |
| Suppliers Who Kept Orders Waiting Query (TPC-H Q21) | 4691.05 |
| Global Sales Opportunity Query (TPC-H Q22) | 143.28 |
### Errors (failed queries)
No errors
### Warnings (result mismatch)
No warnings
### Tests
* TEST passed: Geo Times [s] contains no 0 or NaN
* TEST passed: Power@Size [~Q/h] contains no 0 or NaN
* TEST passed: Throughput@Size contains no 0 or NaN
* TEST passed: No SQL errors
* TEST passed: No SQL warnings
* TEST passed: Workflow as planned
TPC-H Monitoring
bexhoma tpch \
-dbms MySQL \
-sf 10 \
-nlp 8 \
-nlt 8 \
-xii -xic -xis \
-xdt \
-nc 1 \
-ne 1 \
-nbp 1 \
-m \
-mc \
-ms $BEXHOMA_MS \
-t 1200 \
-tr \
-lr 128Gi \
-rr 128Gi \
-rss 150Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_tpch_mysql_2.log
yields (after ca. 15 minutes) something like
testcase_tpch_mysql_2.log
## Show Summary
### Workload
TPC-H Queries SF=10
* Type: tpch
* Duration: 4927s
* Code: 1782333525
* This includes the reading queries of TPC-H.
* This experiment compares run time and resource consumption of TPC-H queries in different DBMS.
* TPC-H (SF=10) data is loaded and benchmark is executed.
* Query ordering is Q1 - Q22.
* All instances use the same query parameters.
* Timeout per query is 1200.
* Import sets indexes and constraints after loading and recomputes statistics.
* Experiment uses bexhoma version 0.10.0.
* System metrics are monitored by a cluster-wide installation.
* Experiment is limited to DBMS ['MySQL'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker38.
* Loading is tested with [8] threads, split into [8] pods.
* Benchmarking is tested with [1] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* MySQL-1-1-1-1-1 uses docker image mysql:8.4.0
* RAM:540492877824
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:280818
* cpu_list:0-127
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:128Gi
* limits_memory:128Gi
* eval_parameters
* code:1782333525
### Workflow
#### Actual
* DBMS MySQL-1 - Experiment 1 Client 1: tpch (1 pods)
#### Planned
* DBMS MySQL-1 - Experiment 1 Client 1: tpch (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:----------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| MySQL-1-1 | 1 | 10 | 999.00 | 3.00 | 22.00 | 174.00 | 798.00 | 8 | 0 | | None | 0 | False | 36.04 |
### Execution
#### Per Connection
| | configuration | phase | job | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id | pod |
|:----------------|:----------------|:------------|:--------------|-----------------:|---------:|----------------:|------------:|------:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|:----------------|
| MySQL-1-1-1-1-1 | MySQL-1 | MySQL-1-1-1 | MySQL-1-1-1-1 | 1 | 1 | 1 | 1 | 10.00 | 22 | 583 | 11.12 | 3302.77 | 1358.49 | -1 | MySQL-1-1-1-1-1 |
#### Per Phase
| | phase | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id |
|:------------|:------------|-----------------:|---------:|----------------:|------------:|------:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|
| MySQL-1-1-1 | MySQL-1-1-1 | 1 | 1 | 1 | 1 | 10.00 | 22 | 583 | 11.12 | 3302.77 | 1358.49 | -1 |
### Latency of Timer Execution [ms]
| Queries | MySQL-1-1-1-1-1 |
|:----------------------------------------------------|------------------:|
| Pricing Summary Report (TPC-H Q1) | 78397.68 |
| Minimum Cost Supplier Query (TPC-H Q2) | 1132.37 |
| Shipping Priority (TPC-H Q3) | 15992.19 |
| Order Priority Checking Query (TPC-H Q4) | 4625.08 |
| Local Supplier Volume (TPC-H Q5) | 13016.57 |
| Forecasting Revenue Change (TPC-H Q6) | 11604.60 |
| Volume Shipping Query (TPC-H Q7) | 9224.89 |
| National Market Share (TPC-H Q8) | 36335.05 |
| Product Type Profit Measure (TPC-H Q9) | 26303.56 |
| Returned Item Reporting Query (TPC-H Q10) | 16749.76 |
| Important Stock Identification (TPC-H Q11) | 2484.84 |
| Shipping Modes and Order Priority (TPC-H Q12) | 18967.48 |
| Customer Distribution (TPC-H Q13) | 80492.53 |
| Promotion Effect Query (TPC-H Q14) | 15317.14 |
| Top Supplier Query (TPC-H Q15) | 163969.15 |
| Parts/Supplier Relationship (TPC-H Q16) | 3084.69 |
| Small-Quantity-Order Revenue (TPC-H Q17) | 3258.76 |
| Large Volume Customer (TPC-H Q18) | 17093.37 |
| Discounted Revenue (TPC-H Q19) | 1373.38 |
| Potential Part Promotion (TPC-H Q20) | 2822.47 |
| Suppliers Who Kept Orders Waiting Query (TPC-H Q21) | 51820.62 |
| Global Sales Opportunity Query (TPC-H Q22) | 1382.15 |
### Errors (failed queries)
No errors
### Warnings (result mismatch)
No warnings
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 3755.41 | 9.06 | 25.92 | 52.97 |
### Loading phase: component data generator
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 0.00 | 0.00 | 0.00 | 0.00 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 24.86 | 0.45 | 0.01 | 1.14 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 568.65 | 1.02 | 26.65 | 53.71 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 20.83 | 0.22 | 0.40 | 0.41 |
### Tests
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST skipped: Loading phase: component data generator contains 0 or NaN in CPU [CPUs] (data pre-existing)
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Geo Times [s] contains no 0 or NaN
* TEST passed: Power@Size [~Q/h] contains no 0 or NaN
* TEST passed: Throughput@Size contains no 0 or NaN
* TEST passed: No SQL errors
* TEST passed: No SQL warnings
* TEST passed: Workflow as planned
TPC-H Throughput Test
bexhoma tpch \
-dbms MySQL \
-sf 10 \
-nlp 8 \
-nlt 8 \
-xii -xic -xis \
-xdt \
-nc 2 \
-ne 1,2 \
-nbp 1 \
-m \
-mc \
-ms $BEXHOMA_MS \
-t 1200 \
-tr \
-lr 128Gi \
-rr 128Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rss 150Gi \
-rsr \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_tpch_mysql_3.log
yields (after ca. 15 minutes) something like
testcase_tpch_mysql_3.log
## Show Summary
### Workload
TPC-H Queries SF=1
Type: tpch
Duration: 1241s
Code: 1748912641
This includes the reading queries of TPC-H.
This experiment compares run time and resource consumption of TPC-H queries in different DBMS.
TPC-H (SF=1) data is loaded and benchmark is executed.
Query ordering is Q1 - Q22.
All instances use the same query parameters.
Timeout per query is 1200.
Import sets indexes and constraints after loading and recomputes statistics.
Experiment uses bexhoma version 0.8.7.
System metrics are monitored by a cluster-wide installation.
Benchmark is limited to DBMS ['MySQL'].
Import is handled by 8 processes (pods).
Loading is fixed to cl-worker19.
Benchmarking is fixed to cl-worker19.
SUT is fixed to cl-worker11.
Database is persisted to disk of type shared and size 30Gi.
Loading is tested with [1] threads, split into [8] pods.
Benchmarking is tested with [1] threads, split into [1] pods.
Benchmarking is run as [1, 2] times the number of benchmarking pods.
Experiment is run 2 times.
### Connections
MySQL-BHT-8-1-1-1 uses docker image mysql:8.4.0
RAM:541008568320
Cores:64
host:5.15.0-140-generic
node:cl-worker11
disk:317384432
datadisk:8286
volume_size:30G
volume_used:8.1G
requests_cpu:4
requests_memory:16Gi
eval_parameters
code:1748912641
MySQL-BHT-8-1-2-1 uses docker image mysql:8.4.0
RAM:541008568320
Cores:64
host:5.15.0-140-generic
node:cl-worker11
disk:317384432
datadisk:8286
volume_size:30G
volume_used:8.1G
requests_cpu:4
requests_memory:16Gi
eval_parameters
code:1748912641
MySQL-BHT-8-1-2-2 uses docker image mysql:8.4.0
RAM:541008568320
Cores:64
host:5.15.0-140-generic
node:cl-worker11
disk:317384432
datadisk:8286
volume_size:30G
volume_used:8.1G
requests_cpu:4
requests_memory:16Gi
eval_parameters
code:1748912641
MySQL-BHT-8-2-1-1 uses docker image mysql:8.4.0
RAM:541008568320
Cores:64
host:5.15.0-140-generic
node:cl-worker11
disk:317384416
datadisk:8286
volume_size:30G
volume_used:8.1G
requests_cpu:4
requests_memory:16Gi
eval_parameters
code:1748912641
MySQL-BHT-8-2-2-1 uses docker image mysql:8.4.0
RAM:541008568320
Cores:64
host:5.15.0-140-generic
node:cl-worker11
disk:317384420
datadisk:8286
volume_size:30G
volume_used:8.1G
requests_cpu:4
requests_memory:16Gi
eval_parameters
code:1748912641
MySQL-BHT-8-2-2-2 uses docker image mysql:8.4.0
RAM:541008568320
Cores:64
host:5.15.0-140-generic
node:cl-worker11
disk:317384420
datadisk:8286
volume_size:30G
volume_used:8.1G
requests_cpu:4
requests_memory:16Gi
eval_parameters
code:1748912641
### Errors (failed queries)
No errors
### Warnings (result mismatch)
No warnings
### Latency of Timer Execution [ms]
DBMS MySQL-BHT-8-1-1-1 MySQL-BHT-8-1-2-1 MySQL-BHT-8-1-2-2 MySQL-BHT-8-2-1-1 MySQL-BHT-8-2-2-1 MySQL-BHT-8-2-2-2
Pricing Summary Report (TPC-H Q1) 77.71 71.17 113.27 90.10 68.93 81.12
Minimum Cost Supplier Query (TPC-H Q2) 4.07 3.44 3.64 16.93 4.26 3.92
Shipping Priority (TPC-H Q3) 2.18 1.87 2.57 3.00 1.75 2.78
Order Priority Checking Query (TPC-H Q4) 1.64 1.53 3.12 3.66 1.49 2.79
Local Supplier Volume (TPC-H Q5) 1.87 1.74 3.57 2.56 1.77 3.41
Forecasting Revenue Change (TPC-H Q6) 1.33 1.32 2.57 2.35 1.45 2.29
Forecasting Revenue Change (TPC-H Q7) 1.90 2.24 3.55 2.75 1.87 2.75
National Market Share (TPC-H Q8) 3.18 1.92 3.07 3.26 2.51 2.98
Product Type Profit Measure (TPC-H Q9) 1.94 1.69 2.81 2.76 1.92 2.73
Forecasting Revenue Change (TPC-H Q10) 2.34 1.73 2.02 3.02 1.69 2.82
Important Stock Identification (TPC-H Q11) 2.17 1.62 2.62 2.36 1.46 2.78
Shipping Modes and Order Priority (TPC-H Q12) 2.92 1.41 3.14 2.75 1.71 2.73
Customer Distribution (TPC-H Q13) 2.79 1.47 2.62 2.19 1.32 2.65
Forecasting Revenue Change (TPC-H Q14) 1.91 1.28 2.55 2.31 1.13 2.48
Top Supplier Query (TPC-H Q15) 4.25 2.78 4.84 5.21 3.27 4.68
Parts/Supplier Relationship (TPC-H Q16) 2.79 1.84 3.10 2.31 2.02 3.30
Small-Quantity-Order Revenue (TPC-H Q17) 2.13 1.24 5.14 2.00 1.73 2.81
Large Volume Customer (TPC-H Q18) 2.22 1.51 2.48 2.56 2.05 3.17
Discounted Revenue (TPC-H Q19) 2.26 1.71 2.63 2.48 2.09 2.37
Potential Part Promotion (TPC-H Q20) 2.61 1.49 3.36 2.88 1.79 3.23
Suppliers Who Kept Orders Waiting Query (TPC-H Q21) 3.06 1.52 3.19 3.00 1.87 3.14
Global Sales Opportunity Query (TPC-H Q22) 2.52 1.74 2.90 2.54 2.01 2.95
### Loading [s]
timeGenerate timeIngesting timeSchema timeIndex timeLoad
MySQL-BHT-8-1-1-1 1.0 4.0 9.0 20.0 41.0
MySQL-BHT-8-1-2-1 1.0 4.0 9.0 20.0 41.0
MySQL-BHT-8-1-2-2 1.0 4.0 9.0 20.0 41.0
MySQL-BHT-8-2-1-1 1.0 4.0 9.0 20.0 41.0
MySQL-BHT-8-2-2-1 1.0 4.0 9.0 20.0 41.0
MySQL-BHT-8-2-2-2 1.0 4.0 9.0 20.0 41.0
### Geometric Mean of Medians of Timer Run [s]
Geo Times [s]
DBMS
MySQL-BHT-8-1-1-1 0.0
MySQL-BHT-8-1-2-1 0.0
MySQL-BHT-8-1-2-2 0.0
MySQL-BHT-8-2-1-1 0.0
MySQL-BHT-8-2-2-1 0.0
MySQL-BHT-8-2-2-2 0.0
### Power@Size ((3600*SF)/(geo times))
Power@Size [~Q/h]
DBMS
MySQL-BHT-8-1-1-1 1287048.59
MySQL-BHT-8-1-2-1 1775137.53
MySQL-BHT-8-1-2-2 1003150.78
MySQL-BHT-8-2-1-1 1034211.47
MySQL-BHT-8-2-2-1 1630827.76
MySQL-BHT-8-2-2-2 1050194.94
### Throughput@Size ((queries*streams*3600*SF)/(span of time))
time [s] count SF Throughput@Size
DBMS SF num_experiment num_client
MySQL-BHT-8-1-1 1 1 1 2 1 1 39600.0
MySQL-BHT-8-1-2 1 1 2 3 2 1 52800.0
MySQL-BHT-8-2-1 1 2 1 2 1 1 39600.0
MySQL-BHT-8-2-2 1 2 2 3 2 1 52800.0
### Workflow
#### Actual
DBMS MySQL-BHT-8 - Pods [[1, 2], [1, 2]]
#### Planned
DBMS MySQL-BHT-8 - Pods [[1, 2], [1, 2]]
### Ingestion - SUT
CPU [CPUs] Max CPU Max RAM [Gb] Max RAM Cached [Gb]
MySQL-BHT-8-1-1 4.72 0.02 37.47 45.51
MySQL-BHT-8-1-2 4.72 0.02 37.47 45.51
### Ingestion - Loader
CPU [CPUs] Max CPU Max RAM [Gb] Max RAM Cached [Gb]
MySQL-BHT-8-1-1 0 0 0.0 0.0
MySQL-BHT-8-1-2 0 0 0.0 0.0
### Execution - SUT
CPU [CPUs] Max CPU Max RAM [Gb] Max RAM Cached [Gb]
MySQL-BHT-8-1-1 0.00 0.00 37.47 45.51
MySQL-BHT-8-1-2 1.35 0.00 37.75 45.78
MySQL-BHT-8-2-1 255.00 0.00 75.20 91.06
MySQL-BHT-8-2-2 0.00 0.02 37.46 45.29
### Execution - Benchmarker
CPU [CPUs] Max CPU Max RAM [Gb] Max RAM Cached [Gb]
MySQL-BHT-8-1-1 0.00 0.0 0.0 0.0
MySQL-BHT-8-1-2 0.00 0.0 0.0 0.0
MySQL-BHT-8-2-1 0.00 0.0 0.0 0.0
MySQL-BHT-8-2-2 0.03 0.0 0.0 0.0
### Tests
TEST passed: Geo Times [s] contains no 0 or NaN
TEST passed: Power@Size [~Q/h] contains no 0 or NaN
TEST passed: Throughput@Size contains no 0 or NaN
TEST passed: No SQL errors
TEST passed: No SQL warnings
TEST passed: Ingestion SUT contains no 0 or NaN in CPU [CPUs]
TEST failed: Ingestion Loader contains 0 or NaN in CPU [CPUs]
TEST failed: Execution SUT contains 0 or NaN in CPU [CPUs]
TEST failed: Execution Benchmarker contains 0 or NaN in CPU [CPUs]
TEST passed: Workflow as planned
TPC-H RAM Disk Test
This loads TPC-H data (SF=3) into a database that is stored on a RAM disk. The disk has size 50GB. Make sure you have enough RAM.
bexhoma tpch \
-dbms MySQL \
-sf 10 \
-nlp 8 \
-nlt 8 \
-xii -xic -xis \
-xdt \
-m \
-mc \
-ma \
-ms $BEXHOMA_MS \
-t 1200 \
-tr \
-lr 128Gi \
-rr 128Gi \
-rst ramdisk \
-rss 150Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_tpch_mysql_ramdisk.log
yields (after ca. 15 minutes) something like
testcase_tpch_mysql_ramdisk.log
## Show Summary
### Workload
TPC-H Queries SF=10
* Type: tpch
* Duration: 1863s
* Code: 1782406715
* This includes the reading queries of TPC-H.
* This experiment compares run time and resource consumption of TPC-H queries in different DBMS.
* TPC-H (SF=10) data is loaded and benchmark is executed.
* Query ordering is Q1 - Q22.
* All instances use the same query parameters.
* Timeout per query is 1200.
* Import sets indexes and constraints after loading and recomputes statistics.
* Experiment uses bexhoma version 0.10.0.
* System metrics are monitored by a cluster-wide installation.
* Application metrics are monitored by sidecar containers.
* Experiment is limited to DBMS ['MySQL'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker38.
* Database is persisted to disk of type ramdisk and size 100Gi.
* Loading is tested with [8] threads, split into [8] pods.
* Benchmarking is tested with [1] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* MySQL-1-1-1-1-1 uses docker image mysql:8.4.0
* RAM:540492877824
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220477
* cpu_list:0-127
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--skip-log-bin', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:128Gi
* limits_memory:128Gi
* eval_parameters
* code:1782406715
### Workflow
#### Actual
* DBMS MySQL-1 - Experiment 1 Client 1: tpch (1 pods)
#### Planned
* DBMS MySQL-1 - Experiment 1 Client 1: tpch (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:----------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| MySQL-1-1 | 1 | 10 | 968.00 | 2.00 | 16.00 | 143.00 | 803.00 | 8 | 0 | | None | 0 | False | 37.19 |
### Execution
#### Per Connection
| | configuration | phase | job | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id | pod |
|:----------------|:----------------|:------------|:--------------|-----------------:|---------:|----------------:|------------:|------:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|:----------------|
| MySQL-1-1-1-1-1 | MySQL-1 | MySQL-1-1-1 | MySQL-1-1-1-1 | 1 | 1 | 1 | 1 | 10.00 | 22 | 586 | 11.21 | 3262.50 | 1351.54 | -1 | MySQL-1-1-1-1-1 |
#### Per Phase
| | phase | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id |
|:------------|:------------|-----------------:|---------:|----------------:|------------:|------:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|
| MySQL-1-1-1 | MySQL-1-1-1 | 1 | 1 | 1 | 1 | 10.00 | 22 | 586 | 11.21 | 3262.50 | 1351.54 | -1 |
### Latency of Timer Execution [ms]
| Queries | MySQL-1-1-1-1-1 |
|:----------------------------------------------------|------------------:|
| Pricing Summary Report (TPC-H Q1) | 77080.85 |
| Minimum Cost Supplier Query (TPC-H Q2) | 1046.80 |
| Shipping Priority (TPC-H Q3) | 15651.20 |
| Order Priority Checking Query (TPC-H Q4) | 4487.97 |
| Local Supplier Volume (TPC-H Q5) | 12609.95 |
| Forecasting Revenue Change (TPC-H Q6) | 12032.87 |
| Volume Shipping Query (TPC-H Q7) | 8701.39 |
| National Market Share (TPC-H Q8) | 34697.33 |
| Product Type Profit Measure (TPC-H Q9) | 24026.03 |
| Returned Item Reporting Query (TPC-H Q10) | 16264.46 |
| Important Stock Identification (TPC-H Q11) | 2368.88 |
| Shipping Modes and Order Priority (TPC-H Q12) | 19935.53 |
| Customer Distribution (TPC-H Q13) | 78864.00 |
| Promotion Effect Query (TPC-H Q14) | 14272.63 |
| Top Supplier Query (TPC-H Q15) | 165623.67 |
| Parts/Supplier Relationship (TPC-H Q16) | 3191.36 |
| Small-Quantity-Order Revenue (TPC-H Q17) | 3602.01 |
| Large Volume Customer (TPC-H Q18) | 19186.95 |
| Discounted Revenue (TPC-H Q19) | 1519.51 |
| Potential Part Promotion (TPC-H Q20) | 3022.54 |
| Suppliers Who Kept Orders Waiting Query (TPC-H Q21) | 58908.85 |
| Global Sales Opportunity Query (TPC-H Q22) | 1635.99 |
### Errors (failed queries)
No errors
### Warnings (result mismatch)
No warnings
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 3966.62 | 9.51 | 75.54 | 75.54 |
### Loading phase: component data generator
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 0.00 | 0.00 | 0.00 | 0.00 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 25.07 | 0.30 | 0.03 | 1.14 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 575.47 | 1.02 | 76.32 | 76.32 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 22.67 | 0.48 | 0.42 | 0.42 |
### Application Metrics
#### Loading phase: SUT deployment
| DBMS | InnoDB Buffer Pool Hit Ratio | Queries Per Second (QPS) | Connection Usage Ratio | Slow Queries Rate | InnoDB Log Waits Rate |
|:--------------|-------------------------------:|---------------------------:|-------------------------:|--------------------:|------------------------:|
| MySQL-1-1-1-1 | 1.00 | 1.33 | 0.01 | 0.09 | 0.00 |
#### Execution phase: SUT deployment
| DBMS | InnoDB Buffer Pool Hit Ratio | Queries Per Second (QPS) | Connection Usage Ratio | Slow Queries Rate | InnoDB Log Waits Rate |
|:--------------|-------------------------------:|---------------------------:|-------------------------:|--------------------:|------------------------:|
| MySQL-1-1-1-1 | 1.00 | 0.84 | 0.00 | 0.03 | 0.00 |
### Tests
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST skipped: Loading phase: component data generator contains 0 or NaN in CPU [CPUs] (data pre-existing)
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Geo Times [s] contains no 0 or NaN
* TEST passed: Power@Size [~Q/h] contains no 0 or NaN
* TEST passed: Throughput@Size contains no 0 or NaN
* TEST passed: No SQL errors
* TEST passed: No SQL warnings
* TEST passed: Workflow as planned
MariaDB
TPC-H Simple
bexhoma tpch \
-dbms MariaDB \
-sf 1 \
-nlp 8 \
-nlt 8 \
-xii -xic -xis \
-xdt \
-nc 1 \
-ne 1 \
-nbp 1 \
-ms $BEXHOMA_MS \
-t 1200 \
-tr \
-lr 128Gi \
-rr 128Gi \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_tpch_mariadb_1.log
yields (after ca. 10 minutes) something like
testcase_tpch_mariadb_1.log
## Show Summary
### Workload
TPC-H Queries SF=1
* Type: tpch
* Duration: 708s
* Code: 1782332846
* This includes the reading queries of TPC-H.
* This experiment compares run time and resource consumption of TPC-H queries in different DBMS.
* TPC-H (SF=1) data is loaded and benchmark is executed.
* Query ordering is Q1 - Q22.
* All instances use the same query parameters.
* Timeout per query is 1200.
* Import sets indexes and constraints after loading and recomputes statistics.
* Experiment uses bexhoma version 0.10.0.
* Experiment is limited to DBMS ['MariaDB'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker38.
* Loading is tested with [8] threads, split into [8] pods.
* Benchmarking is tested with [1] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* MariaDB-1-1-1-1-1 uses docker image mariadb:11.4.7
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:265547
* cpu_list:0-127
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782332846
### Workflow
#### Actual
* DBMS MariaDB-1 - Experiment 1 Client 1: tpch (1 pods)
#### Planned
* DBMS MariaDB-1 - Experiment 1 Client 1: tpch (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:------------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| MariaDB-1-1 | 1 | 1 | 367.00 | 1.00 | 25.00 | 36.00 | 302.00 | 8 | 0 | | None | 0 | False | 9.81 |
### Execution
#### Per Connection
| | configuration | phase | job | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id | pod |
|:------------------|:----------------|:--------------|:----------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|:------------------|
| MariaDB-1-1-1-1-1 | MariaDB-1 | MariaDB-1-1-1 | MariaDB-1-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 22 | 107 | 1.15 | 3301.61 | 740.19 | -1 | MariaDB-1-1-1-1-1 |
#### Per Phase
| | phase | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id |
|:--------------|:--------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|
| MariaDB-1-1-1 | MariaDB-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 22 | 107 | 1.15 | 3301.61 | 740.19 | -1 |
### Latency of Timer Execution [ms]
| Queries | MariaDB-1-1-1-1-1 |
|:----------------------------------------------------|--------------------:|
| Pricing Summary Report (TPC-H Q1) | 8140.58 |
| Minimum Cost Supplier Query (TPC-H Q2) | 542.66 |
| Shipping Priority (TPC-H Q3) | 1629.37 |
| Order Priority Checking Query (TPC-H Q4) | 372.68 |
| Local Supplier Volume (TPC-H Q5) | 1190.27 |
| Forecasting Revenue Change (TPC-H Q6) | 1298.28 |
| Volume Shipping Query (TPC-H Q7) | 1265.87 |
| National Market Share (TPC-H Q8) | 2202.44 |
| Product Type Profit Measure (TPC-H Q9) | 2062.75 |
| Returned Item Reporting Query (TPC-H Q10) | 896.30 |
| Important Stock Identification (TPC-H Q11) | 150.67 |
| Shipping Modes and Order Priority (TPC-H Q12) | 3196.19 |
| Customer Distribution (TPC-H Q13) | 3191.68 |
| Promotion Effect Query (TPC-H Q14) | 12756.89 |
| Top Supplier Query (TPC-H Q15) | 2071.70 |
| Parts/Supplier Relationship (TPC-H Q16) | 238.70 |
| Small-Quantity-Order Revenue (TPC-H Q17) | 62.08 |
| Large Volume Customer (TPC-H Q18) | 3098.08 |
| Discounted Revenue (TPC-H Q19) | 97.96 |
| Potential Part Promotion (TPC-H Q20) | 227.04 |
| Suppliers Who Kept Orders Waiting Query (TPC-H Q21) | 57943.45 |
| Global Sales Opportunity Query (TPC-H Q22) | 130.66 |
### Errors (failed queries)
No errors
### Warnings (result mismatch)
No warnings
### Tests
* TEST passed: Geo Times [s] contains no 0 or NaN
* TEST passed: Power@Size [~Q/h] contains no 0 or NaN
* TEST passed: Throughput@Size contains no 0 or NaN
* TEST passed: No SQL errors
* TEST passed: No SQL warnings
* TEST passed: Workflow as planned
TPC-H Monitoring
bexhoma tpch \
-dbms MariaDB \
-sf 1 \
-nlp 8 \
-nlt 8 \
-xii -xic -xis \
-xdt \
-nc 1 \
-ne 1 \
-nbp 1 \
-m \
-mc \
-ms $BEXHOMA_MS \
-t 1200 \
-tr \
-lr 128Gi \
-rr 128Gi \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_tpch_mariadb_2.log
yields (after ca. 15 minutes) something like
testcase_tpch_mariadb_2.log
## Show Summary
### Workload
TPC-H Queries SF=1
* Type: tpch
* Duration: 725s
* Code: 1782333577
* This includes the reading queries of TPC-H.
* This experiment compares run time and resource consumption of TPC-H queries in different DBMS.
* TPC-H (SF=1) data is loaded and benchmark is executed.
* Query ordering is Q1 - Q22.
* All instances use the same query parameters.
* Timeout per query is 1200.
* Import sets indexes and constraints after loading and recomputes statistics.
* Experiment uses bexhoma version 0.10.0.
* System metrics are monitored by a cluster-wide installation.
* Experiment is limited to DBMS ['MariaDB'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker38.
* Loading is tested with [8] threads, split into [8] pods.
* Benchmarking is tested with [1] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* MariaDB-1-1-1-1-1 uses docker image mariadb:11.4.7
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:254316
* cpu_list:0-127
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782333577
### Workflow
#### Actual
* DBMS MariaDB-1 - Experiment 1 Client 1: tpch (1 pods)
#### Planned
* DBMS MariaDB-1 - Experiment 1 Client 1: tpch (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:------------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| MariaDB-1-1 | 1 | 1 | 361.00 | 1.00 | 18.00 | 47.00 | 292.00 | 8 | 0 | | None | 0 | False | 9.97 |
### Execution
#### Per Connection
| | configuration | phase | job | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id | pod |
|:------------------|:----------------|:--------------|:----------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|:------------------|
| MariaDB-1-1-1-1-1 | MariaDB-1 | MariaDB-1-1-1 | MariaDB-1-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 22 | 107 | 1.12 | 3444.13 | 740.19 | -1 | MariaDB-1-1-1-1-1 |
#### Per Phase
| | phase | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id |
|:--------------|:--------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|
| MariaDB-1-1-1 | MariaDB-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 22 | 107 | 1.12 | 3444.13 | 740.19 | -1 |
### Latency of Timer Execution [ms]
| Queries | MariaDB-1-1-1-1-1 |
|:----------------------------------------------------|--------------------:|
| Pricing Summary Report (TPC-H Q1) | 7826.42 |
| Minimum Cost Supplier Query (TPC-H Q2) | 508.07 |
| Shipping Priority (TPC-H Q3) | 1652.33 |
| Order Priority Checking Query (TPC-H Q4) | 400.18 |
| Local Supplier Volume (TPC-H Q5) | 1065.60 |
| Forecasting Revenue Change (TPC-H Q6) | 897.29 |
| Volume Shipping Query (TPC-H Q7) | 1131.93 |
| National Market Share (TPC-H Q8) | 2075.71 |
| Product Type Profit Measure (TPC-H Q9) | 2037.49 |
| Returned Item Reporting Query (TPC-H Q10) | 904.34 |
| Important Stock Identification (TPC-H Q11) | 139.67 |
| Shipping Modes and Order Priority (TPC-H Q12) | 3119.38 |
| Customer Distribution (TPC-H Q13) | 3115.70 |
| Promotion Effect Query (TPC-H Q14) | 12671.14 |
| Top Supplier Query (TPC-H Q15) | 1997.16 |
| Parts/Supplier Relationship (TPC-H Q16) | 210.77 |
| Small-Quantity-Order Revenue (TPC-H Q17) | 59.44 |
| Large Volume Customer (TPC-H Q18) | 3205.99 |
| Discounted Revenue (TPC-H Q19) | 106.26 |
| Potential Part Promotion (TPC-H Q20) | 222.20 |
| Suppliers Who Kept Orders Waiting Query (TPC-H Q21) | 58675.88 |
| Global Sales Opportunity Query (TPC-H Q22) | 127.26 |
### Errors (failed queries)
No errors
### Warnings (result mismatch)
No warnings
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1-1 | 519.88 | 6.76 | 5.68 | 5.78 |
### Loading phase: component data generator
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1-1 | 0.00 | 0.00 | 0.00 | 0.00 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1-1 | 2.75 | 0.05 | 0.00 | 0.11 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1-1 | 83.67 | 1.00 | 5.91 | 6.02 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1-1 | 14.63 | 0.46 | 0.30 | 0.31 |
### Tests
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST skipped: Loading phase: component data generator contains 0 or NaN in CPU [CPUs] (data pre-existing)
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Geo Times [s] contains no 0 or NaN
* TEST passed: Power@Size [~Q/h] contains no 0 or NaN
* TEST passed: Throughput@Size contains no 0 or NaN
* TEST passed: No SQL errors
* TEST passed: No SQL warnings
* TEST passed: Workflow as planned
TPC-H Throughput Test
bexhoma tpch \
-dbms MariaDB \
-sf 1 \
-nlp 8 \
-nlt 8 \
-xii -xic -xis \
-xdt \
-nc 2 \
-ne 1,2 \
-nbp 1 \
-m \
-mc \
-ms $BEXHOMA_MS \
-t 1200 \
-tr \
-lr 128Gi \
-rr 128Gi \
-rsr \
-rst $BEXHOMA_STORAGE_CLASS \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_tpch_mariadb_3.log
yields (after ca. 15 minutes) something like
testcase_tpch_mariadb_3.log
## Show Summary
### Workload
TPC-H Queries SF=1
* Type: tpch
* Duration: 2627s
* Code: 1782334385
* This includes the reading queries of TPC-H.
* This experiment compares run time and resource consumption of TPC-H queries in different DBMS.
* TPC-H (SF=1) data is loaded and benchmark is executed.
* Query ordering is Q1 - Q22.
* All instances use the same query parameters.
* Timeout per query is 1200.
* Import sets indexes and constraints after loading and recomputes statistics.
* Experiment uses bexhoma version 0.10.0.
* System metrics are monitored by a cluster-wide installation.
* Experiment is limited to DBMS ['MariaDB'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker38.
* Database is persisted to disk of type shared and size 50Gi.
* Loading is tested with [8] threads, split into [8] pods.
* Benchmarking is tested with [1] threads, split into [1] pods.
* Benchmarking is run as [1, 2] times the number of benchmarking pods.
* Experiment is run 2 times.
### Connections
* MariaDB-1-1-1-1-1 uses docker image mariadb:11.4.7
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220467
* volume_size:50G
* volume_used:2.1G
* cpu_list:0-127
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782334385
* MariaDB-1-1-2-1-1 uses docker image mariadb:11.4.7
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220467
* volume_size:50G
* volume_used:2.1G
* cpu_list:0-127
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782334385
* MariaDB-1-1-2-1-2 uses docker image mariadb:11.4.7
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220467
* volume_size:50G
* volume_used:2.1G
* cpu_list:0-127
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782334385
* MariaDB-1-2-1-1-1 uses docker image mariadb:11.4.7
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220467
* volume_size:50G
* volume_used:2.1G
* cpu_list:0-127
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782334385
* MariaDB-1-2-2-1-1 uses docker image mariadb:11.4.7
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:253328
* volume_size:50G
* volume_used:2.1G
* cpu_list:0-127
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782334385
* MariaDB-1-2-2-1-2 uses docker image mariadb:11.4.7
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:253328
* volume_size:50G
* volume_used:2.1G
* cpu_list:0-127
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782334385
### Workflow
#### Actual
* DBMS MariaDB-1 - Experiment 1 Client 1: tpch (1 pods)
* DBMS MariaDB-1 - Experiment 1 Client 2: tpch (2 pods)
* DBMS MariaDB-1 - Experiment 2 Client 1: tpch (1 pods)
* DBMS MariaDB-1 - Experiment 2 Client 2: tpch (2 pods)
#### Planned
* DBMS MariaDB-1 - Experiment 1 Client 1: tpch (1 pods)
* DBMS MariaDB-1 - Experiment 1 Client 2: tpch (2 pods)
* DBMS MariaDB-1 - Experiment 2 Client 1: tpch (1 pods)
* DBMS MariaDB-1 - Experiment 2 Client 2: tpch (2 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:------------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| MariaDB-1-1 | 1 | 1 | 1725.00 | 3.00 | 33.00 | 510.00 | 1172.00 | 8 | 0 | | None | 0 | False | 2.09 |
| MariaDB-1-2 | 2 | 1 | 1725.00 | 3.00 | 33.00 | 510.00 | 1172.00 | 8 | 0 | | None | 0 | False | 2.09 |
### Execution
#### Per Connection
| | configuration | phase | job | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id | pod |
|:------------------|:----------------|:--------------|:----------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|:------------------|
| MariaDB-1-1-1-1-1 | MariaDB-1 | MariaDB-1-1-1 | MariaDB-1-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 22 | 103 | 1.08 | 3525.92 | 768.93 | -1 | MariaDB-1-1-1-1-1 |
| MariaDB-1-1-2-1-1 | MariaDB-1 | MariaDB-1-1-2 | MariaDB-1-1-2-1 | 1 | 2 | 1 | 1 | 1.00 | 22 | 121 | 1.17 | 3264.87 | 654.55 | -1 | MariaDB-1-1-2-1-1 |
| MariaDB-1-1-2-1-2 | MariaDB-1 | MariaDB-1-1-2 | MariaDB-1-1-2-1 | 1 | 2 | 1 | 1 | 1.00 | 22 | 119 | 1.18 | 3237.13 | 665.55 | -1 | MariaDB-1-1-2-1-2 |
| MariaDB-1-2-1-1-1 | MariaDB-1 | MariaDB-1-2-1 | MariaDB-1-2-1-1 | 2 | 1 | 1 | 1 | 1.00 | 22 | 107 | 1.16 | 3324.21 | 740.19 | -1 | MariaDB-1-2-1-1-1 |
| MariaDB-1-2-2-1-1 | MariaDB-1 | MariaDB-1-2-2 | MariaDB-1-2-2-1 | 2 | 2 | 1 | 1 | 1.00 | 22 | 135 | 1.37 | 2770.37 | 586.67 | -1 | MariaDB-1-2-2-1-1 |
| MariaDB-1-2-2-1-2 | MariaDB-1 | MariaDB-1-2-2 | MariaDB-1-2-2-1 | 2 | 2 | 1 | 1 | 1.00 | 22 | 134 | 1.43 | 2674.87 | 591.04 | -1 | MariaDB-1-2-2-1-2 |
#### Per Phase
| | phase | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id |
|:--------------|:--------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|
| MariaDB-1-1-1 | MariaDB-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 22 | 103 | 1.08 | 3525.92 | 768.93 | -1 |
| MariaDB-1-1-2 | MariaDB-1-1-2 | 1 | 2 | 1 | 2 | 1.00 | 44 | 121 | 1.18 | 3250.97 | 1309.09 | -1 |
| MariaDB-1-2-1 | MariaDB-1-2-1 | 2 | 1 | 1 | 1 | 1.00 | 22 | 107 | 1.16 | 3324.21 | 740.19 | -1 |
| MariaDB-1-2-2 | MariaDB-1-2-2 | 2 | 2 | 1 | 2 | 1.00 | 44 | 135 | 1.40 | 2722.20 | 1173.33 | -1 |
### Latency of Timer Execution [ms]
| Queries | MariaDB-1-1-1-1-1 | MariaDB-1-1-2-1-1 | MariaDB-1-1-2-1-2 | MariaDB-1-2-1-1-1 | MariaDB-1-2-2-1-1 | MariaDB-1-2-2-1-2 |
|:----------------------------------------------------|--------------------:|--------------------:|--------------------:|--------------------:|--------------------:|--------------------:|
| Pricing Summary Report (TPC-H Q1) | 7835.73 | 7460.30 | 7576.32 | 7445.58 | 7798.95 | 7445.41 |
| Minimum Cost Supplier Query (TPC-H Q2) | 445.24 | 413.40 | 452.21 | 562.07 | 698.00 | 629.33 |
| Shipping Priority (TPC-H Q3) | 1551.76 | 1716.63 | 1743.74 | 1536.82 | 2642.48 | 2428.26 |
| Order Priority Checking Query (TPC-H Q4) | 339.95 | 350.44 | 357.45 | 351.30 | 398.86 | 386.22 |
| Local Supplier Volume (TPC-H Q5) | 966.73 | 1102.55 | 1067.68 | 965.93 | 1496.52 | 1452.50 |
| Forecasting Revenue Change (TPC-H Q6) | 826.30 | 813.59 | 799.97 | 806.65 | 802.49 | 802.31 |
| Volume Shipping Query (TPC-H Q7) | 1078.50 | 1090.08 | 1134.41 | 1098.53 | 1394.57 | 1357.76 |
| National Market Share (TPC-H Q8) | 1828.43 | 2211.54 | 2201.69 | 1924.68 | 3119.33 | 3003.46 |
| Product Type Profit Measure (TPC-H Q9) | 3999.56 | 4395.40 | 4387.79 | 4681.89 | 5739.51 | 5963.52 |
| Returned Item Reporting Query (TPC-H Q10) | 927.14 | 925.49 | 928.07 | 925.57 | 1174.82 | 1175.41 |
| Important Stock Identification (TPC-H Q11) | 134.54 | 147.96 | 141.73 | 146.11 | 217.12 | 218.17 |
| Shipping Modes and Order Priority (TPC-H Q12) | 3029.25 | 3415.46 | 3422.66 | 3044.02 | 4549.91 | 4266.76 |
| Customer Distribution (TPC-H Q13) | 2995.40 | 3887.61 | 3889.33 | 3221.74 | 4031.94 | 3630.11 |
| Promotion Effect Query (TPC-H Q14) | 12074.97 | 14588.29 | 14297.04 | 14396.01 | 18364.82 | 17723.74 |
| Top Supplier Query (TPC-H Q15) | 1780.15 | 1809.90 | 1816.64 | 1751.99 | 1732.48 | 2195.46 |
| Parts/Supplier Relationship (TPC-H Q16) | 220.32 | 221.36 | 206.89 | 213.05 | 234.73 | 226.32 |
| Small-Quantity-Order Revenue (TPC-H Q17) | 69.54 | 61.56 | 65.50 | 75.99 | 62.19 | 61.79 |
| Large Volume Customer (TPC-H Q18) | 2930.59 | 3922.40 | 3912.35 | 2979.81 | 3533.70 | 3592.21 |
| Discounted Revenue (TPC-H Q19) | 95.73 | 104.56 | 105.85 | 103.57 | 96.93 | 161.23 |
| Potential Part Promotion (TPC-H Q20) | 206.68 | 235.15 | 220.62 | 242.60 | 218.74 | 385.86 |
| Suppliers Who Kept Orders Waiting Query (TPC-H Q21) | 55401.87 | 66972.36 | 65984.58 | 56422.19 | 71639.16 | 71949.04 |
| Global Sales Opportunity Query (TPC-H Q22) | 113.79 | 115.17 | 139.42 | 144.84 | 158.60 | 159.10 |
### Errors (failed queries)
No errors
### Warnings (result mismatch)
No warnings
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1-1 | 1541.04 | 2.44 | 5.93 | 6.04 |
| MariaDB-1-1-2-1 | 1541.04 | 2.44 | 5.93 | 6.04 |
### Loading phase: component data generator
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1-1 | 0.00 | 0.00 | 0.00 | 0.00 |
| MariaDB-1-1-2-1 | 0.00 | 0.00 | 0.00 | 0.00 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1-1 | 4.79 | 0.05 | 0.00 | 0.13 |
| MariaDB-1-1-2-1 | 4.79 | 0.05 | 0.00 | 0.13 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1-1 | 69.86 | 0.99 | 5.99 | 6.09 |
| MariaDB-1-1-2-1 | 205.20 | 2.00 | 6.00 | 6.10 |
| MariaDB-1-2-1-1 | 92.51 | 1.00 | 3.72 | 3.75 |
| MariaDB-1-2-2-1 | 224.20 | 2.00 | 3.73 | 3.76 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1-1 | 15.09 | 0.17 | 0.30 | 0.30 |
| MariaDB-1-1-2-1 | 29.66 | 0.21 | 0.33 | 0.34 |
| MariaDB-1-2-1-1 | 16.76 | 0.46 | 0.34 | 0.35 |
| MariaDB-1-2-2-1 | 30.93 | 0.24 | 0.34 | 0.35 |
### Tests
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST skipped: Loading phase: component data generator contains 0 or NaN in CPU [CPUs] (data pre-existing)
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Geo Times [s] contains no 0 or NaN
* TEST passed: Power@Size [~Q/h] contains no 0 or NaN
* TEST passed: Throughput@Size contains no 0 or NaN
* TEST passed: No SQL errors
* TEST passed: No SQL warnings
* TEST passed: Workflow as planned
TPC-H RAM Disk Test
This loads TPC-H data (SF=3) into a database that is stored on a RAM disk. The disk has size 50GB. Make sure you have enough RAM.
bexhoma tpch \
-dbms MariaDB \
-sf 10 \
-nlp 8 \
-nlt 8 \
-xii -xic -xis \
-xdt \
-m \
-mc \
-ma \
-ms $BEXHOMA_MS \
-t 1200 \
-tr \
-lr 128Gi \
-rr 128Gi \
-rst ramdisk \
-rss 150Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_tpch_mariadb_ramdisk.log
yields (after ca. 15 minutes) something like
testcase_tpch_mariadb_ramdisk.log
## Show Summary
### Workload
TPC-H Queries SF=10
* Type: tpch
* Duration: 3579s
* Code: 1782340884
* This includes the reading queries of TPC-H.
* This experiment compares run time and resource consumption of TPC-H queries in different DBMS.
* TPC-H (SF=10) data is loaded and benchmark is executed.
* Query ordering is Q1 - Q22.
* All instances use the same query parameters.
* Timeout per query is 1200.
* Import sets indexes and constraints after loading and recomputes statistics.
* Experiment uses bexhoma version 0.10.0.
* System metrics are monitored by a cluster-wide installation.
* Application metrics are monitored by sidecar containers.
* Experiment is limited to DBMS ['MariaDB'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker38.
* Database is persisted to disk of type ramdisk and size 100Gi.
* Loading is tested with [8] threads, split into [8] pods.
* Benchmarking is tested with [1] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* MariaDB-1-1-1-1-1 uses docker image mariadb:11.4.7
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:225014
* cpu_list:0-127
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:128Gi
* limits_memory:128Gi
* eval_parameters
* code:1782340884
### Workflow
#### Actual
* DBMS MariaDB-1 - Experiment 1 Client 1: tpch (1 pods)
#### Planned
* DBMS MariaDB-1 - Experiment 1 Client 1: tpch (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:------------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| MariaDB-1-1 | 1 | 10 | 2433.00 | 2.00 | 25.00 | 384.00 | 2018.00 | 8 | 0 | | None | 0 | False | 14.80 |
### Execution
#### Per Connection
| | configuration | phase | job | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id | pod |
|:------------------|:----------------|:--------------|:----------------|-----------------:|---------:|----------------:|------------:|------:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|:------------------|
| MariaDB-1-1-1-1-1 | MariaDB-1 | MariaDB-1-1-1 | MariaDB-1-1-1-1 | 1 | 1 | 1 | 1 | 10.00 | 22 | 1148 | 12.78 | 2877.44 | 689.90 | -1 | MariaDB-1-1-1-1-1 |
#### Per Phase
| | phase | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id |
|:--------------|:--------------|-----------------:|---------:|----------------:|------------:|------:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|
| MariaDB-1-1-1 | MariaDB-1-1-1 | 1 | 1 | 1 | 1 | 10.00 | 22 | 1148 | 12.78 | 2877.44 | 689.90 | -1 |
### Latency of Timer Execution [ms]
| Queries | MariaDB-1-1-1-1-1 |
|:----------------------------------------------------|--------------------:|
| Pricing Summary Report (TPC-H Q1) | 70985.56 |
| Minimum Cost Supplier Query (TPC-H Q2) | 5073.32 |
| Shipping Priority (TPC-H Q3) | 20450.51 |
| Order Priority Checking Query (TPC-H Q4) | 21473.07 |
| Local Supplier Volume (TPC-H Q5) | 11632.52 |
| Forecasting Revenue Change (TPC-H Q6) | 8384.30 |
| Volume Shipping Query (TPC-H Q7) | 12469.46 |
| National Market Share (TPC-H Q8) | 25097.32 |
| Product Type Profit Measure (TPC-H Q9) | 23869.39 |
| Returned Item Reporting Query (TPC-H Q10) | 10857.62 |
| Important Stock Identification (TPC-H Q11) | 1819.67 |
| Shipping Modes and Order Priority (TPC-H Q12) | 32846.25 |
| Customer Distribution (TPC-H Q13) | 42904.62 |
| Promotion Effect Query (TPC-H Q14) | 162749.93 |
| Top Supplier Query (TPC-H Q15) | 18011.23 |
| Parts/Supplier Relationship (TPC-H Q16) | 2189.07 |
| Small-Quantity-Order Revenue (TPC-H Q17) | 623.10 |
| Large Volume Customer (TPC-H Q18) | 39322.71 |
| Discounted Revenue (TPC-H Q19) | 1202.30 |
| Potential Part Promotion (TPC-H Q20) | 3050.72 |
| Suppliers Who Kept Orders Waiting Query (TPC-H Q21) | 624716.37 |
| Global Sales Opportunity Query (TPC-H Q22) | 1195.83 |
### Errors (failed queries)
No errors
### Warnings (result mismatch)
No warnings
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1-1 | 7280.38 | 11.95 | 74.05 | 74.05 |
### Loading phase: component data generator
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1-1 | 0.02 | 0.00 | 0.00 | 0.00 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1-1 | 29.18 | 0.22 | 0.01 | 1.31 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1-1 | 1143.73 | 1.07 | 59.70 | 59.70 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1-1 | 18.50 | 0.09 | 0.34 | 0.34 |
### Tests
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component data generator contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Geo Times [s] contains no 0 or NaN
* TEST passed: Power@Size [~Q/h] contains no 0 or NaN
* TEST passed: Throughput@Size contains no 0 or NaN
* TEST passed: No SQL errors
* TEST passed: No SQL warnings
* TEST passed: Workflow as planned
TPC-DS
Compare
TPC-DS Compare
bexhoma tpcds \
-sf 1 \
-nlp 8 \
-nlt 8 \
-xii -xic -xis \
-xdt \
-ms $BEXHOMA_MS \
-t 1200 \
-tr \
-lr 128Gi \
-rr 128Gi \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_tpcds_compare.log
yields (after ca. 520 minutes) something like
testcase_tpcds_compare.log
## Show Summary
### Workload
TPC-DS Queries SF=1
* Type: tpcds
* Duration: 5464s
* Code: 1782340910
* This includes the reading queries of TPC-DS.
* This experiment compares run time and resource consumption of TPC-DS queries in different DBMS.
* TPC-DS (SF=1) data is loaded and benchmark is executed.
* Query ordering is Q1 - Q99.
* All instances use the same query parameters.
* Timeout per query is 1200.
* Import sets indexes and constraints after loading and recomputes statistics.
* Experiment uses bexhoma version 0.10.0.
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker38.
* Loading is tested with [8] threads, split into [8] pods.
* Benchmarking is tested with [1] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* MariaDB-1-1-1-1-1 uses docker image mariadb:11.4.7
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:225014
* cpu_list:0-127
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1782340910
* MonetDB-1-1-1-1-1 uses docker image monetdb/monetdb:Dec2025-SP1
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:262263
* cpu_list:0-127
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1782340910
* MySQL-1-1-1-1-1 uses docker image mysql:8.4.0
* RAM:540492877824
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:263247
* cpu_list:0-127
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1782340910
* PostgreSQL-1-1-1-1-1 uses docker image postgres:18.3
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:259748
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1782340910
### Workflow
#### Actual
* DBMS MariaDB-1 - Experiment 1 Client 1: tpcds (1 pods)
* DBMS MonetDB-1 - Experiment 1 Client 1: tpcds (1 pods)
* DBMS MySQL-1 - Experiment 1 Client 1: tpcds (1 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 1: tpcds (1 pods)
#### Planned
* DBMS PostgreSQL-1 - Experiment 1 Client 1: tpcds (1 pods)
* DBMS MonetDB-1 - Experiment 1 Client 1: tpcds (1 pods)
* DBMS MariaDB-1 - Experiment 1 Client 1: tpcds (1 pods)
* DBMS MySQL-1 - Experiment 1 Client 1: tpcds (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:---------------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| MariaDB-1-1 | 1 | 1 | 685.00 | 1.00 | 1.00 | 62.00 | 616.00 | 8 | 0 | | None | 0 | False | 5.26 |
| MonetDB-1-1 | 1 | 1 | 267.00 | 2.00 | 1.00 | 59.00 | 200.00 | 8 | 0 | | None | 0 | False | 13.48 |
| MySQL-1-1 | 1 | 1 | 826.00 | 2.00 | 1.00 | 90.00 | 726.00 | 8 | 0 | | None | 0 | False | 4.36 |
| PostgreSQL-1-1 | 1 | 1 | 302.00 | 1.00 | 1.00 | 49.00 | 243.00 | 8 | 0 | | None | 0 | False | 11.92 |
### Execution
#### Per Connection
| | configuration | phase | job | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id | pod |
|:---------------------|:----------------|:-----------------|:-------------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|:---------------------|
| MariaDB-1-1-1-1-1 | MariaDB-1 | MariaDB-1-1-1 | MariaDB-1-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 97 | 3220 | 1.01 | 3618.98 | 108.45 | -1 | MariaDB-1-1-1-1-1 |
| MonetDB-1-1-1-1-1 | MonetDB-1 | MonetDB-1-1-1 | MonetDB-1-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 97 | 22 | 0.06 | 60422.01 | 15872.73 | -1 | MonetDB-1-1-1-1-1 |
| MySQL-1-1-1-1-1 | MySQL-1 | MySQL-1-1-1 | MySQL-1-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 97 | 517 | 0.84 | 4337.55 | 675.44 | -1 | MySQL-1-1-1-1-1 |
| PostgreSQL-1-1-1-1-1 | PostgreSQL-1 | PostgreSQL-1-1-1 | PostgreSQL-1-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 97 | 215 | 0.42 | 8799.20 | 1624.19 | -1 | PostgreSQL-1-1-1-1-1 |
#### Per Phase
| | phase | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id |
|:-----------------|:-----------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|
| MariaDB-1-1-1 | MariaDB-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 97 | 3220 | 1.01 | 3618.98 | 108.45 | -1 |
| MonetDB-1-1-1 | MonetDB-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 97 | 22 | 0.06 | 60422.01 | 15872.73 | -1 |
| MySQL-1-1-1 | MySQL-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 97 | 517 | 0.84 | 4337.55 | 675.44 | -1 |
| PostgreSQL-1-1-1 | PostgreSQL-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 97 | 215 | 0.42 | 8799.20 | 1624.19 | -1 |
### Latency of Timer Execution [ms]
| Queries | MariaDB-1-1-1-1-1 | MonetDB-1-1-1-1-1 | MySQL-1-1-1-1-1 | PostgreSQL-1-1-1-1-1 |
|:--------------|--------------------:|--------------------:|------------------:|-----------------------:|
| TPC-DS Q1 | 21.07 | 55.26 | 67.17 | 152.02 |
| TPC-DS Q2 | 5803.54 | 99.61 | 5400.13 | 318.81 |
| TPC-DS Q3 | 13.89 | 24.17 | 16.76 | 187.35 |
| TPC-DS Q4 | 13724.83 | 591.97 | 28228.22 | 9527.46 |
| TPC-DS Q5 | 9699.03 | 82.07 | 12899.93 | 512.05 |
| TPC-DS Q6 | 910.05 | 46.80 | 76171.86 | 58650.09 |
| TPC-DS Q7 | 4163.95 | 46.20 | 767.16 | 384.39 |
| TPC-DS Q8 | 373.86 | 26.99 | 398.33 | 71.92 |
| TPC-DS Q9 | 4233.44 | 47.19 | 4433.47 | 2453.94 |
| TPC-DS Q10 | 69.69 | 39.45 | 50.23 | 1241.57 |
| TPC-DS Q11 | 9526.71 | 290.42 | 18897.74 | 5740.73 |
| TPC-DS Q12 | 281.70 | 22.44 | 316.18 | 70.59 |
| TPC-DS Q13 | 1242.60 | 42.27 | 1559.52 | 669.91 |
| TPC-DS Q14a+b | 52023.38 | 1623.37 | 52238.94 | 2203.49 |
| TPC-DS Q15 | 168.84 | 25.55 | 199.70 | 135.23 |
| TPC-DS Q16 | 11165.10 | 34.09 | 62.38 | 196.40 |
| TPC-DS Q17 | 711.14 | 88.00 | 483.22 | 310.12 |
| TPC-DS Q18 | 1970.04 | 54.44 | 954.09 | 415.76 |
| TPC-DS Q19 | 240.11 | 38.34 | 335.27 | 160.99 |
| TPC-DS Q20 | 465.63 | 27.99 | 588.79 | 102.95 |
| TPC-DS Q21 | 21609.24 | 94.85 | 27884.28 | 253.48 |
| TPC-DS Q22 | 19818.31 | 707.33 | 5119.50 | 3602.36 |
| TPC-DS Q23a+b | 53298.29 | 977.49 | 37681.68 | 4651.22 |
| TPC-DS Q24a+b | 25.72 | 221.42 | 1307.35 | 687.98 |
| TPC-DS Q25 | 158.90 | 92.36 | 146.32 | 319.96 |
| TPC-DS Q26 | 905.16 | 23.92 | 5428.34 | 276.28 |
| TPC-DS Q27 | 1620.45 | 62.92 | 623.18 | 35.32 |
| TPC-DS Q28 | 2887.82 | 41.03 | 3676.00 | 880.69 |
| TPC-DS Q29 | 98.79 | 90.62 | 85.77 | 350.02 |
| TPC-DS Q30 | 113.54 | 24.25 | 479.41 | 10200.61 |
| TPC-DS Q31 | 1481.64 | 95.09 | 12681.09 | 1858.14 |
| TPC-DS Q32 | 12.02 | 17.87 | 137.53 | 118.79 |
| TPC-DS Q33 | 154.27 | 23.98 | 225.23 | 446.59 |
| TPC-DS Q34 | 3114.09 | 29.74 | 1078.53 | 37.94 |
| TPC-DS Q35 | 1221.68 | 72.71 | 15290.84 | 1451.16 |
| TPC-DS Q36 | 2938.42 | 103.00 | 2955.39 | 429.56 |
| TPC-DS Q37 | 3727.47 | 63.45 | 14.13 | 351.82 |
| TPC-DS Q38 | 7216.85 | 122.45 | 8440.30 | 1800.39 |
| TPC-DS Q39a+b | 1325.36 | 984.92 | 2256.26 | 2867.46 |
| TPC-DS Q40 | 177.18 | 59.73 | 182.75 | 142.59 |
| TPC-DS Q41 | 433.50 | 8.30 | 1449.47 | 1274.81 |
| TPC-DS Q42 | 228.59 | 53.15 | 28.12 | 99.66 |
| TPC-DS Q43 | 1161.45 | 40.72 | 2.47 | 34.65 |
| TPC-DS Q44 | 2.80 | 27.29 | 2.84 | 4.33 |
| TPC-DS Q45 | 123.79 | 15.95 | 129.26 | 105.13 |
| TPC-DS Q46 | 3343.31 | 34.21 | 854.27 | 48.80 |
| TPC-DS Q47 | 14000.60 | 173.92 | 7504.01 | 2290.67 |
| TPC-DS Q48 | 1213.90 | 33.09 | 1023.27 | 816.02 |
| TPC-DS Q49 | 98.84 | 67.00 | 144.06 | 570.17 |
| TPC-DS Q50 | 30.46 | 95.72 | 34.04 | 522.41 |
| TPC-DS Q51 | 7726.22 | 298.44 | 6481.72 | 1212.11 |
| TPC-DS Q52 | 241.61 | 52.72 | 31.34 | 100.63 |
| TPC-DS Q53 | 137.48 | 24.66 | 247.29 | 125.98 |
| TPC-DS Q54 | 1041.57 | 25.03 | 2888.43 | 110.26 |
| TPC-DS Q55 | 232.88 | 17.69 | 20.79 | 97.20 |
| TPC-DS Q56 | 297.55 | 20.46 | 255.40 | 515.70 |
| TPC-DS Q57 | 7031.33 | 102.55 | 3618.49 | 1217.90 |
| TPC-DS Q58 | 5844.27 | 53.04 | 6950.81 | 485.39 |
| TPC-DS Q59 | 9224.35 | 97.24 | 6548.77 | 522.90 |
| TPC-DS Q60 | 315.44 | 21.29 | 493.63 | 455.90 |
| TPC-DS Q61 | 408.92 | 27.47 | 3.25 | 200.50 |
| TPC-DS Q62 | 1694.59 | 33.87 | 3010.61 | 136.49 |
| TPC-DS Q63 | 183.99 | 26.25 | 264.85 | 130.30 |
| TPC-DS Q64 | 864.69 | 229.33 | 844.38 | 1077.57 |
| TPC-DS Q65 | 5950.28 | 84.82 | 7780.19 | 788.41 |
| TPC-DS Q66 | 1201.54 | 102.77 | 1529.50 | 260.41 |
| TPC-DS Q67 | 7342.74 | 227.18 | 8726.65 | 3833.80 |
| TPC-DS Q68 | 3129.20 | 33.12 | 358.69 | 51.43 |
| TPC-DS Q69 | 408.21 | 50.25 | 491.36 | 320.53 |
| TPC-DS Q70 | 8599.59 | 77.83 | 13044.67 | 530.28 |
| TPC-DS Q71 | 467.25 | 27.97 | 548.84 | 421.98 |
| TPC-DS Q72 | 402504.84 | 193.61 | 13769.05 | 1377.94 |
| TPC-DS Q73 | 2897.10 | 21.17 | 1109.43 | 37.22 |
| TPC-DS Q74 | 5990.92 | 89.73 | 5425.75 | 1344.35 |
| TPC-DS Q75 | 5687.24 | 271.19 | 1794.57 | 1031.77 |
| TPC-DS Q76 | 470.49 | 32.05 | 454.91 | 157.88 |
| TPC-DS Q77 | 6140.32 | 50.71 | 9968.00 | 424.99 |
| TPC-DS Q78 | 5760.60 | 397.72 | 12489.20 | 2499.30 |
| TPC-DS Q79 | 3410.66 | 55.51 | 5240.94 | 251.68 |
| TPC-DS Q80 | 534.09 | 293.81 | 9304.36 | 586.29 |
| TPC-DS Q81 | 215.52 | 38.86 | 2210.80 | 51833.27 |
| TPC-DS Q82 | 3690.94 | 52.44 | 16.92 | 371.25 |
| TPC-DS Q83 | 916.24 | 13.48 | 885.12 | 123.44 |
| TPC-DS Q84 | 59.94 | 20.42 | 66.85 | 105.39 |
| TPC-DS Q85 | 127.50 | 174.22 | 109.63 | 402.61 |
| TPC-DS Q86 | 920.11 | 33.54 | 1198.13 | 265.17 |
| TPC-DS Q87 | 7192.41 | 146.16 | 8526.64 | 1738.31 |
| TPC-DS Q88 | 23827.06 | 41.49 | 2029.98 | 3558.78 |
| TPC-DS Q89 | 1681.72 | 37.15 | 284.53 | 164.21 |
| TPC-DS Q90 | 357.77 | 14.74 | 377.34 | 154.39 |
| TPC-DS Q91 | 18.86 | 22.27 | 22.12 | 177.27 |
| TPC-DS Q92 | 11.04 | 11.17 | 63.66 | 64.48 |
| TPC-DS Q93 | 39.09 | 71.28 | 45.82 | 197.28 |
| TPC-DS Q96 | 2703.00 | 16.61 | 159.09 | 105.49 |
| TPC-DS Q97 | 5213.55 | 129.45 | 6318.19 | 417.12 |
| TPC-DS Q98 | 871.98 | 43.38 | 1138.54 | 182.52 |
| TPC-DS Q99 | 5054.21 | 48.56 | 12817.66 | 179.08 |
### Errors (failed queries)
| | TPC-DS Q1 | TPC-DS Q2 | TPC-DS Q3 | TPC-DS Q4 | TPC-DS Q5 | TPC-DS Q6 | TPC-DS Q7 | TPC-DS Q8 | TPC-DS Q9 | TPC-DS Q10 | TPC-DS Q11 | TPC-DS Q12 | TPC-DS Q13 | TPC-DS Q14a+b | TPC-DS Q15 | TPC-DS Q16 | TPC-DS Q17 | TPC-DS Q18 | TPC-DS Q19 | TPC-DS Q20 | TPC-DS Q21 | TPC-DS Q22 | TPC-DS Q23a+b | TPC-DS Q24a+b | TPC-DS Q25 | TPC-DS Q26 | TPC-DS Q27 | TPC-DS Q28 | TPC-DS Q29 | TPC-DS Q30 | TPC-DS Q31 | TPC-DS Q32 | TPC-DS Q33 | TPC-DS Q34 | TPC-DS Q35 | TPC-DS Q36 | TPC-DS Q37 | TPC-DS Q38 | TPC-DS Q39a+b | TPC-DS Q40 | TPC-DS Q41 | TPC-DS Q42 | TPC-DS Q43 | TPC-DS Q44 | TPC-DS Q45 | TPC-DS Q46 | TPC-DS Q47 | TPC-DS Q48 | TPC-DS Q49 | TPC-DS Q50 | TPC-DS Q51 | TPC-DS Q52 | TPC-DS Q53 | TPC-DS Q54 | TPC-DS Q55 | TPC-DS Q56 | TPC-DS Q57 | TPC-DS Q58 | TPC-DS Q59 | TPC-DS Q60 | TPC-DS Q61 | TPC-DS Q62 | TPC-DS Q63 | TPC-DS Q64 | TPC-DS Q65 | TPC-DS Q66 | TPC-DS Q67 | TPC-DS Q68 | TPC-DS Q69 | TPC-DS Q70 | TPC-DS Q71 | TPC-DS Q72 | TPC-DS Q73 | TPC-DS Q74 | TPC-DS Q75 | TPC-DS Q76 | TPC-DS Q77 | TPC-DS Q78 | TPC-DS Q79 | TPC-DS Q80 | TPC-DS Q81 | TPC-DS Q82 | TPC-DS Q83 | TPC-DS Q84 | TPC-DS Q85 | TPC-DS Q86 | TPC-DS Q87 | TPC-DS Q88 | TPC-DS Q89 | TPC-DS Q90 | TPC-DS Q91 | TPC-DS Q92 | TPC-DS Q93 | TPC-DS Q94 | TPC-DS Q95 | TPC-DS Q96 | TPC-DS Q97 | TPC-DS Q98 | TPC-DS Q99 |
|:------------------|------------:|------------:|------------:|------------:|------------:|------------:|------------:|------------:|------------:|-------------:|-------------:|-------------:|-------------:|----------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|----------------:|----------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|----------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|
| MariaDB-1-1-1-1-1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 |
* TPC-DS Q94
* MariaDB-1-1-1-1-1: numRun 1: : java.sql.SQLTimeoutException: (conn=61) Query execution was interrupted (max_statement_time exceeded)
* TPC-DS Q95
* MariaDB-1-1-1-1-1: numRun 1: : java.sql.SQLTimeoutException: (conn=61) Query execution was interrupted (max_statement_time exceeded)
### Warnings (result mismatch)
| | TPC-DS Q1 | TPC-DS Q2 | TPC-DS Q3 | TPC-DS Q4 | TPC-DS Q5 | TPC-DS Q6 | TPC-DS Q7 | TPC-DS Q8 | TPC-DS Q9 | TPC-DS Q10 | TPC-DS Q11 | TPC-DS Q12 | TPC-DS Q13 | TPC-DS Q14a+b | TPC-DS Q15 | TPC-DS Q16 | TPC-DS Q17 | TPC-DS Q18 | TPC-DS Q19 | TPC-DS Q20 | TPC-DS Q21 | TPC-DS Q22 | TPC-DS Q23a+b | TPC-DS Q24a+b | TPC-DS Q25 | TPC-DS Q26 | TPC-DS Q27 | TPC-DS Q28 | TPC-DS Q29 | TPC-DS Q30 | TPC-DS Q31 | TPC-DS Q32 | TPC-DS Q33 | TPC-DS Q34 | TPC-DS Q35 | TPC-DS Q36 | TPC-DS Q37 | TPC-DS Q38 | TPC-DS Q39a+b | TPC-DS Q40 | TPC-DS Q41 | TPC-DS Q42 | TPC-DS Q43 | TPC-DS Q44 | TPC-DS Q45 | TPC-DS Q46 | TPC-DS Q47 | TPC-DS Q48 | TPC-DS Q49 | TPC-DS Q50 | TPC-DS Q51 | TPC-DS Q52 | TPC-DS Q53 | TPC-DS Q54 | TPC-DS Q55 | TPC-DS Q56 | TPC-DS Q57 | TPC-DS Q58 | TPC-DS Q59 | TPC-DS Q60 | TPC-DS Q61 | TPC-DS Q62 | TPC-DS Q63 | TPC-DS Q64 | TPC-DS Q65 | TPC-DS Q66 | TPC-DS Q67 | TPC-DS Q68 | TPC-DS Q69 | TPC-DS Q70 | TPC-DS Q71 | TPC-DS Q72 | TPC-DS Q73 | TPC-DS Q74 | TPC-DS Q75 | TPC-DS Q76 | TPC-DS Q77 | TPC-DS Q78 | TPC-DS Q79 | TPC-DS Q80 | TPC-DS Q81 | TPC-DS Q82 | TPC-DS Q83 | TPC-DS Q84 | TPC-DS Q85 | TPC-DS Q86 | TPC-DS Q87 | TPC-DS Q88 | TPC-DS Q89 | TPC-DS Q90 | TPC-DS Q91 | TPC-DS Q92 | TPC-DS Q93 | TPC-DS Q94 | TPC-DS Q95 | TPC-DS Q96 | TPC-DS Q97 | TPC-DS Q98 | TPC-DS Q99 |
|:---------------------|------------:|------------:|------------:|------------:|------------:|------------:|------------:|------------:|------------:|-------------:|-------------:|-------------:|-------------:|----------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|----------------:|----------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|----------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|
| MariaDB-1-1-1-1-1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| MonetDB-1-1-1-1-1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 |
| PostgreSQL-1-1-1-1-1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
### Tests
* TEST passed: Geo Times [s] contains no 0 or NaN
* TEST passed: Power@Size [~Q/h] contains no 0 or NaN
* TEST passed: Throughput@Size contains no 0 or NaN
* TEST failed: SQL errors
* TEST failed: SQL warnings (result mismatch)
* TEST passed: Workflow as planned
PostgreSQL
TPC-DS Simple
bexhoma tpcds \
-dbms PostgreSQL \
-sf 1 \
-nlp 8 \
-nlt 8 \
-xii -xic -xis \
-xdt \
-nc 1 \
-ne 1 \
-nbp 1 \
-ms $BEXHOMA_MS \
-t 1200 \
-tr \
-lr 128Gi \
-rr 128Gi \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_tpcds_postgresql_1.log
yields (after ca. 10 minutes) something like
testcase_tpcds_postgresql_1.log
## Show Summary
### Workload
TPC-DS Queries SF=1
* Type: tpcds
* Duration: 1037s
* Code: 1782552070
* This includes the reading queries of TPC-DS.
* This experiment compares run time and resource consumption of TPC-DS queries in different DBMS.
* TPC-DS (SF=1) data is loaded and benchmark is executed.
* Query ordering is Q1 - Q99.
* All instances use the same query parameters.
* Timeout per query is 1200.
* Import sets indexes and constraints after loading and recomputes statistics.
* Experiment uses bexhoma version 0.10.2.
* Experiment is limited to DBMS ['PostgreSQL'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* Database uses ephemeral storage of size 10Gi.
* Loading is tested with [8] threads, split into [8] pods.
* Benchmarking is tested with [1] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* PostgreSQL-1-1-1-1-1 uses docker image postgres:18.3
* RAM:1077381271552
* CPU:AMD EPYC 7742 64-Core Processor
* Cores:256
* host:6.8.0-111-generic
* node:cl-worker27
* disk:1391288
* cpu_list:0-255
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:128Gi
* limits_memory:128Gi
* eval_parameters
* code:1782552070
### SUT Container Restarts
* bexhoma-sut-postgresql-1-1782552070-695847f58-fdfqc: 0 0
### Workflow
#### Actual
* DBMS PostgreSQL-1 - Experiment 1 Client 1: tpcds (1 pods)
#### Planned
* DBMS PostgreSQL-1 - Experiment 1 Client 1: tpcds (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:---------------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| PostgreSQL-1-1 | 1 | 1 | 504.00 | 4.00 | 1.00 | 82.00 | 409.00 | 8 | 0 | | None | 0 | False | 7.14 |
### Execution
#### Per Connection
| | configuration | phase | job | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id | pod |
|:---------------------|:----------------|:-----------------|:-------------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|:---------------------|
| PostgreSQL-1-1-1-1-1 | PostgreSQL-1 | PostgreSQL-1-1-1 | PostgreSQL-1-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 99 | 248 | 0.46 | 7992.02 | 1437.10 | -1 | PostgreSQL-1-1-1-1-1 |
#### Per Phase
| | phase | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id |
|:-----------------|:-----------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|
| PostgreSQL-1-1-1 | PostgreSQL-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 99 | 248 | 0.46 | 7992.02 | 1437.10 | -1 |
### Latency of Timer Execution [ms]
| Queries | PostgreSQL-1-1-1-1-1 |
|:--------------|-----------------------:|
| TPC-DS Q1 | 124.57 |
| TPC-DS Q2 | 359.84 |
| TPC-DS Q3 | 229.97 |
| TPC-DS Q4 | 10549.67 |
| TPC-DS Q5 | 672.73 |
| TPC-DS Q6 | 56874.73 |
| TPC-DS Q7 | 486.99 |
| TPC-DS Q8 | 81.12 |
| TPC-DS Q9 | 2684.87 |
| TPC-DS Q10 | 1301.89 |
| TPC-DS Q11 | 6665.48 |
| TPC-DS Q12 | 95.25 |
| TPC-DS Q13 | 804.45 |
| TPC-DS Q14a+b | 2738.69 |
| TPC-DS Q15 | 179.81 |
| TPC-DS Q16 | 280.22 |
| TPC-DS Q17 | 445.36 |
| TPC-DS Q18 | 583.33 |
| TPC-DS Q19 | 274.57 |
| TPC-DS Q20 | 153.22 |
| TPC-DS Q21 | 285.06 |
| TPC-DS Q22 | 4279.16 |
| TPC-DS Q23a+b | 6031.94 |
| TPC-DS Q24a+b | 1021.23 |
| TPC-DS Q25 | 485.61 |
| TPC-DS Q26 | 385.94 |
| TPC-DS Q27 | 30.05 |
| TPC-DS Q28 | 978.31 |
| TPC-DS Q29 | 516.98 |
| TPC-DS Q30 | 17176.13 |
| TPC-DS Q31 | 2390.55 |
| TPC-DS Q32 | 95.10 |
| TPC-DS Q33 | 601.09 |
| TPC-DS Q34 | 32.98 |
| TPC-DS Q35 | 1459.86 |
| TPC-DS Q36 | 31.54 |
| TPC-DS Q37 | 900.87 |
| TPC-DS Q38 | 1874.15 |
| TPC-DS Q39a+b | 3096.16 |
| TPC-DS Q40 | 192.42 |
| TPC-DS Q41 | 333.00 |
| TPC-DS Q42 | 111.92 |
| TPC-DS Q43 | 32.49 |
| TPC-DS Q44 | 1.88 |
| TPC-DS Q45 | 136.40 |
| TPC-DS Q46 | 43.19 |
| TPC-DS Q47 | 2288.70 |
| TPC-DS Q48 | 837.74 |
| TPC-DS Q49 | 602.92 |
| TPC-DS Q50 | 772.83 |
| TPC-DS Q51 | 1236.34 |
| TPC-DS Q52 | 118.93 |
| TPC-DS Q53 | 136.49 |
| TPC-DS Q54 | 97.60 |
| TPC-DS Q55 | 114.28 |
| TPC-DS Q56 | 669.88 |
| TPC-DS Q57 | 1165.56 |
| TPC-DS Q58 | 601.35 |
| TPC-DS Q59 | 496.81 |
| TPC-DS Q60 | 579.08 |
| TPC-DS Q61 | 169.56 |
| TPC-DS Q62 | 139.90 |
| TPC-DS Q63 | 132.85 |
| TPC-DS Q64 | 857.70 |
| TPC-DS Q65 | 687.14 |
| TPC-DS Q66 | 333.55 |
| TPC-DS Q67 | 3873.62 |
| TPC-DS Q68 | 43.25 |
| TPC-DS Q69 | 273.79 |
| TPC-DS Q70 | 532.23 |
| TPC-DS Q71 | 518.14 |
| TPC-DS Q72 | 1535.14 |
| TPC-DS Q73 | 33.15 |
| TPC-DS Q74 | 1502.64 |
| TPC-DS Q75 | 1045.67 |
| TPC-DS Q76 | 289.81 |
| TPC-DS Q77 | 416.18 |
| TPC-DS Q78 | 1940.28 |
| TPC-DS Q79 | 143.38 |
| TPC-DS Q80 | 642.15 |
| TPC-DS Q81 | 72550.22 |
| TPC-DS Q82 | 407.96 |
| TPC-DS Q83 | 152.31 |
| TPC-DS Q84 | 123.65 |
| TPC-DS Q85 | 405.92 |
| TPC-DS Q86 | 296.71 |
| TPC-DS Q87 | 1875.36 |
| TPC-DS Q88 | 3031.38 |
| TPC-DS Q89 | 148.67 |
| TPC-DS Q90 | 197.72 |
| TPC-DS Q91 | 208.77 |
| TPC-DS Q92 | 120.61 |
| TPC-DS Q93 | 281.55 |
| TPC-DS Q94 | 254.51 |
| TPC-DS Q95 | 5029.72 |
| TPC-DS Q96 | 122.94 |
| TPC-DS Q97 | 485.89 |
| TPC-DS Q98 | 269.11 |
| TPC-DS Q99 | 189.39 |
### Errors (failed queries)
No errors
### Warnings (result mismatch)
No warnings
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Geo Times [s] contains no 0 or NaN
* TEST passed: Power@Size [~Q/h] contains no 0 or NaN
* TEST passed: Throughput@Size contains no 0 or NaN
* TEST passed: No SQL errors
* TEST passed: No SQL warnings
* TEST passed: Workflow as planned
TPC-DS Monitoring
bexhoma tpcds \
-dbms PostgreSQL \
-sf 10 \
-nlp 8 \
-nlt 8 \
-xii -xic -xis \
-xdt \
-nc 1 \
-ne 1 \
-nbp 1 \
-m \
-mc \
-ms $BEXHOMA_MS \
-t 1200 \
-tr \
-lr 128Gi \
-rr 128Gi \
-rss 150Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_tpcds_postgresql_2.log
yields (after ca. 15 minutes) something like
testcase_tpcds_postgresql_2.log
## Show Summary
### Workload
TPC-DS Queries SF=10
* Type: tpcds
* Duration: 7048s
* Code: 1782595993
* This includes the reading queries of TPC-DS.
* This experiment compares run time and resource consumption of TPC-DS queries in different DBMS.
* TPC-DS (SF=10) data is loaded and benchmark is executed.
* Query ordering is Q1 - Q99.
* All instances use the same query parameters.
* Timeout per query is 1200.
* Import sets indexes and constraints after loading and recomputes statistics.
* Experiment uses bexhoma version 0.10.2.
* System metrics are monitored by a cluster-wide installation.
* Experiment is limited to DBMS ['PostgreSQL'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* Database uses ephemeral storage of size 150Gi.
* Loading is tested with [8] threads, split into [8] pods.
* Benchmarking is tested with [1] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* PostgreSQL-1-1-1-1-1 uses docker image postgres:18.3
* RAM:1081742745600
* CPU:AMD EPYC 7502 32-Core Processor
* Cores:128
* host:6.8.0-117-generic
* node:cl-worker29
* disk:660803
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:128Gi
* limits_memory:128Gi
* eval_parameters
* code:1782595993
### SUT Container Restarts
* bexhoma-sut-postgresql-1-1782595993-7677b5d699-zfm6g: 0 0
### Workflow
#### Actual
* DBMS PostgreSQL-1 - Experiment 1 Client 1: tpcds (1 pods)
#### Planned
* DBMS PostgreSQL-1 - Experiment 1 Client 1: tpcds (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:---------------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| PostgreSQL-1-1 | 1 | 10 | 1653.00 | 6.00 | 0.00 | 496.00 | 1142.00 | 8 | 0 | | None | 0 | False | 21.78 |
### Execution
#### Per Connection
| | configuration | phase | job | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id | pod |
|:---------------------|:----------------|:-----------------|:-------------------|-----------------:|---------:|----------------:|------------:|------:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|:---------------------|
| PostgreSQL-1-1-1-1-1 | PostgreSQL-1 | PostgreSQL-1-1-1 | PostgreSQL-1-1-1-1 | 1 | 1 | 1 | 1 | 10.00 | 96 | 5506 | 3.16 | 11477.20 | 627.68 | -1 | PostgreSQL-1-1-1-1-1 |
#### Per Phase
| | phase | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id |
|:-----------------|:-----------------|-----------------:|---------:|----------------:|------------:|------:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|
| PostgreSQL-1-1-1 | PostgreSQL-1-1-1 | 1 | 1 | 1 | 1 | 10.00 | 96 | 5506 | 3.16 | 11477.20 | 627.68 | -1 |
### Latency of Timer Execution [ms]
| Queries | PostgreSQL-1-1-1-1-1 |
|:--------------|-----------------------:|
| TPC-DS Q2 | 2473.69 |
| TPC-DS Q3 | 1454.28 |
| TPC-DS Q4 | 98028.18 |
| TPC-DS Q5 | 5054.95 |
| TPC-DS Q7 | 2403.56 |
| TPC-DS Q8 | 1715.23 |
| TPC-DS Q9 | 7782.47 |
| TPC-DS Q10 | 2819.92 |
| TPC-DS Q11 | 58417.79 |
| TPC-DS Q12 | 594.50 |
| TPC-DS Q13 | 3714.13 |
| TPC-DS Q14a+b | 23174.54 |
| TPC-DS Q15 | 1004.68 |
| TPC-DS Q16 | 2525.16 |
| TPC-DS Q17 | 3268.41 |
| TPC-DS Q18 | 2255.10 |
| TPC-DS Q19 | 2213.33 |
| TPC-DS Q20 | 1063.13 |
| TPC-DS Q21 | 2360.77 |
| TPC-DS Q22 | 48712.96 |
| TPC-DS Q23a+b | 63793.55 |
| TPC-DS Q24a+b | 6047.96 |
| TPC-DS Q25 | 3360.84 |
| TPC-DS Q26 | 1221.89 |
| TPC-DS Q27 | 2566.88 |
| TPC-DS Q28 | 6417.05 |
| TPC-DS Q29 | 3919.92 |
| TPC-DS Q31 | 7771.48 |
| TPC-DS Q32 | 875.20 |
| TPC-DS Q33 | 4951.72 |
| TPC-DS Q34 | 340.16 |
| TPC-DS Q35 | 3546.01 |
| TPC-DS Q36 | 427.48 |
| TPC-DS Q37 | 2715.66 |
| TPC-DS Q38 | 11662.60 |
| TPC-DS Q39a+b | 40993.66 |
| TPC-DS Q40 | 988.24 |
| TPC-DS Q41 | 61020.81 |
| TPC-DS Q42 | 1226.30 |
| TPC-DS Q43 | 2038.20 |
| TPC-DS Q44 | 2.10 |
| TPC-DS Q45 | 681.53 |
| TPC-DS Q46 | 574.82 |
| TPC-DS Q47 | 11807.07 |
| TPC-DS Q48 | 3609.40 |
| TPC-DS Q49 | 4531.62 |
| TPC-DS Q50 | 7187.90 |
| TPC-DS Q51 | 13219.13 |
| TPC-DS Q52 | 1209.94 |
| TPC-DS Q53 | 1297.56 |
| TPC-DS Q54 | 1008.60 |
| TPC-DS Q55 | 1152.27 |
| TPC-DS Q56 | 4257.17 |
| TPC-DS Q57 | 8388.62 |
| TPC-DS Q58 | 4194.10 |
| TPC-DS Q59 | 3613.69 |
| TPC-DS Q60 | 4830.12 |
| TPC-DS Q61 | 1907.61 |
| TPC-DS Q62 | 933.52 |
| TPC-DS Q63 | 1344.40 |
| TPC-DS Q64 | 7382.88 |
| TPC-DS Q65 | 8075.58 |
| TPC-DS Q66 | 4374.84 |
| TPC-DS Q67 | 40688.35 |
| TPC-DS Q68 | 596.04 |
| TPC-DS Q69 | 2265.34 |
| TPC-DS Q70 | 4738.52 |
| TPC-DS Q71 | 4191.63 |
| TPC-DS Q72 | 14392.62 |
| TPC-DS Q73 | 328.37 |
| TPC-DS Q74 | 12506.90 |
| TPC-DS Q75 | 8205.31 |
| TPC-DS Q76 | 2750.68 |
| TPC-DS Q77 | 3409.65 |
| TPC-DS Q78 | 13398.11 |
| TPC-DS Q79 | 1602.84 |
| TPC-DS Q80 | 5602.13 |
| TPC-DS Q81 | 1093594.34 |
| TPC-DS Q82 | 2769.49 |
| TPC-DS Q83 | 630.68 |
| TPC-DS Q84 | 134.30 |
| TPC-DS Q85 | 1106.60 |
| TPC-DS Q86 | 2264.56 |
| TPC-DS Q87 | 10954.12 |
| TPC-DS Q88 | 6770.17 |
| TPC-DS Q89 | 1605.55 |
| TPC-DS Q90 | 714.22 |
| TPC-DS Q91 | 334.94 |
| TPC-DS Q92 | 354.22 |
| TPC-DS Q93 | 3284.99 |
| TPC-DS Q94 | 1465.73 |
| TPC-DS Q95 | 47447.73 |
| TPC-DS Q96 | 1050.60 |
| TPC-DS Q97 | 3918.92 |
| TPC-DS Q98 | 2262.77 |
| TPC-DS Q99 | 1458.33 |
### Errors (failed queries)
| | TPC-DS Q1 | TPC-DS Q2 | TPC-DS Q3 | TPC-DS Q4 | TPC-DS Q5 | TPC-DS Q6 | TPC-DS Q7 | TPC-DS Q8 | TPC-DS Q9 | TPC-DS Q10 | TPC-DS Q11 | TPC-DS Q12 | TPC-DS Q13 | TPC-DS Q14a+b | TPC-DS Q15 | TPC-DS Q16 | TPC-DS Q17 | TPC-DS Q18 | TPC-DS Q19 | TPC-DS Q20 | TPC-DS Q21 | TPC-DS Q22 | TPC-DS Q23a+b | TPC-DS Q24a+b | TPC-DS Q25 | TPC-DS Q26 | TPC-DS Q27 | TPC-DS Q28 | TPC-DS Q29 | TPC-DS Q30 | TPC-DS Q31 | TPC-DS Q32 | TPC-DS Q33 | TPC-DS Q34 | TPC-DS Q35 | TPC-DS Q36 | TPC-DS Q37 | TPC-DS Q38 | TPC-DS Q39a+b | TPC-DS Q40 | TPC-DS Q41 | TPC-DS Q42 | TPC-DS Q43 | TPC-DS Q44 | TPC-DS Q45 | TPC-DS Q46 | TPC-DS Q47 | TPC-DS Q48 | TPC-DS Q49 | TPC-DS Q50 | TPC-DS Q51 | TPC-DS Q52 | TPC-DS Q53 | TPC-DS Q54 | TPC-DS Q55 | TPC-DS Q56 | TPC-DS Q57 | TPC-DS Q58 | TPC-DS Q59 | TPC-DS Q60 | TPC-DS Q61 | TPC-DS Q62 | TPC-DS Q63 | TPC-DS Q64 | TPC-DS Q65 | TPC-DS Q66 | TPC-DS Q67 | TPC-DS Q68 | TPC-DS Q69 | TPC-DS Q70 | TPC-DS Q71 | TPC-DS Q72 | TPC-DS Q73 | TPC-DS Q74 | TPC-DS Q75 | TPC-DS Q76 | TPC-DS Q77 | TPC-DS Q78 | TPC-DS Q79 | TPC-DS Q80 | TPC-DS Q81 | TPC-DS Q82 | TPC-DS Q83 | TPC-DS Q84 | TPC-DS Q85 | TPC-DS Q86 | TPC-DS Q87 | TPC-DS Q88 | TPC-DS Q89 | TPC-DS Q90 | TPC-DS Q91 | TPC-DS Q92 | TPC-DS Q93 | TPC-DS Q94 | TPC-DS Q95 | TPC-DS Q96 | TPC-DS Q97 | TPC-DS Q98 | TPC-DS Q99 |
|:---------------------|------------:|------------:|------------:|------------:|------------:|------------:|------------:|------------:|------------:|-------------:|-------------:|-------------:|-------------:|----------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|----------------:|----------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|----------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|
| PostgreSQL-1-1-1-1-1 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
* TPC-DS Q1
* PostgreSQL-1-1-1-1-1: numRun 1: : org.postgresql.util.PSQLException: ERROR: canceling statement due to user request
* TPC-DS Q6
* PostgreSQL-1-1-1-1-1: numRun 1: : org.postgresql.util.PSQLException: ERROR: canceling statement due to user request
* TPC-DS Q30
* PostgreSQL-1-1-1-1-1: numRun 1: : org.postgresql.util.PSQLException: ERROR: canceling statement due to user request
### Warnings (result mismatch)
No warnings
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 2568.70 | 4.41 | 19.59 | 48.11 |
### Loading phase: component data generator
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 0.00 | 0.00 | 0.00 | 0.00 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 132.67 | 0.77 | 0.01 | 2.82 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 7048.38 | 5.58 | 21.33 | 53.06 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 28.48 | 0.43 | 0.31 | 0.32 |
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST skipped: Loading phase: component data generator contains 0 or NaN in CPU [CPUs] (data pre-existing)
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Geo Times [s] contains no 0 or NaN
* TEST passed: Power@Size [~Q/h] contains no 0 or NaN
* TEST passed: Throughput@Size contains no 0 or NaN
* TEST failed: SQL errors
* TEST passed: No SQL warnings
* TEST passed: Workflow as planned
TPC-H Throughput Test
bexhoma tpcds \
-dbms PostgreSQL \
-sf 10 \
-nlp 8 \
-nlt 8 \
-xii -xic -xis \
-xdt \
-nc 2 \
-ne 1,2 \
-nbp 1 \
-m \
-mc \
-ms $BEXHOMA_MS \
-t 1200 \
-tr \
-lr 128Gi \
-rr 128Gi \
-rsr \
-rst $BEXHOMA_STORAGE_CLASS \
-rss 150Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_tpcds_postgresql_3.log
yields (after ca. 15 minutes) something like
testcase_tpcds_postgresql_3.log
## Show Summary
### Workload
TPC-DS Queries SF=10
* Type: tpcds
* Duration: 20047s
* Code: 1782560700
* This includes the reading queries of TPC-DS.
* This experiment compares run time and resource consumption of TPC-DS queries in different DBMS.
* TPC-DS (SF=10) data is loaded and benchmark is executed.
* Query ordering is Q1 - Q99.
* All instances use the same query parameters.
* Timeout per query is 1200.
* Import sets indexes and constraints after loading and recomputes statistics.
* Experiment uses bexhoma version 0.10.2.
* System metrics are monitored by a cluster-wide installation.
* Experiment is limited to DBMS ['PostgreSQL'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* Database is persisted to disk of type shared and size 100Gi.
* Loading is tested with [8] threads, split into [8] pods.
* Benchmarking is tested with [1] threads, split into [1] pods.
* Benchmarking is run as [1, 2] times the number of benchmarking pods.
* Experiment is run 2 times.
### Connections
* PostgreSQL-1-1-1-1-1 uses docker image postgres:18.3
* RAM:1081742745600
* CPU:AMD EPYC 7502 32-Core Processor
* Cores:128
* host:6.8.0-117-generic
* node:cl-worker29
* disk:598935
* volume_size:100G
* volume_used:54G
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:128Gi
* limits_memory:128Gi
* eval_parameters
* code:1782560700
* PostgreSQL-1-1-2-1-1 uses docker image postgres:18.3
* RAM:1081742745600
* CPU:AMD EPYC 7502 32-Core Processor
* Cores:128
* host:6.8.0-117-generic
* node:cl-worker29
* disk:598940
* volume_size:100G
* volume_used:54G
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:128Gi
* limits_memory:128Gi
* eval_parameters
* code:1782560700
* PostgreSQL-1-1-2-1-2 uses docker image postgres:18.3
* RAM:1081742745600
* CPU:AMD EPYC 7502 32-Core Processor
* Cores:128
* host:6.8.0-117-generic
* node:cl-worker29
* disk:598940
* volume_size:100G
* volume_used:54G
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:128Gi
* limits_memory:128Gi
* eval_parameters
* code:1782560700
* PostgreSQL-1-2-1-1-1 uses docker image postgres:18.3
* RAM:1081742745600
* CPU:AMD EPYC 7502 32-Core Processor
* Cores:128
* host:6.8.0-117-generic
* node:cl-worker29
* disk:599040
* volume_size:100G
* volume_used:54G
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:128Gi
* limits_memory:128Gi
* eval_parameters
* code:1782560700
* PostgreSQL-1-2-2-1-1 uses docker image postgres:18.3
* RAM:1081742745600
* CPU:AMD EPYC 7502 32-Core Processor
* Cores:128
* host:6.8.0-117-generic
* node:cl-worker29
* disk:598938
* volume_size:100G
* volume_used:54G
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:128Gi
* limits_memory:128Gi
* eval_parameters
* code:1782560700
* PostgreSQL-1-2-2-1-2 uses docker image postgres:18.3
* RAM:1081742745600
* CPU:AMD EPYC 7502 32-Core Processor
* Cores:128
* host:6.8.0-117-generic
* node:cl-worker29
* disk:598938
* volume_size:100G
* volume_used:54G
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:128Gi
* limits_memory:128Gi
* eval_parameters
* code:1782560700
### SUT Container Restarts
* bexhoma-sut-postgresql-1-1782560700-5f7cf4fb57-8zdgk: 0 0
### Workflow
#### Actual
* DBMS PostgreSQL-1 - Experiment 1 Client 1: tpcds (1 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 2: tpcds (2 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 1: tpcds (1 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 2: tpcds (2 pods)
#### Planned
* DBMS PostgreSQL-1 - Experiment 1 Client 1: tpcds (1 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 2: tpcds (2 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 1: tpcds (1 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 2: tpcds (2 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:---------------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| PostgreSQL-1-1 | 1 | 10 | 1940.00 | 7.00 | 1.00 | 770.00 | 1154.00 | 0 | 0 | | None | 0 | False | 18.56 |
| PostgreSQL-1-2 | 2 | 10 | 1940.00 | 7.00 | 1.00 | 770.00 | 1154.00 | 0 | 0 | | None | 0 | False | 18.56 |
### Execution
#### Per Connection
| | configuration | phase | job | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id | pod |
|:---------------------|:----------------|:-----------------|:-------------------|-----------------:|---------:|----------------:|------------:|------:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|:---------------------|
| PostgreSQL-1-1-1-1-1 | PostgreSQL-1 | PostgreSQL-1-1-1 | PostgreSQL-1-1-1-1 | 1 | 1 | 1 | 1 | 10.00 | 96 | 4602 | 3.82 | 9510.63 | 750.98 | -1 | PostgreSQL-1-1-1-1-1 |
| PostgreSQL-1-1-2-1-1 | PostgreSQL-1 | PostgreSQL-1-1-2 | PostgreSQL-1-1-2-1 | 1 | 2 | 1 | 1 | 10.00 | 96 | 4557 | 3.48 | 10430.18 | 758.39 | -1 | PostgreSQL-1-1-2-1-1 |
| PostgreSQL-1-1-2-1-2 | PostgreSQL-1 | PostgreSQL-1-1-2 | PostgreSQL-1-1-2-1 | 1 | 2 | 1 | 1 | 10.00 | 96 | 4514 | 3.44 | 10560.34 | 765.62 | -1 | PostgreSQL-1-1-2-1-2 |
| PostgreSQL-1-2-1-1-1 | PostgreSQL-1 | PostgreSQL-1-2-1 | PostgreSQL-1-2-1-1 | 2 | 1 | 1 | 1 | 10.00 | 96 | 4592 | 3.74 | 9700.45 | 752.61 | -1 | PostgreSQL-1-2-1-1-1 |
| PostgreSQL-1-2-2-1-1 | PostgreSQL-1 | PostgreSQL-1-2-2 | PostgreSQL-1-2-2-1 | 2 | 2 | 1 | 1 | 10.00 | 96 | 4498 | 3.28 | 11073.94 | 768.34 | -1 | PostgreSQL-1-2-2-1-1 |
| PostgreSQL-1-2-2-1-2 | PostgreSQL-1 | PostgreSQL-1-2-2 | PostgreSQL-1-2-2-1 | 2 | 2 | 1 | 1 | 10.00 | 96 | 4470 | 3.30 | 11025.68 | 773.15 | -1 | PostgreSQL-1-2-2-1-2 |
#### Per Phase
| | phase | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id |
|:-----------------|:-----------------|-----------------:|---------:|----------------:|------------:|------:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|
| PostgreSQL-1-1-1 | PostgreSQL-1-1-1 | 1 | 1 | 1 | 1 | 10.00 | 96 | 4602 | 3.82 | 9510.63 | 750.98 | -1 |
| PostgreSQL-1-1-2 | PostgreSQL-1-1-2 | 1 | 2 | 1 | 2 | 10.00 | 192 | 4557 | 3.46 | 10495.06 | 1516.79 | -1 |
| PostgreSQL-1-2-1 | PostgreSQL-1-2-1 | 2 | 1 | 1 | 1 | 10.00 | 96 | 4592 | 3.74 | 9700.45 | 752.61 | -1 |
| PostgreSQL-1-2-2 | PostgreSQL-1-2-2 | 2 | 2 | 1 | 2 | 10.00 | 192 | 4498 | 3.29 | 11049.78 | 1536.68 | -1 |
### Latency of Timer Execution [ms]
| Queries | PostgreSQL-1-1-1-1-1 | PostgreSQL-1-1-2-1-1 | PostgreSQL-1-1-2-1-2 | PostgreSQL-1-2-1-1-1 | PostgreSQL-1-2-2-1-1 | PostgreSQL-1-2-2-1-2 |
|:--------------|-----------------------:|-----------------------:|-----------------------:|-----------------------:|-----------------------:|-----------------------:|
| TPC-DS Q1 | 6864.23 | 1287.53 | 1288.40 | 6654.30 | 1284.29 | 1286.04 |
| TPC-DS Q2 | 49063.73 | 2862.42 | 3051.10 | 46396.92 | 2700.57 | 2626.98 |
| TPC-DS Q3 | 44381.79 | 1224.51 | 1255.87 | 42144.18 | 1181.86 | 1521.76 |
| TPC-DS Q4 | 75612.30 | 77866.59 | 75360.98 | 75636.84 | 77479.30 | 75644.91 |
| TPC-DS Q5 | 7234.88 | 5328.44 | 5613.47 | 8387.75 | 4999.13 | 5328.28 |
| TPC-DS Q7 | 3595.14 | 2419.30 | 2234.55 | 5089.92 | 2305.50 | 2320.12 |
| TPC-DS Q8 | 1717.48 | 1757.33 | 1738.09 | 1701.99 | 1706.75 | 1707.59 |
| TPC-DS Q9 | 7566.32 | 8323.89 | 11121.54 | 7471.68 | 8743.24 | 8781.13 |
| TPC-DS Q10 | 2996.21 | 3266.27 | 3153.45 | 2942.08 | 2898.98 | 2846.80 |
| TPC-DS Q11 | 57857.73 | 59763.26 | 58652.92 | 58882.46 | 62006.06 | 58236.27 |
| TPC-DS Q12 | 364.88 | 373.65 | 374.99 | 364.07 | 377.86 | 380.44 |
| TPC-DS Q13 | 5128.88 | 5211.34 | 5192.79 | 5359.39 | 5192.40 | 5251.56 |
| TPC-DS Q14a+b | 39135.49 | 61241.38 | 53772.58 | 37671.80 | 58073.87 | 41862.17 |
| TPC-DS Q15 | 1221.31 | 1197.09 | 2909.40 | 958.93 | 937.38 | 921.76 |
| TPC-DS Q16 | 2665.81 | 2660.81 | 2642.42 | 2545.98 | 2536.33 | 2650.40 |
| TPC-DS Q17 | 2710.42 | 2641.08 | 2645.08 | 2482.44 | 2543.91 | 3356.72 |
| TPC-DS Q18 | 2451.72 | 2302.06 | 2300.66 | 2105.79 | 2104.93 | 2066.25 |
| TPC-DS Q19 | 2781.88 | 2800.15 | 2846.43 | 2665.16 | 2712.20 | 2668.90 |
| TPC-DS Q20 | 779.45 | 786.99 | 772.91 | 589.66 | 590.76 | 1070.03 |
| TPC-DS Q21 | 66640.93 | 2513.24 | 2511.14 | 53656.03 | 2306.05 | 4260.89 |
| TPC-DS Q22 | 47052.41 | 48444.27 | 47674.00 | 46772.88 | 47195.68 | 50293.06 |
| TPC-DS Q23a+b | 47551.82 | 60276.55 | 48565.95 | 47620.30 | 48063.57 | 47679.92 |
| TPC-DS Q24a+b | 4143.61 | 4171.59 | 4122.02 | 4128.40 | 4109.26 | 4115.34 |
| TPC-DS Q25 | 3786.14 | 3995.89 | 4141.58 | 3414.56 | 3574.53 | 3658.17 |
| TPC-DS Q26 | 1436.64 | 1615.51 | 1468.76 | 1141.46 | 1251.56 | 1238.39 |
| TPC-DS Q27 | 2783.28 | 2843.87 | 2751.33 | 2755.13 | 2702.43 | 2714.70 |
| TPC-DS Q28 | 6263.95 | 6397.14 | 7657.10 | 6305.08 | 6307.15 | 7691.57 |
| TPC-DS Q29 | 3875.03 | 4234.14 | 3973.39 | 3492.17 | 3598.11 | 3677.84 |
| TPC-DS Q31 | 8505.10 | 9778.24 | 7715.16 | 7731.79 | 7689.59 | 7671.33 |
| TPC-DS Q32 | 2610.27 | 1250.90 | 985.23 | 2423.38 | 814.70 | 831.41 |
| TPC-DS Q33 | 4902.11 | 5190.66 | 5002.44 | 4691.17 | 5508.79 | 4768.15 |
| TPC-DS Q34 | 311.24 | 317.08 | 323.26 | 318.01 | 316.97 | 318.10 |
| TPC-DS Q35 | 3579.45 | 5782.56 | 3561.91 | 3464.86 | 4924.19 | 3461.54 |
| TPC-DS Q36 | 815.60 | 810.03 | 789.67 | 812.40 | 836.64 | 810.67 |
| TPC-DS Q37 | 2945.51 | 3767.18 | 2945.75 | 2696.74 | 2677.08 | 2659.72 |
| TPC-DS Q38 | 8609.82 | 22697.25 | 22080.25 | 8245.00 | 22477.01 | 10472.46 |
| TPC-DS Q39a+b | 48948.87 | 49826.09 | 49506.95 | 49330.32 | 49783.56 | 49438.64 |
| TPC-DS Q40 | 1007.20 | 1026.41 | 940.35 | 783.18 | 789.61 | 793.38 |
| TPC-DS Q41 | 55621.69 | 102419.45 | 86183.00 | 71947.66 | 74858.26 | 75313.36 |
| TPC-DS Q42 | 1527.45 | 1218.97 | 1196.26 | 1217.23 | 1211.82 | 1186.73 |
| TPC-DS Q43 | 798.42 | 815.32 | 800.67 | 809.10 | 804.86 | 799.96 |
| TPC-DS Q44 | 973.61 | 749.81 | 729.27 | 857.93 | 730.68 | 735.34 |
| TPC-DS Q45 | 822.08 | 814.71 | 676.74 | 688.96 | 770.96 | 775.52 |
| TPC-DS Q46 | 848.43 | 869.99 | 845.16 | 873.35 | 853.51 | 859.15 |
| TPC-DS Q47 | 11318.78 | 11876.79 | 11120.03 | 11291.17 | 11301.36 | 11414.76 |
| TPC-DS Q48 | 5699.55 | 5711.18 | 5048.38 | 5248.63 | 5348.71 | 5489.29 |
| TPC-DS Q49 | 4956.01 | 11022.77 | 4793.98 | 4678.23 | 4647.56 | 4608.71 |
| TPC-DS Q50 | 5385.14 | 5318.68 | 4858.67 | 4858.55 | 4986.40 | 4972.04 |
| TPC-DS Q51 | 10883.67 | 10971.47 | 10774.89 | 10661.73 | 11178.37 | 11168.31 |
| TPC-DS Q52 | 1501.88 | 1220.08 | 1496.87 | 1202.65 | 1191.77 | 1213.59 |
| TPC-DS Q53 | 3178.00 | 3272.05 | 2944.67 | 2938.67 | 3088.27 | 3021.13 |
| TPC-DS Q54 | 2796.10 | 2928.12 | 3626.40 | 2852.52 | 2878.84 | 2655.80 |
| TPC-DS Q55 | 1419.93 | 1145.70 | 1388.88 | 1141.36 | 1156.20 | 1150.54 |
| TPC-DS Q56 | 4950.04 | 5274.79 | 4825.59 | 4912.74 | 4921.27 | 4793.01 |
| TPC-DS Q57 | 6474.38 | 6167.74 | 6394.88 | 6053.59 | 6103.45 | 6034.22 |
| TPC-DS Q58 | 4363.92 | 5309.79 | 4989.19 | 4501.11 | 5044.58 | 5834.71 |
| TPC-DS Q59 | 3554.90 | 3576.52 | 3539.98 | 3552.68 | 3551.43 | 3557.24 |
| TPC-DS Q60 | 4496.69 | 4876.58 | 4452.13 | 4401.33 | 4461.10 | 4412.26 |
| TPC-DS Q61 | 4735.82 | 4698.12 | 4532.63 | 4415.84 | 4472.78 | 4533.64 |
| TPC-DS Q62 | 920.80 | 890.17 | 928.60 | 908.79 | 892.00 | 924.20 |
| TPC-DS Q63 | 3318.60 | 3304.18 | 3006.72 | 3091.61 | 3152.51 | 3102.28 |
| TPC-DS Q64 | 6483.38 | 6760.75 | 10967.58 | 6548.49 | 7419.66 | 7369.33 |
| TPC-DS Q65 | 7839.14 | 7955.65 | 12793.19 | 7802.83 | 8357.48 | 12884.65 |
| TPC-DS Q66 | 4060.93 | 3996.26 | 3686.85 | 3802.28 | 3723.29 | 3732.09 |
| TPC-DS Q67 | 68652.68 | 68711.16 | 67083.98 | 67254.59 | 67996.00 | 67955.81 |
| TPC-DS Q68 | 864.10 | 867.02 | 854.43 | 854.27 | 860.76 | 874.68 |
| TPC-DS Q69 | 1175.07 | 1189.56 | 1183.52 | 1168.55 | 1161.32 | 1156.47 |
| TPC-DS Q70 | 4806.47 | 5337.94 | 5266.56 | 4671.18 | 5251.21 | 4655.96 |
| TPC-DS Q71 | 4407.62 | 4457.89 | 4043.10 | 3909.76 | 3943.92 | 3954.00 |
| TPC-DS Q72 | 7358.33 | 7709.52 | 7106.23 | 7232.08 | 7275.88 | 7170.91 |
| TPC-DS Q73 | 332.92 | 331.59 | 312.72 | 315.60 | 314.14 | 313.19 |
| TPC-DS Q74 | 19316.14 | 19040.57 | 18910.83 | 19252.78 | 19660.40 | 18953.04 |
| TPC-DS Q75 | 10727.66 | 10550.22 | 11419.10 | 10490.25 | 14674.50 | 10490.73 |
| TPC-DS Q76 | 3430.54 | 3228.12 | 3264.92 | 3423.29 | 3164.43 | 3166.86 |
| TPC-DS Q77 | 3769.78 | 3767.69 | 3745.39 | 3646.42 | 3587.28 | 3598.48 |
| TPC-DS Q78 | 14209.02 | 13732.35 | 15094.61 | 13473.19 | 13529.95 | 14639.56 |
| TPC-DS Q79 | 2406.82 | 2282.81 | 2220.32 | 2244.28 | 2254.92 | 2238.01 |
| TPC-DS Q80 | 5097.76 | 5073.22 | 4845.86 | 4755.09 | 4741.17 | 4723.89 |
| TPC-DS Q82 | 2875.87 | 2818.41 | 2849.90 | 2712.46 | 2730.22 | 2702.60 |
| TPC-DS Q83 | 603.11 | 609.69 | 603.87 | 552.42 | 553.00 | 570.45 |
| TPC-DS Q84 | 315.27 | 302.44 | 310.12 | 296.05 | 298.62 | 293.17 |
| TPC-DS Q85 | 1203.19 | 1179.43 | 1230.17 | 1137.51 | 1217.50 | 1187.26 |
| TPC-DS Q86 | 1156.22 | 1166.51 | 1220.56 | 1161.21 | 1164.48 | 1158.14 |
| TPC-DS Q87 | 7733.10 | 7682.83 | 7996.92 | 7583.80 | 7633.41 | 7604.10 |
| TPC-DS Q88 | 6695.01 | 6676.59 | 6746.99 | 6587.64 | 6594.14 | 6569.07 |
| TPC-DS Q89 | 1692.60 | 1735.20 | 1719.66 | 1717.03 | 1726.41 | 1708.78 |
| TPC-DS Q90 | 661.17 | 630.31 | 627.92 | 622.64 | 630.32 | 620.38 |
| TPC-DS Q91 | 390.26 | 379.81 | 393.65 | 373.85 | 360.70 | 369.92 |
| TPC-DS Q92 | 322.09 | 309.28 | 322.33 | 3252.58 | 311.12 | 318.59 |
| TPC-DS Q93 | 5622.66 | 5689.11 | 5692.37 | 5704.70 | 5635.01 | 5684.27 |
| TPC-DS Q94 | 1705.39 | 1664.03 | 1634.94 | 1627.97 | 1597.49 | 1599.61 |
| TPC-DS Q95 | 61528.79 | 62564.16 | 62576.78 | 61486.30 | 61783.12 | 61984.23 |
| TPC-DS Q96 | 960.20 | 983.35 | 943.46 | 944.36 | 940.36 | 925.18 |
| TPC-DS Q97 | 3959.34 | 3967.73 | 3918.11 | 3785.47 | 3778.98 | 3792.19 |
| TPC-DS Q98 | 1249.29 | 1250.72 | 1207.50 | 1244.56 | 1236.85 | 1210.30 |
| TPC-DS Q99 | 1715.49 | 1671.38 | 1626.19 | 1463.42 | 1439.39 | 1448.02 |
### Errors (failed queries)
| | TPC-DS Q1 | TPC-DS Q2 | TPC-DS Q3 | TPC-DS Q4 | TPC-DS Q5 | TPC-DS Q6 | TPC-DS Q7 | TPC-DS Q8 | TPC-DS Q9 | TPC-DS Q10 | TPC-DS Q11 | TPC-DS Q12 | TPC-DS Q13 | TPC-DS Q14a+b | TPC-DS Q15 | TPC-DS Q16 | TPC-DS Q17 | TPC-DS Q18 | TPC-DS Q19 | TPC-DS Q20 | TPC-DS Q21 | TPC-DS Q22 | TPC-DS Q23a+b | TPC-DS Q24a+b | TPC-DS Q25 | TPC-DS Q26 | TPC-DS Q27 | TPC-DS Q28 | TPC-DS Q29 | TPC-DS Q30 | TPC-DS Q31 | TPC-DS Q32 | TPC-DS Q33 | TPC-DS Q34 | TPC-DS Q35 | TPC-DS Q36 | TPC-DS Q37 | TPC-DS Q38 | TPC-DS Q39a+b | TPC-DS Q40 | TPC-DS Q41 | TPC-DS Q42 | TPC-DS Q43 | TPC-DS Q44 | TPC-DS Q45 | TPC-DS Q46 | TPC-DS Q47 | TPC-DS Q48 | TPC-DS Q49 | TPC-DS Q50 | TPC-DS Q51 | TPC-DS Q52 | TPC-DS Q53 | TPC-DS Q54 | TPC-DS Q55 | TPC-DS Q56 | TPC-DS Q57 | TPC-DS Q58 | TPC-DS Q59 | TPC-DS Q60 | TPC-DS Q61 | TPC-DS Q62 | TPC-DS Q63 | TPC-DS Q64 | TPC-DS Q65 | TPC-DS Q66 | TPC-DS Q67 | TPC-DS Q68 | TPC-DS Q69 | TPC-DS Q70 | TPC-DS Q71 | TPC-DS Q72 | TPC-DS Q73 | TPC-DS Q74 | TPC-DS Q75 | TPC-DS Q76 | TPC-DS Q77 | TPC-DS Q78 | TPC-DS Q79 | TPC-DS Q80 | TPC-DS Q81 | TPC-DS Q82 | TPC-DS Q83 | TPC-DS Q84 | TPC-DS Q85 | TPC-DS Q86 | TPC-DS Q87 | TPC-DS Q88 | TPC-DS Q89 | TPC-DS Q90 | TPC-DS Q91 | TPC-DS Q92 | TPC-DS Q93 | TPC-DS Q94 | TPC-DS Q95 | TPC-DS Q96 | TPC-DS Q97 | TPC-DS Q98 | TPC-DS Q99 |
|:---------------------|------------:|------------:|------------:|------------:|------------:|------------:|------------:|------------:|------------:|-------------:|-------------:|-------------:|-------------:|----------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|----------------:|----------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|----------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|
| PostgreSQL-1-1-1-1-1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| PostgreSQL-1-1-2-1-1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| PostgreSQL-1-1-2-1-2 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| PostgreSQL-1-2-1-1-1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| PostgreSQL-1-2-2-1-1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| PostgreSQL-1-2-2-1-2 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
* TPC-DS Q6
* PostgreSQL-1-1-2-1-1: numRun 1: : org.postgresql.util.PSQLException: ERROR: canceling statement due to user request
* PostgreSQL-1-1-2-1-2: numRun 1: : org.postgresql.util.PSQLException: ERROR: canceling statement due to user request
* PostgreSQL-1-2-2-1-1: numRun 1: : org.postgresql.util.PSQLException: ERROR: canceling statement due to user request
* PostgreSQL-1-1-1-1-1: numRun 1: : org.postgresql.util.PSQLException: ERROR: canceling statement due to user request
* PostgreSQL-1-2-2-1-2: numRun 1: : org.postgresql.util.PSQLException: ERROR: canceling statement due to user request
* PostgreSQL-1-2-1-1-1: numRun 1: : org.postgresql.util.PSQLException: ERROR: canceling statement due to user request
* TPC-DS Q30
* PostgreSQL-1-1-2-1-1: numRun 1: : org.postgresql.util.PSQLException: ERROR: canceling statement due to user request
* PostgreSQL-1-1-2-1-2: numRun 1: : org.postgresql.util.PSQLException: ERROR: canceling statement due to user request
* PostgreSQL-1-2-2-1-1: numRun 1: : org.postgresql.util.PSQLException: ERROR: canceling statement due to user request
* PostgreSQL-1-1-1-1-1: numRun 1: : org.postgresql.util.PSQLException: ERROR: canceling statement due to user request
* PostgreSQL-1-2-2-1-2: numRun 1: : org.postgresql.util.PSQLException: ERROR: canceling statement due to user request
* PostgreSQL-1-2-1-1-1: numRun 1: : org.postgresql.util.PSQLException: ERROR: canceling statement due to user request
* TPC-DS Q81
* PostgreSQL-1-1-2-1-1: numRun 1: : org.postgresql.util.PSQLException: ERROR: canceling statement due to user request
* PostgreSQL-1-1-2-1-2: numRun 1: : org.postgresql.util.PSQLException: ERROR: canceling statement due to user request
* PostgreSQL-1-2-2-1-1: numRun 1: : org.postgresql.util.PSQLException: ERROR: canceling statement due to user request
* PostgreSQL-1-1-1-1-1: numRun 1: : org.postgresql.util.PSQLException: ERROR: canceling statement due to user request
* PostgreSQL-1-2-2-1-2: numRun 1: : org.postgresql.util.PSQLException: ERROR: canceling statement due to user request
* PostgreSQL-1-2-1-1-1: numRun 1: : org.postgresql.util.PSQLException: ERROR: canceling statement due to user request
### Warnings (result mismatch)
| | TPC-DS Q1 | TPC-DS Q2 | TPC-DS Q3 | TPC-DS Q4 | TPC-DS Q5 | TPC-DS Q6 | TPC-DS Q7 | TPC-DS Q8 | TPC-DS Q9 | TPC-DS Q10 | TPC-DS Q11 | TPC-DS Q12 | TPC-DS Q13 | TPC-DS Q14a+b | TPC-DS Q15 | TPC-DS Q16 | TPC-DS Q17 | TPC-DS Q18 | TPC-DS Q19 | TPC-DS Q20 | TPC-DS Q21 | TPC-DS Q22 | TPC-DS Q23a+b | TPC-DS Q24a+b | TPC-DS Q25 | TPC-DS Q26 | TPC-DS Q27 | TPC-DS Q28 | TPC-DS Q29 | TPC-DS Q30 | TPC-DS Q31 | TPC-DS Q32 | TPC-DS Q33 | TPC-DS Q34 | TPC-DS Q35 | TPC-DS Q36 | TPC-DS Q37 | TPC-DS Q38 | TPC-DS Q39a+b | TPC-DS Q40 | TPC-DS Q41 | TPC-DS Q42 | TPC-DS Q43 | TPC-DS Q44 | TPC-DS Q45 | TPC-DS Q46 | TPC-DS Q47 | TPC-DS Q48 | TPC-DS Q49 | TPC-DS Q50 | TPC-DS Q51 | TPC-DS Q52 | TPC-DS Q53 | TPC-DS Q54 | TPC-DS Q55 | TPC-DS Q56 | TPC-DS Q57 | TPC-DS Q58 | TPC-DS Q59 | TPC-DS Q60 | TPC-DS Q61 | TPC-DS Q62 | TPC-DS Q63 | TPC-DS Q64 | TPC-DS Q65 | TPC-DS Q66 | TPC-DS Q67 | TPC-DS Q68 | TPC-DS Q69 | TPC-DS Q70 | TPC-DS Q71 | TPC-DS Q72 | TPC-DS Q73 | TPC-DS Q74 | TPC-DS Q75 | TPC-DS Q76 | TPC-DS Q77 | TPC-DS Q78 | TPC-DS Q79 | TPC-DS Q80 | TPC-DS Q81 | TPC-DS Q82 | TPC-DS Q83 | TPC-DS Q84 | TPC-DS Q85 | TPC-DS Q86 | TPC-DS Q87 | TPC-DS Q88 | TPC-DS Q89 | TPC-DS Q90 | TPC-DS Q91 | TPC-DS Q92 | TPC-DS Q93 | TPC-DS Q94 | TPC-DS Q95 | TPC-DS Q96 | TPC-DS Q97 | TPC-DS Q98 | TPC-DS Q99 |
|:---------------------|------------:|------------:|------------:|------------:|------------:|------------:|------------:|------------:|------------:|-------------:|-------------:|-------------:|-------------:|----------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|----------------:|----------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|----------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|
| PostgreSQL-1-1-1-1-1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| PostgreSQL-1-1-2-1-2 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| PostgreSQL-1-2-1-1-1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| PostgreSQL-1-2-2-1-1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| PostgreSQL-1-2-2-1-2 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
### Monitoring
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 6234.14 | 5.73 | 20.46 | 61.87 |
| PostgreSQL-1-1-2-1 | 12248.90 | 11.73 | 24.40 | 63.28 |
| PostgreSQL-1-2-1-1 | 18784.44 | 6.05 | 16.67 | 47.31 |
| PostgreSQL-1-2-2-1 | 12076.53 | 12.04 | 17.70 | 40.99 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 26.57 | 0.38 | 0.34 | 0.34 |
| PostgreSQL-1-1-2-1 | 27.42 | 1.13 | 0.39 | 0.39 |
| PostgreSQL-1-2-1-1 | 26.40 | 0.07 | 0.35 | 0.35 |
| PostgreSQL-1-2-2-1 | 29.03 | 0.16 | 0.35 | 0.36 |
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Geo Times [s] contains no 0 or NaN
* TEST passed: Power@Size [~Q/h] contains no 0 or NaN
* TEST passed: Throughput@Size contains no 0 or NaN
* TEST failed: SQL errors
* TEST failed: SQL warnings (result mismatch)
* TEST passed: Workflow as planned
MySQL
TPC-DS Simple
bexhoma tpcds \
-dbms MySQL \
-sf 1 \
-nlp 8 \
-nlt 8 \
-xii -xic -xis \
-xdt \
-nc 1 \
-ne 1 \
-nbp 1 \
-ms $BEXHOMA_MS \
-t 1200 \
-tr \
-lr 128Gi \
-rr 128Gi \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_tpcds_mysql_1.log
yields (after ca. 10 minutes) something like
testcase_tpcds_mysql_1.log
## Show Summary
### Workload
TPC-DS Queries SF=1
* Type: tpcds
* Duration: 1233s
* Code: 1782320326
* This includes the reading queries of TPC-DS.
* This experiment compares run time and resource consumption of TPC-DS queries in different DBMS.
* TPC-DS (SF=1) data is loaded and benchmark is executed.
* Query ordering is Q1 - Q99.
* All instances use the same query parameters.
* Timeout per query is 1200.
* Import sets indexes and constraints after loading and recomputes statistics.
* Experiment uses bexhoma version 0.10.0.
* Experiment is limited to DBMS ['MySQL'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker38.
* Database uses ephemeral storage of size 10Gi.
* Loading is tested with [8] threads, split into [8] pods.
* Benchmarking is tested with [1] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* MySQL-1-1-1-1-1 uses docker image mysql:8.4.0
* RAM:540492877824
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:258698
* cpu_list:0-127
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782320326
### Workflow
#### Actual
* DBMS MySQL-1 - Experiment 1 Client 1: tpcds (1 pods)
#### Planned
* DBMS MySQL-1 - Experiment 1 Client 1: tpcds (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:----------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| MySQL-1-1 | 1 | 1 | 861.00 | 2.00 | 1.00 | 89.00 | 762.00 | 8 | 0 | | None | 0 | False | 4.18 |
### Execution
#### Per Connection
| | configuration | phase | job | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id | pod |
|:----------------|:----------------|:------------|:--------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|:----------------|
| MySQL-1-1-1-1-1 | MySQL-1 | MySQL-1-1-1 | MySQL-1-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 3 | 24 | 0.20 | 19181.42 | 450.00 | -1 | MySQL-1-1-1-1-1 |
#### Per Phase
| | phase | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id |
|:------------|:------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|
| MySQL-1-1-1 | MySQL-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 3 | 24 | 0.20 | 19181.42 | 450.00 | -1 |
### Latency of Timer Execution [ms]
| Queries | MySQL-1-1-1-1-1 |
|:----------|------------------:|
| TPC-DS Q1 | 67.99 |
| TPC-DS Q2 | 5271.93 |
| TPC-DS Q3 | 18.44 |
### Errors (failed queries)
| | TPC-DS Q1 | TPC-DS Q2 | TPC-DS Q3 | TPC-DS Q4 | TPC-DS Q5 | TPC-DS Q6 | TPC-DS Q7 | TPC-DS Q8 | TPC-DS Q9 | TPC-DS Q10 | TPC-DS Q11 | TPC-DS Q12 | TPC-DS Q13 | TPC-DS Q14a+b | TPC-DS Q15 | TPC-DS Q16 | TPC-DS Q17 | TPC-DS Q18 | TPC-DS Q19 | TPC-DS Q20 | TPC-DS Q21 | TPC-DS Q22 | TPC-DS Q23a+b | TPC-DS Q24a+b | TPC-DS Q25 | TPC-DS Q26 | TPC-DS Q27 | TPC-DS Q28 | TPC-DS Q29 | TPC-DS Q30 | TPC-DS Q31 | TPC-DS Q32 | TPC-DS Q33 | TPC-DS Q34 | TPC-DS Q35 | TPC-DS Q36 | TPC-DS Q37 | TPC-DS Q38 | TPC-DS Q39a+b | TPC-DS Q40 | TPC-DS Q41 | TPC-DS Q42 | TPC-DS Q43 | TPC-DS Q44 | TPC-DS Q45 | TPC-DS Q46 | TPC-DS Q47 | TPC-DS Q48 | TPC-DS Q49 | TPC-DS Q50 | TPC-DS Q51 | TPC-DS Q52 | TPC-DS Q53 | TPC-DS Q54 | TPC-DS Q55 | TPC-DS Q56 | TPC-DS Q57 | TPC-DS Q58 | TPC-DS Q59 | TPC-DS Q60 | TPC-DS Q61 | TPC-DS Q62 | TPC-DS Q63 | TPC-DS Q64 | TPC-DS Q65 | TPC-DS Q66 | TPC-DS Q67 | TPC-DS Q68 | TPC-DS Q69 | TPC-DS Q70 | TPC-DS Q71 | TPC-DS Q72 | TPC-DS Q73 | TPC-DS Q74 | TPC-DS Q75 | TPC-DS Q76 | TPC-DS Q77 | TPC-DS Q78 | TPC-DS Q79 | TPC-DS Q80 | TPC-DS Q81 | TPC-DS Q82 | TPC-DS Q83 | TPC-DS Q84 | TPC-DS Q85 | TPC-DS Q86 | TPC-DS Q87 | TPC-DS Q88 | TPC-DS Q89 | TPC-DS Q90 | TPC-DS Q91 | TPC-DS Q92 | TPC-DS Q93 | TPC-DS Q94 | TPC-DS Q95 | TPC-DS Q96 | TPC-DS Q97 | TPC-DS Q98 | TPC-DS Q99 |
|:----------------|------------:|------------:|------------:|------------:|------------:|------------:|------------:|------------:|------------:|-------------:|-------------:|-------------:|-------------:|----------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|----------------:|----------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|----------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|
| MySQL-1-1-1-1-1 | 0.00 | 0.00 | 0.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
bexhoma : Traceback (most recent call last):
In Zeile:1 Zeichen:1
+ bexhoma tpcds `
+ ~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (Traceback (most recent call last)::String) [], RemoteException
+ FullyQualifiedErrorId : NativeCommandError
File "C:\Users\Patrick\eclipse-workspace\Benchmark-Experiment-Host-Manager\tpcds.py", line 250, in <module>
experiment.process()
File "C:\Users\Patrick\eclipse-workspace\Benchmark-Experiment-Host-Manager\bexhoma\experiments\base.py", line 291, in process
self.show_summary()
File "C:\Users\Patrick\eclipse-workspace\Benchmark-Experiment-Host-Manager\bexhoma\experiments\dbmsbenchmarker.py", line 120, in
show_summary
primary.show_summary(self)
File "C:\Users\Patrick\eclipse-workspace\Benchmark-Experiment-Host-Manager\bexhoma\benchmarks\base.py", line 170, in show_summary
extra_context = self._show_extra_sections(experiment, df_aggregated_reduced)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Patrick\eclipse-workspace\Benchmark-Experiment-Host-Manager\bexhoma\benchmarks\base.py", line 290, in
_show_extra_sections
list_errors = self.evaluator.evaluation.get_error(numQuery)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Patrick\anaconda3\envs\bexhoma\Lib\site-packages\dbmsbenchmarker\inspector.py", line 450, in get_error
return self.benchmarks.getError(numQuery, connection)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Patrick\anaconda3\envs\bexhoma\Lib\site-packages\dbmsbenchmarker\benchmarker.py", line 1926, in getError
return self.protocol['query'][str(query)]['errors']
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: 'ySQL-1-1-1-1-1'
TPC-DS Monitoring
bexhoma tpcds \
-dbms MySQL \
-sf 10 \
-nlp 8 \
-nlt 8 \
-xii -xic -xis \
-xdt \
-nc 1 \
-ne 1 \
-nbp 1 \
-m \
-mc \
-ms $BEXHOMA_MS \
-t 1200 \
-tr \
-lr 128Gi \
-rr 128Gi \
-rsr \
-rss 150Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_tpcds_mysql_2.log
yields (after ca. 15 minutes) something like
testcase_tpcds_mysql_2.log
## Show Summary
### Workload
TPC-DS Queries SF=10
* Type: tpcds
* Duration: 683s
* Code: 1782321594
* This includes the reading queries of TPC-DS.
* This experiment compares run time and resource consumption of TPC-DS queries in different DBMS.
* TPC-DS (SF=10) data is loaded and benchmark is executed.
* Query ordering is Q1 - Q99.
* All instances use the same query parameters.
* Timeout per query is 1200.
* Import sets indexes and constraints after loading and recomputes statistics.
* Experiment uses bexhoma version 0.10.0.
* System metrics are monitored by a cluster-wide installation.
* Experiment is limited to DBMS ['MySQL'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker38.
* Database uses ephemeral storage of size 10Gi.
* Loading is tested with [8] threads, split into [8] pods.
* Benchmarking is tested with [1] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* MySQL-1-1-1-1-1 uses docker image mysql:8.4.0
* RAM:540492877824
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:297170
* cpu_list:0-127
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782321594
### Workflow
#### Actual
* DBMS MySQL-1 - Experiment 1 Client 1: tpcds (1 pods)
#### Planned
* DBMS MySQL-1 - Experiment 1 Client 1: tpcds (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:----------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| MySQL-1-1 | 1 | 10 | 373.00 | 1.00 | 1.00 | 129.00 | 236.00 | 8 | 0 | | None | 0 | False | 96.51 |
### Execution
#### Per Connection
#### Per Phase
### Latency of Timer Execution [ms]
### Errors (failed queries)
No errors
### Warnings (result mismatch)
No warnings
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 703.91 | 10.56 | 15.38 | 16.00 |
### Loading phase: component data generator
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 0.31 | 0.00 | 0.00 | 0.00 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 17.43 | 0.13 | 0.01 | 0.80 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 0.00 | 7.33 | 15.38 | 16.00 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 0.00 | 0.00 | 0.00 | 0.00 |
### Tests
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component data generator contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST failed: Execution phase: SUT deployment contains 0 or NaN in CPU [CPUs]
* TEST failed: Execution phase: component benchmarker contains 0 or NaN in CPU [CPUs]
* TEST failed: Geo Times [s] contains 0 or NaN
* TEST failed: Power@Size [~Q/h] contains 0 or NaN
* TEST failed: Throughput@Size contains 0 or NaN
* TEST passed: No SQL errors
* TEST passed: No SQL warnings
* TEST passed: Workflow as planned
TPC-DS Throughput Test
bexhoma tpcds \
-dbms MySQL \
-sf 10 \
-nlp 8 \
-nlt 8 \
-xii -xic -xis \
-xdt \
-nc 2 \
-ne 1,2 \
-nbp 1 \
-m \
-mc \
-ms $BEXHOMA_MS \
-t 1200 \
-tr \
-lr 128Gi \
-rr 128Gi \
-rsr \
-rst $BEXHOMA_STORAGE_CLASS \
-rss 150Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_tpcds_mysql_3.log
yields (after ca. 15 minutes) something like
testcase_tpcds_mysql_3.log
## Show Summary
### Workload
TPC-DS Queries SF=10
Type: tpcds
Duration: 1381s
Code: 1750148746
This includes the reading queries of TPC-DS.
This experiment compares run time and resource consumption of TPC-DS queries in different DBMS.
TPC-DS (SF=10) data is loaded and benchmark is executed.
Query ordering is Q1 - Q99.
All instances use the same query parameters.
Timeout per query is 1200.
Import sets indexes and constraints after loading and recomputes statistics.
Experiment uses bexhoma version 0.8.8.
System metrics are monitored by a cluster-wide installation.
Experiment is limited to DBMS ['MySQL'].
Import is handled by 8 processes (pods).
Loading is fixed to cl-worker19.
Benchmarking is fixed to cl-worker19.
SUT is fixed to cl-worker11.
Database is persisted to disk of type shared and size 100Gi.
Loading is tested with [1] threads, split into [8] pods.
Benchmarking is tested with [1] threads, split into [1] pods.
Benchmarking is run as [1, 2] times the number of benchmarking pods.
Experiment is run 2 times.
### Connections
MySQL-BHT-8-1-1-1 uses docker image mysql:8.4.0
RAM:541008568320
Cores:64
host:5.15.0-140-generic
node:cl-worker11
disk:393510268
datadisk:8289
volume_size:100G
volume_used:8.1G
requests_cpu:4
requests_memory:16Gi
eval_parameters
code:1750148746
MySQL-BHT-8-1-2-1 uses docker image mysql:8.4.0
RAM:541008568320
Cores:64
host:5.15.0-140-generic
node:cl-worker11
disk:393510692
datadisk:8289
volume_size:100G
volume_used:8.1G
requests_cpu:4
requests_memory:16Gi
eval_parameters
code:1750148746
MySQL-BHT-8-1-2-2 uses docker image mysql:8.4.0
RAM:541008568320
Cores:64
host:5.15.0-140-generic
node:cl-worker11
disk:393510692
datadisk:8289
volume_size:100G
volume_used:8.1G
requests_cpu:4
requests_memory:16Gi
eval_parameters
code:1750148746
MySQL-BHT-8-2-1-1 uses docker image mysql:8.4.0
RAM:541008568320
Cores:64
host:5.15.0-140-generic
node:cl-worker11
disk:393512176
datadisk:8289
volume_size:100G
volume_used:8.1G
requests_cpu:4
requests_memory:16Gi
eval_parameters
code:1750148746
MySQL-BHT-8-2-2-1 uses docker image mysql:8.4.0
RAM:541008568320
Cores:64
host:5.15.0-140-generic
node:cl-worker11
disk:393220484
datadisk:8289
volume_size:100G
volume_used:8.1G
requests_cpu:4
requests_memory:16Gi
eval_parameters
code:1750148746
MySQL-BHT-8-2-2-2 uses docker image mysql:8.4.0
RAM:541008568320
Cores:64
host:5.15.0-140-generic
node:cl-worker11
disk:393220484
datadisk:8289
volume_size:100G
volume_used:8.1G
requests_cpu:4
requests_memory:16Gi
eval_parameters
code:1750148746
### Errors (failed queries)
No errors
### Warnings (result mismatch)
MySQL-BHT-8-1-1-1 MySQL-BHT-8-1-2-1 MySQL-BHT-8-1-2-2 MySQL-BHT-8-2-1-1 MySQL-BHT-8-2-2-1 MySQL-BHT-8-2-2-2
TPC-DS Q39a+b True True True True False True
### Latency of Timer Execution [ms]
DBMS MySQL-BHT-8-1-1-1 MySQL-BHT-8-1-2-1 MySQL-BHT-8-1-2-2 MySQL-BHT-8-2-1-1 MySQL-BHT-8-2-2-1 MySQL-BHT-8-2-2-2
TPC-DS Q1 93.99 80.91 87.56 229.54 91.51 82.45
TPC-DS Q2 9.65 8.10 7.65 18.15 7.36 7.91
TPC-DS Q3 2.49 2.68 3.18 14.24 2.90 2.79
TPC-DS Q4 18.67 21.01 18.71 18.88 20.74 21.40
TPC-DS Q5 10.73 11.70 11.91 13.84 15.50 11.75
TPC-DS Q6 2.09 2.88 3.41 21.14 2.73 2.87
TPC-DS Q7 2.17 3.76 3.30 28.42 2.83 2.91
TPC-DS Q8 3.91 5.85 5.75 7.36 4.98 5.28
TPC-DS Q9 2.88 3.92 4.34 12.08 3.63 3.89
TPC-DS Q10 3.56 5.69 5.96 7.50 5.58 6.53
TPC-DS Q11 8.66 10.95 10.72 10.28 10.66 11.25
TPC-DS Q12 2.60 3.63 3.35 2.71 3.02 3.15
TPC-DS Q13 2.74 3.91 3.55 19.57 3.78 3.87
TPC-DS Q14a+b 21.01 17.50 23.73 30.14 21.51 23.72
TPC-DS Q15 2.32 2.82 2.87 2.35 2.54 2.84
TPC-DS Q16 2.69 3.32 3.53 3.98 2.71 2.71
TPC-DS Q17 2.96 3.86 3.95 3.36 3.48 4.39
TPC-DS Q18 2.69 3.66 3.74 3.01 3.13 4.14
TPC-DS Q19 2.49 2.78 2.78 2.21 2.27 3.24
TPC-DS Q20 2.17 3.03 2.70 2.02 2.20 2.48
TPC-DS Q21 2.71 3.39 3.43 11.40 2.16 2.43
TPC-DS Q22 2.32 2.87 2.35 1.90 1.76 1.97
TPC-DS Q23a+b 12.75 13.30 12.55 10.45 10.11 12.37
TPC-DS Q24a+b 8.26 9.59 8.68 7.60 6.97 7.95
TPC-DS Q25 3.35 3.39 2.97 2.46 2.30 2.57
TPC-DS Q26 2.36 2.37 2.50 1.94 1.94 2.29
TPC-DS Q27 3.02 2.91 3.44 2.17 1.94 2.61
TPC-DS Q28 3.69 3.72 4.31 3.21 3.05 3.55
TPC-DS Q29 4.49 3.01 3.06 2.57 2.25 2.48
TPC-DS Q30 4.68 4.73 5.06 3.58 3.26 3.06
TPC-DS Q31 7.68 7.21 8.02 7.31 6.03 5.80
TPC-DS Q32 2.24 2.18 2.88 1.98 1.49 1.81
TPC-DS Q33 6.22 5.85 6.61 7.57 4.26 4.28
TPC-DS Q34 3.50 2.73 2.92 2.50 2.00 2.35
TPC-DS Q35 4.29 3.90 4.18 3.36 2.81 3.18
TPC-DS Q36 3.22 2.80 3.12 2.21 2.00 2.20
TPC-DS Q37 2.58 2.48 2.70 1.73 1.71 1.64
TPC-DS Q38 3.14 2.88 2.77 1.91 2.77 2.79
TPC-DS Q39a+b 9.48 6.11 9.11 7.42 6.66 7.38
TPC-DS Q40 2.86 2.59 2.96 2.14 2.21 2.58
TPC-DS Q41 2.80 2.60 3.02 2.25 2.38 2.49
TPC-DS Q42 2.65 2.05 2.26 2.31 1.73 2.10
TPC-DS Q43 2.97 2.31 2.89 1.84 2.82 2.30
TPC-DS Q44 3.03 2.84 2.82 3.55 2.86 2.62
TPC-DS Q45 2.81 3.06 2.45 2.94 2.17 1.95
TPC-DS Q46 2.75 3.32 3.11 2.75 2.20 2.15
TPC-DS Q47 6.13 6.32 6.53 7.54 5.35 4.99
TPC-DS Q48 2.64 2.81 2.46 2.91 2.04 2.02
TPC-DS Q49 8.06 4.90 4.77 4.95 3.74 5.79
TPC-DS Q50 3.09 2.88 3.29 3.19 2.09 2.63
TPC-DS Q51 4.56 4.02 4.34 5.11 3.63 3.92
TPC-DS Q52 2.54 1.98 2.29 2.47 1.65 1.70
TPC-DS Q53 2.92 3.05 2.66 2.66 2.06 1.95
TPC-DS Q54 4.58 3.68 4.05 3.82 3.17 3.28
TPC-DS Q55 2.52 1.79 2.55 1.84 1.48 1.89
TPC-DS Q56 5.00 4.45 4.72 4.69 3.85 3.99
TPC-DS Q57 5.75 5.44 5.32 5.77 4.53 4.31
TPC-DS Q58 5.38 4.91 4.90 8.15 3.75 3.73
TPC-DS Q59 4.86 4.35 7.21 4.27 3.46 3.40
TPC-DS Q60 4.62 4.87 4.33 4.00 3.50 3.91
TPC-DS Q61 3.11 3.02 2.80 2.86 2.79 2.57
TPC-DS Q62 3.37 3.31 3.70 8.59 2.56 2.48
TPC-DS Q63 3.00 2.66 3.31 2.16 2.07 1.95
TPC-DS Q64 7.05 7.23 7.41 8.12 5.85 5.78
TPC-DS Q65 2.74 2.89 2.83 2.22 1.90 2.70
TPC-DS Q66 5.81 5.82 5.90 22.32 10.37 5.56
TPC-DS Q67 2.41 2.70 2.82 2.27 2.20 2.37
TPC-DS Q68 3.09 2.73 2.61 2.44 2.29 2.90
TPC-DS Q69 2.53 3.09 2.90 2.36 2.50 2.52
TPC-DS Q70 3.08 5.55 2.51 2.27 2.19 2.36
TPC-DS Q71 2.85 3.15 3.32 2.57 2.13 2.32
TPC-DS Q72 3.32 2.42 2.74 2.37 2.20 2.77
TPC-DS Q73 2.38 2.88 2.51 2.19 2.05 2.33
TPC-DS Q74 5.54 5.39 4.92 8.53 4.43 4.61
TPC-DS Q75 5.30 5.42 5.17 4.74 4.26 4.38
TPC-DS Q76 2.79 3.07 2.80 3.18 1.99 2.22
TPC-DS Q77 5.83 6.05 5.84 5.90 4.74 4.57
TPC-DS Q78 4.20 4.18 4.06 4.36 3.28 3.46
TPC-DS Q79 2.47 2.28 3.43 2.97 1.85 2.03
TPC-DS Q80 4.35 4.08 5.06 5.22 3.54 3.84
TPC-DS Q81 3.34 3.07 3.32 4.68 2.35 2.49
TPC-DS Q82 2.46 1.96 2.31 2.17 1.59 1.50
TPC-DS Q83 3.82 3.62 3.81 5.89 3.01 2.73
TPC-DS Q84 2.28 2.36 2.45 1.82 1.54 1.64
TPC-DS Q85 3.38 3.52 3.48 3.09 2.37 2.47
TPC-DS Q86 2.66 3.04 2.15 3.28 1.74 1.69
TPC-DS Q87 2.73 2.91 2.75 2.01 1.82 1.92
TPC-DS Q88 6.07 6.06 5.47 5.06 5.53 5.68
TPC-DS Q89 3.12 3.12 2.48 1.86 2.03 2.23
TPC-DS Q90 2.78 2.68 2.51 1.86 1.76 1.89
TPC-DS Q91 2.44 2.93 2.46 4.06 1.95 1.92
TPC-DS Q92 2.25 2.40 2.34 2.17 1.61 1.56
TPC-DS Q93 2.62 2.65 2.54 2.39 1.73 7.51
TPC-DS Q94 2.92 2.62 2.60 2.32 1.78 2.32
TPC-DS Q95 3.23 3.13 3.46 2.70 2.34 2.80
TPC-DS Q96 2.33 2.14 1.99 1.34 1.89 1.78
TPC-DS Q97 3.17 3.42 3.79 3.30 3.00 2.81
TPC-DS Q98 2.63 2.72 2.70 1.56 2.45 2.15
TPC-DS Q99 2.69 2.69 2.80 1.59 2.31 2.18
### Loading [s]
timeGenerate timeIngesting timeSchema timeIndex timeLoad
MySQL-BHT-8-1-1-1 0.0 26.0 21.0 253.0 308.0
MySQL-BHT-8-1-2-1 0.0 26.0 21.0 253.0 308.0
MySQL-BHT-8-1-2-2 0.0 26.0 21.0 253.0 308.0
MySQL-BHT-8-2-1-1 0.0 26.0 21.0 253.0 308.0
MySQL-BHT-8-2-2-1 0.0 26.0 21.0 253.0 308.0
MySQL-BHT-8-2-2-2 0.0 26.0 21.0 253.0 308.0
### Geometric Mean of Medians of Timer Run [s]
Geo Times [s]
DBMS
MySQL-BHT-8-1-1-1 0.0
MySQL-BHT-8-1-2-1 0.0
MySQL-BHT-8-1-2-2 0.0
MySQL-BHT-8-2-1-1 0.0
MySQL-BHT-8-2-2-1 0.0
MySQL-BHT-8-2-2-2 0.0
### Power@Size ((3600*SF)/(geo times))
Power@Size [~Q/h]
DBMS
MySQL-BHT-8-1-1-1 9592942.85
MySQL-BHT-8-1-2-1 9429060.52
MySQL-BHT-8-1-2-2 9236386.35
MySQL-BHT-8-2-1-1 8773386.10
MySQL-BHT-8-2-2-1 11741502.08
MySQL-BHT-8-2-2-2 10964590.31
### Throughput@Size ((queries*streams*3600*SF)/(span of time))
time [s] count SF Throughput@Size
DBMS SF num_experiment num_client
MySQL-BHT-8-1-1 10.0 1 1 7 1 10.0 509142.86
MySQL-BHT-8-1-2 10.0 1 2 17 2 10.0 419294.12
MySQL-BHT-8-2-1 10.0 2 1 4 1 10.0 891000.00
MySQL-BHT-8-2-2 10.0 2 2 4 2 10.0 1782000.00
### Workflow
#### Actual
DBMS MySQL-BHT-8 - Pods [[1, 2], [1, 2]]
#### Planned
DBMS MySQL-BHT-8 - Pods [[1, 2], [1, 2]]
### Ingestion - SUT
CPU [CPUs] Max CPU Max RAM [Gb] Max RAM Cached [Gb]
MySQL-BHT-8-1-1 188.77 1.11 37.69 45.73
MySQL-BHT-8-1-2 188.77 1.11 37.69 45.73
### Ingestion - Loader
CPU [CPUs] Max CPU Max RAM [Gb] Max RAM Cached [Gb]
MySQL-BHT-8-1-1 0 0 0.0 0.0
MySQL-BHT-8-1-2 0 0 0.0 0.0
### Execution - SUT
CPU [CPUs] Max CPU Max RAM [Gb] Max RAM Cached [Gb]
MySQL-BHT-8-1-1 0.00 0.00 37.69 45.73
MySQL-BHT-8-1-2 1.71 0.00 37.70 45.73
MySQL-BHT-8-2-1 475.11 0.00 75.15 91.12
MySQL-BHT-8-2-2 0.00 0.02 37.47 45.41
### Execution - Benchmarker
CPU [CPUs] Max CPU Max RAM [Gb] Max RAM Cached [Gb]
MySQL-BHT-8-1-1 0.00 0.0 0.0 0.0
MySQL-BHT-8-1-2 0.00 0.0 0.0 0.0
MySQL-BHT-8-2-1 0.00 0.0 0.0 0.0
MySQL-BHT-8-2-2 0.03 0.0 0.0 0.0
### Tests
TEST passed: Geo Times [s] contains no 0 or NaN
TEST passed: Power@Size [~Q/h] contains no 0 or NaN
TEST passed: Throughput@Size contains no 0 or NaN
TEST passed: No SQL errors
TEST failed: SQL warnings (result mismatch)
TEST passed: Workflow as planned
TEST passed: Ingestion SUT contains no 0 or NaN in CPU [CPUs]
TEST failed: Ingestion Loader contains 0 or NaN in CPU [CPUs]
TEST failed: Execution SUT contains 0 or NaN in CPU [CPUs]
TEST failed: Execution Benchmarker contains 0 or NaN in CPU [CPUs]
MariaDB
TPC-DS Simple
bexhoma tpcds \
-dbms MariaDB \
-sf 1 \
-nlp 8 \
-nlt 8 \
-xii -xic -xis \
-xdt \
-nc 1 \
-ne 1 \
-nbp 1 \
-ms $BEXHOMA_MS \
-t 1200 \
-tr \
-lr 128Gi \
-rr 128Gi \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_tpcds_mariadb_1.log
yields (after ca. 10 minutes) something like
testcase_tpcds_mariadb_1.log
## Show Summary
### Workload
TPC-DS Queries SF=1
* Type: tpcds
* Duration: 4037s
* Code: 1782833902
* This includes the reading queries of TPC-DS.
* This experiment compares run time and resource consumption of TPC-DS queries in different DBMS.
* TPC-DS (SF=1) data is loaded and benchmark is executed.
* Query ordering is Q1 - Q99.
* All instances use the same query parameters.
* Timeout per query is 1200.
* Import sets indexes and constraints after loading and recomputes statistics.
* Experiment uses bexhoma version 0.10.2.
* Experiment is limited to DBMS ['MariaDB'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker38.
* Database uses ephemeral storage of size 50Gi.
* Loading is tested with [8] threads, split into [8] pods.
* Benchmarking is tested with [1] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* MariaDB-1-1-1-1-1 uses docker image mariadb:11.4.7
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:268035
* datadisk:4543
* cpu_list:0-127
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:128Gi
* limits_memory:128Gi
* eval_parameters
* code:1782833902
### SUT Container Restarts
* bexhoma-sut-mariadb-1-1782833902-fdb6f66d8-5zpn7: 0
### Workflow
#### Actual
* DBMS MariaDB-1 - Experiment 1 Client 1: tpcds (1 pods)
#### Planned
* DBMS MariaDB-1 - Experiment 1 Client 1: tpcds (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:------------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| MariaDB-1-1 | 1 | 1 | 585.00 | 0.00 | 0.00 | 66.00 | 512.00 | 8 | 0 | | None | 0 | False | 6.15 |
### Execution
#### Per Connection
| | configuration | phase | job | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id | pod |
|:------------------|:----------------|:--------------|:----------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|:------------------|
| MariaDB-1-1-1-1-1 | MariaDB-1 | MariaDB-1-1-1 | MariaDB-1-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 97 | 3204 | 0.99 | 3680.65 | 108.99 | -1 | MariaDB-1-1-1-1-1 |
#### Per Phase
| | phase | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id |
|:--------------|:--------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|
| MariaDB-1-1-1 | MariaDB-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 97 | 3204 | 0.99 | 3680.65 | 108.99 | -1 |
### Latency of Timer Execution [ms]
| Queries | MariaDB-1-1-1-1-1 |
|:--------------|--------------------:|
| TPC-DS Q1 | 21.54 |
| TPC-DS Q2 | 4688.76 |
| TPC-DS Q3 | 19.31 |
| TPC-DS Q4 | 12070.81 |
| TPC-DS Q5 | 8657.12 |
| TPC-DS Q6 | 741.86 |
| TPC-DS Q7 | 4017.22 |
| TPC-DS Q8 | 381.48 |
| TPC-DS Q9 | 3799.48 |
| TPC-DS Q10 | 60.42 |
| TPC-DS Q11 | 8152.01 |
| TPC-DS Q12 | 258.03 |
| TPC-DS Q13 | 1067.62 |
| TPC-DS Q14a+b | 49638.93 |
| TPC-DS Q15 | 175.56 |
| TPC-DS Q16 | 11121.39 |
| TPC-DS Q17 | 707.26 |
| TPC-DS Q18 | 1952.32 |
| TPC-DS Q19 | 309.52 |
| TPC-DS Q20 | 433.49 |
| TPC-DS Q21 | 21572.00 |
| TPC-DS Q22 | 19869.36 |
| TPC-DS Q23a+b | 57465.28 |
| TPC-DS Q24a+b | 32.58 |
| TPC-DS Q25 | 153.86 |
| TPC-DS Q26 | 868.66 |
| TPC-DS Q27 | 1665.35 |
| TPC-DS Q28 | 2837.08 |
| TPC-DS Q29 | 94.89 |
| TPC-DS Q30 | 106.37 |
| TPC-DS Q31 | 1408.80 |
| TPC-DS Q32 | 12.06 |
| TPC-DS Q33 | 158.76 |
| TPC-DS Q34 | 3045.86 |
| TPC-DS Q35 | 1288.52 |
| TPC-DS Q36 | 1771.70 |
| TPC-DS Q37 | 3663.48 |
| TPC-DS Q38 | 7218.75 |
| TPC-DS Q39a+b | 1314.93 |
| TPC-DS Q40 | 179.28 |
| TPC-DS Q41 | 408.73 |
| TPC-DS Q42 | 281.63 |
| TPC-DS Q43 | 1109.11 |
| TPC-DS Q44 | 1198.02 |
| TPC-DS Q45 | 134.35 |
| TPC-DS Q46 | 3230.52 |
| TPC-DS Q47 | 15022.31 |
| TPC-DS Q48 | 1248.50 |
| TPC-DS Q49 | 109.78 |
| TPC-DS Q50 | 35.45 |
| TPC-DS Q51 | 7916.88 |
| TPC-DS Q52 | 314.14 |
| TPC-DS Q53 | 153.86 |
| TPC-DS Q54 | 1086.61 |
| TPC-DS Q55 | 247.21 |
| TPC-DS Q56 | 160.67 |
| TPC-DS Q57 | 6159.56 |
| TPC-DS Q58 | 5881.91 |
| TPC-DS Q59 | 9180.81 |
| TPC-DS Q60 | 663.17 |
| TPC-DS Q61 | 402.53 |
| TPC-DS Q62 | 1681.06 |
| TPC-DS Q63 | 136.88 |
| TPC-DS Q64 | 548.86 |
| TPC-DS Q65 | 5857.24 |
| TPC-DS Q66 | 1228.97 |
| TPC-DS Q67 | 7050.47 |
| TPC-DS Q68 | 3057.05 |
| TPC-DS Q69 | 3.60 |
| TPC-DS Q70 | 8193.80 |
| TPC-DS Q71 | 529.04 |
| TPC-DS Q72 | 403601.33 |
| TPC-DS Q73 | 3509.42 |
| TPC-DS Q74 | 6182.05 |
| TPC-DS Q75 | 5714.65 |
| TPC-DS Q76 | 465.15 |
| TPC-DS Q77 | 6261.85 |
| TPC-DS Q78 | 5726.20 |
| TPC-DS Q79 | 3437.41 |
| TPC-DS Q80 | 548.97 |
| TPC-DS Q81 | 220.27 |
| TPC-DS Q82 | 3659.84 |
| TPC-DS Q83 | 935.74 |
| TPC-DS Q84 | 59.40 |
| TPC-DS Q85 | 146.99 |
| TPC-DS Q86 | 919.30 |
| TPC-DS Q87 | 7199.08 |
| TPC-DS Q88 | 16154.30 |
| TPC-DS Q89 | 1615.79 |
| TPC-DS Q90 | 119.56 |
| TPC-DS Q91 | 24.38 |
| TPC-DS Q92 | 9.40 |
| TPC-DS Q93 | 45.70 |
| TPC-DS Q96 | 807.61 |
| TPC-DS Q97 | 5338.46 |
| TPC-DS Q98 | 875.80 |
| TPC-DS Q99 | 4938.86 |
### Errors (failed queries)
| | TPC-DS Q1 | TPC-DS Q2 | TPC-DS Q3 | TPC-DS Q4 | TPC-DS Q5 | TPC-DS Q6 | TPC-DS Q7 | TPC-DS Q8 | TPC-DS Q9 | TPC-DS Q10 | TPC-DS Q11 | TPC-DS Q12 | TPC-DS Q13 | TPC-DS Q14a+b | TPC-DS Q15 | TPC-DS Q16 | TPC-DS Q17 | TPC-DS Q18 | TPC-DS Q19 | TPC-DS Q20 | TPC-DS Q21 | TPC-DS Q22 | TPC-DS Q23a+b | TPC-DS Q24a+b | TPC-DS Q25 | TPC-DS Q26 | TPC-DS Q27 | TPC-DS Q28 | TPC-DS Q29 | TPC-DS Q30 | TPC-DS Q31 | TPC-DS Q32 | TPC-DS Q33 | TPC-DS Q34 | TPC-DS Q35 | TPC-DS Q36 | TPC-DS Q37 | TPC-DS Q38 | TPC-DS Q39a+b | TPC-DS Q40 | TPC-DS Q41 | TPC-DS Q42 | TPC-DS Q43 | TPC-DS Q44 | TPC-DS Q45 | TPC-DS Q46 | TPC-DS Q47 | TPC-DS Q48 | TPC-DS Q49 | TPC-DS Q50 | TPC-DS Q51 | TPC-DS Q52 | TPC-DS Q53 | TPC-DS Q54 | TPC-DS Q55 | TPC-DS Q56 | TPC-DS Q57 | TPC-DS Q58 | TPC-DS Q59 | TPC-DS Q60 | TPC-DS Q61 | TPC-DS Q62 | TPC-DS Q63 | TPC-DS Q64 | TPC-DS Q65 | TPC-DS Q66 | TPC-DS Q67 | TPC-DS Q68 | TPC-DS Q69 | TPC-DS Q70 | TPC-DS Q71 | TPC-DS Q72 | TPC-DS Q73 | TPC-DS Q74 | TPC-DS Q75 | TPC-DS Q76 | TPC-DS Q77 | TPC-DS Q78 | TPC-DS Q79 | TPC-DS Q80 | TPC-DS Q81 | TPC-DS Q82 | TPC-DS Q83 | TPC-DS Q84 | TPC-DS Q85 | TPC-DS Q86 | TPC-DS Q87 | TPC-DS Q88 | TPC-DS Q89 | TPC-DS Q90 | TPC-DS Q91 | TPC-DS Q92 | TPC-DS Q93 | TPC-DS Q94 | TPC-DS Q95 | TPC-DS Q96 | TPC-DS Q97 | TPC-DS Q98 | TPC-DS Q99 |
|:------------------|------------:|------------:|------------:|------------:|------------:|------------:|------------:|------------:|------------:|-------------:|-------------:|-------------:|-------------:|----------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|----------------:|----------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|----------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|
| MariaDB-1-1-1-1-1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 |
* TPC-DS Q94
* MariaDB-1-1-1-1-1: numRun 1: : java.sql.SQLTimeoutException: (conn=59) Query execution was interrupted (max_statement_time exceeded)
* TPC-DS Q95
* MariaDB-1-1-1-1-1: numRun 1: : java.sql.SQLTimeoutException: (conn=59) Query execution was interrupted (max_statement_time exceeded)
### Warnings (result mismatch)
No warnings
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Geo Times [s] contains no 0 or NaN
* TEST passed: Power@Size [~Q/h] contains no 0 or NaN
* TEST passed: Throughput@Size contains no 0 or NaN
* TEST failed: SQL errors
* TEST passed: No SQL warnings
* TEST passed: Workflow as planned
TPC-DS Monitoring
bexhoma tpcds \
-dbms MariaDB \
-sf 1 \
-nlp 8 \
-nlt 8 \
-xii -xic -xis \
-xdt \
-nc 1 \
-ne 1 \
-nbp 1 \
-m \
-mc \
-ms $BEXHOMA_MS \
-t 1200 \
-tr \
-lr 128Gi \
-rr 128Gi \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_tpcds_mariadb_2.log
yields (after ca. 15 minutes) something like
testcase_tpcds_mariadb_2.log
## Show Summary
### Workload
TPC-DS Queries SF=1
* Type: tpcds
* Duration: 4175s
* Code: 1782408033
* This includes the reading queries of TPC-DS.
* This experiment compares run time and resource consumption of TPC-DS queries in different DBMS.
* TPC-DS (SF=1) data is loaded and benchmark is executed.
* Query ordering is Q1 - Q99.
* All instances use the same query parameters.
* Timeout per query is 1200.
* Import sets indexes and constraints after loading and recomputes statistics.
* Experiment uses bexhoma version 0.10.0.
* System metrics are monitored by a cluster-wide installation.
* Experiment is limited to DBMS ['MariaDB'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker38.
* Loading is tested with [8] threads, split into [8] pods.
* Benchmarking is tested with [1] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* MariaDB-1-1-1-1-1 uses docker image mariadb:11.4.7
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:225003
* cpu_list:0-127
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782408033
### Workflow
#### Actual
* DBMS MariaDB-1 - Experiment 1 Client 1: tpcds (1 pods)
#### Planned
* DBMS MariaDB-1 - Experiment 1 Client 1: tpcds (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:------------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| MariaDB-1-1 | 1 | 1 | 659.00 | 2.00 | 1.00 | 63.00 | 586.00 | 8 | 0 | | None | 0 | False | 5.46 |
### Execution
#### Per Connection
| | configuration | phase | job | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id | pod |
|:------------------|:----------------|:--------------|:----------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|:------------------|
| MariaDB-1-1-1-1-1 | MariaDB-1 | MariaDB-1-1-1 | MariaDB-1-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 97 | 3222 | 1.00 | 3615.10 | 108.38 | -1 | MariaDB-1-1-1-1-1 |
#### Per Phase
| | phase | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id |
|:--------------|:--------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|
| MariaDB-1-1-1 | MariaDB-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 97 | 3222 | 1.00 | 3615.10 | 108.38 | -1 |
### Latency of Timer Execution [ms]
| Queries | MariaDB-1-1-1-1-1 |
|:--------------|--------------------:|
| TPC-DS Q1 | 20.58 |
| TPC-DS Q2 | 5155.74 |
| TPC-DS Q3 | 17.24 |
| TPC-DS Q4 | 12760.85 |
| TPC-DS Q5 | 8530.52 |
| TPC-DS Q6 | 788.63 |
| TPC-DS Q7 | 3533.69 |
| TPC-DS Q8 | 364.70 |
| TPC-DS Q9 | 3826.11 |
| TPC-DS Q10 | 47.09 |
| TPC-DS Q11 | 8029.01 |
| TPC-DS Q12 | 249.82 |
| TPC-DS Q13 | 1138.95 |
| TPC-DS Q14a+b | 49164.50 |
| TPC-DS Q15 | 173.35 |
| TPC-DS Q16 | 11244.66 |
| TPC-DS Q17 | 803.62 |
| TPC-DS Q18 | 1934.76 |
| TPC-DS Q19 | 255.67 |
| TPC-DS Q20 | 437.01 |
| TPC-DS Q21 | 21254.04 |
| TPC-DS Q22 | 19635.03 |
| TPC-DS Q23a+b | 58108.26 |
| TPC-DS Q24a+b | 2072.30 |
| TPC-DS Q25 | 158.64 |
| TPC-DS Q26 | 898.32 |
| TPC-DS Q27 | 1680.10 |
| TPC-DS Q28 | 2833.70 |
| TPC-DS Q29 | 97.75 |
| TPC-DS Q30 | 103.69 |
| TPC-DS Q31 | 1445.23 |
| TPC-DS Q32 | 10.71 |
| TPC-DS Q33 | 204.18 |
| TPC-DS Q34 | 3192.23 |
| TPC-DS Q35 | 1189.62 |
| TPC-DS Q36 | 1688.02 |
| TPC-DS Q37 | 3737.26 |
| TPC-DS Q38 | 7163.57 |
| TPC-DS Q39a+b | 1324.60 |
| TPC-DS Q40 | 186.38 |
| TPC-DS Q41 | 413.10 |
| TPC-DS Q42 | 239.73 |
| TPC-DS Q43 | 1148.65 |
| TPC-DS Q44 | 2.52 |
| TPC-DS Q45 | 128.51 |
| TPC-DS Q46 | 3546.82 |
| TPC-DS Q47 | 13981.45 |
| TPC-DS Q48 | 1259.37 |
| TPC-DS Q49 | 114.33 |
| TPC-DS Q50 | 30.74 |
| TPC-DS Q51 | 7630.46 |
| TPC-DS Q52 | 224.42 |
| TPC-DS Q53 | 161.65 |
| TPC-DS Q54 | 1039.59 |
| TPC-DS Q55 | 218.12 |
| TPC-DS Q56 | 133.75 |
| TPC-DS Q57 | 6727.82 |
| TPC-DS Q58 | 5732.05 |
| TPC-DS Q59 | 9403.49 |
| TPC-DS Q60 | 663.03 |
| TPC-DS Q61 | 370.34 |
| TPC-DS Q62 | 1684.59 |
| TPC-DS Q63 | 152.48 |
| TPC-DS Q64 | 566.63 |
| TPC-DS Q65 | 5794.88 |
| TPC-DS Q66 | 1281.28 |
| TPC-DS Q67 | 7257.30 |
| TPC-DS Q68 | 3197.45 |
| TPC-DS Q69 | 3.45 |
| TPC-DS Q70 | 8595.11 |
| TPC-DS Q71 | 522.66 |
| TPC-DS Q72 | 406943.14 |
| TPC-DS Q73 | 2959.08 |
| TPC-DS Q74 | 6295.58 |
| TPC-DS Q75 | 5770.01 |
| TPC-DS Q76 | 845.91 |
| TPC-DS Q77 | 6337.57 |
| TPC-DS Q78 | 5962.35 |
| TPC-DS Q79 | 3277.33 |
| TPC-DS Q80 | 842.18 |
| TPC-DS Q81 | 316.31 |
| TPC-DS Q82 | 4404.48 |
| TPC-DS Q83 | 948.80 |
| TPC-DS Q84 | 59.16 |
| TPC-DS Q85 | 148.73 |
| TPC-DS Q86 | 980.04 |
| TPC-DS Q87 | 7542.86 |
| TPC-DS Q88 | 25828.87 |
| TPC-DS Q89 | 1677.75 |
| TPC-DS Q90 | 395.91 |
| TPC-DS Q91 | 28.21 |
| TPC-DS Q92 | 10.74 |
| TPC-DS Q93 | 44.28 |
| TPC-DS Q96 | 2678.46 |
| TPC-DS Q97 | 5174.98 |
| TPC-DS Q98 | 857.55 |
| TPC-DS Q99 | 4862.09 |
### Errors (failed queries)
| | TPC-DS Q1 | TPC-DS Q2 | TPC-DS Q3 | TPC-DS Q4 | TPC-DS Q5 | TPC-DS Q6 | TPC-DS Q7 | TPC-DS Q8 | TPC-DS Q9 | TPC-DS Q10 | TPC-DS Q11 | TPC-DS Q12 | TPC-DS Q13 | TPC-DS Q14a+b | TPC-DS Q15 | TPC-DS Q16 | TPC-DS Q17 | TPC-DS Q18 | TPC-DS Q19 | TPC-DS Q20 | TPC-DS Q21 | TPC-DS Q22 | TPC-DS Q23a+b | TPC-DS Q24a+b | TPC-DS Q25 | TPC-DS Q26 | TPC-DS Q27 | TPC-DS Q28 | TPC-DS Q29 | TPC-DS Q30 | TPC-DS Q31 | TPC-DS Q32 | TPC-DS Q33 | TPC-DS Q34 | TPC-DS Q35 | TPC-DS Q36 | TPC-DS Q37 | TPC-DS Q38 | TPC-DS Q39a+b | TPC-DS Q40 | TPC-DS Q41 | TPC-DS Q42 | TPC-DS Q43 | TPC-DS Q44 | TPC-DS Q45 | TPC-DS Q46 | TPC-DS Q47 | TPC-DS Q48 | TPC-DS Q49 | TPC-DS Q50 | TPC-DS Q51 | TPC-DS Q52 | TPC-DS Q53 | TPC-DS Q54 | TPC-DS Q55 | TPC-DS Q56 | TPC-DS Q57 | TPC-DS Q58 | TPC-DS Q59 | TPC-DS Q60 | TPC-DS Q61 | TPC-DS Q62 | TPC-DS Q63 | TPC-DS Q64 | TPC-DS Q65 | TPC-DS Q66 | TPC-DS Q67 | TPC-DS Q68 | TPC-DS Q69 | TPC-DS Q70 | TPC-DS Q71 | TPC-DS Q72 | TPC-DS Q73 | TPC-DS Q74 | TPC-DS Q75 | TPC-DS Q76 | TPC-DS Q77 | TPC-DS Q78 | TPC-DS Q79 | TPC-DS Q80 | TPC-DS Q81 | TPC-DS Q82 | TPC-DS Q83 | TPC-DS Q84 | TPC-DS Q85 | TPC-DS Q86 | TPC-DS Q87 | TPC-DS Q88 | TPC-DS Q89 | TPC-DS Q90 | TPC-DS Q91 | TPC-DS Q92 | TPC-DS Q93 | TPC-DS Q94 | TPC-DS Q95 | TPC-DS Q96 | TPC-DS Q97 | TPC-DS Q98 | TPC-DS Q99 |
|:------------------|------------:|------------:|------------:|------------:|------------:|------------:|------------:|------------:|------------:|-------------:|-------------:|-------------:|-------------:|----------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|----------------:|----------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|----------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|
| MariaDB-1-1-1-1-1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 |
* TPC-DS Q94
* MariaDB-1-1-1-1-1: numRun 1: : java.sql.SQLTimeoutException: (conn=61) Query execution was interrupted (max_statement_time exceeded)
* TPC-DS Q95
* MariaDB-1-1-1-1-1: numRun 1: : java.sql.SQLTimeoutException: (conn=61) Query execution was interrupted (max_statement_time exceeded)
### Warnings (result mismatch)
No warnings
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1-1 | 752.71 | 4.40 | 8.68 | 8.79 |
### Loading phase: component data generator
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1-1 | 0.00 | 0.00 | 0.00 | 0.00 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1-1 | 2.14 | 0.04 | 0.01 | 0.76 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1-1 | 3193.56 | 1.00 | 8.93 | 9.04 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1-1 | 20.80 | 0.02 | 0.30 | 0.31 |
### Tests
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST skipped: Loading phase: component data generator contains 0 or NaN in CPU [CPUs] (data pre-existing)
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Geo Times [s] contains no 0 or NaN
* TEST passed: Power@Size [~Q/h] contains no 0 or NaN
* TEST passed: Throughput@Size contains no 0 or NaN
* TEST failed: SQL errors
* TEST passed: No SQL warnings
* TEST passed: Workflow as planned
TPC-DS Throughput Test
bexhoma tpcds \
-dbms MariaDB \
-sf 1 \
-nlp 8 \
-nlt 8 \
-xii -xic -xis \
-xdt \
-nc 2 \
-ne 1,2 \
-nbp 1 \
-m \
-mc \
-ms $BEXHOMA_MS \
-t 1200 \
-tr \
-lr 128Gi \
-rr 128Gi \
-rsr \
-rst $BEXHOMA_STORAGE_CLASS \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_tpcds_mariadb_3.log
yields (after ca. 15 minutes) something like
testcase_tpcds_mariadb_3.log
## Show Summary
### Workload
TPC-DS Queries SF=1
* Type: tpcds
* Duration: 16025s
* Code: 1782408084
* This includes the reading queries of TPC-DS.
* This experiment compares run time and resource consumption of TPC-DS queries in different DBMS.
* TPC-DS (SF=1) data is loaded and benchmark is executed.
* Query ordering is Q1 - Q99.
* All instances use the same query parameters.
* Timeout per query is 1200.
* Import sets indexes and constraints after loading and recomputes statistics.
* Experiment uses bexhoma version 0.10.0.
* System metrics are monitored by a cluster-wide installation.
* Experiment is limited to DBMS ['MariaDB'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker38.
* Database is persisted to disk of type shared and size 10Gi.
* Loading is tested with [8] threads, split into [8] pods.
* Benchmarking is tested with [1] threads, split into [1] pods.
* Benchmarking is run as [1, 2] times the number of benchmarking pods.
* Experiment is run 2 times.
### Connections
* MariaDB-1-1-1-1-1 uses docker image mariadb:11.4.7
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:225003
* volume_size:10G
* volume_used:4.5G
* cpu_list:0-127
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782408084
* MariaDB-1-1-2-1-1 uses docker image mariadb:11.4.7
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220477
* volume_size:10G
* volume_used:4.5G
* cpu_list:0-127
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782408084
* MariaDB-1-1-2-1-2 uses docker image mariadb:11.4.7
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220477
* volume_size:10G
* volume_used:4.5G
* cpu_list:0-127
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782408084
* MariaDB-1-2-1-1-1 uses docker image mariadb:11.4.7
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220478
* volume_size:10G
* volume_used:4.5G
* cpu_list:0-127
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782408084
* MariaDB-1-2-2-1-1 uses docker image mariadb:11.4.7
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220478
* volume_size:10G
* volume_used:4.5G
* cpu_list:0-127
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782408084
* MariaDB-1-2-2-1-2 uses docker image mariadb:11.4.7
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220478
* volume_size:10G
* volume_used:4.5G
* cpu_list:0-127
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782408084
### Workflow
#### Actual
* DBMS MariaDB-1 - Experiment 1 Client 1: tpcds (1 pods)
* DBMS MariaDB-1 - Experiment 1 Client 2: tpcds (2 pods)
* DBMS MariaDB-1 - Experiment 2 Client 1: tpcds (1 pods)
* DBMS MariaDB-1 - Experiment 2 Client 2: tpcds (2 pods)
#### Planned
* DBMS MariaDB-1 - Experiment 1 Client 1: tpcds (1 pods)
* DBMS MariaDB-1 - Experiment 1 Client 2: tpcds (2 pods)
* DBMS MariaDB-1 - Experiment 2 Client 1: tpcds (1 pods)
* DBMS MariaDB-1 - Experiment 2 Client 2: tpcds (2 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:------------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| MariaDB-1-1 | 1 | 1 | 2391.00 | 5.00 | 1.00 | 464.00 | 1913.00 | 8 | 0 | | None | 0 | False | 1.51 |
| MariaDB-1-2 | 2 | 1 | 2391.00 | 5.00 | 1.00 | 464.00 | 1913.00 | 8 | 0 | | None | 0 | False | 1.51 |
### Execution
#### Per Connection
| | configuration | phase | job | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id | pod |
|:------------------|:----------------|:--------------|:----------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|:------------------|
| MariaDB-1-1-1-1-1 | MariaDB-1 | MariaDB-1-1-1 | MariaDB-1-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 97 | 3190 | 1.03 | 3534.75 | 109.47 | -1 | MariaDB-1-1-1-1-1 |
| MariaDB-1-1-2-1-1 | MariaDB-1 | MariaDB-1-1-2 | MariaDB-1-1-2-1 | 1 | 2 | 1 | 1 | 1.00 | 97 | 3269 | 1.10 | 3286.12 | 106.82 | -1 | MariaDB-1-1-2-1-1 |
| MariaDB-1-1-2-1-2 | MariaDB-1 | MariaDB-1-1-2 | MariaDB-1-1-2-1 | 1 | 2 | 1 | 1 | 1.00 | 97 | 3265 | 1.09 | 3332.89 | 106.95 | -1 | MariaDB-1-1-2-1-2 |
| MariaDB-1-2-1-1-1 | MariaDB-1 | MariaDB-1-2-1 | MariaDB-1-2-1-1 | 2 | 1 | 1 | 1 | 1.00 | 97 | 3176 | 1.03 | 3536.56 | 109.95 | -1 | MariaDB-1-2-1-1-1 |
| MariaDB-1-2-2-1-1 | MariaDB-1 | MariaDB-1-2-2 | MariaDB-1-2-2-1 | 2 | 2 | 1 | 1 | 1.00 | 97 | 3309 | 1.16 | 3112.08 | 105.53 | -1 | MariaDB-1-2-2-1-1 |
| MariaDB-1-2-2-1-2 | MariaDB-1 | MariaDB-1-2-2 | MariaDB-1-2-2-1 | 2 | 2 | 1 | 1 | 1.00 | 97 | 3310 | 1.13 | 3204.20 | 105.50 | -1 | MariaDB-1-2-2-1-2 |
#### Per Phase
| | phase | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id |
|:--------------|:--------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|
| MariaDB-1-1-1 | MariaDB-1-1-1 | 1 | 1 | 1 | 1 | 1.00 | 97 | 3190 | 1.03 | 3534.75 | 109.47 | -1 |
| MariaDB-1-1-2 | MariaDB-1-1-2 | 1 | 2 | 1 | 2 | 1.00 | 194 | 3269 | 1.10 | 3309.42 | 213.64 | -1 |
| MariaDB-1-2-1 | MariaDB-1-2-1 | 2 | 1 | 1 | 1 | 1.00 | 97 | 3176 | 1.03 | 3536.56 | 109.95 | -1 |
| MariaDB-1-2-2 | MariaDB-1-2-2 | 2 | 2 | 1 | 2 | 1.00 | 194 | 3310 | 1.15 | 3157.80 | 211.00 | -1 |
### Latency of Timer Execution [ms]
| Queries | MariaDB-1-1-1-1-1 | MariaDB-1-1-2-1-1 | MariaDB-1-1-2-1-2 | MariaDB-1-2-1-1-1 | MariaDB-1-2-2-1-1 | MariaDB-1-2-2-1-2 |
|:--------------|--------------------:|--------------------:|--------------------:|--------------------:|--------------------:|--------------------:|
| TPC-DS Q1 | 21.25 | 21.99 | 20.65 | 156.55 | 19.74 | 19.42 |
| TPC-DS Q2 | 4572.82 | 4580.97 | 4652.58 | 4663.55 | 4672.21 | 4425.77 |
| TPC-DS Q3 | 12.23 | 10.62 | 10.31 | 29.80 | 9.75 | 10.38 |
| TPC-DS Q4 | 11814.81 | 12501.07 | 12437.42 | 12102.38 | 12928.04 | 12800.08 |
| TPC-DS Q5 | 8366.21 | 9842.79 | 9910.66 | 9389.33 | 10867.63 | 10874.60 |
| TPC-DS Q6 | 739.85 | 1265.71 | 849.79 | 748.56 | 1320.35 | 1188.27 |
| TPC-DS Q7 | 3376.76 | 4458.41 | 4393.45 | 3429.95 | 5104.82 | 5209.71 |
| TPC-DS Q8 | 343.92 | 367.82 | 338.62 | 363.22 | 492.11 | 486.03 |
| TPC-DS Q9 | 3799.43 | 3882.50 | 3799.12 | 3794.98 | 3878.79 | 3896.92 |
| TPC-DS Q10 | 35.46 | 54.46 | 29.46 | 37.15 | 57.30 | 30.34 |
| TPC-DS Q11 | 8050.92 | 9397.69 | 9382.76 | 8350.30 | 9387.13 | 9359.84 |
| TPC-DS Q12 | 248.72 | 362.98 | 362.63 | 249.00 | 348.20 | 349.12 |
| TPC-DS Q13 | 1054.74 | 1461.56 | 1461.12 | 1191.68 | 1474.66 | 1474.37 |
| TPC-DS Q14a+b | 48552.25 | 62756.87 | 62759.34 | 48582.69 | 63169.03 | 62933.92 |
| TPC-DS Q15 | 163.26 | 196.61 | 181.96 | 177.04 | 166.73 | 180.11 |
| TPC-DS Q16 | 10847.69 | 14153.72 | 14151.81 | 10820.47 | 12814.62 | 13007.66 |
| TPC-DS Q17 | 728.31 | 1050.29 | 1050.16 | 767.33 | 1008.07 | 940.85 |
| TPC-DS Q18 | 1844.73 | 2495.47 | 2494.17 | 1850.02 | 2570.09 | 2592.19 |
| TPC-DS Q19 | 238.02 | 255.75 | 275.81 | 231.32 | 244.95 | 263.61 |
| TPC-DS Q20 | 458.56 | 591.78 | 601.10 | 442.84 | 568.52 | 571.26 |
| TPC-DS Q21 | 21074.79 | 33681.57 | 32755.89 | 21235.75 | 34368.12 | 34372.27 |
| TPC-DS Q22 | 19298.32 | 23518.30 | 22760.57 | 19338.94 | 24807.46 | 25263.17 |
| TPC-DS Q23a+b | 48300.77 | 60779.03 | 58433.79 | 47628.03 | 60414.22 | 60098.87 |
| TPC-DS Q24a+b | 2055.66 | 2286.62 | 2203.90 | 2019.86 | 2589.10 | 2494.32 |
| TPC-DS Q25 | 141.25 | 148.99 | 158.71 | 149.85 | 174.74 | 151.71 |
| TPC-DS Q26 | 831.80 | 833.36 | 875.27 | 873.59 | 1117.71 | 1120.46 |
| TPC-DS Q27 | 1652.02 | 1693.30 | 2310.39 | 1698.96 | 2212.60 | 2216.17 |
| TPC-DS Q28 | 2802.26 | 2788.12 | 2818.67 | 2791.98 | 2903.12 | 2926.11 |
| TPC-DS Q29 | 88.25 | 93.80 | 96.65 | 88.78 | 126.71 | 127.19 |
| TPC-DS Q30 | 102.43 | 139.73 | 108.13 | 105.74 | 127.20 | 129.53 |
| TPC-DS Q31 | 1396.46 | 1451.24 | 1466.10 | 1386.77 | 1709.62 | 1711.92 |
| TPC-DS Q32 | 10.50 | 10.73 | 9.95 | 9.58 | 10.85 | 10.44 |
| TPC-DS Q33 | 164.08 | 183.65 | 178.22 | 170.11 | 229.13 | 219.65 |
| TPC-DS Q34 | 2912.74 | 3104.45 | 2968.59 | 2874.04 | 3912.62 | 3927.87 |
| TPC-DS Q35 | 1133.96 | 1218.13 | 1243.14 | 1140.08 | 1648.53 | 1643.21 |
| TPC-DS Q36 | 1620.78 | 1799.61 | 1786.51 | 1645.18 | 2414.60 | 3016.97 |
| TPC-DS Q37 | 3538.03 | 3763.53 | 3979.29 | 3637.88 | 4191.12 | 5101.84 |
| TPC-DS Q38 | 7006.01 | 8248.53 | 8141.00 | 7148.33 | 8886.38 | 8664.93 |
| TPC-DS Q39a+b | 1168.31 | 1189.52 | 1202.60 | 1223.29 | 1203.41 | 1191.13 |
| TPC-DS Q40 | 175.99 | 180.18 | 237.27 | 180.80 | 183.92 | 203.87 |
| TPC-DS Q41 | 397.18 | 400.24 | 402.33 | 391.30 | 390.42 | 394.80 |
| TPC-DS Q42 | 223.24 | 233.10 | 246.26 | 216.27 | 232.34 | 263.56 |
| TPC-DS Q43 | 1152.75 | 1409.22 | 1312.66 | 1126.98 | 1276.68 | 1215.07 |
| TPC-DS Q44 | 1165.07 | 1181.96 | 1174.86 | 1154.83 | 1181.82 | 1146.20 |
| TPC-DS Q45 | 217.63 | 143.93 | 140.66 | 135.37 | 168.65 | 138.83 |
| TPC-DS Q46 | 5502.38 | 3800.02 | 4170.88 | 3534.17 | 4205.41 | 4251.95 |
| TPC-DS Q47 | 14345.68 | 13441.87 | 13457.65 | 12648.00 | 13383.53 | 13805.53 |
| TPC-DS Q48 | 1272.92 | 1339.87 | 1379.83 | 1206.61 | 1347.37 | 1324.44 |
| TPC-DS Q49 | 111.87 | 96.78 | 111.16 | 113.47 | 110.14 | 89.87 |
| TPC-DS Q50 | 34.56 | 37.21 | 35.83 | 34.25 | 51.22 | 34.18 |
| TPC-DS Q51 | 7653.39 | 8029.37 | 8192.90 | 7680.66 | 9296.93 | 8714.28 |
| TPC-DS Q52 | 246.52 | 264.49 | 249.46 | 239.00 | 238.53 | 279.04 |
| TPC-DS Q53 | 143.85 | 155.63 | 143.53 | 143.00 | 142.95 | 155.68 |
| TPC-DS Q54 | 1048.88 | 1048.92 | 1034.33 | 1034.47 | 1101.29 | 1060.13 |
| TPC-DS Q55 | 229.28 | 247.48 | 227.60 | 224.44 | 255.68 | 246.92 |
| TPC-DS Q56 | 187.73 | 195.42 | 186.39 | 183.33 | 204.72 | 195.20 |
| TPC-DS Q57 | 6036.70 | 6392.01 | 6499.07 | 6089.04 | 6472.27 | 6442.59 |
| TPC-DS Q58 | 5944.94 | 7312.86 | 6721.07 | 5767.65 | 7401.82 | 7508.00 |
| TPC-DS Q59 | 11047.43 | 10566.60 | 10791.14 | 9049.94 | 10927.35 | 10747.04 |
| TPC-DS Q60 | 832.57 | 622.17 | 609.61 | 586.81 | 605.42 | 626.48 |
| TPC-DS Q61 | 390.04 | 386.09 | 385.62 | 363.84 | 413.20 | 377.21 |
| TPC-DS Q62 | 2319.48 | 2007.23 | 1930.39 | 1674.97 | 1926.94 | 1942.67 |
| TPC-DS Q63 | 262.15 | 207.31 | 164.45 | 162.91 | 200.26 | 171.74 |
| TPC-DS Q64 | 992.38 | 572.70 | 573.98 | 581.89 | 617.66 | 577.34 |
| TPC-DS Q65 | 6354.56 | 7478.49 | 7259.60 | 5843.99 | 7971.14 | 7874.61 |
| TPC-DS Q66 | 1235.07 | 1197.58 | 1205.87 | 1191.99 | 1378.54 | 1305.24 |
| TPC-DS Q67 | 7011.01 | 7170.33 | 7799.37 | 7045.88 | 8046.68 | 7982.74 |
| TPC-DS Q68 | 3061.65 | 3885.38 | 3029.25 | 3057.08 | 4063.73 | 4067.12 |
| TPC-DS Q69 | 3.57 | 2.98 | 2.98 | 3.60 | 3.69 | 2.96 |
| TPC-DS Q70 | 9522.08 | 9712.56 | 10397.34 | 8257.08 | 10838.14 | 10661.19 |
| TPC-DS Q71 | 597.96 | 477.95 | 475.43 | 485.90 | 582.93 | 495.58 |
| TPC-DS Q72 | 395212.23 | 409650.17 | 409844.91 | 391651.39 | 433132.47 | 435601.07 |
| TPC-DS Q73 | 2878.01 | 3006.95 | 3124.16 | 2908.24 | 3292.84 | 3213.67 |
| TPC-DS Q74 | 6129.58 | 6673.63 | 6618.97 | 5944.10 | 6730.25 | 6691.07 |
| TPC-DS Q75 | 5740.42 | 6206.04 | 6408.75 | 5536.32 | 6355.60 | 6155.40 |
| TPC-DS Q76 | 477.30 | 483.95 | 464.51 | 436.51 | 476.69 | 482.97 |
| TPC-DS Q77 | 6169.36 | 7420.22 | 7319.68 | 6029.10 | 7686.24 | 7760.36 |
| TPC-DS Q78 | 5676.84 | 6071.62 | 6103.93 | 5591.84 | 6073.52 | 5978.85 |
| TPC-DS Q79 | 3413.31 | 3822.91 | 3841.43 | 3421.33 | 3922.41 | 3980.36 |
| TPC-DS Q80 | 523.73 | 562.09 | 652.70 | 534.16 | 662.63 | 563.60 |
| TPC-DS Q81 | 220.88 | 220.16 | 221.36 | 212.61 | 219.78 | 212.44 |
| TPC-DS Q82 | 3862.53 | 4060.44 | 4096.51 | 3763.48 | 4172.12 | 4060.20 |
| TPC-DS Q83 | 913.73 | 1526.93 | 943.23 | 886.67 | 934.27 | 1058.47 |
| TPC-DS Q84 | 56.86 | 58.31 | 58.37 | 56.50 | 60.29 | 57.42 |
| TPC-DS Q85 | 121.47 | 130.24 | 136.11 | 121.23 | 130.96 | 128.22 |
| TPC-DS Q86 | 903.70 | 1164.70 | 964.87 | 906.07 | 1017.96 | 1163.04 |
| TPC-DS Q87 | 6966.96 | 8403.44 | 8802.19 | 7092.20 | 8822.29 | 8679.90 |
| TPC-DS Q88 | 15895.84 | 18404.17 | 18551.36 | 15518.00 | 21083.81 | 21158.54 |
| TPC-DS Q89 | 1555.02 | 1577.60 | 1782.64 | 1584.78 | 1755.49 | 1539.85 |
| TPC-DS Q90 | 341.08 | 331.89 | 389.58 | 358.96 | 396.67 | 328.63 |
| TPC-DS Q91 | 16.67 | 16.61 | 17.02 | 19.73 | 21.68 | 16.29 |
| TPC-DS Q92 | 10.02 | 9.39 | 9.23 | 9.23 | 9.79 | 9.45 |
| TPC-DS Q93 | 38.31 | 38.44 | 40.50 | 44.05 | 44.03 | 37.44 |
| TPC-DS Q96 | 779.19 | 749.65 | 800.53 | 782.43 | 763.98 | 735.48 |
| TPC-DS Q97 | 5163.54 | 5996.10 | 5872.14 | 5179.33 | 6084.32 | 6250.16 |
| TPC-DS Q98 | 879.55 | 1046.87 | 1113.54 | 883.05 | 1178.87 | 1030.93 |
| TPC-DS Q99 | 4918.81 | 5354.58 | 5563.41 | 4911.81 | 5592.88 | 5456.76 |
### Errors (failed queries)
| | TPC-DS Q1 | TPC-DS Q2 | TPC-DS Q3 | TPC-DS Q4 | TPC-DS Q5 | TPC-DS Q6 | TPC-DS Q7 | TPC-DS Q8 | TPC-DS Q9 | TPC-DS Q10 | TPC-DS Q11 | TPC-DS Q12 | TPC-DS Q13 | TPC-DS Q14a+b | TPC-DS Q15 | TPC-DS Q16 | TPC-DS Q17 | TPC-DS Q18 | TPC-DS Q19 | TPC-DS Q20 | TPC-DS Q21 | TPC-DS Q22 | TPC-DS Q23a+b | TPC-DS Q24a+b | TPC-DS Q25 | TPC-DS Q26 | TPC-DS Q27 | TPC-DS Q28 | TPC-DS Q29 | TPC-DS Q30 | TPC-DS Q31 | TPC-DS Q32 | TPC-DS Q33 | TPC-DS Q34 | TPC-DS Q35 | TPC-DS Q36 | TPC-DS Q37 | TPC-DS Q38 | TPC-DS Q39a+b | TPC-DS Q40 | TPC-DS Q41 | TPC-DS Q42 | TPC-DS Q43 | TPC-DS Q44 | TPC-DS Q45 | TPC-DS Q46 | TPC-DS Q47 | TPC-DS Q48 | TPC-DS Q49 | TPC-DS Q50 | TPC-DS Q51 | TPC-DS Q52 | TPC-DS Q53 | TPC-DS Q54 | TPC-DS Q55 | TPC-DS Q56 | TPC-DS Q57 | TPC-DS Q58 | TPC-DS Q59 | TPC-DS Q60 | TPC-DS Q61 | TPC-DS Q62 | TPC-DS Q63 | TPC-DS Q64 | TPC-DS Q65 | TPC-DS Q66 | TPC-DS Q67 | TPC-DS Q68 | TPC-DS Q69 | TPC-DS Q70 | TPC-DS Q71 | TPC-DS Q72 | TPC-DS Q73 | TPC-DS Q74 | TPC-DS Q75 | TPC-DS Q76 | TPC-DS Q77 | TPC-DS Q78 | TPC-DS Q79 | TPC-DS Q80 | TPC-DS Q81 | TPC-DS Q82 | TPC-DS Q83 | TPC-DS Q84 | TPC-DS Q85 | TPC-DS Q86 | TPC-DS Q87 | TPC-DS Q88 | TPC-DS Q89 | TPC-DS Q90 | TPC-DS Q91 | TPC-DS Q92 | TPC-DS Q93 | TPC-DS Q94 | TPC-DS Q95 | TPC-DS Q96 | TPC-DS Q97 | TPC-DS Q98 | TPC-DS Q99 |
|:------------------|------------:|------------:|------------:|------------:|------------:|------------:|------------:|------------:|------------:|-------------:|-------------:|-------------:|-------------:|----------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|----------------:|----------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|----------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|
| MariaDB-1-1-1-1-1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| MariaDB-1-1-2-1-1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| MariaDB-1-1-2-1-2 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| MariaDB-1-2-1-1-1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| MariaDB-1-2-2-1-1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| MariaDB-1-2-2-1-2 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 |
* TPC-DS Q94
* MariaDB-1-1-2-1-1: numRun 1: : java.sql.SQLTimeoutException: (conn=140) Query execution was interrupted (max_statement_time exceeded)
* MariaDB-1-1-2-1-2: numRun 1: : java.sql.SQLTimeoutException: (conn=141) Query execution was interrupted (max_statement_time exceeded)
* MariaDB-1-2-1-1-1: numRun 1: : java.sql.SQLTimeoutException: (conn=8) Query execution was interrupted (max_statement_time exceeded)
* MariaDB-1-1-1-1-1: numRun 1: : java.sql.SQLTimeoutException: (conn=83) Query execution was interrupted (max_statement_time exceeded)
* MariaDB-1-2-2-1-2: numRun 1: : java.sql.SQLTimeoutException: (conn=65) Query execution was interrupted (max_statement_time exceeded)
* MariaDB-1-2-2-1-1: numRun 1: : java.sql.SQLTimeoutException: (conn=64) Query execution was interrupted (max_statement_time exceeded)
* TPC-DS Q95
* MariaDB-1-1-2-1-1: numRun 1: : java.sql.SQLTimeoutException: (conn=140) Query execution was interrupted (max_statement_time exceeded)
* MariaDB-1-1-2-1-2: numRun 1: : java.sql.SQLTimeoutException: (conn=141) Query execution was interrupted (max_statement_time exceeded)
* MariaDB-1-2-1-1-1: numRun 1: : java.sql.SQLTimeoutException: (conn=8) Query execution was interrupted (max_statement_time exceeded)
* MariaDB-1-1-1-1-1: numRun 1: : java.sql.SQLTimeoutException: (conn=83) Query execution was interrupted (max_statement_time exceeded)
* MariaDB-1-2-2-1-2: numRun 1: : java.sql.SQLTimeoutException: (conn=65) Query execution was interrupted (max_statement_time exceeded)
* MariaDB-1-2-2-1-1: numRun 1: : java.sql.SQLTimeoutException: (conn=64) Query execution was interrupted (max_statement_time exceeded)
### Warnings (result mismatch)
| | TPC-DS Q1 | TPC-DS Q2 | TPC-DS Q3 | TPC-DS Q4 | TPC-DS Q5 | TPC-DS Q6 | TPC-DS Q7 | TPC-DS Q8 | TPC-DS Q9 | TPC-DS Q10 | TPC-DS Q11 | TPC-DS Q12 | TPC-DS Q13 | TPC-DS Q14a+b | TPC-DS Q15 | TPC-DS Q16 | TPC-DS Q17 | TPC-DS Q18 | TPC-DS Q19 | TPC-DS Q20 | TPC-DS Q21 | TPC-DS Q22 | TPC-DS Q23a+b | TPC-DS Q24a+b | TPC-DS Q25 | TPC-DS Q26 | TPC-DS Q27 | TPC-DS Q28 | TPC-DS Q29 | TPC-DS Q30 | TPC-DS Q31 | TPC-DS Q32 | TPC-DS Q33 | TPC-DS Q34 | TPC-DS Q35 | TPC-DS Q36 | TPC-DS Q37 | TPC-DS Q38 | TPC-DS Q39a+b | TPC-DS Q40 | TPC-DS Q41 | TPC-DS Q42 | TPC-DS Q43 | TPC-DS Q44 | TPC-DS Q45 | TPC-DS Q46 | TPC-DS Q47 | TPC-DS Q48 | TPC-DS Q49 | TPC-DS Q50 | TPC-DS Q51 | TPC-DS Q52 | TPC-DS Q53 | TPC-DS Q54 | TPC-DS Q55 | TPC-DS Q56 | TPC-DS Q57 | TPC-DS Q58 | TPC-DS Q59 | TPC-DS Q60 | TPC-DS Q61 | TPC-DS Q62 | TPC-DS Q63 | TPC-DS Q64 | TPC-DS Q65 | TPC-DS Q66 | TPC-DS Q67 | TPC-DS Q68 | TPC-DS Q69 | TPC-DS Q70 | TPC-DS Q71 | TPC-DS Q72 | TPC-DS Q73 | TPC-DS Q74 | TPC-DS Q75 | TPC-DS Q76 | TPC-DS Q77 | TPC-DS Q78 | TPC-DS Q79 | TPC-DS Q80 | TPC-DS Q81 | TPC-DS Q82 | TPC-DS Q83 | TPC-DS Q84 | TPC-DS Q85 | TPC-DS Q86 | TPC-DS Q87 | TPC-DS Q88 | TPC-DS Q89 | TPC-DS Q90 | TPC-DS Q91 | TPC-DS Q92 | TPC-DS Q93 | TPC-DS Q94 | TPC-DS Q95 | TPC-DS Q96 | TPC-DS Q97 | TPC-DS Q98 | TPC-DS Q99 |
|:------------------|------------:|------------:|------------:|------------:|------------:|------------:|------------:|------------:|------------:|-------------:|-------------:|-------------:|-------------:|----------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|----------------:|----------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|----------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|
| MariaDB-1-1-1-1-1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| MariaDB-1-1-2-1-2 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| MariaDB-1-2-1-1-1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| MariaDB-1-2-2-1-1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| MariaDB-1-2-2-1-2 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1-1 | 2090.86 | 2.86 | 8.74 | 8.85 |
| MariaDB-1-1-2-1 | 2090.86 | 2.86 | 8.74 | 8.85 |
### Loading phase: component data generator
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1-1 | 0.00 | 0.00 | 0.00 | 0.00 |
| MariaDB-1-1-2-1 | 0.00 | 0.00 | 0.00 | 0.00 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1-1 | 3.46 | 0.04 | 0.00 | 0.00 |
| MariaDB-1-1-2-1 | 3.46 | 0.04 | 0.00 | 0.00 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1-1 | 3178.46 | 1.00 | 8.84 | 8.94 |
| MariaDB-1-1-2-1 | 6509.84 | 2.00 | 9.32 | 9.45 |
| MariaDB-1-2-1-1 | 3150.31 | 1.00 | 5.85 | 5.89 |
| MariaDB-1-2-2-1 | 6573.27 | 2.00 | 6.18 | 6.23 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1-1 | 21.13 | 0.41 | 0.32 | 0.32 |
| MariaDB-1-1-2-1 | 27.56 | 0.07 | 0.33 | 0.33 |
| MariaDB-1-2-1-1 | 20.90 | 0.38 | 0.31 | 0.31 |
| MariaDB-1-2-2-1 | 26.77 | 0.06 | 0.33 | 0.33 |
### Tests
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST skipped: Loading phase: component data generator contains 0 or NaN in CPU [CPUs] (data pre-existing)
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Geo Times [s] contains no 0 or NaN
* TEST passed: Power@Size [~Q/h] contains no 0 or NaN
* TEST passed: Throughput@Size contains no 0 or NaN
* TEST failed: SQL errors
* TEST failed: SQL warnings (result mismatch)
* TEST passed: Workflow as planned
MonetDB
TPC-DS Simple
bexhoma tpcds \
-dbms MonetDB \
-sf 3 \
-nlp 8 \
-nlt 1 \
-xii -xic -xis \
-nc 1 \
-ne 1 \
-nbp 1 \
-ms $BEXHOMA_MS \
-tr \
-lr 128Gi \
-rr 128Gi \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_tpcds_monetdb_1.log
yields (after ca. 15 minutes) something like
testcase_tpcds_monetdb_1.log
## Show Summary
### Workload
TPC-DS Queries SF=3
* Type: tpcds
* Duration: 812s
* Code: 1782552004
* This includes the reading queries of TPC-DS.
* This experiment compares run time and resource consumption of TPC-DS queries in different DBMS.
* TPC-DS (SF=3) data is loaded and benchmark is executed.
* Query ordering is Q1 - Q99.
* All instances use the same query parameters.
* Timeout per query is 600.
* Import sets indexes and constraints after loading and recomputes statistics.
* Experiment uses bexhoma version 0.10.2.
* Experiment is limited to DBMS ['MonetDB'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* Database uses ephemeral storage of size 30Gi.
* Loading is tested with [1] threads, split into [8] pods.
* Benchmarking is tested with [1] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* MonetDB-1-1-1-1-1 uses docker image monetdb/monetdb:Dec2025-SP1
* RAM:540590804992
* CPU:AMD EPYC 7352 24-Core Processor
* Cores:96
* host:6.8.0-124-generic
* node:cl-worker24
* disk:144619
* cpu_list:0-95
* requests_cpu:4
* requests_memory:128Gi
* limits_memory:128Gi
* eval_parameters
* code:1782552004
### SUT Container Restarts
* bexhoma-sut-monetdb-1-1782552004-5c996dfb7c-wd4v6: 0
### Workflow
#### Actual
* DBMS MonetDB-1 - Experiment 1 Client 1: tpcds (1 pods)
#### Planned
* DBMS MonetDB-1 - Experiment 1 Client 1: tpcds (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:------------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| MonetDB-1-1 | 1 | 3 | 654.00 | 14.00 | 1.00 | 185.00 | 447.00 | 8 | 0 | | None | 0 | False | 16.51 |
### Execution
#### Per Connection
| | configuration | phase | job | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id | pod |
|:------------------|:----------------|:--------------|:----------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|:------------------|
| MonetDB-1-1-1-1-1 | MonetDB-1 | MonetDB-1-1-1 | MonetDB-1-1-1-1 | 1 | 1 | 1 | 1 | 3.00 | 99 | 34 | 0.10 | 104905.69 | 31447.06 | -1 | MonetDB-1-1-1-1-1 |
#### Per Phase
| | phase | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id |
|:--------------|:--------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|
| MonetDB-1-1-1 | MonetDB-1-1-1 | 1 | 1 | 1 | 1 | 3.00 | 99 | 34 | 0.10 | 104905.69 | 31447.06 | -1 |
### Latency of Timer Execution [ms]
| Queries | MonetDB-1-1-1-1-1 |
|:--------------|--------------------:|
| TPC-DS Q1 | 54.86 |
| TPC-DS Q2 | 209.02 |
| TPC-DS Q3 | 48.00 |
| TPC-DS Q4 | 1971.25 |
| TPC-DS Q5 | 148.98 |
| TPC-DS Q6 | 59.29 |
| TPC-DS Q7 | 69.35 |
| TPC-DS Q8 | 32.75 |
| TPC-DS Q9 | 129.60 |
| TPC-DS Q10 | 48.36 |
| TPC-DS Q11 | 931.53 |
| TPC-DS Q12 | 22.02 |
| TPC-DS Q13 | 102.36 |
| TPC-DS Q14a+b | 4524.03 |
| TPC-DS Q15 | 29.51 |
| TPC-DS Q16 | 179.23 |
| TPC-DS Q17 | 188.79 |
| TPC-DS Q18 | 103.31 |
| TPC-DS Q19 | 69.30 |
| TPC-DS Q20 | 28.38 |
| TPC-DS Q21 | 63.45 |
| TPC-DS Q22 | 1542.90 |
| TPC-DS Q23a+b | 4430.20 |
| TPC-DS Q24a+b | 257.63 |
| TPC-DS Q25 | 246.42 |
| TPC-DS Q26 | 44.24 |
| TPC-DS Q27 | 173.61 |
| TPC-DS Q28 | 117.85 |
| TPC-DS Q29 | 265.82 |
| TPC-DS Q30 | 23.41 |
| TPC-DS Q31 | 299.83 |
| TPC-DS Q32 | 25.34 |
| TPC-DS Q33 | 31.68 |
| TPC-DS Q34 | 47.38 |
| TPC-DS Q35 | 140.97 |
| TPC-DS Q36 | 190.23 |
| TPC-DS Q37 | 47.09 |
| TPC-DS Q38 | 317.58 |
| TPC-DS Q39a+b | 2405.50 |
| TPC-DS Q40 | 84.39 |
| TPC-DS Q41 | 8.07 |
| TPC-DS Q42 | 34.15 |
| TPC-DS Q43 | 90.39 |
| TPC-DS Q44 | 63.12 |
| TPC-DS Q45 | 33.01 |
| TPC-DS Q46 | 71.48 |
| TPC-DS Q47 | 268.98 |
| TPC-DS Q48 | 74.90 |
| TPC-DS Q49 | 158.44 |
| TPC-DS Q50 | 172.03 |
| TPC-DS Q51 | 726.73 |
| TPC-DS Q52 | 34.32 |
| TPC-DS Q53 | 34.13 |
| TPC-DS Q54 | 39.24 |
| TPC-DS Q55 | 27.84 |
| TPC-DS Q56 | 27.85 |
| TPC-DS Q57 | 98.63 |
| TPC-DS Q58 | 145.24 |
| TPC-DS Q59 | 134.58 |
| TPC-DS Q60 | 36.69 |
| TPC-DS Q61 | 68.46 |
| TPC-DS Q62 | 30.11 |
| TPC-DS Q63 | 33.63 |
| TPC-DS Q64 | 552.20 |
| TPC-DS Q65 | 225.18 |
| TPC-DS Q66 | 183.05 |
| TPC-DS Q67 | 807.36 |
| TPC-DS Q68 | 77.33 |
| TPC-DS Q69 | 40.74 |
| TPC-DS Q70 | 278.28 |
| TPC-DS Q71 | 49.01 |
| TPC-DS Q72 | 184.12 |
| TPC-DS Q73 | 31.17 |
| TPC-DS Q74 | 912.41 |
| TPC-DS Q75 | 828.82 |
| TPC-DS Q76 | 58.75 |
| TPC-DS Q77 | 95.79 |
| TPC-DS Q78 | 1790.85 |
| TPC-DS Q79 | 79.57 |
| TPC-DS Q80 | 1042.49 |
| TPC-DS Q81 | 21.07 |
| TPC-DS Q82 | 42.47 |
| TPC-DS Q83 | 11.62 |
| TPC-DS Q84 | 9.66 |
| TPC-DS Q85 | 149.28 |
| TPC-DS Q86 | 44.46 |
| TPC-DS Q87 | 373.37 |
| TPC-DS Q88 | 99.66 |
| TPC-DS Q89 | 56.99 |
| TPC-DS Q90 | 10.69 |
| TPC-DS Q91 | 16.97 |
| TPC-DS Q92 | 13.81 |
| TPC-DS Q93 | 183.97 |
| TPC-DS Q94 | 53.36 |
| TPC-DS Q95 | 788.74 |
| TPC-DS Q96 | 18.60 |
| TPC-DS Q97 | 362.71 |
| TPC-DS Q98 | 61.70 |
| TPC-DS Q99 | 55.50 |
### Errors (failed queries)
No errors
### Warnings (result mismatch)
No warnings
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Geo Times [s] contains no 0 or NaN
* TEST passed: Power@Size [~Q/h] contains no 0 or NaN
* TEST passed: Throughput@Size contains no 0 or NaN
* TEST passed: No SQL errors
* TEST passed: No SQL warnings
* TEST passed: Workflow as planned
TPC-DS Monitoring
bexhoma tpcds \
-dbms MonetDB \
-sf 3 \
-nlp 8 \
-nlt 1 \
-xii -xic -xis \
-nc 1 \
-ne 1 \
-nbp 1 \
-m \
-mc \
-ms $BEXHOMA_MS \
-tr \
-lr 128Gi \
-rr 128Gi \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_tpcds_monetdb_2.log
yields (after ca. 15 minutes) something like
testcase_tpcds_monetdb_2.log
## Show Summary
### Workload
TPC-DS Queries SF=3
* Type: tpcds
* Duration: 851s
* Code: 1782552906
* This includes the reading queries of TPC-DS.
* This experiment compares run time and resource consumption of TPC-DS queries in different DBMS.
* TPC-DS (SF=3) data is loaded and benchmark is executed.
* Query ordering is Q1 - Q99.
* All instances use the same query parameters.
* Timeout per query is 600.
* Import sets indexes and constraints after loading and recomputes statistics.
* Experiment uses bexhoma version 0.10.2.
* System metrics are monitored by a cluster-wide installation.
* Experiment is limited to DBMS ['MonetDB'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* Database uses ephemeral storage of size 30Gi.
* Loading is tested with [1] threads, split into [8] pods.
* Benchmarking is tested with [1] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* MonetDB-1-1-1-1-1 uses docker image monetdb/monetdb:Dec2025-SP1
* RAM:540590804992
* CPU:AMD EPYC 7352 24-Core Processor
* Cores:96
* host:6.8.0-124-generic
* node:cl-worker24
* disk:145493
* cpu_list:0-95
* requests_cpu:4
* requests_memory:128Gi
* limits_memory:128Gi
* eval_parameters
* code:1782552906
### SUT Container Restarts
* bexhoma-sut-monetdb-1-1782552906-68b7dfbc8d-6tfmh: 0
### Workflow
#### Actual
* DBMS MonetDB-1 - Experiment 1 Client 1: tpcds (1 pods)
#### Planned
* DBMS MonetDB-1 - Experiment 1 Client 1: tpcds (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:------------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| MonetDB-1-1 | 1 | 3 | 609.00 | 6.00 | 1.00 | 163.00 | 431.00 | 8 | 0 | | None | 0 | False | 17.73 |
### Execution
#### Per Connection
| | configuration | phase | job | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id | pod |
|:------------------|:----------------|:--------------|:----------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|:------------------|
| MonetDB-1-1-1-1-1 | MonetDB-1 | MonetDB-1-1-1 | MonetDB-1-1-1-1 | 1 | 1 | 1 | 1 | 3.00 | 99 | 33 | 0.10 | 106426.81 | 32400.00 | -1 | MonetDB-1-1-1-1-1 |
#### Per Phase
| | phase | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id |
|:--------------|:--------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|
| MonetDB-1-1-1 | MonetDB-1-1-1 | 1 | 1 | 1 | 1 | 3.00 | 99 | 33 | 0.10 | 106426.81 | 32400.00 | -1 |
### Latency of Timer Execution [ms]
| Queries | MonetDB-1-1-1-1-1 |
|:--------------|--------------------:|
| TPC-DS Q1 | 49.66 |
| TPC-DS Q2 | 212.79 |
| TPC-DS Q3 | 49.18 |
| TPC-DS Q4 | 1940.56 |
| TPC-DS Q5 | 154.15 |
| TPC-DS Q6 | 61.13 |
| TPC-DS Q7 | 65.65 |
| TPC-DS Q8 | 30.64 |
| TPC-DS Q9 | 126.67 |
| TPC-DS Q10 | 51.22 |
| TPC-DS Q11 | 943.33 |
| TPC-DS Q12 | 20.98 |
| TPC-DS Q13 | 98.89 |
| TPC-DS Q14a+b | 4880.73 |
| TPC-DS Q15 | 36.63 |
| TPC-DS Q16 | 164.85 |
| TPC-DS Q17 | 179.46 |
| TPC-DS Q18 | 98.87 |
| TPC-DS Q19 | 67.41 |
| TPC-DS Q20 | 27.32 |
| TPC-DS Q21 | 62.00 |
| TPC-DS Q22 | 1527.16 |
| TPC-DS Q23a+b | 4248.51 |
| TPC-DS Q24a+b | 258.72 |
| TPC-DS Q25 | 162.29 |
| TPC-DS Q26 | 41.07 |
| TPC-DS Q27 | 171.67 |
| TPC-DS Q28 | 120.18 |
| TPC-DS Q29 | 244.89 |
| TPC-DS Q30 | 22.01 |
| TPC-DS Q31 | 299.47 |
| TPC-DS Q32 | 22.50 |
| TPC-DS Q33 | 31.41 |
| TPC-DS Q34 | 46.36 |
| TPC-DS Q35 | 129.07 |
| TPC-DS Q36 | 197.30 |
| TPC-DS Q37 | 44.87 |
| TPC-DS Q38 | 287.26 |
| TPC-DS Q39a+b | 2459.20 |
| TPC-DS Q40 | 85.07 |
| TPC-DS Q41 | 4.93 |
| TPC-DS Q42 | 33.72 |
| TPC-DS Q43 | 92.25 |
| TPC-DS Q44 | 80.96 |
| TPC-DS Q45 | 30.56 |
| TPC-DS Q46 | 74.57 |
| TPC-DS Q47 | 300.76 |
| TPC-DS Q48 | 73.36 |
| TPC-DS Q49 | 170.34 |
| TPC-DS Q50 | 194.34 |
| TPC-DS Q51 | 736.53 |
| TPC-DS Q52 | 34.56 |
| TPC-DS Q53 | 32.25 |
| TPC-DS Q54 | 39.01 |
| TPC-DS Q55 | 24.29 |
| TPC-DS Q56 | 30.98 |
| TPC-DS Q57 | 89.37 |
| TPC-DS Q58 | 181.14 |
| TPC-DS Q59 | 132.64 |
| TPC-DS Q60 | 29.34 |
| TPC-DS Q61 | 68.41 |
| TPC-DS Q62 | 29.15 |
| TPC-DS Q63 | 32.50 |
| TPC-DS Q64 | 518.04 |
| TPC-DS Q65 | 209.89 |
| TPC-DS Q66 | 187.58 |
| TPC-DS Q67 | 737.45 |
| TPC-DS Q68 | 74.40 |
| TPC-DS Q69 | 46.68 |
| TPC-DS Q70 | 229.30 |
| TPC-DS Q71 | 50.04 |
| TPC-DS Q72 | 168.39 |
| TPC-DS Q73 | 30.35 |
| TPC-DS Q74 | 232.17 |
| TPC-DS Q75 | 835.34 |
| TPC-DS Q76 | 242.43 |
| TPC-DS Q77 | 122.10 |
| TPC-DS Q78 | 1818.65 |
| TPC-DS Q79 | 73.30 |
| TPC-DS Q80 | 915.01 |
| TPC-DS Q81 | 30.50 |
| TPC-DS Q82 | 55.53 |
| TPC-DS Q83 | 9.17 |
| TPC-DS Q84 | 10.31 |
| TPC-DS Q85 | 164.30 |
| TPC-DS Q86 | 43.47 |
| TPC-DS Q87 | 337.60 |
| TPC-DS Q88 | 98.23 |
| TPC-DS Q89 | 55.26 |
| TPC-DS Q90 | 10.75 |
| TPC-DS Q91 | 18.40 |
| TPC-DS Q92 | 13.46 |
| TPC-DS Q93 | 168.71 |
| TPC-DS Q94 | 49.15 |
| TPC-DS Q95 | 782.70 |
| TPC-DS Q96 | 16.58 |
| TPC-DS Q97 | 389.19 |
| TPC-DS Q98 | 61.43 |
| TPC-DS Q99 | 56.24 |
### Errors (failed queries)
No errors
### Warnings (result mismatch)
No warnings
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MonetDB-1-1-1-1 | 330.33 | 2.55 | 4.53 | 6.05 |
### Loading phase: component data generator
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MonetDB-1-1-1-1 | 0.24 | 0.00 | 0.00 | 0.00 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MonetDB-1-1-1-1 | 26.43 | 0.24 | 0.01 | 2.21 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MonetDB-1-1-1-1 | 86.74 | 4.41 | 10.46 | 12.00 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MonetDB-1-1-1-1 | 0.02 | 0.00 | 0.00 | 0.00 |
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component data generator contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Geo Times [s] contains no 0 or NaN
* TEST passed: Power@Size [~Q/h] contains no 0 or NaN
* TEST passed: Throughput@Size contains no 0 or NaN
* TEST passed: No SQL errors
* TEST passed: No SQL warnings
* TEST passed: Workflow as planned
TPC-DS Throughput Test
bexhoma tpcds \
-dbms MonetDB \
-sf 3 \
-nlp 8 \
-nlt 1 \
-xii -xic -xis \
-nc 2 \
-ne 1,2 \
-nbp 1 \
-m \
-mc \
-ms $BEXHOMA_MS \
-tr \
-lr 128Gi \
-rr 128Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_tpcds_monetdb_3.log
yields (after ca. 20 minutes) something like
testcase_tpcds_monetdb_3.log
## Show Summary
### Workload
TPC-DS Queries SF=3
Type: tpcds
Duration: 995s
Code: 1728675428
This includes the reading queries of TPC-DS.
This experiment compares run time and resource consumption of TPC-DS queries in different DBMS.
TPC-DS (SF=3) data is loaded and benchmark is executed.
Query ordering is Q1 - Q99.
All instances use the same query parameters.
Import sets indexes and constraints after loading and recomputes statistics.
System metrics are monitored by a cluster-wide installation.
Benchmark is limited to DBMS ['MonetDB'].
Import is handled by 8 processes (pods).
Loading is fixed to cl-worker19.
Benchmarking is fixed to cl-worker19.
SUT is fixed to cl-worker11.
Database is persisted to disk of type shared and size 30Gi.
Loading is tested with [1] threads, split into [8] pods.
Benchmarking is tested with [1] threads, split into [1] pods.
Benchmarking is run as [1, 2] times the number of benchmarking pods.
Experiment is run 2 times.
### Connections
MonetDB-BHT-8-1-1-1 uses docker image monetdb/monetdb:Dec2023
RAM:541008605184
CPU:AMD Opteron(tm) Processor 6378
Cores:64
host:5.15.0-116-generic
node:cl-worker11
disk:249634108
datadisk:5486698
volume_size:100G
volume_used:5.3G
requests_cpu:4
requests_memory:16Gi
MonetDB-BHT-8-1-2-1 uses docker image monetdb/monetdb:Dec2023
RAM:541008605184
CPU:AMD Opteron(tm) Processor 6378
Cores:64
host:5.15.0-116-generic
node:cl-worker11
disk:249634108
datadisk:5558770
volume_size:100G
volume_used:5.3G
requests_cpu:4
requests_memory:16Gi
MonetDB-BHT-8-1-2-2 uses docker image monetdb/monetdb:Dec2023
RAM:541008605184
CPU:AMD Opteron(tm) Processor 6378
Cores:64
host:5.15.0-116-generic
node:cl-worker11
disk:249634108
datadisk:5558770
volume_size:100G
volume_used:5.3G
requests_cpu:4
requests_memory:16Gi
MonetDB-BHT-8-2-1-1 uses docker image monetdb/monetdb:Dec2023
RAM:541008605184
CPU:AMD Opteron(tm) Processor 6378
Cores:64
host:5.15.0-116-generic
node:cl-worker11
disk:249634108
datadisk:5558771
volume_size:100G
volume_used:5.4G
requests_cpu:4
requests_memory:16Gi
MonetDB-BHT-8-2-2-1 uses docker image monetdb/monetdb:Dec2023
RAM:541008605184
CPU:AMD Opteron(tm) Processor 6378
Cores:64
host:5.15.0-116-generic
node:cl-worker11
disk:249634108
datadisk:5558197
volume_size:100G
volume_used:5.4G
requests_cpu:4
requests_memory:16Gi
MonetDB-BHT-8-2-2-2 uses docker image monetdb/monetdb:Dec2023
RAM:541008605184
CPU:AMD Opteron(tm) Processor 6378
Cores:64
host:5.15.0-116-generic
node:cl-worker11
disk:249634108
datadisk:5558197
volume_size:100G
volume_used:5.4G
requests_cpu:4
requests_memory:16Gi
### Errors (failed queries)
No errors
### Warnings (result mismatch)
MonetDB-BHT-8-1-1-1 MonetDB-BHT-8-1-2-1 MonetDB-BHT-8-1-2-2 MonetDB-BHT-8-2-1-1 MonetDB-BHT-8-2-2-1 MonetDB-BHT-8-2-2-2
TPC-DS Q1 True True True True True False
TPC-DS Q25 True True True True True False
TPC-DS Q34 True True True True True False
TPC-DS Q43 True True True True True False
TPC-DS Q44 True True True True True False
TPC-DS Q46 True True True True True False
TPC-DS Q54 True True True True True False
TPC-DS Q68 True True True True True False
TPC-DS Q73 True True True True True False
### Latency of Timer Execution [ms]
DBMS MonetDB-BHT-8-1-1-1 MonetDB-BHT-8-1-2-1 MonetDB-BHT-8-1-2-2 MonetDB-BHT-8-2-1-1 MonetDB-BHT-8-2-2-1 MonetDB-BHT-8-2-2-2
TPC-DS Q1 1457.90 55.71 52.77 1036.26 53.04 64.40
TPC-DS Q2 8897.17 433.92 417.83 6701.04 416.75 425.14
TPC-DS Q3 2492.94 43.05 33.73 1810.61 40.69 38.67
TPC-DS Q4 12439.46 3795.57 3938.42 10047.81 4018.64 3859.93
TPC-DS Q5 6301.59 740.40 634.56 3798.51 689.95 622.01
TPC-DS Q6 582.82 217.37 175.64 486.61 155.69 150.46
TPC-DS Q7 2761.20 96.81 82.15 2552.40 86.28 122.09
TPC-DS Q8 1220.89 128.82 120.85 1012.07 95.54 119.90
TPC-DS Q9 635.23 230.87 222.55 513.18 195.19 176.37
TPC-DS Q10 1944.22 66.97 74.34 1442.27 64.76 115.25
TPC-DS Q11 1995.57 1960.02 1840.68 1973.40 1940.35 1893.88
TPC-DS Q12 901.53 37.77 39.34 811.98 32.13 36.08
TPC-DS Q13 298.44 184.98 173.74 196.24 136.51 151.76
TPC-DS Q14a+b 7875.23 7310.76 6835.78 6959.55 6064.90 5986.83
TPC-DS Q15 317.83 40.80 36.44 183.75 33.21 57.51
TPC-DS Q16 1582.59 280.60 283.50 1249.30 284.59 274.86
TPC-DS Q17 1246.26 329.36 365.05 1072.00 312.44 437.02
TPC-DS Q18 949.22 220.84 266.57 1394.53 223.82 229.80
TPC-DS Q19 163.75 82.94 77.59 125.20 81.99 86.57
TPC-DS Q20 58.07 52.46 51.33 52.19 48.89 50.37
TPC-DS Q21 7896.30 66.97 93.68 5718.54 93.54 103.29
TPC-DS Q22 2789.12 2970.36 2736.95 2785.88 2820.73 2821.87
TPC-DS Q23a+b 6467.52 6817.87 7425.04 6773.74 6912.46 6692.53
TPC-DS Q24a+b 786.52 360.59 328.72 930.28 354.57 310.32
TPC-DS Q25 390.95 382.50 379.47 832.69 378.67 372.53
TPC-DS Q26 414.53 62.61 113.07 230.80 60.97 153.19
TPC-DS Q27 391.67 336.29 388.75 485.71 368.16 340.48
TPC-DS Q28 810.17 195.01 184.20 651.45 175.24 175.50
TPC-DS Q29 348.06 408.32 421.04 349.08 362.12 466.38
TPC-DS Q30 256.94 51.04 48.23 135.65 65.69 32.77
TPC-DS Q31 853.96 411.53 457.30 899.25 449.57 464.16
TPC-DS Q32 39.75 39.67 40.41 64.72 34.82 34.99
TPC-DS Q33 158.07 56.13 49.50 92.59 53.07 53.46
TPC-DS Q34 667.98 44.71 44.78 439.87 45.05 51.55
TPC-DS Q35 193.79 227.84 208.86 223.47 218.79 191.00
TPC-DS Q37 183.74 66.45 108.31 363.26 65.78 62.99
TPC-DS Q38 559.49 612.53 612.32 649.48 577.21 605.82
TPC-DS Q39a+b 2795.62 2621.09 2604.48 2728.80 2441.65 2736.29
TPC-DS Q40 454.41 84.46 95.53 672.49 81.85 82.76
TPC-DS Q41 8.78 12.81 8.59 9.38 9.06 8.77
TPC-DS Q42 36.26 38.52 89.67 37.75 50.32 46.54
TPC-DS Q43 99.81 105.60 94.91 101.64 105.66 116.34
TPC-DS Q44 67.60 65.70 80.38 70.85 78.03 63.74
TPC-DS Q45 36.77 48.37 41.96 39.58 41.19 41.62
TPC-DS Q46 293.60 112.61 80.49 126.50 72.92 107.04
TPC-DS Q47 652.86 629.48 689.78 665.06 646.98 666.18
TPC-DS Q48 129.64 126.69 134.31 453.37 129.62 150.58
TPC-DS Q49 994.84 254.84 265.41 631.35 224.06 234.91
TPC-DS Q50 411.80 257.95 239.26 331.69 232.09 340.69
TPC-DS Q51 1810.95 1761.45 1819.17 1792.25 1836.54 1803.62
TPC-DS Q52 38.61 37.25 37.28 73.31 36.40 38.77
TPC-DS Q53 63.88 57.65 57.54 66.58 56.20 70.13
TPC-DS Q54 295.37 36.98 43.54 232.27 41.14 46.92
TPC-DS Q55 36.21 29.34 29.12 30.14 29.58 34.00
TPC-DS Q56 65.69 45.06 46.22 42.60 57.03 43.49
TPC-DS Q57 258.10 182.61 187.59 196.75 187.37 228.58
TPC-DS Q58 120.90 118.55 131.27 118.36 118.08 142.61
TPC-DS Q59 279.80 517.06 524.65 249.02 256.63 260.00
TPC-DS Q60 173.97 58.92 58.14 182.61 62.58 60.17
TPC-DS Q61 132.04 67.16 66.30 91.42 167.82 64.46
TPC-DS Q62 729.19 283.22 281.48 559.81 52.02 46.38
TPC-DS Q63 54.77 76.32 74.40 53.21 50.57 62.17
TPC-DS Q64 1324.58 643.39 614.00 1151.97 582.28 597.60
TPC-DS Q65 411.61 419.52 409.04 397.73 425.15 440.10
TPC-DS Q66 1267.26 538.76 627.83 716.93 306.93 279.70
TPC-DS Q67 2420.32 2473.85 2260.13 2270.07 2176.51 2418.12
TPC-DS Q68 733.11 79.67 72.15 235.31 76.59 73.66
TPC-DS Q69 286.03 71.44 76.99 80.09 76.24 83.76
TPC-DS Q71 493.94 60.72 72.69 104.31 64.92 420.97
TPC-DS Q72 2568.27 3801.24 3849.62 2638.45 3913.62 2981.85
TPC-DS Q73 52.42 42.56 42.55 42.67 44.51 44.01
TPC-DS Q74 553.73 613.68 598.28 553.55 564.35 579.13
TPC-DS Q75 4278.21 2325.02 2354.85 3556.34 2403.58 2328.77
TPC-DS Q76 72.60 76.63 73.81 67.52 83.05 72.04
TPC-DS Q77 413.48 173.82 180.13 350.36 180.41 188.84
TPC-DS Q78 3199.09 3134.83 3575.71 3106.40 2970.40 3732.60
TPC-DS Q79 211.67 88.75 85.18 174.57 78.34 83.01
TPC-DS Q80 2100.88 2233.29 2029.86 2173.84 2227.43 2372.94
TPC-DS Q81 192.89 33.27 60.99 99.19 34.15 32.81
TPC-DS Q82 544.84 69.25 72.32 437.80 76.63 67.48
TPC-DS Q83 133.69 21.77 21.10 176.84 21.54 22.19
TPC-DS Q84 335.91 16.58 14.86 364.21 16.59 15.92
TPC-DS Q85 198.42 70.18 74.61 86.76 95.47 82.30
TPC-DS Q87 750.08 777.95 754.70 755.24 754.57 751.73
TPC-DS Q88 789.86 191.05 199.44 504.00 180.22 205.46
TPC-DS Q89 83.55 81.68 75.47 77.83 80.40 75.74
TPC-DS Q90 746.61 22.87 20.18 423.59 20.78 32.37
TPC-DS Q91 189.19 26.79 25.40 69.64 27.00 27.08
TPC-DS Q92 27.13 25.20 24.32 24.93 25.59 26.29
TPC-DS Q93 353.73 446.38 438.20 335.19 328.46 317.22
TPC-DS Q94 441.25 63.27 65.71 295.77 67.04 62.92
TPC-DS Q95 200.01 255.47 252.36 193.54 186.92 208.63
TPC-DS Q96 25.11 25.25 24.40 24.63 24.59 25.76
TPC-DS Q97 888.44 955.01 970.64 865.66 1030.15 947.86
TPC-DS Q98 85.35 83.53 86.75 86.76 87.75 89.26
TPC-DS Q99 91.98 101.54 94.49 94.38 97.89 90.17
### Loading [s]
timeGenerate timeIngesting timeSchema timeIndex timeLoad
MonetDB-BHT-8-1-1-1 1.0 275.0 5.0 105.0 393.0
MonetDB-BHT-8-1-2-1 1.0 275.0 5.0 105.0 393.0
MonetDB-BHT-8-1-2-2 1.0 275.0 5.0 105.0 393.0
MonetDB-BHT-8-2-1-1 1.0 275.0 5.0 105.0 393.0
MonetDB-BHT-8-2-2-1 1.0 275.0 5.0 105.0 393.0
MonetDB-BHT-8-2-2-2 1.0 275.0 5.0 105.0 393.0
### Geometric Mean of Medians of Timer Run [s]
Geo Times [s]
DBMS
MonetDB-BHT-8-1-1-1 0.43
MonetDB-BHT-8-1-2-1 0.18
MonetDB-BHT-8-1-2-2 0.18
MonetDB-BHT-8-2-1-1 0.38
MonetDB-BHT-8-2-2-1 0.17
MonetDB-BHT-8-2-2-2 0.18
### Power@Size
Power@Size [~Q/h]
DBMS
MonetDB-BHT-8-1-1-1 25917.90
MonetDB-BHT-8-1-2-1 64333.08
MonetDB-BHT-8-1-2-2 63407.94
MonetDB-BHT-8-2-1-1 30183.86
MonetDB-BHT-8-2-2-1 67882.00
MonetDB-BHT-8-2-2-2 64231.37
### Throughput@Size
time [s] count SF Throughput@Size [~GB/h]
DBMS SF num_experiment num_client
MonetDB-BHT-8-1-1 3 1 1 129 1 3 1841.86
MonetDB-BHT-8-1-2 3 1 2 68 2 3 6988.24
MonetDB-BHT-8-2-1 3 2 1 114 1 3 2084.21
MonetDB-BHT-8-2-2 3 2 2 65 2 3 7310.77
### Workflow
#### Actual
DBMS MonetDB-BHT-8 - Pods [[1, 2], [1, 2]]
#### Planned
DBMS MonetDB-BHT-8 - Pods [[1, 2], [1, 2]]
### Execution - SUT
CPU [CPUs] Max CPU Max RAM [Gb] Max RAM Cached [Gb]
MonetDB-BHT-8-1-1 238.66 0.00 3.32 8.23
MonetDB-BHT-8-1-2 565.72 9.99 16.86 22.39
MonetDB-BHT-8-2-1 1154.98 0.00 7.93 16.96
MonetDB-BHT-8-2-2 334.40 0.00 5.01 10.34
### Execution - Benchmarker
CPU [CPUs] Max CPU Max RAM [Gb] Max RAM Cached [Gb]
MonetDB-BHT-8-1-1 14.10 0.05 0.25 0.26
MonetDB-BHT-8-1-2 34.05 0.33 0.75 0.78
MonetDB-BHT-8-2-1 34.05 0.00 0.74 0.77
MonetDB-BHT-8-2-2 27.37 0.00 0.73 0.76
### Tests
TEST passed: Geo Times [s] contains no 0 or NaN
TEST passed: Power@Size [~Q/h] contains no 0 or NaN
TEST passed: Throughput@Size [~GB/h] contains no 0 or NaN
TEST passed: Execution SUT contains no 0 or NaN in CPU [CPUs]
TEST passed: Execution Benchmarker contains no 0 or NaN in CPU [CPUs]
TEST passed: Workflow as planned
TPC-DS Power Test Large
bexhoma tpcds \
-dbms MonetDB \
-sf 100 \
-nlp 8 \
-nlt 8 \
-xii -xic -xis \
-nc 1 \
-ne 1 \
-nbp 1 \
-m \
-mc \
-ms $BEXHOMA_MS \
-tr \
-lr 128Gi \
-rr 128Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rss 1000Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_tpcds_monetdb_4.log
yields (after ca. 110 minutes) something like
testcase_tpcds_monetdb_4.log
## Show Summary
### Workload
TPC-DS Queries SF=100
Type: tpcds
Duration: 6331s
Code: 1728676629
This includes the reading queries of TPC-DS.
This experiment compares run time and resource consumption of TPC-DS queries in different DBMS.
TPC-DS (SF=100) data is loaded and benchmark is executed.
Query ordering is Q1 - Q99.
All instances use the same query parameters.
Import sets indexes and constraints after loading and recomputes statistics.
System metrics are monitored by a cluster-wide installation.
Benchmark is limited to DBMS ['MonetDB'].
Import is handled by 8 processes (pods).
Loading is fixed to cl-worker19.
Benchmarking is fixed to cl-worker19.
SUT is fixed to cl-worker11.
Database is persisted to disk of type shared and size 300Gi.
Loading is tested with [8] threads, split into [8] pods.
Benchmarking is tested with [1] threads, split into [1] pods.
Benchmarking is run as [1] times the number of benchmarking pods.
Experiment is run once.
### Connections
MonetDB-BHT-8-1-1 uses docker image monetdb/monetdb:Dec2023
RAM:541008605184
CPU:AMD Opteron(tm) Processor 6378
Cores:64
host:5.15.0-116-generic
node:cl-worker11
disk:249634108
datadisk:162530732
volume_size:300G
volume_used:155G
requests_cpu:4
requests_memory:16Gi
### Errors (failed queries)
No errors
### Warnings (result mismatch)
No warnings
### Latency of Timer Execution [ms]
DBMS MonetDB-BHT-8-1-1
TPC-DS Q1 21540.10
TPC-DS Q2 234536.17
TPC-DS Q3 52161.56
TPC-DS Q4 383862.53
TPC-DS Q5 130445.75
TPC-DS Q6 12467.55
TPC-DS Q7 73421.64
TPC-DS Q8 47626.63
TPC-DS Q9 35999.02
TPC-DS Q10 13875.78
TPC-DS Q11 87521.23
TPC-DS Q12 2813.57
TPC-DS Q13 7253.94
TPC-DS Q14a+b 410993.67
TPC-DS Q15 11345.34
TPC-DS Q16 20444.94
TPC-DS Q17 112165.36
TPC-DS Q18 24168.99
TPC-DS Q19 5743.73
TPC-DS Q20 1023.79
TPC-DS Q21 91704.61
TPC-DS Q22 86337.22
TPC-DS Q23a+b 2736895.84
TPC-DS Q24a+b 6107.79
TPC-DS Q25 20943.89
TPC-DS Q26 1762.98
TPC-DS Q27 28616.77
TPC-DS Q28 24479.10
TPC-DS Q29 14225.98
TPC-DS Q30 2905.96
TPC-DS Q31 22703.53
TPC-DS Q32 535.63
TPC-DS Q33 14858.80
TPC-DS Q34 2961.26
TPC-DS Q35 7629.06
TPC-DS Q37 34972.06
TPC-DS Q38 29446.64
TPC-DS Q39a+b 79446.39
TPC-DS Q40 5005.79
TPC-DS Q41 340.97
TPC-DS Q42 2261.78
TPC-DS Q43 2160.14
TPC-DS Q44 3335.84
TPC-DS Q45 1007.63
TPC-DS Q46 2493.34
TPC-DS Q47 5878.41
TPC-DS Q48 1175.38
TPC-DS Q49 38294.50
TPC-DS Q50 3741.81
TPC-DS Q51 50318.14
TPC-DS Q52 1265.41
TPC-DS Q53 2107.47
TPC-DS Q54 7608.31
TPC-DS Q55 2060.64
TPC-DS Q56 12970.61
TPC-DS Q57 2303.47
TPC-DS Q58 5467.10
TPC-DS Q59 11463.27
TPC-DS Q60 3737.78
TPC-DS Q61 4874.03
TPC-DS Q62 3992.80
TPC-DS Q63 950.49
TPC-DS Q64 45759.04
TPC-DS Q65 25361.62
TPC-DS Q66 21174.82
TPC-DS Q67 112971.25
TPC-DS Q68 9680.79
TPC-DS Q69 3257.94
TPC-DS Q71 4878.81
TPC-DS Q72 32102.01
TPC-DS Q73 322.64
TPC-DS Q74 24826.16
TPC-DS Q75 187073.16
TPC-DS Q76 14043.38
TPC-DS Q77 8627.66
TPC-DS Q78 191797.32
TPC-DS Q79 5074.40
TPC-DS Q80 117303.11
TPC-DS Q81 2790.04
TPC-DS Q82 28911.03
TPC-DS Q83 1952.11
TPC-DS Q84 879.36
TPC-DS Q85 3602.37
TPC-DS Q87 59126.34
TPC-DS Q88 3332.16
TPC-DS Q89 3645.28
TPC-DS Q90 1185.67
TPC-DS Q91 1299.77
TPC-DS Q92 2001.95
TPC-DS Q93 28101.76
TPC-DS Q94 2069.09
TPC-DS Q95 18037.05
TPC-DS Q96 119.80
TPC-DS Q97 42468.49
TPC-DS Q98 3413.94
TPC-DS Q99 1930.03
### Loading [s]
timeGenerate timeIngesting timeSchema timeIndex timeLoad
MonetDB-BHT-8-1-1 4023.0 1362.0 6.0 2703.0 8096.0
### Geometric Mean of Medians of Timer Run [s]
Geo Times [s]
DBMS
MonetDB-BHT-8-1-1 10.34
### Power@Size
Power@Size [~Q/h]
DBMS
MonetDB-BHT-8-1-1 35243.87
### Throughput@Size
time [s] count SF Throughput@Size [~GB/h]
DBMS SF num_experiment num_client
MonetDB-BHT-8-1 100 1 1 6117 1 100 1294.75
### Workflow
#### Actual
DBMS MonetDB-BHT-8 - Pods [[1]]
#### Planned
DBMS MonetDB-BHT-8 - Pods [[1]]
### Execution - SUT
CPU [CPUs] Max CPU Max RAM [Gb] Max RAM Cached [Gb]
MonetDB-BHT-8-1 20233.87 16.8 133.36 245.14
### Execution - Benchmarker
CPU [CPUs] Max CPU Max RAM [Gb] Max RAM Cached [Gb]
MonetDB-BHT-8-1 42.02 0.06 0.39 0.4
### Tests
TEST passed: Geo Times [s] contains no 0 or NaN
TEST passed: Power@Size [~Q/h] contains no 0 or NaN
TEST passed: Throughput@Size [~GB/h] contains no 0 or NaN
TEST passed: Execution SUT contains no 0 or NaN in CPU [CPUs]
TEST passed: Execution Benchmarker contains no 0 or NaN in CPU [CPUs]
TEST passed: Workflow as planned
TPC-DS Throughput Test Large
bexhoma tpcds \
-dbms MonetDB \
-sf 100 \
-nlp 8 \
-nlt 8 \
-xii -xic -xis \
-nc 2 \
-ne 1,5 \
-nbp 1 \
-m \
-mc \
-ms $BEXHOMA_MS \
-tr \
-lr 128Gi \
-rr 128Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rss 1000Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_tpcds_monetdb_5.log
yields (after ca. 360 minutes) something like
testcase_tpcds_monetdb_5.log
## Show Summary
### Workload
TPC-DS Queries SF=100
Type: tpcds
Duration: 20752s
Code: 1729497306
This includes the reading queries of TPC-DS.
This experiment compares run time and resource consumption of TPC-DS queries in different DBMS.
TPC-DS (SF=100) data is loaded and benchmark is executed.
Query ordering is Q1 - Q99.
All instances use the same query parameters.
Import sets indexes and constraints after loading and recomputes statistics.
System metrics are monitored by a cluster-wide installation.
Benchmark is limited to DBMS ['MonetDB'].
Import is handled by 8 processes (pods).
Loading is fixed to cl-worker19.
Benchmarking is fixed to cl-worker19.
SUT is fixed to cl-worker11.
Database is persisted to disk of type shared and size 300Gi.
Loading is tested with [8] threads, split into [8] pods.
Benchmarking is tested with [1] threads, split into [1] pods.
Benchmarking is run as [1, 1, 5, 5] times the number of benchmarking pods.
Experiment is run once.
### Connections
MonetDB-BHT-8-1-1 uses docker image monetdb/monetdb:Dec2023
RAM:541008605184
CPU:AMD Opteron(tm) Processor 6378
Cores:64
host:5.15.0-116-generic
node:cl-worker11
disk:250321764
datadisk:288744504
volume_size:300G
volume_used:276G
requests_cpu:4
requests_memory:16Gi
MonetDB-BHT-8-2-1 uses docker image monetdb/monetdb:Dec2023
RAM:541008605184
CPU:AMD Opteron(tm) Processor 6378
Cores:64
host:5.15.0-116-generic
node:cl-worker11
disk:250322104
datadisk:157561721
volume_size:300G
volume_used:151G
requests_cpu:4
requests_memory:16Gi
MonetDB-BHT-8-3-1 uses docker image monetdb/monetdb:Dec2023
RAM:541008605184
CPU:AMD Opteron(tm) Processor 6378
Cores:64
host:5.15.0-116-generic
node:cl-worker11
disk:250322276
datadisk:157561721
volume_size:300G
volume_used:151G
requests_cpu:4
requests_memory:16Gi
MonetDB-BHT-8-3-2 uses docker image monetdb/monetdb:Dec2023
RAM:541008605184
CPU:AMD Opteron(tm) Processor 6378
Cores:64
host:5.15.0-116-generic
node:cl-worker11
disk:250322276
datadisk:157561721
volume_size:300G
volume_used:151G
requests_cpu:4
requests_memory:16Gi
MonetDB-BHT-8-3-3 uses docker image monetdb/monetdb:Dec2023
RAM:541008605184
CPU:AMD Opteron(tm) Processor 6378
Cores:64
host:5.15.0-116-generic
node:cl-worker11
disk:250322276
datadisk:157561721
volume_size:300G
volume_used:151G
requests_cpu:4
requests_memory:16Gi
MonetDB-BHT-8-3-4 uses docker image monetdb/monetdb:Dec2023
RAM:541008605184
CPU:AMD Opteron(tm) Processor 6378
Cores:64
host:5.15.0-116-generic
node:cl-worker11
disk:250322276
datadisk:157561721
volume_size:300G
volume_used:151G
requests_cpu:4
requests_memory:16Gi
MonetDB-BHT-8-3-5 uses docker image monetdb/monetdb:Dec2023
RAM:541008605184
CPU:AMD Opteron(tm) Processor 6378
Cores:64
host:5.15.0-116-generic
node:cl-worker11
disk:250322276
datadisk:157561721
volume_size:300G
volume_used:151G
requests_cpu:4
requests_memory:16Gi
MonetDB-BHT-8-4-1 uses docker image monetdb/monetdb:Dec2023
RAM:541008605184
CPU:AMD Opteron(tm) Processor 6378
Cores:64
host:5.15.0-116-generic
node:cl-worker11
disk:250322616
datadisk:232899461
volume_size:300G
volume_used:223G
requests_cpu:4
requests_memory:16Gi
MonetDB-BHT-8-4-2 uses docker image monetdb/monetdb:Dec2023
RAM:541008605184
CPU:AMD Opteron(tm) Processor 6378
Cores:64
host:5.15.0-116-generic
node:cl-worker11
disk:250322616
datadisk:232899461
volume_size:300G
volume_used:223G
requests_cpu:4
requests_memory:16Gi
MonetDB-BHT-8-4-3 uses docker image monetdb/monetdb:Dec2023
RAM:541008605184
CPU:AMD Opteron(tm) Processor 6378
Cores:64
host:5.15.0-116-generic
node:cl-worker11
disk:250322616
datadisk:232899461
volume_size:300G
volume_used:223G
requests_cpu:4
requests_memory:16Gi
MonetDB-BHT-8-4-4 uses docker image monetdb/monetdb:Dec2023
RAM:541008605184
CPU:AMD Opteron(tm) Processor 6378
Cores:64
host:5.15.0-116-generic
node:cl-worker11
disk:250322616
datadisk:232899461
volume_size:300G
volume_used:223G
requests_cpu:4
requests_memory:16Gi
MonetDB-BHT-8-4-5 uses docker image monetdb/monetdb:Dec2023
RAM:541008605184
CPU:AMD Opteron(tm) Processor 6378
Cores:64
host:5.15.0-116-generic
node:cl-worker11
disk:250322616
datadisk:232899461
volume_size:300G
volume_used:223G
requests_cpu:4
requests_memory:16Gi
### Errors (failed queries)
MonetDB-BHT-8-1-1 MonetDB-BHT-8-2-1 MonetDB-BHT-8-3-1 MonetDB-BHT-8-3-2 MonetDB-BHT-8-3-3 MonetDB-BHT-8-3-4 MonetDB-BHT-8-3-5 MonetDB-BHT-8-4-1 MonetDB-BHT-8-4-2 MonetDB-BHT-8-4-3 MonetDB-BHT-8-4-4 MonetDB-BHT-8-4-5
TPC-DS Q23a+b False False True True True True True True True True True True
TPC-DS Q24a+b False False True False True False False False False False True False
TPC-DS Q25 False False True False True False False False False False True False
TPC-DS Q26 False False True False True False False False False False True False
TPC-DS Q27 False False True False True False False False False False True False
TPC-DS Q28 False False True False True False False False False False True False
TPC-DS Q29 False False True False True False False False False False True False
TPC-DS Q30 False False True False True False False False False False True False
TPC-DS Q31 False False True False True False False False False False True False
TPC-DS Q32 False False True False True False False False False False True False
TPC-DS Q33 False False True False True False False False False False True False
TPC-DS Q34 False False True False True False False False False False True False
TPC-DS Q35 False False True False True False False False False False True False
TPC-DS Q37 False False True False True False False False False False True False
TPC-DS Q38 False False True False True False False False False False True False
TPC-DS Q39a+b False False True False True False False False False False True False
TPC-DS Q40 False False True False True False False False False False True False
TPC-DS Q41 False False True False True False False False False False True False
TPC-DS Q42 False False True False True False False False False False True False
TPC-DS Q43 False False True False True False False False False False True False
TPC-DS Q44 False False True False True False False False False False True False
TPC-DS Q45 False False True False True False False False False False True False
TPC-DS Q46 False False True False True False False False False False True False
TPC-DS Q47 False False True False True False False False False False True False
TPC-DS Q48 False False True False True False False False False False True False
TPC-DS Q49 False False True False True False False False False False True False
TPC-DS Q50 False False True False True False False False False False True False
TPC-DS Q51 False False True False True False False False False False True False
TPC-DS Q52 False False True False True False False False False False True False
TPC-DS Q53 False False True False True False False False False False True False
TPC-DS Q54 False False True False True False False False False False True False
TPC-DS Q55 False False True False True False False False False False True False
TPC-DS Q56 False False True False True False False False False False True False
TPC-DS Q57 False False True False True False False False False False True False
TPC-DS Q58 False False True False True False False False False False True False
TPC-DS Q59 False False True False True False False False False False True False
TPC-DS Q60 False False True False True False False False False False True False
TPC-DS Q61 False False True False True False False False False False True False
TPC-DS Q62 False False True False True False False False False False True False
TPC-DS Q63 False False True False True False False False False False True False
TPC-DS Q64 False False True False True False False False False False True False
TPC-DS Q65 False False True False True False False False False False True False
TPC-DS Q66 False False True False True False False False False False True False
TPC-DS Q67 False False True False True False False False False False True False
TPC-DS Q68 False False True True True False False False False False True False
TPC-DS Q69 False False True True True False False False False False True False
TPC-DS Q71 False False True True True False False False False False True False
TPC-DS Q72 False False True True True False False False False False True False
TPC-DS Q73 False False True True True False False False True True True False
TPC-DS Q74 False False True True True False False False True True True False
TPC-DS Q75 False False True True True False False False True True True True
TPC-DS Q76 False False True True True False False True True True True True
TPC-DS Q77 False False True True True False False True True True True True
TPC-DS Q78 False False True True True True True True True True True True
TPC-DS Q79 False False True True True True True True True True True True
TPC-DS Q80 False False True True True True True True True True True True
TPC-DS Q81 False False True True True True True True True True True True
TPC-DS Q82 False False True True True True True True True True True True
TPC-DS Q83 False False True True True True True True True True True True
TPC-DS Q84 False False True True True True True True True True True True
TPC-DS Q85 False False True True True True True True True True True True
TPC-DS Q87 False False True True True True True True True True True True
TPC-DS Q88 False False True True True True True True True True True True
TPC-DS Q89 False False True True True True True True True True True True
TPC-DS Q90 False False True True True True True True True True True True
TPC-DS Q91 False False True True True True True True True True True True
TPC-DS Q92 False False True True True True True True True True True True
TPC-DS Q93 False False True True True True True True True True True True
TPC-DS Q94 False False True True True True True True True True True True
TPC-DS Q95 False False True True True True True True True True True True
TPC-DS Q96 False False True True True True True True True True True True
TPC-DS Q97 False False True True True True True True True True True True
TPC-DS Q98 False False True True True True True True True True True True
TPC-DS Q99 False False True True True True True True True True True True
### Warnings (result mismatch)
MonetDB-BHT-8-1-1 MonetDB-BHT-8-2-1 MonetDB-BHT-8-3-1 MonetDB-BHT-8-3-2 MonetDB-BHT-8-3-3 MonetDB-BHT-8-3-4 MonetDB-BHT-8-3-5 MonetDB-BHT-8-4-1 MonetDB-BHT-8-4-2 MonetDB-BHT-8-4-3 MonetDB-BHT-8-4-4 MonetDB-BHT-8-4-5
TPC-DS Q1 True True True False True True True True True True True True
TPC-DS Q34 True True False False False True True True True True False True
TPC-DS Q46 True True False False False True True True True True False True
TPC-DS Q54 True True False False False True True True True True False True
TPC-DS Q68 True True False False False True True True True True False True
TPC-DS Q69 True True False False False True True True True True False True
TPC-DS Q71 True True False False False True True True True True False True
TPC-DS Q72 True True False False False True True True True True False True
TPC-DS Q73 True True False False False True True True False False False True
TPC-DS Q74 True True False False False True True True False False False True
TPC-DS Q75 True True False False False True True True False False False False
TPC-DS Q76 True True False False False True True False False False False False
TPC-DS Q77 True True False False False True True False False False False False
TPC-DS Q78 True True False False False False False False False False False False
TPC-DS Q79 True True False False False False False False False False False False
TPC-DS Q80 True True False False False False False False False False False False
TPC-DS Q81 True True False False False False False False False False False False
TPC-DS Q82 True True False False False False False False False False False False
TPC-DS Q83 True True False False False False False False False False False False
TPC-DS Q84 True True False False False False False False False False False False
TPC-DS Q85 True True False False False False False False False False False False
TPC-DS Q87 True True False False False False False False False False False False
TPC-DS Q88 True True False False False False False False False False False False
TPC-DS Q89 True True False False False False False False False False False False
TPC-DS Q90 True True False False False False False False False False False False
TPC-DS Q91 True True False False False False False False False False False False
TPC-DS Q92 True True False False False False False False False False False False
TPC-DS Q93 True True False False False False False False False False False False
TPC-DS Q94 True True False False False False False False False False False False
TPC-DS Q95 True True False False False False False False False False False False
TPC-DS Q96 True True False False False False False False False False False False
TPC-DS Q97 True True False False False False False False False False False False
TPC-DS Q98 True True False False False False False False False False False False
TPC-DS Q99 True True False False False False False False False False False False
### Latency of Timer Execution [ms]
DBMS MonetDB-BHT-8-1-1 MonetDB-BHT-8-2-1 MonetDB-BHT-8-3-1 MonetDB-BHT-8-3-2 MonetDB-BHT-8-3-3 MonetDB-BHT-8-3-4 MonetDB-BHT-8-3-5 MonetDB-BHT-8-4-1 MonetDB-BHT-8-4-2 MonetDB-BHT-8-4-3 MonetDB-BHT-8-4-4 MonetDB-BHT-8-4-5
TPC-DS Q1 27370.75 2398.34 3427.48 3548.13 2621.80 3349.98 2533.30 42994.62 43099.49 42921.18 42831.04 42908.51
TPC-DS Q2 290821.48 12968.89 42420.72 43545.27 43077.55 42017.09 43019.55 587520.51 587142.92 589163.04 587448.87 588656.35
TPC-DS Q3 113297.32 2534.75 2226.94 1311.94 2827.32 2754.30 2898.16 126104.75 126592.93 124695.38 126524.33 125210.92
TPC-DS Q4 419012.49 151417.82 841027.33 841979.72 771544.18 758497.43 987181.05 863825.28 842226.46 875548.69 841795.93 864986.88
TPC-DS Q5 152472.27 25913.86 38700.56 38704.71 75992.76 89474.50 35704.63 280847.09 302657.12 272871.72 309269.37 290256.00
TPC-DS Q6 10465.72 6148.11 8494.14 8372.64 20295.71 19794.75 10755.25 39960.65 38099.00 35536.35 36848.37 32571.94
TPC-DS Q7 85464.02 1717.00 656.49 659.43 14569.19 14941.80 779.82 125905.42 127393.38 126724.35 122928.01 123021.42
TPC-DS Q8 31537.49 2868.56 51567.86 50598.40 57879.85 57629.18 4131.33 37375.11 37821.85 37375.05 37374.59 37374.76
TPC-DS Q9 24558.10 2959.41 9601.51 7915.98 8954.79 8555.44 2609.90 39832.01 37006.15 39783.22 39745.52 39686.99
TPC-DS Q10 4798.43 1708.85 10055.94 12059.05 11024.65 11410.25 1068.77 8830.09 11203.75 8863.00 8856.28 8982.50
TPC-DS Q11 74948.80 74442.53 127970.28 129086.83 119531.67 127820.74 98889.67 128465.69 116448.46 123230.43 115624.77 126471.84
TPC-DS Q12 1922.56 586.98 489.47 429.42 8194.10 615.13 359.98 302.72 711.24 3474.69 1657.43 523.10
TPC-DS Q13 2705.39 1601.06 3541.62 2587.14 4190.26 3541.01 356.11 335.97 8615.20 375.12 8625.05 414.59
TPC-DS Q14a+b 331389.83 274582.58 495565.35 450771.53 450517.42 450518.97 498161.82 544996.30 520469.11 543487.22 534403.23 570664.29
TPC-DS Q15 11234.26 1133.29 539.39 3274.29 3533.18 3535.34 704.67 639.50 14143.56 606.52 869.53 1274.78
TPC-DS Q16 25139.98 863.51 783.43 5917.83 5936.80 5181.27 795.14 43898.62 57727.55 46832.57 57696.69 19303.74
TPC-DS Q17 57273.54 34438.89 136125.32 173456.87 179543.43 175108.63 85005.01 218938.52 223817.53 219048.18 220848.61 219440.73
TPC-DS Q18 26480.21 8663.66 22465.62 22848.76 15768.86 21216.73 21039.15 51344.16 46580.72 51222.66 49248.84 50783.59
TPC-DS Q19 2424.20 2259.17 6801.66 6127.00 6794.42 6798.48 6977.89 9677.23 9660.20 9704.29 9707.72 9694.76
TPC-DS Q20 941.11 498.07 859.07 876.10 1179.98 1178.25 1177.47 731.59 729.09 735.39 730.50 730.95
TPC-DS Q21 115406.12 950.95 6428.35 6422.47 6412.70 6404.55 6426.69 142631.61 142621.07 142619.30 142613.04 142602.51
TPC-DS Q22 91135.38 81955.21 97082.60 112905.33 112828.09 107411.61 97016.26 97696.66 110253.74 97281.24 91222.80 121667.95
### Loading [s]
timeGenerate timeIngesting timeSchema timeIndex timeLoad
MonetDB-BHT-8-1-1 4023.0 1362.0 6.0 2703.0 8096.0
MonetDB-BHT-8-2-1 4023.0 1362.0 6.0 2703.0 8096.0
MonetDB-BHT-8-3-1 4023.0 1362.0 6.0 2703.0 8096.0
MonetDB-BHT-8-3-2 4023.0 1362.0 6.0 2703.0 8096.0
MonetDB-BHT-8-3-3 4023.0 1362.0 6.0 2703.0 8096.0
MonetDB-BHT-8-3-4 4023.0 1362.0 6.0 2703.0 8096.0
MonetDB-BHT-8-3-5 4023.0 1362.0 6.0 2703.0 8096.0
MonetDB-BHT-8-4-1 4023.0 1362.0 6.0 2703.0 8096.0
MonetDB-BHT-8-4-2 4023.0 1362.0 6.0 2703.0 8096.0
MonetDB-BHT-8-4-3 4023.0 1362.0 6.0 2703.0 8096.0
MonetDB-BHT-8-4-4 4023.0 1362.0 6.0 2703.0 8096.0
MonetDB-BHT-8-4-5 4023.0 1362.0 6.0 2703.0 8096.0
### Geometric Mean of Medians of Timer Run [s]
Geo Times [s]
DBMS
MonetDB-BHT-8-1-1 29.76
MonetDB-BHT-8-2-1 5.57
MonetDB-BHT-8-3-1 11.35
MonetDB-BHT-8-3-2 13.07
MonetDB-BHT-8-3-3 19.37
MonetDB-BHT-8-3-4 17.48
MonetDB-BHT-8-3-5 7.76
MonetDB-BHT-8-4-1 35.40
MonetDB-BHT-8-4-2 49.88
MonetDB-BHT-8-4-3 39.40
MonetDB-BHT-8-4-4 44.92
MonetDB-BHT-8-4-5 36.51
### Power@Size
Power@Size [~Q/h]
DBMS
MonetDB-BHT-8-1-1 12115.25
MonetDB-BHT-8-2-1 64912.63
MonetDB-BHT-8-3-1 31815.26
MonetDB-BHT-8-3-2 27619.23
MonetDB-BHT-8-3-3 18609.96
MonetDB-BHT-8-3-4 20642.87
MonetDB-BHT-8-3-5 46558.91
MonetDB-BHT-8-4-1 10194.85
MonetDB-BHT-8-4-2 7229.98
MonetDB-BHT-8-4-3 9147.06
MonetDB-BHT-8-4-4 8024.32
MonetDB-BHT-8-4-5 9884.43
### Throughput@Size
time [s] count SF Throughput@Size [~GB/h]
DBMS SF num_experiment num_client
MonetDB-BHT-8-1 100 1 1 6889 1 100 1149.66
MonetDB-BHT-8-2 100 1 2 3711 1 100 2134.20
MonetDB-BHT-8-3 100 1 3 4040 5 100 9801.98
MonetDB-BHT-8-4 100 1 4 5477 5 100 7230.24
### Workflow
#### Actual
DBMS MonetDB-BHT-8 - Pods [[1, 1, 5, 5]]
#### Planned
DBMS MonetDB-BHT-8 - Pods [[1, 1, 5, 5]]
### Execution - SUT
CPU [CPUs] Max CPU Max RAM [Gb] Max RAM Cached [Gb]
MonetDB-BHT-8-1 21745.36 17.29 173.03 283.57
MonetDB-BHT-8-2 18120.57 20.21 188.28 290.39
MonetDB-BHT-8-3 43989.47 42.21 461.98 484.15
MonetDB-BHT-8-4 50356.12 45.19 457.93 484.00
### Execution - Benchmarker
CPU [CPUs] Max CPU Max RAM [Gb] Max RAM Cached [Gb]
MonetDB-BHT-8-1 39.84 0.10 0.38 0.41
MonetDB-BHT-8-2 39.84 0.01 0.63 0.66
MonetDB-BHT-8-3 80.27 0.03 1.58 1.62
MonetDB-BHT-8-4 126.49 0.12 1.78 1.82
### Tests
TEST passed: Geo Times [s] contains no 0 or NaN
TEST passed: Power@Size [~Q/h] contains no 0 or NaN
TEST passed: Throughput@Size [~GB/h] contains no 0 or NaN
TEST failed: SQL errors
TEST failed: SQL warnings (result mismatch)
TEST passed: Execution SUT contains no 0 or NaN in CPU [CPUs]
TEST passed: Execution Benchmarker contains no 0 or NaN in CPU [CPUs]
TEST passed: Workflow as planned
Benchbase
PostgreSQL
Benchbase Simple
bexhoma benchbase \
-dbms PostgreSQL \
-sf 16 \
-xsd 5 \
-xtb 1024 \
-nc 1 \
-ne 1 \
-nbp 1 \
-nbt 160 \
-xnbf 8 \
-ms $BEXHOMA_MS \
-tr \
-lr 64Gi \
-rr 64Gi \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_benchbase_postgresql_1.log
yields (after ca. 10 minutes) something like
testcase_benchbase_postgresql_1.log
## Show Summary
### Workload
Benchbase Workload tpcc SF=16
* Type: benchbase
* Duration: 847s
* Code: 1782361771
* Benchbase runs a TPC-C experiment.
* This experiment compares run time and resource consumption of Benchbase queries in different DBMS.
* Benchbase data is generated and loaded using several threads.
* Benchmark is 'tpcc'. Scaling factor is 16. Target is based on multiples of '1024'. Factors for benchmarking are [8]. Benchmarking runs for 5 minutes.
* Experiment uses bexhoma version 0.10.0.
* Experiment is limited to DBMS ['PostgreSQL'].
* Import is handled by 1 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker38.
* Loading is tested with [1] threads, split into [1] pods.
* Benchmarking is tested with [160] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* PostgreSQL-1-1-1-1 uses docker image postgres:18.3
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:261336
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782361771
* TENANT_VOL:False
### Workflow
#### Actual
* DBMS PostgreSQL-1 - Experiment 1 Client 1: benchbase (1 pods)
#### Planned
* DBMS PostgreSQL-1 - Experiment 1 Client 1: benchbase (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:---------------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| PostgreSQL-1-1 | 1 | 16 | 333.00 | 1.00 | 0.00 | 139.00 | 193.00 | 1 | 1 | | | 0 | False | 172.97 |
### Execution
#### Per Connection
| DBMS | phase | job | experiment_run | terminals | target | client | benchmark_run | child | tenant_id | time | num_errors | Throughput (requests/second) | Goodput (requests/second) | efficiency | Latency Distribution.95th Percentile Latency (microseconds) | Latency Distribution.Average Latency (microseconds) |
|:---------------------|:-----------------|:-------------------|-----------------:|------------:|---------:|---------:|----------------:|--------:|------------:|-------:|-------------:|-------------------------------:|----------------------------:|-------------:|--------------------------------------------------------------:|------------------------------------------------------:|
| PostgreSQL-1-1-1-1-1 | PostgreSQL-1-1-1 | PostgreSQL-1-1-1-1 | 1 | 160 | 8192 | 1 | 1 | 1 | 0 | 300.00 | 56 | 4792.09 | 4717.51 | 0.00 | 112400.00 | 32132.00 |
#### Per Phase
| DBMS | phase | experiment_run | terminals | target | benchmark_run | pod_count | tenant_id | time | num_errors | Throughput (requests/second) | Goodput (requests/second) | efficiency | Latency Distribution.95th Percentile Latency (microseconds) | Latency Distribution.Average Latency (microseconds) |
|:-----------------|:-----------------|-----------------:|------------:|---------:|----------------:|------------:|------------:|-------:|-------------:|-------------------------------:|----------------------------:|-------------:|--------------------------------------------------------------:|------------------------------------------------------:|
| PostgreSQL-1-1-1 | PostgreSQL-1-1-1 | 1 | 160 | 8192 | 1 | 1 | 0 | 300.00 | 56 | 4792.09 | 4717.51 | 0.00 | 112400.00 | 32132.00 |
### Tests
* TEST passed: Throughput (requests/second) contains no 0 or NaN
* TEST passed: Workflow as planned
Benchbase Persistency
bexhoma benchbase \
-dbms PostgreSQL \
-sf 16 \
-xsd 1 \
-xtb 1024 \
-nc 2 \
-ne 1 \
-nbp 1 \
-nbt 160 \
-xnbf 8 \
-ms $BEXHOMA_MS \
-lr 64Gi \
-rr 64Gi \
-rsr \
-rst $BEXHOMA_STORAGE_CLASS \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_benchbase_postgresql_2.log
yields (after ca. 10 minutes) something like
testcase_benchbase_postgresql_2.log
## Show Summary
### Workload
Benchbase Workload tpcc SF=16
* Type: benchbase
* Duration: 824s
* Code: 1782362661
* Benchbase runs a TPC-C experiment.
* This experiment compares run time and resource consumption of Benchbase queries in different DBMS.
* Benchbase data is generated and loaded using several threads.
* Benchmark is 'tpcc'. Scaling factor is 16. Target is based on multiples of '1024'. Factors for benchmarking are [8]. Benchmarking runs for 1 minutes.
* Experiment uses bexhoma version 0.10.0.
* Experiment is limited to DBMS ['PostgreSQL'].
* Import is handled by 1 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker38.
* Database is persisted to disk of type shared and size 30Gi.
* Loading is tested with [1] threads, split into [1] pods.
* Benchmarking is tested with [160] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run 2 times.
### Connections
* PostgreSQL-1-1-1-1 uses docker image postgres:18.3
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:244255
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782362661
* TENANT_VOL:False
* PostgreSQL-1-2-1-1 uses docker image postgres:18.3
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:253332
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782362661
* TENANT_VOL:False
### Workflow
#### Actual
* DBMS PostgreSQL-1 - Experiment 1 Client 1: benchbase (1 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 1: benchbase (1 pods)
#### Planned
* DBMS PostgreSQL-1 - Experiment 1 Client 1: benchbase (1 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 1: benchbase (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:---------------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| PostgreSQL-1-1 | 1 | 16 | 749.00 | 1.00 | 0.00 | 348.00 | 400.00 | 0 | 1 | | | 0 | False | 76.90 |
| PostgreSQL-1-2 | 2 | 16 | 749.00 | 1.00 | 0.00 | 348.00 | 400.00 | 0 | 1 | | | 0 | False | 76.90 |
### Execution
#### Per Connection
| DBMS | phase | job | experiment_run | terminals | target | client | benchmark_run | child | tenant_id | time | num_errors | Throughput (requests/second) | Goodput (requests/second) | efficiency | Latency Distribution.95th Percentile Latency (microseconds) | Latency Distribution.Average Latency (microseconds) |
|:---------------------|:-----------------|:-------------------|-----------------:|------------:|---------:|---------:|----------------:|--------:|------------:|-------:|-------------:|-------------------------------:|----------------------------:|-------------:|--------------------------------------------------------------:|------------------------------------------------------:|
| PostgreSQL-1-1-1-1-1 | PostgreSQL-1-1-1 | PostgreSQL-1-1-1-1 | 1 | 160 | 8192 | 1 | 1 | 1 | 0 | 60.00 | 0 | 225.15 | 225.35 | 0.00 | 2749686.00 | 707141.00 |
| PostgreSQL-1-2-1-1-1 | PostgreSQL-1-2-1 | PostgreSQL-1-2-1-1 | 2 | 160 | 8192 | 1 | 1 | 1 | 0 | 60.00 | 0 | 548.00 | 545.47 | 0.00 | 1122286.00 | 290238.00 |
#### Per Phase
| DBMS | phase | experiment_run | terminals | target | benchmark_run | pod_count | tenant_id | time | num_errors | Throughput (requests/second) | Goodput (requests/second) | efficiency | Latency Distribution.95th Percentile Latency (microseconds) | Latency Distribution.Average Latency (microseconds) |
|:-----------------|:-----------------|-----------------:|------------:|---------:|----------------:|------------:|------------:|-------:|-------------:|-------------------------------:|----------------------------:|-------------:|--------------------------------------------------------------:|------------------------------------------------------:|
| PostgreSQL-1-1-1 | PostgreSQL-1-1-1 | 1 | 160 | 8192 | 1 | 1 | 0 | 60.00 | 0 | 225.15 | 225.35 | 0.00 | 2749686.00 | 707141.00 |
| PostgreSQL-1-2-1 | PostgreSQL-1-2-1 | 2 | 160 | 8192 | 1 | 1 | 0 | 60.00 | 0 | 548.00 | 545.47 | 0.00 | 1122286.00 | 290238.00 |
### Tests
* TEST passed: Throughput (requests/second) contains no 0 or NaN
* TEST passed: Workflow as planned
Benchbase Monitoring
bexhoma benchbase \
-dbms PostgreSQL \
-sf 16 \
-xsd 5 \
-xtb 1024 \
-nc 1 \
-ne 1 \
-nbp 1 \
-nbt 160 \
-xnbf 8 \
-m \
-mc \
-ms $BEXHOMA_MS \
-tr \
-lr 64Gi \
-rr 64Gi \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_benchbase_postgresql_3.log
yields (after ca. 10 minutes) something like
testcase_benchbase_postgresql_3.log
## Show Summary
### Workload
Benchbase Workload tpcc SF=16
* Type: benchbase
* Duration: 798s
* Code: 1782363495
* Benchbase runs a TPC-C experiment.
* This experiment compares run time and resource consumption of Benchbase queries in different DBMS.
* Benchbase data is generated and loaded using several threads.
* Benchmark is 'tpcc'. Scaling factor is 16. Target is based on multiples of '1024'. Factors for benchmarking are [8]. Benchmarking runs for 5 minutes.
* Experiment uses bexhoma version 0.10.0.
* System metrics are monitored by a cluster-wide installation.
* Experiment is limited to DBMS ['PostgreSQL'].
* Import is handled by 1 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker38.
* Loading is tested with [1] threads, split into [1] pods.
* Benchmarking is tested with [160] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* PostgreSQL-1-1-1-1 uses docker image postgres:18.3
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:261715
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782363495
* TENANT_VOL:False
### Workflow
#### Actual
* DBMS PostgreSQL-1 - Experiment 1 Client 1: benchbase (1 pods)
#### Planned
* DBMS PostgreSQL-1 - Experiment 1 Client 1: benchbase (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:---------------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| PostgreSQL-1-1 | 1 | 16 | 276.00 | 1.00 | 0.00 | 113.00 | 162.00 | 1 | 1 | | | 0 | False | 208.70 |
### Execution
#### Per Connection
| DBMS | phase | job | experiment_run | terminals | target | client | benchmark_run | child | tenant_id | time | num_errors | Throughput (requests/second) | Goodput (requests/second) | efficiency | Latency Distribution.95th Percentile Latency (microseconds) | Latency Distribution.Average Latency (microseconds) |
|:---------------------|:-----------------|:-------------------|-----------------:|------------:|---------:|---------:|----------------:|--------:|------------:|-------:|-------------:|-------------------------------:|----------------------------:|-------------:|--------------------------------------------------------------:|------------------------------------------------------:|
| PostgreSQL-1-1-1-1-1 | PostgreSQL-1-1-1 | PostgreSQL-1-1-1-1 | 1 | 160 | 8192 | 1 | 1 | 1 | 0 | 300.00 | 83 | 8172.76 | 8052.75 | 0.00 | 50599.00 | 15381.00 |
#### Per Phase
| DBMS | phase | experiment_run | terminals | target | benchmark_run | pod_count | tenant_id | time | num_errors | Throughput (requests/second) | Goodput (requests/second) | efficiency | Latency Distribution.95th Percentile Latency (microseconds) | Latency Distribution.Average Latency (microseconds) |
|:-----------------|:-----------------|-----------------:|------------:|---------:|----------------:|------------:|------------:|-------:|-------------:|-------------------------------:|----------------------------:|-------------:|--------------------------------------------------------------:|------------------------------------------------------:|
| PostgreSQL-1-1-1 | PostgreSQL-1-1-1 | 1 | 160 | 8192 | 1 | 1 | 0 | 300.00 | 83 | 8172.76 | 8052.75 | 0.00 | 50599.00 | 15381.00 |
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 329.45 | 5.91 | 2.05 | 3.69 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 1244.38 | 13.91 | 0.25 | 0.25 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 4782.46 | 17.72 | 5.83 | 9.53 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 3821.02 | 14.14 | 0.91 | 0.91 |
### Tests
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Throughput (requests/second) contains no 0 or NaN
* TEST passed: Workflow as planned
Benchbase Complex
bexhoma benchbase \
-dbms PostgreSQL \
-sf 16 \
-xsd 2 \
-xtb 1024 \
-nc 2 \
-ne 1,2 \
-nbp 1,2 \
-nbt 160 \
-xnbf 8 \
-m \
-mc \
-ms $BEXHOMA_MS \
-lr 64Gi \
-rr 64Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_benchbase_postgresql_4.log
yields (after ca. 30 minutes) something like
testcase_benchbase_postgresql_4.log
## Show Summary
### Workload
Benchbase Workload tpcc SF=16
* Type: benchbase
* Duration: 2471s
* Code: 1782364325
* Benchbase runs a TPC-C experiment.
* This experiment compares run time and resource consumption of Benchbase queries in different DBMS.
* Benchbase data is generated and loaded using several threads.
* Benchmark is 'tpcc'. Scaling factor is 16. Target is based on multiples of '1024'. Factors for benchmarking are [8]. Benchmarking runs for 2 minutes.
* Experiment uses bexhoma version 0.10.0.
* System metrics are monitored by a cluster-wide installation.
* Experiment is limited to DBMS ['PostgreSQL'].
* Import is handled by 1 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker38.
* Database is persisted to disk of type shared and size 30Gi.
* Loading is tested with [1] threads, split into [1] pods.
* Benchmarking is tested with [160] threads, split into [1, 2] pods.
* Benchmarking is run as [1, 2] times the number of benchmarking pods.
* Experiment is run 2 times.
### Connections
* PostgreSQL-1-1-1-1 uses docker image postgres:18.3
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:222623
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782364325
* TENANT_VOL:False
* PostgreSQL-1-1-2-1 uses docker image postgres:18.3
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:223817
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782364325
* TENANT_VOL:False
* PostgreSQL-1-1-3-1 uses docker image postgres:18.3
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220471
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782364325
* TENANT_VOL:False
* PostgreSQL-1-1-4-1 uses docker image postgres:18.3
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220471
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782364325
* TENANT_VOL:False
* PostgreSQL-1-2-1-1 uses docker image postgres:18.3
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220472
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782364325
* TENANT_VOL:False
* PostgreSQL-1-2-2-1 uses docker image postgres:18.3
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220472
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782364325
* TENANT_VOL:False
* PostgreSQL-1-2-3-1 uses docker image postgres:18.3
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220472
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782364325
* TENANT_VOL:False
* PostgreSQL-1-2-4-1 uses docker image postgres:18.3
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220472
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782364325
* TENANT_VOL:False
### Workflow
#### Actual
* DBMS PostgreSQL-1 - Experiment 1 Client 1: benchbase (1 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 2: benchbase (2 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 3: benchbase (2 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 4: benchbase (4 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 1: benchbase (1 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 2: benchbase (2 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 3: benchbase (2 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 4: benchbase (4 pods)
#### Planned
* DBMS PostgreSQL-1 - Experiment 1 Client 1: benchbase (1 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 2: benchbase (2 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 3: benchbase (2 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 4: benchbase (4 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 1: benchbase (1 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 2: benchbase (2 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 3: benchbase (2 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 4: benchbase (4 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:---------------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| PostgreSQL-1-1 | 1 | 16 | 749.00 | 1.00 | 0.00 | 348.00 | 400.00 | 0 | 1 | | | 0 | False | 76.90 |
| PostgreSQL-1-2 | 2 | 16 | 749.00 | 1.00 | 0.00 | 348.00 | 400.00 | 0 | 1 | | | 0 | False | 76.90 |
### Execution
#### Per Connection
| DBMS | phase | job | experiment_run | terminals | target | client | benchmark_run | child | tenant_id | time | num_errors | Throughput (requests/second) | Goodput (requests/second) | efficiency | Latency Distribution.95th Percentile Latency (microseconds) | Latency Distribution.Average Latency (microseconds) |
|:---------------------|:-----------------|:-------------------|-----------------:|------------:|---------:|---------:|----------------:|--------:|------------:|-------:|-------------:|-------------------------------:|----------------------------:|-------------:|--------------------------------------------------------------:|------------------------------------------------------:|
| PostgreSQL-1-1-1-1-1 | PostgreSQL-1-1-1 | PostgreSQL-1-1-1-1 | 1 | 160 | 8192 | 1 | 1 | 1 | 0 | 120.00 | 0 | 636.60 | 630.76 | 0.00 | 1090371.00 | 249829.00 |
| PostgreSQL-1-1-2-1-1 | PostgreSQL-1-1-2 | PostgreSQL-1-1-2-1 | 1 | 160 | 8192 | 2 | 1 | 1 | 0 | 120.00 | 1 | 477.71 | 473.88 | 0.00 | 1685708.00 | 332123.00 |
| PostgreSQL-1-1-2-1-2 | PostgreSQL-1-1-2 | PostgreSQL-1-1-2-1 | 1 | 160 | 8192 | 2 | 1 | 2 | 0 | 120.00 | 0 | 485.80 | 481.89 | 0.00 | 1652145.00 | 326588.00 |
| PostgreSQL-1-1-3-1-1 | PostgreSQL-1-1-3 | PostgreSQL-1-1-3-1 | 1 | 80 | 4096 | 3 | 1 | 1 | 0 | 120.00 | 0 | 505.12 | 500.57 | 0.00 | 641983.00 | 157003.00 |
| PostgreSQL-1-1-3-1-2 | PostgreSQL-1-1-3 | PostgreSQL-1-1-3-1 | 1 | 80 | 4096 | 3 | 1 | 2 | 0 | 120.00 | 0 | 494.12 | 489.75 | 0.00 | 658908.00 | 161178.00 |
| PostgreSQL-1-1-4-1-1 | PostgreSQL-1-1-4 | PostgreSQL-1-1-4-1 | 1 | 80 | 4096 | 4 | 1 | 1 | 0 | 120.00 | 1 | 204.44 | 203.08 | 0.00 | 1848759.00 | 386955.00 |
| PostgreSQL-1-1-4-1-2 | PostgreSQL-1-1-4 | PostgreSQL-1-1-4-1 | 1 | 80 | 4096 | 4 | 1 | 2 | 0 | 120.00 | 0 | 210.92 | 209.42 | 0.00 | 1797703.00 | 374969.00 |
| PostgreSQL-1-1-4-1-3 | PostgreSQL-1-1-4 | PostgreSQL-1-1-4-1 | 1 | 80 | 4096 | 4 | 1 | 3 | 0 | 120.00 | 0 | 208.47 | 207.16 | 0.00 | 1882198.00 | 380525.00 |
| PostgreSQL-1-1-4-1-4 | PostgreSQL-1-1-4 | PostgreSQL-1-1-4-1 | 1 | 80 | 4096 | 4 | 1 | 4 | 0 | 120.00 | 1 | 205.55 | 204.31 | 0.00 | 1848898.00 | 384630.00 |
| PostgreSQL-1-2-1-1-1 | PostgreSQL-1-2-1 | PostgreSQL-1-2-1-1 | 2 | 160 | 8192 | 1 | 1 | 1 | 0 | 120.00 | 0 | 489.02 | 485.34 | 0.00 | 1401511.00 | 326610.00 |
| PostgreSQL-1-2-2-1-1 | PostgreSQL-1-2-2 | PostgreSQL-1-2-2-1 | 2 | 160 | 8192 | 2 | 1 | 1 | 0 | 120.00 | 0 | 588.40 | 583.32 | 0.00 | 1262257.00 | 269739.00 |
| PostgreSQL-1-2-2-1-2 | PostgreSQL-1-2-2 | PostgreSQL-1-2-2-1 | 2 | 160 | 8192 | 2 | 1 | 2 | 0 | 120.00 | 0 | 580.57 | 575.71 | 0.00 | 1278093.00 | 272905.00 |
| PostgreSQL-1-2-3-1-1 | PostgreSQL-1-2-3 | PostgreSQL-1-2-3-1 | 2 | 80 | 4096 | 3 | 1 | 1 | 0 | 120.00 | 0 | 616.69 | 611.22 | 0.00 | 461185.00 | 129369.00 |
| PostgreSQL-1-2-3-1-2 | PostgreSQL-1-2-3 | PostgreSQL-1-2-3-1 | 2 | 80 | 4096 | 3 | 1 | 2 | 0 | 120.00 | 0 | 616.21 | 610.55 | 0.00 | 457243.00 | 129474.00 |
| PostgreSQL-1-2-4-1-1 | PostgreSQL-1-2-4 | PostgreSQL-1-2-4-1 | 2 | 80 | 4096 | 4 | 1 | 1 | 0 | 120.00 | 0 | 204.86 | 203.47 | 0.00 | 1865151.00 | 387985.00 |
| PostgreSQL-1-2-4-1-2 | PostgreSQL-1-2-4 | PostgreSQL-1-2-4-1 | 2 | 80 | 4096 | 4 | 1 | 2 | 0 | 120.00 | 0 | 209.17 | 207.83 | 0.00 | 1812033.00 | 381303.00 |
| PostgreSQL-1-2-4-1-3 | PostgreSQL-1-2-4 | PostgreSQL-1-2-4-1 | 2 | 80 | 4096 | 4 | 1 | 3 | 0 | 120.00 | 0 | 205.87 | 204.22 | 0.00 | 1825412.00 | 385903.00 |
| PostgreSQL-1-2-4-1-4 | PostgreSQL-1-2-4 | PostgreSQL-1-2-4-1 | 2 | 80 | 4096 | 4 | 1 | 4 | 0 | 120.00 | 0 | 212.42 | 211.01 | 0.00 | 1775657.00 | 374683.00 |
#### Per Phase
| DBMS | phase | experiment_run | terminals | target | benchmark_run | pod_count | tenant_id | time | num_errors | Throughput (requests/second) | Goodput (requests/second) | efficiency | Latency Distribution.95th Percentile Latency (microseconds) | Latency Distribution.Average Latency (microseconds) |
|:-----------------|:-----------------|-----------------:|------------:|---------:|----------------:|------------:|------------:|-------:|-------------:|-------------------------------:|----------------------------:|-------------:|--------------------------------------------------------------:|------------------------------------------------------:|
| PostgreSQL-1-1-1 | PostgreSQL-1-1-1 | 1 | 160 | 8192 | 1 | 1 | 0 | 120.00 | 0 | 636.60 | 630.76 | 0.00 | 1090371.00 | 249829.00 |
| PostgreSQL-1-1-2 | PostgreSQL-1-1-2 | 1 | 320 | 16384 | 1 | 2 | 0 | 120.00 | 1 | 963.51 | 955.77 | 0.00 | 1685708.00 | 329355.50 |
| PostgreSQL-1-1-3 | PostgreSQL-1-1-3 | 1 | 160 | 8192 | 1 | 2 | 0 | 120.00 | 0 | 999.23 | 990.32 | 0.00 | 658908.00 | 159090.50 |
| PostgreSQL-1-1-4 | PostgreSQL-1-1-4 | 1 | 320 | 16384 | 1 | 4 | 0 | 120.00 | 2 | 829.38 | 823.97 | 0.00 | 1882198.00 | 381769.75 |
| PostgreSQL-1-2-1 | PostgreSQL-1-2-1 | 2 | 160 | 8192 | 1 | 1 | 0 | 120.00 | 0 | 489.02 | 485.34 | 0.00 | 1401511.00 | 326610.00 |
| PostgreSQL-1-2-2 | PostgreSQL-1-2-2 | 2 | 320 | 16384 | 1 | 2 | 0 | 120.00 | 0 | 1168.97 | 1159.03 | 0.00 | 1278093.00 | 271322.00 |
| PostgreSQL-1-2-3 | PostgreSQL-1-2-3 | 2 | 160 | 8192 | 1 | 2 | 0 | 120.00 | 0 | 1232.90 | 1221.77 | 0.00 | 461185.00 | 129421.50 |
| PostgreSQL-1-2-4 | PostgreSQL-1-2-4 | 2 | 320 | 16384 | 1 | 4 | 0 | 120.00 | 0 | 832.32 | 826.52 | 0.00 | 1865151.00 | 382468.50 |
### Monitoring
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 236.54 | 2.99 | 2.46 | 3.76 |
| PostgreSQL-1-1-2-1 | 564.62 | 5.99 | 3.89 | 5.51 |
| PostgreSQL-1-1-3-1 | 516.64 | 5.85 | 3.32 | 5.23 |
| PostgreSQL-1-1-4-1 | 551.32 | 6.39 | 4.54 | 6.64 |
| PostgreSQL-1-2-1-1 | 2029.18 | 3.83 | 2.47 | 4.59 |
| PostgreSQL-1-2-2-1 | 928.13 | 8.71 | 3.88 | 5.48 |
| PostgreSQL-1-2-3-1 | 782.87 | 7.32 | 3.40 | 5.33 |
| PostgreSQL-1-2-4-1 | 579.57 | 6.13 | 4.51 | 6.64 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 107.31 | 1.48 | 0.65 | 0.65 |
| PostgreSQL-1-1-2-1 | 164.51 | 2.76 | 0.70 | 0.70 |
| PostgreSQL-1-1-3-1 | 150.72 | 2.63 | 0.70 | 0.70 |
| PostgreSQL-1-1-4-1 | 175.11 | 2.44 | 0.41 | 0.41 |
| PostgreSQL-1-2-1-1 | 80.68 | 1.15 | 0.65 | 0.65 |
| PostgreSQL-1-2-2-1 | 159.50 | 3.15 | 0.68 | 0.68 |
| PostgreSQL-1-2-3-1 | 177.14 | 3.20 | 0.68 | 0.68 |
| PostgreSQL-1-2-4-1 | 156.33 | 3.24 | 0.41 | 0.41 |
### Tests
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Throughput (requests/second) contains no 0 or NaN
* TEST passed: Workflow as planned
MySQL
Benchbase Simple
bexhoma benchbase \
-dbms MySQL \
-sf 16 \
-xsd 5 \
-xtb 1024 \
-nc 1 \
-ne 1 \
-nbp 1 \
-nbt 160 \
-xnbf 8 \
-ms $BEXHOMA_MS \
-tr \
-lr 64Gi \
-rr 64Gi \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_benchbase_mysql_1.log
yields (after ca. 10 minutes) something like
testcase_benchbase_mysql_1.log
## Show Summary
### Workload
Benchbase Workload tpcc SF=16
* Type: benchbase
* Duration: 945s
* Code: 1782361781
* Benchbase runs a TPC-C experiment.
* This experiment compares run time and resource consumption of Benchbase queries in different DBMS.
* Benchbase data is generated and loaded using several threads.
* Benchmark is 'tpcc'. Scaling factor is 16. Target is based on multiples of '1024'. Factors for benchmarking are [8]. Benchmarking runs for 5 minutes.
* Experiment uses bexhoma version 0.10.0.
* Experiment is limited to DBMS ['MySQL'].
* Import is handled by 1 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker38.
* Loading is tested with [1] threads, split into [1] pods.
* Benchmarking is tested with [160] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* MySQL-1-1-1-1 uses docker image mysql:8.4.0
* RAM:540492877824
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:267301
* cpu_list:0-127
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782361781
* TENANT_VOL:False
### Workflow
#### Actual
* DBMS MySQL-1 - Experiment 1 Client 1: benchbase (1 pods)
#### Planned
* DBMS MySQL-1 - Experiment 1 Client 1: benchbase (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:----------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| MySQL-1-1 | 1 | 16 | 340.00 | 2.00 | 0.00 | 148.00 | 190.00 | 1 | 1 | | | 0 | False | 169.41 |
### Execution
#### Per Connection
| DBMS | phase | job | experiment_run | terminals | target | client | benchmark_run | child | tenant_id | time | num_errors | Throughput (requests/second) | Goodput (requests/second) | efficiency | Latency Distribution.95th Percentile Latency (microseconds) | Latency Distribution.Average Latency (microseconds) |
|:----------------|:------------|:--------------|-----------------:|------------:|---------:|---------:|----------------:|--------:|------------:|-------:|-------------:|-------------------------------:|----------------------------:|-------------:|--------------------------------------------------------------:|------------------------------------------------------:|
| MySQL-1-1-1-1-1 | MySQL-1-1-1 | MySQL-1-1-1-1 | 1 | 160 | 8192 | 1 | 1 | 1 | -1 | 300.00 | 0 | 5342.18 | 5259.56 | 0.00 | 88779.00 | 28617.00 |
#### Per Phase
| DBMS | phase | experiment_run | terminals | target | benchmark_run | pod_count | tenant_id | time | num_errors | Throughput (requests/second) | Goodput (requests/second) | efficiency | Latency Distribution.95th Percentile Latency (microseconds) | Latency Distribution.Average Latency (microseconds) |
|:------------|:------------|-----------------:|------------:|---------:|----------------:|------------:|------------:|-------:|-------------:|-------------------------------:|----------------------------:|-------------:|--------------------------------------------------------------:|------------------------------------------------------:|
| MySQL-1-1-1 | MySQL-1-1-1 | 1 | 160 | 8192 | 1 | 1 | -1 | 300.00 | 0 | 5342.18 | 5259.56 | 0.00 | 88779.00 | 28617.00 |
### Tests
* TEST passed: Throughput (requests/second) contains no 0 or NaN
* TEST passed: Workflow as planned
Benchbase Persistency
bexhoma benchbase \
-dbms MySQL \
-sf 16 \
-xsd 1 \
-xtb 1024 \
-nc 2 \
-ne 1 \
-nbp 1 \
-nbt 160 \
-xnbf 8 \
-ms $BEXHOMA_MS \
-lr 64Gi \
-rr 64Gi \
-rsr \
-rst $BEXHOMA_STORAGE_CLASS \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_benchbase_mysql_2.log
yields (after ca. 10 minutes) something like
testcase_benchbase_mysql_2.log
## Show Summary
### Workload
Benchbase Workload SF=16
Type: benchbase
Duration: 8930s
Code: 1748934307
Benchbase runs the TPC-C benchmark.
This experiment compares run time and resource consumption of Benchbase queries in different DBMS.
Benchbase data is generated and loaded using several threads.
Benchmark is 'tpcc'. Scaling factor is 16. Benchmarking runs for 1 minutes. Target is based on multiples of '1024'. Factors for benchmarking are [8].
Experiment uses bexhoma version 0.8.7.
Benchmark is limited to DBMS ['MySQL'].
Import is handled by 1 processes (pods).
Loading is fixed to cl-worker19.
Benchmarking is fixed to cl-worker19.
SUT is fixed to cl-worker11.
Database is persisted to disk of type shared and size 30Gi.
Loading is tested with [1] threads, split into [1] pods.
Benchmarking is tested with [16] threads, split into [1] pods.
Benchmarking is run as [1] times the number of benchmarking pods.
Experiment is run 2 times.
### Connections
MySQL-1-1-1024-1-1 uses docker image mysql:8.4.0
RAM:541008568320
Cores:64
host:5.15.0-140-generic
node:cl-worker11
disk:317590004
datadisk:11132
volume_size:30G
volume_used:11G
requests_cpu:4
requests_memory:16Gi
client:1
numExperiment:1
eval_parameters
code:1748934307
MySQL-1-1-1024-2-1 uses docker image mysql:8.4.0
RAM:541008568320
Cores:64
host:5.15.0-140-generic
node:cl-worker11
disk:317590184
datadisk:11165
volume_size:30G
volume_used:11G
requests_cpu:4
requests_memory:16Gi
client:1
numExperiment:2
eval_parameters
code:1748934307
### Execution
experiment_run terminals target pod_count time num_errors Throughput (requests/second) Goodput (requests/second) efficiency Latency Distribution.95th Percentile Latency (microseconds) Latency Distribution.Average Latency (microseconds)
MySQL-1-1-1024-1-1 1 16 8192 1 60.0 0 1.00 1.23 0.0 47671586.0 8847058.0
MySQL-1-1-1024-2-1 2 16 8192 1 60.0 0 4.52 4.78 0.0 15558733.0 3001960.0
### Workflow
#### Actual
DBMS MySQL-1-1-1024 - Pods [[1], [1]]
#### Planned
DBMS MySQL-1-1-1024 - Pods [[1], [1]]
### Loading
time_load terminals pods Throughput [SF/h]
MySQL-1-1-1024-1-1 7641.0 1.0 1.0 7.53828
MySQL-1-1-1024-2-1 7641.0 1.0 1.0 7.53828
### Tests
TEST passed: Throughput (requests/second) contains no 0 or NaN
TEST passed: Workflow as planned
Benchbase Monitoring
bexhoma benchbase \
-dbms MySQL \
-sf 16 \
-xsd 5 \
-xtb 1024 \
-nc 1 \
-ne 1 \
-nbp 1 \
-nbt 160 \
-xnbf 8 \
-m \
-mc \
-ms $BEXHOMA_MS \
-tr \
-lr 64Gi \
-rr 64Gi \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_benchbase_mysql_3.log
yields (after ca. 10 minutes) something like
testcase_benchbase_mysql_3.log
## Show Summary
### Workload
Benchbase Workload tpcc SF=16
* Type: benchbase
* Duration: 977s
* Code: 1782363268
* Benchbase runs a TPC-C experiment.
* This experiment compares run time and resource consumption of Benchbase queries in different DBMS.
* Benchbase data is generated and loaded using several threads.
* Benchmark is 'tpcc'. Scaling factor is 16. Target is based on multiples of '1024'. Factors for benchmarking are [8]. Benchmarking runs for 5 minutes.
* Experiment uses bexhoma version 0.10.0.
* System metrics are monitored by a cluster-wide installation.
* Experiment is limited to DBMS ['MySQL'].
* Import is handled by 1 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker38.
* Loading is tested with [1] threads, split into [1] pods.
* Benchmarking is tested with [160] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* MySQL-1-1-1-1 uses docker image mysql:8.4.0
* RAM:540492877824
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:260482
* cpu_list:0-127
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782363268
* TENANT_VOL:False
### Workflow
#### Actual
* DBMS MySQL-1 - Experiment 1 Client 1: benchbase (1 pods)
#### Planned
* DBMS MySQL-1 - Experiment 1 Client 1: benchbase (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:----------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| MySQL-1-1 | 1 | 16 | 355.00 | 2.00 | 0.00 | 159.00 | 194.00 | 1 | 1 | | | 0 | False | 162.25 |
### Execution
#### Per Connection
| DBMS | phase | job | experiment_run | terminals | target | client | benchmark_run | child | tenant_id | time | num_errors | Throughput (requests/second) | Goodput (requests/second) | efficiency | Latency Distribution.95th Percentile Latency (microseconds) | Latency Distribution.Average Latency (microseconds) |
|:----------------|:------------|:--------------|-----------------:|------------:|---------:|---------:|----------------:|--------:|------------:|-------:|-------------:|-------------------------------:|----------------------------:|-------------:|--------------------------------------------------------------:|------------------------------------------------------:|
| MySQL-1-1-1-1-1 | MySQL-1-1-1 | MySQL-1-1-1-1 | 1 | 160 | 8192 | 1 | 1 | 1 | -1 | 300.00 | 0 | 5472.97 | 5384.40 | 0.00 | 81681.00 | 29194.00 |
#### Per Phase
| DBMS | phase | experiment_run | terminals | target | benchmark_run | pod_count | tenant_id | time | num_errors | Throughput (requests/second) | Goodput (requests/second) | efficiency | Latency Distribution.95th Percentile Latency (microseconds) | Latency Distribution.Average Latency (microseconds) |
|:------------|:------------|-----------------:|------------:|---------:|----------------:|------------:|------------:|-------:|-------------:|-------------------------------:|----------------------------:|-------------:|--------------------------------------------------------------:|------------------------------------------------------:|
| MySQL-1-1-1 | MySQL-1-1-1 | 1 | 160 | 8192 | 1 | 1 | -1 | 300.00 | 0 | 5472.97 | 5384.40 | 0.00 | 81681.00 | 29194.00 |
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 414.07 | 7.90 | 9.58 | 12.83 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 1724.24 | 14.72 | 0.55 | 0.55 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 6946.42 | 26.99 | 11.03 | 16.00 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 5191.00 | 21.74 | 1.36 | 1.36 |
### Tests
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Throughput (requests/second) contains no 0 or NaN
* TEST passed: Workflow as planned
Benchbase Complex
bexhoma benchbase \
-dbms MySQL \
-sf 16 \
-xsd 2 \
-xtb 1024 \
-nc 2 \
-ne 1,2 \
-nbp 1,2 \
-nbt 160 \
-xnbf 8 \
-m \
-mc \
-ms $BEXHOMA_MS \
-lr 64Gi \
-rr 64Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_benchbase_mysql_4.log
yields (after ca. 30 minutes) something like
testcase_benchbase_mysql_4.log
## Show Summary
### Workload
Benchbase Workload SF=16
Type: benchbase
Duration: 2183s
Code: 1748944810
Benchbase runs the TPC-C benchmark.
This experiment compares run time and resource consumption of Benchbase queries in different DBMS.
Benchbase data is generated and loaded using several threads.
Benchmark is 'tpcc'. Scaling factor is 16. Benchmarking runs for 2 minutes. Target is based on multiples of '1024'. Factors for benchmarking are [8].
Experiment uses bexhoma version 0.8.7.
System metrics are monitored by a cluster-wide installation.
Benchmark is limited to DBMS ['MySQL'].
Import is handled by 1 processes (pods).
Loading is fixed to cl-worker19.
Benchmarking is fixed to cl-worker19.
SUT is fixed to cl-worker11.
Database is persisted to disk of type shared and size 30Gi.
Loading is tested with [1] threads, split into [1] pods.
Benchmarking is tested with [8] threads, split into [1, 2] pods.
Benchmarking is run as [1, 2] times the number of benchmarking pods.
Experiment is run 2 times.
### Connections
MySQL-1-1-1024-1-1 uses docker image mysql:8.4.0
RAM:541008568320
Cores:64
host:5.15.0-140-generic
node:cl-worker11
disk:317590204
datadisk:11187
volume_size:30G
volume_used:11G
requests_cpu:4
requests_memory:16Gi
client:1
numExperiment:1
eval_parameters
code:1748944810
MySQL-1-1-1024-1-2 uses docker image mysql:8.4.0
RAM:541008568320
Cores:64
host:5.15.0-140-generic
node:cl-worker11
disk:317590204
datadisk:11220
volume_size:30G
volume_used:11G
requests_cpu:4
requests_memory:16Gi
client:2
numExperiment:1
eval_parameters
code:1748944810
MySQL-1-1-1024-1-3 uses docker image mysql:8.4.0
RAM:541008568320
Cores:64
host:5.15.0-140-generic
node:cl-worker11
disk:317590208
datadisk:11256
volume_size:30G
volume_used:11G
requests_cpu:4
requests_memory:16Gi
client:3
numExperiment:1
eval_parameters
code:1748944810
MySQL-1-1-1024-1-4 uses docker image mysql:8.4.0
RAM:541008568320
Cores:64
host:5.15.0-140-generic
node:cl-worker11
disk:317590208
datadisk:11308
volume_size:30G
volume_used:11G
requests_cpu:4
requests_memory:16Gi
client:4
numExperiment:1
eval_parameters
code:1748944810
MySQL-1-1-1024-2-1 uses docker image mysql:8.4.0
RAM:541008568320
Cores:64
host:5.15.0-140-generic
node:cl-worker11
disk:317590216
datadisk:11339
volume_size:30G
volume_used:12G
requests_cpu:4
requests_memory:16Gi
client:1
numExperiment:2
eval_parameters
code:1748944810
MySQL-1-1-1024-2-2 uses docker image mysql:8.4.0
RAM:541008568320
Cores:64
host:5.15.0-140-generic
node:cl-worker11
disk:317590216
datadisk:11374
volume_size:30G
volume_used:12G
requests_cpu:4
requests_memory:16Gi
client:2
numExperiment:2
eval_parameters
code:1748944810
MySQL-1-1-1024-2-3 uses docker image mysql:8.4.0
RAM:541008568320
Cores:64
host:5.15.0-140-generic
node:cl-worker11
disk:317590220
datadisk:11458
volume_size:30G
volume_used:12G
requests_cpu:4
requests_memory:16Gi
client:3
numExperiment:2
eval_parameters
code:1748944810
MySQL-1-1-1024-2-4 uses docker image mysql:8.4.0
RAM:541008568320
Cores:64
host:5.15.0-140-generic
node:cl-worker11
disk:317590396
datadisk:11489
volume_size:30G
volume_used:12G
requests_cpu:4
requests_memory:16Gi
client:4
numExperiment:2
eval_parameters
code:1748944810
### Execution
experiment_run terminals target pod_count time num_errors Throughput (requests/second) Goodput (requests/second) efficiency Latency Distribution.95th Percentile Latency (microseconds) Latency Distribution.Average Latency (microseconds)
MySQL-1-1-1024-1-1 1 8 8192 1 120.0 0 36.02 35.85 0.0 610937.0 220741.00
MySQL-1-1-1024-1-2 1 16 16384 2 120.0 0 39.64 39.47 0.0 820427.0 285537.50
MySQL-1-1-1024-1-3 1 8 8192 2 120.0 0 69.92 69.46 0.0 394377.0 114272.50
MySQL-1-1-1024-1-4 1 16 16384 4 120.0 0 37.22 36.94 0.0 1876083.0 429714.25
MySQL-1-1-1024-2-1 2 8 8192 1 120.0 0 45.04 44.85 0.0 500014.0 177493.00
MySQL-1-1-1024-2-2 2 16 16384 2 120.0 0 106.22 105.62 0.0 483418.0 150073.00
MySQL-1-1-1024-2-3 2 8 8192 2 120.0 0 39.84 39.67 0.0 952677.0 200600.50
MySQL-1-1-1024-2-4 2 16 16384 4 120.0 0 49.48 49.02 0.0 940977.0 213649.25
### Workflow
#### Actual
DBMS MySQL-1-1-1024 - Pods [[1, 2, 2, 4], [4, 2, 2, 1]]
#### Planned
DBMS MySQL-1-1-1024 - Pods [[1, 2, 2, 4], [1, 2, 2, 4]]
### Loading
time_load terminals pods Throughput [SF/h]
MySQL-1-1-1024-1-1 7641.0 1.0 1.0 7.53828
MySQL-1-1-1024-1-2 7641.0 1.0 2.0 7.53828
MySQL-1-1-1024-1-3 7641.0 1.0 2.0 7.53828
MySQL-1-1-1024-1-4 7641.0 1.0 4.0 7.53828
MySQL-1-1-1024-2-1 7641.0 1.0 1.0 7.53828
MySQL-1-1-1024-2-2 7641.0 1.0 2.0 7.53828
MySQL-1-1-1024-2-3 7641.0 1.0 2.0 7.53828
MySQL-1-1-1024-2-4 7641.0 1.0 4.0 7.53828
### Execution - SUT
CPU [CPUs] Max CPU Max RAM [Gb] Max RAM Cached [Gb]
MySQL-1-1-1024-1-1 50.42 0.79 37.96 45.81
MySQL-1-1-1024-1-2 78.16 0.28 38.17 46.08
MySQL-1-1-1024-1-3 86.48 0.10 38.23 46.20
MySQL-1-1-1024-1-4 49.59 0.73 38.27 46.26
MySQL-1-1-1024-2-1 51.82 0.00 37.95 45.98
MySQL-1-1-1024-2-2 112.04 1.48 38.17 46.30
MySQL-1-1-1024-2-3 100.06 0.47 38.21 46.40
MySQL-1-1-1024-2-4 123.16 1.35 38.26 46.51
### Execution - Benchmarker
CPU [CPUs] Max CPU Max RAM [Gb] Max RAM Cached [Gb]
MySQL-1-1-1024-1-1 33.15 0.00 0.28 0.28
MySQL-1-1-1024-1-2 61.65 0.00 0.80 0.80
MySQL-1-1-1024-1-3 33.28 0.00 1.19 1.19
MySQL-1-1-1024-1-4 38.84 0.08 1.45 1.45
MySQL-1-1-1024-2-1 35.98 0.00 0.30 0.30
MySQL-1-1-1024-2-2 87.11 0.37 0.97 0.97
MySQL-1-1-1024-2-3 35.98 0.07 1.18 1.18
MySQL-1-1-1024-2-4 117.93 0.51 4.46 4.47
### Tests
TEST passed: Throughput (requests/second) contains no 0 or NaN
TEST passed: Execution SUT contains no 0 or NaN in CPU [CPUs]
TEST passed: Execution Benchmarker contains no 0 or NaN in CPU [CPUs]
TEST passed: Workflow as planned
MariaDB
Benchbase Simple
bexhoma benchbase \
-dbms MariaDB \
-sf 16 \
-xsd 5 \
-xtb 1024 \
-nc 1 \
-ne 1 \
-nbp 1 \
-nbt 160 \
-xnbf 8 \
-ms $BEXHOMA_MS \
-tr \
-lr 64Gi \
-rr 64Gi \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_benchbase_mariadb_1.log
yields (after ca. 10 minutes) something like
testcase_benchbase_mariadb_1.log
## Show Summary
### Workload
Benchbase Workload tpcc SF=16
* Type: benchbase
* Duration: 835s
* Code: 1782361791
* Benchbase runs a TPC-C experiment.
* This experiment compares run time and resource consumption of Benchbase queries in different DBMS.
* Benchbase data is generated and loaded using several threads.
* Benchmark is 'tpcc'. Scaling factor is 16. Target is based on multiples of '1024'. Factors for benchmarking are [8]. Benchmarking runs for 5 minutes.
* Experiment uses bexhoma version 0.10.0.
* Experiment is limited to DBMS ['MariaDB'].
* Import is handled by 1 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker38.
* Loading is tested with [1] threads, split into [1] pods.
* Benchmarking is tested with [160] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* MariaDB-1-1-1-1 uses docker image mariadb:11.4.7
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:261844
* cpu_list:0-127
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782361791
### Workflow
#### Actual
* DBMS MariaDB-1 - Experiment 1 Client 1: benchbase (1 pods)
#### Planned
* DBMS MariaDB-1 - Experiment 1 Client 1: benchbase (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:------------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| MariaDB-1-1 | 1 | 16 | 265.00 | 2.00 | 0.00 | 114.00 | 149.00 | 1 | 1 | | None | 0 | False | 217.36 |
### Execution
#### Per Connection
| DBMS | phase | job | experiment_run | terminals | target | client | benchmark_run | child | tenant_id | time | num_errors | Throughput (requests/second) | Goodput (requests/second) | efficiency | Latency Distribution.95th Percentile Latency (microseconds) | Latency Distribution.Average Latency (microseconds) |
|:------------------|:--------------|:----------------|-----------------:|------------:|---------:|---------:|----------------:|--------:|------------:|-------:|-------------:|-------------------------------:|----------------------------:|-------------:|--------------------------------------------------------------:|------------------------------------------------------:|
| MariaDB-1-1-1-1-1 | MariaDB-1-1-1 | MariaDB-1-1-1-1 | 1 | 160 | 8192 | 1 | 1 | 1 | -1 | 300.00 | 0 | 6489.12 | 6382.97 | 0.00 | 68949.00 | 21665.00 |
#### Per Phase
| DBMS | phase | experiment_run | terminals | target | benchmark_run | pod_count | tenant_id | time | num_errors | Throughput (requests/second) | Goodput (requests/second) | efficiency | Latency Distribution.95th Percentile Latency (microseconds) | Latency Distribution.Average Latency (microseconds) |
|:--------------|:--------------|-----------------:|------------:|---------:|----------------:|------------:|------------:|-------:|-------------:|-------------------------------:|----------------------------:|-------------:|--------------------------------------------------------------:|------------------------------------------------------:|
| MariaDB-1-1-1 | MariaDB-1-1-1 | 1 | 160 | 8192 | 1 | 1 | -1 | 300.00 | 0 | 6489.12 | 6382.97 | 0.00 | 68949.00 | 21665.00 |
### Tests
* TEST passed: Throughput (requests/second) contains no 0 or NaN
* TEST passed: Workflow as planned
Benchbase Persistency
bexhoma benchbase \
-dbms MariaDB \
-sf 16 \
-xsd 1 \
-xtb 1024 \
-nc 2 \
-ne 1 \
-nbp 1 \
-nbt 160 \
-xnbf 8 \
-ms $BEXHOMA_MS \
-lr 64Gi \
-rr 64Gi \
-rsr \
-rst $BEXHOMA_STORAGE_CLASS \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_benchbase_mariadb_2.log
yields (after ca. 10 minutes) something like
testcase_benchbase_mariadb_2.log
## Show Summary
### Workload
Benchbase Workload tpcc SF=16
* Type: benchbase
* Duration: 1331s
* Code: 1782362666
* Benchbase runs a TPC-C experiment.
* This experiment compares run time and resource consumption of Benchbase queries in different DBMS.
* Benchbase data is generated and loaded using several threads.
* Benchmark is 'tpcc'. Scaling factor is 16. Target is based on multiples of '1024'. Factors for benchmarking are [8]. Benchmarking runs for 1 minutes.
* Experiment uses bexhoma version 0.10.0.
* Experiment is limited to DBMS ['MariaDB'].
* Import is handled by 1 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker38.
* Database is persisted to disk of type shared and size 30Gi.
* Loading is tested with [1] threads, split into [1] pods.
* Benchmarking is tested with [160] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run 2 times.
### Connections
* MariaDB-1-1-1-1 uses docker image mariadb:11.4.7
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:253332
* volume_size:30G
* volume_used:1.9G
* cpu_list:0-127
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782362666
* MariaDB-1-2-1-1 uses docker image mariadb:11.4.7
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:260794
* volume_size:30G
* volume_used:2.0G
* cpu_list:0-127
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782362666
### Workflow
#### Actual
* DBMS MariaDB-1 - Experiment 1 Client 1: benchbase (1 pods)
* DBMS MariaDB-1 - Experiment 2 Client 1: benchbase (1 pods)
#### Planned
* DBMS MariaDB-1 - Experiment 1 Client 1: benchbase (1 pods)
* DBMS MariaDB-1 - Experiment 2 Client 1: benchbase (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:------------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| MariaDB-1-1 | 1 | 16 | 761.00 | 2.00 | 0.00 | 356.00 | 403.00 | 1 | 1 | | None | 0 | False | 75.69 |
| MariaDB-1-2 | 2 | 16 | 761.00 | 2.00 | 0.00 | 356.00 | 403.00 | 1 | 1 | | None | 0 | False | 75.69 |
### Execution
#### Per Connection
| DBMS | phase | job | experiment_run | terminals | target | client | benchmark_run | child | tenant_id | time | num_errors | Throughput (requests/second) | Goodput (requests/second) | efficiency | Latency Distribution.95th Percentile Latency (microseconds) | Latency Distribution.Average Latency (microseconds) |
|:------------------|:--------------|:----------------|-----------------:|------------:|---------:|---------:|----------------:|--------:|------------:|-------:|-------------:|-------------------------------:|----------------------------:|-------------:|--------------------------------------------------------------:|------------------------------------------------------:|
| MariaDB-1-1-1-1-1 | MariaDB-1-1-1 | MariaDB-1-1-1-1 | 1 | 160 | 8192 | 1 | 1 | 1 | -1 | 60.00 | 0 | 1143.50 | 1127.18 | 0.00 | 542141.00 | 137600.00 |
| MariaDB-1-2-1-1-1 | MariaDB-1-2-1 | MariaDB-1-2-1-1 | 2 | 160 | 8192 | 1 | 1 | 1 | -1 | 60.00 | 0 | 1116.67 | 1101.28 | 0.00 | 250549.00 | 130235.00 |
#### Per Phase
| DBMS | phase | experiment_run | terminals | target | benchmark_run | pod_count | tenant_id | time | num_errors | Throughput (requests/second) | Goodput (requests/second) | efficiency | Latency Distribution.95th Percentile Latency (microseconds) | Latency Distribution.Average Latency (microseconds) |
|:--------------|:--------------|-----------------:|------------:|---------:|----------------:|------------:|------------:|-------:|-------------:|-------------------------------:|----------------------------:|-------------:|--------------------------------------------------------------:|------------------------------------------------------:|
| MariaDB-1-1-1 | MariaDB-1-1-1 | 1 | 160 | 8192 | 1 | 1 | -1 | 60.00 | 0 | 1143.50 | 1127.18 | 0.00 | 542141.00 | 137600.00 |
| MariaDB-1-2-1 | MariaDB-1-2-1 | 2 | 160 | 8192 | 1 | 1 | -1 | 60.00 | 0 | 1116.67 | 1101.28 | 0.00 | 250549.00 | 130235.00 |
### Tests
* TEST passed: Throughput (requests/second) contains no 0 or NaN
* TEST passed: Workflow as planned
Benchbase Monitoring
bexhoma benchbase \
-dbms MariaDB \
-sf 16 \
-xsd 5 \
-xtb 1024 \
-nc 1 \
-ne 1 \
-nbp 1 \
-nbt 160 \
-xnbf 8 \
-m \
-mc \
-ms $BEXHOMA_MS \
-tr \
-lr 64Gi \
-rr 64Gi \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_benchbase_mariadb_3.log
yields (after ca. 10 minutes) something like
testcase_benchbase_mariadb_3.log
## Show Summary
### Workload
Benchbase Workload tpcc SF=16
* Type: benchbase
* Duration: 893s
* Code: 1782364006
* Benchbase runs a TPC-C experiment.
* This experiment compares run time and resource consumption of Benchbase queries in different DBMS.
* Benchbase data is generated and loaded using several threads.
* Benchmark is 'tpcc'. Scaling factor is 16. Target is based on multiples of '1024'. Factors for benchmarking are [8]. Benchmarking runs for 5 minutes.
* Experiment uses bexhoma version 0.10.0.
* System metrics are monitored by a cluster-wide installation.
* Experiment is limited to DBMS ['MariaDB'].
* Import is handled by 1 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker38.
* Loading is tested with [1] threads, split into [1] pods.
* Benchmarking is tested with [160] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* MariaDB-1-1-1-1 uses docker image mariadb:11.4.7
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:222370
* cpu_list:0-127
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782364006
### Workflow
#### Actual
* DBMS MariaDB-1 - Experiment 1 Client 1: benchbase (1 pods)
#### Planned
* DBMS MariaDB-1 - Experiment 1 Client 1: benchbase (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:------------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| MariaDB-1-1 | 1 | 16 | 332.00 | 2.00 | 0.00 | 142.00 | 188.00 | 1 | 1 | | None | 0 | False | 173.49 |
### Execution
#### Per Connection
| DBMS | phase | job | experiment_run | terminals | target | client | benchmark_run | child | tenant_id | time | num_errors | Throughput (requests/second) | Goodput (requests/second) | efficiency | Latency Distribution.95th Percentile Latency (microseconds) | Latency Distribution.Average Latency (microseconds) |
|:------------------|:--------------|:----------------|-----------------:|------------:|---------:|---------:|----------------:|--------:|------------:|-------:|-------------:|-------------------------------:|----------------------------:|-------------:|--------------------------------------------------------------:|------------------------------------------------------:|
| MariaDB-1-1-1-1-1 | MariaDB-1-1-1 | MariaDB-1-1-1-1 | 1 | 160 | 8192 | 1 | 1 | 1 | -1 | 300.00 | 0 | 8191.94 | 8102.43 | 0.00 | 10659.00 | 5096.00 |
#### Per Phase
| DBMS | phase | experiment_run | terminals | target | benchmark_run | pod_count | tenant_id | time | num_errors | Throughput (requests/second) | Goodput (requests/second) | efficiency | Latency Distribution.95th Percentile Latency (microseconds) | Latency Distribution.Average Latency (microseconds) |
|:--------------|:--------------|-----------------:|------------:|---------:|----------------:|------------:|------------:|-------:|-------------:|-------------------------------:|----------------------------:|-------------:|--------------------------------------------------------------:|------------------------------------------------------:|
| MariaDB-1-1-1 | MariaDB-1-1-1 | 1 | 160 | 8192 | 1 | 1 | -1 | 300.00 | 0 | 8191.94 | 8102.43 | 0.00 | 10659.00 | 5096.00 |
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1-1 | 267.20 | 3.31 | 3.79 | 3.89 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1-1 | 1835.23 | 15.32 | 0.24 | 0.24 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1-1 | 5273.25 | 18.16 | 5.49 | 5.60 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1-1 | 4183.33 | 14.61 | 1.67 | 1.67 |
### Tests
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Throughput (requests/second) contains no 0 or NaN
* TEST passed: Workflow as planned
Benchbase Complex
bexhoma benchbase \
-dbms MariaDB \
-sf 16 \
-xsd 2 \
-xtb 1024 \
-nc 2 \
-ne 1,2 \
-nbp 1,2 \
-nbt 160 \
-xnbf 8 \
-m \
-mc \
-ms $BEXHOMA_MS \
-lr 64Gi \
-rr 64Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_benchbase_mariadb_4.log
yields (after ca. 30 minutes) something like
testcase_benchbase_mariadb_4.log
## Show Summary
### Workload
Benchbase Workload tpcc SF=16
* Type: benchbase
* Duration: 2495s
* Code: 1782364932
* Benchbase runs a TPC-C experiment.
* This experiment compares run time and resource consumption of Benchbase queries in different DBMS.
* Benchbase data is generated and loaded using several threads.
* Benchmark is 'tpcc'. Scaling factor is 16. Target is based on multiples of '1024'. Factors for benchmarking are [8]. Benchmarking runs for 2 minutes.
* Experiment uses bexhoma version 0.10.0.
* System metrics are monitored by a cluster-wide installation.
* Experiment is limited to DBMS ['MariaDB'].
* Import is handled by 1 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker38.
* Database is persisted to disk of type shared and size 30Gi.
* Loading is tested with [1] threads, split into [1] pods.
* Benchmarking is tested with [160] threads, split into [1, 2] pods.
* Benchmarking is run as [1, 2] times the number of benchmarking pods.
* Experiment is run 2 times.
### Connections
* MariaDB-1-1-1-1 uses docker image mariadb:11.4.7
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220471
* volume_size:30G
* volume_used:2.0G
* cpu_list:0-127
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782364932
* MariaDB-1-1-2-1 uses docker image mariadb:11.4.7
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220471
* volume_size:30G
* volume_used:2.0G
* cpu_list:0-127
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782364932
* MariaDB-1-1-3-1 uses docker image mariadb:11.4.7
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220471
* volume_size:30G
* volume_used:2.0G
* cpu_list:0-127
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782364932
* MariaDB-1-1-4-1 uses docker image mariadb:11.4.7
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220472
* volume_size:30G
* volume_used:2.0G
* cpu_list:0-127
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782364932
* MariaDB-1-2-1-1 uses docker image mariadb:11.4.7
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220472
* volume_size:30G
* volume_used:2.3G
* cpu_list:0-127
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782364932
* MariaDB-1-2-2-1 uses docker image mariadb:11.4.7
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220472
* volume_size:30G
* volume_used:2.3G
* cpu_list:0-127
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782364932
* MariaDB-1-2-3-1 uses docker image mariadb:11.4.7
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220471
* volume_size:30G
* volume_used:2.3G
* cpu_list:0-127
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782364932
* MariaDB-1-2-4-1 uses docker image mariadb:11.4.7
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220471
* volume_size:30G
* volume_used:2.3G
* cpu_list:0-127
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782364932
### Workflow
#### Actual
* DBMS MariaDB-1 - Experiment 1 Client 1: benchbase (1 pods)
* DBMS MariaDB-1 - Experiment 1 Client 2: benchbase (2 pods)
* DBMS MariaDB-1 - Experiment 1 Client 3: benchbase (2 pods)
* DBMS MariaDB-1 - Experiment 1 Client 4: benchbase (4 pods)
* DBMS MariaDB-1 - Experiment 2 Client 1: benchbase (1 pods)
* DBMS MariaDB-1 - Experiment 2 Client 2: benchbase (2 pods)
* DBMS MariaDB-1 - Experiment 2 Client 3: benchbase (2 pods)
* DBMS MariaDB-1 - Experiment 2 Client 4: benchbase (4 pods)
#### Planned
* DBMS MariaDB-1 - Experiment 1 Client 1: benchbase (1 pods)
* DBMS MariaDB-1 - Experiment 1 Client 2: benchbase (2 pods)
* DBMS MariaDB-1 - Experiment 1 Client 3: benchbase (2 pods)
* DBMS MariaDB-1 - Experiment 1 Client 4: benchbase (4 pods)
* DBMS MariaDB-1 - Experiment 2 Client 1: benchbase (1 pods)
* DBMS MariaDB-1 - Experiment 2 Client 2: benchbase (2 pods)
* DBMS MariaDB-1 - Experiment 2 Client 3: benchbase (2 pods)
* DBMS MariaDB-1 - Experiment 2 Client 4: benchbase (4 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:------------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| MariaDB-1-1 | 1 | 16 | 761.00 | 2.00 | 0.00 | 356.00 | 403.00 | 0 | 1 | | None | 0 | False | 75.69 |
| MariaDB-1-2 | 2 | 16 | 761.00 | 2.00 | 0.00 | 356.00 | 403.00 | 0 | 1 | | None | 0 | False | 75.69 |
### Execution
#### Per Connection
| DBMS | phase | job | experiment_run | terminals | target | client | benchmark_run | child | tenant_id | time | num_errors | Throughput (requests/second) | Goodput (requests/second) | efficiency | Latency Distribution.95th Percentile Latency (microseconds) | Latency Distribution.Average Latency (microseconds) |
|:------------------|:--------------|:----------------|-----------------:|------------:|---------:|---------:|----------------:|--------:|------------:|-------:|-------------:|-------------------------------:|----------------------------:|-------------:|--------------------------------------------------------------:|------------------------------------------------------:|
| MariaDB-1-1-1-1-1 | MariaDB-1-1-1 | MariaDB-1-1-1-1 | 1 | 160 | 8192 | 1 | 1 | 1 | -1 | 120.00 | 0 | 967.06 | 952.57 | 0.00 | 492745.00 | 164974.00 |
| MariaDB-1-1-2-1-1 | MariaDB-1-1-2 | MariaDB-1-1-2-1 | 1 | 160 | 8192 | 2 | 1 | 1 | -1 | 120.00 | 0 | 320.49 | 316.64 | 0.00 | 2093763.00 | 494056.00 |
| MariaDB-1-1-2-1-2 | MariaDB-1-1-2 | MariaDB-1-1-2-1 | 1 | 160 | 8192 | 2 | 1 | 2 | -1 | 120.00 | 0 | 242.21 | 239.85 | 0.00 | 2804421.00 | 659793.00 |
| MariaDB-1-1-3-1-1 | MariaDB-1-1-3 | MariaDB-1-1-3-1 | 1 | 80 | 4096 | 3 | 1 | 1 | -1 | 120.00 | 0 | 451.47 | 445.15 | 0.00 | 135576.00 | 169888.00 |
| MariaDB-1-1-3-1-2 | MariaDB-1-1-3 | MariaDB-1-1-3-1 | 1 | 80 | 4096 | 3 | 1 | 2 | -1 | 120.00 | 0 | 425.37 | 419.38 | 0.00 | 141231.00 | 179604.00 |
| MariaDB-1-1-4-1-1 | MariaDB-1-1-4 | MariaDB-1-1-4-1 | 1 | 80 | 4096 | 4 | 1 | 1 | -1 | 120.00 | 0 | 210.65 | 208.06 | 0.00 | 1383547.00 | 379692.00 |
| MariaDB-1-1-4-1-2 | MariaDB-1-1-4 | MariaDB-1-1-4-1 | 1 | 80 | 4096 | 4 | 1 | 2 | -1 | 120.00 | 0 | 232.05 | 229.02 | 0.00 | 1189542.00 | 343797.00 |
| MariaDB-1-1-4-1-3 | MariaDB-1-1-4 | MariaDB-1-1-4-1 | 1 | 80 | 4096 | 4 | 1 | 3 | -1 | 120.00 | 0 | 213.38 | 210.43 | 0.00 | 1355253.00 | 374383.00 |
| MariaDB-1-1-4-1-4 | MariaDB-1-1-4 | MariaDB-1-1-4-1 | 1 | 80 | 4096 | 4 | 1 | 4 | -1 | 120.00 | 0 | 225.12 | 222.47 | 0.00 | 1238071.00 | 353898.00 |
| MariaDB-1-2-1-1-1 | MariaDB-1-2-1 | MariaDB-1-2-1-1 | 2 | 160 | 8192 | 1 | 1 | 1 | -1 | 120.00 | 0 | 920.43 | 906.26 | 0.00 | 571446.00 | 172230.00 |
| MariaDB-1-2-2-1-1 | MariaDB-1-2-2 | MariaDB-1-2-2-1 | 2 | 160 | 8192 | 2 | 1 | 1 | -1 | 120.00 | 0 | 427.24 | 421.13 | 0.00 | 1017372.00 | 372815.00 |
| MariaDB-1-2-2-1-2 | MariaDB-1-2-2 | MariaDB-1-2-2-1 | 2 | 160 | 8192 | 2 | 1 | 2 | -1 | 120.00 | 0 | 357.92 | 353.82 | 0.00 | 1227140.00 | 445660.00 |
| MariaDB-1-2-3-1-1 | MariaDB-1-2-3 | MariaDB-1-2-3-1 | 2 | 80 | 4096 | 3 | 1 | 1 | -1 | 120.00 | 0 | 379.06 | 373.49 | 0.00 | 917359.00 | 199443.00 |
| MariaDB-1-2-3-1-2 | MariaDB-1-2-3 | MariaDB-1-2-3-1 | 2 | 80 | 4096 | 3 | 1 | 2 | -1 | 120.00 | 0 | 348.27 | 343.62 | 0.00 | 987067.00 | 216431.00 |
| MariaDB-1-2-4-1-1 | MariaDB-1-2-4 | MariaDB-1-2-4-1 | 2 | 80 | 4096 | 4 | 1 | 1 | -1 | 120.00 | 0 | 200.48 | 197.60 | 0.00 | 1357630.00 | 397247.00 |
| MariaDB-1-2-4-1-2 | MariaDB-1-2-4 | MariaDB-1-2-4-1 | 2 | 80 | 4096 | 4 | 1 | 2 | -1 | 120.00 | 0 | 220.92 | 217.98 | 0.00 | 1163093.00 | 360457.00 |
| MariaDB-1-2-4-1-3 | MariaDB-1-2-4 | MariaDB-1-2-4-1 | 2 | 80 | 4096 | 4 | 1 | 3 | -1 | 120.00 | 0 | 202.23 | 199.54 | 0.00 | 1305829.00 | 395153.00 |
| MariaDB-1-2-4-1-4 | MariaDB-1-2-4 | MariaDB-1-2-4-1 | 2 | 80 | 4096 | 4 | 1 | 4 | -1 | 120.00 | 0 | 207.67 | 205.04 | 0.00 | 1262583.00 | 384669.00 |
#### Per Phase
| DBMS | phase | experiment_run | terminals | target | benchmark_run | pod_count | tenant_id | time | num_errors | Throughput (requests/second) | Goodput (requests/second) | efficiency | Latency Distribution.95th Percentile Latency (microseconds) | Latency Distribution.Average Latency (microseconds) |
|:--------------|:--------------|-----------------:|------------:|---------:|----------------:|------------:|------------:|-------:|-------------:|-------------------------------:|----------------------------:|-------------:|--------------------------------------------------------------:|------------------------------------------------------:|
| MariaDB-1-1-1 | MariaDB-1-1-1 | 1 | 160 | 8192 | 1 | 1 | -1 | 120.00 | 0 | 967.06 | 952.57 | 0.00 | 492745.00 | 164974.00 |
| MariaDB-1-1-2 | MariaDB-1-1-2 | 1 | 320 | 16384 | 1 | 2 | -1 | 120.00 | 0 | 562.70 | 556.49 | 0.00 | 2804421.00 | 576924.50 |
| MariaDB-1-1-3 | MariaDB-1-1-3 | 1 | 160 | 8192 | 1 | 2 | -1 | 120.00 | 0 | 876.85 | 864.53 | 0.00 | 141231.00 | 174746.00 |
| MariaDB-1-1-4 | MariaDB-1-1-4 | 1 | 320 | 16384 | 1 | 4 | -1 | 120.00 | 0 | 881.21 | 869.98 | 0.00 | 1383547.00 | 362942.50 |
| MariaDB-1-2-1 | MariaDB-1-2-1 | 2 | 160 | 8192 | 1 | 1 | -1 | 120.00 | 0 | 920.43 | 906.26 | 0.00 | 571446.00 | 172230.00 |
| MariaDB-1-2-2 | MariaDB-1-2-2 | 2 | 320 | 16384 | 1 | 2 | -1 | 120.00 | 0 | 785.17 | 774.95 | 0.00 | 1227140.00 | 409237.50 |
| MariaDB-1-2-3 | MariaDB-1-2-3 | 2 | 160 | 8192 | 1 | 2 | -1 | 120.00 | 0 | 727.33 | 717.11 | 0.00 | 987067.00 | 207937.00 |
| MariaDB-1-2-4 | MariaDB-1-2-4 | 2 | 320 | 16384 | 1 | 4 | -1 | 120.00 | 0 | 831.32 | 820.17 | 0.00 | 1357630.00 | 384381.50 |
### Monitoring
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1-1 | 2031.17 | 28.21 | 4.00 | 4.10 |
| MariaDB-1-1-2-1 | 1322.28 | 18.92 | 4.23 | 4.33 |
| MariaDB-1-1-3-1 | 1209.81 | 13.68 | 4.33 | 4.43 |
| MariaDB-1-1-4-1 | 1048.15 | 11.39 | 4.49 | 4.59 |
| MariaDB-1-2-1-1 | 1304.29 | 14.17 | 4.26 | 4.36 |
| MariaDB-1-2-2-1 | 1065.69 | 15.92 | 4.55 | 4.65 |
| MariaDB-1-2-3-1 | 926.95 | 10.39 | 4.62 | 4.72 |
| MariaDB-1-2-4-1 | 1001.09 | 12.19 | 4.80 | 4.90 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1-1 | 194.52 | 2.35 | 0.70 | 0.70 |
| MariaDB-1-1-2-1 | 168.41 | 2.97 | 0.74 | 0.74 |
| MariaDB-1-1-3-1 | 213.01 | 3.25 | 0.74 | 0.74 |
| MariaDB-1-1-4-1 | 253.99 | 3.43 | 0.50 | 0.50 |
| MariaDB-1-2-1-1 | 197.18 | 2.65 | 0.71 | 0.71 |
| MariaDB-1-2-2-1 | 228.81 | 4.92 | 0.79 | 0.79 |
| MariaDB-1-2-3-1 | 171.83 | 4.99 | 0.79 | 0.79 |
| MariaDB-1-2-4-1 | 235.27 | 4.42 | 0.49 | 0.49 |
### Tests
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Throughput (requests/second) contains no 0 or NaN
* TEST passed: Workflow as planned
HammerDB
PostgreSQL
HammerDB Simple
bexhoma hammerdb \
-dbms PostgreSQL \
-sf 16 \
-nlt 8 \
-nc 1 \
-ne 1 \
-nbp 1 \
-nbt 16 \
-ms $BEXHOMA_MS \
-tr \
-lr 64Gi \
-rr 64Gi \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_hammerdb_postgresql_1.log
yields (after ca. 10 minutes)
testcase_hammerdb_postgresql_1.log
## Show Summary
### Workload
HammerDB Workload SF=16 (warehouses for TPC-C)
* Type: tpcc
* Duration: 867s
* Code: 1782362075
* HammerDB runs the benchmark.
* This experiment compares run time and resource consumption of TPC-C queries in different DBMS.
* TPC-C data is generated and loaded using several threads.
* Scaling factor (i.e., number of warehouses) is 16. Benchmarking runs for 5 minutes.
* Experiment uses bexhoma version 0.10.0.
* Experiment is limited to DBMS ['PostgreSQL'].
* Import is handled by 1 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker38.
* Loading is tested with [8] threads, split into [1] pods.
* Benchmarking is tested with [16] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* PostgreSQL-1-1-1-1 uses docker image postgres:18.3
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:271027
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782362075
### Workflow
#### Actual
* DBMS PostgreSQL-1 - Experiment 1 Client 1: hammerdb (1 pods)
#### Planned
* DBMS PostgreSQL-1 - Experiment 1 Client 1: hammerdb (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:---------------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| PostgreSQL-1-1 | 1 | 16 | 200.00 | 2.00 | 0.00 | 80.00 | 118.00 | 1 | 8 | | None | 0 | False | 288.00 |
### Execution
#### Per Connection
| DBMS | phase | job | experiment_run | vusers | client | benchmark_run | child | NOPM | TPM | efficiency | duration | errors |
|:---------------------|:-----------------|:-------------------|-----------------:|---------:|---------:|----------------:|--------:|-------:|--------:|-------------:|-----------:|---------:|
| PostgreSQL-1-1-1-1-1 | PostgreSQL-1-1-1 | PostgreSQL-1-1-1-1 | 1 | 16 | 1 | 1 | 1 | 461948 | 1061903 | 0.00 | 5 | 0 |
#### Per Phase
| DBMS | phase | experiment_run | vusers | client | benchmark_run | pod_count | efficiency | NOPM | TPM | duration | errors |
|:-----------------|:-----------------|-----------------:|---------:|---------:|----------------:|------------:|-------------:|----------:|-----------:|-----------:|---------:|
| PostgreSQL-1-1-1 | PostgreSQL-1-1-1 | 1 | 16 | 1 | 1 | 1 | 0.00 | 461948.00 | 1061903.00 | 5 | 0 |
### Tests
* TEST passed: NOPM contains no 0 or NaN
* TEST passed: Workflow as planned
HammerDB Monitoring
bexhoma hammerdb \
-dbms PostgreSQL \
-sf 16 \
-nlt 8 \
-nc 1 \
-ne 1 \
-nbp 1 \
-nbt 16 \
-m \
-mc \
-ms $BEXHOMA_MS \
-lr 64Gi \
-rr 64Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_hammerdb_postgresql_2.log
yields (after ca. 15 minutes)
testcase_hammerdb_postgresql_2.log
## Show Summary
### Workload
HammerDB Workload SF=16 (warehouses for TPC-C)
* Type: tpcc
* Duration: 997s
* Code: 1782362983
* HammerDB runs the benchmark.
* This experiment compares run time and resource consumption of TPC-C queries in different DBMS.
* TPC-C data is generated and loaded using several threads.
* Scaling factor (i.e., number of warehouses) is 16. Benchmarking runs for 5 minutes.
* Experiment uses bexhoma version 0.10.0.
* System metrics are monitored by a cluster-wide installation.
* Experiment is limited to DBMS ['PostgreSQL'].
* Import is handled by 1 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker38.
* Database is persisted to disk of type shared and size 30Gi.
* Loading is tested with [8] threads, split into [1] pods.
* Benchmarking is tested with [16] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* PostgreSQL-1-1-1-1 uses docker image postgres:18.3
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:253332
* volume_size:30G
* volume_used:3.3G
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782362983
### Workflow
#### Actual
* DBMS PostgreSQL-1 - Experiment 1 Client 1: hammerdb (1 pods)
#### Planned
* DBMS PostgreSQL-1 - Experiment 1 Client 1: hammerdb (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:---------------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| PostgreSQL-1-1 | 1 | 16 | 364.00 | 2.00 | 0.00 | 165.00 | 197.00 | 1 | 8 | | None | 0 | False | 158.24 |
### Execution
#### Per Connection
| DBMS | phase | job | experiment_run | vusers | client | benchmark_run | child | NOPM | TPM | efficiency | duration | errors |
|:---------------------|:-----------------|:-------------------|-----------------:|---------:|---------:|----------------:|--------:|-------:|------:|-------------:|-----------:|---------:|
| PostgreSQL-1-1-1-1-1 | PostgreSQL-1-1-1 | PostgreSQL-1-1-1-1 | 1 | 16 | 1 | 1 | 1 | 14626 | 33661 | 0.00 | 5 | 0 |
#### Per Phase
| DBMS | phase | experiment_run | vusers | client | benchmark_run | pod_count | efficiency | NOPM | TPM | duration | errors |
|:-----------------|:-----------------|-----------------:|---------:|---------:|----------------:|------------:|-------------:|---------:|---------:|-----------:|---------:|
| PostgreSQL-1-1-1 | PostgreSQL-1-1-1 | 1 | 16 | 1 | 1 | 1 | 0.00 | 14626.00 | 33661.00 | 5 | 0 |
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 84.21 | 0.73 | 1.69 | 3.22 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 253.02 | 4.30 | 0.09 | 0.09 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 360.15 | 1.28 | 2.14 | 3.93 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 35.38 | 0.11 | 0.07 | 0.07 |
### Tests
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: NOPM contains no 0 or NaN
* TEST passed: Workflow as planned
HammerDB Complex
bexhoma hammerdb \
-dbms PostgreSQL \
-sf 16 \
-xsd 2 \
-nlt 8 \
-nc 2 \
-ne 1,2 \
-nbp 1,2 \
-nbt 16 \
-m \
-mc \
-ms $BEXHOMA_MS \
-lr 64Gi \
-rr 64Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_hammerdb_postgresql_3.log
yields (after ca. 60 minutes)
testcase_hammerdb_postgresql_3.log
## Show Summary
### Workload
HammerDB Workload SF=16 (warehouses for TPC-C)
* Type: tpcc
* Duration: 3463s
* Code: 1782364015
* HammerDB runs the benchmark.
* This experiment compares run time and resource consumption of TPC-C queries in different DBMS.
* TPC-C data is generated and loaded using several threads.
* Scaling factor (i.e., number of warehouses) is 16. Benchmarking runs for 2 minutes.
* Experiment uses bexhoma version 0.10.0.
* System metrics are monitored by a cluster-wide installation.
* Experiment is limited to DBMS ['PostgreSQL'].
* Import is handled by 1 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker38.
* Database is persisted to disk of type shared and size 30Gi.
* Loading is tested with [8] threads, split into [1] pods.
* Benchmarking is tested with [16] threads, split into [1, 2] pods.
* Benchmarking is run as [1, 2] times the number of benchmarking pods.
* Experiment is run 2 times.
### Connections
* PostgreSQL-1-1-1-1 uses docker image postgres:18.3
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:277941
* volume_size:30G
* volume_used:4.1G
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782364015
* PostgreSQL-1-1-2-1 uses docker image postgres:18.3
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:222745
* volume_size:30G
* volume_used:4.1G
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782364015
* PostgreSQL-1-1-3-1 uses docker image postgres:18.3
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220471
* volume_size:30G
* volume_used:4.1G
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782364015
* PostgreSQL-1-1-4-1 uses docker image postgres:18.3
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220471
* volume_size:30G
* volume_used:4.1G
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782364015
* PostgreSQL-1-2-1-1 uses docker image postgres:18.3
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220472
* volume_size:30G
* volume_used:4.8G
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782364015
* PostgreSQL-1-2-2-1 uses docker image postgres:18.3
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220472
* volume_size:30G
* volume_used:4.8G
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782364015
* PostgreSQL-1-2-3-1 uses docker image postgres:18.3
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220472
* volume_size:30G
* volume_used:4.8G
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782364015
* PostgreSQL-1-2-4-1 uses docker image postgres:18.3
* RAM:540492877824
* CPU:Intel(R) Xeon(R) Gold 6430
* Cores:128
* host:6.8.0-111-generic
* node:cl-worker38
* disk:220471
* volume_size:30G
* volume_used:4.8G
* cpu_list:0-127
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1782364015
### Workflow
#### Actual
* DBMS PostgreSQL-1 - Experiment 1 Client 1: hammerdb (1 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 2: hammerdb (2 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 3: hammerdb (2 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 4: hammerdb (4 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 1: hammerdb (1 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 2: hammerdb (2 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 3: hammerdb (2 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 4: hammerdb (4 pods)
#### Planned
* DBMS PostgreSQL-1 - Experiment 1 Client 1: hammerdb (1 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 2: hammerdb (2 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 3: hammerdb (2 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 4: hammerdb (4 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 1: hammerdb (1 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 2: hammerdb (2 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 3: hammerdb (2 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 4: hammerdb (4 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:---------------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| PostgreSQL-1-1 | 1 | 16 | 364.00 | 2.00 | 0.00 | 165.00 | 197.00 | 0 | 8 | | None | 0 | False | 158.24 |
| PostgreSQL-1-2 | 2 | 16 | 364.00 | 2.00 | 0.00 | 165.00 | 197.00 | 0 | 8 | | None | 0 | False | 158.24 |
### Execution
#### Per Connection
| DBMS | phase | job | experiment_run | vusers | client | benchmark_run | child | NOPM | TPM | efficiency | duration | errors |
|:---------------------|:-----------------|:-------------------|-----------------:|---------:|---------:|----------------:|--------:|-------:|------:|-------------:|-----------:|---------:|
| PostgreSQL-1-1-1-1-1 | PostgreSQL-1-1-1 | PostgreSQL-1-1-1-1 | 1 | 16 | 1 | 1 | 1 | 13209 | 30504 | 0.00 | 2 | 0 |
| PostgreSQL-1-1-2-1-1 | PostgreSQL-1-1-2 | PostgreSQL-1-1-2-1 | 1 | 16 | 2 | 1 | 1 | 15790 | 36578 | 0.00 | 2 | 0 |
| PostgreSQL-1-1-2-1-1 | PostgreSQL-1-1-2 | PostgreSQL-1-1-2-1 | 1 | 16 | 2 | 1 | 1 | 15782 | 36339 | 0.00 | 2 | 0 |
| PostgreSQL-1-1-3-1-1 | PostgreSQL-1-1-3 | PostgreSQL-1-1-3-1 | 1 | 8 | 3 | 1 | 1 | 11178 | 25963 | 0.00 | 2 | 0 |
| PostgreSQL-1-1-3-1-1 | PostgreSQL-1-1-3 | PostgreSQL-1-1-3-1 | 1 | 8 | 3 | 1 | 1 | 11180 | 25823 | 0.00 | 2 | 0 |
| PostgreSQL-1-1-4-1-1 | PostgreSQL-1-1-4 | PostgreSQL-1-1-4-1 | 1 | 8 | 4 | 1 | 1 | 13528 | 31357 | 0.00 | 2 | 0 |
| PostgreSQL-1-1-4-1-1 | PostgreSQL-1-1-4 | PostgreSQL-1-1-4-1 | 1 | 8 | 4 | 1 | 1 | 13555 | 31424 | 0.00 | 2 | 0 |
| PostgreSQL-1-1-4-1-1 | PostgreSQL-1-1-4 | PostgreSQL-1-1-4-1 | 1 | 8 | 4 | 1 | 1 | 13558 | 31395 | 0.00 | 2 | 0 |
| PostgreSQL-1-1-4-1-1 | PostgreSQL-1-1-4 | PostgreSQL-1-1-4-1 | 1 | 8 | 4 | 1 | 1 | 13582 | 31502 | 0.00 | 2 | 0 |
| PostgreSQL-1-2-1-1-1 | PostgreSQL-1-2-1 | PostgreSQL-1-2-1-1 | 2 | 16 | 1 | 1 | 1 | 8849 | 20432 | 0.00 | 2 | 0 |
| PostgreSQL-1-2-2-1-1 | PostgreSQL-1-2-2 | PostgreSQL-1-2-2-1 | 2 | 16 | 2 | 1 | 1 | 13977 | 32490 | 0.00 | 2 | 0 |
| PostgreSQL-1-2-2-1-1 | PostgreSQL-1-2-2 | PostgreSQL-1-2-2-1 | 2 | 16 | 2 | 1 | 1 | 13912 | 32276 | 0.00 | 2 | 0 |
| PostgreSQL-1-2-3-1-1 | PostgreSQL-1-2-3 | PostgreSQL-1-2-3-1 | 2 | 8 | 3 | 1 | 1 | 11789 | 27254 | 0.00 | 2 | 0 |
| PostgreSQL-1-2-3-1-1 | PostgreSQL-1-2-3 | PostgreSQL-1-2-3-1 | 2 | 8 | 3 | 1 | 1 | 11806 | 27287 | 0.00 | 2 | 0 |
| PostgreSQL-1-2-4-1-1 | PostgreSQL-1-2-4 | PostgreSQL-1-2-4-1 | 2 | 8 | 4 | 1 | 1 | 14962 | 34299 | 0.00 | 2 | 0 |
| PostgreSQL-1-2-4-1-1 | PostgreSQL-1-2-4 | PostgreSQL-1-2-4-1 | 2 | 8 | 4 | 1 | 1 | 14852 | 33988 | 0.00 | 2 | 0 |
| PostgreSQL-1-2-4-1-1 | PostgreSQL-1-2-4 | PostgreSQL-1-2-4-1 | 2 | 8 | 4 | 1 | 1 | 14883 | 34103 | 0.00 | 2 | 0 |
| PostgreSQL-1-2-4-1-1 | PostgreSQL-1-2-4 | PostgreSQL-1-2-4-1 | 2 | 8 | 4 | 1 | 1 | 14925 | 34225 | 0.00 | 2 | 0 |
#### Per Phase
| DBMS | phase | experiment_run | vusers | client | benchmark_run | pod_count | efficiency | NOPM | TPM | duration | errors |
|:-----------------|:-----------------|-----------------:|---------:|---------:|----------------:|------------:|-------------:|---------:|---------:|-----------:|---------:|
| PostgreSQL-1-1-1 | PostgreSQL-1-1-1 | 1 | 16 | 1 | 1 | 1 | 0.00 | 13209.00 | 30504.00 | 2 | 0 |
| PostgreSQL-1-1-2 | PostgreSQL-1-1-2 | 1 | 32 | 2 | 1 | 2 | 0.00 | 15786.00 | 36458.50 | 2 | 0 |
| PostgreSQL-1-1-3 | PostgreSQL-1-1-3 | 1 | 16 | 3 | 1 | 2 | 0.00 | 11179.00 | 25893.00 | 2 | 0 |
| PostgreSQL-1-1-4 | PostgreSQL-1-1-4 | 1 | 32 | 4 | 1 | 4 | 0.00 | 13555.75 | 31419.50 | 2 | 0 |
| PostgreSQL-1-2-1 | PostgreSQL-1-2-1 | 2 | 16 | 1 | 1 | 1 | 0.00 | 8849.00 | 20432.00 | 2 | 0 |
| PostgreSQL-1-2-2 | PostgreSQL-1-2-2 | 2 | 32 | 2 | 1 | 2 | 0.00 | 13944.50 | 32383.00 | 2 | 0 |
| PostgreSQL-1-2-3 | PostgreSQL-1-2-3 | 2 | 16 | 3 | 1 | 2 | 0.00 | 11797.50 | 27270.50 | 2 | 0 |
| PostgreSQL-1-2-4 | PostgreSQL-1-2-4 | 2 | 32 | 4 | 1 | 4 | 0.00 | 14905.50 | 34153.75 | 2 | 0 |
### Monitoring
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 165.55 | 0.82 | 1.26 | 2.95 |
| PostgreSQL-1-1-2-1 | 135.36 | 1.02 | 1.82 | 3.76 |
| PostgreSQL-1-1-3-1 | 325.31 | 1.74 | 2.19 | 4.24 |
| PostgreSQL-1-1-4-1 | 1809.44 | 11.38 | 2.74 | 4.91 |
| PostgreSQL-1-2-1-1 | 2484.00 | 5.72 | 2.15 | 4.34 |
| PostgreSQL-1-2-2-1 | 2138.16 | 11.27 | 2.27 | 4.59 |
| PostgreSQL-1-2-3-1 | 1298.34 | 7.16 | 2.18 | 4.59 |
| PostgreSQL-1-2-4-1 | 2151.17 | 12.22 | 2.72 | 5.27 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 13.46 | 0.08 | 0.07 | 0.07 |
| PostgreSQL-1-1-2-1 | 13.82 | 0.20 | 0.07 | 0.07 |
| PostgreSQL-1-1-3-1 | 21.41 | 0.19 | 0.07 | 0.07 |
| PostgreSQL-1-1-4-1 | 18.60 | 0.28 | 0.04 | 0.05 |
| PostgreSQL-1-2-1-1 | 9.53 | 0.05 | 0.07 | 0.07 |
| PostgreSQL-1-2-2-1 | 13.03 | 0.12 | 0.07 | 0.07 |
| PostgreSQL-1-2-3-1 | 21.69 | 0.14 | 0.07 | 0.07 |
| PostgreSQL-1-2-4-1 | 17.99 | 0.24 | 0.04 | 0.05 |
### Tests
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: NOPM contains no 0 or NaN
* TEST passed: Workflow as planned
MySQL
HammerDB Simple
bexhoma hammerdb \
-dbms MySQL \
-sf 16 \
-nlt 8 \
-nc 1 \
-ne 1 \
-nbp 1 \
-nbt 16 \
-ms $BEXHOMA_MS \
-tr \
-lr 64Gi \
-rr 64Gi \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_hammerdb_mysql_1.log
yields (after ca. 10 minutes)
testcase_hammerdb_mysql_1.log
## Show Summary
### Workload
HammerDB Workload SF=16 (warehouses for TPC-C)
* Type: tpcc
* Duration: 1347s
* Code: 1782626077
* HammerDB runs the benchmark.
* This experiment compares run time and resource consumption of TPC-C queries in different DBMS.
* TPC-C data is generated and loaded using several threads.
* Scaling factor (i.e., number of warehouses) is 16. Benchmarking runs for 5 minutes.
* Experiment uses bexhoma version 0.10.2.
* Experiment is limited to DBMS ['MySQL'].
* Import is handled by 1 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* Database uses ephemeral storage of size 16Gi.
* Loading is tested with [8] threads, split into [1] pods.
* Benchmarking is tested with [16] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* MySQL-1-1-1-1 uses docker image mysql:8.4.0
* RAM:1077381271552
* Cores:256
* host:6.8.0-111-generic
* node:cl-worker27
* disk:1407972
* cpu_list:0-255
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--skip-log-bin', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1782626077
### SUT Container Restarts
* bexhoma-sut-mysql-1-1782626077-58dcb6d95-bkq4d: 0 0
### Workflow
#### Actual
* DBMS MySQL-1 - Experiment 1 Client 1: hammerdb (1 pods)
#### Planned
* DBMS MySQL-1 - Experiment 1 Client 1: hammerdb (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:----------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| MySQL-1-1 | 1 | 16 | 257.00 | 14.00 | 0.00 | 85.00 | 158.00 | 1 | 8 | | None | 0 | False | 224.12 |
### Execution
#### Per Connection
| DBMS | phase | job | experiment_run | vusers | client | benchmark_run | child | NOPM | TPM | efficiency | duration | errors |
|:----------------|:------------|:--------------|-----------------:|---------:|---------:|----------------:|--------:|-------:|-------:|-------------:|-----------:|---------:|
| MySQL-1-1-1-1-1 | MySQL-1-1-1 | MySQL-1-1-1-1 | 1 | 16 | 1 | 1 | 1 | 181106 | 421095 | 0.00 | 5 | 0 |
#### Per Phase
| DBMS | phase | experiment_run | vusers | client | benchmark_run | pod_count | efficiency | NOPM | TPM | duration | errors |
|:------------|:------------|-----------------:|---------:|---------:|----------------:|------------:|-------------:|----------:|----------:|-----------:|---------:|
| MySQL-1-1-1 | MySQL-1-1-1 | 1 | 16 | 1 | 1 | 1 | 0.00 | 181106.00 | 421095.00 | 5 | 0 |
### Tests
* TEST passed: No SUT container restarts
* TEST passed: NOPM contains no 0 or NaN
* TEST passed: Workflow as planned
HammerDB Monitoring
bexhoma hammerdb \
-dbms MySQL \
-sf 16 \
-nlt 8 \
-nc 1 \
-ne 1 \
-nbp 1 \
-nbt 16 \
-m \
-mc \
-ms $BEXHOMA_MS \
-lr 64Gi \
-rr 64Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rss 128Gi \
-rsr \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_hammerdb_mysql_2.log
yields (after ca. 15 minutes)
testcase_hammerdb_mysql_2.log
## Show Summary
### Workload
HammerDB Workload SF=16 (warehouses for TPC-C)
* Type: tpcc
* Duration: 15959s
* Code: 1782644541
* HammerDB runs the benchmark.
* This experiment compares run time and resource consumption of TPC-C queries in different DBMS.
* TPC-C data is generated and loaded using several threads.
* Scaling factor (i.e., number of warehouses) is 16. Benchmarking runs for 5 minutes.
* Experiment uses bexhoma version 0.10.2.
* System metrics are monitored by a cluster-wide installation.
* Experiment is limited to DBMS ['MySQL'].
* Import is handled by 1 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* Database is persisted to disk of type shared and size 128Gi. Persistent storage is removed at experiment start.
* Loading is tested with [8] threads, split into [1] pods.
* Benchmarking is tested with [16] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* MySQL-1-1-1-1 uses docker image mysql:8.4.0
* RAM:540590804992
* Cores:96
* host:6.8.0-124-generic
* node:cl-worker24
* disk:162631
* volume_size:128G
* volume_used:34G
* cpu_list:0-95
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--skip-log-bin', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1782644541
### SUT Container Restarts
* bexhoma-sut-mysql-1-1782644541-784d9855cc-22v5m: 0 0
### Workflow
#### Actual
* DBMS MySQL-1 - Experiment 1 Client 1: hammerdb (1 pods)
#### Planned
* DBMS MySQL-1 - Experiment 1 Client 1: hammerdb (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:----------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| MySQL-1-1 | 1 | 16 | 28313.00 | 15.00 | 0.00 | 14125.00 | 14173.00 | 1 | 8 | | None | 0 | False | 2.03 |
### Execution
#### Per Connection
| DBMS | phase | job | experiment_run | vusers | client | benchmark_run | child | NOPM | TPM | efficiency | duration | errors |
|:----------------|:------------|:--------------|-----------------:|---------:|---------:|----------------:|--------:|-------:|------:|-------------:|-----------:|---------:|
| MySQL-1-1-1-1-1 | MySQL-1-1-1 | MySQL-1-1-1-1 | 1 | 16 | 1 | 1 | 1 | 6 | 11 | 0.00 | 5 | 0 |
#### Per Phase
| DBMS | phase | experiment_run | vusers | client | benchmark_run | pod_count | efficiency | NOPM | TPM | duration | errors |
|:------------|:------------|-----------------:|---------:|---------:|----------------:|------------:|-------------:|-------:|------:|-----------:|---------:|
| MySQL-1-1-1 | MySQL-1-1-1 | 1 | 16 | 1 | 1 | 1 | 0.00 | 6.00 | 11.00 | 5 | 0 |
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 236.29 | 0.46 | 22.11 | 54.13 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 289.80 | 0.19 | 0.10 | 0.10 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 3.29 | 0.02 | 22.17 | 54.19 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 1.09 | 0.05 | 0.07 | 0.07 |
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: NOPM contains no 0 or NaN
* TEST passed: Workflow as planned
HammerDB Complex
bexhoma hammerdb \
-dbms MySQL \
-sf 16 \
-xsd 2 \
-nlt 8 \
-nc 2 \
-ne 1,2 \
-nbp 1,2 \
-nbt 16 \
-m \
-mc \
-ms $BEXHOMA_MS \
-lr 64Gi \
-rr 64Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_hammerdb_mysql_3.log
yields (after ca. 60 minutes)
testcase_hammerdb_mysql_3.log
## Show Summary
### Workload
HammerDB Workload SF=16 (warehouses for TPC-C)
Type: tpcc
Duration: 3902s
Code: 1750098823
HammerDB runs the benchmark.
This experiment compares run time and resource consumption of TPC-C queries in different DBMS.
TPC-C data is generated and loaded using several threads.
Scaling factor (i.e., number of warehouses) is 16. Benchmarking runs for 2 minutes.
Experiment uses bexhoma version 0.8.8.
System metrics are monitored by a cluster-wide installation.
Experiment is limited to DBMS ['MySQL'].
Import is handled by 1 processes (pods).
Loading is fixed to cl-worker19.
Benchmarking is fixed to cl-worker19.
SUT is fixed to cl-worker11.
Database is persisted to disk of type shared and size 30Gi.
Loading is tested with [8] threads, split into [1] pods.
Benchmarking is tested with [16] threads, split into [1, 2] pods.
Benchmarking is run as [1, 2] times the number of benchmarking pods.
Experiment is run 2 times.
### Connections
MySQL-BHT-8-1-1-1 uses docker image mysql:8.4.0
RAM:541008568320
Cores:64
host:5.15.0-140-generic
node:cl-worker11
disk:392962848
datadisk:11037
volume_size:30G
volume_used:11G
requests_cpu:4
requests_memory:16Gi
eval_parameters
code:1750098823
MySQL-BHT-8-1-1-2 uses docker image mysql:8.4.0
RAM:541008568320
Cores:64
host:5.15.0-140-generic
node:cl-worker11
disk:392965728
datadisk:11230
volume_size:30G
volume_used:11G
requests_cpu:4
requests_memory:16Gi
eval_parameters
code:1750098823
MySQL-BHT-8-1-1-3 uses docker image mysql:8.4.0
RAM:541008568320
Cores:64
host:5.15.0-140-generic
node:cl-worker11
disk:393131856
datadisk:11531
volume_size:30G
volume_used:11G
requests_cpu:4
requests_memory:16Gi
eval_parameters
code:1750098823
MySQL-BHT-8-1-1-4 uses docker image mysql:8.4.0
RAM:541008568320
Cores:64
host:5.15.0-140-generic
node:cl-worker11
disk:393133892
datadisk:11702
volume_size:30G
volume_used:11G
requests_cpu:4
requests_memory:16Gi
eval_parameters
code:1750098823
MySQL-BHT-8-1-2-1 uses docker image mysql:8.4.0
RAM:541008568320
Cores:64
host:5.15.0-140-generic
node:cl-worker11
disk:393140456
datadisk:12083
volume_size:30G
volume_used:12G
requests_cpu:4
requests_memory:16Gi
eval_parameters
code:1750098823
MySQL-BHT-8-1-2-2 uses docker image mysql:8.4.0
RAM:541008568320
Cores:64
host:5.15.0-140-generic
node:cl-worker11
disk:393141932
datadisk:12259
volume_size:30G
volume_used:12G
requests_cpu:4
requests_memory:16Gi
eval_parameters
code:1750098823
MySQL-BHT-8-1-2-3 uses docker image mysql:8.4.0
RAM:541008568320
Cores:64
host:5.15.0-140-generic
node:cl-worker11
disk:393149976
datadisk:12569
volume_size:30G
volume_used:12G
requests_cpu:4
requests_memory:16Gi
eval_parameters
code:1750098823
MySQL-BHT-8-1-2-4 uses docker image mysql:8.4.0
RAM:541008568320
Cores:64
host:5.15.0-140-generic
node:cl-worker11
disk:393150500
datadisk:12729
volume_size:30G
volume_used:12G
requests_cpu:4
requests_memory:16Gi
eval_parameters
code:1750098823
### Execution
experiment_run vusers client pod_count efficiency NOPM TPM duration errors
MySQL-BHT-8-1-1-1 1 16 1 1 0.0 2752.0 6382.00 2 0
MySQL-BHT-8-1-1-2 1 32 2 2 0.0 4722.5 10913.00 2 0
MySQL-BHT-8-1-1-3 1 16 3 2 0.0 1650.0 3857.50 2 0
MySQL-BHT-8-1-1-4 1 32 4 4 0.0 7990.5 18509.75 2 0
MySQL-BHT-8-1-2-1 2 16 1 1 0.0 2395.0 5645.00 2 0
MySQL-BHT-8-1-2-2 2 32 2 2 0.0 3964.0 9193.00 2 0
MySQL-BHT-8-1-2-3 2 16 3 2 0.0 2639.5 6071.50 2 0
MySQL-BHT-8-1-2-4 2 32 4 4 0.0 4926.0 11356.25 2 0
Warehouses: 16
### Workflow
#### Actual
DBMS MySQL-BHT-8-1 - Pods [[4, 2, 1, 2], [2, 4, 1, 2]]
#### Planned
DBMS MySQL-BHT-8-1 - Pods [[1, 2, 2, 4], [1, 2, 2, 4]]
### Loading
time_load terminals pods Imported warehouses [1/h]
MySQL-BHT-8-1-1-1 6604.0 1.0 1.0 8.721987
MySQL-BHT-8-1-1-2 6604.0 1.0 2.0 8.721987
MySQL-BHT-8-1-1-3 6604.0 1.0 2.0 8.721987
MySQL-BHT-8-1-1-4 6604.0 1.0 4.0 8.721987
MySQL-BHT-8-1-2-1 6604.0 1.0 1.0 8.721987
MySQL-BHT-8-1-2-2 6604.0 1.0 2.0 8.721987
MySQL-BHT-8-1-2-3 6604.0 1.0 2.0 8.721987
MySQL-BHT-8-1-2-4 6604.0 1.0 4.0 8.721987
### Execution - SUT
CPU [CPUs] Max CPU Max RAM [Gb] Max RAM Cached [Gb]
MySQL-BHT-8-1-1-1 482.55 1.39 38.37 46.48
MySQL-BHT-8-1-1-2 490.84 2.18 38.66 47.15
MySQL-BHT-8-1-1-3 308.68 1.21 38.73 47.36
MySQL-BHT-8-1-1-4 499.53 1.67 38.95 47.92
MySQL-BHT-8-1-2-1 2618.47 1.74 77.01 94.82
MySQL-BHT-8-1-2-2 477.33 1.67 38.66 47.99
MySQL-BHT-8-1-2-3 347.58 1.74 38.81 48.26
MySQL-BHT-8-1-2-4 443.20 1.76 38.92 48.66
### Execution - Benchmarker
CPU [CPUs] Max CPU Max RAM [Gb] Max RAM Cached [Gb]
MySQL-BHT-8-1-1-1 12.44 0.03 0.07 0.07
MySQL-BHT-8-1-1-2 12.86 0.05 0.21 0.22
MySQL-BHT-8-1-1-3 19.67 0.02 0.23 0.23
MySQL-BHT-8-1-1-4 19.70 0.09 0.26 0.27
MySQL-BHT-8-1-2-1 11.75 0.04 0.07 0.07
MySQL-BHT-8-1-2-2 16.38 0.06 0.21 0.22
MySQL-BHT-8-1-2-3 22.10 0.05 0.23 0.23
MySQL-BHT-8-1-2-4 11.96 0.04 0.26 0.27
### Tests
TEST passed: Execution SUT contains no 0 or NaN in CPU [CPUs]
TEST passed: Execution Benchmarker contains no 0 or NaN in CPU [CPUs]
TEST passed: NOPM contains no 0 or NaN
TEST passed: Workflow as planned
MariaDB
HammerDB Simple
bexhoma hammerdb \
-dbms MariaDB \
-sf 16 \
-nlt 8 \
-nc 1 \
-ne 1 \
-nbp 1 \
-nbt 16 \
-ms $BEXHOMA_MS \
-tr \
-lr 64Gi \
-rr 64Gi \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_hammerdb_mariadb_1.log
yields (after ca. 10 minutes)
testcase_hammerdb_mariadb_1.log
## Show Summary
### Workload
HammerDB Workload SF=16 (warehouses for TPC-C)
* Type: tpcc
* Duration: 885s
* Code: 1780052904
* HammerDB runs the benchmark.
* This experiment compares run time and resource consumption of TPC-C queries in different DBMS.
* TPC-C data is generated and loaded using several threads.
* Scaling factor (i.e., number of warehouses) is 16. Benchmarking runs for 5 minutes.
* Experiment uses bexhoma version 0.9.9.
* Experiment is limited to DBMS ['MariaDB'].
* Import is handled by 1 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker3.
* Loading is tested with [8] threads, split into [1] pods.
* Benchmarking is tested with [16] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* MariaDB-1-1-1 uses docker image mariadb:11.4.7
* RAM:541006622720
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:59776
* cpu_list:0-63
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=256G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1780052904
### Workflow
#### Actual
* DBMS MariaDB-1 - Pods [[1]]
#### Planned
* DBMS MariaDB-1 - Pods [[1]]
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:------------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:---------|:---------------|--------------:|:--------------|--------------------:|
| MariaDB-1-1 | 1 | 16 | 190.00 | 1.00 | 0.00 | 80.00 | 109.00 | 1 | 8 | | None | 0 | False | 303.16 |
### Execution
#### Per Connection
| DBMS | experiment_run | vusers | client | child | NOPM | TPM | efficiency | duration | errors |
|:----------------|-----------------:|---------:|---------:|--------:|-------:|-------:|-------------:|-----------:|---------:|
| MariaDB-1-1-1-1 | 1 | 16 | 1 | 1 | 63390 | 147366 | 0.00 | 5 | 0 |
#### Per Phase
| DBMS | experiment_run | vusers | client | pod_count | efficiency | NOPM | TPM | duration | errors |
|:--------------|-----------------:|---------:|---------:|------------:|-------------:|---------:|----------:|-----------:|---------:|
| MariaDB-1-1-1 | 1.00 | 16.00 | 1.00 | 1.00 | 0.00 | 63390.00 | 147366.00 | 5.00 | 0.00 |
### Tests
* TEST passed: NOPM contains no 0 or NaN
* TEST passed: Workflow as planned
HammerDB Monitoring
bexhoma hammerdb \
-dbms MariaDB \
-sf 16 \
-nlt 8 \
-nc 1 \
-ne 1 \
-nbp 1 \
-nbt 16 \
-m \
-mc \
-ms $BEXHOMA_MS \
-lr 64Gi \
-rr 64Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_hammerdb_mariadb_2.log
yields (after ca. 15 minutes)
testcase_hammerdb_mariadb_2.log
## Show Summary
### Workload
HammerDB Workload SF=16 (warehouses for TPC-C)
* Type: tpcc
* Duration: 1155s
* Code: 1780053795
* HammerDB runs the benchmark.
* This experiment compares run time and resource consumption of TPC-C queries in different DBMS.
* TPC-C data is generated and loaded using several threads.
* Scaling factor (i.e., number of warehouses) is 16. Benchmarking runs for 5 minutes.
* Experiment uses bexhoma version 0.9.9.
* System metrics are monitored by a cluster-wide installation.
* Experiment is limited to DBMS ['MariaDB'].
* Import is handled by 1 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker3.
* Database is persisted to disk of type shared and size 30Gi. Persistent storage is removed at experiment start.
* Loading is tested with [8] threads, split into [1] pods.
* Benchmarking is tested with [16] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* MariaDB-1-1-1 uses docker image mariadb:11.4.7
* RAM:541006622720
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58089
* volume_size:30G
* volume_used:1.7G
* cpu_list:0-63
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=256G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1780053795
### Workflow
#### Actual
* DBMS MariaDB-1 - Pods [[1]]
#### Planned
* DBMS MariaDB-1 - Pods [[1]]
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:------------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:---------|:---------------|--------------:|:--------------|--------------------:|
| MariaDB-1-1 | 1 | 16 | 700.00 | 1.00 | 0.00 | 335.00 | 364.00 | 1 | 8 | | None | 0 | False | 82.29 |
### Execution
#### Per Connection
| DBMS | experiment_run | vusers | client | child | NOPM | TPM | efficiency | duration | errors |
|:----------------|-----------------:|---------:|---------:|--------:|-------:|------:|-------------:|-----------:|---------:|
| MariaDB-1-1-1-1 | 1 | 16 | 1 | 1 | 9805 | 22794 | 0.00 | 5 | 0 |
#### Per Phase
| DBMS | experiment_run | vusers | client | pod_count | efficiency | NOPM | TPM | duration | errors |
|:--------------|-----------------:|---------:|---------:|------------:|-------------:|--------:|---------:|-----------:|---------:|
| MariaDB-1-1-1 | 1.00 | 16.00 | 1.00 | 1.00 | 0.00 | 9805.00 | 22794.00 | 5.00 | 0.00 |
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1 | 845.08 | 3.35 | 7.06 | 7.16 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1 | 271.99 | 1.16 | 0.10 | 0.10 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1 | 947.62 | 3.68 | 7.16 | 7.26 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1 | 40.35 | 0.15 | 0.07 | 0.07 |
### Tests
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: NOPM contains no 0 or NaN
* TEST passed: Workflow as planned
HammerDB Complex
bexhoma hammerdb \
-dbms MariaDB \
-sf 16 \
-xsd 2 \
-nlt 8 \
-nc 2 \
-ne 1,2 \
-nbp 1,2 \
-nbt 16 \
-m \
-mc \
-ms $BEXHOMA_MS \
-lr 64Gi \
-rr 64Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_hammerdb_mariadb_3.log
yields (after ca. 60 minutes)
testcase_hammerdb_mariadb_3.log
## Show Summary
### Workload
HammerDB Workload SF=16 (warehouses for TPC-C)
* Type: tpcc
* Duration: 3224s
* Code: 1780054976
* HammerDB runs the benchmark.
* This experiment compares run time and resource consumption of TPC-C queries in different DBMS.
* TPC-C data is generated and loaded using several threads.
* Scaling factor (i.e., number of warehouses) is 16. Benchmarking runs for 2 minutes.
* Experiment uses bexhoma version 0.9.9.
* System metrics are monitored by a cluster-wide installation.
* Experiment is limited to DBMS ['MariaDB'].
* Import is handled by 1 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker3.
* Database is persisted to disk of type shared and size 30Gi.
* Loading is tested with [8] threads, split into [1] pods.
* Benchmarking is tested with [16] threads, split into [1, 2] pods.
* Benchmarking is run as [1, 2] times the number of benchmarking pods.
* Experiment is run 2 times.
### Connections
* MariaDB-1-1-1 uses docker image mariadb:11.4.7
* RAM:541006622720
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58089
* volume_size:30G
* volume_used:1.8G
* cpu_list:0-63
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=256G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1780054976
* MariaDB-1-1-2 uses docker image mariadb:11.4.7
* RAM:541006622720
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58089
* volume_size:30G
* volume_used:1.8G
* cpu_list:0-63
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=256G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1780054976
* MariaDB-1-1-3 uses docker image mariadb:11.4.7
* RAM:541006622720
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58089
* volume_size:30G
* volume_used:1.8G
* cpu_list:0-63
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=256G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1780054976
* MariaDB-1-1-4 uses docker image mariadb:11.4.7
* RAM:541006622720
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58089
* volume_size:30G
* volume_used:1.8G
* cpu_list:0-63
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=256G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1780054976
* MariaDB-1-2-1 uses docker image mariadb:11.4.7
* RAM:541006622720
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58089
* volume_size:30G
* volume_used:1.9G
* cpu_list:0-63
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=256G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1780054976
* MariaDB-1-2-2 uses docker image mariadb:11.4.7
* RAM:541006622720
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58089
* volume_size:30G
* volume_used:1.9G
* cpu_list:0-63
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=256G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1780054976
* MariaDB-1-2-3 uses docker image mariadb:11.4.7
* RAM:541006622720
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58089
* volume_size:30G
* volume_used:1.9G
* cpu_list:0-63
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=256G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1780054976
* MariaDB-1-2-4 uses docker image mariadb:11.4.7
* RAM:541006622720
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58089
* volume_size:30G
* volume_used:1.9G
* cpu_list:0-63
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=256G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1780054976
### Workflow
#### Actual
* DBMS MariaDB-1 - Pods [[1, 2, 2, 4], [1, 2, 2, 4]]
#### Planned
* DBMS MariaDB-1 - Pods [[1, 2, 2, 4], [1, 2, 2, 4]]
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:------------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:---------|:---------------|--------------:|:--------------|--------------------:|
| MariaDB-1-1 | 1 | 16 | 700.00 | 1.00 | 0.00 | 335.00 | 364.00 | 0 | 8 | | None | 0 | False | 82.29 |
| MariaDB-1-2 | 2 | 16 | 700.00 | 1.00 | 0.00 | 335.00 | 364.00 | 0 | 8 | | None | 0 | False | 82.29 |
### Execution
#### Per Connection
| DBMS | experiment_run | vusers | client | child | NOPM | TPM | efficiency | duration | errors |
|:----------------|-----------------:|---------:|---------:|--------:|-------:|------:|-------------:|-----------:|---------:|
| MariaDB-1-1-1-1 | 1 | 16 | 1 | 1 | 12123 | 28299 | 0.00 | 2 | 0 |
| MariaDB-1-1-2-1 | 1 | 16 | 2 | 1 | 9914 | 22893 | 0.00 | 2 | 0 |
| MariaDB-1-1-2-1 | 1 | 16 | 2 | 1 | 10146 | 23427 | 0.00 | 2 | 0 |
| MariaDB-1-1-3-1 | 1 | 8 | 3 | 1 | 7436 | 17246 | 0.00 | 2 | 0 |
| MariaDB-1-1-3-1 | 1 | 8 | 3 | 1 | 7440 | 17252 | 0.00 | 2 | 0 |
| MariaDB-1-1-4-1 | 1 | 8 | 4 | 1 | 6528 | 15148 | 0.00 | 2 | 0 |
| MariaDB-1-1-4-1 | 1 | 8 | 4 | 1 | 6591 | 15283 | 0.00 | 2 | 0 |
| MariaDB-1-1-4-1 | 1 | 8 | 4 | 1 | 6535 | 15169 | 0.00 | 2 | 0 |
| MariaDB-1-1-4-1 | 1 | 8 | 4 | 1 | 6560 | 15217 | 0.00 | 2 | 0 |
| MariaDB-1-2-1-1 | 2 | 16 | 1 | 1 | 9101 | 21227 | 0.00 | 2 | 0 |
| MariaDB-1-2-2-1 | 2 | 16 | 2 | 1 | 9011 | 20796 | 0.00 | 2 | 0 |
| MariaDB-1-2-2-1 | 2 | 16 | 2 | 1 | 9077 | 20938 | 0.00 | 2 | 0 |
| MariaDB-1-2-3-1 | 2 | 8 | 3 | 1 | 8350 | 19461 | 0.00 | 2 | 0 |
| MariaDB-1-2-3-1 | 2 | 8 | 3 | 1 | 8295 | 19358 | 0.00 | 2 | 0 |
| MariaDB-1-2-4-1 | 2 | 8 | 4 | 1 | 8527 | 19937 | 0.00 | 2 | 0 |
| MariaDB-1-2-4-1 | 2 | 8 | 4 | 1 | 8431 | 19717 | 0.00 | 2 | 0 |
| MariaDB-1-2-4-1 | 2 | 8 | 4 | 1 | 8498 | 19865 | 0.00 | 2 | 0 |
| MariaDB-1-2-4-1 | 2 | 8 | 4 | 1 | 8431 | 19717 | 0.00 | 2 | 0 |
#### Per Phase
| DBMS | experiment_run | vusers | client | pod_count | efficiency | NOPM | TPM | duration | errors |
|:--------------|-----------------:|---------:|---------:|------------:|-------------:|---------:|---------:|-----------:|---------:|
| MariaDB-1-1-1 | 1.00 | 16.00 | 1.00 | 1.00 | 0.00 | 12123.00 | 28299.00 | 2.00 | 0.00 |
| MariaDB-1-1-2 | 1.00 | 32.00 | 2.00 | 2.00 | 0.00 | 10030.00 | 23160.00 | 2.00 | 0.00 |
| MariaDB-1-1-3 | 1.00 | 16.00 | 3.00 | 2.00 | 0.00 | 7438.00 | 17249.00 | 2.00 | 0.00 |
| MariaDB-1-1-4 | 1.00 | 32.00 | 4.00 | 4.00 | 0.00 | 6553.50 | 15204.25 | 2.00 | 0.00 |
| MariaDB-1-2-1 | 2.00 | 16.00 | 1.00 | 1.00 | 0.00 | 9101.00 | 21227.00 | 2.00 | 0.00 |
| MariaDB-1-2-2 | 2.00 | 32.00 | 2.00 | 2.00 | 0.00 | 9044.00 | 20867.00 | 2.00 | 0.00 |
| MariaDB-1-2-3 | 2.00 | 16.00 | 3.00 | 2.00 | 0.00 | 8322.50 | 19409.50 | 2.00 | 0.00 |
| MariaDB-1-2-4 | 2.00 | 32.00 | 4.00 | 4.00 | 0.00 | 8471.75 | 19809.00 | 2.00 | 0.00 |
### Monitoring
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1 | 585.20 | 2.97 | 6.84 | 6.94 |
| MariaDB-1-1-2 | 871.45 | 6.59 | 6.95 | 7.05 |
| MariaDB-1-1-3 | 506.27 | 4.93 | 6.98 | 7.08 |
| MariaDB-1-1-4 | 940.30 | 5.49 | 7.06 | 7.16 |
| MariaDB-1-2-1 | 3027.35 | 2.36 | 7.06 | 7.16 |
| MariaDB-1-2-2 | 863.04 | 4.29 | 7.06 | 7.16 |
| MariaDB-1-2-3 | 418.66 | 2.17 | 7.09 | 7.19 |
| MariaDB-1-2-4 | 798.13 | 3.86 | 7.18 | 7.28 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1 | 24.52 | 0.13 | 0.07 | 0.07 |
| MariaDB-1-1-2 | 24.52 | 0.34 | 0.07 | 0.07 |
| MariaDB-1-1-3 | 21.49 | 0.28 | 0.07 | 0.07 |
| MariaDB-1-1-4 | 16.11 | 0.25 | 0.04 | 0.05 |
| MariaDB-1-2-1 | 22.71 | 0.13 | 0.07 | 0.07 |
| MariaDB-1-2-2 | 22.71 | 0.27 | 0.07 | 0.07 |
| MariaDB-1-2-3 | 22.87 | 0.21 | 0.07 | 0.07 |
| MariaDB-1-2-4 | 18.35 | 0.26 | 0.04 | 0.05 |
### Tests
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: NOPM contains no 0 or NaN
* TEST passed: Workflow as planned
YCSB
YCSB Loader Test for Scaling the Driver
bexhoma ycsb \
-dbms PostgreSQL \
-sf 1 \
-xwl a \
-xtb 1024 \
-nc 1 \
-ne 1 \
-nlp 4,8 \
-nlt 32,64 \
-xnlf 1 \
-nbp 1 \
-nbt 64 \
-xnbf 1 \
-ms $BEXHOMA_MS \
-tr \
-lr 64Gi \
-rr 64Gi \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_ycsb_postgresql_1.log
yields (after ca. 15 minutes) something like
testcase_ycsb_postgresql_1.log
## Show Summary
### Workload
YCSB SF=1
* Type: ycsb
* Duration: 2942s
* Code: 1782552120
* YCSB driver runs the experiment.
* This experiment compares run time and resource consumption of YCSB queries.
* Workload is 'A'.
* Number of rows to insert is 1000000.
* Ordering of inserts is hashed.
* Number of operations is 1000000.
* Batch size is ''.
* Target is based on multiples of '1024'.
* Factors for loading are [1].
* Factors for benchmarking are [1].
* Experiment uses bexhoma version 0.10.2.
* Experiment is limited to DBMS ['PostgreSQL'].
* Import is handled by 4 and 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* Database uses ephemeral storage of size 5Gi.
* Loading is tested with [32, 64] threads, split into [4, 8] pods.
* Benchmarking is tested with [64] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* PostgreSQL-1-1-1-1 uses docker image postgres:18.3
* RAM:540590804992
* CPU:AMD EPYC 7352 24-Core Processor
* Cores:96
* host:6.8.0-124-generic
* node:cl-worker24
* disk:145501
* cpu_list:0-95
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1782552120
* PostgreSQL-2-1-1-1 uses docker image postgres:18.3
* RAM:1081649803264
* CPU:AMD EPYC 7453 28-Core Processor
* Cores:56
* host:6.8.0-111-generic
* node:cl-worker34
* disk:330807
* cpu_list:0-55
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1782552120
* PostgreSQL-3-1-1-1 uses docker image postgres:18.3
* RAM:1077382602752
* CPU:AMD EPYC 7742 64-Core Processor
* Cores:256
* host:6.8.0-1052-nvidia
* node:cl-worker28
* disk:384441
* cpu_list:0-255
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1782552120
* PostgreSQL-4-1-1-1 uses docker image postgres:18.3
* RAM:1077381271552
* CPU:AMD EPYC 7742 64-Core Processor
* Cores:256
* host:6.8.0-111-generic
* node:cl-worker27
* disk:1392951
* cpu_list:0-255
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1782552120
### SUT Container Restarts
* bexhoma-sut-postgresql-1-1782552120-5f7f6fc6cb-xtg2v: 0 0
* bexhoma-sut-postgresql-2-1782552120-59796c5bc-m2scp: 0 0
* bexhoma-sut-postgresql-3-1782552120-86bd88959d-5g6j5: 0 0
* bexhoma-sut-postgresql-4-1782552120-865fb556d5-7mqzd: 0 0
### Workflow
#### Actual
* DBMS PostgreSQL-1 - Experiment 1 Client 1: ycsb (1 pods)
* DBMS PostgreSQL-2 - Experiment 1 Client 1: ycsb (1 pods)
* DBMS PostgreSQL-3 - Experiment 1 Client 1: ycsb (1 pods)
* DBMS PostgreSQL-4 - Experiment 1 Client 1: ycsb (1 pods)
#### Planned
* DBMS PostgreSQL-1 - Experiment 1 Client 1: ycsb (1 pods)
* DBMS PostgreSQL-2 - Experiment 1 Client 1: ycsb (1 pods)
* DBMS PostgreSQL-3 - Experiment 1 Client 1: ycsb (1 pods)
* DBMS PostgreSQL-4 - Experiment 1 Client 1: ycsb (1 pods)
### Loading
#### Per Connection
| connection | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) | sf | Throughput [SF/h] |
|:---------------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|-----:|--------------------:|
| PostgreSQL-1-1-0-1-1 | 1.00 | 8.00 | 256.00 | 4.00 | 0.00 | 255.94 | 976806.00 | 250000.00 | 65215.00 | 1.00 | 3.69 |
| PostgreSQL-1-1-0-1-2 | 1.00 | 8.00 | 256.00 | 4.00 | 0.00 | 255.94 | 976807.00 | 250000.00 | 65183.00 | 1.00 | 3.69 |
| PostgreSQL-1-1-0-1-3 | 1.00 | 8.00 | 256.00 | 4.00 | 0.00 | 255.94 | 976807.00 | 250000.00 | 65247.00 | 1.00 | 3.69 |
| PostgreSQL-1-1-0-1-4 | 1.00 | 8.00 | 256.00 | 4.00 | 0.00 | 255.94 | 976804.00 | 250000.00 | 66047.00 | 1.00 | 3.69 |
| PostgreSQL-2-1-0-1-1 | 1.00 | 4.00 | 128.00 | 8.00 | 0.00 | 127.97 | 976804.00 | 125000.00 | 1610.00 | 1.00 | 3.69 |
| PostgreSQL-2-1-0-1-2 | 1.00 | 4.00 | 128.00 | 8.00 | 0.00 | 127.97 | 976801.00 | 125000.00 | 1675.00 | 1.00 | 3.69 |
| PostgreSQL-2-1-0-1-3 | 1.00 | 4.00 | 128.00 | 8.00 | 0.00 | 127.97 | 976814.00 | 125000.00 | 1707.00 | 1.00 | 3.69 |
| PostgreSQL-2-1-0-1-4 | 1.00 | 4.00 | 128.00 | 8.00 | 0.00 | 127.97 | 976809.00 | 125000.00 | 1694.00 | 1.00 | 3.69 |
| PostgreSQL-2-1-0-1-5 | 1.00 | 4.00 | 128.00 | 8.00 | 0.00 | 127.97 | 976806.00 | 125000.00 | 1627.00 | 1.00 | 3.69 |
| PostgreSQL-2-1-0-1-6 | 1.00 | 4.00 | 128.00 | 8.00 | 0.00 | 127.97 | 976811.00 | 125000.00 | 1668.00 | 1.00 | 3.69 |
| PostgreSQL-2-1-0-1-7 | 1.00 | 4.00 | 128.00 | 8.00 | 0.00 | 127.97 | 976813.00 | 125000.00 | 1638.00 | 1.00 | 3.69 |
| PostgreSQL-2-1-0-1-8 | 1.00 | 4.00 | 128.00 | 8.00 | 0.00 | 127.97 | 976810.00 | 125000.00 | 1769.00 | 1.00 | 3.69 |
| PostgreSQL-3-1-0-1-1 | 1.00 | 16.00 | 256.00 | 4.00 | 0.00 | 255.92 | 976855.00 | 250000.00 | 93375.00 | 1.00 | 3.69 |
| PostgreSQL-3-1-0-1-2 | 1.00 | 16.00 | 256.00 | 4.00 | 0.00 | 255.92 | 976855.00 | 250000.00 | 93311.00 | 1.00 | 3.69 |
| PostgreSQL-3-1-0-1-3 | 1.00 | 16.00 | 256.00 | 4.00 | 0.00 | 255.92 | 976859.00 | 250000.00 | 93439.00 | 1.00 | 3.69 |
| PostgreSQL-3-1-0-1-4 | 1.00 | 16.00 | 256.00 | 4.00 | 0.00 | 255.92 | 976859.00 | 250000.00 | 93695.00 | 1.00 | 3.69 |
| PostgreSQL-4-1-0-1-1 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 127.96 | 976850.00 | 125000.00 | 206207.00 | 1.00 | 3.69 |
| PostgreSQL-4-1-0-1-2 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 127.95 | 976958.00 | 125000.00 | 206207.00 | 1.00 | 3.68 |
| PostgreSQL-4-1-0-1-3 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 127.96 | 976837.00 | 125000.00 | 205823.00 | 1.00 | 3.69 |
| PostgreSQL-4-1-0-1-4 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 127.96 | 976841.00 | 125000.00 | 206463.00 | 1.00 | 3.69 |
| PostgreSQL-4-1-0-1-5 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 127.96 | 976841.00 | 125000.00 | 207359.00 | 1.00 | 3.69 |
| PostgreSQL-4-1-0-1-6 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 127.96 | 976851.00 | 125000.00 | 206079.00 | 1.00 | 3.69 |
| PostgreSQL-4-1-0-1-7 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 127.96 | 976860.00 | 125000.00 | 205951.00 | 1.00 | 3.69 |
| PostgreSQL-4-1-0-1-8 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 127.97 | 976821.00 | 125000.00 | 207231.00 | 1.00 | 3.69 |
#### Per Run
| DBMS | experiment_run | threads | target | pod_count | exceptions | sf | Throughput [SF/h] | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) |
|:---------------|-----------------:|----------:|---------:|------------:|-------------:|-----:|--------------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|
| PostgreSQL-1-1 | 1.00 | 32.00 | 1024.00 | 4.00 | 0.00 | 1.00 | 3.69 | 1023.74 | 976807.00 | 1000000.00 | 65423.00 |
| PostgreSQL-2-1 | 1.00 | 32.00 | 1024.00 | 8.00 | 0.00 | 1.00 | 3.69 | 1023.74 | 976814.00 | 1000000.00 | 1673.50 |
| PostgreSQL-3-1 | 1.00 | 64.00 | 1024.00 | 4.00 | 0.00 | 1.00 | 3.69 | 1023.69 | 976859.00 | 1000000.00 | 93455.00 |
| PostgreSQL-4-1 | 1.00 | 64.00 | 1024.00 | 8.00 | 0.00 | 1.00 | 3.68 | 1023.69 | 976958.00 | 1000000.00 | 206415.00 |
### Execution
#### Per Connection
| DBMS | phase | job | configuration | experiment_run | client | benchmark_run | child | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:---------------------|:-----------------|:-------------------|:----------------|-----------------:|---------:|----------------:|--------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| PostgreSQL-1-1-1-1-1 | PostgreSQL-1-1-1 | PostgreSQL-1-1-1-1 | PostgreSQL-1 | 1 | 1 | 1 | 1 | 64 | 1024 | 1 | 0 | 1023.64 | 976902.00 | 499751 | 511.00 | 500249 | 71999.00 |
| PostgreSQL-2-1-1-1-1 | PostgreSQL-2-1-1 | PostgreSQL-2-1-1-1 | PostgreSQL-2 | 1 | 1 | 1 | 1 | 64 | 1024 | 1 | 0 | 1023.64 | 976903.00 | 499702 | 591.00 | 500298 | 1421.00 |
| PostgreSQL-3-1-1-1-1 | PostgreSQL-3-1-1 | PostgreSQL-3-1-1-1 | PostgreSQL-3 | 1 | 1 | 1 | 1 | 64 | 1024 | 1 | 0 | 1023.63 | 976917.00 | 500130 | 483.00 | 499870 | 106047.00 |
| PostgreSQL-4-1-1-1-1 | PostgreSQL-4-1-1 | PostgreSQL-4-1-1-1 | PostgreSQL-4 | 1 | 1 | 1 | 1 | 64 | 1024 | 1 | 0 | 1023.64 | 976902.00 | 499298 | 706.00 | 500702 | 232831.00 |
#### Per Phase
| DBMS | phase | experiment_run | threads | target | benchmark_run | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:-----------------|:-----------------|-----------------:|----------:|---------:|----------------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| PostgreSQL-1-1-1 | PostgreSQL-1-1-1 | 1 | 64 | 1024 | 1 | 1 | 0 | 1023.64 | 976902.00 | 499751 | 511.00 | 500249 | 71999.00 |
| PostgreSQL-2-1-1 | PostgreSQL-2-1-1 | 1 | 64 | 1024 | 1 | 1 | 0 | 1023.64 | 976903.00 | 499702 | 591.00 | 500298 | 1421.00 |
| PostgreSQL-3-1-1 | PostgreSQL-3-1-1 | 1 | 64 | 1024 | 1 | 1 | 0 | 1023.63 | 976917.00 | 500130 | 483.00 | 499870 | 106047.00 |
| PostgreSQL-4-1-1 | PostgreSQL-4-1-1 | 1 | 64 | 1024 | 1 | 1 | 0 | 1023.64 | 976902.00 | 499298 | 706.00 | 500702 | 232831.00 |
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Loading Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Execution Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: contains no FAILED column
YCSB Loader Test for Persistency
bexhoma ycsb \
-dbms PostgreSQL \
-sf 1 \
-xwl a \
-xtb 1024 \
-nc 2 \
-ne 1 \
-nlp 8 \
-nlt 64 \
-xnlf 1 \
-nbp 1 \
-nbt 64 \
-xnbf 1 \
-ms $BEXHOMA_MS \
-lr 64Gi \
-rr 64Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_ycsb_postgresql_2.log
yields (after ca. 10 minutes) something like
testcase_ycsb_postgresql_2.log
## Show Summary
### Workload
YCSB SF=1
* Type: ycsb
* Duration: 7051s
* Code: 1782625813
* YCSB driver runs the experiment.
* This experiment compares run time and resource consumption of YCSB queries.
* Workload is 'A'.
* Number of rows to insert is 1000000.
* Ordering of inserts is hashed.
* Number of operations is 1000000.
* Batch size is ''.
* Target is based on multiples of '1024'.
* Factors for loading are [1].
* Factors for benchmarking are [1].
* Experiment uses bexhoma version 0.10.2.
* Experiment is limited to DBMS ['PostgreSQL'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* Database is persisted to disk of type shared and size 10Gi.
* Loading is tested with [64] threads, split into [8] pods.
* Benchmarking is tested with [64] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run 2 times.
### Connections
* PostgreSQL-1-1-1-1 uses docker image postgres:18.3
* RAM:540590804992
* CPU:AMD EPYC 7352 24-Core Processor
* Cores:96
* host:6.8.0-124-generic
* node:cl-worker24
* disk:174612
* volume_size:10G
* volume_used:2.4G
* cpu_list:0-95
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1782625813
* PostgreSQL-1-2-1-1 uses docker image postgres:18.3
* RAM:540590804992
* CPU:AMD EPYC 7352 24-Core Processor
* Cores:96
* host:6.8.0-124-generic
* node:cl-worker24
* disk:170849
* volume_size:10G
* volume_used:2.9G
* cpu_list:0-95
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1782625813
### SUT Container Restarts
* bexhoma-sut-postgresql-1-1782625813-797b6c7595-hp4rp: 0 0
### Workflow
#### Actual
* DBMS PostgreSQL-1 - Experiment 1 Client 1: ycsb (1 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 1: ycsb (1 pods)
#### Planned
* DBMS PostgreSQL-1 - Experiment 1 Client 1: ycsb (1 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 1: ycsb (1 pods)
### Loading
#### Per Connection
| connection | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) | sf | Throughput [SF/h] |
|:---------------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|-----:|--------------------:|
| PostgreSQL-1-1-0-1-1 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 60.84 | 2054437.00 | 125000.00 | 804863.00 | 1.00 | 1.75 |
| PostgreSQL-1-1-0-1-2 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 60.86 | 2053808.00 | 125000.00 | 804863.00 | 1.00 | 1.75 |
| PostgreSQL-1-1-0-1-3 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 60.87 | 2053449.00 | 125000.00 | 805887.00 | 1.00 | 1.75 |
| PostgreSQL-1-1-0-1-4 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 60.84 | 2054502.00 | 125000.00 | 803839.00 | 1.00 | 1.75 |
| PostgreSQL-1-1-0-1-5 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 60.86 | 2053746.00 | 125000.00 | 806399.00 | 1.00 | 1.75 |
| PostgreSQL-1-1-0-1-6 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 60.90 | 2052683.00 | 125000.00 | 804351.00 | 1.00 | 1.75 |
| PostgreSQL-1-1-0-1-7 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 60.84 | 2054721.00 | 125000.00 | 806911.00 | 1.00 | 1.75 |
| PostgreSQL-1-1-0-1-8 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 60.87 | 2053557.00 | 125000.00 | 805887.00 | 1.00 | 1.75 |
#### Per Run
| DBMS | experiment_run | threads | target | pod_count | exceptions | sf | Throughput [SF/h] | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) |
|:---------------|-----------------:|----------:|---------:|------------:|-------------:|-----:|--------------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|
| PostgreSQL-1-1 | 1.00 | 64.00 | 1024.00 | 8.00 | 0.00 | 1.00 | 1.75 | 486.89 | 2054721.00 | 1000000.00 | 805375.00 |
### Execution
#### Per Connection
| DBMS | phase | job | configuration | experiment_run | client | benchmark_run | child | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:---------------------|:-----------------|:-------------------|:----------------|-----------------:|---------:|----------------:|--------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| PostgreSQL-1-1-1-1-1 | PostgreSQL-1-1-1 | PostgreSQL-1-1-1-1 | PostgreSQL-1 | 1 | 1 | 1 | 1 | 64 | 1024 | 1 | 0 | 517.76 | 1931414.00 | 499132 | 675.00 | 500868 | 3633151.00 |
| PostgreSQL-1-2-1-1-1 | PostgreSQL-1-2-1 | PostgreSQL-1-2-1-1 | PostgreSQL-1 | 2 | 1 | 1 | 1 | 64 | 1024 | 1 | 0 | 477.08 | 2096088.00 | 500345 | 687.00 | 499655 | 4354047.00 |
#### Per Phase
| DBMS | phase | experiment_run | threads | target | benchmark_run | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:-----------------|:-----------------|-----------------:|----------:|---------:|----------------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| PostgreSQL-1-1-1 | PostgreSQL-1-1-1 | 1 | 64 | 1024 | 1 | 1 | 0 | 517.76 | 1931414.00 | 499132 | 675.00 | 500868 | 3633151.00 |
| PostgreSQL-1-2-1 | PostgreSQL-1-2-1 | 2 | 64 | 1024 | 1 | 1 | 0 | 477.08 | 2096088.00 | 500345 | 687.00 | 499655 | 4354047.00 |
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Loading Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Execution Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: contains no FAILED column
YCSB Execution for Scaling and Repetition
bexhoma ycsb \
-dbms PostgreSQL \
-sf 1 \
-xwl a \
-xtb 1024 \
-nc 2 \
-ne 1,2 \
-nlp 8 \
-nlt 64 \
-xnlf 1 \
-nbp 1,8 \
-nbt 64 \
-xnbf 1 \
-ms $BEXHOMA_MS \
-lr 64Gi \
-rr 64Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_ycsb_postgresql_3.log
yields (after ca. 15 minutes) something like
testcase_ycsb_postgresql_3.log
## Show Summary
### Workload
YCSB SF=1
* Type: ycsb
* Duration: 24081s
* Code: 1782632881
* YCSB driver runs the experiment.
* This experiment compares run time and resource consumption of YCSB queries.
* Workload is 'A'.
* Number of rows to insert is 1000000.
* Ordering of inserts is hashed.
* Number of operations is 1000000.
* Batch size is ''.
* Target is based on multiples of '1024'.
* Factors for loading are [1].
* Factors for benchmarking are [1].
* Experiment uses bexhoma version 0.10.2.
* Experiment is limited to DBMS ['PostgreSQL'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* Database is persisted to disk of type shared and size 10Gi.
* Loading is tested with [64] threads, split into [8] pods.
* Benchmarking is tested with [64] threads, split into [1, 8] pods.
* Benchmarking is run as [1, 2] times the number of benchmarking pods.
* Experiment is run 2 times.
### Connections
* PostgreSQL-1-1-1-1 uses docker image postgres:18.3
* RAM:540590804992
* CPU:AMD EPYC 7352 24-Core Processor
* Cores:96
* host:6.8.0-124-generic
* node:cl-worker24
* disk:178701
* volume_size:10G
* volume_used:3.0G
* cpu_list:0-95
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1782632881
* PostgreSQL-1-1-2-1 uses docker image postgres:18.3
* RAM:540590804992
* CPU:AMD EPYC 7352 24-Core Processor
* Cores:96
* host:6.8.0-124-generic
* node:cl-worker24
* disk:161399
* volume_size:10G
* volume_used:3.0G
* cpu_list:0-95
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1782632881
* PostgreSQL-1-1-3-1 uses docker image postgres:18.3
* RAM:540590804992
* CPU:AMD EPYC 7352 24-Core Processor
* Cores:96
* host:6.8.0-124-generic
* node:cl-worker24
* disk:167246
* volume_size:10G
* volume_used:4.3G
* cpu_list:0-95
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1782632881
* PostgreSQL-1-1-4-1 uses docker image postgres:18.3
* RAM:540590804992
* CPU:AMD EPYC 7352 24-Core Processor
* Cores:96
* host:6.8.0-124-generic
* node:cl-worker24
* disk:167758
* volume_size:10G
* volume_used:4.7G
* cpu_list:0-95
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1782632881
* PostgreSQL-1-2-1-1 uses docker image postgres:18.3
* RAM:1077381271552
* CPU:AMD EPYC 7742 64-Core Processor
* Cores:256
* host:6.8.0-111-generic
* node:cl-worker27
* disk:1393138
* volume_size:10G
* volume_used:5.0G
* cpu_list:0-255
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1782632881
* PostgreSQL-1-2-2-1 uses docker image postgres:18.3
* RAM:1077381271552
* CPU:AMD EPYC 7742 64-Core Processor
* Cores:256
* host:6.8.0-111-generic
* node:cl-worker27
* disk:1396241
* volume_size:10G
* volume_used:5.0G
* cpu_list:0-255
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1782632881
* PostgreSQL-1-2-3-1 uses docker image postgres:18.3
* RAM:1077381271552
* CPU:AMD EPYC 7742 64-Core Processor
* Cores:256
* host:6.8.0-111-generic
* node:cl-worker27
* disk:1413952
* volume_size:10G
* volume_used:5.0G
* cpu_list:0-255
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1782632881
* PostgreSQL-1-2-4-1 uses docker image postgres:18.3
* RAM:1077381271552
* CPU:AMD EPYC 7742 64-Core Processor
* Cores:256
* host:6.8.0-111-generic
* node:cl-worker27
* disk:1383878
* volume_size:10G
* volume_used:5.0G
* cpu_list:0-255
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1782632881
### SUT Container Restarts
* bexhoma-sut-postgresql-1-1782632881-5df5b978db-k8skm: 0 0
### Workflow
#### Actual
* DBMS PostgreSQL-1 - Experiment 1 Client 1: ycsb (1 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 2: ycsb (2 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 3: ycsb (8 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 4: ycsb (16 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 1: ycsb (1 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 2: ycsb (2 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 3: ycsb (8 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 4: ycsb (16 pods)
#### Planned
* DBMS PostgreSQL-1 - Experiment 1 Client 1: ycsb (1 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 2: ycsb (2 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 3: ycsb (8 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 4: ycsb (16 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 1: ycsb (1 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 2: ycsb (2 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 3: ycsb (8 pods)
* DBMS PostgreSQL-1 - Experiment 2 Client 4: ycsb (16 pods)
### Execution
#### Per Connection
| DBMS | phase | job | configuration | experiment_run | client | benchmark_run | child | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:----------------------|:-----------------|:-------------------|:----------------|-----------------:|---------:|----------------:|--------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| PostgreSQL-1-1-1-1-1 | PostgreSQL-1-1-1 | PostgreSQL-1-1-1-1 | PostgreSQL-1 | 1 | 1 | 1 | 1 | 64 | 1024 | 1 | 0 | 409.08 | 2444507.00 | 500518 | 694.00 | 499482 | 5558271.00 |
| PostgreSQL-1-1-2-1-2 | PostgreSQL-1-1-2 | PostgreSQL-1-1-2-1 | PostgreSQL-1 | 1 | 2 | 1 | 2 | 64 | 1024 | 2 | 0 | 204.22 | 2448343.00 | 249625 | 1272.00 | 250375 | 14630911.00 |
| PostgreSQL-1-1-2-1-1 | PostgreSQL-1-1-2 | PostgreSQL-1-1-2-1 | PostgreSQL-1 | 1 | 2 | 1 | 1 | 64 | 1024 | 2 | 0 | 203.04 | 2462554.00 | 250589 | 1267.00 | 249411 | 14581759.00 |
| PostgreSQL-1-1-3-1-1 | PostgreSQL-1-1-3 | PostgreSQL-1-1-3-1 | PostgreSQL-1 | 1 | 3 | 1 | 1 | 8 | 128 | 8 | 0 | 47.51 | 2631037.00 | 62690 | 686.00 | 62310 | 5517311.00 |
| PostgreSQL-1-1-3-1-6 | PostgreSQL-1-1-3 | PostgreSQL-1-1-3-1 | PostgreSQL-1 | 1 | 3 | 1 | 6 | 8 | 128 | 8 | 0 | 47.61 | 2625729.00 | 62662 | 631.00 | 62338 | 5804031.00 |
| PostgreSQL-1-1-3-1-8 | PostgreSQL-1-1-3 | PostgreSQL-1-1-3-1 | PostgreSQL-1 | 1 | 3 | 1 | 8 | 8 | 128 | 8 | 0 | 47.35 | 2640020.00 | 63043 | 639.00 | 61957 | 5799935.00 |
| PostgreSQL-1-1-3-1-3 | PostgreSQL-1-1-3 | PostgreSQL-1-1-3-1 | PostgreSQL-1 | 1 | 3 | 1 | 3 | 8 | 128 | 8 | 0 | 47.59 | 2626391.00 | 62717 | 657.00 | 62283 | 5648383.00 |
| PostgreSQL-1-1-3-1-7 | PostgreSQL-1-1-3 | PostgreSQL-1-1-3-1 | PostgreSQL-1 | 1 | 3 | 1 | 7 | 8 | 128 | 8 | 0 | 47.27 | 2644274.00 | 62604 | 616.00 | 62396 | 5480447.00 |
| PostgreSQL-1-1-3-1-2 | PostgreSQL-1-1-3 | PostgreSQL-1-1-3-1 | PostgreSQL-1 | 1 | 3 | 1 | 2 | 8 | 128 | 8 | 0 | 47.26 | 2645004.00 | 62616 | 679.00 | 62384 | 5304319.00 |
| PostgreSQL-1-1-3-1-4 | PostgreSQL-1-1-3 | PostgreSQL-1-1-3-1 | PostgreSQL-1 | 1 | 3 | 1 | 4 | 8 | 128 | 8 | 0 | 47.58 | 2627348.00 | 62389 | 691.00 | 62611 | 5459967.00 |
| PostgreSQL-1-1-3-1-5 | PostgreSQL-1-1-3 | PostgreSQL-1-1-3-1 | PostgreSQL-1 | 1 | 3 | 1 | 5 | 8 | 128 | 8 | 0 | 47.58 | 2627366.00 | 62489 | 659.00 | 62511 | 5582847.00 |
| PostgreSQL-1-1-4-1-9 | PostgreSQL-1-1-4 | PostgreSQL-1-1-4-1 | PostgreSQL-1 | 1 | 4 | 1 | 9 | 8 | 128 | 16 | 0 | 26.83 | 2329350.00 | 31430 | 1251.00 | 31070 | 14270463.00 |
| PostgreSQL-1-1-4-1-1 | PostgreSQL-1-1-4 | PostgreSQL-1-1-4-1 | PostgreSQL-1 | 1 | 4 | 1 | 1 | 8 | 128 | 16 | 0 | 25.74 | 2428147.00 | 31171 | 1215.00 | 31329 | 13180927.00 |
| PostgreSQL-1-1-4-1-2 | PostgreSQL-1-1-4 | PostgreSQL-1-1-4-1 | PostgreSQL-1 | 1 | 4 | 1 | 2 | 8 | 128 | 16 | 0 | 25.74 | 2428251.00 | 31245 | 1262.00 | 31255 | 14041087.00 |
| PostgreSQL-1-1-4-1-12 | PostgreSQL-1-1-4 | PostgreSQL-1-1-4-1 | PostgreSQL-1 | 1 | 4 | 1 | 12 | 8 | 128 | 16 | 0 | 25.75 | 2427551.00 | 31277 | 1242.00 | 31223 | 12746751.00 |
| PostgreSQL-1-1-4-1-13 | PostgreSQL-1-1-4 | PostgreSQL-1-1-4-1 | PostgreSQL-1 | 1 | 4 | 1 | 13 | 8 | 128 | 16 | 0 | 25.75 | 2427478.00 | 31434 | 1263.00 | 31066 | 14163967.00 |
| PostgreSQL-1-1-4-1-11 | PostgreSQL-1-1-4 | PostgreSQL-1-1-4-1 | PostgreSQL-1 | 1 | 4 | 1 | 11 | 8 | 128 | 16 | 0 | 26.45 | 2362761.00 | 31153 | 1240.00 | 31347 | 13877247.00 |
| PostgreSQL-1-1-4-1-5 | PostgreSQL-1-1-4 | PostgreSQL-1-1-4-1 | PostgreSQL-1 | 1 | 4 | 1 | 5 | 8 | 128 | 16 | 0 | 26.35 | 2372150.00 | 31182 | 1252.00 | 31318 | 13221887.00 |
| PostgreSQL-1-1-4-1-4 | PostgreSQL-1-1-4 | PostgreSQL-1-1-4-1 | PostgreSQL-1 | 1 | 4 | 1 | 4 | 8 | 128 | 16 | 0 | 24.95 | 2505089.00 | 30954 | 1223.00 | 31546 | 13484031.00 |
| PostgreSQL-1-1-4-1-3 | PostgreSQL-1-1-4 | PostgreSQL-1-1-4-1 | PostgreSQL-1 | 1 | 4 | 1 | 3 | 8 | 128 | 16 | 0 | 25.91 | 2412041.00 | 31168 | 1208.00 | 31332 | 13721599.00 |
| PostgreSQL-1-1-4-1-10 | PostgreSQL-1-1-4 | PostgreSQL-1-1-4-1 | PostgreSQL-1 | 1 | 4 | 1 | 10 | 8 | 128 | 16 | 0 | 26.21 | 2385031.00 | 31157 | 1235.00 | 31343 | 13148159.00 |
| PostgreSQL-1-1-4-1-6 | PostgreSQL-1-1-4 | PostgreSQL-1-1-4-1 | PostgreSQL-1 | 1 | 4 | 1 | 6 | 8 | 128 | 16 | 0 | 25.46 | 2454429.00 | 31178 | 1215.00 | 31322 | 13811711.00 |
| PostgreSQL-1-1-4-1-14 | PostgreSQL-1-1-4 | PostgreSQL-1-1-4-1 | PostgreSQL-1 | 1 | 4 | 1 | 14 | 8 | 128 | 16 | 0 | 25.67 | 2435068.00 | 31253 | 1239.00 | 31247 | 15745023.00 |
| PostgreSQL-1-1-4-1-16 | PostgreSQL-1-1-4 | PostgreSQL-1-1-4-1 | PostgreSQL-1 | 1 | 4 | 1 | 16 | 8 | 128 | 16 | 0 | 25.75 | 2427155.00 | 31391 | 1232.00 | 31109 | 14540799.00 |
| PostgreSQL-1-1-4-1-15 | PostgreSQL-1-1-4 | PostgreSQL-1-1-4-1 | PostgreSQL-1 | 1 | 4 | 1 | 15 | 8 | 128 | 16 | 0 | 25.28 | 2472055.00 | 31007 | 1229.00 | 31493 | 14401535.00 |
| PostgreSQL-1-1-4-1-8 | PostgreSQL-1-1-4 | PostgreSQL-1-1-4-1 | PostgreSQL-1 | 1 | 4 | 1 | 8 | 8 | 128 | 16 | 0 | 25.44 | 2457206.00 | 31160 | 1244.00 | 31340 | 14352383.00 |
| PostgreSQL-1-1-4-1-7 | PostgreSQL-1-1-4 | PostgreSQL-1-1-4-1 | PostgreSQL-1 | 1 | 4 | 1 | 7 | 8 | 128 | 16 | 0 | 26.94 | 2320320.00 | 31419 | 1229.00 | 31081 | 14131199.00 |
| PostgreSQL-1-2-4-1-12 | PostgreSQL-1-2-4 | PostgreSQL-1-2-4-1 | PostgreSQL-1 | 2 | 4 | 1 | 12 | 8 | 128 | 16 | 0 | 31.23 | 2001503.00 | 31090 | 922.00 | 31410 | 12017663.00 |
| PostgreSQL-1-2-4-1-2 | PostgreSQL-1-2-4 | PostgreSQL-1-2-4-1 | PostgreSQL-1 | 2 | 4 | 1 | 2 | 8 | 128 | 16 | 0 | 30.96 | 2019013.00 | 31098 | 946.00 | 31402 | 11911167.00 |
| PostgreSQL-1-2-4-1-16 | PostgreSQL-1-2-4 | PostgreSQL-1-2-4-1 | PostgreSQL-1 | 2 | 4 | 1 | 16 | 8 | 128 | 16 | 0 | 30.73 | 2033755.00 | 31395 | 915.00 | 31105 | 12967935.00 |
| PostgreSQL-1-2-4-1-6 | PostgreSQL-1-2-4 | PostgreSQL-1-2-4-1 | PostgreSQL-1 | 2 | 4 | 1 | 6 | 8 | 128 | 16 | 0 | 30.76 | 2031871.00 | 31317 | 952.00 | 31183 | 13271039.00 |
| PostgreSQL-1-2-4-1-15 | PostgreSQL-1-2-4 | PostgreSQL-1-2-4-1 | PostgreSQL-1 | 2 | 4 | 1 | 15 | 8 | 128 | 16 | 0 | 31.54 | 1981363.00 | 31070 | 959.00 | 31430 | 12337151.00 |
| PostgreSQL-1-2-4-1-8 | PostgreSQL-1-2-4 | PostgreSQL-1-2-4-1 | PostgreSQL-1 | 2 | 4 | 1 | 8 | 8 | 128 | 16 | 0 | 30.73 | 2033902.00 | 31079 | 966.00 | 31421 | 13336575.00 |
| PostgreSQL-1-2-4-1-14 | PostgreSQL-1-2-4 | PostgreSQL-1-2-4-1 | PostgreSQL-1 | 2 | 4 | 1 | 14 | 8 | 128 | 16 | 0 | 31.09 | 2009972.00 | 31089 | 925.00 | 31411 | 11952127.00 |
| PostgreSQL-1-2-4-1-5 | PostgreSQL-1-2-4 | PostgreSQL-1-2-4-1 | PostgreSQL-1 | 2 | 4 | 1 | 5 | 8 | 128 | 16 | 0 | 31.13 | 2007880.00 | 31192 | 909.00 | 31308 | 12754943.00 |
| PostgreSQL-1-2-4-1-1 | PostgreSQL-1-2-4 | PostgreSQL-1-2-4-1 | PostgreSQL-1 | 2 | 4 | 1 | 1 | 8 | 128 | 16 | 0 | 31.45 | 1987359.00 | 31317 | 959.00 | 31183 | 11460607.00 |
| PostgreSQL-1-2-4-1-11 | PostgreSQL-1-2-4 | PostgreSQL-1-2-4-1 | PostgreSQL-1 | 2 | 4 | 1 | 11 | 8 | 128 | 16 | 0 | 31.13 | 2007546.00 | 31233 | 992.00 | 31267 | 12976127.00 |
| PostgreSQL-1-2-4-1-13 | PostgreSQL-1-2-4 | PostgreSQL-1-2-4-1 | PostgreSQL-1 | 2 | 4 | 1 | 13 | 8 | 128 | 16 | 0 | 30.72 | 2034498.00 | 31362 | 961.00 | 31138 | 12525567.00 |
| PostgreSQL-1-2-4-1-10 | PostgreSQL-1-2-4 | PostgreSQL-1-2-4-1 | PostgreSQL-1 | 2 | 4 | 1 | 10 | 8 | 128 | 16 | 0 | 31.19 | 2003693.00 | 31226 | 926.00 | 31274 | 11616255.00 |
| PostgreSQL-1-2-4-1-9 | PostgreSQL-1-2-4 | PostgreSQL-1-2-4-1 | PostgreSQL-1 | 2 | 4 | 1 | 9 | 8 | 128 | 16 | 0 | 31.35 | 1993781.00 | 31256 | 961.00 | 31244 | 12296191.00 |
| PostgreSQL-1-2-4-1-3 | PostgreSQL-1-2-4 | PostgreSQL-1-2-4-1 | PostgreSQL-1 | 2 | 4 | 1 | 3 | 8 | 128 | 16 | 0 | 31.14 | 2007356.00 | 31252 | 947.00 | 31248 | 12287999.00 |
| PostgreSQL-1-2-1-1-1 | PostgreSQL-1-2-1 | PostgreSQL-1-2-1-1 | PostgreSQL-1 | 2 | 1 | 1 | 1 | 64 | 1024 | 1 | 0 | 293.80 | 3403654.00 | 500362 | 550.00 | 499638 | 6004735.00 |
| PostgreSQL-1-2-2-1-1 | PostgreSQL-1-2-2 | PostgreSQL-1-2-2-1 | PostgreSQL-1 | 2 | 2 | 1 | 1 | 64 | 1024 | 2 | 0 | 194.12 | 2575755.00 | 249767 | 973.00 | 250233 | 13049855.00 |
| PostgreSQL-1-2-2-1-2 | PostgreSQL-1-2-2 | PostgreSQL-1-2-2-1 | PostgreSQL-1 | 2 | 2 | 1 | 2 | 64 | 1024 | 2 | 0 | 192.69 | 2594816.00 | 250075 | 976.00 | 249925 | 13762559.00 |
| PostgreSQL-1-2-3-1-8 | PostgreSQL-1-2-3 | PostgreSQL-1-2-3-1 | PostgreSQL-1 | 2 | 3 | 1 | 8 | 8 | 128 | 8 | 0 | 47.44 | 2634641.00 | 62675 | 536.00 | 62325 | 5615615.00 |
| PostgreSQL-1-2-3-1-5 | PostgreSQL-1-2-3 | PostgreSQL-1-2-3-1 | PostgreSQL-1 | 2 | 3 | 1 | 5 | 8 | 128 | 8 | 0 | 47.23 | 2646444.00 | 62631 | 555.00 | 62369 | 5484543.00 |
| PostgreSQL-1-2-3-1-2 | PostgreSQL-1-2-3 | PostgreSQL-1-2-3-1 | PostgreSQL-1 | 2 | 3 | 1 | 2 | 8 | 128 | 8 | 0 | 50.45 | 2477686.00 | 62650 | 571.00 | 62350 | 5505023.00 |
| PostgreSQL-1-2-3-1-3 | PostgreSQL-1-2-3 | PostgreSQL-1-2-3-1 | PostgreSQL-1 | 2 | 3 | 1 | 3 | 8 | 128 | 8 | 0 | 49.98 | 2501034.00 | 62716 | 536.00 | 62284 | 5603327.00 |
| PostgreSQL-1-2-3-1-7 | PostgreSQL-1-2-3 | PostgreSQL-1-2-3-1 | PostgreSQL-1 | 2 | 3 | 1 | 7 | 8 | 128 | 8 | 0 | 48.39 | 2583337.00 | 62631 | 526.00 | 62369 | 5476351.00 |
| PostgreSQL-1-2-3-1-6 | PostgreSQL-1-2-3 | PostgreSQL-1-2-3-1 | PostgreSQL-1 | 2 | 3 | 1 | 6 | 8 | 128 | 8 | 0 | 51.43 | 2430276.00 | 62580 | 546.00 | 62420 | 5427199.00 |
| PostgreSQL-1-2-3-1-1 | PostgreSQL-1-2-3 | PostgreSQL-1-2-3-1 | PostgreSQL-1 | 2 | 3 | 1 | 1 | 8 | 128 | 8 | 0 | 47.90 | 2609558.00 | 62552 | 540.00 | 62448 | 5595135.00 |
| PostgreSQL-1-2-3-1-4 | PostgreSQL-1-2-3 | PostgreSQL-1-2-3-1 | PostgreSQL-1 | 2 | 3 | 1 | 4 | 8 | 128 | 8 | 0 | 49.77 | 2511498.00 | 62489 | 548.00 | 62511 | 5779455.00 |
| PostgreSQL-1-2-4-1-7 | PostgreSQL-1-2-4 | PostgreSQL-1-2-4-1 | PostgreSQL-1 | 2 | 4 | 1 | 7 | 8 | 128 | 16 | 0 | 31.51 | 1983676.00 | 31276 | 960.00 | 31224 | 12287999.00 |
| PostgreSQL-1-2-4-1-4 | PostgreSQL-1-2-4 | PostgreSQL-1-2-4-1 | PostgreSQL-1 | 2 | 4 | 1 | 4 | 8 | 128 | 16 | 0 | 31.45 | 1987123.00 | 31319 | 923.00 | 31181 | 11968511.00 |
#### Per Phase
| DBMS | phase | experiment_run | threads | target | benchmark_run | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:-----------------|:-----------------|-----------------:|----------:|---------:|----------------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| PostgreSQL-1-1-1 | PostgreSQL-1-1-1 | 1 | 64 | 1024 | 1 | 1 | 0 | 409.08 | 2444507.00 | 500518 | 694.00 | 499482 | 5558271.00 |
| PostgreSQL-1-1-2 | PostgreSQL-1-1-2 | 1 | 128 | 2048 | 1 | 2 | 0 | 407.26 | 2462554.00 | 500214 | 1272.00 | 499786 | 14630911.00 |
| PostgreSQL-1-1-3 | PostgreSQL-1-1-3 | 1 | 64 | 1024 | 1 | 8 | 0 | 379.74 | 2645004.00 | 501210 | 691.00 | 498790 | 5804031.00 |
| PostgreSQL-1-1-4 | PostgreSQL-1-1-4 | 1 | 128 | 2048 | 1 | 16 | 0 | 414.20 | 2505089.00 | 499579 | 1263.00 | 500421 | 15745023.00 |
| PostgreSQL-1-2-1 | PostgreSQL-1-2-1 | 2 | 64 | 1024 | 1 | 1 | 0 | 293.80 | 3403654.00 | 500362 | 550.00 | 499638 | 6004735.00 |
| PostgreSQL-1-2-2 | PostgreSQL-1-2-2 | 2 | 128 | 2048 | 1 | 2 | 0 | 386.81 | 2594816.00 | 499842 | 976.00 | 500158 | 13762559.00 |
| PostgreSQL-1-2-3 | PostgreSQL-1-2-3 | 2 | 64 | 1024 | 1 | 8 | 0 | 392.60 | 2646444.00 | 500924 | 571.00 | 499076 | 5779455.00 |
| PostgreSQL-1-2-4 | PostgreSQL-1-2-4 | 2 | 128 | 2048 | 1 | 16 | 0 | 498.11 | 2034498.00 | 499571 | 992.00 | 500429 | 13336575.00 |
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Execution Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: contains no FAILED column
YCSB Execution Different Workload
bexhoma ycsb \
-dbms PostgreSQL \
-sf 1 \
-xwl e \
-xtb 1024 \
-nc 1 \
-ne 1 \
-nlp 8 \
-nlt 64 \
-xnlf 1 \
-nbp 8 \
-nbt 64 \
-xnbf 1 \
-ms $BEXHOMA_MS \
-lr 64Gi \
-rr 64Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_ycsb_postgresql_4.log
yields (after ca. 5 minutes) something like
testcase_ycsb_postgresql_4.log
## Show Summary
### Workload
YCSB SF=1
Type: ycsb
Duration: 1302s
Code: 1749146590
YCSB tool runs the benchmark.
This experiment compares run time and resource consumption of YCSB queries.
Workload is 'E'.
Number of rows to insert is 1000000.
Ordering of inserts is hashed.
Number of operations is 1000000.
Batch size is ''.
Target is based on multiples of '1024'.
Factors for loading are [1].
Factors for benchmarking are [1].
Experiment uses bexhoma version 0.8.7.
Benchmark is limited to DBMS ['PostgreSQL'].
Import is handled by 8 processes (pods).
Loading is fixed to cl-worker19.
Benchmarking is fixed to cl-worker19.
SUT is fixed to cl-worker11.
Database is persisted to disk of type shared and size 50Gi.
Loading is tested with [64] threads, split into [8] pods.
Benchmarking is tested with [64] threads, split into [8] pods.
Benchmarking is run as [1] times the number of benchmarking pods.
Experiment is run once.
### Connections
PostgreSQL-64-8-1024-1 uses docker image postgres:16.1
RAM:541008568320
CPU:AMD Opteron(tm) Processor 6378
Cores:64
host:5.15.0-140-generic
node:cl-worker11
disk:359248556
datadisk:3047
volume_size:50G
volume_used:3.0G
requests_cpu:4
requests_memory:16Gi
client:1
numExperiment:1
eval_parameters
code:1749146590
### Execution
experiment_run threads target pod_count exceptions [OVERALL].Throughput(ops/sec) [OVERALL].RunTime(ms) [INSERT].Return=OK [INSERT].99thPercentileLatency(us) [SCAN].Return=OK [SCAN].99thPercentileLatency(us) [INSERT-FAILED].Operations [INSERT-FAILED].99thPercentileLatency(us)
PostgreSQL-64-8-1024-1 1 64 1024 8 0 1023.69 976884.0 49901 1124.0 949867 2323.0 232 15863.0
### Workflow
#### Actual
DBMS PostgreSQL-64-8-1024 - Pods [[8]]
#### Planned
DBMS PostgreSQL-64-8-1024 - Pods [[8]]
### Tests
TEST passed: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
TEST passed: Workflow as planned
TEST failed: Result contains FAILED column
YCSB Execution Monitoring
bexhoma ycsb \
-dbms PostgreSQL \
-sf 1 \
-xwl a \
-xtb 1024 \
-nc 1 \
-ne 1 \
-nlp 8 \
-nlt 64 \
-xnlf 1 \
-nbp 1,8 \
-nbt 64 \
-xnbf 1 \
-m \
-mc \
-ms $BEXHOMA_MS \
-lr 64Gi \
-rr 64Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_ycsb_postgresql_5.log
yields (after ca. 10 minutes) something like
testcase_ycsb_postgresql_5.log
## Show Summary
### Workload
YCSB SF=1
* Type: ycsb
* Duration: 5251s
* Code: 1782659235
* YCSB driver runs the experiment.
* This experiment compares run time and resource consumption of YCSB queries.
* Workload is 'A'.
* Number of rows to insert is 1000000.
* Ordering of inserts is hashed.
* Number of operations is 1000000.
* Batch size is ''.
* Target is based on multiples of '1024'.
* Factors for loading are [1].
* Factors for benchmarking are [1].
* Experiment uses bexhoma version 0.10.2.
* System metrics are monitored by a cluster-wide installation.
* Experiment is limited to DBMS ['PostgreSQL'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* Database is persisted to disk of type shared and size 10Gi.
* Loading is tested with [64] threads, split into [8] pods.
* Benchmarking is tested with [64] threads, split into [1, 8] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* PostgreSQL-1-1-1-1 uses docker image postgres:18.3
* RAM:1077381271552
* CPU:AMD EPYC 7742 64-Core Processor
* Cores:256
* host:6.8.0-111-generic
* node:cl-worker27
* disk:1387355
* volume_size:10G
* volume_used:4.9G
* cpu_list:0-255
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1782659235
* PostgreSQL-1-1-2-1 uses docker image postgres:18.3
* RAM:1077381271552
* CPU:AMD EPYC 7742 64-Core Processor
* Cores:256
* host:6.8.0-111-generic
* node:cl-worker27
* disk:1383455
* volume_size:10G
* volume_used:4.9G
* cpu_list:0-255
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1782659235
### SUT Container Restarts
* bexhoma-sut-postgresql-1-1782659235-5fdfc78c8f-4trxk: 0 0
### Workflow
#### Actual
* DBMS PostgreSQL-1 - Experiment 1 Client 1: ycsb (1 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 2: ycsb (8 pods)
#### Planned
* DBMS PostgreSQL-1 - Experiment 1 Client 1: ycsb (1 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 2: ycsb (8 pods)
### Execution
#### Per Connection
| DBMS | phase | job | configuration | experiment_run | client | benchmark_run | child | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:---------------------|:-----------------|:-------------------|:----------------|-----------------:|---------:|----------------:|--------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| PostgreSQL-1-1-1-1-1 | PostgreSQL-1-1-1 | PostgreSQL-1-1-1-1 | PostgreSQL-1 | 1 | 1 | 1 | 1 | 64 | 1024 | 1 | 0 | 511.19 | 1956221.00 | 499847 | 536.00 | 500153 | 4247551.00 |
| PostgreSQL-1-1-2-1-4 | PostgreSQL-1-1-2 | PostgreSQL-1-1-2-1 | PostgreSQL-1 | 1 | 2 | 1 | 4 | 8 | 128 | 8 | 0 | 61.13 | 2044871.00 | 62639 | 542.00 | 62361 | 4098047.00 |
| PostgreSQL-1-1-2-1-3 | PostgreSQL-1-1-2 | PostgreSQL-1-1-2-1 | PostgreSQL-1 | 1 | 2 | 1 | 3 | 8 | 128 | 8 | 0 | 61.17 | 2043384.00 | 62553 | 551.00 | 62447 | 3981311.00 |
| PostgreSQL-1-1-2-1-1 | PostgreSQL-1-1-2 | PostgreSQL-1-1-2-1 | PostgreSQL-1 | 1 | 2 | 1 | 1 | 8 | 128 | 8 | 0 | 61.22 | 2041922.00 | 62893 | 546.00 | 62107 | 4337663.00 |
| PostgreSQL-1-1-2-1-8 | PostgreSQL-1-1-2 | PostgreSQL-1-1-2-1 | PostgreSQL-1 | 1 | 2 | 1 | 8 | 8 | 128 | 8 | 0 | 61.18 | 2043121.00 | 62707 | 547.00 | 62293 | 4139007.00 |
| PostgreSQL-1-1-2-1-2 | PostgreSQL-1-1-2 | PostgreSQL-1-1-2-1 | PostgreSQL-1 | 1 | 2 | 1 | 2 | 8 | 128 | 8 | 0 | 62.85 | 1988752.00 | 62246 | 544.00 | 62754 | 4036607.00 |
| PostgreSQL-1-1-2-1-6 | PostgreSQL-1-1-2 | PostgreSQL-1-1-2-1 | PostgreSQL-1 | 1 | 2 | 1 | 6 | 8 | 128 | 8 | 0 | 61.15 | 2044050.00 | 62734 | 532.00 | 62266 | 4407295.00 |
| PostgreSQL-1-1-2-1-7 | PostgreSQL-1-1-2 | PostgreSQL-1-1-2-1 | PostgreSQL-1 | 1 | 2 | 1 | 7 | 8 | 128 | 8 | 0 | 61.14 | 2044435.00 | 62809 | 529.00 | 62191 | 4395007.00 |
| PostgreSQL-1-1-2-1-5 | PostgreSQL-1-1-2 | PostgreSQL-1-1-2-1 | PostgreSQL-1 | 1 | 2 | 1 | 5 | 8 | 128 | 8 | 0 | 61.11 | 2045474.00 | 62515 | 531.00 | 62485 | 4116479.00 |
#### Per Phase
| DBMS | phase | experiment_run | threads | target | benchmark_run | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:-----------------|:-----------------|-----------------:|----------:|---------:|----------------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| PostgreSQL-1-1-1 | PostgreSQL-1-1-1 | 1 | 64 | 1024 | 1 | 1 | 0 | 511.19 | 1956221.00 | 499847 | 536.00 | 500153 | 4247551.00 |
| PostgreSQL-1-1-2 | PostgreSQL-1-1-2 | 1 | 64 | 1024 | 1 | 8 | 0 | 490.96 | 2045474.00 | 501096 | 551.00 | 498904 | 4407295.00 |
### Monitoring
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 194.18 | 0.17 | 2.51 | 2.62 |
| PostgreSQL-1-1-2-1 | 190.23 | 0.22 | 2.86 | 3.08 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 94.09 | 0.25 | 0.13 | 0.13 |
| PostgreSQL-1-1-2-1 | 110.20 | 0.73 | 0.12 | 0.13 |
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: contains no FAILED column
MySQL
YCSB Loader Test for Scaling the Driver
bexhoma ycsb \
-dbms MySQL \
-sf 1 \
-xwl a \
-xtb 1024 \
-nc 1 \
-ne 1 \
-nlp 4,8 \
-nlt 32,64 \
-xnlf 1 \
-nbp 1 \
-nbt 64 \
-xnbf 1 \
-ms $BEXHOMA_MS \
-tr \
-lr 64Gi \
-rr 64Gi \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_ycsb_mysql_1.log
yields (after ca. 165 minutes) something like
testcase_ycsb_mysql_1.log
## Show Summary
### Workload
YCSB SF=1
* Type: ycsb
* Duration: 9694s
* Code: 1780391716
* YCSB driver runs the experiment.
* This experiment compares run time and resource consumption of YCSB queries.
* Workload is 'A'.
* Number of rows to insert is 1000000.
* Ordering of inserts is hashed.
* Number of operations is 1000000.
* Batch size is ''.
* Target is based on multiples of '1024'.
* Factors for loading are [1].
* Factors for benchmarking are [1].
* Experiment uses bexhoma version 0.9.9.
* Experiment is limited to DBMS ['MySQL'].
* Import is handled by 4 and 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker3.
* Loading is tested with [32, 64] threads, split into [4, 8] pods.
* Benchmarking is tested with [64] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* MySQL-1-1-1 uses docker image mysql:8.4.0
* RAM:541006622720
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:94107
* cpu_list:0-63
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1780391716
* MySQL-2-1-1 uses docker image mysql:8.4.0
* RAM:541006622720
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:94075
* cpu_list:0-63
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1780391716
* MySQL-3-1-1 uses docker image mysql:8.4.0
* RAM:541006622720
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:94108
* cpu_list:0-63
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1780391716
* MySQL-4-1-1 uses docker image mysql:8.4.0
* RAM:541006622720
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:94080
* cpu_list:0-63
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1780391716
### Workflow
#### Actual
* DBMS MySQL-1 - Pods [[1]]
* DBMS MySQL-2 - Pods [[1]]
* DBMS MySQL-3 - Pods [[1]]
* DBMS MySQL-4 - Pods [[1]]
#### Planned
* DBMS MySQL-1 - Pods [[1]]
* DBMS MySQL-2 - Pods [[1]]
* DBMS MySQL-3 - Pods [[1]]
* DBMS MySQL-4 - Pods [[1]]
### Loading
#### Per Connection
| connection | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) |
|:--------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|
| MySQL-1-1-0-1 | 1.00 | 8.00 | 256.00 | 4.00 | 0.00 | 254.24 | 983341.00 | 250000.00 | 16495.00 |
| MySQL-1-1-0-2 | 1.00 | 8.00 | 256.00 | 4.00 | 0.00 | 254.41 | 982675.00 | 250000.00 | 16559.00 |
| MySQL-1-1-0-3 | 1.00 | 8.00 | 256.00 | 4.00 | 0.00 | 254.31 | 983041.00 | 250000.00 | 16335.00 |
| MySQL-1-1-0-4 | 1.00 | 8.00 | 256.00 | 4.00 | 0.00 | 254.21 | 983433.00 | 250000.00 | 16511.00 |
| MySQL-2-1-0-1 | 1.00 | 4.00 | 128.00 | 8.00 | 0.00 | 127.90 | 977335.00 | 125000.00 | 16127.00 |
| MySQL-2-1-0-2 | 1.00 | 4.00 | 128.00 | 8.00 | 0.00 | 127.90 | 977360.00 | 125000.00 | 16135.00 |
| MySQL-2-1-0-3 | 1.00 | 4.00 | 128.00 | 8.00 | 0.00 | 127.89 | 977383.00 | 125000.00 | 16119.00 |
| MySQL-2-1-0-4 | 1.00 | 4.00 | 128.00 | 8.00 | 0.00 | 127.89 | 977431.00 | 125000.00 | 16079.00 |
| MySQL-2-1-0-5 | 1.00 | 4.00 | 128.00 | 8.00 | 0.00 | 127.89 | 977384.00 | 125000.00 | 16383.00 |
| MySQL-2-1-0-6 | 1.00 | 4.00 | 128.00 | 8.00 | 0.00 | 127.90 | 977356.00 | 125000.00 | 16255.00 |
| MySQL-2-1-0-7 | 1.00 | 4.00 | 128.00 | 8.00 | 0.00 | 127.90 | 977329.00 | 125000.00 | 16231.00 |
| MySQL-2-1-0-8 | 1.00 | 4.00 | 128.00 | 8.00 | 0.00 | 127.89 | 977377.00 | 125000.00 | 15999.00 |
| MySQL-3-1-0-1 | 1.00 | 16.00 | 256.00 | 4.00 | 0.00 | 254.10 | 983854.00 | 250000.00 | 16447.00 |
| MySQL-3-1-0-2 | 1.00 | 16.00 | 256.00 | 4.00 | 0.00 | 254.26 | 983244.00 | 250000.00 | 16703.00 |
| MySQL-3-1-0-3 | 1.00 | 16.00 | 256.00 | 4.00 | 0.00 | 254.17 | 983604.00 | 250000.00 | 16575.00 |
| MySQL-3-1-0-4 | 1.00 | 16.00 | 256.00 | 4.00 | 0.00 | 254.06 | 984030.00 | 250000.00 | 16447.00 |
| MySQL-4-1-0-1 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 127.87 | 977529.00 | 125000.00 | 16575.00 |
| MySQL-4-1-0-2 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 127.88 | 977501.00 | 125000.00 | 16927.00 |
| MySQL-4-1-0-3 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 127.88 | 977470.00 | 125000.00 | 16303.00 |
| MySQL-4-1-0-4 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 127.87 | 977549.00 | 125000.00 | 16287.00 |
| MySQL-4-1-0-5 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 127.87 | 977536.00 | 125000.00 | 16327.00 |
| MySQL-4-1-0-6 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 127.87 | 977566.00 | 125000.00 | 16143.00 |
| MySQL-4-1-0-7 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 127.88 | 977473.00 | 125000.00 | 16623.00 |
| MySQL-4-1-0-8 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 127.85 | 977696.00 | 125000.00 | 16167.00 |
#### Per Run
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) |
|:----------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|
| MySQL-1-1 | 1.00 | 32.00 | 1024.00 | 4.00 | 0.00 | 1017.17 | 983433.00 | 1000000.00 | 16475.00 |
| MySQL-2-1 | 1.00 | 32.00 | 1024.00 | 8.00 | 0.00 | 1023.15 | 977431.00 | 1000000.00 | 16166.00 |
| MySQL-3-1 | 1.00 | 64.00 | 1024.00 | 4.00 | 0.00 | 1016.59 | 984030.00 | 1000000.00 | 16543.00 |
| MySQL-4-1 | 1.00 | 64.00 | 1024.00 | 8.00 | 0.00 | 1022.98 | 977696.00 | 1000000.00 | 16419.00 |
### Execution
#### Per Connection
| DBMS | configuration | experiment_run | client | child | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:--------------|:----------------|-----------------:|---------:|--------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| MySQL-1-1-1-1 | MySQL-1 | 1 | 1 | 1 | 64 | 1024 | 1 | 0 | 1022.03 | 978441.00 | 499891 | 1225.00 | 500109 | 14967.00 |
| MySQL-2-1-1-1 | MySQL-2 | 1 | 1 | 1 | 64 | 1024 | 1 | 0 | 1022.07 | 978404.00 | 498987 | 1218.00 | 501013 | 15103.00 |
| MySQL-3-1-1-1 | MySQL-3 | 1 | 1 | 1 | 64 | 1024 | 1 | 0 | 1022.21 | 978274.00 | 499498 | 1215.00 | 500502 | 15111.00 |
| MySQL-4-1-1-1 | MySQL-4 | 1 | 1 | 1 | 64 | 1024 | 1 | 0 | 1021.89 | 978579.00 | 500273 | 1174.00 | 499727 | 15015.00 |
#### Per Phase
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| MySQL-1-1-1 | 1.00 | 64.00 | 1024.00 | 1.00 | 0.00 | 1022.03 | 978441.00 | 499891.00 | 1225.00 | 500109.00 | 14967.00 |
| MySQL-2-1-1 | 1.00 | 64.00 | 1024.00 | 1.00 | 0.00 | 1022.07 | 978404.00 | 498987.00 | 1218.00 | 501013.00 | 15103.00 |
| MySQL-3-1-1 | 1.00 | 64.00 | 1024.00 | 1.00 | 0.00 | 1022.21 | 978274.00 | 499498.00 | 1215.00 | 500502.00 | 15111.00 |
| MySQL-4-1-1 | 1.00 | 64.00 | 1024.00 | 1.00 | 0.00 | 1021.89 | 978579.00 | 500273.00 | 1174.00 | 499727.00 | 15015.00 |
### Tests
* TEST passed: Loading Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Execution Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: contains no FAILED column
YCSB Loader Test for Persistency
bexhoma ycsb \
-dbms MySQL \
-sf 1 \
-xwl a \
-xtb 1024 \
-nc 2 \
-ne 1 \
-nlp 8 \
-nlt 64 \
-xnlf 1 \
-nbp 1 \
-nbt 64 \
-xnbf 1 \
-ms $BEXHOMA_MS \
-lr 64Gi \
-rr 64Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_ycsb_mysql_2.log
yields (after ca. 310 minutes) something like
testcase_ycsb_mysql_2.log
## Show Summary
### Workload
YCSB SF=1
* Type: ycsb
* Duration: 18369s
* Code: 1780401423
* YCSB driver runs the experiment.
* This experiment compares run time and resource consumption of YCSB queries.
* Workload is 'A'.
* Number of rows to insert is 1000000.
* Ordering of inserts is hashed.
* Number of operations is 1000000.
* Batch size is ''.
* Target is based on multiples of '1024'.
* Factors for loading are [1].
* Factors for benchmarking are [1].
* Experiment uses bexhoma version 0.9.9.
* Experiment is limited to DBMS ['MySQL'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker3.
* Database is persisted to disk of type shared and size 50Gi. Persistent storage is removed at experiment start.
* Loading is tested with [64] threads, split into [8] pods.
* Benchmarking is tested with [64] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run 2 times.
### Connections
* MySQL-1-1-1 uses docker image mysql:8.4.0
* RAM:541006622720
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58111
* volume_size:50G
* volume_used:36G
* cpu_list:0-63
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1780401423
* MySQL-1-2-1 uses docker image mysql:8.4.0
* RAM:541006622720
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58111
* volume_size:50G
* volume_used:37G
* cpu_list:0-63
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1780401423
### Workflow
#### Actual
* DBMS MySQL-1 - Pods [[1], [1]]
#### Planned
* DBMS MySQL-1 - Pods [[1], [1]]
### Loading
#### Per Connection
| connection | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) |
|:--------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|
| MySQL-1-1-0-1 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 9.02 | 13859599.00 | 125000.00 | 15671295.00 |
| MySQL-1-1-0-2 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 9.02 | 13859976.00 | 125000.00 | 15704063.00 |
| MySQL-1-1-0-3 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 9.02 | 13859372.00 | 125000.00 | 15302655.00 |
| MySQL-1-1-0-4 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 9.02 | 13860975.00 | 125000.00 | 15597567.00 |
| MySQL-1-1-0-5 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 9.03 | 13838752.00 | 125000.00 | 15359999.00 |
| MySQL-1-1-0-6 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 9.03 | 13850251.00 | 125000.00 | 15376383.00 |
| MySQL-1-1-0-7 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 9.02 | 13859977.00 | 125000.00 | 15278079.00 |
| MySQL-1-1-0-8 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 9.02 | 13861550.00 | 125000.00 | 15294463.00 |
#### Per Run
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) |
|:----------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|
| MySQL-1-1 | 1.00 | 64.00 | 1024.00 | 8.00 | 0.00 | 72.17 | 13861550.00 | 1000000.00 | 15448063.00 |
### Execution
#### Per Connection
| DBMS | configuration | experiment_run | client | child | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:--------------|:----------------|-----------------:|---------:|--------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| MySQL-1-1-1-1 | MySQL-1 | 1 | 1 | 1 | 64 | 1024 | 1 | 0 | 870.36 | 1148951.00 | 500404 | 1849.00 | 499596 | 2787327.00 |
| MySQL-1-2-1-1 | MySQL-1 | 2 | 1 | 1 | 64 | 1024 | 1 | 0 | 721.94 | 1385149.00 | 500619 | 2027.00 | 499381 | 3078143.00 |
#### Per Phase
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| MySQL-1-1-1 | 1.00 | 64.00 | 1024.00 | 1.00 | 0.00 | 870.36 | 1148951.00 | 500404.00 | 1849.00 | 499596.00 | 2787327.00 |
| MySQL-1-2-1 | 2.00 | 64.00 | 1024.00 | 1.00 | 0.00 | 721.94 | 1385149.00 | 500619.00 | 2027.00 | 499381.00 | 3078143.00 |
### Tests
* TEST passed: Loading Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Execution Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: contains no FAILED column
YCSB Execution for Scaling and Repetition
bexhoma ycsb \
-dbms MySQL \
-sf 1 \
-xwl a \
-xtb 1024 \
-nc 2 \
-ne 1,2 \
-nlp 8 \
-nlt 64 \
-xnlf 1 \
-nbp 1,8 \
-nbt 64 \
-xnbf 1 \
-ms $BEXHOMA_MS \
-lr 64Gi \
-rr 64Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_ycsb_mysql_3.log
yields (after ca. 240 minutes) something like
testcase_ycsb_mysql_3.log
## Show Summary
### Workload
YCSB SF=1
* Type: ycsb
* Duration: 14280s
* Code: 1780419828
* YCSB driver runs the experiment.
* This experiment compares run time and resource consumption of YCSB queries.
* Workload is 'A'.
* Number of rows to insert is 1000000.
* Ordering of inserts is hashed.
* Number of operations is 1000000.
* Batch size is ''.
* Target is based on multiples of '1024'.
* Factors for loading are [1].
* Factors for benchmarking are [1].
* Experiment uses bexhoma version 0.9.9.
* Experiment is limited to DBMS ['MySQL'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker3.
* Database is persisted to disk of type shared and size 50Gi.
* Loading is tested with [64] threads, split into [8] pods.
* Benchmarking is tested with [64] threads, split into [1, 8] pods.
* Benchmarking is run as [1, 2] times the number of benchmarking pods.
* Experiment is run 2 times.
### Connections
* MySQL-1-1-1 uses docker image mysql:8.4.0
* RAM:541006622720
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58111
* volume_size:50G
* volume_used:38G
* cpu_list:0-63
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1780419828
* MySQL-1-1-2 uses docker image mysql:8.4.0
* RAM:541006622720
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58111
* volume_size:50G
* volume_used:39G
* cpu_list:0-63
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1780419828
* MySQL-1-1-3 uses docker image mysql:8.4.0
* RAM:541006622720
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58111
* volume_size:50G
* volume_used:40G
* cpu_list:0-63
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1780419828
* MySQL-1-1-4 uses docker image mysql:8.4.0
* RAM:541006622720
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58112
* volume_size:50G
* volume_used:41G
* cpu_list:0-63
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1780419828
* MySQL-1-2-1 uses docker image mysql:8.4.0
* RAM:541006622720
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58111
* volume_size:50G
* volume_used:42G
* cpu_list:0-63
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1780419828
* MySQL-1-2-2 uses docker image mysql:8.4.0
* RAM:541006622720
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58112
* volume_size:50G
* volume_used:43G
* cpu_list:0-63
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1780419828
* MySQL-1-2-3 uses docker image mysql:8.4.0
* RAM:541006622720
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58112
* volume_size:50G
* volume_used:45G
* cpu_list:0-63
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1780419828
* MySQL-1-2-4 uses docker image mysql:8.4.0
* RAM:541006622720
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58112
* volume_size:50G
* volume_used:46G
* cpu_list:0-63
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1780419828
### Workflow
#### Actual
* DBMS MySQL-1 - Pods [[1, 2, 8, 16], [1, 2, 8, 16]]
#### Planned
* DBMS MySQL-1 - Pods [[1, 2, 8, 16], [1, 2, 8, 16]]
### Execution
#### Per Connection
| DBMS | configuration | experiment_run | client | child | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) | [UPDATE-FAILED].Operations | [UPDATE-FAILED].99thPercentileLatency(us) |
|:---------------|:----------------|-----------------:|---------:|--------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|-----------------------------:|--------------------------------------------:|
| MySQL-1-1-1-1 | MySQL-1 | 1 | 1 | 1 | 64 | 1024 | 1 | 0 | 799.12 | 1251370.00 | 499858 | 3715.00 | 500142 | 2648063.00 | 0 | 0.00 |
| MySQL-1-1-2-1 | MySQL-1 | 1 | 2 | 1 | 64 | 1024 | 2 | 0 | 317.95 | 1572571.00 | 249489 | 1995.00 | 250511 | 10338303.00 | 0 | 0.00 |
| MySQL-1-1-2-2 | MySQL-1 | 1 | 2 | 2 | 64 | 1024 | 2 | 0 | 317.75 | 1573578.00 | 249971 | 2020.00 | 250029 | 10264575.00 | 0 | 0.00 |
| MySQL-1-1-3-6 | MySQL-1 | 1 | 3 | 6 | 8 | 128 | 8 | 0 | 86.87 | 1438964.00 | 62549 | 2001.00 | 62451 | 3461119.00 | 0 | 0.00 |
| MySQL-1-1-3-5 | MySQL-1 | 1 | 3 | 5 | 8 | 128 | 8 | 0 | 86.65 | 1442551.00 | 62392 | 2019.00 | 62608 | 3383295.00 | 0 | 0.00 |
| MySQL-1-1-3-4 | MySQL-1 | 1 | 3 | 4 | 8 | 128 | 8 | 0 | 86.71 | 1441583.00 | 62464 | 1996.00 | 62536 | 3477503.00 | 0 | 0.00 |
| MySQL-1-1-3-3 | MySQL-1 | 1 | 3 | 3 | 8 | 128 | 8 | 0 | 86.71 | 1441629.00 | 62683 | 2003.00 | 62317 | 3522559.00 | 0 | 0.00 |
| MySQL-1-1-3-1 | MySQL-1 | 1 | 3 | 1 | 8 | 128 | 8 | 0 | 87.12 | 1434739.00 | 62433 | 1967.00 | 62567 | 3463167.00 | 0 | 0.00 |
| MySQL-1-1-3-7 | MySQL-1 | 1 | 3 | 7 | 8 | 128 | 8 | 0 | 86.74 | 1441125.00 | 62557 | 2067.00 | 62443 | 3471359.00 | 0 | 0.00 |
| MySQL-1-1-3-2 | MySQL-1 | 1 | 3 | 2 | 8 | 128 | 8 | 0 | 86.83 | 1439515.00 | 62275 | 2027.00 | 62725 | 3309567.00 | 0 | 0.00 |
| MySQL-1-1-3-8 | MySQL-1 | 1 | 3 | 8 | 8 | 128 | 8 | 0 | 86.64 | 1442721.00 | 62417 | 1967.00 | 62583 | 3383295.00 | 0 | 0.00 |
| MySQL-1-1-4-15 | MySQL-1 | 1 | 4 | 15 | 8 | 128 | 16 | 0 | 44.02 | 1419810.00 | 31207 | 1996.00 | 31293 | 9428991.00 | 0 | 0.00 |
| MySQL-1-1-4-12 | MySQL-1 | 1 | 4 | 12 | 8 | 128 | 16 | 0 | 44.06 | 1418399.00 | 31251 | 1992.00 | 31249 | 9297919.00 | 0 | 0.00 |
| MySQL-1-1-4-14 | MySQL-1 | 1 | 4 | 14 | 8 | 128 | 16 | 0 | 44.01 | 1420260.00 | 31367 | 2003.00 | 31133 | 9150463.00 | 0 | 0.00 |
| MySQL-1-1-4-16 | MySQL-1 | 1 | 4 | 16 | 8 | 128 | 16 | 0 | 43.91 | 1423264.00 | 31373 | 2010.00 | 31127 | 9191423.00 | 0 | 0.00 |
| MySQL-1-1-4-7 | MySQL-1 | 1 | 4 | 7 | 8 | 128 | 16 | 0 | 43.87 | 1424547.00 | 31150 | 1932.00 | 31350 | 9576447.00 | 0 | 0.00 |
| MySQL-1-1-4-6 | MySQL-1 | 1 | 4 | 6 | 8 | 128 | 16 | 0 | 43.94 | 1422349.00 | 31287 | 1985.00 | 31213 | 9093119.00 | 0 | 0.00 |
| MySQL-1-1-4-8 | MySQL-1 | 1 | 4 | 8 | 8 | 128 | 16 | 0 | 43.87 | 1424554.00 | 31353 | 1924.00 | 31147 | 9347071.00 | 0 | 0.00 |
| MySQL-1-1-4-13 | MySQL-1 | 1 | 4 | 13 | 8 | 128 | 16 | 0 | 43.81 | 1426530.00 | 31151 | 1978.00 | 31349 | 9682943.00 | 0 | 0.00 |
| MySQL-1-1-4-2 | MySQL-1 | 1 | 4 | 2 | 8 | 128 | 16 | 0 | 43.92 | 1422925.00 | 31166 | 2025.00 | 31334 | 9510911.00 | 0 | 0.00 |
| MySQL-1-1-4-10 | MySQL-1 | 1 | 4 | 10 | 8 | 128 | 16 | 0 | 43.91 | 1423479.00 | 31278 | 2003.00 | 31222 | 9650175.00 | 0 | 0.00 |
| MySQL-1-1-4-11 | MySQL-1 | 1 | 4 | 11 | 8 | 128 | 16 | 0 | 43.95 | 1422210.00 | 31181 | 1981.00 | 31319 | 9428991.00 | 0 | 0.00 |
| MySQL-1-1-4-3 | MySQL-1 | 1 | 4 | 3 | 8 | 128 | 16 | 0 | 43.92 | 1423130.00 | 31370 | 1924.00 | 31130 | 9437183.00 | 0 | 0.00 |
| MySQL-1-1-4-1 | MySQL-1 | 1 | 4 | 1 | 8 | 128 | 16 | 0 | 43.97 | 1421566.00 | 31036 | 1974.00 | 31464 | 9535487.00 | 0 | 0.00 |
| MySQL-1-1-4-5 | MySQL-1 | 1 | 4 | 5 | 8 | 128 | 16 | 0 | 43.82 | 1426251.00 | 31139 | 1990.00 | 31361 | 9347071.00 | 0 | 0.00 |
| MySQL-1-1-4-4 | MySQL-1 | 1 | 4 | 4 | 8 | 128 | 16 | 0 | 44.14 | 1415857.00 | 31329 | 1977.00 | 31171 | 9355263.00 | 0 | 0.00 |
| MySQL-1-1-4-9 | MySQL-1 | 1 | 4 | 9 | 8 | 128 | 16 | 0 | 43.97 | 1421577.00 | 31304 | 1916.00 | 31196 | 9347071.00 | 0 | 0.00 |
| MySQL-1-2-4-6 | MySQL-1 | 2 | 4 | 6 | 8 | 128 | 16 | 0 | 47.23 | 1323341.00 | 31286 | 1927.00 | 31214 | 7749631.00 | 0 | 0.00 |
| MySQL-1-2-4-10 | MySQL-1 | 2 | 4 | 10 | 8 | 128 | 16 | 0 | 47.09 | 1327367.00 | 30946 | 2001.00 | 31553 | 7737343.00 | 1 | 50102271.00 |
| MySQL-1-2-4-11 | MySQL-1 | 2 | 4 | 11 | 8 | 128 | 16 | 0 | 47.12 | 1326408.00 | 31134 | 1946.00 | 31365 | 7688191.00 | 1 | 50888703.00 |
| MySQL-1-2-4-4 | MySQL-1 | 2 | 4 | 4 | 8 | 128 | 16 | 0 | 47.31 | 1321003.00 | 31401 | 1920.00 | 31098 | 7692287.00 | 1 | 50888703.00 |
| MySQL-1-2-4-12 | MySQL-1 | 2 | 4 | 12 | 8 | 128 | 16 | 0 | 47.33 | 1320481.00 | 31286 | 1999.00 | 31214 | 7647231.00 | 0 | 0.00 |
| MySQL-1-2-4-16 | MySQL-1 | 2 | 4 | 16 | 8 | 128 | 16 | 0 | 47.15 | 1325516.00 | 31368 | 2042.00 | 31132 | 8011775.00 | 0 | 0.00 |
| MySQL-1-2-4-13 | MySQL-1 | 2 | 4 | 13 | 8 | 128 | 16 | 0 | 47.14 | 1325750.00 | 31170 | 1972.00 | 31330 | 7647231.00 | 0 | 0.00 |
| MySQL-1-2-4-5 | MySQL-1 | 2 | 4 | 5 | 8 | 128 | 16 | 0 | 47.18 | 1324636.00 | 31034 | 1946.00 | 31466 | 7827455.00 | 0 | 0.00 |
| MySQL-1-2-4-1 | MySQL-1 | 2 | 4 | 1 | 8 | 128 | 16 | 0 | 47.14 | 1325850.00 | 31057 | 1946.00 | 31443 | 7622655.00 | 0 | 0.00 |
| MySQL-1-2-4-7 | MySQL-1 | 2 | 4 | 7 | 8 | 128 | 16 | 0 | 47.18 | 1324723.00 | 31258 | 1944.00 | 31241 | 7651327.00 | 1 | 50167807.00 |
| MySQL-1-2-4-8 | MySQL-1 | 2 | 4 | 8 | 8 | 128 | 16 | 0 | 47.20 | 1324175.00 | 31333 | 2021.00 | 31167 | 7778303.00 | 0 | 0.00 |
| MySQL-1-2-4-3 | MySQL-1 | 2 | 4 | 3 | 8 | 128 | 16 | 0 | 47.22 | 1323649.00 | 30973 | 1989.00 | 31526 | 7770111.00 | 1 | 50102271.00 |
| MySQL-1-2-4-2 | MySQL-1 | 2 | 4 | 2 | 8 | 128 | 16 | 0 | 47.24 | 1323105.00 | 31411 | 1912.00 | 31089 | 7757823.00 | 0 | 0.00 |
| MySQL-1-2-4-9 | MySQL-1 | 2 | 4 | 9 | 8 | 128 | 16 | 0 | 47.11 | 1326685.00 | 31164 | 1924.00 | 31333 | 7704575.00 | 3 | 50364415.00 |
| MySQL-1-2-1-1 | MySQL-1 | 2 | 1 | 1 | 64 | 1024 | 1 | 0 | 673.11 | 1485637.00 | 500509 | 2063.00 | 499491 | 3090431.00 | 0 | 0.00 |
| MySQL-1-2-2-1 | MySQL-1 | 2 | 2 | 1 | 64 | 1024 | 2 | 0 | 351.74 | 1421499.00 | 249566 | 1984.00 | 250434 | 8437759.00 | 0 | 0.00 |
| MySQL-1-2-2-2 | MySQL-1 | 2 | 2 | 2 | 64 | 1024 | 2 | 0 | 351.75 | 1421453.00 | 249874 | 1988.00 | 250126 | 8536063.00 | 0 | 0.00 |
| MySQL-1-2-3-1 | MySQL-1 | 2 | 3 | 1 | 8 | 128 | 8 | 0 | 77.33 | 1616421.00 | 62630 | 1859.00 | 62370 | 3301375.00 | 0 | 0.00 |
| MySQL-1-2-3-6 | MySQL-1 | 2 | 3 | 6 | 8 | 128 | 8 | 0 | 79.69 | 1568582.00 | 62561 | 1917.00 | 62439 | 3235839.00 | 0 | 0.00 |
| MySQL-1-2-3-7 | MySQL-1 | 2 | 3 | 7 | 8 | 128 | 8 | 0 | 80.53 | 1552262.00 | 62129 | 1912.00 | 62871 | 3102719.00 | 0 | 0.00 |
| MySQL-1-2-3-2 | MySQL-1 | 2 | 3 | 2 | 8 | 128 | 8 | 0 | 80.40 | 1554681.00 | 62478 | 1911.00 | 62522 | 3170303.00 | 0 | 0.00 |
| MySQL-1-2-3-5 | MySQL-1 | 2 | 3 | 5 | 8 | 128 | 8 | 0 | 80.05 | 1561613.00 | 62351 | 1923.00 | 62649 | 3207167.00 | 0 | 0.00 |
| MySQL-1-2-3-4 | MySQL-1 | 2 | 3 | 4 | 8 | 128 | 8 | 0 | 79.21 | 1578158.00 | 62447 | 1878.00 | 62553 | 3235839.00 | 0 | 0.00 |
| MySQL-1-2-3-3 | MySQL-1 | 2 | 3 | 3 | 8 | 128 | 8 | 0 | 79.12 | 1579887.00 | 62739 | 1873.00 | 62261 | 3223551.00 | 0 | 0.00 |
| MySQL-1-2-3-8 | MySQL-1 | 2 | 3 | 8 | 8 | 128 | 8 | 0 | 80.33 | 1556101.00 | 62439 | 1948.00 | 62561 | 3172351.00 | 0 | 0.00 |
| MySQL-1-2-4-14 | MySQL-1 | 2 | 4 | 14 | 8 | 128 | 16 | 0 | 47.18 | 1324659.00 | 31349 | 1887.00 | 31151 | 7749631.00 | 0 | 0.00 |
| MySQL-1-2-4-15 | MySQL-1 | 2 | 4 | 15 | 8 | 128 | 16 | 0 | 47.10 | 1326985.00 | 31303 | 1909.00 | 31197 | 7770111.00 | 0 | 0.00 |
#### Per Phase
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) | [UPDATE-FAILED].Operations | [UPDATE-FAILED].99thPercentileLatency(us) |
|:------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|-----------------------------:|--------------------------------------------:|
| MySQL-1-1-1 | 1.00 | 64.00 | 1024.00 | 1.00 | 0.00 | 799.12 | 1251370.00 | 499858.00 | 3715.00 | 500142.00 | 2648063.00 | 0.00 | 0.00 |
| MySQL-1-1-2 | 1.00 | 128.00 | 2048.00 | 2.00 | 0.00 | 635.70 | 1573578.00 | 499460.00 | 2020.00 | 500540.00 | 10338303.00 | 0.00 | 0.00 |
| MySQL-1-1-3 | 1.00 | 64.00 | 1024.00 | 8.00 | 0.00 | 694.28 | 1442721.00 | 499770.00 | 2067.00 | 500230.00 | 3522559.00 | 0.00 | 0.00 |
| MySQL-1-1-4 | 1.00 | 128.00 | 2048.00 | 16.00 | 0.00 | 703.09 | 1426530.00 | 499942.00 | 2025.00 | 500058.00 | 9682943.00 | 0.00 | 0.00 |
| MySQL-1-2-1 | 2.00 | 64.00 | 1024.00 | 1.00 | 0.00 | 673.11 | 1485637.00 | 500509.00 | 2063.00 | 499491.00 | 3090431.00 | 0.00 | 0.00 |
| MySQL-1-2-2 | 2.00 | 128.00 | 2048.00 | 2.00 | 0.00 | 703.49 | 1421499.00 | 499440.00 | 1988.00 | 500560.00 | 8536063.00 | 0.00 | 0.00 |
| MySQL-1-2-3 | 2.00 | 64.00 | 1024.00 | 8.00 | 0.00 | 636.65 | 1616421.00 | 499774.00 | 1948.00 | 500226.00 | 3301375.00 | 0.00 | 0.00 |
| MySQL-1-2-4 | 2.00 | 128.00 | 2048.00 | 16.00 | 0.00 | 754.92 | 1327367.00 | 499473.00 | 2042.00 | 500519.00 | 8011775.00 | 8.00 | 50888703.00 |
### Tests
* TEST passed: Execution Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Workflow as planned
* TEST failed: Execution Phase: contains FAILED column
YCSB Execution Different Workload
bexhoma ycsb \
-dbms MySQL \
-sf 1 \
-xwl e \
-xtb 1024 \
-nc 1 \
-ne 1 \
-nlp 8 \
-nlt 64 \
-xnlf 1 \
-nbp 8 \
-nbt 64 \
-xnbf 1 \
-ms $BEXHOMA_MS \
-lr 64Gi \
-rr 64Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_ycsb_mysql_4.log
yields (after ca. 35 minutes) something like
testcase_ycsb_mysql_4.log
## Show Summary
### Workload
YCSB SF=1
* Type: ycsb
* Duration: 2060s
* Code: 1780434280
* YCSB driver runs the experiment.
* This experiment compares run time and resource consumption of YCSB queries.
* Workload is 'E'.
* Number of rows to insert is 1000000.
* Ordering of inserts is hashed.
* Number of operations is 1000000.
* Batch size is ''.
* Target is based on multiples of '1024'.
* Factors for loading are [1].
* Factors for benchmarking are [1].
* Experiment uses bexhoma version 0.9.9.
* Experiment is limited to DBMS ['MySQL'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker3.
* Database is persisted to disk of type shared and size 50Gi.
* Loading is tested with [64] threads, split into [8] pods.
* Benchmarking is tested with [64] threads, split into [8] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* MySQL-1-1-1 uses docker image mysql:8.4.0
* RAM:541006622720
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58112
* volume_size:50G
* volume_used:47G
* cpu_list:0-63
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1780434280
### Workflow
#### Actual
* DBMS MySQL-1 - Pods [[8]]
#### Planned
* DBMS MySQL-1 - Pods [[8]]
### Execution
#### Per Connection
| DBMS | configuration | experiment_run | client | child | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) | [SCAN].Return=OK | [SCAN].99thPercentileLatency(us) | [INSERT-FAILED].Operations | [INSERT-FAILED].99thPercentileLatency(us) |
|:--------------|:----------------|-----------------:|---------:|--------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|-------------------:|-----------------------------------:|-----------------------------:|--------------------------------------------:|
| MySQL-1-1-1-6 | MySQL-1 | 1 | 1 | 6 | 8 | 128 | 8 | 0 | 120.68 | 1035758.00 | 6204 | 500991.00 | 118796 | 9911.00 | 0 | 0.00 |
| MySQL-1-1-1-8 | MySQL-1 | 1 | 1 | 8 | 8 | 128 | 8 | 0 | 120.76 | 1035129.00 | 6294 | 499199.00 | 118706 | 10063.00 | 0 | 0.00 |
| MySQL-1-1-1-2 | MySQL-1 | 1 | 1 | 2 | 8 | 128 | 8 | 0 | 120.68 | 1035786.00 | 6163 | 520703.00 | 118837 | 9663.00 | 0 | 0.00 |
| MySQL-1-1-1-1 | MySQL-1 | 1 | 1 | 1 | 8 | 128 | 8 | 0 | 120.92 | 1033741.00 | 6156 | 499199.00 | 118844 | 9575.00 | 0 | 0.00 |
| MySQL-1-1-1-4 | MySQL-1 | 1 | 1 | 4 | 8 | 128 | 8 | 0 | 120.61 | 1036361.00 | 6206 | 506623.00 | 118794 | 10783.00 | 0 | 0.00 |
| MySQL-1-1-1-5 | MySQL-1 | 1 | 1 | 5 | 8 | 128 | 8 | 0 | 120.80 | 1034731.00 | 6250 | 503295.00 | 118666 | 10231.00 | 84 | 8019.00 |
| MySQL-1-1-1-3 | MySQL-1 | 1 | 1 | 3 | 8 | 128 | 8 | 0 | 120.77 | 1035015.00 | 6185 | 516607.00 | 118815 | 10183.00 | 0 | 0.00 |
| MySQL-1-1-1-7 | MySQL-1 | 1 | 1 | 7 | 8 | 128 | 8 | 0 | 120.68 | 1035769.00 | 6250 | 502527.00 | 118678 | 10631.00 | 72 | 17871.00 |
#### Per Phase
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) | [SCAN].Return=OK | [SCAN].99thPercentileLatency(us) | [INSERT-FAILED].Operations | [INSERT-FAILED].99thPercentileLatency(us) |
|:------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|-------------------:|-----------------------------------:|-----------------------------:|--------------------------------------------:|
| MySQL-1-1-1 | 1.00 | 64.00 | 1024.00 | 8.00 | 0.00 | 965.92 | 1036361.00 | 49708.00 | 520703.00 | 950136.00 | 10783.00 | 156.00 | 17871.00 |
### Tests
* TEST passed: Execution Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Workflow as planned
* TEST failed: Execution Phase: contains FAILED column
YCSB Execution Monitoring
bexhoma ycsb \
-dbms MySQL \
-sf 1 \
-xwl a \
-xtb 1024 \
-nc 1 \
-ne 1 \
-nlp 8 \
-nlt 64 \
-xnlf 1 \
-nbp 1,8 \
-nbt 64 \
-xnbf 1 \
-m \
-mc \
-ms $BEXHOMA_MS \
-lr 64Gi \
-rr 64Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_ycsb_mysql_5.log
yields (after ca. 70 minutes) something like
testcase_ycsb_mysql_5.log
## Show Summary
### Workload
YCSB SF=1
* Type: ycsb
* Duration: 4009s
* Code: 1780436349
* YCSB driver runs the experiment.
* This experiment compares run time and resource consumption of YCSB queries.
* Workload is 'A'.
* Number of rows to insert is 1000000.
* Ordering of inserts is hashed.
* Number of operations is 1000000.
* Batch size is ''.
* Target is based on multiples of '1024'.
* Factors for loading are [1].
* Factors for benchmarking are [1].
* Experiment uses bexhoma version 0.9.9.
* System metrics are monitored by a cluster-wide installation.
* Experiment is limited to DBMS ['MySQL'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker3.
* Database is persisted to disk of type shared and size 50Gi.
* Loading is tested with [64] threads, split into [8] pods.
* Benchmarking is tested with [64] threads, split into [1, 8] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* MySQL-1-1-1 uses docker image mysql:8.4.0
* RAM:541006622720
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58112
* volume_size:50G
* volume_used:47G
* cpu_list:0-63
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1780436349
* MySQL-1-1-2 uses docker image mysql:8.4.0
* RAM:541006622720
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58112
* volume_size:50G
* volume_used:48G
* cpu_list:0-63
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1780436349
### Workflow
#### Actual
* DBMS MySQL-1 - Pods [[1, 8]]
#### Planned
* DBMS MySQL-1 - Pods [[1, 8]]
### Execution
#### Per Connection
| DBMS | configuration | experiment_run | client | child | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:--------------|:----------------|-----------------:|---------:|--------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| MySQL-1-1-1-1 | MySQL-1 | 1 | 1 | 1 | 64 | 1024 | 1 | 0 | 652.25 | 1533148.00 | 500318 | 46815.00 | 499682 | 3174399.00 |
| MySQL-1-1-2-8 | MySQL-1 | 1 | 2 | 8 | 8 | 128 | 8 | 0 | 81.87 | 1526776.00 | 62363 | 1977.00 | 62637 | 3074047.00 |
| MySQL-1-1-2-2 | MySQL-1 | 1 | 2 | 2 | 8 | 128 | 8 | 0 | 81.80 | 1528065.00 | 62418 | 1949.00 | 62582 | 3078143.00 |
| MySQL-1-1-2-4 | MySQL-1 | 1 | 2 | 4 | 8 | 128 | 8 | 0 | 81.97 | 1524923.00 | 62545 | 1914.00 | 62455 | 3176447.00 |
| MySQL-1-1-2-6 | MySQL-1 | 1 | 2 | 6 | 8 | 128 | 8 | 0 | 82.01 | 1524218.00 | 62333 | 1935.00 | 62667 | 3194879.00 |
| MySQL-1-1-2-1 | MySQL-1 | 1 | 2 | 1 | 8 | 128 | 8 | 0 | 81.97 | 1525016.00 | 62588 | 1965.00 | 62412 | 3112959.00 |
| MySQL-1-1-2-3 | MySQL-1 | 1 | 2 | 3 | 8 | 128 | 8 | 0 | 81.88 | 1526538.00 | 62284 | 1931.00 | 62716 | 3258367.00 |
| MySQL-1-1-2-5 | MySQL-1 | 1 | 2 | 5 | 8 | 128 | 8 | 0 | 82.03 | 1523873.00 | 62546 | 1947.00 | 62454 | 3299327.00 |
| MySQL-1-1-2-7 | MySQL-1 | 1 | 2 | 7 | 8 | 128 | 8 | 0 | 81.85 | 1527182.00 | 62627 | 1922.00 | 62373 | 3217407.00 |
#### Per Phase
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| MySQL-1-1-1 | 1.00 | 64.00 | 1024.00 | 1.00 | 0.00 | 652.25 | 1533148.00 | 500318.00 | 46815.00 | 499682.00 | 3174399.00 |
| MySQL-1-1-2 | 1.00 | 64.00 | 1024.00 | 8.00 | 0.00 | 655.38 | 1528065.00 | 499704.00 | 1977.00 | 500296.00 | 3299327.00 |
### Monitoring
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1 | 934.57 | 1.20 | 9.66 | 42.24 |
| MySQL-1-1-2 | 887.17 | 1.20 | 9.68 | 43.42 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1 | 213.59 | 0.21 | 0.17 | 0.17 |
| MySQL-1-1-2 | 213.59 | 0.79 | 0.16 | 0.16 |
### Tests
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: contains no FAILED column
MariaDB
YCSB Loader Test for Scaling the Driver
bexhoma ycsb \
-dbms MariaDB \
-sf 1 \
-xwl a \
-xtb 1024 \
-nc 1 \
-ne 1 \
-nlp 4,8 \
-nlt 32,64 \
-xnlf 1 \
-nbp 1 \
-nbt 64 \
-xnbf 1 \
-ms $BEXHOMA_MS \
-tr \
-lr 64Gi \
-rr 64Gi \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_ycsb_mariadb_1.log
yields (after ca. 45 minutes) something like
testcase_ycsb_mariadb_1.log
## Show Summary
### Workload
YCSB SF=1
Type: ycsb
Duration: 2456s
Code: 1767887403
Intro: YCSB driver runs the experiment.
This experiment compares run time and resource consumption of YCSB queries.
Workload is 'A'.
Number of rows to insert is 1000000.
Ordering of inserts is hashed.
Number of operations is 1000000.
Batch size is ''.
Target is based on multiples of '1024'.
Factors for loading are [1].
Factors for benchmarking are [1].
Experiment uses bexhoma version 0.8.20.
Experiment is limited to DBMS ['MariaDB'].
Import is handled by 4 and 8 processes (pods).
Loading is fixed to cl-worker19.
Benchmarking is fixed to cl-worker19.
SUT is fixed to cl-worker21.
Loading is tested with [32, 64] threads, split into [4, 8] pods.
Benchmarking is tested with [64] threads, split into [1] pods.
Benchmarking is run as [1] times the number of benchmarking pods.
Experiment is run once.
### Connections
MariaDB-32-4-1024-1 uses docker image mariadb:11.4.7
RAM:608117153792
CPU:AMD EPYC 7542 32-Core Processor
Cores:64
host:6.8.0-90-generic
node:cl-worker21
disk:137746
datadisk:1794
cpu_list:0-63
args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=256G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
requests_cpu:4
requests_memory:16Gi
client:1
numExperiment:1
eval_parameters
code:1767887403
MariaDB-32-8-1024-1 uses docker image mariadb:11.4.7
RAM:608117153792
CPU:AMD EPYC 7542 32-Core Processor
Cores:64
host:6.8.0-90-generic
node:cl-worker21
disk:137746
datadisk:1770
cpu_list:0-63
args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=256G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
requests_cpu:4
requests_memory:16Gi
client:1
numExperiment:1
eval_parameters
code:1767887403
MariaDB-64-4-1024-1 uses docker image mariadb:11.4.7
RAM:608117153792
CPU:AMD EPYC 7542 32-Core Processor
Cores:64
host:6.8.0-90-generic
node:cl-worker21
disk:137746
datadisk:1794
cpu_list:0-63
args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=256G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
requests_cpu:4
requests_memory:16Gi
client:1
numExperiment:1
eval_parameters
code:1767887403
MariaDB-64-8-1024-1 uses docker image mariadb:11.4.7
RAM:608117153792
CPU:AMD EPYC 7542 32-Core Processor
Cores:64
host:6.8.0-90-generic
node:cl-worker21
disk:137746
datadisk:1770
cpu_list:0-63
args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=256G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
requests_cpu:4
requests_memory:16Gi
client:1
numExperiment:1
eval_parameters
code:1767887403
### Loading
experiment_run threads target pod_count exceptions [OVERALL].Throughput(ops/sec) [OVERALL].RunTime(ms) [INSERT].Return=OK [INSERT].99thPercentileLatency(us)
MariaDB-32-4-1024 1 32 1024 4 0 1023.703077 976855.0 1000000 2736.50
MariaDB-64-4-1024 1 64 1024 4 0 1023.675307 976880.0 1000000 2816.50
MariaDB-32-8-1024 1 32 1024 8 0 1023.715915 976848.0 1000000 3304.50
MariaDB-64-8-1024 1 64 1024 8 0 1023.679236 976881.0 1000000 3643.75
### Execution
experiment_run threads target pod_count exceptions [OVERALL].Throughput(ops/sec) [OVERALL].RunTime(ms) [READ].Return=OK [READ].99thPercentileLatency(us) [UPDATE].Return=OK [UPDATE].99thPercentileLatency(us)
MariaDB-32-4-1024-1 1 64 1024 1 0 1023.59 976957.0 500781 415.0 499219 431.0
MariaDB-32-8-1024-1 1 64 1024 1 0 1023.55 976996.0 500435 409.0 499565 422.0
MariaDB-64-4-1024-1 1 64 1024 1 0 1023.58 976966.0 499977 397.0 500023 402.0
MariaDB-64-8-1024-1 1 64 1024 1 0 1023.57 976969.0 500456 400.0 499544 405.0
### Workflow
#### Actual
DBMS MariaDB-64-4-1024 - Pods [[1]]
DBMS MariaDB-32-4-1024 - Pods [[1]]
DBMS MariaDB-64-8-1024 - Pods [[1]]
DBMS MariaDB-32-8-1024 - Pods [[1]]
#### Planned
DBMS MariaDB-32-4-1024 - Pods [[1]]
DBMS MariaDB-32-8-1024 - Pods [[1]]
DBMS MariaDB-64-4-1024 - Pods [[1]]
DBMS MariaDB-64-8-1024 - Pods [[1]]
### Tests
TEST passed: Loading Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
TEST passed: Execution Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
TEST passed: Workflow as planned
TEST passed: Execution Phase: contains no FAILED column
YCSB Loader Test for Persistency
bexhoma ycsb \
-dbms MariaDB \
-sf 1 \
-xwl a \
-xtb 1024 \
-nc 2 \
-ne 1 \
-nlp 8 \
-nlt 64 \
-xnlf 1 \
-nbp 1 \
-nbt 64 \
-xnbf 1 \
-ms $BEXHOMA_MS \
-lr 64Gi \
-rr 64Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_ycsb_mariadb_2.log
yields (after ca. 70 minutes) something like
testcase_ycsb_mariadb_2.log
## Show Summary
### Workload
YCSB SF=1
* Type: ycsb
* Duration: 4065s
* Code: 1780345196
* YCSB driver runs the experiment.
* This experiment compares run time and resource consumption of YCSB queries.
* Workload is 'A'.
* Number of rows to insert is 1000000.
* Ordering of inserts is hashed.
* Number of operations is 1000000.
* Batch size is ''.
* Target is based on multiples of '1024'.
* Factors for loading are [1].
* Factors for benchmarking are [1].
* Experiment uses bexhoma version 0.9.9.
* Experiment is limited to DBMS ['MariaDB'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker3.
* Database is persisted to disk of type shared and size 30Gi. Persistent storage is removed at experiment start.
* Loading is tested with [64] threads, split into [8] pods.
* Benchmarking is tested with [64] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run 2 times.
### Connections
* MariaDB-1-1-1 uses docker image mariadb:11.4.7
* RAM:541006622720
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58107
* volume_size:30G
* volume_used:1.8G
* cpu_list:0-63
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=256G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1780345196
* MariaDB-1-2-1 uses docker image mariadb:11.4.7
* RAM:541006622720
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58107
* volume_size:30G
* volume_used:1.8G
* cpu_list:0-63
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=256G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1780345196
### Workflow
#### Actual
* DBMS MariaDB-1 - Pods [[1], [1]]
#### Planned
* DBMS MariaDB-1 - Pods [[1], [1]]
### Loading
#### Per Connection
| connection | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) |
|:----------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|
| MariaDB-1-1-0-1 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 106.16 | 1177492.00 | 125000.00 | 1288191.00 |
| MariaDB-1-1-0-2 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 106.11 | 1177978.00 | 125000.00 | 1309695.00 |
| MariaDB-1-1-0-3 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 106.17 | 1177394.00 | 125000.00 | 1287167.00 |
| MariaDB-1-1-0-4 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 106.13 | 1177808.00 | 125000.00 | 1286143.00 |
| MariaDB-1-1-0-5 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 106.10 | 1178188.00 | 125000.00 | 1304575.00 |
| MariaDB-1-1-0-6 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 106.16 | 1177486.00 | 125000.00 | 1298431.00 |
| MariaDB-1-1-0-7 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 106.12 | 1177939.00 | 125000.00 | 1311743.00 |
| MariaDB-1-1-0-8 | 1.00 | 8.00 | 128.00 | 8.00 | 0.00 | 106.07 | 1178413.00 | 125000.00 | 1309695.00 |
#### Per Run
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) |
|:------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|
| MariaDB-1-1 | 1.00 | 64.00 | 1024.00 | 8.00 | 0.00 | 849.01 | 1178413.00 | 1000000.00 | 1299455.00 |
### Execution
#### Per Connection
| DBMS | configuration | experiment_run | client | child | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:----------------|:----------------|-----------------:|---------:|--------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| MariaDB-1-1-1-1 | MariaDB-1 | 1 | 1 | 1 | 64 | 1024 | 1 | 0 | 1023.26 | 977266.00 | 499819 | 1833.00 | 500181 | 229759.00 |
| MariaDB-1-2-1-1 | MariaDB-1 | 2 | 1 | 1 | 64 | 1024 | 1 | 0 | 1023.44 | 977094.00 | 499534 | 1602.00 | 500466 | 212863.00 |
#### Per Phase
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:--------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| MariaDB-1-1-1 | 1.00 | 64.00 | 1024.00 | 1.00 | 0.00 | 1023.26 | 977266.00 | 499819.00 | 1833.00 | 500181.00 | 229759.00 |
| MariaDB-1-2-1 | 2.00 | 64.00 | 1024.00 | 1.00 | 0.00 | 1023.44 | 977094.00 | 499534.00 | 1602.00 | 500466.00 | 212863.00 |
### Tests
* TEST passed: Loading Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Execution Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: contains no FAILED column
YCSB Execution for Scaling and Repetition
bexhoma ycsb \
-dbms MariaDB \
-sf 1 \
-xwl a \
-xtb 1024 \
-nc 2 \
-ne 1,2 \
-nlp 8 \
-nlt 64 \
-xnlf 1 \
-nbp 1,8 \
-nbt 64 \
-xnbf 1 \
-ms $BEXHOMA_MS \
-lr 64Gi \
-rr 64Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_ycsb_mariadb_3.log
yields (after ca. 120 minutes) something like
testcase_ycsb_mariadb_3.log
## Show Summary
### Workload
YCSB SF=1
* Type: ycsb
* Duration: 7175s
* Code: 1780349271
* YCSB driver runs the experiment.
* This experiment compares run time and resource consumption of YCSB queries.
* Workload is 'A'.
* Number of rows to insert is 1000000.
* Ordering of inserts is hashed.
* Number of operations is 1000000.
* Batch size is ''.
* Target is based on multiples of '1024'.
* Factors for loading are [1].
* Factors for benchmarking are [1].
* Experiment uses bexhoma version 0.9.9.
* Experiment is limited to DBMS ['MariaDB'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker3.
* Database is persisted to disk of type shared and size 30Gi.
* Loading is tested with [64] threads, split into [8] pods.
* Benchmarking is tested with [64] threads, split into [1, 8] pods.
* Benchmarking is run as [1, 2] times the number of benchmarking pods.
* Experiment is run 2 times.
### Connections
* MariaDB-1-1-1 uses docker image mariadb:11.4.7
* RAM:541006622720
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58107
* volume_size:30G
* volume_used:1.8G
* cpu_list:0-63
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=256G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1780349271
* MariaDB-1-1-2 uses docker image mariadb:11.4.7
* RAM:541006622720
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58107
* volume_size:30G
* volume_used:1.8G
* cpu_list:0-63
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=256G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1780349271
* MariaDB-1-1-3 uses docker image mariadb:11.4.7
* RAM:541006622720
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58107
* volume_size:30G
* volume_used:1.8G
* cpu_list:0-63
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=256G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1780349271
* MariaDB-1-1-4 uses docker image mariadb:11.4.7
* RAM:541006622720
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58107
* volume_size:30G
* volume_used:1.8G
* cpu_list:0-63
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=256G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1780349271
* MariaDB-1-2-1 uses docker image mariadb:11.4.7
* RAM:541006622720
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58107
* volume_size:30G
* volume_used:1.8G
* cpu_list:0-63
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=256G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1780349271
* MariaDB-1-2-2 uses docker image mariadb:11.4.7
* RAM:541006622720
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58107
* volume_size:30G
* volume_used:1.8G
* cpu_list:0-63
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=256G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1780349271
* MariaDB-1-2-3 uses docker image mariadb:11.4.7
* RAM:541006622720
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58107
* volume_size:30G
* volume_used:1.8G
* cpu_list:0-63
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=256G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1780349271
* MariaDB-1-2-4 uses docker image mariadb:11.4.7
* RAM:541006622720
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58107
* volume_size:30G
* volume_used:1.8G
* cpu_list:0-63
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=256G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1780349271
### Workflow
#### Actual
* DBMS MariaDB-1 - Pods [[1, 2, 8, 16], [1, 2, 8, 16]]
#### Planned
* DBMS MariaDB-1 - Pods [[1, 2, 8, 16], [1, 2, 8, 16]]
### Loading
#### Per Connection
#### Per Run
### Execution
#### Per Connection
| DBMS | configuration | experiment_run | client | child | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:-----------------|:----------------|-----------------:|---------:|--------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| MariaDB-1-1-1-1 | MariaDB-1 | 1 | 1 | 1 | 64 | 1024 | 1 | 0 | 1002.46 | 997550.00 | 500951 | 1455.00 | 499049 | 143871.00 |
| MariaDB-1-1-2-1 | MariaDB-1 | 1 | 2 | 1 | 64 | 1024 | 2 | 0 | 1023.09 | 488715.00 | 249284 | 3319.00 | 250716 | 299007.00 |
| MariaDB-1-1-2-2 | MariaDB-1 | 1 | 2 | 2 | 64 | 1024 | 2 | 0 | 1023.08 | 488719.00 | 250286 | 3287.00 | 249714 | 299007.00 |
| MariaDB-1-1-3-2 | MariaDB-1 | 1 | 3 | 2 | 8 | 128 | 8 | 0 | 127.95 | 976954.00 | 62348 | 2089.00 | 62652 | 304639.00 |
| MariaDB-1-1-3-7 | MariaDB-1 | 1 | 3 | 7 | 8 | 128 | 8 | 0 | 127.95 | 976942.00 | 62564 | 2325.00 | 62436 | 305663.00 |
| MariaDB-1-1-3-6 | MariaDB-1 | 1 | 3 | 6 | 8 | 128 | 8 | 0 | 127.96 | 976864.00 | 62447 | 2407.00 | 62553 | 304383.00 |
| MariaDB-1-1-3-5 | MariaDB-1 | 1 | 3 | 5 | 8 | 128 | 8 | 0 | 127.92 | 977171.00 | 62548 | 2335.00 | 62452 | 303615.00 |
| MariaDB-1-1-3-4 | MariaDB-1 | 1 | 3 | 4 | 8 | 128 | 8 | 0 | 127.96 | 976864.00 | 62396 | 2413.00 | 62604 | 305919.00 |
| MariaDB-1-1-3-1 | MariaDB-1 | 1 | 3 | 1 | 8 | 128 | 8 | 0 | 127.96 | 976886.00 | 62623 | 2353.00 | 62377 | 310015.00 |
| MariaDB-1-1-3-3 | MariaDB-1 | 1 | 3 | 3 | 8 | 128 | 8 | 0 | 127.96 | 976870.00 | 62348 | 2393.00 | 62652 | 305151.00 |
| MariaDB-1-1-3-8 | MariaDB-1 | 1 | 3 | 8 | 8 | 128 | 8 | 0 | 127.96 | 976863.00 | 62610 | 2303.00 | 62390 | 306431.00 |
| MariaDB-1-1-4-15 | MariaDB-1 | 1 | 4 | 15 | 8 | 128 | 16 | 0 | 127.88 | 488737.00 | 31204 | 3049.00 | 31296 | 406015.00 |
| MariaDB-1-1-4-16 | MariaDB-1 | 1 | 4 | 16 | 8 | 128 | 16 | 0 | 127.87 | 488794.00 | 31355 | 3041.00 | 31145 | 413951.00 |
| MariaDB-1-1-4-14 | MariaDB-1 | 1 | 4 | 14 | 8 | 128 | 16 | 0 | 127.91 | 488638.00 | 31159 | 3261.00 | 31341 | 416511.00 |
| MariaDB-1-1-4-4 | MariaDB-1 | 1 | 4 | 4 | 8 | 128 | 16 | 0 | 127.91 | 488627.00 | 31470 | 2983.00 | 31030 | 424191.00 |
| MariaDB-1-1-4-2 | MariaDB-1 | 1 | 4 | 2 | 8 | 128 | 16 | 0 | 127.91 | 488626.00 | 31261 | 3121.00 | 31239 | 431615.00 |
| MariaDB-1-1-4-8 | MariaDB-1 | 1 | 4 | 8 | 8 | 128 | 16 | 0 | 127.91 | 488631.00 | 31232 | 3163.00 | 31268 | 403967.00 |
| MariaDB-1-1-4-5 | MariaDB-1 | 1 | 4 | 5 | 8 | 128 | 16 | 0 | 127.89 | 488707.00 | 31283 | 3075.00 | 31217 | 404991.00 |
| MariaDB-1-1-4-13 | MariaDB-1 | 1 | 4 | 13 | 8 | 128 | 16 | 0 | 127.91 | 488617.00 | 31291 | 3079.00 | 31209 | 406527.00 |
| MariaDB-1-1-4-11 | MariaDB-1 | 1 | 4 | 11 | 8 | 128 | 16 | 0 | 127.91 | 488622.00 | 31249 | 3075.00 | 31251 | 412415.00 |
| MariaDB-1-1-4-1 | MariaDB-1 | 1 | 4 | 1 | 8 | 128 | 16 | 0 | 127.91 | 488610.00 | 31250 | 3179.00 | 31250 | 415231.00 |
| MariaDB-1-1-4-3 | MariaDB-1 | 1 | 4 | 3 | 8 | 128 | 16 | 0 | 127.91 | 488632.00 | 31171 | 3411.00 | 31329 | 418303.00 |
| MariaDB-1-1-4-6 | MariaDB-1 | 1 | 4 | 6 | 8 | 128 | 16 | 0 | 127.87 | 488793.00 | 31242 | 3169.00 | 31258 | 417023.00 |
| MariaDB-1-1-4-10 | MariaDB-1 | 1 | 4 | 10 | 8 | 128 | 16 | 0 | 127.89 | 488720.00 | 31154 | 3255.00 | 31346 | 406015.00 |
| MariaDB-1-1-4-12 | MariaDB-1 | 1 | 4 | 12 | 8 | 128 | 16 | 0 | 127.91 | 488635.00 | 31241 | 3187.00 | 31259 | 400127.00 |
| MariaDB-1-1-4-9 | MariaDB-1 | 1 | 4 | 9 | 8 | 128 | 16 | 0 | 127.91 | 488634.00 | 31322 | 3125.00 | 31178 | 397823.00 |
| MariaDB-1-1-4-7 | MariaDB-1 | 1 | 4 | 7 | 8 | 128 | 16 | 0 | 127.91 | 488631.00 | 31173 | 3031.00 | 31327 | 398335.00 |
| MariaDB-1-2-4-8 | MariaDB-1 | 2 | 4 | 8 | 8 | 128 | 16 | 0 | 127.91 | 488641.00 | 31555 | 2179.00 | 30945 | 276991.00 |
| MariaDB-1-2-4-14 | MariaDB-1 | 2 | 4 | 14 | 8 | 128 | 16 | 0 | 127.79 | 489093.00 | 31258 | 2339.00 | 31242 | 289023.00 |
| MariaDB-1-2-4-2 | MariaDB-1 | 2 | 4 | 2 | 8 | 128 | 16 | 0 | 127.78 | 489121.00 | 31369 | 2251.00 | 31131 | 278015.00 |
| MariaDB-1-2-4-13 | MariaDB-1 | 2 | 4 | 13 | 8 | 128 | 16 | 0 | 127.91 | 488619.00 | 31472 | 2311.00 | 31028 | 274687.00 |
| MariaDB-1-2-4-6 | MariaDB-1 | 2 | 4 | 6 | 8 | 128 | 16 | 0 | 127.91 | 488609.00 | 31370 | 2107.00 | 31130 | 271615.00 |
| MariaDB-1-2-4-7 | MariaDB-1 | 2 | 4 | 7 | 8 | 128 | 16 | 0 | 127.91 | 488627.00 | 31332 | 2243.00 | 31168 | 277759.00 |
| MariaDB-1-2-4-11 | MariaDB-1 | 2 | 4 | 11 | 8 | 128 | 16 | 0 | 127.91 | 488635.00 | 31340 | 2271.00 | 31160 | 276223.00 |
| MariaDB-1-2-4-3 | MariaDB-1 | 2 | 4 | 3 | 8 | 128 | 16 | 0 | 127.91 | 488614.00 | 31296 | 2323.00 | 31204 | 273407.00 |
| MariaDB-1-2-4-5 | MariaDB-1 | 2 | 4 | 5 | 8 | 128 | 16 | 0 | 127.91 | 488615.00 | 31224 | 2247.00 | 31276 | 275455.00 |
| MariaDB-1-2-4-10 | MariaDB-1 | 2 | 4 | 10 | 8 | 128 | 16 | 0 | 127.92 | 488603.00 | 31133 | 2235.00 | 31367 | 277503.00 |
| MariaDB-1-2-4-16 | MariaDB-1 | 2 | 4 | 16 | 8 | 128 | 16 | 0 | 127.91 | 488610.00 | 31219 | 2261.00 | 31281 | 282879.00 |
| MariaDB-1-2-4-12 | MariaDB-1 | 2 | 4 | 12 | 8 | 128 | 16 | 0 | 127.91 | 488629.00 | 31060 | 2203.00 | 31440 | 288511.00 |
| MariaDB-1-2-4-15 | MariaDB-1 | 2 | 4 | 15 | 8 | 128 | 16 | 0 | 127.91 | 488634.00 | 31091 | 2049.00 | 31409 | 260735.00 |
| MariaDB-1-2-4-1 | MariaDB-1 | 2 | 4 | 1 | 8 | 128 | 16 | 0 | 127.91 | 488610.00 | 31096 | 2295.00 | 31404 | 269055.00 |
| MariaDB-1-2-1-1 | MariaDB-1 | 2 | 1 | 1 | 64 | 1024 | 1 | 0 | 1023.58 | 976960.00 | 499579 | 1637.00 | 500421 | 242303.00 |
| MariaDB-1-2-2-2 | MariaDB-1 | 2 | 2 | 2 | 64 | 1024 | 2 | 0 | 1023.11 | 488708.00 | 249669 | 2597.00 | 250331 | 297727.00 |
| MariaDB-1-2-2-1 | MariaDB-1 | 2 | 2 | 1 | 64 | 1024 | 2 | 0 | 1023.10 | 488713.00 | 249729 | 2765.00 | 250271 | 297215.00 |
| MariaDB-1-2-3-5 | MariaDB-1 | 2 | 3 | 5 | 8 | 128 | 8 | 0 | 127.96 | 976858.00 | 62480 | 1990.00 | 62520 | 285439.00 |
| MariaDB-1-2-3-7 | MariaDB-1 | 2 | 3 | 7 | 8 | 128 | 8 | 0 | 127.96 | 976874.00 | 62805 | 1902.00 | 62195 | 286207.00 |
| MariaDB-1-2-3-4 | MariaDB-1 | 2 | 3 | 4 | 8 | 128 | 8 | 0 | 127.96 | 976866.00 | 62311 | 1836.00 | 62689 | 283135.00 |
| MariaDB-1-2-3-3 | MariaDB-1 | 2 | 3 | 3 | 8 | 128 | 8 | 0 | 127.96 | 976866.00 | 62568 | 1918.00 | 62432 | 286975.00 |
| MariaDB-1-2-3-1 | MariaDB-1 | 2 | 3 | 1 | 8 | 128 | 8 | 0 | 127.96 | 976881.00 | 62561 | 1886.00 | 62439 | 287743.00 |
| MariaDB-1-2-3-2 | MariaDB-1 | 2 | 3 | 2 | 8 | 128 | 8 | 0 | 127.96 | 976860.00 | 62406 | 1962.00 | 62594 | 290047.00 |
| MariaDB-1-2-3-6 | MariaDB-1 | 2 | 3 | 6 | 8 | 128 | 8 | 0 | 127.96 | 976867.00 | 62272 | 1985.00 | 62728 | 283391.00 |
| MariaDB-1-2-3-8 | MariaDB-1 | 2 | 3 | 8 | 8 | 128 | 8 | 0 | 127.96 | 976843.00 | 62658 | 1892.00 | 62342 | 283135.00 |
| MariaDB-1-2-4-4 | MariaDB-1 | 2 | 4 | 4 | 8 | 128 | 16 | 0 | 127.82 | 488982.00 | 31224 | 2263.00 | 31276 | 281343.00 |
| MariaDB-1-2-4-9 | MariaDB-1 | 2 | 4 | 9 | 8 | 128 | 16 | 0 | 127.80 | 489064.00 | 31164 | 2137.00 | 31336 | 261759.00 |
#### Per Phase
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:--------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| MariaDB-1-1-1 | 1.00 | 64.00 | 1024.00 | 1.00 | 0.00 | 1002.46 | 997550.00 | 500951.00 | 1455.00 | 499049.00 | 143871.00 |
| MariaDB-1-1-2 | 1.00 | 128.00 | 2048.00 | 2.00 | 0.00 | 2046.17 | 488719.00 | 499570.00 | 3319.00 | 500430.00 | 299007.00 |
| MariaDB-1-1-3 | 1.00 | 64.00 | 1024.00 | 8.00 | 0.00 | 1023.62 | 977171.00 | 499884.00 | 2413.00 | 500116.00 | 310015.00 |
| MariaDB-1-1-4 | 1.00 | 128.00 | 2048.00 | 16.00 | 0.00 | 2046.39 | 488794.00 | 500057.00 | 3411.00 | 499943.00 | 431615.00 |
| MariaDB-1-2-1 | 2.00 | 64.00 | 1024.00 | 1.00 | 0.00 | 1023.58 | 976960.00 | 499579.00 | 1637.00 | 500421.00 | 242303.00 |
| MariaDB-1-2-2 | 2.00 | 128.00 | 2048.00 | 2.00 | 0.00 | 2046.20 | 488713.00 | 499398.00 | 2765.00 | 500602.00 | 297727.00 |
| MariaDB-1-2-3 | 2.00 | 64.00 | 1024.00 | 8.00 | 0.00 | 1023.68 | 976881.00 | 500061.00 | 1990.00 | 499939.00 | 290047.00 |
| MariaDB-1-2-4 | 2.00 | 128.00 | 2048.00 | 16.00 | 0.00 | 2046.11 | 489121.00 | 500203.00 | 2339.00 | 499797.00 | 289023.00 |
### Tests
* TEST failed: Loading Phase: [OVERALL].Throughput(ops/sec) contains 0 or NaN
* TEST passed: Execution Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: contains no FAILED column
YCSB Execution Different Workload
bexhoma ycsb \
-dbms MariaDB \
-sf 1 \
-xwl e \
-xtb 1024 \
-nc 1 \
-ne 1 \
-nlp 8 \
-nlt 64 \
-xnlf 1 \
-nbp 8 \
-nbt 64 \
-xnbf 1 \
-ms $BEXHOMA_MS \
-lr 64Gi \
-rr 64Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_ycsb_mariadb_4.log
yields (after ca. 25 minutes) something like
testcase_ycsb_mariadb_4.log
## Show Summary
### Workload
YCSB SF=1
* Type: ycsb
* Duration: 1317s
* Code: 1780356598
* YCSB driver runs the experiment.
* This experiment compares run time and resource consumption of YCSB queries.
* Workload is 'E'.
* Number of rows to insert is 1000000.
* Ordering of inserts is hashed.
* Number of operations is 1000000.
* Batch size is ''.
* Target is based on multiples of '1024'.
* Factors for loading are [1].
* Factors for benchmarking are [1].
* Experiment uses bexhoma version 0.9.9.
* Experiment is limited to DBMS ['MariaDB'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker3.
* Database is persisted to disk of type shared and size 30Gi.
* Loading is tested with [64] threads, split into [8] pods.
* Benchmarking is tested with [64] threads, split into [8] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* MariaDB-1-1-1 uses docker image mariadb:11.4.7
* RAM:541006622720
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58107
* volume_size:30G
* volume_used:1.8G
* cpu_list:0-63
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=256G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1780356598
### Workflow
#### Actual
* DBMS MariaDB-1 - Pods [[8]]
#### Planned
* DBMS MariaDB-1 - Pods [[8]]
### Loading
#### Per Connection
#### Per Run
### Execution
#### Per Connection
| DBMS | configuration | experiment_run | client | child | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) | [SCAN].Return=OK | [SCAN].99thPercentileLatency(us) | [INSERT-FAILED].Operations | [INSERT-FAILED].99thPercentileLatency(us) |
|:----------------|:----------------|-----------------:|---------:|--------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|-------------------:|-----------------------------------:|-----------------------------:|--------------------------------------------:|
| MariaDB-1-1-1-7 | MariaDB-1 | 1 | 1 | 7 | 8 | 128 | 8 | 0 | 127.96 | 976885.00 | 6177 | 182399.00 | 118823 | 2427.00 | 0 | 0.00 |
| MariaDB-1-1-1-8 | MariaDB-1 | 1 | 1 | 8 | 8 | 128 | 8 | 0 | 127.96 | 976879.00 | 6352 | 159743.00 | 118648 | 2379.00 | 0 | 0.00 |
| MariaDB-1-1-1-6 | MariaDB-1 | 1 | 1 | 6 | 8 | 128 | 8 | 0 | 127.96 | 976871.00 | 6143 | 172927.00 | 118857 | 2359.00 | 0 | 0.00 |
| MariaDB-1-1-1-3 | MariaDB-1 | 1 | 1 | 3 | 8 | 128 | 8 | 0 | 127.96 | 976861.00 | 6250 | 154367.00 | 118695 | 2339.00 | 55 | 1623.00 |
| MariaDB-1-1-1-4 | MariaDB-1 | 1 | 1 | 4 | 8 | 128 | 8 | 0 | 127.96 | 976863.00 | 6250 | 174463.00 | 118559 | 2399.00 | 191 | 1504.00 |
| MariaDB-1-1-1-1 | MariaDB-1 | 1 | 1 | 1 | 8 | 128 | 8 | 0 | 127.96 | 976859.00 | 6250 | 155135.00 | 118605 | 2363.00 | 145 | 1684.00 |
| MariaDB-1-1-1-5 | MariaDB-1 | 1 | 1 | 5 | 8 | 128 | 8 | 0 | 127.96 | 976869.00 | 6250 | 168703.00 | 118748 | 2437.00 | 2 | 7987.00 |
| MariaDB-1-1-1-2 | MariaDB-1 | 1 | 1 | 2 | 8 | 128 | 8 | 0 | 127.96 | 976866.00 | 6232 | 171647.00 | 118768 | 2281.00 | 0 | 0.00 |
#### Per Phase
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) | [SCAN].Return=OK | [SCAN].99thPercentileLatency(us) | [INSERT-FAILED].Operations | [INSERT-FAILED].99thPercentileLatency(us) |
|:--------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|-------------------:|-----------------------------------:|-----------------------------:|--------------------------------------------:|
| MariaDB-1-1-1 | 1.00 | 64.00 | 1024.00 | 8.00 | 0.00 | 1023.68 | 976885.00 | 49904.00 | 182399.00 | 949703.00 | 2437.00 | 393.00 | 7987.00 |
### Tests
* TEST failed: Loading Phase: [OVERALL].Throughput(ops/sec) contains 0 or NaN
* TEST passed: Execution Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Workflow as planned
* TEST failed: Execution Phase: contains FAILED column
YCSB Execution Monitoring
bexhoma ycsb \
-dbms MariaDB \
-sf 1 \
-xwl a \
-xtb 1024 \
-nc 1 \
-ne 1 \
-nlp 8 \
-nlt 64 \
-xnlf 1 \
-nbp 1,8 \
-nbt 64 \
-xnbf 1 \
-m \
-mc \
-ms $BEXHOMA_MS \
-lr 64Gi \
-rr 64Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_ycsb_mariadb_5.log
yields (after ca. 45 minutes) something like
testcase_ycsb_mariadb_5.log
## Show Summary
### Workload
YCSB SF=1
* Type: ycsb
* Duration: 2404s
* Code: 1780357927
* YCSB driver runs the experiment.
* This experiment compares run time and resource consumption of YCSB queries.
* Workload is 'A'.
* Number of rows to insert is 1000000.
* Ordering of inserts is hashed.
* Number of operations is 1000000.
* Batch size is ''.
* Target is based on multiples of '1024'.
* Factors for loading are [1].
* Factors for benchmarking are [1].
* Experiment uses bexhoma version 0.9.9.
* System metrics are monitored by a cluster-wide installation.
* Experiment is limited to DBMS ['MariaDB'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker3.
* Database is persisted to disk of type shared and size 30Gi.
* Loading is tested with [64] threads, split into [8] pods.
* Benchmarking is tested with [64] threads, split into [1, 8] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* MariaDB-1-1-1 uses docker image mariadb:11.4.7
* RAM:541006622720
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58107
* volume_size:30G
* volume_used:1.9G
* cpu_list:0-63
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=256G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1780357927
* MariaDB-1-1-2 uses docker image mariadb:11.4.7
* RAM:541006622720
* CPU:AMD Opteron(tm) Processor 6378
* Cores:64
* host:6.8.0-111-generic
* node:cl-worker3
* disk:58107
* volume_size:30G
* volume_used:1.9G
* cpu_list:0-63
* args:['--max_connections=1500', '--innodb-read-io-threads=64', '--innodb-write-io-threads=64', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=256G', '--innodb-buffer-pool-chunk-size=2G', '--innodb-io-capacity=200', '--innodb-io-capacity-max=1000', '--innodb-log-buffer-size=1G', '--innodb-flush-log-at-trx-commit=2', '--sync-binlog=0', '--tmp-table-size=1G', '--max-heap-table-size=1G', '--innodb-doublewrite=0']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1780357927
### Workflow
#### Actual
* DBMS MariaDB-1 - Pods [[1, 8]]
#### Planned
* DBMS MariaDB-1 - Pods [[1, 8]]
### Loading
#### Per Connection
#### Per Run
### Execution
#### Per Connection
| DBMS | configuration | experiment_run | client | child | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:----------------|:----------------|-----------------:|---------:|--------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| MariaDB-1-1-1-1 | MariaDB-1 | 1 | 1 | 1 | 64 | 1024 | 1 | 0 | 1023.41 | 977124.00 | 499187 | 1738.00 | 500813 | 271103.00 |
| MariaDB-1-1-2-6 | MariaDB-1 | 1 | 2 | 6 | 8 | 128 | 8 | 0 | 127.96 | 976875.00 | 62590 | 2109.00 | 62410 | 342783.00 |
| MariaDB-1-1-2-2 | MariaDB-1 | 1 | 2 | 2 | 8 | 128 | 8 | 0 | 127.96 | 976849.00 | 62447 | 2097.00 | 62553 | 340479.00 |
| MariaDB-1-1-2-8 | MariaDB-1 | 1 | 2 | 8 | 8 | 128 | 8 | 0 | 127.96 | 976868.00 | 62478 | 2187.00 | 62522 | 343807.00 |
| MariaDB-1-1-2-1 | MariaDB-1 | 1 | 2 | 1 | 8 | 128 | 8 | 0 | 127.96 | 976861.00 | 62654 | 2149.00 | 62346 | 343039.00 |
| MariaDB-1-1-2-4 | MariaDB-1 | 1 | 2 | 4 | 8 | 128 | 8 | 0 | 127.96 | 976854.00 | 62596 | 2063.00 | 62404 | 345087.00 |
| MariaDB-1-1-2-3 | MariaDB-1 | 1 | 2 | 3 | 8 | 128 | 8 | 0 | 127.96 | 976867.00 | 62520 | 2129.00 | 62480 | 337407.00 |
| MariaDB-1-1-2-5 | MariaDB-1 | 1 | 2 | 5 | 8 | 128 | 8 | 0 | 127.96 | 976862.00 | 62215 | 2243.00 | 62785 | 346111.00 |
| MariaDB-1-1-2-7 | MariaDB-1 | 1 | 2 | 7 | 8 | 128 | 8 | 0 | 127.96 | 976856.00 | 62386 | 2161.00 | 62614 | 346111.00 |
#### Per Phase
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:--------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| MariaDB-1-1-1 | 1.00 | 64.00 | 1024.00 | 1.00 | 0.00 | 1023.41 | 977124.00 | 499187.00 | 1738.00 | 500813.00 | 271103.00 |
| MariaDB-1-1-2 | 1.00 | 64.00 | 1024.00 | 8.00 | 0.00 | 1023.69 | 976875.00 | 499886.00 | 2243.00 | 500114.00 | 346111.00 |
### Monitoring
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1 | 607.29 | 2.85 | 6.86 | 6.96 |
| MariaDB-1-1-2 | 545.60 | 0.72 | 6.86 | 6.96 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MariaDB-1-1-1 | 143.19 | 0.37 | 0.13 | 0.13 |
| MariaDB-1-1-2 | 143.19 | 0.63 | 0.13 | 0.13 |
### Tests
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST failed: Loading Phase: [OVERALL].Throughput(ops/sec) contains 0 or NaN
* TEST passed: Execution Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: contains no FAILED column
Hardware
Hardware runs fio, sysbench, sockperf, or netperf directly against a dedicated SUT container,
bypassing any database engine entirely — see Example-Hardware.md for what
each tool is for, how to use it in bexhoma, and typical use cases for all four tools. All sysbench,
netperf, and sockperf examples are covered there; the eight fio sweeps below are the ones that did
not fit as “typical examples” on that page (queue-depth/numjobs/block-size refinement, the
PostgreSQL-page-size depth sweep, WAL fdatasync/group-commit/record-size sweeps, checkpoint
writeback bandwidth, and the OLTP/WAL contention proxy).
The results are not official benchmark results. Exact performance depends on a number of parameters, including the underlying storage class, node hardware, and cluster load at the time of the run. These examples are solely to illustrate how to use bexhoma and show the result evaluation.
You will have to change the node selectors to names of nodes that exist in your cluster (or leave the corresponding parameters out):
BEXHOMA_NODE_SUT="cl-worker36"
BEXHOMA_NODE_BENCHMARK="cl-worker19"
LOG_DIR="./logs_tests"
BEXHOMA_MS=1
BEXHOMA_STORAGE_CLASS="shared"
mkdir -p $LOG_DIR
Fio
1. Depth-sweep refinement around the elbow
The coarse sweep above only localizes the elbow to “somewhere between 64 and 128” — each doubling step covers a wide range. This does a linear pass inside that bracket to pinpoint the actual knee instead of just the bracket containing it.
bexhoma hardware \
-dbms Hardware \
-xht fio \
-xts 4G \
-xtd 60 \
-xfrw randread,randwrite \
-xfbs 4k \
-xfid 64,80,96,112,128 \
-xfe libaio \
-nbp 1 \
-nbt 1 \
-ne 1 \
-m \
-ms $BEXHOMA_MS \
-tr \
-rsr \
-rss 50Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rnn $BEXHOMA_NODE_SUT -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_hardware_fio_depth_sweep_refine.log
10 rounds (2 patterns × 5 depths) ≈ 10 minutes.
Result
testcase_hardware_fio_depth_sweep_refine.log
## Show Summary
### Workload
Hardware Benchmark (fio)
* Type: hardware
* Duration: 1436s
* Code: 1783194420
* fio/sysbench driver runs the experiment.
* This experiment measures raw hardware I/O (fio) or CPU/memory (sysbench) performance.
* Benchmark tool: fio.
* Test file size is '4G', duration per round is 60s.
* I/O pattern(s) swept: ['randread', 'randwrite'].
* Block size(s) swept: ['4k'].
* Queue depth(s) swept: [64, 80, 96, 112, 128].
* I/O engine(s) swept: ['libaio'].
* Fsync interval(s) swept: [0].
* Fdatasync interval(s) swept: [0].
* Experiment uses bexhoma version 0.10.2.
* System metrics are monitored by sidecar containers.
* Experiment is limited to DBMS ['Hardware'].
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker36.
* Database is persisted to disk of type shared and size 50Gi. Persistent storage is removed at experiment start.
* Benchmarking is tested with [1] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* Hardware-1-1-1-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:822252
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783194420
* Hardware-1-1-10-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:830607
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783194420
* Hardware-1-1-2-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:825422
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783194420
* Hardware-1-1-3-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:830646
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783194420
* Hardware-1-1-4-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:826119
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783194420
* Hardware-1-1-5-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:821292
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783194420
* Hardware-1-1-6-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:819472
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783194420
* Hardware-1-1-7-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:819602
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783194420
* Hardware-1-1-8-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:817519
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783194420
* Hardware-1-1-9-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:827330
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783194420
### SUT Container Restarts
* bexhoma-sut-hardware-1-1783194420-65c8bf6b44-5bmdx: 0
### Workflow
#### Actual
* DBMS Hardware-1 - Experiment 1 Client 1: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 2: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 3: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 4: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 5: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 6: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 7: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 8: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 9: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 10: hardware (1 pods)
#### Planned
* DBMS Hardware-1 - Experiment 1 Client 1: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 2: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 3: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 4: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 5: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 6: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 7: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 8: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 9: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 10: hardware (1 pods)
### Execution
#### Per Connection
| DBMS | phase | job | experiment_run | client | benchmark_run | child | duration | hardware_fio_rw | hardware_fio_bs | hardware_fio_iodepth | hardware_fio_engine | hardware_fio_fsync | hardware_fio_fdatasync | hardware_fio_rwmixread | hardware_fio_numjobs | hardware_fio_read_iops | hardware_fio_write_iops | hardware_fio_read_lat_p95_ms | hardware_fio_write_lat_p95_ms | hardware_fio_read_lat_p99_ms | hardware_fio_write_lat_p99_ms | hardware_threads | hardware_sysbench_cpu_events_per_sec | hardware_sysbench_cpu_total_time_s | hardware_sysbench_cpu_lat_p95_ms | hardware_sysbench_memory_ops_per_sec | hardware_sysbench_memory_throughput_mibps | hardware_sysbench_memory_lat_p95_ms | errors |
|:--------------------|:----------------|:------------------|-----------------:|---------:|----------------:|--------:|-----------:|:------------------|:------------------|-----------------------:|:----------------------|---------------------:|-------------------------:|-------------------------:|-----------------------:|-------------------------:|--------------------------:|-------------------------------:|--------------------------------:|-------------------------------:|--------------------------------:|-------------------:|---------------------------------------:|-------------------------------------:|-----------------------------------:|---------------------------------------:|--------------------------------------------:|--------------------------------------:|---------:|
| Hardware-1-1-1-1-1 | Hardware-1-1-1 | Hardware-1-1-1-1 | 1 | 1 | 1 | 1 | 72 | randread | 4k | 64 | libaio | 0 | 0 | 50 | 1 | 3313.16 | 0.00 | 50.59 | 0.00 | 175.11 | 0.00 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-2-1-1 | Hardware-1-1-2 | Hardware-1-1-2-1 | 1 | 2 | 1 | 1 | 71 | randread | 4k | 80 | libaio | 0 | 0 | 50 | 1 | 4361.20 | 0.00 | 49.02 | 0.00 | 206.57 | 0.00 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-3-1-1 | Hardware-1-1-3 | Hardware-1-1-3-1 | 1 | 3 | 1 | 1 | 72 | randread | 4k | 96 | libaio | 0 | 0 | 50 | 1 | 4002.56 | 0.00 | 65.27 | 0.00 | 396.36 | 0.00 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-4-1-1 | Hardware-1-1-4 | Hardware-1-1-4-1 | 1 | 4 | 1 | 1 | 72 | randread | 4k | 112 | libaio | 0 | 0 | 50 | 1 | 4200.43 | 0.00 | 79.17 | 0.00 | 413.14 | 0.00 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-5-1-1 | Hardware-1-1-5 | Hardware-1-1-5-1 | 1 | 5 | 1 | 1 | 70 | randread | 4k | 128 | libaio | 0 | 0 | 50 | 1 | 5286.49 | 0.00 | 78.12 | 0.00 | 438.30 | 0.00 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-6-1-1 | Hardware-1-1-6 | Hardware-1-1-6-1 | 1 | 6 | 1 | 1 | 65 | randwrite | 4k | 64 | libaio | 0 | 0 | 50 | 1 | 0.00 | 1822.94 | 0.00 | 141.56 | 0.00 | 250.61 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-7-1-1 | Hardware-1-1-7 | Hardware-1-1-7-1 | 1 | 7 | 1 | 1 | 64 | randwrite | 4k | 80 | libaio | 0 | 0 | 50 | 1 | 0.00 | 2241.12 | 0.00 | 152.04 | 0.00 | 250.61 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-8-1-1 | Hardware-1-1-8 | Hardware-1-1-8-1 | 1 | 8 | 1 | 1 | 63 | randwrite | 4k | 96 | libaio | 0 | 0 | 50 | 1 | 0.00 | 2578.38 | 0.00 | 152.04 | 0.00 | 263.19 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-9-1-1 | Hardware-1-1-9 | Hardware-1-1-9-1 | 1 | 9 | 1 | 1 | 63 | randwrite | 4k | 112 | libaio | 0 | 0 | 50 | 1 | 0.00 | 2552.31 | 0.00 | 173.02 | 0.00 | 341.84 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-10-1-1 | Hardware-1-1-10 | Hardware-1-1-10-1 | 1 | 10 | 1 | 1 | 64 | randwrite | 4k | 128 | libaio | 0 | 0 | 50 | 1 | 0.00 | 3076.56 | 0.00 | 168.82 | 0.00 | 291.50 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
#### Per Phase
| DBMS | phase | experiment_run | client | benchmark_run | pod_count | duration | hardware_fio_rw | hardware_fio_bs | hardware_fio_iodepth | hardware_fio_engine | hardware_fio_fsync | hardware_fio_fdatasync | hardware_fio_rwmixread | hardware_fio_read_iops | hardware_fio_write_iops | hardware_fio_read_lat_p95_ms | hardware_fio_write_lat_p95_ms | hardware_fio_read_lat_p99_ms | hardware_fio_write_lat_p99_ms | hardware_threads | hardware_sysbench_cpu_events_per_sec | hardware_sysbench_cpu_total_time_s | hardware_sysbench_cpu_lat_p95_ms | hardware_sysbench_memory_ops_per_sec | hardware_sysbench_memory_throughput_mibps | hardware_sysbench_memory_lat_p95_ms | errors |
|:----------------|:----------------|-----------------:|---------:|----------------:|------------:|-----------:|:------------------|:------------------|-----------------------:|:----------------------|---------------------:|-------------------------:|-------------------------:|-------------------------:|--------------------------:|-------------------------------:|--------------------------------:|-------------------------------:|--------------------------------:|-------------------:|---------------------------------------:|-------------------------------------:|-----------------------------------:|---------------------------------------:|--------------------------------------------:|--------------------------------------:|---------:|
| Hardware-1-1-1 | Hardware-1-1-1 | 1 | 1 | 1 | 1 | 72 | randread | 4k | 64 | libaio | 0 | 0 | 50 | 3313.16 | 0.00 | 50.59 | 0.00 | 175.11 | 0.00 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-2 | Hardware-1-1-2 | 1 | 2 | 1 | 1 | 71 | randread | 4k | 80 | libaio | 0 | 0 | 50 | 4361.20 | 0.00 | 49.02 | 0.00 | 206.57 | 0.00 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-3 | Hardware-1-1-3 | 1 | 3 | 1 | 1 | 72 | randread | 4k | 96 | libaio | 0 | 0 | 50 | 4002.56 | 0.00 | 65.27 | 0.00 | 396.36 | 0.00 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-4 | Hardware-1-1-4 | 1 | 4 | 1 | 1 | 72 | randread | 4k | 112 | libaio | 0 | 0 | 50 | 4200.43 | 0.00 | 79.17 | 0.00 | 413.14 | 0.00 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-5 | Hardware-1-1-5 | 1 | 5 | 1 | 1 | 70 | randread | 4k | 128 | libaio | 0 | 0 | 50 | 5286.49 | 0.00 | 78.12 | 0.00 | 438.30 | 0.00 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-6 | Hardware-1-1-6 | 1 | 6 | 1 | 1 | 65 | randwrite | 4k | 64 | libaio | 0 | 0 | 50 | 0.00 | 1822.94 | 0.00 | 141.56 | 0.00 | 250.61 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-7 | Hardware-1-1-7 | 1 | 7 | 1 | 1 | 64 | randwrite | 4k | 80 | libaio | 0 | 0 | 50 | 0.00 | 2241.12 | 0.00 | 152.04 | 0.00 | 250.61 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-8 | Hardware-1-1-8 | 1 | 8 | 1 | 1 | 63 | randwrite | 4k | 96 | libaio | 0 | 0 | 50 | 0.00 | 2578.38 | 0.00 | 152.04 | 0.00 | 263.19 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-9 | Hardware-1-1-9 | 1 | 9 | 1 | 1 | 63 | randwrite | 4k | 112 | libaio | 0 | 0 | 50 | 0.00 | 2552.31 | 0.00 | 173.02 | 0.00 | 341.84 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-10 | Hardware-1-1-10 | 1 | 10 | 1 | 1 | 64 | randwrite | 4k | 128 | libaio | 0 | 0 | 50 | 0.00 | 3076.56 | 0.00 | 168.82 | 0.00 | 291.50 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
### Monitoring
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------|-------------:|----------:|---------------:|----------------------:|
| Hardware-1-1-1-1 | 67.31 | 2.51 | 0.02 | 0.02 |
| Hardware-1-1-2-1 | 75.69 | 4.20 | 0.02 | 0.02 |
| Hardware-1-1-3-1 | 75.00 | 3.93 | 0.03 | 2.17 |
| Hardware-1-1-4-1 | 64.23 | 1.85 | 0.02 | 0.02 |
| Hardware-1-1-5-1 | 72.15 | 2.09 | 0.03 | 4.03 |
| Hardware-1-1-6-1 | 66.85 | 3.03 | 0.02 | 0.02 |
| Hardware-1-1-7-1 | 62.88 | 2.02 | 0.02 | 0.02 |
| Hardware-1-1-8-1 | 73.73 | 2.75 | 0.02 | 0.02 |
| Hardware-1-1-9-1 | 69.06 | 2.72 | 0.02 | 0.02 |
| Hardware-1-1-10-1 | 68.84 | 1.95 | 0.02 | 0.02 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------|-------------:|----------:|---------------:|----------------------:|
| Hardware-1-1-1-1 | 0.50 | 0.02 | 0.00 | 0.00 |
| Hardware-1-1-2-1 | 0.54 | 0.00 | 0.00 | 0.00 |
| Hardware-1-1-3-1 | 0.52 | 0.02 | 0.00 | 0.00 |
| Hardware-1-1-4-1 | 0.54 | 0.00 | 0.00 | 0.00 |
| Hardware-1-1-5-1 | 0.52 | 0.00 | 0.00 | 0.00 |
| Hardware-1-1-6-1 | 0.53 | 0.01 | 0.00 | 0.00 |
| Hardware-1-1-7-1 | 0.47 | 0.01 | 0.00 | 0.00 |
| Hardware-1-1-8-1 | 0.44 | 0.01 | 0.00 | 0.00 |
| Hardware-1-1-9-1 | 0.53 | 0.00 | 0.00 | 0.00 |
| Hardware-1-1-10-1 | 0.51 | 0.01 | 0.00 | 0.00 |
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: every round has non-zero read or write IOPS
-rnn pins the SUT pod to a specific node; without it, the scheduler may place different commands
on different nodes, so results are only directly comparable across commands if -rnn is set
consistently.
2. Numjobs sweep at fixed queue depth (elbow check)
Fixes -xfid 64 (the elbow found above) and sweeps -nbt (numjobs per pod) instead of depth: if
IOPS keep climbing with more threads at the same depth, 64 was a per-queue submission limit, not
a real device ceiling; if IOPS stay flat, 64 is the actual hardware limit.
bexhoma hardware \
-dbms Hardware \
-xht fio \
-xts 4G \
-xtd 60 \
-xfrw randread,randwrite \
-xfbs 4k \
-xfid 64 \
-xfe libaio \
-nbp 1 \
-nbt 1,2,4,8,16 \
-ne 1 \
-m \
-ms $BEXHOMA_MS \
-tr \
-rsr \
-rss 80Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rnn $BEXHOMA_NODE_SUT -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_hardware_fio_numjobs_sweep.log
Result
testcase_hardware_fio_numjobs_sweep.log
## Show Summary
### Workload
Hardware Benchmark (fio)
* Type: hardware
* Duration: 1662s
* Code: 1783190728
* fio/sysbench driver runs the experiment.
* This experiment measures raw hardware I/O (fio) or CPU/memory (sysbench) performance.
* Benchmark tool: fio.
* Test file size is '4G', duration per round is 60s.
* I/O pattern(s) swept: ['randread', 'randwrite'].
* Block size(s) swept: ['4k'].
* Queue depth(s) swept: [64].
* I/O engine(s) swept: ['libaio'].
* Fsync interval(s) swept: [0].
* Fdatasync interval(s) swept: [0].
* Experiment uses bexhoma version 0.10.2.
* System metrics are monitored by sidecar containers.
* Experiment is limited to DBMS ['Hardware'].
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker36.
* Database is persisted to disk of type shared and size 80Gi. Persistent storage is removed at experiment start.
* Benchmarking is tested with [1, 2, 4, 8, 16] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* Hardware-1-1-1-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:823734
* volume_size:80.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783190728
* Hardware-1-1-10-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:815983
* volume_size:80.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783190728
* Hardware-1-1-2-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:826418
* volume_size:80.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783190728
* Hardware-1-1-3-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:821858
* volume_size:80.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783190728
* Hardware-1-1-4-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:828969
* volume_size:80.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783190728
* Hardware-1-1-5-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:821433
* volume_size:80.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783190728
* Hardware-1-1-6-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:818500
* volume_size:80.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783190728
* Hardware-1-1-7-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:827363
* volume_size:80.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783190728
* Hardware-1-1-8-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:818057
* volume_size:80.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783190728
* Hardware-1-1-9-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:821903
* volume_size:80.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783190728
### SUT Container Restarts
* bexhoma-sut-hardware-1-1783190728-748c5bf8b6-6grqm: 0
### Workflow
#### Actual
* DBMS Hardware-1 - Experiment 1 Client 1: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 2: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 3: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 4: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 5: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 6: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 7: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 8: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 9: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 10: hardware (1 pods)
#### Planned
* DBMS Hardware-1 - Experiment 1 Client 1: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 2: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 3: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 4: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 5: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 6: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 7: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 8: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 9: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 10: hardware (1 pods)
### Execution
#### Per Connection
| DBMS | phase | job | experiment_run | client | benchmark_run | child | duration | hardware_fio_rw | hardware_fio_bs | hardware_fio_iodepth | hardware_fio_engine | hardware_fio_fsync | hardware_fio_fdatasync | hardware_fio_rwmixread | hardware_fio_numjobs | hardware_fio_read_iops | hardware_fio_write_iops | hardware_fio_read_lat_p95_ms | hardware_fio_write_lat_p95_ms | hardware_fio_read_lat_p99_ms | hardware_fio_write_lat_p99_ms | hardware_threads | hardware_sysbench_cpu_events_per_sec | hardware_sysbench_cpu_total_time_s | hardware_sysbench_cpu_lat_p95_ms | hardware_sysbench_memory_ops_per_sec | hardware_sysbench_memory_throughput_mibps | hardware_sysbench_memory_lat_p95_ms | errors |
|:--------------------|:----------------|:------------------|-----------------:|---------:|----------------:|--------:|-----------:|:------------------|:------------------|-----------------------:|:----------------------|---------------------:|-------------------------:|-------------------------:|-----------------------:|-------------------------:|--------------------------:|-------------------------------:|--------------------------------:|-------------------------------:|--------------------------------:|-------------------:|---------------------------------------:|-------------------------------------:|-----------------------------------:|---------------------------------------:|--------------------------------------------:|--------------------------------------:|---------:|
| Hardware-1-1-1-1-1 | Hardware-1-1-1 | Hardware-1-1-1-1 | 1 | 1 | 1 | 1 | 71 | randread | 4k | 64 | libaio | 0 | 0 | 50 | 1 | 4338.31 | 0.00 | 54.26 | 0.00 | 152.04 | 0.00 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-2-1-1 | Hardware-1-1-2 | Hardware-1-1-2-1 | 1 | 2 | 1 | 1 | 79 | randread | 4k | 64 | libaio | 0 | 0 | 50 | 2 | 4703.97 | 0.00 | 89.65 | 0.00 | 497.03 | 0.00 | 2 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-3-1-1 | Hardware-1-1-3 | Hardware-1-1-3-1 | 1 | 3 | 1 | 1 | 94 | randread | 4k | 64 | libaio | 0 | 0 | 50 | 4 | 5166.99 | 0.00 | 113.77 | 0.00 | 952.11 | 0.00 | 4 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-4-1-1 | Hardware-1-1-4 | Hardware-1-1-4-1 | 1 | 4 | 1 | 1 | 122 | randread | 4k | 64 | libaio | 0 | 0 | 50 | 8 | 5773.17 | 0.00 | 196.08 | 0.00 | 1803.55 | 0.00 | 8 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-5-1-1 | Hardware-1-1-5 | Hardware-1-1-5-1 | 1 | 5 | 1 | 1 | 187 | randread | 4k | 64 | libaio | 0 | 0 | 50 | 16 | 5295.18 | 0.00 | 1333.79 | 0.00 | 4731.17 | 0.00 | 16 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-6-1-1 | Hardware-1-1-6 | Hardware-1-1-6-1 | 1 | 6 | 1 | 1 | 63 | randwrite | 4k | 64 | libaio | 0 | 0 | 50 | 1 | 0.00 | 1878.38 | 0.00 | 141.56 | 0.00 | 256.90 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-7-1-1 | Hardware-1-1-7 | Hardware-1-1-7-1 | 1 | 7 | 1 | 1 | 64 | randwrite | 4k | 64 | libaio | 0 | 0 | 50 | 2 | 0.00 | 3213.22 | 0.00 | 168.82 | 0.00 | 265.29 | 2 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-8-1-1 | Hardware-1-1-8 | Hardware-1-1-8-1 | 1 | 8 | 1 | 1 | 63 | randwrite | 4k | 64 | libaio | 0 | 0 | 50 | 4 | 0.00 | 3792.92 | 0.00 | 214.96 | 0.00 | 750.78 | 4 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-9-1-1 | Hardware-1-1-9 | Hardware-1-1-9-1 | 1 | 9 | 1 | 1 | 63 | randwrite | 4k | 64 | libaio | 0 | 0 | 50 | 8 | 0.00 | 4919.47 | 0.00 | 320.86 | 0.00 | 1484.78 | 8 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-10-1-1 | Hardware-1-1-10 | Hardware-1-1-10-1 | 1 | 10 | 1 | 1 | 64 | randwrite | 4k | 64 | libaio | 0 | 0 | 50 | 16 | 0.00 | 6830.33 | 0.00 | 488.64 | 0.00 | 2768.24 | 16 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
#### Per Phase
| DBMS | phase | experiment_run | client | benchmark_run | pod_count | duration | hardware_fio_rw | hardware_fio_bs | hardware_fio_iodepth | hardware_fio_engine | hardware_fio_fsync | hardware_fio_fdatasync | hardware_fio_rwmixread | hardware_fio_read_iops | hardware_fio_write_iops | hardware_fio_read_lat_p95_ms | hardware_fio_write_lat_p95_ms | hardware_fio_read_lat_p99_ms | hardware_fio_write_lat_p99_ms | hardware_threads | hardware_sysbench_cpu_events_per_sec | hardware_sysbench_cpu_total_time_s | hardware_sysbench_cpu_lat_p95_ms | hardware_sysbench_memory_ops_per_sec | hardware_sysbench_memory_throughput_mibps | hardware_sysbench_memory_lat_p95_ms | errors |
|:----------------|:----------------|-----------------:|---------:|----------------:|------------:|-----------:|:------------------|:------------------|-----------------------:|:----------------------|---------------------:|-------------------------:|-------------------------:|-------------------------:|--------------------------:|-------------------------------:|--------------------------------:|-------------------------------:|--------------------------------:|-------------------:|---------------------------------------:|-------------------------------------:|-----------------------------------:|---------------------------------------:|--------------------------------------------:|--------------------------------------:|---------:|
| Hardware-1-1-1 | Hardware-1-1-1 | 1 | 1 | 1 | 1 | 71 | randread | 4k | 64 | libaio | 0 | 0 | 50 | 4338.31 | 0.00 | 54.26 | 0.00 | 152.04 | 0.00 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-2 | Hardware-1-1-2 | 1 | 2 | 1 | 1 | 79 | randread | 4k | 64 | libaio | 0 | 0 | 50 | 4703.97 | 0.00 | 89.65 | 0.00 | 497.03 | 0.00 | 2 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-3 | Hardware-1-1-3 | 1 | 3 | 1 | 1 | 94 | randread | 4k | 64 | libaio | 0 | 0 | 50 | 5166.99 | 0.00 | 113.77 | 0.00 | 952.11 | 0.00 | 4 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-4 | Hardware-1-1-4 | 1 | 4 | 1 | 1 | 122 | randread | 4k | 64 | libaio | 0 | 0 | 50 | 5773.17 | 0.00 | 196.08 | 0.00 | 1803.55 | 0.00 | 8 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-5 | Hardware-1-1-5 | 1 | 5 | 1 | 1 | 187 | randread | 4k | 64 | libaio | 0 | 0 | 50 | 5295.18 | 0.00 | 1333.79 | 0.00 | 4731.17 | 0.00 | 16 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-6 | Hardware-1-1-6 | 1 | 6 | 1 | 1 | 63 | randwrite | 4k | 64 | libaio | 0 | 0 | 50 | 0.00 | 1878.38 | 0.00 | 141.56 | 0.00 | 256.90 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-7 | Hardware-1-1-7 | 1 | 7 | 1 | 1 | 64 | randwrite | 4k | 64 | libaio | 0 | 0 | 50 | 0.00 | 3213.22 | 0.00 | 168.82 | 0.00 | 265.29 | 2 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-8 | Hardware-1-1-8 | 1 | 8 | 1 | 1 | 63 | randwrite | 4k | 64 | libaio | 0 | 0 | 50 | 0.00 | 3792.92 | 0.00 | 214.96 | 0.00 | 750.78 | 4 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-9 | Hardware-1-1-9 | 1 | 9 | 1 | 1 | 63 | randwrite | 4k | 64 | libaio | 0 | 0 | 50 | 0.00 | 4919.47 | 0.00 | 320.86 | 0.00 | 1484.78 | 8 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-10 | Hardware-1-1-10 | 1 | 10 | 1 | 1 | 64 | randwrite | 4k | 64 | libaio | 0 | 0 | 50 | 0.00 | 6830.33 | 0.00 | 488.64 | 0.00 | 2768.24 | 16 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
### Monitoring
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------|-------------:|----------:|---------------:|----------------------:|
| Hardware-1-1-1-1 | 70.17 | 2.29 | 0.02 | 0.02 |
| Hardware-1-1-2-1 | 70.96 | 1.99 | 0.03 | 4.03 |
| Hardware-1-1-3-1 | 81.90 | 3.40 | 0.03 | 4.03 |
| Hardware-1-1-4-1 | 91.72 | 1.83 | 0.03 | 4.03 |
| Hardware-1-1-5-1 | 108.61 | 1.85 | 0.03 | 4.03 |
| Hardware-1-1-6-1 | 68.42 | 2.56 | 0.02 | 0.02 |
| Hardware-1-1-7-1 | 73.46 | 2.33 | 0.02 | 0.02 |
| Hardware-1-1-8-1 | 71.69 | 4.00 | 0.03 | 0.03 |
| Hardware-1-1-9-1 | 72.63 | 1.74 | 0.03 | 0.03 |
| Hardware-1-1-10-1 | 61.99 | 1.61 | 0.03 | 0.04 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------|-------------:|----------:|---------------:|----------------------:|
| Hardware-1-1-1-1 | 0.54 | 0.01 | 0.00 | 0.00 |
| Hardware-1-1-2-1 | 0.57 | 0.00 | 0.00 | 0.00 |
| Hardware-1-1-3-1 | 0.54 | 0.01 | 0.00 | 0.00 |
| Hardware-1-1-4-1 | 0.57 | 0.00 | 0.00 | 0.00 |
| Hardware-1-1-5-1 | 0.58 | 0.02 | 0.00 | 0.00 |
| Hardware-1-1-6-1 | 0.44 | 0.01 | 0.00 | 0.00 |
| Hardware-1-1-7-1 | 0.58 | 0.01 | 0.00 | 0.00 |
| Hardware-1-1-8-1 | 0.55 | 0.02 | 0.00 | 0.00 |
| Hardware-1-1-9-1 | 0.55 | 0.01 | 0.00 | 0.00 |
| Hardware-1-1-10-1 | 0.50 | 0.01 | 0.00 | 0.00 |
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: every round has non-zero read or write IOPS
The Per Phase table lists one row per -nbt (numjobs) value, at the fixed queue depth set by
-xfid.
3. Depth sweep at PostgreSQL’s page size (8k)
The sweeps above use bs=4k as a generic device-IOPS probe. PostgreSQL always issues 8kB pages
(BLCKSZ), so this re-anchors the depth sweep at the actual unit of I/O Postgres uses — the
number that calibrates effective_io_concurrency / maintenance_io_concurrency, PostgreSQL’s
own prefetch-depth knobs.
bexhoma hardware \
-dbms Hardware \
-xht fio \
-xts 4G \
-xtd 60 \
-xfrw randread,randwrite \
-xfbs 8k \
-xfid 1,2,4,8,16,32,64,128 \
-xfe libaio \
-nbp 1 \
-nbt 1 \
-ne 1 \
-m \
-ms $BEXHOMA_MS \
-tr \
-rsr \
-rss 50Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rnn $BEXHOMA_NODE_SUT -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_hardware_fio_depth_sweep_8k.log
Result
testcase_hardware_fio_depth_sweep_8k.log
## Show Summary
### Workload
Hardware Benchmark (fio)
* Type: hardware
* Duration: 2212s
* Code: 1783195883
* fio/sysbench driver runs the experiment.
* This experiment measures raw hardware I/O (fio) or CPU/memory (sysbench) performance.
* Benchmark tool: fio.
* Test file size is '4G', duration per round is 60s.
* I/O pattern(s) swept: ['randread', 'randwrite'].
* Block size(s) swept: ['8k'].
* Queue depth(s) swept: [1, 2, 4, 8, 16, 32, 64, 128].
* I/O engine(s) swept: ['libaio'].
* Fsync interval(s) swept: [0].
* Fdatasync interval(s) swept: [0].
* Experiment uses bexhoma version 0.10.2.
* System metrics are monitored by sidecar containers.
* Experiment is limited to DBMS ['Hardware'].
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker36.
* Database is persisted to disk of type shared and size 50Gi. Persistent storage is removed at experiment start.
* Benchmarking is tested with [1] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* Hardware-1-1-1-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:822044
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783195883
* Hardware-1-1-10-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:825002
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783195883
* Hardware-1-1-11-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:816640
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783195883
* Hardware-1-1-12-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:823446
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783195883
* Hardware-1-1-13-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:824851
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783195883
* Hardware-1-1-14-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:826533
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783195883
* Hardware-1-1-15-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:825407
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783195883
* Hardware-1-1-16-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:820455
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783195883
* Hardware-1-1-2-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:823479
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783195883
* Hardware-1-1-3-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:829372
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783195883
* Hardware-1-1-4-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:826037
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783195883
* Hardware-1-1-5-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:829244
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783195883
* Hardware-1-1-6-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:825635
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783195883
* Hardware-1-1-7-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:823397
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783195883
* Hardware-1-1-8-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:819244
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783195883
* Hardware-1-1-9-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:821267
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783195883
### SUT Container Restarts
* bexhoma-sut-hardware-1-1783195883-97d957696-qdj82: 0
### Workflow
#### Actual
* DBMS Hardware-1 - Experiment 1 Client 1: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 2: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 3: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 4: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 5: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 6: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 7: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 8: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 9: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 10: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 11: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 12: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 13: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 14: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 15: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 16: hardware (1 pods)
#### Planned
* DBMS Hardware-1 - Experiment 1 Client 1: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 2: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 3: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 4: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 5: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 6: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 7: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 8: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 9: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 10: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 11: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 12: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 13: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 14: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 15: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 16: hardware (1 pods)
### Execution
#### Per Connection
| DBMS | phase | job | experiment_run | client | benchmark_run | child | duration | hardware_fio_rw | hardware_fio_bs | hardware_fio_iodepth | hardware_fio_engine | hardware_fio_fsync | hardware_fio_fdatasync | hardware_fio_rwmixread | hardware_fio_numjobs | hardware_fio_read_iops | hardware_fio_write_iops | hardware_fio_read_lat_p95_ms | hardware_fio_write_lat_p95_ms | hardware_fio_read_lat_p99_ms | hardware_fio_write_lat_p99_ms | hardware_threads | hardware_sysbench_cpu_events_per_sec | hardware_sysbench_cpu_total_time_s | hardware_sysbench_cpu_lat_p95_ms | hardware_sysbench_memory_ops_per_sec | hardware_sysbench_memory_throughput_mibps | hardware_sysbench_memory_lat_p95_ms | errors |
|:--------------------|:----------------|:------------------|-----------------:|---------:|----------------:|--------:|-----------:|:------------------|:------------------|-----------------------:|:----------------------|---------------------:|-------------------------:|-------------------------:|-----------------------:|-------------------------:|--------------------------:|-------------------------------:|--------------------------------:|-------------------------------:|--------------------------------:|-------------------:|---------------------------------------:|-------------------------------------:|-----------------------------------:|---------------------------------------:|--------------------------------------------:|--------------------------------------:|---------:|
| Hardware-1-1-1-1-1 | Hardware-1-1-1 | Hardware-1-1-1-1 | 1 | 1 | 1 | 1 | 70 | randread | 8k | 1 | libaio | 0 | 0 | 50 | 1 | 61.98 | 0.00 | 36.44 | 0.00 | 90.70 | 0.00 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-2-1-1 | Hardware-1-1-2 | Hardware-1-1-2-1 | 1 | 2 | 1 | 1 | 69 | randread | 8k | 2 | libaio | 0 | 0 | 50 | 1 | 167.54 | 0.00 | 32.11 | 0.00 | 76.02 | 0.00 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-3-1-1 | Hardware-1-1-3 | Hardware-1-1-3-1 | 1 | 3 | 1 | 1 | 70 | randread | 8k | 4 | libaio | 0 | 0 | 50 | 1 | 403.22 | 0.00 | 26.35 | 0.00 | 66.32 | 0.00 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-4-1-1 | Hardware-1-1-4 | Hardware-1-1-4-1 | 1 | 4 | 1 | 1 | 72 | randread | 8k | 8 | libaio | 0 | 0 | 50 | 1 | 546.25 | 0.00 | 35.39 | 0.00 | 117.96 | 0.00 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-5-1-1 | Hardware-1-1-5 | Hardware-1-1-5-1 | 1 | 5 | 1 | 1 | 70 | randread | 8k | 16 | libaio | 0 | 0 | 50 | 1 | 1337.27 | 0.00 | 42.21 | 0.00 | 107.48 | 0.00 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-6-1-1 | Hardware-1-1-6 | Hardware-1-1-6-1 | 1 | 6 | 1 | 1 | 70 | randread | 8k | 32 | libaio | 0 | 0 | 50 | 1 | 2537.31 | 0.00 | 31.33 | 0.00 | 91.75 | 0.00 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-7-1-1 | Hardware-1-1-7 | Hardware-1-1-7-1 | 1 | 7 | 1 | 1 | 71 | randread | 8k | 64 | libaio | 0 | 0 | 50 | 1 | 4781.67 | 0.00 | 43.25 | 0.00 | 127.40 | 0.00 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-8-1-1 | Hardware-1-1-8 | Hardware-1-1-8-1 | 1 | 8 | 1 | 1 | 71 | randread | 8k | 128 | libaio | 0 | 0 | 50 | 1 | 4743.46 | 0.00 | 66.85 | 0.00 | 734.00 | 0.00 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-9-1-1 | Hardware-1-1-9 | Hardware-1-1-9-1 | 1 | 9 | 1 | 1 | 63 | randwrite | 8k | 1 | libaio | 0 | 0 | 50 | 1 | 0.00 | 100.86 | 0.00 | 38.01 | 0.00 | 71.83 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-10-1-1 | Hardware-1-1-10 | Hardware-1-1-10-1 | 1 | 10 | 1 | 1 | 63 | randwrite | 8k | 2 | libaio | 0 | 0 | 50 | 1 | 0.00 | 172.16 | 0.00 | 43.78 | 0.00 | 83.36 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-11-1-1 | Hardware-1-1-11 | Hardware-1-1-11-1 | 1 | 11 | 1 | 1 | 63 | randwrite | 8k | 4 | libaio | 0 | 0 | 50 | 1 | 0.00 | 279.84 | 0.00 | 50.59 | 0.00 | 112.72 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-12-1-1 | Hardware-1-1-12 | Hardware-1-1-12-1 | 1 | 12 | 1 | 1 | 63 | randwrite | 8k | 8 | libaio | 0 | 0 | 50 | 1 | 0.00 | 490.15 | 0.00 | 55.84 | 0.00 | 149.95 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-13-1-1 | Hardware-1-1-13 | Hardware-1-1-13-1 | 1 | 13 | 1 | 1 | 65 | randwrite | 8k | 16 | libaio | 0 | 0 | 50 | 1 | 0.00 | 696.68 | 0.00 | 70.78 | 0.00 | 193.99 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-14-1-1 | Hardware-1-1-14 | Hardware-1-1-14-1 | 1 | 14 | 1 | 1 | 63 | randwrite | 8k | 32 | libaio | 0 | 0 | 50 | 1 | 0.00 | 1175.87 | 0.00 | 88.60 | 0.00 | 223.35 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-15-1-1 | Hardware-1-1-15 | Hardware-1-1-15-1 | 1 | 15 | 1 | 1 | 64 | randwrite | 8k | 64 | libaio | 0 | 0 | 50 | 1 | 0.00 | 1859.32 | 0.00 | 122.16 | 0.00 | 246.42 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-16-1-1 | Hardware-1-1-16 | Hardware-1-1-16-1 | 1 | 16 | 1 | 1 | 64 | randwrite | 8k | 128 | libaio | 0 | 0 | 50 | 1 | 0.00 | 2449.04 | 0.00 | 162.53 | 0.00 | 434.11 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
#### Per Phase
| DBMS | phase | experiment_run | client | benchmark_run | pod_count | duration | hardware_fio_rw | hardware_fio_bs | hardware_fio_iodepth | hardware_fio_engine | hardware_fio_fsync | hardware_fio_fdatasync | hardware_fio_rwmixread | hardware_fio_read_iops | hardware_fio_write_iops | hardware_fio_read_lat_p95_ms | hardware_fio_write_lat_p95_ms | hardware_fio_read_lat_p99_ms | hardware_fio_write_lat_p99_ms | hardware_threads | hardware_sysbench_cpu_events_per_sec | hardware_sysbench_cpu_total_time_s | hardware_sysbench_cpu_lat_p95_ms | hardware_sysbench_memory_ops_per_sec | hardware_sysbench_memory_throughput_mibps | hardware_sysbench_memory_lat_p95_ms | errors |
|:----------------|:----------------|-----------------:|---------:|----------------:|------------:|-----------:|:------------------|:------------------|-----------------------:|:----------------------|---------------------:|-------------------------:|-------------------------:|-------------------------:|--------------------------:|-------------------------------:|--------------------------------:|-------------------------------:|--------------------------------:|-------------------:|---------------------------------------:|-------------------------------------:|-----------------------------------:|---------------------------------------:|--------------------------------------------:|--------------------------------------:|---------:|
| Hardware-1-1-1 | Hardware-1-1-1 | 1 | 1 | 1 | 1 | 70 | randread | 8k | 1 | libaio | 0 | 0 | 50 | 61.98 | 0.00 | 36.44 | 0.00 | 90.70 | 0.00 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-2 | Hardware-1-1-2 | 1 | 2 | 1 | 1 | 69 | randread | 8k | 2 | libaio | 0 | 0 | 50 | 167.54 | 0.00 | 32.11 | 0.00 | 76.02 | 0.00 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-3 | Hardware-1-1-3 | 1 | 3 | 1 | 1 | 70 | randread | 8k | 4 | libaio | 0 | 0 | 50 | 403.22 | 0.00 | 26.35 | 0.00 | 66.32 | 0.00 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-4 | Hardware-1-1-4 | 1 | 4 | 1 | 1 | 72 | randread | 8k | 8 | libaio | 0 | 0 | 50 | 546.25 | 0.00 | 35.39 | 0.00 | 117.96 | 0.00 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-5 | Hardware-1-1-5 | 1 | 5 | 1 | 1 | 70 | randread | 8k | 16 | libaio | 0 | 0 | 50 | 1337.27 | 0.00 | 42.21 | 0.00 | 107.48 | 0.00 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-6 | Hardware-1-1-6 | 1 | 6 | 1 | 1 | 70 | randread | 8k | 32 | libaio | 0 | 0 | 50 | 2537.31 | 0.00 | 31.33 | 0.00 | 91.75 | 0.00 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-7 | Hardware-1-1-7 | 1 | 7 | 1 | 1 | 71 | randread | 8k | 64 | libaio | 0 | 0 | 50 | 4781.67 | 0.00 | 43.25 | 0.00 | 127.40 | 0.00 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-8 | Hardware-1-1-8 | 1 | 8 | 1 | 1 | 71 | randread | 8k | 128 | libaio | 0 | 0 | 50 | 4743.46 | 0.00 | 66.85 | 0.00 | 734.00 | 0.00 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-9 | Hardware-1-1-9 | 1 | 9 | 1 | 1 | 63 | randwrite | 8k | 1 | libaio | 0 | 0 | 50 | 0.00 | 100.86 | 0.00 | 38.01 | 0.00 | 71.83 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-10 | Hardware-1-1-10 | 1 | 10 | 1 | 1 | 63 | randwrite | 8k | 2 | libaio | 0 | 0 | 50 | 0.00 | 172.16 | 0.00 | 43.78 | 0.00 | 83.36 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-11 | Hardware-1-1-11 | 1 | 11 | 1 | 1 | 63 | randwrite | 8k | 4 | libaio | 0 | 0 | 50 | 0.00 | 279.84 | 0.00 | 50.59 | 0.00 | 112.72 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-12 | Hardware-1-1-12 | 1 | 12 | 1 | 1 | 63 | randwrite | 8k | 8 | libaio | 0 | 0 | 50 | 0.00 | 490.15 | 0.00 | 55.84 | 0.00 | 149.95 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-13 | Hardware-1-1-13 | 1 | 13 | 1 | 1 | 65 | randwrite | 8k | 16 | libaio | 0 | 0 | 50 | 0.00 | 696.68 | 0.00 | 70.78 | 0.00 | 193.99 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-14 | Hardware-1-1-14 | 1 | 14 | 1 | 1 | 63 | randwrite | 8k | 32 | libaio | 0 | 0 | 50 | 0.00 | 1175.87 | 0.00 | 88.60 | 0.00 | 223.35 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-15 | Hardware-1-1-15 | 1 | 15 | 1 | 1 | 64 | randwrite | 8k | 64 | libaio | 0 | 0 | 50 | 0.00 | 1859.32 | 0.00 | 122.16 | 0.00 | 246.42 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-16 | Hardware-1-1-16 | 1 | 16 | 1 | 1 | 64 | randwrite | 8k | 128 | libaio | 0 | 0 | 50 | 0.00 | 2449.04 | 0.00 | 162.53 | 0.00 | 434.11 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
### Monitoring
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------|-------------:|----------:|---------------:|----------------------:|
| Hardware-1-1-1-1 | 73.15 | 2.30 | 0.02 | 0.02 |
| Hardware-1-1-2-1 | 65.66 | 2.04 | 0.03 | 4.03 |
| Hardware-1-1-3-1 | 73.03 | 2.69 | 0.02 | 0.02 |
| Hardware-1-1-4-1 | 65.29 | 1.79 | 0.02 | 0.02 |
| Hardware-1-1-5-1 | 71.24 | 1.98 | 0.02 | 0.02 |
| Hardware-1-1-6-1 | 65.16 | 2.14 | 0.02 | 0.02 |
| Hardware-1-1-7-1 | 72.52 | 4.75 | 0.02 | 0.02 |
| Hardware-1-1-8-1 | 75.68 | 2.36 | 0.03 | 2.44 |
| Hardware-1-1-9-1 | 65.55 | 2.49 | 0.02 | 0.02 |
| Hardware-1-1-10-1 | 68.96 | 2.28 | 0.08 | 0.08 |
| Hardware-1-1-11-1 | 66.96 | 1.66 | 0.02 | 0.02 |
| Hardware-1-1-12-1 | 58.58 | 1.69 | 0.02 | 0.02 |
| Hardware-1-1-13-1 | 65.96 | 1.79 | 0.02 | 0.02 |
| Hardware-1-1-14-1 | 64.07 | 2.03 | 0.03 | 0.03 |
| Hardware-1-1-15-1 | 67.91 | 1.47 | 0.02 | 0.02 |
| Hardware-1-1-16-1 | 64.59 | 1.68 | 0.02 | 0.02 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------|-------------:|----------:|---------------:|----------------------:|
| Hardware-1-1-1-1 | 0.49 | 0.00 | 0.00 | 0.00 |
| Hardware-1-1-2-1 | 0.50 | 0.00 | 0.00 | 0.00 |
| Hardware-1-1-3-1 | 0.52 | 0.00 | 0.00 | 0.00 |
| Hardware-1-1-4-1 | 0.54 | 0.00 | 0.00 | 0.00 |
| Hardware-1-1-5-1 | 0.52 | 0.01 | 0.00 | 0.00 |
| Hardware-1-1-6-1 | 0.48 | 0.01 | 0.00 | 0.00 |
| Hardware-1-1-7-1 | 0.55 | 0.02 | 0.00 | 0.00 |
| Hardware-1-1-8-1 | 0.43 | 0.02 | 0.00 | 0.00 |
| Hardware-1-1-9-1 | 0.58 | 0.02 | 0.00 | 0.00 |
| Hardware-1-1-10-1 | 0.51 | 0.00 | 0.00 | 0.00 |
| Hardware-1-1-11-1 | 0.54 | 0.02 | 0.00 | 0.00 |
| Hardware-1-1-12-1 | 0.44 | 0.02 | 0.00 | 0.00 |
| Hardware-1-1-13-1 | 0.50 | 0.02 | 0.00 | 0.00 |
| Hardware-1-1-14-1 | 0.52 | 0.01 | 0.00 | 0.00 |
| Hardware-1-1-15-1 | 0.50 | 0.01 | 0.00 | 0.00 |
| Hardware-1-1-16-1 | 0.57 | 0.01 | 0.00 | 0.00 |
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: every round has non-zero read or write IOPS
The Per Phase table lists one row per queue depth, at the 8k block size set by -xfbs.
4. WAL sync-write latency (fdatasync)
Same as above but fdatasync instead of fsync. fdatasync skips the inode-metadata sync fsync
does, and is PostgreSQL’s Linux default (wal_sync_method=fdatasync) — compare its latency
against the fsync run above to confirm it is actually cheaper on this storage.
bexhoma hardware \
-dbms Hardware \
-xht fio \
-xts 4G \
-xtd 60 \
-xfrw write \
-xfbs 8k \
-xfid 1 \
-xfe libaio \
-xffd 1 \
-nbp 1 \
-nbt 1 \
-ne 1 \
-m \
-ms $BEXHOMA_MS \
-tr \
-rsr \
-rss 50Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rnn $BEXHOMA_NODE_SUT -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_hardware_fio_wal_sync_fdatasync.log
Result
testcase_hardware_fio_wal_sync_fdatasync.log
## Show Summary
### Workload
Hardware Benchmark (fio)
* Type: hardware
* Duration: 182s
* Code: 1783198702
* fio/sysbench driver runs the experiment.
* This experiment measures raw hardware I/O (fio) or CPU/memory (sysbench) performance.
* Benchmark tool: fio.
* Test file size is '4G', duration per round is 60s.
* I/O pattern(s) swept: ['write'].
* Block size(s) swept: ['8k'].
* Queue depth(s) swept: [1].
* I/O engine(s) swept: ['libaio'].
* Fsync interval(s) swept: [0].
* Fdatasync interval(s) swept: [1].
* Experiment uses bexhoma version 0.10.2.
* System metrics are monitored by sidecar containers.
* Experiment is limited to DBMS ['Hardware'].
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker36.
* Database is persisted to disk of type shared and size 50Gi. Persistent storage is removed at experiment start.
* Benchmarking is tested with [1] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* Hardware-1-1-1-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:825046
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783198702
### SUT Container Restarts
* bexhoma-sut-hardware-1-1783198702-85dd59cf54-f67jc: 0
### Workflow
#### Actual
* DBMS Hardware-1 - Experiment 1 Client 1: hardware (1 pods)
#### Planned
* DBMS Hardware-1 - Experiment 1 Client 1: hardware (1 pods)
### Execution
#### Per Connection
| DBMS | phase | job | experiment_run | client | benchmark_run | child | duration | hardware_fio_rw | hardware_fio_bs | hardware_fio_iodepth | hardware_fio_engine | hardware_fio_fsync | hardware_fio_fdatasync | hardware_fio_rwmixread | hardware_fio_numjobs | hardware_fio_read_iops | hardware_fio_write_iops | hardware_fio_read_lat_p95_ms | hardware_fio_write_lat_p95_ms | hardware_fio_read_lat_p99_ms | hardware_fio_write_lat_p99_ms | hardware_threads | hardware_sysbench_cpu_events_per_sec | hardware_sysbench_cpu_total_time_s | hardware_sysbench_cpu_lat_p95_ms | hardware_sysbench_memory_ops_per_sec | hardware_sysbench_memory_throughput_mibps | hardware_sysbench_memory_lat_p95_ms | errors |
|:-------------------|:---------------|:-----------------|-----------------:|---------:|----------------:|--------:|-----------:|:------------------|:------------------|-----------------------:|:----------------------|---------------------:|-------------------------:|-------------------------:|-----------------------:|-------------------------:|--------------------------:|-------------------------------:|--------------------------------:|-------------------------------:|--------------------------------:|-------------------:|---------------------------------------:|-------------------------------------:|-----------------------------------:|---------------------------------------:|--------------------------------------------:|--------------------------------------:|---------:|
| Hardware-1-1-1-1-1 | Hardware-1-1-1 | Hardware-1-1-1-1 | 1 | 1 | 1 | 1 | 63 | write | 8k | 1 | libaio | 0 | 1 | 50 | 1 | 0.00 | 177.61 | 0.00 | 25.03 | 0.00 | 58.46 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
#### Per Phase
| DBMS | phase | experiment_run | client | benchmark_run | pod_count | duration | hardware_fio_rw | hardware_fio_bs | hardware_fio_iodepth | hardware_fio_engine | hardware_fio_fsync | hardware_fio_fdatasync | hardware_fio_rwmixread | hardware_fio_read_iops | hardware_fio_write_iops | hardware_fio_read_lat_p95_ms | hardware_fio_write_lat_p95_ms | hardware_fio_read_lat_p99_ms | hardware_fio_write_lat_p99_ms | hardware_threads | hardware_sysbench_cpu_events_per_sec | hardware_sysbench_cpu_total_time_s | hardware_sysbench_cpu_lat_p95_ms | hardware_sysbench_memory_ops_per_sec | hardware_sysbench_memory_throughput_mibps | hardware_sysbench_memory_lat_p95_ms | errors |
|:---------------|:---------------|-----------------:|---------:|----------------:|------------:|-----------:|:------------------|:------------------|-----------------------:|:----------------------|---------------------:|-------------------------:|-------------------------:|-------------------------:|--------------------------:|-------------------------------:|--------------------------------:|-------------------------------:|--------------------------------:|-------------------:|---------------------------------------:|-------------------------------------:|-----------------------------------:|---------------------------------------:|--------------------------------------------:|--------------------------------------:|---------:|
| Hardware-1-1-1 | Hardware-1-1-1 | 1 | 1 | 1 | 1 | 63 | write | 8k | 1 | libaio | 0 | 1 | 50 | 0.00 | 177.61 | 0.00 | 25.03 | 0.00 | 58.46 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
### Monitoring
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-----------------|-------------:|----------:|---------------:|----------------------:|
| Hardware-1-1-1-1 | 64.00 | 2.58 | 0.02 | 0.02 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-----------------|-------------:|----------:|---------------:|----------------------:|
| Hardware-1-1-1-1 | 0.50 | 0.01 | 0.00 | 0.00 |
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: every round has non-zero read or write IOPS
hardware_fio_write_iops and the write latency percentiles are the relevant columns to compare
against the WAL sync-write fsync example in Example-Hardware.md.
5. WAL group-commit scaling
Same sync-write profile, sweeping concurrent committing backends (-nbt) instead of a single
one. If aggregate fsyncs/sec keeps climbing with more concurrent writers, the storage/controller
coalesces concurrent commits well; if it flattens immediately, tune commit_delay /
commit_siblings in Postgres to force batching in software instead.
bexhoma hardware \
-dbms Hardware \
-xht fio \
-xts 4G \
-xtd 60 \
-xfrw write \
-xfbs 8k \
-xfid 1 \
-xfe libaio \
-xfsy 1 \
-nbp 1 \
-nbt 1,2,4,8,16,32 \
-ne 1 \
-m \
-ms $BEXHOMA_MS \
-tr \
-rsr \
-rss 150Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rnn $BEXHOMA_NODE_SUT -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_hardware_fio_wal_group_commit.log
Result
testcase_hardware_fio_wal_group_commit.log
## Show Summary
### Workload
Hardware Benchmark (fio)
* Type: hardware
* Duration: 856s
* Code: 1783198907
* fio/sysbench driver runs the experiment.
* This experiment measures raw hardware I/O (fio) or CPU/memory (sysbench) performance.
* Benchmark tool: fio.
* Test file size is '4G', duration per round is 60s.
* I/O pattern(s) swept: ['write'].
* Block size(s) swept: ['8k'].
* Queue depth(s) swept: [1].
* I/O engine(s) swept: ['libaio'].
* Fsync interval(s) swept: [1].
* Fdatasync interval(s) swept: [0].
* Experiment uses bexhoma version 0.10.2.
* System metrics are monitored by sidecar containers.
* Experiment is limited to DBMS ['Hardware'].
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker36.
* Database is persisted to disk of type shared and size 150Gi. Persistent storage is removed at experiment start.
* Benchmarking is tested with [1, 2, 4, 8, 16, 32] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* Hardware-1-1-1-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:826576
* volume_size:150.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783198907
* Hardware-1-1-2-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:824462
* volume_size:150.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783198907
* Hardware-1-1-3-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:824318
* volume_size:150.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783198907
* Hardware-1-1-4-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:818035
* volume_size:150.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783198907
* Hardware-1-1-5-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:823012
* volume_size:150.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783198907
* Hardware-1-1-6-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:824871
* volume_size:150.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783198907
### SUT Container Restarts
* bexhoma-sut-hardware-1-1783198907-565b848d8d-p7zpw: 0
### Workflow
#### Actual
* DBMS Hardware-1 - Experiment 1 Client 1: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 2: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 3: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 4: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 5: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 6: hardware (1 pods)
#### Planned
* DBMS Hardware-1 - Experiment 1 Client 1: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 2: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 3: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 4: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 5: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 6: hardware (1 pods)
### Execution
#### Per Connection
| DBMS | phase | job | experiment_run | client | benchmark_run | child | duration | hardware_fio_rw | hardware_fio_bs | hardware_fio_iodepth | hardware_fio_engine | hardware_fio_fsync | hardware_fio_fdatasync | hardware_fio_rwmixread | hardware_fio_numjobs | hardware_fio_read_iops | hardware_fio_write_iops | hardware_fio_read_lat_p95_ms | hardware_fio_write_lat_p95_ms | hardware_fio_read_lat_p99_ms | hardware_fio_write_lat_p99_ms | hardware_threads | hardware_sysbench_cpu_events_per_sec | hardware_sysbench_cpu_total_time_s | hardware_sysbench_cpu_lat_p95_ms | hardware_sysbench_memory_ops_per_sec | hardware_sysbench_memory_throughput_mibps | hardware_sysbench_memory_lat_p95_ms | errors |
|:-------------------|:---------------|:-----------------|-----------------:|---------:|----------------:|--------:|-----------:|:------------------|:------------------|-----------------------:|:----------------------|---------------------:|-------------------------:|-------------------------:|-----------------------:|-------------------------:|--------------------------:|-------------------------------:|--------------------------------:|-------------------------------:|--------------------------------:|-------------------:|---------------------------------------:|-------------------------------------:|-----------------------------------:|---------------------------------------:|--------------------------------------------:|--------------------------------------:|---------:|
| Hardware-1-1-1-1-1 | Hardware-1-1-1 | Hardware-1-1-1-1 | 1 | 1 | 1 | 1 | 64 | write | 8k | 1 | libaio | 1 | 0 | 50 | 1 | 0.00 | 64.94 | 0.00 | 17.43 | 0.00 | 429.92 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-2-1-1 | Hardware-1-1-2 | Hardware-1-1-2-1 | 1 | 2 | 1 | 1 | 63 | write | 8k | 1 | libaio | 1 | 0 | 50 | 2 | 0.00 | 181.98 | 0.00 | 57.93 | 0.00 | 100.14 | 2 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-3-1-1 | Hardware-1-1-3 | Hardware-1-1-3-1 | 1 | 3 | 1 | 1 | 63 | write | 8k | 1 | libaio | 1 | 0 | 50 | 4 | 0.00 | 476.06 | 0.00 | 50.07 | 0.00 | 74.97 | 4 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-4-1-1 | Hardware-1-1-4 | Hardware-1-1-4-1 | 1 | 4 | 1 | 1 | 63 | write | 8k | 1 | libaio | 1 | 0 | 50 | 8 | 0.00 | 852.53 | 0.00 | 50.59 | 0.00 | 104.33 | 8 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-5-1-1 | Hardware-1-1-5 | Hardware-1-1-5-1 | 1 | 5 | 1 | 1 | 63 | write | 8k | 1 | libaio | 1 | 0 | 50 | 16 | 0.00 | 1396.17 | 0.00 | 50.07 | 0.00 | 91.75 | 16 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-6-1-1 | Hardware-1-1-6 | Hardware-1-1-6-1 | 1 | 6 | 1 | 1 | 63 | write | 8k | 1 | libaio | 1 | 0 | 50 | 32 | 0.00 | 2593.54 | 0.00 | 58.46 | 0.00 | 115.87 | 32 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
#### Per Phase
| DBMS | phase | experiment_run | client | benchmark_run | pod_count | duration | hardware_fio_rw | hardware_fio_bs | hardware_fio_iodepth | hardware_fio_engine | hardware_fio_fsync | hardware_fio_fdatasync | hardware_fio_rwmixread | hardware_fio_read_iops | hardware_fio_write_iops | hardware_fio_read_lat_p95_ms | hardware_fio_write_lat_p95_ms | hardware_fio_read_lat_p99_ms | hardware_fio_write_lat_p99_ms | hardware_threads | hardware_sysbench_cpu_events_per_sec | hardware_sysbench_cpu_total_time_s | hardware_sysbench_cpu_lat_p95_ms | hardware_sysbench_memory_ops_per_sec | hardware_sysbench_memory_throughput_mibps | hardware_sysbench_memory_lat_p95_ms | errors |
|:---------------|:---------------|-----------------:|---------:|----------------:|------------:|-----------:|:------------------|:------------------|-----------------------:|:----------------------|---------------------:|-------------------------:|-------------------------:|-------------------------:|--------------------------:|-------------------------------:|--------------------------------:|-------------------------------:|--------------------------------:|-------------------:|---------------------------------------:|-------------------------------------:|-----------------------------------:|---------------------------------------:|--------------------------------------------:|--------------------------------------:|---------:|
| Hardware-1-1-1 | Hardware-1-1-1 | 1 | 1 | 1 | 1 | 64 | write | 8k | 1 | libaio | 1 | 0 | 50 | 0.00 | 64.94 | 0.00 | 17.43 | 0.00 | 429.92 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-2 | Hardware-1-1-2 | 1 | 2 | 1 | 1 | 63 | write | 8k | 1 | libaio | 1 | 0 | 50 | 0.00 | 181.98 | 0.00 | 57.93 | 0.00 | 100.14 | 2 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-3 | Hardware-1-1-3 | 1 | 3 | 1 | 1 | 63 | write | 8k | 1 | libaio | 1 | 0 | 50 | 0.00 | 476.06 | 0.00 | 50.07 | 0.00 | 74.97 | 4 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-4 | Hardware-1-1-4 | 1 | 4 | 1 | 1 | 63 | write | 8k | 1 | libaio | 1 | 0 | 50 | 0.00 | 852.53 | 0.00 | 50.59 | 0.00 | 104.33 | 8 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-5 | Hardware-1-1-5 | 1 | 5 | 1 | 1 | 63 | write | 8k | 1 | libaio | 1 | 0 | 50 | 0.00 | 1396.17 | 0.00 | 50.07 | 0.00 | 91.75 | 16 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-6 | Hardware-1-1-6 | 1 | 6 | 1 | 1 | 63 | write | 8k | 1 | libaio | 1 | 0 | 50 | 0.00 | 2593.54 | 0.00 | 58.46 | 0.00 | 115.87 | 32 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
### Monitoring
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-----------------|-------------:|----------:|---------------:|----------------------:|
| Hardware-1-1-1-1 | 64.08 | 2.65 | 0.02 | 0.02 |
| Hardware-1-1-2-1 | 63.25 | 1.56 | 0.02 | 0.02 |
| Hardware-1-1-3-1 | 60.88 | 1.96 | 0.02 | 0.02 |
| Hardware-1-1-4-1 | 69.21 | 1.75 | 0.02 | 0.02 |
| Hardware-1-1-5-1 | 69.16 | 2.91 | 0.03 | 0.03 |
| Hardware-1-1-6-1 | 71.18 | 3.49 | 0.04 | 0.04 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-----------------|-------------:|----------:|---------------:|----------------------:|
| Hardware-1-1-1-1 | 0.56 | 0.00 | 0.00 | 0.00 |
| Hardware-1-1-2-1 | 0.57 | 0.02 | 0.00 | 0.00 |
| Hardware-1-1-3-1 | 0.49 | 0.02 | 0.00 | 0.00 |
| Hardware-1-1-4-1 | 0.57 | 0.04 | 0.00 | 0.00 |
| Hardware-1-1-5-1 | 0.62 | 0.01 | 0.00 | 0.00 |
| Hardware-1-1-6-1 | 0.52 | 0.04 | 0.00 | 0.00 |
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: every round has non-zero read or write IOPS
The Per Phase table lists one row per concurrent-writer count set by -nbt.
6. WAL record-size sweep
Same sync-write profile, sweeping the WAL record size instead of backend count. Bigger
transactions (or post-checkpoint full_page_writes bursts) write more before fsync — this shows
how sync-write latency grows with record size.
bexhoma hardware \
-dbms Hardware \
-xht fio \
-xts 4G \
-xtd 60 \
-xfrw write \
-xfbs 1k,8k,16k,32k,64k \
-xfid 1 \
-xfe libaio \
-xfsy 1 \
-nbp 1 \
-nbt 1 \
-ne 1 \
-m \
-ms $BEXHOMA_MS \
-tr \
-rsr \
-rss 50Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rnn $BEXHOMA_NODE_SUT -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_hardware_fio_wal_record_size.log
Result
testcase_hardware_fio_wal_record_size.log
## Show Summary
### Workload
Hardware Benchmark (fio)
* Type: hardware
* Duration: 723s
* Code: 1783199789
* fio/sysbench driver runs the experiment.
* This experiment measures raw hardware I/O (fio) or CPU/memory (sysbench) performance.
* Benchmark tool: fio.
* Test file size is '4G', duration per round is 60s.
* I/O pattern(s) swept: ['write'].
* Block size(s) swept: ['1k', '8k', '16k', '32k', '64k'].
* Queue depth(s) swept: [1].
* I/O engine(s) swept: ['libaio'].
* Fsync interval(s) swept: [1].
* Fdatasync interval(s) swept: [0].
* Experiment uses bexhoma version 0.10.2.
* System metrics are monitored by sidecar containers.
* Experiment is limited to DBMS ['Hardware'].
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker36.
* Database is persisted to disk of type shared and size 50Gi. Persistent storage is removed at experiment start.
* Benchmarking is tested with [1] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* Hardware-1-1-1-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:823240
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783199789
* Hardware-1-1-2-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:831169
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783199789
* Hardware-1-1-3-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:824028
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783199789
* Hardware-1-1-4-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:827995
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783199789
* Hardware-1-1-5-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:819882
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783199789
### SUT Container Restarts
* bexhoma-sut-hardware-1-1783199789-d99d9b8c8-dmq44: 0
### Workflow
#### Actual
* DBMS Hardware-1 - Experiment 1 Client 1: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 2: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 3: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 4: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 5: hardware (1 pods)
#### Planned
* DBMS Hardware-1 - Experiment 1 Client 1: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 2: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 3: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 4: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 5: hardware (1 pods)
### Execution
#### Per Connection
| DBMS | phase | job | experiment_run | client | benchmark_run | child | duration | hardware_fio_rw | hardware_fio_bs | hardware_fio_iodepth | hardware_fio_engine | hardware_fio_fsync | hardware_fio_fdatasync | hardware_fio_rwmixread | hardware_fio_numjobs | hardware_fio_read_iops | hardware_fio_write_iops | hardware_fio_read_lat_p95_ms | hardware_fio_write_lat_p95_ms | hardware_fio_read_lat_p99_ms | hardware_fio_write_lat_p99_ms | hardware_threads | hardware_sysbench_cpu_events_per_sec | hardware_sysbench_cpu_total_time_s | hardware_sysbench_cpu_lat_p95_ms | hardware_sysbench_memory_ops_per_sec | hardware_sysbench_memory_throughput_mibps | hardware_sysbench_memory_lat_p95_ms | errors |
|:-------------------|:---------------|:-----------------|-----------------:|---------:|----------------:|--------:|-----------:|:------------------|:------------------|-----------------------:|:----------------------|---------------------:|-------------------------:|-------------------------:|-----------------------:|-------------------------:|--------------------------:|-------------------------------:|--------------------------------:|-------------------------------:|--------------------------------:|-------------------:|---------------------------------------:|-------------------------------------:|-----------------------------------:|---------------------------------------:|--------------------------------------------:|--------------------------------------:|---------:|
| Hardware-1-1-1-1-1 | Hardware-1-1-1 | Hardware-1-1-1-1 | 1 | 1 | 1 | 1 | 63 | write | 1k | 1 | libaio | 1 | 0 | 50 | 1 | 0.00 | 84.38 | 0.00 | 52.17 | 0.00 | 79.17 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-2-1-1 | Hardware-1-1-2 | Hardware-1-1-2-1 | 1 | 2 | 1 | 1 | 63 | write | 8k | 1 | libaio | 1 | 0 | 50 | 1 | 0.00 | 145.23 | 0.00 | 33.42 | 0.00 | 63.70 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-3-1-1 | Hardware-1-1-3 | Hardware-1-1-3-1 | 1 | 3 | 1 | 1 | 63 | write | 16k | 1 | libaio | 1 | 0 | 50 | 1 | 0.00 | 94.12 | 0.00 | 33.42 | 0.00 | 63.18 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-4-1-1 | Hardware-1-1-4 | Hardware-1-1-4-1 | 1 | 4 | 1 | 1 | 63 | write | 32k | 1 | libaio | 1 | 0 | 50 | 1 | 0.00 | 127.30 | 0.00 | 35.39 | 0.00 | 74.97 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-5-1-1 | Hardware-1-1-5 | Hardware-1-1-5-1 | 1 | 5 | 1 | 1 | 62 | write | 64k | 1 | libaio | 1 | 0 | 50 | 1 | 0.00 | 53.67 | 0.00 | 65.80 | 0.00 | 88.60 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
#### Per Phase
| DBMS | phase | experiment_run | client | benchmark_run | pod_count | duration | hardware_fio_rw | hardware_fio_bs | hardware_fio_iodepth | hardware_fio_engine | hardware_fio_fsync | hardware_fio_fdatasync | hardware_fio_rwmixread | hardware_fio_read_iops | hardware_fio_write_iops | hardware_fio_read_lat_p95_ms | hardware_fio_write_lat_p95_ms | hardware_fio_read_lat_p99_ms | hardware_fio_write_lat_p99_ms | hardware_threads | hardware_sysbench_cpu_events_per_sec | hardware_sysbench_cpu_total_time_s | hardware_sysbench_cpu_lat_p95_ms | hardware_sysbench_memory_ops_per_sec | hardware_sysbench_memory_throughput_mibps | hardware_sysbench_memory_lat_p95_ms | errors |
|:---------------|:---------------|-----------------:|---------:|----------------:|------------:|-----------:|:------------------|:------------------|-----------------------:|:----------------------|---------------------:|-------------------------:|-------------------------:|-------------------------:|--------------------------:|-------------------------------:|--------------------------------:|-------------------------------:|--------------------------------:|-------------------:|---------------------------------------:|-------------------------------------:|-----------------------------------:|---------------------------------------:|--------------------------------------------:|--------------------------------------:|---------:|
| Hardware-1-1-1 | Hardware-1-1-1 | 1 | 1 | 1 | 1 | 63 | write | 1k | 1 | libaio | 1 | 0 | 50 | 0.00 | 84.38 | 0.00 | 52.17 | 0.00 | 79.17 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-2 | Hardware-1-1-2 | 1 | 2 | 1 | 1 | 63 | write | 8k | 1 | libaio | 1 | 0 | 50 | 0.00 | 145.23 | 0.00 | 33.42 | 0.00 | 63.70 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-3 | Hardware-1-1-3 | 1 | 3 | 1 | 1 | 63 | write | 16k | 1 | libaio | 1 | 0 | 50 | 0.00 | 94.12 | 0.00 | 33.42 | 0.00 | 63.18 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-4 | Hardware-1-1-4 | 1 | 4 | 1 | 1 | 63 | write | 32k | 1 | libaio | 1 | 0 | 50 | 0.00 | 127.30 | 0.00 | 35.39 | 0.00 | 74.97 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-5 | Hardware-1-1-5 | 1 | 5 | 1 | 1 | 62 | write | 64k | 1 | libaio | 1 | 0 | 50 | 0.00 | 53.67 | 0.00 | 65.80 | 0.00 | 88.60 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
### Monitoring
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-----------------|-------------:|----------:|---------------:|----------------------:|
| Hardware-1-1-1-1 | 69.84 | 2.70 | 0.02 | 0.02 |
| Hardware-1-1-2-1 | 63.40 | 1.72 | 0.02 | 0.02 |
| Hardware-1-1-3-1 | 62.02 | 1.64 | 0.02 | 0.02 |
| Hardware-1-1-4-1 | 69.61 | 1.90 | 0.02 | 0.02 |
| Hardware-1-1-5-1 | 61.78 | 1.75 | 0.02 | 0.02 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-----------------|-------------:|----------:|---------------:|----------------------:|
| Hardware-1-1-1-1 | 0.45 | 0.01 | 0.00 | 0.00 |
| Hardware-1-1-2-1 | 0.55 | 0.01 | 0.00 | 0.00 |
| Hardware-1-1-3-1 | 0.52 | 0.04 | 0.00 | 0.00 |
| Hardware-1-1-4-1 | 0.44 | 0.02 | 0.00 | 0.00 |
| Hardware-1-1-5-1 | 0.52 | 0.02 | 0.00 | 0.00 |
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: every round has non-zero read or write IOPS
The Per Phase table lists one row per block size set by -xfbs, at fixed queue depth 1.
7. Checkpoint writeback bandwidth
Large-block sequential writes without a per-write fsync, approximating how fast checkpointer/bgwriter can flush dirty pages during a checkpoint.
bexhoma hardware \
-dbms Hardware \
-xht fio \
-xts 4G \
-xtd 60 \
-xfrw write \
-xfbs 1M,4M,16M \
-xfid 4,16 \
-xfe libaio \
-nbp 1 \
-nbt 1 \
-ne 1 \
-m \
-ms $BEXHOMA_MS \
-tr \
-rsr \
-rss 50Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rnn $BEXHOMA_NODE_SUT -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_hardware_fio_checkpoint_writeback.log
Result
testcase_hardware_fio_checkpoint_writeback.log
## Show Summary
### Workload
Hardware Benchmark (fio)
* Type: hardware
* Duration: 864s
* Code: 1783200536
* fio/sysbench driver runs the experiment.
* This experiment measures raw hardware I/O (fio) or CPU/memory (sysbench) performance.
* Benchmark tool: fio.
* Test file size is '4G', duration per round is 60s.
* I/O pattern(s) swept: ['write'].
* Block size(s) swept: ['1M', '4M', '16M'].
* Queue depth(s) swept: [4, 16].
* I/O engine(s) swept: ['libaio'].
* Fsync interval(s) swept: [0].
* Fdatasync interval(s) swept: [0].
* Experiment uses bexhoma version 0.10.2.
* System metrics are monitored by sidecar containers.
* Experiment is limited to DBMS ['Hardware'].
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker36.
* Database is persisted to disk of type shared and size 50Gi. Persistent storage is removed at experiment start.
* Benchmarking is tested with [1] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* Hardware-1-1-1-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:831367
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783200536
* Hardware-1-1-2-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:820536
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783200536
* Hardware-1-1-3-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:827659
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783200536
* Hardware-1-1-4-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:821051
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783200536
* Hardware-1-1-5-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:825418
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783200536
* Hardware-1-1-6-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:821889
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783200536
### SUT Container Restarts
* bexhoma-sut-hardware-1-1783200536-f46bc844-kqq79: 0
### Workflow
#### Actual
* DBMS Hardware-1 - Experiment 1 Client 1: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 2: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 3: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 4: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 5: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 6: hardware (1 pods)
#### Planned
* DBMS Hardware-1 - Experiment 1 Client 1: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 2: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 3: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 4: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 5: hardware (1 pods)
* DBMS Hardware-1 - Experiment 1 Client 6: hardware (1 pods)
### Execution
#### Per Connection
| DBMS | phase | job | experiment_run | client | benchmark_run | child | duration | hardware_fio_rw | hardware_fio_bs | hardware_fio_iodepth | hardware_fio_engine | hardware_fio_fsync | hardware_fio_fdatasync | hardware_fio_rwmixread | hardware_fio_numjobs | hardware_fio_read_iops | hardware_fio_write_iops | hardware_fio_read_lat_p95_ms | hardware_fio_write_lat_p95_ms | hardware_fio_read_lat_p99_ms | hardware_fio_write_lat_p99_ms | hardware_threads | hardware_sysbench_cpu_events_per_sec | hardware_sysbench_cpu_total_time_s | hardware_sysbench_cpu_lat_p95_ms | hardware_sysbench_memory_ops_per_sec | hardware_sysbench_memory_throughput_mibps | hardware_sysbench_memory_lat_p95_ms | errors |
|:-------------------|:---------------|:-----------------|-----------------:|---------:|----------------:|--------:|-----------:|:------------------|:------------------|-----------------------:|:----------------------|---------------------:|-------------------------:|-------------------------:|-----------------------:|-------------------------:|--------------------------:|-------------------------------:|--------------------------------:|-------------------------------:|--------------------------------:|-------------------:|---------------------------------------:|-------------------------------------:|-----------------------------------:|---------------------------------------:|--------------------------------------------:|--------------------------------------:|---------:|
| Hardware-1-1-1-1-1 | Hardware-1-1-1 | Hardware-1-1-1-1 | 1 | 1 | 1 | 1 | 63 | write | 1M | 4 | libaio | 0 | 0 | 50 | 1 | 0.00 | 76.80 | 0.00 | 104.33 | 0.00 | 164.63 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-2-1-1 | Hardware-1-1-2 | Hardware-1-1-2-1 | 1 | 2 | 1 | 1 | 63 | write | 1M | 16 | libaio | 0 | 0 | 50 | 1 | 0.00 | 237.50 | 0.00 | 143.65 | 0.00 | 212.86 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-3-1-1 | Hardware-1-1-3 | Hardware-1-1-3-1 | 1 | 3 | 1 | 1 | 63 | write | 4M | 4 | libaio | 0 | 0 | 50 | 1 | 0.00 | 49.68 | 0.00 | 139.46 | 0.00 | 206.57 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-4-1-1 | Hardware-1-1-4 | Hardware-1-1-4-1 | 1 | 4 | 1 | 1 | 63 | write | 4M | 16 | libaio | 0 | 0 | 50 | 1 | 0.00 | 142.58 | 0.00 | 235.93 | 0.00 | 362.81 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-5-1-1 | Hardware-1-1-5 | Hardware-1-1-5-1 | 1 | 5 | 1 | 1 | 63 | write | 16M | 4 | libaio | 0 | 0 | 50 | 1 | 0.00 | 3.11 | 0.00 | 1182.79 | 0.00 | 1283.46 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-6-1-1 | Hardware-1-1-6 | Hardware-1-1-6-1 | 1 | 6 | 1 | 1 | 63 | write | 16M | 16 | libaio | 0 | 0 | 50 | 1 | 0.00 | 3.08 | 0.00 | 5536.48 | 0.00 | 5737.81 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
#### Per Phase
| DBMS | phase | experiment_run | client | benchmark_run | pod_count | duration | hardware_fio_rw | hardware_fio_bs | hardware_fio_iodepth | hardware_fio_engine | hardware_fio_fsync | hardware_fio_fdatasync | hardware_fio_rwmixread | hardware_fio_read_iops | hardware_fio_write_iops | hardware_fio_read_lat_p95_ms | hardware_fio_write_lat_p95_ms | hardware_fio_read_lat_p99_ms | hardware_fio_write_lat_p99_ms | hardware_threads | hardware_sysbench_cpu_events_per_sec | hardware_sysbench_cpu_total_time_s | hardware_sysbench_cpu_lat_p95_ms | hardware_sysbench_memory_ops_per_sec | hardware_sysbench_memory_throughput_mibps | hardware_sysbench_memory_lat_p95_ms | errors |
|:---------------|:---------------|-----------------:|---------:|----------------:|------------:|-----------:|:------------------|:------------------|-----------------------:|:----------------------|---------------------:|-------------------------:|-------------------------:|-------------------------:|--------------------------:|-------------------------------:|--------------------------------:|-------------------------------:|--------------------------------:|-------------------:|---------------------------------------:|-------------------------------------:|-----------------------------------:|---------------------------------------:|--------------------------------------------:|--------------------------------------:|---------:|
| Hardware-1-1-1 | Hardware-1-1-1 | 1 | 1 | 1 | 1 | 63 | write | 1M | 4 | libaio | 0 | 0 | 50 | 0.00 | 76.80 | 0.00 | 104.33 | 0.00 | 164.63 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-2 | Hardware-1-1-2 | 1 | 2 | 1 | 1 | 63 | write | 1M | 16 | libaio | 0 | 0 | 50 | 0.00 | 237.50 | 0.00 | 143.65 | 0.00 | 212.86 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-3 | Hardware-1-1-3 | 1 | 3 | 1 | 1 | 63 | write | 4M | 4 | libaio | 0 | 0 | 50 | 0.00 | 49.68 | 0.00 | 139.46 | 0.00 | 206.57 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-4 | Hardware-1-1-4 | 1 | 4 | 1 | 1 | 63 | write | 4M | 16 | libaio | 0 | 0 | 50 | 0.00 | 142.58 | 0.00 | 235.93 | 0.00 | 362.81 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-5 | Hardware-1-1-5 | 1 | 5 | 1 | 1 | 63 | write | 16M | 4 | libaio | 0 | 0 | 50 | 0.00 | 3.11 | 0.00 | 1182.79 | 0.00 | 1283.46 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| Hardware-1-1-6 | Hardware-1-1-6 | 1 | 6 | 1 | 1 | 63 | write | 16M | 16 | libaio | 0 | 0 | 50 | 0.00 | 3.08 | 0.00 | 5536.48 | 0.00 | 5737.81 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
### Monitoring
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-----------------|-------------:|----------:|---------------:|----------------------:|
| Hardware-1-1-1-1 | 62.71 | 2.49 | 0.03 | 0.03 |
| Hardware-1-1-2-1 | 67.90 | 1.91 | 0.04 | 0.04 |
| Hardware-1-1-3-1 | 59.10 | 1.61 | 0.04 | 0.04 |
| Hardware-1-1-4-1 | 72.89 | 2.15 | 0.09 | 0.09 |
| Hardware-1-1-5-1 | 64.14 | 1.67 | 0.09 | 0.09 |
| Hardware-1-1-6-1 | 62.89 | 2.54 | 0.27 | 0.27 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-----------------|-------------:|----------:|---------------:|----------------------:|
| Hardware-1-1-1-1 | 0.57 | 0.02 | 0.00 | 0.00 |
| Hardware-1-1-2-1 | 0.57 | 0.02 | 0.00 | 0.00 |
| Hardware-1-1-3-1 | 0.57 | 0.02 | 0.00 | 0.00 |
| Hardware-1-1-4-1 | 0.44 | 0.02 | 0.00 | 0.00 |
| Hardware-1-1-5-1 | 0.56 | 0.02 | 0.00 | 0.00 |
| Hardware-1-1-6-1 | 0.53 | 0.02 | 0.00 | 0.00 |
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: every round has non-zero read or write IOPS
The Per Phase table lists one row per block size/queue depth combination; bandwidth can be
derived from hardware_fio_write_iops × hardware_fio_bs.
8. OLTP/WAL contention proxy
A single-profile approximation of foreground OLTP traffic contending with WAL flushes on one
queue: mixed random read/write with fsync on the write side. This is not the same as true
concurrent checkpoint+WAL+OLTP contention — that needs several parallel benchmarker jobs with
different fio profiles running in the same round, which bexhoma does not yet support for Hardware
(the closest existing precedent is the TPC-H refresh-stream mechanism, see
bexhoma/experiments/CLAUDE.md §8, which could be adapted for this in the future). This proxy is
what is achievable today with the single-profile-per-round model used throughout this page.
bexhoma hardware \
-dbms Hardware \
-xht fio \
-xts 4G \
-xtd 60 \
-xfrw randrw \
-xfmx 70 \
-xfbs 8k \
-xfid 64 \
-xfe libaio \
-xfsy 1 \
-nbp 1 \
-nbt 1 \
-ne 1 \
-m \
-ms $BEXHOMA_MS \
-tr \
-rsr \
-rss 50Gi \
-rst $BEXHOMA_STORAGE_CLASS \
-rnn $BEXHOMA_NODE_SUT -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/testcase_hardware_fio_oltp_wal_contention_proxy.log
Result
testcase_hardware_fio_oltp_wal_contention_proxy.log
## Show Summary
### Workload
Hardware Benchmark (fio)
* Type: hardware
* Duration: 183s
* Code: 1783201425
* fio/sysbench driver runs the experiment.
* This experiment measures raw hardware I/O (fio) or CPU/memory (sysbench) performance.
* Benchmark tool: fio.
* Test file size is '4G', duration per round is 60s.
* I/O pattern(s) swept: ['randrw'].
* Block size(s) swept: ['8k'].
* Queue depth(s) swept: [64].
* I/O engine(s) swept: ['libaio'].
* Fsync interval(s) swept: [1].
* Fdatasync interval(s) swept: [0].
* Read mix percentage(s) swept: [70].
* Experiment uses bexhoma version 0.10.2.
* System metrics are monitored by sidecar containers.
* Experiment is limited to DBMS ['Hardware'].
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker36.
* Database is persisted to disk of type shared and size 50Gi. Persistent storage is removed at experiment start.
* Benchmarking is tested with [1] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* Hardware-1-1-1-1 uses docker image bexhoma/sut_hardware:0.10.2
* RAM:2164173246464
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:831834
* volume_size:50.0G
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783201425
### SUT Container Restarts
* bexhoma-sut-hardware-1-1783201425-655946dbcb-qxgkb: 0
### Workflow
#### Actual
* DBMS Hardware-1 - Experiment 1 Client 1: hardware (1 pods)
#### Planned
* DBMS Hardware-1 - Experiment 1 Client 1: hardware (1 pods)
### Execution
#### Per Connection
| DBMS | phase | job | experiment_run | client | benchmark_run | child | duration | hardware_fio_rw | hardware_fio_bs | hardware_fio_iodepth | hardware_fio_engine | hardware_fio_fsync | hardware_fio_fdatasync | hardware_fio_rwmixread | hardware_fio_numjobs | hardware_fio_read_iops | hardware_fio_write_iops | hardware_fio_read_lat_p95_ms | hardware_fio_write_lat_p95_ms | hardware_fio_read_lat_p99_ms | hardware_fio_write_lat_p99_ms | hardware_threads | hardware_sysbench_cpu_events_per_sec | hardware_sysbench_cpu_total_time_s | hardware_sysbench_cpu_lat_p95_ms | hardware_sysbench_memory_ops_per_sec | hardware_sysbench_memory_throughput_mibps | hardware_sysbench_memory_lat_p95_ms | errors |
|:-------------------|:---------------|:-----------------|-----------------:|---------:|----------------:|--------:|-----------:|:------------------|:------------------|-----------------------:|:----------------------|---------------------:|-------------------------:|-------------------------:|-----------------------:|-------------------------:|--------------------------:|-------------------------------:|--------------------------------:|-------------------------------:|--------------------------------:|-------------------:|---------------------------------------:|-------------------------------------:|-----------------------------------:|---------------------------------------:|--------------------------------------------:|--------------------------------------:|---------:|
| Hardware-1-1-1-1-1 | Hardware-1-1-1 | Hardware-1-1-1-1 | 1 | 1 | 1 | 1 | 72 | randrw | 8k | 64 | libaio | 1 | 0 | 70 | 1 | 1955.51 | 835.77 | 49.02 | 100.14 | 170.92 | 316.67 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
#### Per Phase
| DBMS | phase | experiment_run | client | benchmark_run | pod_count | duration | hardware_fio_rw | hardware_fio_bs | hardware_fio_iodepth | hardware_fio_engine | hardware_fio_fsync | hardware_fio_fdatasync | hardware_fio_rwmixread | hardware_fio_read_iops | hardware_fio_write_iops | hardware_fio_read_lat_p95_ms | hardware_fio_write_lat_p95_ms | hardware_fio_read_lat_p99_ms | hardware_fio_write_lat_p99_ms | hardware_threads | hardware_sysbench_cpu_events_per_sec | hardware_sysbench_cpu_total_time_s | hardware_sysbench_cpu_lat_p95_ms | hardware_sysbench_memory_ops_per_sec | hardware_sysbench_memory_throughput_mibps | hardware_sysbench_memory_lat_p95_ms | errors |
|:---------------|:---------------|-----------------:|---------:|----------------:|------------:|-----------:|:------------------|:------------------|-----------------------:|:----------------------|---------------------:|-------------------------:|-------------------------:|-------------------------:|--------------------------:|-------------------------------:|--------------------------------:|-------------------------------:|--------------------------------:|-------------------:|---------------------------------------:|-------------------------------------:|-----------------------------------:|---------------------------------------:|--------------------------------------------:|--------------------------------------:|---------:|
| Hardware-1-1-1 | Hardware-1-1-1 | 1 | 1 | 1 | 1 | 72 | randrw | 8k | 64 | libaio | 1 | 0 | 70 | 1955.51 | 835.77 | 49.02 | 100.14 | 170.92 | 316.67 | 1 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
### Monitoring
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-----------------|-------------:|----------:|---------------:|----------------------:|
| Hardware-1-1-1-1 | 70.17 | 1.80 | 0.02 | 0.02 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-----------------|-------------:|----------:|---------------:|----------------------:|
| Hardware-1-1-1-1 | 0.50 | 0.00 | 0.00 | 0.00 |
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: every round has non-zero read or write IOPS
hardware_fio_read_iops and hardware_fio_write_iops in the single result row can be compared
against the isolated randread result from the PostgreSQL-page-size depth sweep above at the same depth and block size.
Which fio sweep informs which PostgreSQL parameter
Covers all twelve fio sweeps across Example-Hardware.md and the eight above.
Sweep |
Informs |
|---|---|
Depth sweep + refinement |
|
Numjobs sweep |
|
Block-size sweep |
Reasoning about checkpoint/bgwriter I/O coalescing |
8k depth sweep, |
|
WAL sync fsync/fdatasync |
|
WAL group commit |
|
WAL record-size sweep |
Expectations around |
Checkpoint writeback bandwidth |
|
OLTP/WAL contention proxy |
Sanity check under mixed load |