Example: Application Metrics
In the following we demonstrate how to collect application metrics, that is, metrics of a DBMS.
Perform Benchmark
You will have to change the node selectors there (to names of nodes, that exist in your cluster - or to leave out the corresponding parameters):
BEXHOMA_NODE_SUT="cl-worker11"
BEXHOMA_NODE_LOAD="cl-worker19"
BEXHOMA_NODE_BENCHMARK="cl-worker19"
LOG_DIR="./logs_tests"
BEXHOMA_MS=1
BEXHOMA_STORAGE_CLASS="shared"
mkdir -p $LOG_DIR
For performing the experiment we can run the benchbase file.
PostgreSQL
Benchbase’s TPC-C
Example:
bexhoma benchbase \
-dbms PostgreSQL \
-sf 16 \
-xsd 5 \
-xtb 1024 \
-xnbf 16 \
-nbp 1,2 \
-nbt 160 \
-m \
-ma \
-mc \
-ms $BEXHOMA_MS \
-tr \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/docs_benchbase_postgresql_appmetrics.log
This
activates monitoring (
-m) cluster-wide (-mc)starts a clean instance of PostgreSQL (
-dbms)with a sidecar container for monitoring (
-ma)data directory inside a Docker container
starts 1 loader pod (per DBMS) that
creates TPC-C schema in the database
imports data for 16 (
-sf) warehouses into the DBMSusing all threads of driver machine (benchbase setting)
runs streams of TPC-C queries (per DBMS)
running for 5 (
-xsd) minuteseach stream (pod) having 16 threads to simulate 16 users (
-nbt)-nbp: first stream 1 pod, second stream 2 pods (8 threads each)target is 16x(
-ltf) 1024 (-xtb) ops
with a maximum of 1 DBMS per time (
-ms)tests if results match workflow (
-tr)shows a summary
Evaluate Results
Show docs_benchbase_postgresql_appmetrics.log
## Show Summary
### Workload
Benchbase Workload tpcc SF=16
* Type: benchbase
* Duration: 1233s
* Code: 1783869734
* Benchbase runs a TPC-C experiment.
* This experiment compares run time and resource consumption of Benchbase queries in different DBMS.
* Benchbase data is generated and loaded using several threads.
* Benchmark is 'tpcc'. Scaling factor is 16. Target is based on multiples of '1024'. Factors for benchmarking are [16]. Benchmarking runs for 5 minutes.
* Experiment uses bexhoma version 0.10.5.
* System metrics are monitored by a cluster-wide installation.
* Application metrics are monitored by sidecar containers.
* Experiment is limited to DBMS ['PostgreSQL'].
* Import is handled by 1 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker36.
* Database uses ephemeral storage of size 50Gi.
* Loading is tested with [1] threads, split into [1] pods.
* Benchmarking is tested with [160] threads, split into [1, 2] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* PostgreSQL-1-1-1-1 uses docker image postgres:18.3
* RAM:2164173213696
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:1073586
* cpu_list:0-223
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783869734
* TENANT_VOL:False
* PostgreSQL-1-1-2-1 uses docker image postgres:18.3
* RAM:2164173213696
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:1109056
* cpu_list:0-223
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783869734
* TENANT_VOL:False
### SUT Container Restarts
* bexhoma-sut-postgresql-1-1783869734-845d77c9f8-djxzm: 0 0
### Workflow
#### Actual
* DBMS PostgreSQL-1 - Experiment 1 Client 1: benchbase (1 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 2: benchbase (2 pods)
#### Planned
* DBMS PostgreSQL-1 - Experiment 1 Client 1: benchbase (1 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 2: benchbase (2 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:---------------|-----------------:|------:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| PostgreSQL-1-1 | 1 | 16.00 | 325.00 | 0.00 | 0.00 | 150.00 | 175.00 | 1 | 1 | | | 0 | False | 177.23 |
### Execution
#### Per Connection
| DBMS | phase | job | experiment_run | terminals | target | client | benchmark_run | child | tenant_id | time | num_errors | Throughput (requests/second) | Goodput (requests/second) | efficiency | Latency Distribution.95th Percentile Latency (microseconds) | Latency Distribution.Average Latency (microseconds) |
|:---------------------|:-----------------|:-------------------|-----------------:|------------:|---------:|---------:|----------------:|--------:|------------:|-------:|-------------:|-------------------------------:|----------------------------:|-------------:|--------------------------------------------------------------:|------------------------------------------------------:|
| PostgreSQL-1-1-1-1-1 | PostgreSQL-1-1-1 | PostgreSQL-1-1-1-1 | 1 | 160 | 16384 | 1 | 1 | 1 | 0 | 300.00 | 139 | 11427.15 | 11255.35 | 0.00 | 37416.00 | 13979.00 |
| PostgreSQL-1-1-2-1-1 | PostgreSQL-1-1-2 | PostgreSQL-1-1-2-1 | 1 | 80 | 8192 | 2 | 1 | 1 | 0 | 300.00 | 27 | 4378.83 | 4324.69 | 0.00 | 40392.00 | 18262.00 |
| PostgreSQL-1-1-2-1-2 | PostgreSQL-1-1-2 | PostgreSQL-1-1-2-1 | 1 | 80 | 8192 | 2 | 1 | 2 | 0 | 300.00 | 14 | 4396.02 | 4342.27 | 0.00 | 40181.00 | 18190.00 |
#### Per Phase
| DBMS | phase | experiment_run | terminals | target | benchmark_run | pod_count | tenant_id | time | num_errors | Throughput (requests/second) | Goodput (requests/second) | efficiency | Latency Distribution.95th Percentile Latency (microseconds) | Latency Distribution.Average Latency (microseconds) |
|:-----------------|:-----------------|-----------------:|------------:|---------:|----------------:|------------:|------------:|-------:|-------------:|-------------------------------:|----------------------------:|-------------:|--------------------------------------------------------------:|------------------------------------------------------:|
| PostgreSQL-1-1-1 | PostgreSQL-1-1-1 | 1 | 160 | 16384 | 1 | 1 | 0 | 300.00 | 139 | 11427.15 | 11255.35 | 0.00 | 37416.00 | 13979.00 |
| PostgreSQL-1-1-2 | PostgreSQL-1-1-2 | 1 | 160 | 16384 | 1 | 2 | 0 | 300.00 | 41 | 8774.85 | 8666.95 | 0.00 | 40392.00 | 18226.00 |
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 166.96 | 3.15 | 2.07 | 3.64 |
| PostgreSQL-1-1-2-1 | 166.96 | 3.15 | 2.07 | 3.64 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 1870.76 | 15.35 | 0.27 | 0.27 |
| PostgreSQL-1-1-2-1 | 1870.76 | 15.35 | 0.27 | 0.27 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 4175.96 | 16.51 | 7.36 | 12.21 |
| PostgreSQL-1-1-2-1 | 4286.92 | 16.17 | 10.16 | 16.00 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 5491.80 | 21.42 | 1.14 | 1.14 |
| PostgreSQL-1-1-2-1 | 5491.80 | 36.67 | 1.14 | 1.14 |
### Application Metrics
#### Loading phase: SUT deployment
| DBMS | Number of Idle Sessions | Number of Idle-in-transaction Sessions | Number of Idle-in-transaction Aborted Sessions | Number of Active Sessions | Number of Active Application Sessions |
|:-------------------|--------------------------:|-----------------------------------------:|-------------------------------------------------:|----------------------------:|----------------------------------------:|
| PostgreSQL-1-1-1-1 | 16.00 | 0.00 | 0.00 | 9.00 | 8.00 |
| PostgreSQL-1-1-2-1 | 16.00 | 0.00 | 0.00 | 9.00 | 8.00 |
#### Execution phase: SUT deployment
| DBMS | Number of Idle Sessions | Number of Idle-in-transaction Sessions | Number of Idle-in-transaction Aborted Sessions | Number of Active Sessions | Number of Active Application Sessions |
|:-------------------|--------------------------:|-----------------------------------------:|-------------------------------------------------:|----------------------------:|----------------------------------------:|
| PostgreSQL-1-1-1-1 | 9.00 | 76.00 | 0.00 | 160.00 | 160.00 |
| PostgreSQL-1-1-2-1 | 8.00 | 57.00 | 0.00 | 160.00 | 160.00 |
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Throughput (requests/second) contains no 0 or NaN
* TEST passed: Workflow as planned
The summary shows the first 5 application metrics aggregated per execution run. An extensive example for an evaluation is in the repository.
HammerDB’s TPC-C
Example:
bexhoma hammerdb \
-dbms PostgreSQL \
-sf 16 \
-xsd 5 \
-nlt 16 \
-nbp 1,2 \
-nbt 16 \
-xlat \
-m \
-ma \
-mc \
-ms $BEXHOMA_MS \
-tr \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/docs_hammerdb_postgresql_appmetrics.log
Show docs_hammerdb_postgresql_appmetrics.log
## Show Summary
### Workload
HammerDB Workload SF=16 (warehouses for TPC-C)
* Type: tpcc
* Duration: 1371s
* Code: 1783874903
* HammerDB runs the benchmark.
* This experiment compares run time and resource consumption of TPC-C queries in different DBMS.
* TPC-C data is generated and loaded using several threads.
* Scaling factor (i.e., number of warehouses) is 16. Benchmarking runs for 5 minutes. Benchmarking also logs latencies.
* Experiment uses bexhoma version 0.10.5.
* System metrics are monitored by a cluster-wide installation.
* Application metrics are monitored by sidecar containers.
* Experiment is limited to DBMS ['PostgreSQL'].
* Import is handled by 1 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker36.
* Database uses ephemeral storage of size 50Gi.
* Loading is tested with [16] threads, split into [1] pods.
* Benchmarking is tested with [16] threads, split into [1, 2] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* PostgreSQL-1-1-1-1 uses docker image postgres:18.3
* RAM:2164173213696
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:1073870
* cpu_list:0-223
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783874903
* PostgreSQL-1-1-2-1 uses docker image postgres:18.3
* RAM:2164173213696
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:1088087
* cpu_list:0-223
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783874903
### SUT Container Restarts
* bexhoma-sut-postgresql-1-1783874903-7ddd559b5b-qdkqc: 0 0
### Workflow
#### Actual
* DBMS PostgreSQL-1 - Experiment 1 Client 1: hammerdb (1 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 2: hammerdb (2 pods)
#### Planned
* DBMS PostgreSQL-1 - Experiment 1 Client 1: hammerdb (1 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 2: hammerdb (2 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:---------------|-----------------:|------:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| PostgreSQL-1-1 | 1 | 16.00 | 111.00 | 1.00 | 0.00 | 39.00 | 71.00 | 1 | 16 | | None | 0 | False | 518.92 |
### Execution
#### Per Connection
| DBMS | phase | job | experiment_run | vusers | client | benchmark_run | child | NOPM | TPM | efficiency | duration | errors | P95 [ms] | P99 [ms] |
|:---------------------|:-----------------|:-------------------|-----------------:|---------:|---------:|----------------:|--------:|-------:|-------:|-------------:|-----------:|---------:|-----------:|-----------:|
| PostgreSQL-1-1-1-1-1 | PostgreSQL-1-1-1 | PostgreSQL-1-1-1-1 | 1 | 16 | 1 | 1 | 1 | 260019 | 598280 | 0.00 | 5 | 0 | 2.09 | 2.73 |
| PostgreSQL-1-1-2-1-1 | PostgreSQL-1-1-2 | PostgreSQL-1-1-2-1 | 1 | 8 | 2 | 1 | 1 | 49535 | 114420 | 0.00 | 5 | 0 | 1.62 | 1.94 |
| PostgreSQL-1-1-2-1-2 | PostgreSQL-1-1-2 | PostgreSQL-1-1-2-1 | 1 | 8 | 2 | 1 | 2 | 49501 | 114394 | 0.00 | 5 | 0 | 1.64 | 1.95 |
#### Per Phase
| DBMS | phase | experiment_run | vusers | client | benchmark_run | pod_count | P95 [ms] | P99 [ms] | efficiency | NOPM | TPM | duration | errors |
|:-----------------|:-----------------|-----------------:|---------:|---------:|----------------:|------------:|-----------:|-----------:|-------------:|----------:|----------:|-----------:|---------:|
| PostgreSQL-1-1-1 | PostgreSQL-1-1-1 | 1 | 16 | 1 | 1 | 1 | 2.09 | 2.73 | 0.00 | 260019.00 | 598280.00 | 5 | 0 |
| PostgreSQL-1-1-2 | PostgreSQL-1-1-2 | 1 | 16 | 2 | 1 | 2 | 1.64 | 1.95 | 0.00 | 49518.00 | 114407.00 | 5 | 0 |
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 37.23 | 0.94 | 1.67 | 2.38 |
| PostgreSQL-1-1-2-1 | 37.23 | 0.94 | 1.67 | 2.38 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 0.02 | 0.00 | 0.00 | 0.00 |
| PostgreSQL-1-1-2-1 | 0.02 | 0.00 | 0.00 | 0.00 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 2620.30 | 7.65 | 5.56 | 9.53 |
| PostgreSQL-1-1-2-1 | 6174.56 | 14.85 | 6.50 | 11.22 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 626.76 | 1.58 | 1.79 | 1.79 |
| PostgreSQL-1-1-2-1 | 626.76 | 1.84 | 1.79 | 1.79 |
### Application Metrics
#### Loading phase: SUT deployment
| DBMS | Number of Idle Sessions | Number of Idle-in-transaction Sessions | Number of Idle-in-transaction Aborted Sessions | Number of Active Sessions | Number of Active Application Sessions |
|:-------------------|--------------------------:|-----------------------------------------:|-------------------------------------------------:|----------------------------:|----------------------------------------:|
| PostgreSQL-1-1-1-1 | 16.00 | 0.00 | 0.00 | 10.00 | 0.00 |
| PostgreSQL-1-1-2-1 | 16.00 | 0.00 | 0.00 | 10.00 | 0.00 |
#### Execution phase: SUT deployment
| DBMS | Number of Idle Sessions | Number of Idle-in-transaction Sessions | Number of Idle-in-transaction Aborted Sessions | Number of Active Sessions | Number of Active Application Sessions |
|:-------------------|--------------------------:|-----------------------------------------:|-------------------------------------------------:|----------------------------:|----------------------------------------:|
| PostgreSQL-1-1-1-1 | 13.00 | 0.00 | 0.00 | 17.00 | 0.00 |
| PostgreSQL-1-1-2-1 | 8.00 | 0.00 | 0.00 | 17.00 | 0.00 |
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: NOPM contains no 0 or NaN
* TEST passed: Workflow as planned
TPC-H
Example:
bexhoma tpch \
-dbms PostgreSQL \
-sf 3 \
-nlp 2 \
-nlt 8 \
-xii -xic -xis \
-xdt \
-m \
-ma \
-mc \
-ms $BEXHOMA_MS \
-t 1200 \
-tr \
-lr 64Gi \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/docs_tpch_postgresql_appmetrics.log
Show docs_tpch_postgresql_appmetrics.log
## Show Summary
### Workload
TPC-H Queries SF=3
* Type: tpch
* Duration: 593s
* Code: 1783872105
* This includes the reading queries of TPC-H.
* This experiment compares run time and resource consumption of TPC-H queries in different DBMS.
* TPC-H (SF=3) data is loaded and benchmark is executed.
* Query ordering is Q1 - Q22.
* All instances use the same query parameters.
* Timeout per query is 1200.
* Import sets indexes and constraints after loading and recomputes statistics.
* Experiment uses bexhoma version 0.10.5.
* System metrics are monitored by a cluster-wide installation.
* Application metrics are monitored by sidecar containers.
* Experiment is limited to DBMS ['PostgreSQL'].
* Import is handled by 2 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker36.
* Database uses ephemeral storage of size 50Gi.
* Loading is tested with [8] threads, split into [2] pods.
* Benchmarking is tested with [1] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* PostgreSQL-1-1-1-1-1 uses docker image postgres:18.3
* RAM:2164173213696
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:1078840
* cpu_list:0-223
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:16Gi
* limits_memory:64Gi
* eval_parameters
* code:1783872105
### SUT Container Restarts
* bexhoma-sut-postgresql-1-1783872105-dc4958648-8n9lw: 0 0
### Workflow
#### Actual
* DBMS PostgreSQL-1 - Experiment 1 Client 1: tpch (1 pods)
#### Planned
* DBMS PostgreSQL-1 - Experiment 1 Client 1: tpch (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:---------------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| PostgreSQL-1-1 | 1 | 3.00 | 360.00 | 1.00 | 109.00 | 40.00 | 208.00 | 2 | 0 | | | 0 | False | 30.00 |
### Execution
#### Per Connection
| DBMS | configuration | phase | job | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id | pod |
|:---------------------|:----------------|:-----------------|:-------------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|:---------------------|
| PostgreSQL-1-1-1-1-1 | PostgreSQL-1 | PostgreSQL-1-1-1 | PostgreSQL-1-1-1-1 | 1 | 1 | 1 | 1 | 3.00 | 22 | 28 | 0.60 | 19258.27 | 8485.71 | 0 | PostgreSQL-1-1-1-1-1 |
#### Per Phase
| | phase | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id |
|:-----------------|:-----------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|
| PostgreSQL-1-1-1 | PostgreSQL-1-1-1 | 1 | 1 | 1 | 1 | 3.00 | 22 | 28 | 0.60 | 19258.27 | 8485.71 | 0 |
### Latency of Timer Execution [ms]
| Queries | PostgreSQL-1-1-1-1-1 |
|:----------------------------------------------------|-----------------------:|
| Pricing Summary Report (TPC-H Q1) | 1677.85 |
| Minimum Cost Supplier Query (TPC-H Q2) | 501.69 |
| Shipping Priority (TPC-H Q3) | 496.26 |
| Order Priority Checking Query (TPC-H Q4) | 246.86 |
| Local Supplier Volume (TPC-H Q5) | 552.25 |
| Forecasting Revenue Change (TPC-H Q6) | 612.16 |
| Volume Shipping Query (TPC-H Q7) | 795.29 |
| National Market Share (TPC-H Q8) | 353.20 |
| Product Type Profit Measure (TPC-H Q9) | 867.59 |
| Returned Item Reporting Query (TPC-H Q10) | 861.70 |
| Important Stock Identification (TPC-H Q11) | 173.32 |
| Shipping Modes and Order Priority (TPC-H Q12) | 487.12 |
| Customer Distribution (TPC-H Q13) | 2428.73 |
| Promotion Effect Query (TPC-H Q14) | 550.68 |
| Top Supplier Query (TPC-H Q15) | 403.72 |
| Parts/Supplier Relationship (TPC-H Q16) | 388.51 |
| Small-Quantity-Order Revenue (TPC-H Q17) | 1566.50 |
| Large Volume Customer (TPC-H Q18) | 7652.97 |
| Discounted Revenue (TPC-H Q19) | 89.37 |
| Potential Part Promotion (TPC-H Q20) | 295.38 |
| Suppliers Who Kept Orders Waiting Query (TPC-H Q21) | 522.18 |
| Global Sales Opportunity Query (TPC-H Q22) | 138.93 |
### Errors (failed queries)
No errors
### Warnings (result mismatch)
No warnings
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 58.79 | 1.21 | 4.61 | 8.87 |
### Loading phase: component data generator
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 131.13 | 2.00 | 0.30 | 2.27 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 18.28 | 0.00 | 0.00 | 0.00 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 6.38 | 0.23 | 4.41 | 9.05 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 0.02 | 0.00 | 0.00 | 0.00 |
### Application Metrics
#### Loading phase: SUT deployment
| DBMS | Number of Idle Sessions | Number of Idle-in-transaction Sessions | Number of Idle-in-transaction Aborted Sessions | Number of Active Sessions | Number of Active Application Sessions |
|:-------------------|--------------------------:|-----------------------------------------:|-------------------------------------------------:|----------------------------:|----------------------------------------:|
| PostgreSQL-1-1-1-1 | 1.00 | 0.00 | 0.00 | 3.00 | 2.00 |
#### Execution phase: SUT deployment
| DBMS | Number of Idle Sessions | Number of Idle-in-transaction Sessions | Number of Idle-in-transaction Aborted Sessions | Number of Active Sessions | Number of Active Application Sessions |
|:-------------------|--------------------------:|-----------------------------------------:|-------------------------------------------------:|----------------------------:|----------------------------------------:|
| PostgreSQL-1-1-1-1 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 |
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component data generator contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Geo Times [s] contains no 0 or NaN
* TEST passed: Power@Size [~Q/h] contains no 0 or NaN
* TEST passed: Throughput@Size contains no 0 or NaN
* TEST passed: No SQL errors
* TEST passed: No SQL warnings
* TEST passed: Workflow as planned
TPC-DS
Example:
bexhoma tpcds \
-dbms PostgreSQL \
-sf 3 \
-nlp 2 \
-nlt 8 \
-xii -xic -xis \
-xdt \
-m \
-ma \
-mc \
-ms $BEXHOMA_MS \
-t 1200 \
-tr \
-lr 64Gi \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/docs_tpcds_postgresql_appmetrics.log
Show docs_tpcds_postgresql_appmetrics.log
## Show Summary
### Workload
TPC-DS Queries SF=3
* Type: tpcds
* Duration: 1990s
* Code: 1783872787
* This includes the reading queries of TPC-DS.
* This experiment compares run time and resource consumption of TPC-DS queries in different DBMS.
* TPC-DS (SF=3) data is loaded and benchmark is executed.
* Query ordering is Q1 - Q99.
* All instances use the same query parameters.
* Timeout per query is 1200.
* Import sets indexes and constraints after loading and recomputes statistics.
* Experiment uses bexhoma version 0.10.5.
* System metrics are monitored by a cluster-wide installation.
* Application metrics are monitored by sidecar containers.
* Experiment is limited to DBMS ['PostgreSQL'].
* Import is handled by 2 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker36.
* Database uses ephemeral storage of size 50Gi.
* Loading is tested with [8] threads, split into [2] pods.
* Benchmarking is tested with [1] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* PostgreSQL-1-1-1-1-1 uses docker image postgres:18.3
* RAM:2164173213696
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:1084752
* cpu_list:0-223
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:16Gi
* limits_memory:64Gi
* eval_parameters
* code:1783872787
### SUT Container Restarts
* bexhoma-sut-postgresql-1-1783872787-778cd88656-5brqp: 0 0
### Workflow
#### Actual
* DBMS PostgreSQL-1 - Experiment 1 Client 1: tpcds (1 pods)
#### Planned
* DBMS PostgreSQL-1 - Experiment 1 Client 1: tpcds (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:---------------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| PostgreSQL-1-1 | 1 | 3.00 | 1405.00 | 1.00 | 522.00 | 121.00 | 760.00 | 2 | 0 | | None | 0 | False | 7.69 |
### Execution
#### Per Connection
| DBMS | configuration | phase | job | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id | pod |
|:---------------------|:----------------|:-----------------|:-------------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|:---------------------|
| PostgreSQL-1-1-1-1-1 | PostgreSQL-1 | PostgreSQL-1-1-1 | PostgreSQL-1-1-1-1 | 1 | 1 | 1 | 1 | 3.00 | 99 | 812 | 0.75 | 14825.04 | 1316.75 | -1 | PostgreSQL-1-1-1-1-1 |
#### Per Phase
| | phase | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id |
|:-----------------|:-----------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|
| PostgreSQL-1-1-1 | PostgreSQL-1-1-1 | 1 | 1 | 1 | 1 | 3.00 | 99 | 812 | 0.75 | 14825.04 | 1316.75 | -1 |
### Latency of Timer Execution [ms]
| Queries | PostgreSQL-1-1-1-1-1 |
|:--------------|-----------------------:|
| TPC-DS Q1 | 164.94 |
| TPC-DS Q2 | 443.44 |
| TPC-DS Q3 | 1091.85 |
| TPC-DS Q4 | 18701.26 |
| TPC-DS Q5 | 761.15 |
| TPC-DS Q6 | 225482.02 |
| TPC-DS Q7 | 592.82 |
| TPC-DS Q8 | 224.72 |
| TPC-DS Q9 | 2591.35 |
| TPC-DS Q10 | 1266.62 |
| TPC-DS Q11 | 11635.66 |
| TPC-DS Q12 | 148.50 |
| TPC-DS Q13 | 1849.96 |
| TPC-DS Q14a+b | 7542.65 |
| TPC-DS Q15 | 307.19 |
| TPC-DS Q16 | 426.38 |
| TPC-DS Q17 | 718.01 |
| TPC-DS Q18 | 646.54 |
| TPC-DS Q19 | 414.75 |
| TPC-DS Q20 | 259.17 |
| TPC-DS Q21 | 307.20 |
| TPC-DS Q22 | 8227.72 |
| TPC-DS Q23a+b | 14200.24 |
| TPC-DS Q24a+b | 2112.58 |
| TPC-DS Q25 | 772.76 |
| TPC-DS Q26 | 463.25 |
| TPC-DS Q27 | 44.94 |
| TPC-DS Q28 | 2301.12 |
| TPC-DS Q29 | 885.93 |
| TPC-DS Q30 | 75606.63 |
| TPC-DS Q31 | 4178.54 |
| TPC-DS Q32 | 302.05 |
| TPC-DS Q33 | 1085.23 |
| TPC-DS Q34 | 73.09 |
| TPC-DS Q35 | 1387.18 |
| TPC-DS Q36 | 39.13 |
| TPC-DS Q37 | 443.25 |
| TPC-DS Q38 | 2655.52 |
| TPC-DS Q39a+b | 5479.35 |
| TPC-DS Q40 | 244.43 |
| TPC-DS Q41 | 3776.48 |
| TPC-DS Q42 | 174.15 |
| TPC-DS Q43 | 361.42 |
| TPC-DS Q44 | 1.85 |
| TPC-DS Q45 | 188.24 |
| TPC-DS Q46 | 63.10 |
| TPC-DS Q47 | 2708.73 |
| TPC-DS Q48 | 1186.25 |
| TPC-DS Q49 | 720.83 |
| TPC-DS Q50 | 2218.81 |
| TPC-DS Q51 | 3002.19 |
| TPC-DS Q52 | 165.83 |
| TPC-DS Q53 | 246.72 |
| TPC-DS Q54 | 66.09 |
| TPC-DS Q55 | 166.54 |
| TPC-DS Q56 | 704.60 |
| TPC-DS Q57 | 2640.18 |
| TPC-DS Q58 | 810.58 |
| TPC-DS Q59 | 780.08 |
| TPC-DS Q60 | 803.72 |
| TPC-DS Q61 | 136.89 |
| TPC-DS Q62 | 197.40 |
| TPC-DS Q63 | 216.78 |
| TPC-DS Q64 | 1734.62 |
| TPC-DS Q65 | 1269.42 |
| TPC-DS Q66 | 578.22 |
| TPC-DS Q67 | 7467.14 |
| TPC-DS Q68 | 110.36 |
| TPC-DS Q69 | 414.44 |
| TPC-DS Q70 | 844.63 |
| TPC-DS Q71 | 767.72 |
| TPC-DS Q72 | 2368.33 |
| TPC-DS Q73 | 40.01 |
| TPC-DS Q74 | 2314.94 |
| TPC-DS Q75 | 3756.16 |
| TPC-DS Q76 | 323.18 |
| TPC-DS Q77 | 473.31 |
| TPC-DS Q78 | 2954.40 |
| TPC-DS Q79 | 217.06 |
| TPC-DS Q80 | 772.39 |
| TPC-DS Q81 | 325842.78 |
| TPC-DS Q82 | 502.83 |
| TPC-DS Q83 | 149.34 |
| TPC-DS Q84 | 84.64 |
| TPC-DS Q85 | 325.37 |
| TPC-DS Q86 | 766.95 |
| TPC-DS Q87 | 2558.67 |
| TPC-DS Q88 | 3632.09 |
| TPC-DS Q89 | 249.48 |
| TPC-DS Q90 | 333.37 |
| TPC-DS Q91 | 154.33 |
| TPC-DS Q92 | 79.53 |
| TPC-DS Q93 | 487.39 |
| TPC-DS Q94 | 313.52 |
| TPC-DS Q95 | 9248.51 |
| TPC-DS Q96 | 173.65 |
| TPC-DS Q97 | 757.93 |
| TPC-DS Q98 | 422.70 |
| TPC-DS Q99 | 618.82 |
### Errors (failed queries)
No errors
### Warnings (result mismatch)
No warnings
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 219.06 | 2.43 | 6.31 | 13.17 |
### Loading phase: component data generator
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 499.31 | 2.00 | 0.01 | 3.62 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 29.58 | 0.42 | 0.00 | 0.00 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 996.50 | 2.58 | 6.67 | 13.53 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 25.78 | 0.12 | 0.34 | 0.34 |
### Application Metrics
#### Loading phase: SUT deployment
| DBMS | Number of Idle Sessions | Number of Idle-in-transaction Sessions | Number of Idle-in-transaction Aborted Sessions | Number of Active Sessions | Number of Active Application Sessions |
|:-------------------|--------------------------:|-----------------------------------------:|-------------------------------------------------:|----------------------------:|----------------------------------------:|
| PostgreSQL-1-1-1-1 | 1.00 | 0.00 | 0.00 | 6.00 | 5.00 |
#### Execution phase: SUT deployment
| DBMS | Number of Idle Sessions | Number of Idle-in-transaction Sessions | Number of Idle-in-transaction Aborted Sessions | Number of Active Sessions | Number of Active Application Sessions |
|:-------------------|--------------------------:|-----------------------------------------:|-------------------------------------------------:|----------------------------:|----------------------------------------:|
| PostgreSQL-1-1-1-1 | 1.00 | 0.00 | 0.00 | 6.00 | 6.00 |
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component data generator contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Geo Times [s] contains no 0 or NaN
* TEST passed: Power@Size [~Q/h] contains no 0 or NaN
* TEST passed: Throughput@Size contains no 0 or NaN
* TEST passed: No SQL errors
* TEST passed: No SQL warnings
* TEST passed: Workflow as planned
YCSB
Example:
bexhoma ycsb \
-dbms PostgreSQL \
-sf 3 \
-xwl a \
-xtb 16384 \
-xnbf 2,3 \
-xnlf 4 \
-nc 1 \
-ne 1 \
-nlp 8 \
-nlt 64 \
-nbp 1,8 \
-nbt 64 \
-m \
-ma \
-mc \
-ms $BEXHOMA_MS \
-tr \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/docs_ycsb_postgresql_appmetrics.log
Show docs_ycsb_postgresql_appmetrics.log
## Show Summary
### Workload
YCSB SF=3
* Type: ycsb
* Duration: 1074s
* Code: 1783870998
* YCSB driver runs the experiment.
* This experiment compares run time and resource consumption of YCSB queries.
* Workload is 'A'.
* Number of rows to insert is 3000000.
* Ordering of inserts is hashed.
* Number of operations is 3000000.
* Batch size is ''.
* Target is based on multiples of '16384'.
* Factors for loading are [4].
* Factors for benchmarking are [2, 3].
* Experiment uses bexhoma version 0.10.5.
* System metrics are monitored by a cluster-wide installation.
* Application metrics are monitored by sidecar containers.
* Experiment is limited to DBMS ['PostgreSQL'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker36.
* Database uses ephemeral storage of size 50Gi.
* Loading is tested with [64] threads, split into [8] pods.
* Benchmarking is tested with [64] threads, split into [1, 8] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* PostgreSQL-1-1-1-1 uses docker image postgres:18.3
* RAM:2164173213696
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:1098905
* cpu_list:0-223
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783870998
* PostgreSQL-1-1-2-1 uses docker image postgres:18.3
* RAM:2164173213696
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:1102912
* cpu_list:0-223
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783870998
* PostgreSQL-1-1-3-1 uses docker image postgres:18.3
* RAM:2164173213696
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:1079277
* cpu_list:0-223
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783870998
* PostgreSQL-1-1-4-1 uses docker image postgres:18.3
* RAM:2164173213696
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:1080879
* cpu_list:0-223
* args:['-c', 'max_connections=640', '-c', 'max_worker_processes=16', '-c', 'max_parallel_workers=16', '-c', 'max_parallel_workers_per_gather=8', '-c', 'max_parallel_maintenance_workers=4', '-c', 'shared_buffers=16GB', '-c', 'effective_cache_size=40GB', '-c', 'work_mem=512MB', '-c', 'maintenance_work_mem=2GB', '-c', 'autovacuum=off', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'max_wal_size=32GB', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_completion_target=1.0', '-c', 'lock_timeout=30s', '-c', 'idle_in_transaction_session_timeout=30000']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783870998
### SUT Container Restarts
* bexhoma-sut-postgresql-1-1783870998-57864c9499-xwstq: 0 0
### Workflow
#### Actual
* DBMS PostgreSQL-1 - Experiment 1 Client 1: ycsb (1 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 2: ycsb (8 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 3: ycsb (1 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 4: ycsb (8 pods)
#### Planned
* DBMS PostgreSQL-1 - Experiment 1 Client 1: ycsb (1 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 2: ycsb (8 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 3: ycsb (1 pods)
* DBMS PostgreSQL-1 - Experiment 1 Client 4: ycsb (8 pods)
### Loading
#### Per Connection
| connection | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) | sf | Throughput [SF/h] |
|:---------------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|-----:|--------------------:|
| PostgreSQL-1-1-0-1-1 | 1.00 | 8.00 | 8192.00 | 8.00 | 0.00 | 7805.18 | 48045.00 | 375000.00 | 3053.00 | 3.00 | 224.79 |
| PostgreSQL-1-1-0-1-2 | 1.00 | 8.00 | 8192.00 | 8.00 | 0.00 | 7651.97 | 49007.00 | 375000.00 | 2897.00 | 3.00 | 220.38 |
| PostgreSQL-1-1-0-1-3 | 1.00 | 8.00 | 8192.00 | 8.00 | 0.00 | 7857.02 | 47728.00 | 375000.00 | 2911.00 | 3.00 | 226.28 |
| PostgreSQL-1-1-0-1-4 | 1.00 | 8.00 | 8192.00 | 8.00 | 0.00 | 7689.15 | 48770.00 | 375000.00 | 2973.00 | 3.00 | 221.45 |
| PostgreSQL-1-1-0-1-5 | 1.00 | 8.00 | 8192.00 | 8.00 | 0.00 | 7562.77 | 49585.00 | 375000.00 | 3113.00 | 3.00 | 217.81 |
| PostgreSQL-1-1-0-1-6 | 1.00 | 8.00 | 8192.00 | 8.00 | 0.00 | 7377.68 | 50829.00 | 375000.00 | 2981.00 | 3.00 | 212.48 |
| PostgreSQL-1-1-0-1-7 | 1.00 | 8.00 | 8192.00 | 8.00 | 0.00 | 7825.54 | 47920.00 | 375000.00 | 3009.00 | 3.00 | 225.38 |
| PostgreSQL-1-1-0-1-8 | 1.00 | 8.00 | 8192.00 | 8.00 | 0.00 | 7328.37 | 51171.00 | 375000.00 | 3097.00 | 3.00 | 211.06 |
#### Per Run
| DBMS | experiment_run | threads | target | pod_count | exceptions | sf | Throughput [SF/h] | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) |
|:---------------|-----------------:|----------:|---------:|------------:|-------------:|-----:|--------------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|
| PostgreSQL-1-1 | 1.00 | 64.00 | 65536.00 | 8.00 | 0.00 | 3.00 | 211.06 | 61097.69 | 51171.00 | 3000000.00 | 3004.25 |
### Execution
#### Per Connection
| DBMS | phase | job | configuration | experiment_run | client | benchmark_run | child | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:---------------------|:-----------------|:-------------------|:----------------|-----------------:|---------:|----------------:|--------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| PostgreSQL-1-1-1-1-1 | PostgreSQL-1-1-1 | PostgreSQL-1-1-1-1 | PostgreSQL-1 | 1 | 1 | 1 | 1 | 64 | 32768 | 1 | 0 | 32663.02 | 91847.00 | 1499600 | 584.00 | 1500400 | 5335.00 |
| PostgreSQL-1-1-2-1-1 | PostgreSQL-1-1-2 | PostgreSQL-1-1-2-1 | PostgreSQL-1 | 1 | 2 | 1 | 1 | 8 | 4096 | 8 | 0 | 4084.83 | 91803.00 | 187722 | 567.00 | 187278 | 1436.00 |
| PostgreSQL-1-1-2-1-2 | PostgreSQL-1-1-2 | PostgreSQL-1-1-2-1 | PostgreSQL-1 | 1 | 2 | 1 | 2 | 8 | 4096 | 8 | 0 | 4085.68 | 91784.00 | 187180 | 549.00 | 187820 | 1456.00 |
| PostgreSQL-1-1-2-1-3 | PostgreSQL-1-1-2 | PostgreSQL-1-1-2-1 | PostgreSQL-1 | 1 | 2 | 1 | 3 | 8 | 4096 | 8 | 0 | 4085.81 | 91781.00 | 187412 | 583.00 | 187588 | 1459.00 |
| PostgreSQL-1-1-2-1-4 | PostgreSQL-1-1-2 | PostgreSQL-1-1-2-1 | PostgreSQL-1 | 1 | 2 | 1 | 4 | 8 | 4096 | 8 | 0 | 4085.01 | 91799.00 | 187238 | 530.00 | 187762 | 1422.00 |
| PostgreSQL-1-1-2-1-5 | PostgreSQL-1-1-2 | PostgreSQL-1-1-2-1 | PostgreSQL-1 | 1 | 2 | 1 | 5 | 8 | 4096 | 8 | 0 | 4085.46 | 91789.00 | 187414 | 548.00 | 187586 | 1480.00 |
| PostgreSQL-1-1-2-1-6 | PostgreSQL-1-1-2 | PostgreSQL-1-1-2-1 | PostgreSQL-1 | 1 | 2 | 1 | 6 | 8 | 4096 | 8 | 0 | 4084.88 | 91802.00 | 187486 | 615.00 | 187514 | 1467.00 |
| PostgreSQL-1-1-2-1-7 | PostgreSQL-1-1-2 | PostgreSQL-1-1-2-1 | PostgreSQL-1 | 1 | 2 | 1 | 7 | 8 | 4096 | 8 | 0 | 4084.39 | 91813.00 | 187323 | 568.00 | 187677 | 1447.00 |
| PostgreSQL-1-1-2-1-8 | PostgreSQL-1-1-2 | PostgreSQL-1-1-2-1 | PostgreSQL-1 | 1 | 2 | 1 | 8 | 8 | 4096 | 8 | 0 | 4085.19 | 91795.00 | 187479 | 526.00 | 187521 | 1470.00 |
| PostgreSQL-1-1-3-1-1 | PostgreSQL-1-1-3 | PostgreSQL-1-1-3-1 | PostgreSQL-1 | 1 | 3 | 1 | 1 | 64 | 49152 | 1 | 0 | 48924.48 | 61319.00 | 1498875 | 599.00 | 1501125 | 4147.00 |
| PostgreSQL-1-1-4-1-1 | PostgreSQL-1-1-4 | PostgreSQL-1-1-4-1 | PostgreSQL-1 | 1 | 4 | 1 | 1 | 8 | 6144 | 8 | 0 | 6121.45 | 61260.00 | 187286 | 588.00 | 187714 | 3421.00 |
| PostgreSQL-1-1-4-1-2 | PostgreSQL-1-1-4 | PostgreSQL-1-1-4-1 | PostgreSQL-1 | 1 | 4 | 1 | 2 | 8 | 6144 | 8 | 0 | 6120.55 | 61269.00 | 187589 | 616.00 | 187411 | 3711.00 |
| PostgreSQL-1-1-4-1-3 | PostgreSQL-1-1-4 | PostgreSQL-1-1-4-1 | PostgreSQL-1 | 1 | 4 | 1 | 3 | 8 | 6144 | 8 | 0 | 6119.75 | 61277.00 | 187712 | 604.00 | 187288 | 3407.00 |
| PostgreSQL-1-1-4-1-4 | PostgreSQL-1-1-4 | PostgreSQL-1-1-4-1 | PostgreSQL-1 | 1 | 4 | 1 | 4 | 8 | 6144 | 8 | 0 | 6120.25 | 61272.00 | 187227 | 589.00 | 187773 | 3409.00 |
| PostgreSQL-1-1-4-1-5 | PostgreSQL-1-1-4 | PostgreSQL-1-1-4-1 | PostgreSQL-1 | 1 | 4 | 1 | 5 | 8 | 6144 | 8 | 0 | 6116.86 | 61306.00 | 187633 | 681.00 | 187367 | 3627.00 |
| PostgreSQL-1-1-4-1-6 | PostgreSQL-1-1-4 | PostgreSQL-1-1-4-1 | PostgreSQL-1 | 1 | 4 | 1 | 6 | 8 | 6144 | 8 | 0 | 6117.26 | 61302.00 | 187496 | 648.00 | 187504 | 3525.00 |
| PostgreSQL-1-1-4-1-7 | PostgreSQL-1-1-4 | PostgreSQL-1-1-4-1 | PostgreSQL-1 | 1 | 4 | 1 | 7 | 8 | 6144 | 8 | 0 | 6111.37 | 61361.00 | 187230 | 582.00 | 187770 | 3481.00 |
| PostgreSQL-1-1-4-1-8 | PostgreSQL-1-1-4 | PostgreSQL-1-1-4-1 | PostgreSQL-1 | 1 | 4 | 1 | 8 | 8 | 6144 | 8 | 0 | 6120.75 | 61267.00 | 188343 | 623.00 | 186657 | 3493.00 |
#### Per Phase
| DBMS | phase | experiment_run | threads | target | benchmark_run | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:-----------------|:-----------------|-----------------:|----------:|---------:|----------------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| PostgreSQL-1-1-1 | PostgreSQL-1-1-1 | 1 | 64 | 32768 | 1 | 1 | 0 | 32663.02 | 91847.00 | 1499600 | 584.00 | 1500400 | 5335.00 |
| PostgreSQL-1-1-2 | PostgreSQL-1-1-2 | 1 | 64 | 32768 | 1 | 8 | 0 | 32681.25 | 91813.00 | 1499254 | 615.00 | 1500746 | 1480.00 |
| PostgreSQL-1-1-3 | PostgreSQL-1-1-3 | 1 | 64 | 49152 | 1 | 1 | 0 | 48924.48 | 61319.00 | 1498875 | 599.00 | 1501125 | 4147.00 |
| PostgreSQL-1-1-4 | PostgreSQL-1-1-4 | 1 | 64 | 49152 | 1 | 8 | 0 | 48948.24 | 61361.00 | 1500516 | 681.00 | 1499484 | 3711.00 |
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 272.58 | 5.77 | 3.89 | 4.90 |
| PostgreSQL-1-1-2-1 | 272.58 | 5.77 | 3.89 | 4.90 |
| PostgreSQL-1-1-3-1 | 272.58 | 5.77 | 3.89 | 4.90 |
| PostgreSQL-1-1-4-1 | 272.58 | 5.77 | 3.89 | 4.90 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 190.00 | 6.42 | 0.11 | 0.11 |
| PostgreSQL-1-1-2-1 | 190.00 | 6.42 | 0.11 | 0.11 |
| PostgreSQL-1-1-3-1 | 190.00 | 6.42 | 0.11 | 0.11 |
| PostgreSQL-1-1-4-1 | 190.00 | 6.42 | 0.11 | 0.11 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 252.09 | 2.81 | 5.08 | 9.02 |
| PostgreSQL-1-1-2-1 | 219.41 | 2.82 | 5.17 | 9.18 |
| PostgreSQL-1-1-3-1 | 190.85 | 3.97 | 5.22 | 9.28 |
| PostgreSQL-1-1-4-1 | 159.78 | 3.52 | 5.24 | 9.31 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------------|-------------:|----------:|---------------:|----------------------:|
| PostgreSQL-1-1-1-1 | 174.90 | 2.26 | 0.13 | 0.13 |
| PostgreSQL-1-1-2-1 | 204.63 | 5.20 | 0.13 | 0.13 |
| PostgreSQL-1-1-3-1 | 129.60 | 6.13 | 0.13 | 0.13 |
| PostgreSQL-1-1-4-1 | 255.38 | 8.59 | 0.13 | 0.13 |
### Application Metrics
#### Loading phase: SUT deployment
| DBMS | Number of Idle Sessions | Number of Idle-in-transaction Sessions | Number of Idle-in-transaction Aborted Sessions | Number of Active Sessions | Number of Active Application Sessions |
|:-------------------|--------------------------:|-----------------------------------------:|-------------------------------------------------:|----------------------------:|----------------------------------------:|
| PostgreSQL-1-1-1-1 | 42.00 | 0.00 | 0.00 | 64.00 | 64.00 |
| PostgreSQL-1-1-2-1 | 42.00 | 0.00 | 0.00 | 64.00 | 64.00 |
| PostgreSQL-1-1-3-1 | 42.00 | 0.00 | 0.00 | 64.00 | 64.00 |
| PostgreSQL-1-1-4-1 | 42.00 | 0.00 | 0.00 | 64.00 | 64.00 |
#### Execution phase: SUT deployment
| DBMS | Number of Idle Sessions | Number of Idle-in-transaction Sessions | Number of Idle-in-transaction Aborted Sessions | Number of Active Sessions | Number of Active Application Sessions |
|:-------------------|--------------------------:|-----------------------------------------:|-------------------------------------------------:|----------------------------:|----------------------------------------:|
| PostgreSQL-1-1-1-1 | 58.00 | 0.00 | 0.00 | 62.00 | 61.00 |
| PostgreSQL-1-1-2-1 | 60.00 | 0.00 | 0.00 | 8.00 | 8.00 |
| PostgreSQL-1-1-3-1 | 56.00 | 0.00 | 0.00 | 10.00 | 10.00 |
| PostgreSQL-1-1-4-1 | 55.00 | 0.00 | 0.00 | 15.00 | 15.00 |
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Execution Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: contains no FAILED column
MySQL
Benchbase’s TPC-C
Example:
bexhoma benchbase \
-dbms MySQL \
-sf 16 \
-xsd 5 \
-xtb 1024 \
-xnbf 16 \
-nbp 1,2 \
-nbt 160 \
-m \
-ma \
-mc \
-ms $BEXHOMA_MS \
-tr \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/docs_benchbase_mysql_appmetrics.log
This
activates monitoring (
-m) cluster-wide (-mc)starts a clean instance of MySQL (
-dbms)with a sidecar container for monitoring (
-ma)data directory inside a Docker container
starts 1 loader pod (per DBMS) that
creates TPC-C schema in the database
imports data for 16 (
-sf) warehouses into the DBMSusing all threads of driver machine (benchbase setting)
runs streams of TPC-C queries (per DBMS)
running for 5 (
-xsd) minuteseach stream (pod) having 16 threads to simulate 16 users (
-nbt)-nbp: first stream 1 pod, second stream 2 pods (8 threads each)target is 16x(
-ltf) 1024 (-xtb) ops
with a maximum of 1 DBMS per time (
-ms)tests if results match workflow (
-tr)shows a summary
Evaluate Results
Show docs_benchbase_mysql_appmetrics.log
## Show Summary
### Workload
Benchbase Workload tpcc SF=16
* Type: benchbase
* Duration: 1324s
* Code: 1783876300
* Benchbase runs a TPC-C experiment.
* This experiment compares run time and resource consumption of Benchbase queries in different DBMS.
* Benchbase data is generated and loaded using several threads.
* Benchmark is 'tpcc'. Scaling factor is 16. Target is based on multiples of '1024'. Factors for benchmarking are [16]. Benchmarking runs for 5 minutes.
* Experiment uses bexhoma version 0.10.5.
* System metrics are monitored by a cluster-wide installation.
* Application metrics are monitored by sidecar containers.
* Experiment is limited to DBMS ['MySQL'].
* Import is handled by 1 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker36.
* Database uses ephemeral storage of size 50Gi.
* Loading is tested with [1] threads, split into [1] pods.
* Benchmarking is tested with [160] threads, split into [1, 2] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* MySQL-1-1-1-1 uses docker image mysql:8.4.0
* RAM:2164173213696
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:1106015
* cpu_list:0-223
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--skip-log-bin', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783876300
* TENANT_VOL:False
* MySQL-1-1-2-1 uses docker image mysql:8.4.0
* RAM:2164173213696
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:1107744
* cpu_list:0-223
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--skip-log-bin', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1783876300
* TENANT_VOL:False
### SUT Container Restarts
* bexhoma-sut-mysql-1-1783876300-7fb75d884b-8gz9z: 0 0
### Workflow
#### Actual
* DBMS MySQL-1 - Experiment 1 Client 1: benchbase (1 pods)
* DBMS MySQL-1 - Experiment 1 Client 2: benchbase (2 pods)
#### Planned
* DBMS MySQL-1 - Experiment 1 Client 1: benchbase (1 pods)
* DBMS MySQL-1 - Experiment 1 Client 2: benchbase (2 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:----------|-----------------:|------:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| MySQL-1-1 | 1 | 16.00 | 361.00 | 0.00 | 0.00 | 163.00 | 198.00 | 1 | 1 | | | 0 | False | 159.56 |
### Execution
#### Per Connection
| DBMS | phase | job | experiment_run | terminals | target | client | benchmark_run | child | tenant_id | time | num_errors | Throughput (requests/second) | Goodput (requests/second) | efficiency | Latency Distribution.95th Percentile Latency (microseconds) | Latency Distribution.Average Latency (microseconds) |
|:----------------|:------------|:--------------|-----------------:|------------:|---------:|---------:|----------------:|--------:|------------:|-------:|-------------:|-------------------------------:|----------------------------:|-------------:|--------------------------------------------------------------:|------------------------------------------------------:|
| MySQL-1-1-1-1-1 | MySQL-1-1-1 | MySQL-1-1-1-1 | 1 | 160 | 16384 | 1 | 1 | 1 | -1 | 300.00 | 0 | 7888.33 | 7755.95 | 0.00 | 43647.00 | 20271.00 |
| MySQL-1-1-2-1-1 | MySQL-1-1-2 | MySQL-1-1-2-1 | 1 | 80 | 8192 | 2 | 1 | 1 | -1 | 300.00 | 0 | 3515.41 | 3464.91 | 0.00 | 48427.00 | 22746.00 |
| MySQL-1-1-2-1-2 | MySQL-1-1-2 | MySQL-1-1-2-1 | 1 | 80 | 8192 | 2 | 1 | 2 | -1 | 300.00 | 0 | 3445.78 | 3396.70 | 0.00 | 48401.00 | 23206.00 |
#### Per Phase
| DBMS | phase | experiment_run | terminals | target | benchmark_run | pod_count | tenant_id | time | num_errors | Throughput (requests/second) | Goodput (requests/second) | efficiency | Latency Distribution.95th Percentile Latency (microseconds) | Latency Distribution.Average Latency (microseconds) |
|:------------|:------------|-----------------:|------------:|---------:|----------------:|------------:|------------:|-------:|-------------:|-------------------------------:|----------------------------:|-------------:|--------------------------------------------------------------:|------------------------------------------------------:|
| MySQL-1-1-1 | MySQL-1-1-1 | 1 | 160 | 16384 | 1 | 1 | -1 | 300.00 | 0 | 7888.33 | 7755.95 | 0.00 | 43647.00 | 20271.00 |
| MySQL-1-1-2 | MySQL-1-1-2 | 1 | 160 | 16384 | 1 | 2 | -1 | 300.00 | 0 | 6961.19 | 6861.61 | 0.00 | 48427.00 | 22976.00 |
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 177.69 | 4.68 | 9.65 | 12.05 |
| MySQL-1-1-2-1 | 177.69 | 4.68 | 9.65 | 12.05 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 1893.35 | 15.73 | 0.28 | 0.28 |
| MySQL-1-1-2-1 | 1893.35 | 15.73 | 0.28 | 0.28 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 4765.20 | 16.92 | 11.46 | 16.00 |
| MySQL-1-1-2-1 | 4416.17 | 15.31 | 12.88 | 16.00 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 6447.31 | 24.49 | 1.14 | 1.14 |
| MySQL-1-1-2-1 | 6447.31 | 47.08 | 1.14 | 1.14 |
### Application Metrics
#### Loading phase: SUT deployment
| DBMS | InnoDB Buffer Pool Hit Ratio | Queries Per Second (QPS) | Connection Usage Ratio | Slow Queries Rate | InnoDB Log Waits Rate |
|:--------------|-------------------------------:|---------------------------:|-------------------------:|--------------------:|------------------------:|
| MySQL-1-1-1-1 | 1.00 | 447.12 | 0.01 | 0.00 | 0.00 |
| MySQL-1-1-2-1 | 1.00 | 447.12 | 0.01 | 0.00 | 0.00 |
#### Execution phase: SUT deployment
| DBMS | InnoDB Buffer Pool Hit Ratio | Queries Per Second (QPS) | Connection Usage Ratio | Slow Queries Rate | InnoDB Log Waits Rate |
|:--------------|-------------------------------:|---------------------------:|-------------------------:|--------------------:|------------------------:|
| MySQL-1-1-1-1 | 1.00 | 191138.68 | 0.11 | 0.00 | 0.00 |
| MySQL-1-1-2-1 | 0.00 | 170197.24 | 0.11 | 0.00 | 0.00 |
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Throughput (requests/second) contains no 0 or NaN
* TEST passed: Workflow as planned
The summary shows the first 5 application metrics aggregated per execution run. An extensive example for an evaluation is in the repository.
HammerDB’s TPC-C
Example:
bexhoma hammerdb \
-dbms MySQL \
-sf 16 \
-xsd 5 \
-nlt 16 \
-nbp 1,2 \
-nbt 16 \
-xlat \
-m \
-ma \
-mc \
-ms $BEXHOMA_MS \
-tr \
-lr 64Gi \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/docs_hammerdb_mysql_appmetrics.log
Show docs_hammerdb_mysql_appmetrics.log
## Show Summary
### Workload
HammerDB Workload SF=16 (warehouses for TPC-C)
* Type: tpcc
* Duration: 1491s
* Code: 1783883479
* HammerDB runs the benchmark.
* This experiment compares run time and resource consumption of TPC-C queries in different DBMS.
* TPC-C data is generated and loaded using several threads.
* Scaling factor (i.e., number of warehouses) is 16. Benchmarking runs for 5 minutes. Benchmarking also logs latencies.
* Experiment uses bexhoma version 0.10.5.
* System metrics are monitored by a cluster-wide installation.
* Application metrics are monitored by sidecar containers.
* Experiment is limited to DBMS ['MySQL'].
* Import is handled by 1 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker36.
* Database uses ephemeral storage of size 50Gi.
* Loading is tested with [16] threads, split into [1] pods.
* Benchmarking is tested with [16] threads, split into [1, 2] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* MySQL-1-1-1-1 uses docker image mysql:8.4.0
* RAM:2164173213696
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:1105753
* cpu_list:0-223
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--skip-log-bin', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:16Gi
* limits_memory:64Gi
* eval_parameters
* code:1783883479
* MySQL-1-1-2-1 uses docker image mysql:8.4.0
* RAM:2164173213696
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:1107715
* cpu_list:0-223
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--skip-log-bin', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:16Gi
* limits_memory:64Gi
* eval_parameters
* code:1783883479
### SUT Container Restarts
* bexhoma-sut-mysql-1-1783883479-b9f6c6886-7vmw4: 0 0
### Workflow
#### Actual
* DBMS MySQL-1 - Experiment 1 Client 1: hammerdb (1 pods)
* DBMS MySQL-1 - Experiment 1 Client 2: hammerdb (2 pods)
#### Planned
* DBMS MySQL-1 - Experiment 1 Client 1: hammerdb (1 pods)
* DBMS MySQL-1 - Experiment 1 Client 2: hammerdb (2 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:----------|-----------------:|------:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| MySQL-1-1 | 1 | 16.00 | 133.00 | 9.00 | 0.00 | 54.00 | 70.00 | 1 | 16 | | None | 0 | False | 433.08 |
### Execution
#### Per Connection
| DBMS | phase | job | experiment_run | vusers | client | benchmark_run | child | NOPM | TPM | efficiency | duration | errors | P95 [ms] | P99 [ms] |
|:----------------|:------------|:--------------|-----------------:|---------:|---------:|----------------:|--------:|-------:|-------:|-------------:|-----------:|---------:|-----------:|-----------:|
| MySQL-1-1-1-1-1 | MySQL-1-1-1 | MySQL-1-1-1-1 | 1 | 16 | 1 | 1 | 1 | 265908 | 616752 | 0.00 | 5 | 0 | 2.54 | 3.29 |
| MySQL-1-1-2-1-1 | MySQL-1-1-2 | MySQL-1-1-2-1 | 1 | 8 | 2 | 1 | 1 | 268600 | 623860 | 0.00 | 5 | 0 | 2.48 | 3.29 |
| MySQL-1-1-2-1-2 | MySQL-1-1-2 | MySQL-1-1-2-1 | 1 | 8 | 2 | 1 | 2 | 268709 | 624142 | 0.00 | 5 | 0 | 2.51 | 3.33 |
#### Per Phase
| DBMS | phase | experiment_run | vusers | client | benchmark_run | pod_count | P95 [ms] | P99 [ms] | efficiency | NOPM | TPM | duration | errors |
|:------------|:------------|-----------------:|---------:|---------:|----------------:|------------:|-----------:|-----------:|-------------:|----------:|----------:|-----------:|---------:|
| MySQL-1-1-1 | MySQL-1-1-1 | 1 | 16 | 1 | 1 | 1 | 2.54 | 3.28 | 0.00 | 265908.00 | 616752.00 | 5 | 0 |
| MySQL-1-1-2 | MySQL-1-1-2 | 1 | 16 | 2 | 1 | 2 | 2.51 | 3.33 | 0.00 | 268654.50 | 624001.00 | 5 | 0 |
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 96.64 | 2.03 | 22.23 | 24.14 |
| MySQL-1-1-2-1 | 96.64 | 2.03 | 22.23 | 24.14 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 396.60 | 10.87 | 0.17 | 0.17 |
| MySQL-1-1-2-1 | 396.60 | 10.87 | 0.17 | 0.17 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 4061.50 | 10.04 | 24.46 | 39.84 |
| MySQL-1-1-2-1 | 3831.44 | 9.65 | 26.61 | 54.88 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 1063.80 | 2.67 | 1.83 | 1.83 |
| MySQL-1-1-2-1 | 1063.80 | 4.78 | 1.83 | 1.83 |
### Application Metrics
#### Loading phase: SUT deployment
| DBMS | InnoDB Buffer Pool Hit Ratio | Queries Per Second (QPS) | Connection Usage Ratio | Slow Queries Rate | InnoDB Log Waits Rate |
|:--------------|-------------------------------:|---------------------------:|-------------------------:|--------------------:|------------------------:|
| MySQL-1-1-1-1 | 1.00 | 485.37 | 0.01 | 0.00 | 0.00 |
| MySQL-1-1-2-1 | 1.00 | 485.37 | 0.01 | 0.00 | 0.00 |
#### Execution phase: SUT deployment
| DBMS | InnoDB Buffer Pool Hit Ratio | Queries Per Second (QPS) | Connection Usage Ratio | Slow Queries Rate | InnoDB Log Waits Rate |
|:--------------|-------------------------------:|---------------------------:|-------------------------:|--------------------:|------------------------:|
| MySQL-1-1-1-1 | 1.00 | 1497434.76 | 0.01 | 0.00 | 0.00 |
| MySQL-1-1-2-1 | 0.00 | 1480331.98 | 0.01 | 0.00 | 0.00 |
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: NOPM contains no 0 or NaN
* TEST passed: Workflow as planned
TPC-H
Example:
bexhoma tpch \
-dbms MySQL \
-sf 3 \
-nlp 2 \
-nlt 8 \
-xii -xic -xis \
-xdt \
-m \
-ma \
-mc \
-ms $BEXHOMA_MS \
-t 1200 \
-tr \
-lr 64Gi \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/docs_tpch_mysql_appmetrics.log
Show docs_tpch_mysql_appmetrics.log
## Show Summary
### Workload
TPC-H Queries SF=3
* Type: tpch
* Duration: 816s
* Code: 1783878877
* This includes the reading queries of TPC-H.
* This experiment compares run time and resource consumption of TPC-H queries in different DBMS.
* TPC-H (SF=3) data is loaded and benchmark is executed.
* Query ordering is Q1 - Q22.
* All instances use the same query parameters.
* Timeout per query is 1200.
* Import sets indexes and constraints after loading and recomputes statistics.
* Experiment uses bexhoma version 0.10.5.
* System metrics are monitored by a cluster-wide installation.
* Application metrics are monitored by sidecar containers.
* Experiment is limited to DBMS ['MySQL'].
* Import is handled by 2 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker36.
* Database uses ephemeral storage of size 50Gi.
* Loading is tested with [8] threads, split into [2] pods.
* Benchmarking is tested with [1] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* MySQL-1-1-1-1-1 uses docker image mysql:8.4.0
* RAM:2164173213696
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:1109766
* cpu_list:0-223
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--skip-log-bin', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:16Gi
* limits_memory:64Gi
* eval_parameters
* code:1783878877
### SUT Container Restarts
* bexhoma-sut-mysql-1-1783878877-66475bdd65-gvlbz: 0 0
### Workflow
#### Actual
* DBMS MySQL-1 - Experiment 1 Client 1: tpch (1 pods)
#### Planned
* DBMS MySQL-1 - Experiment 1 Client 1: tpch (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:----------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| MySQL-1-1 | 1 | 3.00 | 406.00 | 1.00 | 0.00 | 118.00 | 284.00 | 2 | 0 | | None | 0 | False | 26.60 |
### Execution
#### Per Connection
| DBMS | configuration | phase | job | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id | pod |
|:----------------|:----------------|:------------|:--------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|:----------------|
| MySQL-1-1-1-1-1 | MySQL-1 | MySQL-1-1-1 | MySQL-1-1-1-1 | 1 | 1 | 1 | 1 | 3.00 | 22 | 144 | 2.83 | 3946.39 | 1650.00 | -1 | MySQL-1-1-1-1-1 |
#### Per Phase
| | phase | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id |
|:------------|:------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|
| MySQL-1-1-1 | MySQL-1-1-1 | 1 | 1 | 1 | 1 | 3.00 | 22 | 144 | 2.83 | 3946.39 | 1650.00 | -1 |
### Latency of Timer Execution [ms]
| Queries | MySQL-1-1-1-1-1 |
|:----------------------------------------------------|------------------:|
| Pricing Summary Report (TPC-H Q1) | 20123.99 |
| Minimum Cost Supplier Query (TPC-H Q2) | 241.46 |
| Shipping Priority (TPC-H Q3) | 3974.75 |
| Order Priority Checking Query (TPC-H Q4) | 1173.19 |
| Local Supplier Volume (TPC-H Q5) | 3356.62 |
| Forecasting Revenue Change (TPC-H Q6) | 3259.18 |
| Volume Shipping Query (TPC-H Q7) | 2650.20 |
| National Market Share (TPC-H Q8) | 7595.22 |
| Product Type Profit Measure (TPC-H Q9) | 5867.13 |
| Returned Item Reporting Query (TPC-H Q10) | 3739.28 |
| Important Stock Identification (TPC-H Q11) | 496.78 |
| Shipping Modes and Order Priority (TPC-H Q12) | 5578.34 |
| Customer Distribution (TPC-H Q13) | 16501.59 |
| Promotion Effect Query (TPC-H Q14) | 4310.56 |
| Top Supplier Query (TPC-H Q15) | 37620.28 |
| Parts/Supplier Relationship (TPC-H Q16) | 1061.65 |
| Small-Quantity-Order Revenue (TPC-H Q17) | 805.88 |
| Large Volume Customer (TPC-H Q18) | 5486.47 |
| Discounted Revenue (TPC-H Q19) | 356.80 |
| Potential Part Promotion (TPC-H Q20) | 670.84 |
| Suppliers Who Kept Orders Waiting Query (TPC-H Q21) | 12820.83 |
| Global Sales Opportunity Query (TPC-H Q22) | 355.63 |
### Errors (failed queries)
No errors
### Warnings (result mismatch)
No warnings
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 584.17 | 6.15 | 13.19 | 18.68 |
### Loading phase: component data generator
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 0.00 | 0.00 | 0.00 | 0.00 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 6.76 | 0.07 | 0.07 | 1.42 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 136.01 | 1.01 | 13.65 | 19.15 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 17.66 | 0.18 | 0.36 | 0.37 |
### Application Metrics
#### Loading phase: SUT deployment
| DBMS | InnoDB Buffer Pool Hit Ratio | Queries Per Second (QPS) | Connection Usage Ratio | Slow Queries Rate | InnoDB Log Waits Rate |
|:--------------|-------------------------------:|---------------------------:|-------------------------:|--------------------:|------------------------:|
| MySQL-1-1-1-1 | 1.00 | 0.88 | 0.00 | 0.02 | 0.00 |
#### Execution phase: SUT deployment
| DBMS | InnoDB Buffer Pool Hit Ratio | Queries Per Second (QPS) | Connection Usage Ratio | Slow Queries Rate | InnoDB Log Waits Rate |
|:--------------|-------------------------------:|---------------------------:|-------------------------:|--------------------:|------------------------:|
| MySQL-1-1-1-1 | 1.00 | 1.07 | 0.00 | 0.01 | 0.00 |
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST skipped: Loading phase: component data generator contains 0 or NaN in CPU [CPUs] (data pre-existing)
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Geo Times [s] contains no 0 or NaN
* TEST passed: Power@Size [~Q/h] contains no 0 or NaN
* TEST passed: Throughput@Size contains no 0 or NaN
* TEST passed: No SQL errors
* TEST passed: No SQL warnings
* TEST passed: Workflow as planned
TPC-DS
Example:
bexhoma tpcds \
-dbms MySQL \
-sf 3 \
-nlp 2 \
-nlt 8 \
-xii -xic -xis \
-xdt \
-m \
-ma \
-mc \
-ms $BEXHOMA_MS \
-t 1200 \
-tr \
-lr 64Gi \
-rr 64Gi \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/docs_tpcds_mysql_appmetrics.log
Show docs_tpcds_mysql_appmetrics.log
## Show Summary
### Workload
TPC-DS Queries SF=3
* Type: tpcds
* Duration: 3586s
* Code: 1783879748
* This includes the reading queries of TPC-DS.
* This experiment compares run time and resource consumption of TPC-DS queries in different DBMS.
* TPC-DS (SF=3) data is loaded and benchmark is executed.
* Query ordering is Q1 - Q99.
* All instances use the same query parameters.
* Timeout per query is 1200.
* Import sets indexes and constraints after loading and recomputes statistics.
* Experiment uses bexhoma version 0.10.5.
* System metrics are monitored by a cluster-wide installation.
* Application metrics are monitored by sidecar containers.
* Experiment is limited to DBMS ['MySQL'].
* Import is handled by 2 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker36.
* Database uses ephemeral storage of size 50Gi.
* Loading is tested with [8] threads, split into [2] pods.
* Benchmarking is tested with [1] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* MySQL-1-1-1-1-1 uses docker image mysql:8.4.0
* RAM:2164173213696
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:1116355
* cpu_list:0-223
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--skip-log-bin', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* eval_parameters
* code:1783879748
### SUT Container Restarts
* bexhoma-sut-mysql-1-1783879748-85cd8f4cc4-6d8x6: 0 0
### Workflow
#### Actual
* DBMS MySQL-1 - Experiment 1 Client 1: tpcds (1 pods)
#### Planned
* DBMS MySQL-1 - Experiment 1 Client 1: tpcds (1 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:----------|-----------------:|-----:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| MySQL-1-1 | 1 | 3.00 | 1852.00 | 0.00 | 0.00 | 227.00 | 1622.00 | 2 | 0 | | None | 0 | False | 5.83 |
### Execution
#### Per Connection
| DBMS | configuration | phase | job | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id | pod |
|:----------------|:----------------|:------------|:--------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|:----------------|
| MySQL-1-1-1-1-1 | MySQL-1 | MySQL-1-1-1 | MySQL-1-1-1-1 | 1 | 1 | 1 | 1 | 3.00 | 99 | 1524 | 2.47 | 4404.45 | 701.57 | -1 | MySQL-1-1-1-1-1 |
#### Per Phase
| | phase | experiment_run | client | benchmark_run | pod_count | SF | num_of_queries | time [s] | Geo Times [s] | Power@Size [~Q/h] | Throughput@Size | tenant_id |
|:------------|:------------|-----------------:|---------:|----------------:|------------:|-----:|-----------------:|-----------:|----------------:|--------------------:|------------------:|------------:|
| MySQL-1-1-1 | MySQL-1-1-1 | 1 | 1 | 1 | 1 | 3.00 | 99 | 1524 | 2.47 | 4404.45 | 701.57 | -1 |
### Latency of Timer Execution [ms]
| Queries | MySQL-1-1-1-1-1 |
|:--------------|------------------:|
| TPC-DS Q1 | 78.40 |
| TPC-DS Q2 | 13692.97 |
| TPC-DS Q3 | 23.98 |
| TPC-DS Q4 | 115304.79 |
| TPC-DS Q5 | 30681.82 |
| TPC-DS Q6 | 271783.80 |
| TPC-DS Q7 | 1325.94 |
| TPC-DS Q8 | 862.64 |
| TPC-DS Q9 | 11744.90 |
| TPC-DS Q10 | 913.63 |
| TPC-DS Q11 | 72910.96 |
| TPC-DS Q12 | 835.39 |
| TPC-DS Q13 | 3785.18 |
| TPC-DS Q14a+b | 119243.75 |
| TPC-DS Q15 | 537.02 |
| TPC-DS Q16 | 372.68 |
| TPC-DS Q17 | 1271.80 |
| TPC-DS Q18 | 1420.20 |
| TPC-DS Q19 | 868.80 |
| TPC-DS Q20 | 1544.70 |
| TPC-DS Q21 | 68402.65 |
| TPC-DS Q22 | 10988.92 |
| TPC-DS Q23a+b | 154354.14 |
| TPC-DS Q24a+b | 3005.38 |
| TPC-DS Q25 | 424.24 |
| TPC-DS Q26 | 1625.22 |
| TPC-DS Q27 | 952.77 |
| TPC-DS Q28 | 9833.14 |
| TPC-DS Q29 | 428.40 |
| TPC-DS Q30 | 4206.57 |
| TPC-DS Q31 | 33402.23 |
| TPC-DS Q32 | 991.75 |
| TPC-DS Q33 | 693.03 |
| TPC-DS Q34 | 1674.20 |
| TPC-DS Q35 | 7120.16 |
| TPC-DS Q36 | 3884.16 |
| TPC-DS Q37 | 20.33 |
| TPC-DS Q38 | 24734.98 |
| TPC-DS Q39a+b | 5254.09 |
| TPC-DS Q40 | 462.91 |
| TPC-DS Q41 | 5554.08 |
| TPC-DS Q42 | 1224.59 |
| TPC-DS Q43 | 2.63 |
| TPC-DS Q44 | 2.62 |
| TPC-DS Q45 | 572.31 |
| TPC-DS Q46 | 3907.38 |
| TPC-DS Q47 | 12765.31 |
| TPC-DS Q48 | 3671.38 |
| TPC-DS Q49 | 4306.05 |
| TPC-DS Q50 | 84.74 |
| TPC-DS Q51 | 17169.02 |
| TPC-DS Q52 | 801.73 |
| TPC-DS Q53 | 927.97 |
| TPC-DS Q54 | 7409.39 |
| TPC-DS Q55 | 997.23 |
| TPC-DS Q56 | 655.08 |
| TPC-DS Q57 | 9084.60 |
| TPC-DS Q58 | 19703.14 |
| TPC-DS Q59 | 20165.00 |
| TPC-DS Q60 | 1557.74 |
| TPC-DS Q61 | 2088.12 |
| TPC-DS Q62 | 8449.80 |
| TPC-DS Q63 | 918.79 |
| TPC-DS Q64 | 1170.43 |
| TPC-DS Q65 | 22724.26 |
| TPC-DS Q66 | 6335.36 |
| TPC-DS Q67 | 26202.02 |
| TPC-DS Q68 | 845.31 |
| TPC-DS Q69 | 2.44 |
| TPC-DS Q70 | 37176.09 |
| TPC-DS Q71 | 1336.15 |
| TPC-DS Q72 | 36807.20 |
| TPC-DS Q73 | 840.35 |
| TPC-DS Q74 | 15149.67 |
| TPC-DS Q75 | 6154.93 |
| TPC-DS Q76 | 1225.58 |
| TPC-DS Q77 | 28219.03 |
| TPC-DS Q78 | 33955.47 |
| TPC-DS Q79 | 2744.97 |
| TPC-DS Q80 | 23906.21 |
| TPC-DS Q81 | 20819.57 |
| TPC-DS Q82 | 22.19 |
| TPC-DS Q83 | 1962.47 |
| TPC-DS Q84 | 98.34 |
| TPC-DS Q85 | 268.41 |
| TPC-DS Q86 | 3018.81 |
| TPC-DS Q87 | 24237.59 |
| TPC-DS Q88 | 32229.34 |
| TPC-DS Q89 | 5739.63 |
| TPC-DS Q90 | 993.20 |
| TPC-DS Q91 | 50.74 |
| TPC-DS Q92 | 314.65 |
| TPC-DS Q93 | 113.53 |
| TPC-DS Q94 | 1272.14 |
| TPC-DS Q95 | 11390.68 |
| TPC-DS Q96 | 2780.83 |
| TPC-DS Q97 | 16793.11 |
| TPC-DS Q98 | 2863.12 |
| TPC-DS Q99 | 14961.96 |
### Errors (failed queries)
No errors
### Warnings (result mismatch)
No warnings
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 2936.43 | 6.92 | 27.45 | 52.96 |
### Loading phase: component data generator
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 0.00 | 0.00 | 0.00 | 0.00 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 7.18 | 0.12 | 0.01 | 2.88 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 1480.38 | 1.02 | 31.80 | 57.33 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 27.15 | 0.05 | 0.43 | 0.44 |
### Application Metrics
#### Loading phase: SUT deployment
| DBMS | InnoDB Buffer Pool Hit Ratio | Queries Per Second (QPS) | Connection Usage Ratio | Slow Queries Rate | InnoDB Log Waits Rate |
|:--------------|-------------------------------:|---------------------------:|-------------------------:|--------------------:|------------------------:|
| MySQL-1-1-1-1 | 1.00 | 1.01 | 0.00 | 0.03 | 0.00 |
#### Execution phase: SUT deployment
| DBMS | InnoDB Buffer Pool Hit Ratio | Queries Per Second (QPS) | Connection Usage Ratio | Slow Queries Rate | InnoDB Log Waits Rate |
|:--------------|-------------------------------:|---------------------------:|-------------------------:|--------------------:|------------------------:|
| MySQL-1-1-1-1 | 1.00 | 0.82 | 0.00 | 0.03 | 0.00 |
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST skipped: Loading phase: component data generator contains 0 or NaN in CPU [CPUs] (data pre-existing)
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Geo Times [s] contains no 0 or NaN
* TEST passed: Power@Size [~Q/h] contains no 0 or NaN
* TEST passed: Throughput@Size contains no 0 or NaN
* TEST passed: No SQL errors
* TEST passed: No SQL warnings
* TEST passed: Workflow as planned
YCSB
Example:
bexhoma ycsb \
-dbms MySQL \
-sf 3 \
-xwl a \
-xtb 16384 \
-xnbf 2,3 \
-xnlf 4 \
-nc 1 \
-ne 1 \
-nlp 8 \
-nlt 64 \
-nbp 1,8 \
-nbt 64 \
-m \
-ma \
-mc \
-ms $BEXHOMA_MS \
-tr \
-lr 64Gi \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/docs_ycsb_mysql_appmetrics.log
Show docs_ycsb_mysql_appmetrics.log
## Show Summary
### Workload
YCSB SF=3
* Type: ycsb
* Duration: 1198s
* Code: 1783877650
* YCSB driver runs the experiment.
* This experiment compares run time and resource consumption of YCSB queries.
* Workload is 'A'.
* Number of rows to insert is 3000000.
* Ordering of inserts is hashed.
* Number of operations is 3000000.
* Batch size is ''.
* Target is based on multiples of '16384'.
* Factors for loading are [4].
* Factors for benchmarking are [2, 3].
* Experiment uses bexhoma version 0.10.5.
* System metrics are monitored by a cluster-wide installation.
* Application metrics are monitored by sidecar containers.
* Experiment is limited to DBMS ['MySQL'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker36.
* Database uses ephemeral storage of size 50Gi.
* Loading is tested with [64] threads, split into [8] pods.
* Benchmarking is tested with [64] threads, split into [1, 8] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* MySQL-1-1-1-1 uses docker image mysql:8.4.0
* RAM:2164173213696
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:1109714
* cpu_list:0-223
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--skip-log-bin', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:16Gi
* limits_memory:64Gi
* eval_parameters
* code:1783877650
* MySQL-1-1-2-1 uses docker image mysql:8.4.0
* RAM:2164173213696
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:1109715
* cpu_list:0-223
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--skip-log-bin', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:16Gi
* limits_memory:64Gi
* eval_parameters
* code:1783877650
* MySQL-1-1-3-1 uses docker image mysql:8.4.0
* RAM:2164173213696
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:1109716
* cpu_list:0-223
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--skip-log-bin', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:16Gi
* limits_memory:64Gi
* eval_parameters
* code:1783877650
* MySQL-1-1-4-1 uses docker image mysql:8.4.0
* RAM:2164173213696
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:1109716
* cpu_list:0-223
* args:['--max_connections=1500', '--local-infile=1', '--mysql-native-password=ON', '--innodb-redo-log-capacity=32GB', '--innodb-io-capacity=400', '--innodb-io-capacity_max=2000', '--innodb-read-io-threads=8', '--innodb-write-io-threads=8', '--innodb-use-native-aio=0', '--innodb-buffer-pool-size=96G', '--innodb-buffer-pool-instances=16', '--innodb-buffer-pool-chunk-size=2G', '--innodb-flush-method=O_DIRECT', '--innodb-flush-neighbors=0', '--innodb-flush-log-at-trx-commit=2', '--skip-log-bin', '--innodb-change-buffer-max-size=50', '--innodb-doublewrite=0', '--tmpdir=/mysqltmp']
* requests_cpu:4
* requests_memory:16Gi
* limits_memory:64Gi
* eval_parameters
* code:1783877650
### SUT Container Restarts
* bexhoma-sut-mysql-1-1783877650-7c85cbbc77-xxwd8: 0 0
### Workflow
#### Actual
* DBMS MySQL-1 - Experiment 1 Client 1: ycsb (1 pods)
* DBMS MySQL-1 - Experiment 1 Client 2: ycsb (8 pods)
* DBMS MySQL-1 - Experiment 1 Client 3: ycsb (1 pods)
* DBMS MySQL-1 - Experiment 1 Client 4: ycsb (8 pods)
#### Planned
* DBMS MySQL-1 - Experiment 1 Client 1: ycsb (1 pods)
* DBMS MySQL-1 - Experiment 1 Client 2: ycsb (8 pods)
* DBMS MySQL-1 - Experiment 1 Client 3: ycsb (1 pods)
* DBMS MySQL-1 - Experiment 1 Client 4: ycsb (8 pods)
### Loading
#### Per Connection
| connection | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) | sf | Throughput [SF/h] |
|:----------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|-----:|--------------------:|
| MySQL-1-1-0-1-1 | 1.00 | 8.00 | 8192.00 | 8.00 | 0.00 | 7415.02 | 50573.00 | 375000.00 | 12535.00 | 3.00 | 213.55 |
| MySQL-1-1-0-1-2 | 1.00 | 8.00 | 8192.00 | 8.00 | 0.00 | 7366.37 | 50907.00 | 375000.00 | 12239.00 | 3.00 | 212.15 |
| MySQL-1-1-0-1-3 | 1.00 | 8.00 | 8192.00 | 8.00 | 0.00 | 7283.53 | 51486.00 | 375000.00 | 12383.00 | 3.00 | 209.77 |
| MySQL-1-1-0-1-4 | 1.00 | 8.00 | 8192.00 | 8.00 | 0.00 | 7449.05 | 50342.00 | 375000.00 | 12207.00 | 3.00 | 214.53 |
| MySQL-1-1-0-1-5 | 1.00 | 8.00 | 8192.00 | 8.00 | 0.00 | 7361.31 | 50942.00 | 375000.00 | 12375.00 | 3.00 | 212.01 |
| MySQL-1-1-0-1-6 | 1.00 | 8.00 | 8192.00 | 8.00 | 0.00 | 7460.31 | 50266.00 | 375000.00 | 12191.00 | 3.00 | 214.86 |
| MySQL-1-1-0-1-7 | 1.00 | 8.00 | 8192.00 | 8.00 | 0.00 | 7368.69 | 50891.00 | 375000.00 | 12279.00 | 3.00 | 212.22 |
| MySQL-1-1-0-1-8 | 1.00 | 8.00 | 8192.00 | 8.00 | 0.00 | 7334.10 | 51131.00 | 375000.00 | 12599.00 | 3.00 | 211.22 |
#### Per Run
| DBMS | experiment_run | threads | target | pod_count | exceptions | sf | Throughput [SF/h] | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) |
|:----------|-----------------:|----------:|---------:|------------:|-------------:|-----:|--------------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|
| MySQL-1-1 | 1.00 | 64.00 | 65536.00 | 8.00 | 0.00 | 3.00 | 209.77 | 59038.40 | 51486.00 | 3000000.00 | 12351.00 |
### Execution
#### Per Connection
| DBMS | phase | job | configuration | experiment_run | client | benchmark_run | child | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:----------------|:------------|:--------------|:----------------|-----------------:|---------:|----------------:|--------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| MySQL-1-1-1-1-1 | MySQL-1-1-1 | MySQL-1-1-1-1 | MySQL-1 | 1 | 1 | 1 | 1 | 64 | 32768 | 1 | 0 | 32157.44 | 93291.00 | 1500723 | 751.00 | 1499277 | 680.00 |
| MySQL-1-1-2-1-1 | MySQL-1-1-2 | MySQL-1-1-2-1 | MySQL-1 | 1 | 2 | 1 | 1 | 8 | 4096 | 8 | 0 | 4046.70 | 92668.00 | 187081 | 769.00 | 187919 | 680.00 |
| MySQL-1-1-2-1-2 | MySQL-1-1-2 | MySQL-1-1-2-1 | MySQL-1 | 1 | 2 | 1 | 2 | 8 | 4096 | 8 | 0 | 4051.38 | 92561.00 | 187061 | 895.00 | 187939 | 777.00 |
| MySQL-1-1-2-1-3 | MySQL-1-1-2 | MySQL-1-1-2-1 | MySQL-1 | 1 | 2 | 1 | 3 | 8 | 4096 | 8 | 0 | 4057.39 | 92424.00 | 187159 | 815.00 | 187841 | 702.00 |
| MySQL-1-1-2-1-4 | MySQL-1-1-2 | MySQL-1-1-2-1 | MySQL-1 | 1 | 2 | 1 | 4 | 8 | 4096 | 8 | 0 | 4047.71 | 92645.00 | 188096 | 760.00 | 186904 | 679.00 |
| MySQL-1-1-2-1-5 | MySQL-1-1-2 | MySQL-1-1-2-1 | MySQL-1 | 1 | 2 | 1 | 5 | 8 | 4096 | 8 | 0 | 4052.56 | 92534.00 | 187269 | 879.00 | 187731 | 794.00 |
| MySQL-1-1-2-1-6 | MySQL-1-1-2 | MySQL-1-1-2-1 | MySQL-1 | 1 | 2 | 1 | 6 | 8 | 4096 | 8 | 0 | 4047.49 | 92650.00 | 187430 | 757.00 | 187570 | 685.00 |
| MySQL-1-1-2-1-7 | MySQL-1-1-2 | MySQL-1-1-2-1 | MySQL-1 | 1 | 2 | 1 | 7 | 8 | 4096 | 8 | 0 | 4049.33 | 92608.00 | 187279 | 774.00 | 187721 | 678.00 |
| MySQL-1-1-2-1-8 | MySQL-1-1-2 | MySQL-1-1-2-1 | MySQL-1 | 1 | 2 | 1 | 8 | 8 | 4096 | 8 | 0 | 4054.58 | 92488.00 | 187392 | 826.00 | 187608 | 712.00 |
| MySQL-1-1-3-1-1 | MySQL-1-1-3 | MySQL-1-1-3-1 | MySQL-1 | 1 | 3 | 1 | 1 | 64 | 49152 | 1 | 0 | 47647.79 | 62962.00 | 1500175 | 1045.00 | 1499825 | 818.00 |
| MySQL-1-1-4-1-1 | MySQL-1-1-4 | MySQL-1-1-4-1 | MySQL-1 | 1 | 4 | 1 | 1 | 8 | 6144 | 8 | 0 | 6032.92 | 62159.00 | 187570 | 849.00 | 187430 | 739.00 |
| MySQL-1-1-4-1-2 | MySQL-1-1-4 | MySQL-1-1-4-1 | MySQL-1 | 1 | 4 | 1 | 2 | 8 | 6144 | 8 | 0 | 6048.87 | 61995.00 | 187591 | 1033.00 | 187409 | 853.00 |
| MySQL-1-1-4-1-3 | MySQL-1-1-4 | MySQL-1-1-4-1 | MySQL-1 | 1 | 4 | 1 | 3 | 8 | 6144 | 8 | 0 | 6058.75 | 61894.00 | 187010 | 1128.00 | 187990 | 960.00 |
| MySQL-1-1-4-1-4 | MySQL-1-1-4 | MySQL-1-1-4-1 | MySQL-1 | 1 | 4 | 1 | 4 | 8 | 6144 | 8 | 0 | 6041.96 | 62066.00 | 188050 | 866.00 | 186950 | 744.00 |
| MySQL-1-1-4-1-5 | MySQL-1-1-4 | MySQL-1-1-4-1 | MySQL-1 | 1 | 4 | 1 | 5 | 8 | 6144 | 8 | 0 | 6054.34 | 61939.00 | 187559 | 1057.00 | 187441 | 852.00 |
| MySQL-1-1-4-1-6 | MySQL-1-1-4 | MySQL-1-1-4-1 | MySQL-1 | 1 | 4 | 1 | 6 | 8 | 6144 | 8 | 0 | 6032.04 | 62168.00 | 187159 | 830.00 | 187841 | 706.00 |
| MySQL-1-1-4-1-7 | MySQL-1-1-4 | MySQL-1-1-4-1 | MySQL-1 | 1 | 4 | 1 | 7 | 8 | 6144 | 8 | 0 | 6040.69 | 62079.00 | 187653 | 1001.00 | 187347 | 791.00 |
| MySQL-1-1-4-1-8 | MySQL-1-1-4 | MySQL-1-1-4-1 | MySQL-1 | 1 | 4 | 1 | 8 | 8 | 6144 | 8 | 0 | 6051.90 | 61964.00 | 187073 | 1048.00 | 187927 | 878.00 |
#### Per Phase
| DBMS | phase | experiment_run | threads | target | benchmark_run | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:------------|:------------|-----------------:|----------:|---------:|----------------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| MySQL-1-1-1 | MySQL-1-1-1 | 1 | 64 | 32768 | 1 | 1 | 0 | 32157.44 | 93291.00 | 1500723 | 751.00 | 1499277 | 680.00 |
| MySQL-1-1-2 | MySQL-1-1-2 | 1 | 64 | 32768 | 1 | 8 | 0 | 32407.14 | 92668.00 | 1498767 | 895.00 | 1501233 | 794.00 |
| MySQL-1-1-3 | MySQL-1-1-3 | 1 | 64 | 49152 | 1 | 1 | 0 | 47647.79 | 62962.00 | 1500175 | 1045.00 | 1499825 | 818.00 |
| MySQL-1-1-4 | MySQL-1-1-4 | 1 | 64 | 49152 | 1 | 8 | 0 | 48361.47 | 62168.00 | 1499665 | 1128.00 | 1500335 | 960.00 |
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 611.00 | 12.04 | 25.52 | 31.34 |
| MySQL-1-1-2-1 | 611.00 | 12.04 | 25.52 | 31.34 |
| MySQL-1-1-3-1 | 611.00 | 12.04 | 25.52 | 31.34 |
| MySQL-1-1-4-1 | 611.00 | 12.04 | 25.52 | 31.34 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 319.53 | 11.06 | 0.13 | 0.14 |
| MySQL-1-1-2-1 | 319.53 | 11.06 | 0.13 | 0.14 |
| MySQL-1-1-3-1 | 319.53 | 11.06 | 0.13 | 0.14 |
| MySQL-1-1-4-1 | 319.53 | 11.06 | 0.13 | 0.14 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 324.86 | 3.89 | 25.57 | 32.08 |
| MySQL-1-1-2-1 | 284.84 | 3.90 | 25.60 | 32.78 |
| MySQL-1-1-3-1 | 148.34 | 5.39 | 25.63 | 33.29 |
| MySQL-1-1-4-1 | 259.83 | 5.14 | 25.67 | 34.38 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| MySQL-1-1-1-1 | 371.22 | 4.05 | 0.16 | 0.16 |
| MySQL-1-1-2-1 | 392.49 | 8.94 | 0.16 | 0.16 |
| MySQL-1-1-3-1 | 427.58 | 11.16 | 0.16 | 0.16 |
| MySQL-1-1-4-1 | 238.27 | 14.50 | 0.16 | 0.16 |
### Application Metrics
#### Loading phase: SUT deployment
| DBMS | InnoDB Buffer Pool Hit Ratio | Queries Per Second (QPS) | Connection Usage Ratio | Slow Queries Rate | InnoDB Log Waits Rate |
|:--------------|-------------------------------:|---------------------------:|-------------------------:|--------------------:|------------------------:|
| MySQL-1-1-1-1 | 0.00 | 14306.98 | 0.04 | 0.00 | 0.00 |
| MySQL-1-1-2-1 | 0.00 | 14306.98 | 0.04 | 0.00 | 0.00 |
| MySQL-1-1-3-1 | 0.00 | 14306.98 | 0.04 | 0.00 | 0.00 |
| MySQL-1-1-4-1 | 0.00 | 14306.98 | 0.04 | 0.00 | 0.00 |
#### Execution phase: SUT deployment
| DBMS | InnoDB Buffer Pool Hit Ratio | Queries Per Second (QPS) | Connection Usage Ratio | Slow Queries Rate | InnoDB Log Waits Rate |
|:--------------|-------------------------------:|---------------------------:|-------------------------:|--------------------:|------------------------:|
| MySQL-1-1-1-1 | 1.00 | 18726.11 | 0.04 | 0.00 | 0.00 |
| MySQL-1-1-2-1 | 1.00 | 20525.17 | 0.04 | 0.00 | 0.00 |
| MySQL-1-1-3-1 | 0.00 | 21969.72 | 0.04 | 0.00 | 0.00 |
| MySQL-1-1-4-1 | 0.00 | 20976.62 | 0.04 | 0.00 | 0.00 |
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Execution Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: contains no FAILED column
CockroachDB
YCSB
Example:
bexhoma ycsb \
-dbms CockroachDB \
-sf 10 \
-xwl a \
-xtb 16384 \
-xnbf 4 \
-xnlf 4 \
-nc 1 \
-ne 1 \
-nlp 8 \
-nlt 64 \
-nbp 1 \
-nbt 64 \
-nw 3 \
-nwr 3 \
-xop 10 \
-m \
-ma \
-mc \
-ms $BEXHOMA_MS \
-tr \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/docs_ycsb_cockroachdb_appmetrics.log
Show docs_ycsb_cockroachdb_appmetrics.log
## Show Summary
### Workload
YCSB SF=10
* Type: ycsb
* Duration: 5521s
* Code: 1783884999
* YCSB driver runs the experiment.
* This experiment compares run time and resource consumption of YCSB queries.
* Workload is 'A'.
* Number of rows to insert is 10000000.
* Ordering of inserts is hashed.
* Number of operations is 10000000.
* Batch size is ''.
* Target is based on multiples of '16384'.
* Factors for loading are [4].
* Factors for benchmarking are [4].
* Experiment uses bexhoma version 0.10.5.
* System metrics are monitored by a cluster-wide installation.
* Application metrics are monitored by sidecar containers.
* Experiment is limited to DBMS ['CockroachDB'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker36.
* Database uses ephemeral storage of size 50Gi.
* Loading is tested with [64] threads, split into [8] pods.
* Benchmarking is tested with [64] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* CockroachDB-1-1-1-1 uses docker image cockroachdb/cockroach:v24.2.4
* RAM:2164173213696
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:1071339
* cpu_list:0-223
* args:['-c', 'while true; do echo hello; sleep 10;done']
* requests_cpu:4
* requests_memory:16Gi
* worker 0
* RAM:1077382598656
* Cores:256
* host:6.8.0-1058-nvidia
* node:cl-worker28
* disk:589347
* volume_size:1000G
* volume_used:738G
* cpu_list:0-255
* worker 1
* RAM:1077381287936
* Cores:256
* host:6.8.0-111-generic
* node:cl-worker27
* disk:1383265
* volume_size:1000G
* volume_used:738G
* cpu_list:0-255
* worker 2
* RAM:540590841856
* Cores:96
* host:6.8.0-117-generic
* node:cl-worker25
* disk:171816
* volume_size:1000G
* volume_used:738G
* cpu_list:0-95
* eval_parameters
* code:1783884999
* BEXHOMA_REPLICAS:3
* BEXHOMA_WORKERS:3
### SUT Container Restarts
* bexhoma-sut-cockroachdb-1-1783884999-67f9c75bff-v26xj: 0
* bexhoma-worker-cockroachdb-ycsb-10-0: 0
* bexhoma-worker-cockroachdb-ycsb-10-1: 0
* bexhoma-worker-cockroachdb-ycsb-10-2: 0
### Workflow
#### Actual
* DBMS CockroachDB-1 - Experiment 1 Client 1: ycsb (1 pods)
#### Planned
* DBMS CockroachDB-1 - Experiment 1 Client 1: ycsb (1 pods)
### Loading
#### Per Connection
| connection | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) | sf | Throughput [SF/h] |
|:----------------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|------:|--------------------:|
| CockroachDB-1-1-0-1-1 | 1.00 | 8.00 | 8192.00 | 8.00 | 0.00 | 514.06 | 2431610.00 | 1250000.00 | 171519.00 | 10.00 | 14.81 |
| CockroachDB-1-1-0-1-2 | 1.00 | 8.00 | 8192.00 | 8.00 | 0.00 | 514.55 | 2429292.00 | 1250000.00 | 171135.00 | 10.00 | 14.82 |
| CockroachDB-1-1-0-1-3 | 1.00 | 8.00 | 8192.00 | 8.00 | 0.00 | 514.58 | 2429176.00 | 1250000.00 | 170367.00 | 10.00 | 14.82 |
| CockroachDB-1-1-0-1-4 | 1.00 | 8.00 | 8192.00 | 8.00 | 0.00 | 514.92 | 2427584.00 | 1250000.00 | 170239.00 | 10.00 | 14.83 |
| CockroachDB-1-1-0-1-5 | 1.00 | 8.00 | 8192.00 | 8.00 | 0.00 | 514.95 | 2427403.00 | 1250000.00 | 171391.00 | 10.00 | 14.83 |
| CockroachDB-1-1-0-1-6 | 1.00 | 8.00 | 8192.00 | 8.00 | 0.00 | 513.98 | 2431992.00 | 1250000.00 | 172415.00 | 10.00 | 14.80 |
| CockroachDB-1-1-0-1-7 | 1.00 | 8.00 | 8192.00 | 8.00 | 0.00 | 515.42 | 2425227.00 | 1250000.00 | 171391.00 | 10.00 | 14.84 |
| CockroachDB-1-1-0-1-8 | 1.00 | 8.00 | 8192.00 | 8.00 | 0.00 | 515.16 | 2426435.00 | 1250000.00 | 171391.00 | 10.00 | 14.84 |
#### Per Run
| DBMS | experiment_run | threads | target | pod_count | exceptions | sf | Throughput [SF/h] | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) |
|:----------------|-----------------:|----------:|---------:|------------:|-------------:|------:|--------------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|
| CockroachDB-1-1 | 1.00 | 64.00 | 65536.00 | 8.00 | 0.00 | 10.00 | 14.80 | 4117.62 | 2431992.00 | 10000000.00 | 171231.00 |
### Execution
#### Per Connection
| DBMS | phase | job | configuration | experiment_run | client | benchmark_run | child | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:----------------------|:------------------|:--------------------|:----------------|-----------------:|---------:|----------------:|--------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| CockroachDB-1-1-1-1-1 | CockroachDB-1-1-1 | CockroachDB-1-1-1-1 | CockroachDB-1 | 1 | 1 | 1 | 1 | 64 | 65536 | 1 | 0 | 3748.55 | 2667700.00 | 5002335 | 17183.00 | 4997665 | 425215.00 |
#### Per Phase
| DBMS | phase | experiment_run | threads | target | benchmark_run | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:------------------|:------------------|-----------------:|----------:|---------:|----------------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| CockroachDB-1-1-1 | CockroachDB-1-1-1 | 1 | 64 | 65536 | 1 | 1 | 0 | 3748.55 | 2667700.00 | 5002335 | 17183.00 | 4997665 | 425215.00 |
### Monitoring
### Loading phase: component worker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------------|-------------:|----------:|---------------:|----------------------:|
| CockroachDB-1-1-1-1 | 23002.85 | 21.07 | 20.00 | 47.94 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------------|-------------:|----------:|---------------:|----------------------:|
| CockroachDB-1-1-1-1 | 965.84 | 1.04 | 0.11 | 0.11 |
### Execution phase: component worker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------------|-------------:|----------:|---------------:|----------------------:|
| CockroachDB-1-1-1-1 | 26604.38 | 22.59 | 24.48 | 47.93 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------------|-------------:|----------:|---------------:|----------------------:|
| CockroachDB-1-1-1-1 | 744.34 | 0.66 | 0.14 | 0.14 |
### Application Metrics
#### Loading phase: component worker
| DBMS | Raft Messages Received (AppResp) [msgs/s] | Raft Network In (Bytes/sec) | Raft Recovery Snapshot In (Bytes/sec) | Replicate Queue Adds Attempted [adds/s] | Replicate Queue Purgatory Count |
|:--------------------|--------------------------------------------:|------------------------------:|----------------------------------------:|------------------------------------------:|----------------------------------:|
| CockroachDB-1-1-1-1 | 16927.34 | 17490323.37 | 15306729.59 | 0.00 | 0.00 |
#### Execution phase: component worker
| DBMS | Raft Messages Received (AppResp) [msgs/s] | Raft Network In (Bytes/sec) | Raft Recovery Snapshot In (Bytes/sec) | Replicate Queue Adds Attempted [adds/s] | Replicate Queue Purgatory Count |
|:--------------------|--------------------------------------------:|------------------------------:|----------------------------------------:|------------------------------------------:|----------------------------------:|
| CockroachDB-1-1-1-1 | 8156.09 | 5519962.47 | 0.00 | 0.00 | 0.00 |
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Loading phase: component worker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component worker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Execution Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: contains no FAILED column
Benchbase’s TPC-C
Example:
bexhoma benchbase \
-dbms CockroachDB \
-sf 16 \
-xsd 5 \
-xtb 1024 \
-xnbf 16 \
-nbp 1,2 \
-nbt 16 \
-nw 3 \
-nwr 3 \
-m \
-ma \
-mc \
-ms $BEXHOMA_MS \
-tr \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/docs_benchbase_cockroachdb_appmetrics.log
Show docs_benchbase_cockroachdb_appmetrics.log
## Show Summary
### Workload
Benchbase Workload tpcc SF=16
* Type: benchbase
* Duration: 1574s
* Code: 1783890550
* Benchbase runs a TPC-C experiment.
* This experiment compares run time and resource consumption of Benchbase queries in different DBMS.
* Benchbase data is generated and loaded using several threads.
* Benchmark is 'tpcc'. Scaling factor is 16. Target is based on multiples of '1024'. Factors for benchmarking are [16]. Benchmarking runs for 5 minutes.
* Experiment uses bexhoma version 0.10.5.
* System metrics are monitored by a cluster-wide installation.
* Application metrics are monitored by sidecar containers.
* Experiment is limited to DBMS ['CockroachDB'].
* Import is handled by 1 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker36.
* Database uses ephemeral storage of size 50Gi.
* Loading is tested with [1] threads, split into [1] pods.
* Benchmarking is tested with [16] threads, split into [1, 2] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* CockroachDB-1-1-1-1 uses docker image cockroachdb/cockroach:v24.2.4
* RAM:2164173213696
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:1086935
* cpu_list:0-223
* args:['-c', 'while true; do echo hello; sleep 10;done']
* requests_cpu:4
* requests_memory:16Gi
* worker 0
* RAM:1077382598656
* Cores:256
* host:6.8.0-1058-nvidia
* node:cl-worker28
* disk:576525
* volume_size:1000G
* volume_used:738G
* cpu_list:0-255
* worker 1
* RAM:540590841856
* Cores:96
* host:6.8.0-117-generic
* node:cl-worker25
* disk:162734
* volume_size:1000G
* volume_used:738G
* cpu_list:0-95
* worker 2
* RAM:540590804992
* Cores:96
* host:6.8.0-124-generic
* node:cl-worker24
* disk:135913
* volume_size:1000G
* volume_used:738G
* cpu_list:0-95
* eval_parameters
* code:1783890550
* BEXHOMA_REPLICAS:3
* BEXHOMA_WORKERS:3
* CockroachDB-1-1-2-1 uses docker image cockroachdb/cockroach:v24.2.4
* RAM:2164173213696
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:1086937
* cpu_list:0-223
* args:['-c', 'while true; do echo hello; sleep 10;done']
* requests_cpu:4
* requests_memory:16Gi
* worker 0
* RAM:1077382598656
* Cores:256
* host:6.8.0-1058-nvidia
* node:cl-worker28
* disk:576555
* volume_size:1000G
* volume_used:738G
* cpu_list:0-255
* worker 1
* RAM:540590841856
* Cores:96
* host:6.8.0-117-generic
* node:cl-worker25
* disk:162826
* volume_size:1000G
* volume_used:738G
* cpu_list:0-95
* worker 2
* RAM:540590804992
* Cores:96
* host:6.8.0-124-generic
* node:cl-worker24
* disk:136006
* volume_size:1000G
* volume_used:738G
* cpu_list:0-95
* eval_parameters
* code:1783890550
* BEXHOMA_REPLICAS:3
* BEXHOMA_WORKERS:3
### SUT Container Restarts
* bexhoma-sut-cockroachdb-1-1783890550-67c46fcb89-98x2f: 0
* bexhoma-worker-cockroachdb-benchbase-tpcc-16-0: 0
* bexhoma-worker-cockroachdb-benchbase-tpcc-16-1: 0
* bexhoma-worker-cockroachdb-benchbase-tpcc-16-2: 0
### Workflow
#### Actual
* DBMS CockroachDB-1 - Experiment 1 Client 1: benchbase (1 pods)
* DBMS CockroachDB-1 - Experiment 1 Client 2: benchbase (2 pods)
#### Planned
* DBMS CockroachDB-1 - Experiment 1 Client 1: benchbase (1 pods)
* DBMS CockroachDB-1 - Experiment 1 Client 2: benchbase (2 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:----------------|-----------------:|------:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| CockroachDB-1-1 | 1 | 16.00 | 1043.00 | 1.00 | 0.00 | 513.00 | 529.00 | 1 | 1 | | None | 0 | False | 55.23 |
### Execution
#### Per Connection
| DBMS | phase | job | experiment_run | terminals | target | client | benchmark_run | child | tenant_id | time | num_errors | Throughput (requests/second) | Goodput (requests/second) | efficiency | Latency Distribution.95th Percentile Latency (microseconds) | Latency Distribution.Average Latency (microseconds) |
|:----------------------|:------------------|:--------------------|-----------------:|------------:|---------:|---------:|----------------:|--------:|------------:|-------:|-------------:|-------------------------------:|----------------------------:|-------------:|--------------------------------------------------------------:|------------------------------------------------------:|
| CockroachDB-1-1-1-1-1 | CockroachDB-1-1-1 | CockroachDB-1-1-1-1 | 1 | 16 | 16384 | 1 | 1 | 1 | -1 | 300.00 | 0 | 168.19 | 167.39 | 0.00 | 181754.00 | 95115.00 |
| CockroachDB-1-1-2-1-1 | CockroachDB-1-1-2 | CockroachDB-1-1-2-1 | 1 | 8 | 8192 | 2 | 1 | 1 | -1 | 300.00 | 0 | 65.08 | 64.81 | 0.00 | 226916.00 | 122892.00 |
| CockroachDB-1-1-2-1-2 | CockroachDB-1-1-2 | CockroachDB-1-1-2-1 | 1 | 8 | 8192 | 2 | 1 | 2 | -1 | 300.00 | 0 | 63.54 | 63.26 | 0.00 | 236248.00 | 125877.00 |
#### Per Phase
| DBMS | phase | experiment_run | terminals | target | benchmark_run | pod_count | tenant_id | time | num_errors | Throughput (requests/second) | Goodput (requests/second) | efficiency | Latency Distribution.95th Percentile Latency (microseconds) | Latency Distribution.Average Latency (microseconds) |
|:------------------|:------------------|-----------------:|------------:|---------:|----------------:|------------:|------------:|-------:|-------------:|-------------------------------:|----------------------------:|-------------:|--------------------------------------------------------------:|------------------------------------------------------:|
| CockroachDB-1-1-1 | CockroachDB-1-1-1 | 1 | 16 | 16384 | 1 | 1 | -1 | 300.00 | 0 | 168.19 | 167.39 | 0.00 | 181754.00 | 95115.00 |
| CockroachDB-1-1-2 | CockroachDB-1-1-2 | 1 | 16 | 16384 | 1 | 2 | -1 | 300.00 | 0 | 128.62 | 128.07 | 0.00 | 236248.00 | 124384.50 |
### Monitoring
### Loading phase: component worker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------------|-------------:|----------:|---------------:|----------------------:|
| CockroachDB-1-1-1-1 | 2525.39 | 17.45 | 6.53 | 10.39 |
| CockroachDB-1-1-2-1 | 2525.39 | 17.45 | 6.53 | 10.39 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------------|-------------:|----------:|---------------:|----------------------:|
| CockroachDB-1-1-1-1 | 1430.79 | 10.09 | 0.26 | 0.26 |
| CockroachDB-1-1-2-1 | 1430.79 | 10.09 | 0.26 | 0.26 |
### Execution phase: component worker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------------|-------------:|----------:|---------------:|----------------------:|
| CockroachDB-1-1-1-1 | 2806.82 | 19.97 | 8.06 | 12.28 |
| CockroachDB-1-1-2-1 | 2062.42 | 17.93 | 7.93 | 12.23 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------------|-------------:|----------:|---------------:|----------------------:|
| CockroachDB-1-1-1-1 | 106.82 | 0.71 | 0.31 | 0.31 |
| CockroachDB-1-1-2-1 | 106.83 | 0.71 | 0.31 | 0.31 |
### Application Metrics
#### Loading phase: component worker
| DBMS | Raft Messages Received (AppResp) [msgs/s] | Raft Network In (Bytes/sec) | Raft Recovery Snapshot In (Bytes/sec) | Replicate Queue Adds Attempted [adds/s] | Replicate Queue Purgatory Count |
|:--------------------|--------------------------------------------:|------------------------------:|----------------------------------------:|------------------------------------------:|----------------------------------:|
| CockroachDB-1-1-1-1 | 970.22 | 9806527.16 | 0.00 | 0.07 | 0.00 |
| CockroachDB-1-1-2-1 | 970.22 | 9806527.16 | 0.00 | 0.07 | 0.00 |
#### Execution phase: component worker
| DBMS | Raft Messages Received (AppResp) [msgs/s] | Raft Network In (Bytes/sec) | Raft Recovery Snapshot In (Bytes/sec) | Replicate Queue Adds Attempted [adds/s] | Replicate Queue Purgatory Count |
|:--------------------|--------------------------------------------:|------------------------------:|----------------------------------------:|------------------------------------------:|----------------------------------:|
| CockroachDB-1-1-1-1 | 8106.30 | 3795993.72 | 0.00 | 0.00 | 0.00 |
| CockroachDB-1-1-2-1 | 5253.89 | 2100009.01 | 0.00 | 0.00 | 0.00 |
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Loading phase: component worker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component worker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Throughput (requests/second) contains no 0 or NaN
* TEST passed: Workflow as planned
Redis
YCSB
Example:
bexhoma ycsb \
-dbms Redis \
-sf 1 \
-xwl a \
-xtb 16384 \
-xnbf 4 \
-xnlf 12 \
-nc 1 \
-ne 1 \
-nlp 8 \
-nlt 64 \
-nbp 1 \
-nbt 128 \
-nw 3 \
-nwr 1 \
-xop 10 \
-m \
-ma \
-mc \
-tr \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/docs_ycsb_redis_appmetrics.log
Show docs_ycsb_redis_appmetrics.log
## Show Summary
### Workload
YCSB SF=1
* Type: ycsb
* Duration: 632s
* Code: 1783892151
* YCSB driver runs the experiment.
* This experiment compares run time and resource consumption of YCSB queries.
* Workload is 'A'.
* Number of rows to insert is 1000000.
* Ordering of inserts is hashed.
* Number of operations is 10000000.
* Batch size is ''.
* Target is based on multiples of '16384'.
* Factors for loading are [12].
* Factors for benchmarking are [4].
* Experiment uses bexhoma version 0.10.5.
* System metrics are monitored by a cluster-wide installation.
* Application metrics are monitored by sidecar containers.
* Experiment is limited to DBMS ['Redis'].
* Import is handled by 8 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker36.
* Database uses ephemeral storage of size 50Gi.
* Loading is tested with [64] threads, split into [8] pods.
* Benchmarking is tested with [128] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* Redis-1-1-1-1 uses docker image redis:7.4.2
* RAM:2164173213696
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:1086940
* cpu_list:0-223
* args:['--maxclients', '10000', '--io-threads', '64']
* requests_cpu:4
* requests_memory:16Gi
* worker 0
* RAM:1077382598656
* CPU:AMD EPYC 7742 64-Core Processor
* Cores:256
* host:6.8.0-1058-nvidia
* node:cl-worker28
* disk:578614
* cpu_list:0-255
* worker 1
* RAM:540590841856
* CPU:AMD EPYC 7352 24-Core Processor
* Cores:96
* host:6.8.0-117-generic
* node:cl-worker25
* disk:161323
* cpu_list:0-95
* worker 2
* RAM:1077381287936
* CPU:AMD EPYC 7742 64-Core Processor
* Cores:256
* host:6.8.0-111-generic
* node:cl-worker27
* disk:1371668
* cpu_list:0-255
* worker 3
* RAM:540590804992
* CPU:AMD EPYC 7352 24-Core Processor
* Cores:96
* host:6.8.0-124-generic
* node:cl-worker24
* disk:134433
* cpu_list:0-95
* worker 4
* RAM:540597907456
* CPU:Intel(R) Xeon(R) 6767P
* Cores:256
* host:6.8.0-124-generic
* node:cl-worker39
* disk:321198
* cpu_list:0-255
* worker 5
* RAM:1081742745600
* CPU:AMD EPYC 7502 32-Core Processor
* Cores:128
* host:6.8.0-117-generic
* node:cl-worker29
* disk:942030
* cpu_list:0-127
* eval_parameters
* code:1783892151
* BEXHOMA_REPLICAS:1
* BEXHOMA_WORKERS:3
### SUT Container Restarts
* bexhoma-sut-redis-1-1783892151-65474664c6-5kghb: 0
* bx-w-redis-ycsb-1-0: 0 0
* bx-w-redis-ycsb-1-1: 0 0
* bx-w-redis-ycsb-1-2: 0 0
* bx-w-redis-ycsb-1-3: 0 0
* bx-w-redis-ycsb-1-4: 0 0
* bx-w-redis-ycsb-1-5: 0 0
### Workflow
#### Actual
* DBMS Redis-1 - Experiment 1 Client 1: ycsb (1 pods)
#### Planned
* DBMS Redis-1 - Experiment 1 Client 1: ycsb (1 pods)
### Loading
#### Per Connection
| connection | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) | sf | Throughput [SF/h] |
|:----------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|-----:|--------------------:|
| Redis-1-1-0-1-1 | 1.00 | 8.00 | 24576.00 | 8.00 | 0.00 | 1836.16 | 68077.00 | 125000.00 | 9303.00 | 1.00 | 52.88 |
| Redis-1-1-0-1-2 | 1.00 | 8.00 | 24576.00 | 8.00 | 0.00 | 1833.38 | 68180.00 | 125000.00 | 9463.00 | 1.00 | 52.80 |
| Redis-1-1-0-1-3 | 1.00 | 8.00 | 24576.00 | 8.00 | 0.00 | 1842.43 | 67845.00 | 125000.00 | 9487.00 | 1.00 | 53.06 |
| Redis-1-1-0-1-4 | 1.00 | 8.00 | 24576.00 | 8.00 | 0.00 | 1834.38 | 68143.00 | 125000.00 | 9439.00 | 1.00 | 52.83 |
| Redis-1-1-0-1-5 | 1.00 | 8.00 | 24576.00 | 8.00 | 0.00 | 1835.72 | 68093.00 | 125000.00 | 9599.00 | 1.00 | 52.87 |
| Redis-1-1-0-1-6 | 1.00 | 8.00 | 24576.00 | 8.00 | 0.00 | 1838.51 | 67990.00 | 125000.00 | 9359.00 | 1.00 | 52.95 |
| Redis-1-1-0-1-7 | 1.00 | 8.00 | 24576.00 | 8.00 | 0.00 | 1833.70 | 68168.00 | 125000.00 | 9447.00 | 1.00 | 52.81 |
| Redis-1-1-0-1-8 | 1.00 | 8.00 | 24576.00 | 8.00 | 0.00 | 1858.18 | 67270.00 | 125000.00 | 9447.00 | 1.00 | 53.52 |
#### Per Run
| DBMS | experiment_run | threads | target | pod_count | exceptions | sf | Throughput [SF/h] | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) |
|:----------|-----------------:|----------:|----------:|------------:|-------------:|-----:|--------------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|
| Redis-1-1 | 1.00 | 64.00 | 196608.00 | 8.00 | 0.00 | 1.00 | 52.80 | 14712.47 | 68180.00 | 1000000.00 | 9443.00 |
### Execution
#### Per Connection
| DBMS | phase | job | configuration | experiment_run | client | benchmark_run | child | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:----------------|:------------|:--------------|:----------------|-----------------:|---------:|----------------:|--------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| Redis-1-1-1-1-1 | Redis-1-1-1 | Redis-1-1-1-1 | Redis-1 | 1 | 1 | 1 | 1 | 128 | 65536 | 1 | 0 | 55248.62 | 181000.00 | 4999079 | 6587.00 | 5000921 | 6555.00 |
#### Per Phase
| DBMS | phase | experiment_run | threads | target | benchmark_run | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:------------|:------------|-----------------:|----------:|---------:|----------------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| Redis-1-1-1 | Redis-1-1-1 | 1 | 128 | 65536 | 1 | 1 | 0 | 55248.62 | 181000.00 | 4999079 | 6587.00 | 5000921 | 6555.00 |
### Monitoring
### Loading phase: component worker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| Redis-1-1-1-1 | 225.56 | 3.38 | 3.59 | 3.59 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| Redis-1-1-1-1 | 153.24 | 2.77 | 0.12 | 0.12 |
### Execution phase: component worker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| Redis-1-1-1-1 | 565.16 | 4.29 | 4.04 | 4.06 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:--------------|-------------:|----------:|---------------:|----------------------:|
| Redis-1-1-1-1 | 547.77 | 4.29 | 0.30 | 0.30 |
### Application Metrics
#### Loading phase: component worker
| DBMS | Redis Cluster State | Connected Clients [count] | Redis Memory Used [Gi] | Redis Master Link Status | Redis Operations Rate [ops/s] |
|:--------------|----------------------:|----------------------------:|-------------------------:|---------------------------:|--------------------------------:|
| Redis-1-1-1-1 | 6.00 | 201.00 | 3.46 | 3.00 | 5530.00 |
#### Execution phase: component worker
| DBMS | Redis Cluster State | Connected Clients [count] | Redis Memory Used [Gi] | Redis Master Link Status | Redis Operations Rate [ops/s] |
|:--------------|----------------------:|----------------------------:|-------------------------:|---------------------------:|--------------------------------:|
| Redis-1-1-1-1 | 6.00 | 393.00 | 3.50 | 3.00 | 6813.02 |
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Loading phase: component worker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component worker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Execution Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: contains no FAILED column
TiDB
YCSB
Example:
bexhoma ycsb \
-dbms TiDB \
-sf 1 \
-xwl a \
-xtb 16384 \
-xnbf 1 \
-xnlf 1 \
-nc 1 \
-ne 1 \
-nlp 8 \
-nlt 64 \
-nbp 1 \
-nbt 64 \
-xnsr 3 \
-nw 3 \
-xnpd 3 \
-nwr 3 \
-xop 1 \
-m \
-ma \
-mc \
-ms $BEXHOMA_MS \
-tr \
-rss 50Gi \
run &>$LOG_DIR/docs_ycsb_tidb_appmetrics.log
Show docs_ycsb_tidb_appmetrics.log
## Show Summary
### Workload
YCSB SF=1
* Type: ycsb
* Duration: 826s
* Code: 1783892810
* YCSB driver runs the experiment.
* This experiment compares run time and resource consumption of YCSB queries.
* Workload is 'A'.
* Number of rows to insert is 1000000.
* Ordering of inserts is hashed.
* Number of operations is 1000000.
* Batch size is ''.
* Target is based on multiples of '16384'.
* Factors for loading are [1].
* Factors for benchmarking are [1].
* Experiment uses bexhoma version 0.10.5.
* System metrics are monitored by a cluster-wide installation.
* Application metrics are monitored by sidecar containers.
* Experiment is limited to DBMS ['TiDB'].
* Import is handled by 8 processes (pods).
* Database uses ephemeral storage of size 50Gi.
* Loading is tested with [64] threads, split into [8] pods.
* Benchmarking is tested with [64] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* TiDB-1-1-1-1 uses docker image pingcap/tidb:v7.1.6
* RAM:540590841856
* CPU:AMD EPYC 7352 24-Core Processor
* Cores:96
* host:6.8.0-117-generic
* node:cl-worker25
* disk:170557
* cpu_list:0-95
* requests_cpu:4
* requests_memory:16Gi
* sut 0
* RAM:540590841856
* CPU:AMD EPYC 7352 24-Core Processor
* Cores:96
* host:6.8.0-117-generic
* node:cl-worker25
* disk:170609
* cpu_list:0-95
* sut 1
* RAM:540597907456
* CPU:Intel(R) Xeon(R) 6767P
* Cores:256
* host:6.8.0-124-generic
* node:cl-worker39
* disk:320592
* cpu_list:0-255
* sut 2
* RAM:1077382598656
* CPU:AMD EPYC 7742 64-Core Processor
* Cores:256
* host:6.8.0-1058-nvidia
* node:cl-worker28
* disk:592162
* cpu_list:0-255
* pd 0
* RAM:1077382598656
* CPU:AMD EPYC 7742 64-Core Processor
* Cores:256
* host:6.8.0-1058-nvidia
* node:cl-worker28
* disk:597940
* cpu_list:0-255
* pd 1
* RAM:540590804992
* CPU:AMD EPYC 7352 24-Core Processor
* Cores:96
* host:6.8.0-124-generic
* node:cl-worker24
* disk:133836
* cpu_list:0-95
* pd 2
* RAM:540590841856
* CPU:AMD EPYC 7352 24-Core Processor
* Cores:96
* host:6.8.0-117-generic
* node:cl-worker25
* disk:170647
* cpu_list:0-95
* tikv 0
* RAM:1077382598656
* CPU:AMD EPYC 7742 64-Core Processor
* Cores:256
* host:6.8.0-1058-nvidia
* node:cl-worker28
* disk:592155
* cpu_list:0-255
* tikv 1
* RAM:1077381287936
* CPU:AMD EPYC 7742 64-Core Processor
* Cores:256
* host:6.8.0-111-generic
* node:cl-worker27
* disk:1392000
* cpu_list:0-255
* tikv 2
* RAM:540590841856
* CPU:AMD EPYC 7352 24-Core Processor
* Cores:96
* host:6.8.0-117-generic
* node:cl-worker25
* disk:170586
* cpu_list:0-95
* eval_parameters
* code:1783892810
* BEXHOMA_REPLICAS:3
* BEXHOMA_WORKERS:3
### SUT Container Restarts
* bexhoma-sut-tidb-1-1783892810-6749979d78-jp74c: 0
* bexhoma-sut-tidb-1-1783892810-6749979d78-lh57n: 0
* bexhoma-sut-tidb-1-1783892810-6749979d78-sq9bz: 0
* bexhoma-pd-tidb-ycsb-1-0: 0
* bexhoma-pd-tidb-ycsb-1-1: 0
* bexhoma-pd-tidb-ycsb-1-2: 0
* bexhoma-tikv-tidb-ycsb-1-0: 0
* bexhoma-tikv-tidb-ycsb-1-1: 0
* bexhoma-tikv-tidb-ycsb-1-2: 0
### Workflow
#### Actual
* DBMS TiDB-1 - Experiment 1 Client 1: ycsb (1 pods)
#### Planned
* DBMS TiDB-1 - Experiment 1 Client 1: ycsb (1 pods)
### Loading
#### Per Connection
| connection | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) | sf | Throughput [SF/h] |
|:---------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|-----:|--------------------:|
| TiDB-1-1-0-1-1 | 1.00 | 8.00 | 2048.00 | 8.00 | 0.00 | 706.97 | 176810.00 | 125000.00 | 78911.00 | 1.00 | 20.36 |
| TiDB-1-1-0-1-2 | 1.00 | 8.00 | 2048.00 | 8.00 | 0.00 | 707.59 | 176655.00 | 125000.00 | 77823.00 | 1.00 | 20.38 |
| TiDB-1-1-0-1-3 | 1.00 | 8.00 | 2048.00 | 8.00 | 0.00 | 707.37 | 176712.00 | 125000.00 | 80255.00 | 1.00 | 20.37 |
| TiDB-1-1-0-1-4 | 1.00 | 8.00 | 2048.00 | 8.00 | 0.00 | 706.94 | 176818.00 | 125000.00 | 79615.00 | 1.00 | 20.36 |
| TiDB-1-1-0-1-5 | 1.00 | 8.00 | 2048.00 | 8.00 | 0.00 | 720.36 | 173524.00 | 125000.00 | 77439.00 | 1.00 | 20.75 |
| TiDB-1-1-0-1-6 | 1.00 | 8.00 | 2048.00 | 8.00 | 0.00 | 708.83 | 176346.00 | 125000.00 | 78975.00 | 1.00 | 20.41 |
| TiDB-1-1-0-1-7 | 1.00 | 8.00 | 2048.00 | 8.00 | 0.00 | 707.69 | 176632.00 | 125000.00 | 78463.00 | 1.00 | 20.38 |
| TiDB-1-1-0-1-8 | 1.00 | 8.00 | 2048.00 | 8.00 | 0.00 | 710.32 | 175978.00 | 125000.00 | 77887.00 | 1.00 | 20.46 |
#### Per Run
| DBMS | experiment_run | threads | target | pod_count | exceptions | sf | Throughput [SF/h] | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) |
|:---------|-----------------:|----------:|---------:|------------:|-------------:|-----:|--------------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|
| TiDB-1-1 | 1.00 | 64.00 | 16384.00 | 8.00 | 0.00 | 1.00 | 20.36 | 5676.07 | 176818.00 | 1000000.00 | 78671.00 |
### Execution
#### Per Connection
| DBMS | phase | job | configuration | experiment_run | client | benchmark_run | child | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:---------------|:-----------|:-------------|:----------------|-----------------:|---------:|----------------:|--------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| TiDB-1-1-1-1-1 | TiDB-1-1-1 | TiDB-1-1-1-1 | TiDB-1 | 1 | 1 | 1 | 1 | 64 | 16384 | 1 | 0 | 4162.68 | 240230.00 | 500772 | 7891.00 | 499228 | 331519.00 |
#### Per Phase
| DBMS | phase | experiment_run | threads | target | benchmark_run | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:-----------|:-----------|-----------------:|----------:|---------:|----------------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| TiDB-1-1-1 | TiDB-1-1-1 | 1 | 64 | 16384 | 1 | 1 | 0 | 4162.68 | 240230.00 | 500772 | 7891.00 | 499228 | 331519.00 |
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------|-------------:|----------:|---------------:|----------------------:|
| TiDB-1-1-1-1 | 1208.31 | 8.29 | 2.88 | 3.22 |
### Loading phase: component pd
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------|-------------:|----------:|---------------:|----------------------:|
| TiDB-1-1-1-1 | 122.25 | 0.85 | 0.28 | 0.28 |
### Loading phase: component tikv
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------|-------------:|----------:|---------------:|----------------------:|
| TiDB-1-1-1-1 | 1184.00 | 8.60 | 7.03 | 17.83 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------|-------------:|----------:|---------------:|----------------------:|
| TiDB-1-1-1-1 | 120.75 | 0.89 | 0.23 | 0.23 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------|-------------:|----------:|---------------:|----------------------:|
| TiDB-1-1-1-1 | 1146.49 | 8.73 | 1.28 | 1.63 |
### Execution phase: component pd
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------|-------------:|----------:|---------------:|----------------------:|
| TiDB-1-1-1-1 | 136.37 | 0.93 | 0.27 | 0.27 |
### Execution phase: component tikv
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------|-------------:|----------:|---------------:|----------------------:|
| TiDB-1-1-1-1 | 1332.57 | 9.86 | 9.05 | 22.55 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------|-------------:|----------:|---------------:|----------------------:|
| TiDB-1-1-1-1 | 85.94 | 0.75 | 0.14 | 0.14 |
### Application Metrics
#### Loading phase: SUT deployment
| DBMS | TiDB SQL Statement Throughput [ops/s] | TiDB Avg Query Duration [ms] |
|:-------------|----------------------------------------:|-------------------------------:|
| TiDB-1-1-1-1 | 3446.56 | 10.77 |
#### Loading phase: component pd
| DBMS | PD Cluster Leader Count | PD Leader Balance Actions [ops] |
|:-------------|--------------------------:|----------------------------------:|
| TiDB-1-1-1-1 | 64.00 | 12.00 |
#### Loading phase: component tikv
| DBMS | TiKV Store Used [%] | TiKV Compaction Time Median [s] | TiKV Compaction Flow [Gi] | TiKV Compaction Pending [Gi] |
|:-------------|----------------------:|----------------------------------:|----------------------------:|-------------------------------:|
| TiDB-1-1-1-1 | 0.18 | 104429564.00 | 6.22 | 1.44 |
#### Execution phase: SUT deployment
| DBMS | TiDB SQL Statement Throughput [ops/s] | TiDB Avg Query Duration [ms] |
|:-------------|----------------------------------------:|-------------------------------:|
| TiDB-1-1-1-1 | 1553.79 | 12.37 |
#### Execution phase: component pd
| DBMS | PD Cluster Leader Count | PD Leader Balance Actions [ops] |
|:-------------|--------------------------:|----------------------------------:|
| TiDB-1-1-1-1 | 70.00 | 0.00 |
#### Execution phase: component tikv
| DBMS | TiKV Store Used [%] | TiKV Compaction Time Median [s] | TiKV Compaction Flow [Gi] | TiKV Compaction Pending [Gi] |
|:-------------|----------------------:|----------------------------------:|----------------------------:|-------------------------------:|
| TiDB-1-1-1-1 | 0.26 | 56219681.00 | 2.26 | 1.42 |
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component pd contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component tikv contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component pd contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component tikv contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Execution Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: contains no FAILED column
Benchbase’s TPC-C
Example:
bexhoma benchbase \
-dbms TiDB \
-sf 16 \
-xsd 5 \
-xtb 1024 \
-xnbf 16 \
-nbp 1,2 \
-nbt 16 \
-xnsr 3 \
-nw 3 \
-xnpd 3 \
-nwr 3 \
-m \
-mc \
-ms $BEXHOMA_MS \
-tr \
-rss 50Gi \
run &>$LOG_DIR/docs_benchbase_tidb_appmetrics.log
Show docs_benchbase_tidb_appmetrics.log
## Show Summary
### Workload
Benchbase Workload tpcc SF=16
* Type: benchbase
* Duration: 1419s
* Code: 1783893680
* Benchbase runs a TPC-C experiment.
* This experiment compares run time and resource consumption of Benchbase queries in different DBMS.
* Benchbase data is generated and loaded using several threads.
* Benchmark is 'tpcc'. Scaling factor is 16. Target is based on multiples of '1024'. Factors for benchmarking are [16]. Benchmarking runs for 5 minutes.
* Experiment uses bexhoma version 0.10.5.
* System metrics are monitored by a cluster-wide installation.
* Experiment is limited to DBMS ['TiDB'].
* Import is handled by 1 processes (pods).
* Database uses ephemeral storage of size 50Gi.
* Loading is tested with [1] threads, split into [1] pods.
* Benchmarking is tested with [16] threads, split into [1, 2] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* TiDB-1-1-1-1 uses docker image pingcap/tidb:v7.1.6
* RAM:1077382598656
* CPU:AMD EPYC 7742 64-Core Processor
* Cores:256
* host:6.8.0-1058-nvidia
* node:cl-worker28
* disk:582341
* cpu_list:0-255
* requests_cpu:4
* requests_memory:16Gi
* sut 0
* RAM:1077382598656
* CPU:AMD EPYC 7742 64-Core Processor
* Cores:256
* host:6.8.0-1058-nvidia
* node:cl-worker28
* disk:577312
* cpu_list:0-255
* sut 1
* RAM:540590804992
* CPU:AMD EPYC 7352 24-Core Processor
* Cores:96
* host:6.8.0-124-generic
* node:cl-worker24
* disk:142627
* cpu_list:0-95
* sut 2
* RAM:540590841856
* CPU:AMD EPYC 7352 24-Core Processor
* Cores:96
* host:6.8.0-117-generic
* node:cl-worker25
* disk:161794
* cpu_list:0-95
* pd 0
* RAM:540590841856
* CPU:AMD EPYC 7352 24-Core Processor
* Cores:96
* host:6.8.0-117-generic
* node:cl-worker25
* disk:161794
* cpu_list:0-95
* pd 1
* RAM:540597907456
* CPU:Intel(R) Xeon(R) 6767P
* Cores:256
* host:6.8.0-124-generic
* node:cl-worker39
* disk:320597
* cpu_list:0-255
* pd 2
* RAM:540590804992
* CPU:AMD EPYC 7352 24-Core Processor
* Cores:96
* host:6.8.0-124-generic
* node:cl-worker24
* disk:142627
* cpu_list:0-95
* tikv 0
* RAM:540590804992
* CPU:AMD EPYC 7352 24-Core Processor
* Cores:96
* host:6.8.0-124-generic
* node:cl-worker24
* disk:142627
* cpu_list:0-95
* tikv 1
* RAM:1081742745600
* CPU:AMD EPYC 7502 32-Core Processor
* Cores:128
* host:6.8.0-117-generic
* node:cl-worker29
* disk:965701
* cpu_list:0-127
* tikv 2
* RAM:2164173213696
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:1111265
* cpu_list:0-223
* eval_parameters
* code:1783893680
* BEXHOMA_REPLICAS:3
* BEXHOMA_WORKERS:3
* TiDB-1-1-2-1 uses docker image pingcap/tidb:v7.1.6
* RAM:1077382598656
* CPU:AMD EPYC 7742 64-Core Processor
* Cores:256
* host:6.8.0-1058-nvidia
* node:cl-worker28
* disk:579884
* cpu_list:0-255
* requests_cpu:4
* requests_memory:16Gi
* sut 0
* RAM:1077382598656
* CPU:AMD EPYC 7742 64-Core Processor
* Cores:256
* host:6.8.0-1058-nvidia
* node:cl-worker28
* disk:578508
* cpu_list:0-255
* sut 1
* RAM:540590804992
* CPU:AMD EPYC 7352 24-Core Processor
* Cores:96
* host:6.8.0-124-generic
* node:cl-worker24
* disk:141199
* cpu_list:0-95
* sut 2
* RAM:540590841856
* CPU:AMD EPYC 7352 24-Core Processor
* Cores:96
* host:6.8.0-117-generic
* node:cl-worker25
* disk:161795
* cpu_list:0-95
* pd 0
* RAM:540590841856
* CPU:AMD EPYC 7352 24-Core Processor
* Cores:96
* host:6.8.0-117-generic
* node:cl-worker25
* disk:161795
* cpu_list:0-95
* pd 1
* RAM:540597907456
* CPU:Intel(R) Xeon(R) 6767P
* Cores:256
* host:6.8.0-124-generic
* node:cl-worker39
* disk:320603
* cpu_list:0-255
* pd 2
* RAM:540590804992
* CPU:AMD EPYC 7352 24-Core Processor
* Cores:96
* host:6.8.0-124-generic
* node:cl-worker24
* disk:141230
* cpu_list:0-95
* tikv 0
* RAM:540590804992
* CPU:AMD EPYC 7352 24-Core Processor
* Cores:96
* host:6.8.0-124-generic
* node:cl-worker24
* disk:141466
* cpu_list:0-95
* tikv 1
* RAM:1081742745600
* CPU:AMD EPYC 7502 32-Core Processor
* Cores:128
* host:6.8.0-117-generic
* node:cl-worker29
* disk:964905
* cpu_list:0-127
* tikv 2
* RAM:2164173213696
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:1109491
* cpu_list:0-223
* eval_parameters
* code:1783893680
* BEXHOMA_REPLICAS:3
* BEXHOMA_WORKERS:3
### SUT Container Restarts
* bexhoma-sut-tidb-1-1783893680-7b9cfc84d9-bbs8w: 0
* bexhoma-sut-tidb-1-1783893680-7b9cfc84d9-fz7fv: 0
* bexhoma-sut-tidb-1-1783893680-7b9cfc84d9-jc5df: 0
* bexhoma-pd-tidb-benchbase-tpcc-16-0: 0
* bexhoma-pd-tidb-benchbase-tpcc-16-1: 0
* bexhoma-pd-tidb-benchbase-tpcc-16-2: 0
* bexhoma-tikv-tidb-benchbase-tpcc-16-0: 0
* bexhoma-tikv-tidb-benchbase-tpcc-16-1: 0
* bexhoma-tikv-tidb-benchbase-tpcc-16-2: 0
### Workflow
#### Actual
* DBMS TiDB-1 - Experiment 1 Client 1: benchbase (1 pods)
* DBMS TiDB-1 - Experiment 1 Client 2: benchbase (2 pods)
#### Planned
* DBMS TiDB-1 - Experiment 1 Client 1: benchbase (1 pods)
* DBMS TiDB-1 - Experiment 1 Client 2: benchbase (2 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:---------|-----------------:|------:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| TiDB-1-1 | 1 | 16.00 | 485.00 | 1.00 | 0.00 | 218.00 | 266.00 | 1 | 1 | | None | 0 | False | 118.76 |
### Execution
#### Per Connection
| DBMS | phase | job | experiment_run | terminals | target | client | benchmark_run | child | tenant_id | time | num_errors | Throughput (requests/second) | Goodput (requests/second) | efficiency | Latency Distribution.95th Percentile Latency (microseconds) | Latency Distribution.Average Latency (microseconds) |
|:---------------|:-----------|:-------------|-----------------:|------------:|---------:|---------:|----------------:|--------:|------------:|-------:|-------------:|-------------------------------:|----------------------------:|-------------:|--------------------------------------------------------------:|------------------------------------------------------:|
| TiDB-1-1-1-1-1 | TiDB-1-1-1 | TiDB-1-1-1-1 | 1 | 16 | 16384 | 1 | 1 | 1 | -1 | 300.00 | 0 | 346.82 | 345.22 | 0.00 | 129138.00 | 45625.00 |
| TiDB-1-1-2-1-1 | TiDB-1-1-2 | TiDB-1-1-2-1 | 1 | 8 | 8192 | 2 | 1 | 1 | -1 | 300.00 | 0 | 167.67 | 165.79 | 0.00 | 119802.00 | 47297.00 |
| TiDB-1-1-2-1-2 | TiDB-1-1-2 | TiDB-1-1-2-1 | 1 | 8 | 8192 | 2 | 1 | 2 | -1 | 300.00 | 0 | 168.23 | 166.45 | 0.00 | 123754.00 | 47407.00 |
#### Per Phase
| DBMS | phase | experiment_run | terminals | target | benchmark_run | pod_count | tenant_id | time | num_errors | Throughput (requests/second) | Goodput (requests/second) | efficiency | Latency Distribution.95th Percentile Latency (microseconds) | Latency Distribution.Average Latency (microseconds) |
|:-----------|:-----------|-----------------:|------------:|---------:|----------------:|------------:|------------:|-------:|-------------:|-------------------------------:|----------------------------:|-------------:|--------------------------------------------------------------:|------------------------------------------------------:|
| TiDB-1-1-1 | TiDB-1-1-1 | 1 | 16 | 16384 | 1 | 1 | -1 | 300.00 | 0 | 346.82 | 345.22 | 0.00 | 129138.00 | 45625.00 |
| TiDB-1-1-2 | TiDB-1-1-2 | 1 | 16 | 16384 | 1 | 2 | -1 | 300.00 | 0 | 335.90 | 332.24 | 0.00 | 123754.00 | 47352.00 |
### Monitoring
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------|-------------:|----------:|---------------:|----------------------:|
| TiDB-1-1-1-1 | 1045.59 | 8.60 | 3.99 | 4.32 |
| TiDB-1-1-2-1 | 1045.59 | 8.60 | 3.99 | 4.32 |
### Loading phase: component pd
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------|-------------:|----------:|---------------:|----------------------:|
| TiDB-1-1-1-1 | 119.00 | 0.55 | 0.33 | 0.33 |
| TiDB-1-1-2-1 | 119.00 | 0.55 | 0.33 | 0.33 |
### Loading phase: component tikv
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------|-------------:|----------:|---------------:|----------------------:|
| TiDB-1-1-1-1 | 2840.89 | 18.27 | 10.96 | 30.53 |
| TiDB-1-1-2-1 | 2840.89 | 18.27 | 10.96 | 30.53 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------|-------------:|----------:|---------------:|----------------------:|
| TiDB-1-1-1-1 | 136.53 | 1.55 | 0.68 | 0.68 |
| TiDB-1-1-2-1 | 136.53 | 1.55 | 0.68 | 0.68 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------|-------------:|----------:|---------------:|----------------------:|
| TiDB-1-1-1-1 | 2304.24 | 9.02 | 2.10 | 2.44 |
| TiDB-1-1-2-1 | 2166.24 | 8.52 | 2.82 | 3.16 |
### Execution phase: component pd
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------|-------------:|----------:|---------------:|----------------------:|
| TiDB-1-1-1-1 | 361.50 | 1.41 | 0.32 | 0.32 |
| TiDB-1-1-2-1 | 372.91 | 1.37 | 0.32 | 0.33 |
### Execution phase: component tikv
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------|-------------:|----------:|---------------:|----------------------:|
| TiDB-1-1-1-1 | 3910.02 | 16.52 | 14.30 | 32.54 |
| TiDB-1-1-2-1 | 3875.33 | 15.20 | 16.49 | 35.40 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:-------------|-------------:|----------:|---------------:|----------------------:|
| TiDB-1-1-1-1 | 219.99 | 0.93 | 0.50 | 0.50 |
| TiDB-1-1-2-1 | 219.99 | 1.85 | 0.50 | 0.50 |
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component pd contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component tikv contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component pd contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component tikv contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Throughput (requests/second) contains no 0 or NaN
* TEST passed: Workflow as planned
PGBouncer
YCSB
Example:
bexhoma ycsb \
-dbms PGBouncer \
-sf 16 \
-xwl c \
-xtb 16384 \
-xnbf 11 \
-xnlf 11 \
-nc 1 \
-ne 1 \
-nlp 16 \
-nlt 64 \
-nbp 16 \
-nbt 128 \
-xnpp 4 \
-xnpi 128 \
-xnpo 64 \
-xop 16 \
-m \
-ma \
-mc \
-ms $BEXHOMA_MS \
-tr \
-lr 64Gi \
-rr 64Gi \
-rss 80Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/docs_ycsb_pgbouncer_appmetrics.log
Show docs_ycsb_pgbouncer_appmetrics.log
## Show Summary
### Workload
YCSB SF=16
* Type: ycsb
* Duration: 1144s
* Code: 1783895142
* YCSB driver runs the experiment.
* This experiment compares run time and resource consumption of YCSB queries.
* Workload is 'C'.
* Number of rows to insert is 16000000.
* Ordering of inserts is hashed.
* Number of operations is 16000000.
* Batch size is ''.
* Target is based on multiples of '16384'.
* Factors for loading are [11].
* Factors for benchmarking are [11].
* Experiment uses bexhoma version 0.10.5.
* System metrics are monitored by a cluster-wide installation.
* Application metrics are monitored by sidecar containers.
* Experiment is limited to DBMS ['PGBouncer'].
* Import is handled by 16 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker36.
* Database uses ephemeral storage of size 80Gi.
* Loading is tested with [64] threads, split into [16] pods.
* Benchmarking is tested with [128] threads, split into [16] pods.
* Pooling is done with [4] pods having [128] inbound and [64] outbound connections in total.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* PGBouncer-1-1-1-1 uses docker image postgres:18.3
* RAM:2164173213696
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:1125018
* cpu_list:0-223
* args:['-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'max_wal_size=32GB', '-c', 'shared_buffers=64GB', '-c', 'max_connections=2048', '-c', 'autovacuum_max_workers=10', '-c', 'autovacuum_vacuum_cost_limit=3000', '-c', 'vacuum_cost_limit=1000', '-c', 'checkpoint_completion_target=0.9', '-c', 'cpu_tuple_cost=0.03', '-c', 'effective_cache_size=64GB', '-c', 'maintenance_work_mem=2GB', '-c', 'wal_buffers=1GB', '-c', 'work_mem=32GB', '-c', 'temp_buffers=4GB', '-c', 'autovacuum_work_mem=-1', '-c', 'max_stack_depth=7MB', '-c', 'max_files_per_process=4000', '-c', 'effective_io_concurrency=32', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'synchronous_commit=off', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_warning=0', '-c', 'autovacuum=off', '-c', 'max_locks_per_transaction=64', '-c', 'max_pred_locks_per_transaction=64', '-c', 'default_statistics_target=1000', '-c', 'random_page_cost=60']
* requests_cpu:4
* requests_memory:64Gi
* limits_memory:64Gi
* pool 0
* node:cl-worker39
* args:['-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'max_wal_size=32GB', '-c', 'shared_buffers=64GB', '-c', 'max_connections=2048', '-c', 'autovacuum_max_workers=10', '-c', 'autovacuum_vacuum_cost_limit=3000', '-c', 'vacuum_cost_limit=1000', '-c', 'checkpoint_completion_target=0.9', '-c', 'cpu_tuple_cost=0.03', '-c', 'effective_cache_size=64GB', '-c', 'maintenance_work_mem=2GB', '-c', 'wal_buffers=1GB', '-c', 'work_mem=32GB', '-c', 'temp_buffers=4GB', '-c', 'autovacuum_work_mem=-1', '-c', 'max_stack_depth=7MB', '-c', 'max_files_per_process=4000', '-c', 'effective_io_concurrency=32', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'synchronous_commit=off', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_warning=0', '-c', 'autovacuum=off', '-c', 'max_locks_per_transaction=64', '-c', 'max_pred_locks_per_transaction=64', '-c', 'default_statistics_target=1000', '-c', 'random_page_cost=60']
* pool 1
* node:cl-worker24
* args:['-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'max_wal_size=32GB', '-c', 'shared_buffers=64GB', '-c', 'max_connections=2048', '-c', 'autovacuum_max_workers=10', '-c', 'autovacuum_vacuum_cost_limit=3000', '-c', 'vacuum_cost_limit=1000', '-c', 'checkpoint_completion_target=0.9', '-c', 'cpu_tuple_cost=0.03', '-c', 'effective_cache_size=64GB', '-c', 'maintenance_work_mem=2GB', '-c', 'wal_buffers=1GB', '-c', 'work_mem=32GB', '-c', 'temp_buffers=4GB', '-c', 'autovacuum_work_mem=-1', '-c', 'max_stack_depth=7MB', '-c', 'max_files_per_process=4000', '-c', 'effective_io_concurrency=32', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'synchronous_commit=off', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_warning=0', '-c', 'autovacuum=off', '-c', 'max_locks_per_transaction=64', '-c', 'max_pred_locks_per_transaction=64', '-c', 'default_statistics_target=1000', '-c', 'random_page_cost=60']
* pool 2
* node:cl-worker25
* args:['-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'max_wal_size=32GB', '-c', 'shared_buffers=64GB', '-c', 'max_connections=2048', '-c', 'autovacuum_max_workers=10', '-c', 'autovacuum_vacuum_cost_limit=3000', '-c', 'vacuum_cost_limit=1000', '-c', 'checkpoint_completion_target=0.9', '-c', 'cpu_tuple_cost=0.03', '-c', 'effective_cache_size=64GB', '-c', 'maintenance_work_mem=2GB', '-c', 'wal_buffers=1GB', '-c', 'work_mem=32GB', '-c', 'temp_buffers=4GB', '-c', 'autovacuum_work_mem=-1', '-c', 'max_stack_depth=7MB', '-c', 'max_files_per_process=4000', '-c', 'effective_io_concurrency=32', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'synchronous_commit=off', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_warning=0', '-c', 'autovacuum=off', '-c', 'max_locks_per_transaction=64', '-c', 'max_pred_locks_per_transaction=64', '-c', 'default_statistics_target=1000', '-c', 'random_page_cost=60']
* pool 3
* node:cl-worker28
* args:['-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'max_wal_size=32GB', '-c', 'shared_buffers=64GB', '-c', 'max_connections=2048', '-c', 'autovacuum_max_workers=10', '-c', 'autovacuum_vacuum_cost_limit=3000', '-c', 'vacuum_cost_limit=1000', '-c', 'checkpoint_completion_target=0.9', '-c', 'cpu_tuple_cost=0.03', '-c', 'effective_cache_size=64GB', '-c', 'maintenance_work_mem=2GB', '-c', 'wal_buffers=1GB', '-c', 'work_mem=32GB', '-c', 'temp_buffers=4GB', '-c', 'autovacuum_work_mem=-1', '-c', 'max_stack_depth=7MB', '-c', 'max_files_per_process=4000', '-c', 'effective_io_concurrency=32', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'synchronous_commit=off', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_warning=0', '-c', 'autovacuum=off', '-c', 'max_locks_per_transaction=64', '-c', 'max_pred_locks_per_transaction=64', '-c', 'default_statistics_target=1000', '-c', 'random_page_cost=60']
* eval_parameters
* code:1783895142
### SUT Container Restarts
* bexhoma-pool-pgbouncer-1-1783895142-6c7964bfd4-2g2jp: 0 0
* bexhoma-pool-pgbouncer-1-1783895142-6c7964bfd4-gmznv: 0 0
* bexhoma-pool-pgbouncer-1-1783895142-6c7964bfd4-wvvrq: 0 0
* bexhoma-pool-pgbouncer-1-1783895142-6c7964bfd4-z8p9d: 0 0
* bexhoma-sut-pgbouncer-1-1783895142-9b64d588-9bvbk: 0 0
### Workflow
#### Actual
* DBMS PGBouncer-1 - Experiment 1 Client 1: ycsb (16 pods)
#### Planned
* DBMS PGBouncer-1 - Experiment 1 Client 1: ycsb (16 pods)
### Loading
#### Per Connection
| connection | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) | sf | Throughput [SF/h] |
|:---------------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|------:|--------------------:|
| PGBouncer-1-1-0-1-1 | 1.00 | 4.00 | 11264.00 | 16.00 | 0.00 | 3136.62 | 318815.00 | 1000000.00 | 1969.00 | 16.00 | 180.67 |
| PGBouncer-1-1-0-1-2 | 1.00 | 4.00 | 11264.00 | 16.00 | 0.00 | 3127.47 | 319747.00 | 1000000.00 | 1892.00 | 16.00 | 180.14 |
| PGBouncer-1-1-0-1-3 | 1.00 | 4.00 | 11264.00 | 16.00 | 0.00 | 3140.04 | 318467.00 | 1000000.00 | 2183.00 | 16.00 | 180.87 |
| PGBouncer-1-1-0-1-4 | 1.00 | 4.00 | 11264.00 | 16.00 | 0.00 | 3234.82 | 309136.00 | 1000000.00 | 1937.00 | 16.00 | 186.33 |
| PGBouncer-1-1-0-1-5 | 1.00 | 4.00 | 11264.00 | 16.00 | 0.00 | 3144.30 | 318036.00 | 1000000.00 | 2447.00 | 16.00 | 181.11 |
| PGBouncer-1-1-0-1-6 | 1.00 | 4.00 | 11264.00 | 16.00 | 0.00 | 3203.07 | 312200.00 | 1000000.00 | 2327.00 | 16.00 | 184.50 |
| PGBouncer-1-1-0-1-7 | 1.00 | 4.00 | 11264.00 | 16.00 | 0.00 | 3108.66 | 321682.00 | 1000000.00 | 2157.00 | 16.00 | 179.06 |
| PGBouncer-1-1-0-1-8 | 1.00 | 4.00 | 11264.00 | 16.00 | 0.00 | 3650.26 | 273953.00 | 1000000.00 | 2103.00 | 16.00 | 210.26 |
| PGBouncer-1-1-0-1-9 | 1.00 | 4.00 | 11264.00 | 16.00 | 0.00 | 3584.26 | 278998.00 | 1000000.00 | 1971.00 | 16.00 | 206.45 |
| PGBouncer-1-1-0-1-10 | 1.00 | 4.00 | 11264.00 | 16.00 | 0.00 | 3330.23 | 300280.00 | 1000000.00 | 1945.00 | 16.00 | 191.82 |
| PGBouncer-1-1-0-1-11 | 1.00 | 4.00 | 11264.00 | 16.00 | 0.00 | 3145.75 | 317889.00 | 1000000.00 | 2031.00 | 16.00 | 181.20 |
| PGBouncer-1-1-0-1-12 | 1.00 | 4.00 | 11264.00 | 16.00 | 0.00 | 3534.57 | 282920.00 | 1000000.00 | 2279.00 | 16.00 | 203.59 |
| PGBouncer-1-1-0-1-13 | 1.00 | 4.00 | 11264.00 | 16.00 | 0.00 | 3323.05 | 300928.00 | 1000000.00 | 2335.00 | 16.00 | 191.41 |
| PGBouncer-1-1-0-1-14 | 1.00 | 4.00 | 11264.00 | 16.00 | 0.00 | 3174.60 | 315000.00 | 1000000.00 | 2165.00 | 16.00 | 182.86 |
| PGBouncer-1-1-0-1-15 | 1.00 | 4.00 | 11264.00 | 16.00 | 0.00 | 3138.89 | 318584.00 | 1000000.00 | 2143.00 | 16.00 | 180.80 |
| PGBouncer-1-1-0-1-16 | 1.00 | 4.00 | 11264.00 | 16.00 | 0.00 | 3603.60 | 277500.00 | 1000000.00 | 2101.00 | 16.00 | 207.57 |
#### Per Run
| DBMS | experiment_run | threads | target | pod_count | exceptions | sf | Throughput [SF/h] | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) |
|:--------------|-----------------:|----------:|----------:|------------:|-------------:|------:|--------------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|
| PGBouncer-1-1 | 1.00 | 64.00 | 180224.00 | 16.00 | 0.00 | 16.00 | 179.06 | 52580.20 | 321682.00 | 16000000.00 | 2124.06 |
### Execution
#### Per Connection
| DBMS | phase | job | configuration | experiment_run | client | benchmark_run | child | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) |
|:---------------------|:----------------|:------------------|:----------------|-----------------:|---------:|----------------:|--------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|
| PGBouncer-1-1-1-1-1 | PGBouncer-1-1-1 | PGBouncer-1-1-1-1 | PGBouncer-1 | 1 | 1 | 1 | 1 | 8 | 11264 | 16 | 0 | 3234.30 | 309186.00 | 1000000 | 2755.00 |
| PGBouncer-1-1-1-1-2 | PGBouncer-1-1-1 | PGBouncer-1-1-1-1 | PGBouncer-1 | 1 | 1 | 1 | 2 | 8 | 11264 | 16 | 0 | 3239.39 | 308700.00 | 1000000 | 2789.00 |
| PGBouncer-1-1-1-1-3 | PGBouncer-1-1-1 | PGBouncer-1-1-1-1 | PGBouncer-1 | 1 | 1 | 1 | 3 | 8 | 11264 | 16 | 0 | 3238.58 | 308777.00 | 1000000 | 2829.00 |
| PGBouncer-1-1-1-1-4 | PGBouncer-1-1-1 | PGBouncer-1-1-1-1 | PGBouncer-1 | 1 | 1 | 1 | 4 | 8 | 11264 | 16 | 0 | 3240.76 | 308570.00 | 1000000 | 2713.00 |
| PGBouncer-1-1-1-1-5 | PGBouncer-1-1-1 | PGBouncer-1-1-1-1 | PGBouncer-1 | 1 | 1 | 1 | 5 | 8 | 11264 | 16 | 0 | 3235.26 | 309094.00 | 1000000 | 2819.00 |
| PGBouncer-1-1-1-1-6 | PGBouncer-1-1-1 | PGBouncer-1-1-1-1 | PGBouncer-1 | 1 | 1 | 1 | 6 | 8 | 11264 | 16 | 0 | 3276.39 | 305214.00 | 1000000 | 2647.00 |
| PGBouncer-1-1-1-1-7 | PGBouncer-1-1-1 | PGBouncer-1-1-1-1 | PGBouncer-1 | 1 | 1 | 1 | 7 | 8 | 11264 | 16 | 0 | 3231.05 | 309497.00 | 1000000 | 2807.00 |
| PGBouncer-1-1-1-1-8 | PGBouncer-1-1-1 | PGBouncer-1-1-1-1 | PGBouncer-1 | 1 | 1 | 1 | 8 | 8 | 11264 | 16 | 0 | 3248.01 | 307881.00 | 1000000 | 2691.00 |
| PGBouncer-1-1-1-1-9 | PGBouncer-1-1-1 | PGBouncer-1-1-1-1 | PGBouncer-1 | 1 | 1 | 1 | 9 | 8 | 11264 | 16 | 0 | 3255.82 | 307142.00 | 1000000 | 2739.00 |
| PGBouncer-1-1-1-1-10 | PGBouncer-1-1-1 | PGBouncer-1-1-1-1 | PGBouncer-1 | 1 | 1 | 1 | 10 | 8 | 11264 | 16 | 0 | 4336.85 | 230582.00 | 1000000 | 2303.00 |
| PGBouncer-1-1-1-1-11 | PGBouncer-1-1-1 | PGBouncer-1-1-1-1 | PGBouncer-1 | 1 | 1 | 1 | 11 | 8 | 11264 | 16 | 0 | 3300.35 | 302998.00 | 1000000 | 2643.00 |
| PGBouncer-1-1-1-1-12 | PGBouncer-1-1-1 | PGBouncer-1-1-1-1 | PGBouncer-1 | 1 | 1 | 1 | 12 | 8 | 11264 | 16 | 0 | 3236.37 | 308988.00 | 1000000 | 2751.00 |
| PGBouncer-1-1-1-1-13 | PGBouncer-1-1-1 | PGBouncer-1-1-1-1 | PGBouncer-1 | 1 | 1 | 1 | 13 | 8 | 11264 | 16 | 0 | 3290.21 | 303932.00 | 1000000 | 2631.00 |
| PGBouncer-1-1-1-1-14 | PGBouncer-1-1-1 | PGBouncer-1-1-1-1 | PGBouncer-1 | 1 | 1 | 1 | 14 | 8 | 11264 | 16 | 0 | 3241.52 | 308497.00 | 1000000 | 2731.00 |
| PGBouncer-1-1-1-1-15 | PGBouncer-1-1-1 | PGBouncer-1-1-1-1 | PGBouncer-1 | 1 | 1 | 1 | 15 | 8 | 11264 | 16 | 0 | 3250.44 | 307651.00 | 1000000 | 2725.00 |
| PGBouncer-1-1-1-1-16 | PGBouncer-1-1-1 | PGBouncer-1-1-1-1 | PGBouncer-1 | 1 | 1 | 1 | 16 | 8 | 11264 | 16 | 0 | 3242.00 | 308452.00 | 1000000 | 2751.00 |
#### Per Phase
| DBMS | phase | experiment_run | threads | target | benchmark_run | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) |
|:----------------|:----------------|-----------------:|----------:|---------:|----------------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|
| PGBouncer-1-1-1 | PGBouncer-1-1-1 | 1 | 128 | 180224 | 1 | 16 | 0 | 53097.30 | 309497.00 | 16000000 | 2829.00 |
### Monitoring
### Loading phase: component pool
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------|-------------:|----------:|---------------:|----------------------:|
| PGBouncer-1-1-1-1 | 1.42 | 0.01 | 0.08 | 0.08 |
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------|-------------:|----------:|---------------:|----------------------:|
| PGBouncer-1-1-1-1 | 1403.46 | 6.51 | 23.91 | 42.17 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------|-------------:|----------:|---------------:|----------------------:|
| PGBouncer-1-1-1-1 | 1352.86 | 7.56 | 0.11 | 0.11 |
### Execution phase: component pool
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------|-------------:|----------:|---------------:|----------------------:|
| PGBouncer-1-1-1-1 | 0.91 | 0.01 | 0.08 | 0.08 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------|-------------:|----------:|---------------:|----------------------:|
| PGBouncer-1-1-1-1 | 965.82 | 4.77 | 26.76 | 45.03 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------|-------------:|----------:|---------------:|----------------------:|
| PGBouncer-1-1-1-1 | 1188.78 | 6.15 | 0.10 | 0.11 |
### Application Metrics
#### Loading phase: component pool
| DBMS | PgBouncer Query Throughput [queries/s] | PgBouncer Waiting Clients [s] | PgBouncer Waiting Clients [clients] | PgBouncer Idle Connections [connections] | PgBouncer Pool Load Pressure [%] |
|:------------------|-----------------------------------------:|--------------------------------:|--------------------------------------:|-------------------------------------------:|-----------------------------------:|
| PGBouncer-1-1-1-1 | 53474.10 | 0.28 | 0.00 | 157.00 | 100.00 |
#### Loading phase: SUT deployment
| DBMS | Number of Idle Sessions | Number of Idle-in-transaction Sessions | Number of Idle-in-transaction Aborted Sessions | Number of Active Sessions | Number of Active Application Sessions |
|:------------------|--------------------------:|-----------------------------------------:|-------------------------------------------------:|----------------------------:|----------------------------------------:|
| PGBouncer-1-1-1-1 | 257.00 | 0.00 | 0.00 | 32.00 | 32.00 |
#### Execution phase: component pool
| DBMS | PgBouncer Query Throughput [queries/s] | PgBouncer Waiting Clients [s] | PgBouncer Waiting Clients [clients] | PgBouncer Idle Connections [connections] | PgBouncer Pool Load Pressure [%] |
|:------------------|-----------------------------------------:|--------------------------------:|--------------------------------------:|-------------------------------------------:|-----------------------------------:|
| PGBouncer-1-1-1-1 | 57082.92 | 0.11 | 0.00 | 56.00 | 100.00 |
#### Execution phase: SUT deployment
| DBMS | Number of Idle Sessions | Number of Idle-in-transaction Sessions | Number of Idle-in-transaction Aborted Sessions | Number of Active Sessions | Number of Active Application Sessions |
|:------------------|--------------------------:|-----------------------------------------:|-------------------------------------------------:|----------------------------:|----------------------------------------:|
| PGBouncer-1-1-1-1 | 257.00 | 0.00 | 0.00 | 5.00 | 5.00 |
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Loading phase: component pool contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component pool contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Execution Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: contains no FAILED column
Benchbase’s TPC-C
Example:
bexhoma benchbase \
-dbms PGBouncer \
-sf 16 \
-xsd 10 \
-xtb 1024 \
-xnbf 16 \
-nbp 1,2 \
-nbt 32 \
-xnpp 2 \
-xnpi 32 \
-xnpo 32 \
-xconn \
-m \
-ma \
-mc \
-ms $BEXHOMA_MS \
-tr \
-rss 50Gi \
-rnn $BEXHOMA_NODE_SUT -rnl $BEXHOMA_NODE_LOAD -rnb $BEXHOMA_NODE_BENCHMARK \
run &>$LOG_DIR/docs_benchbase_pgbouncer_appmetrics.log
Show docs_benchbase_pgbouncer_appmetrics.log
## Show Summary
### Workload
Benchbase Workload tpcc SF=16
* Type: benchbase
* Duration: 1760s
* Code: 1783896325
* Benchbase runs a TPC-C experiment.
* This experiment compares run time and resource consumption of Benchbase queries in different DBMS.
* Benchbase data is generated and loaded using several threads.
* Benchmark is 'tpcc'. Scaling factor is 16. Target is based on multiples of '1024'. Factors for benchmarking are [16]. There is a reconnect for each transaction. Benchmarking runs for 10 minutes.
* Experiment uses bexhoma version 0.10.5.
* System metrics are monitored by a cluster-wide installation.
* Application metrics are monitored by sidecar containers.
* Experiment is limited to DBMS ['PGBouncer'].
* Import is handled by 1 processes (pods).
* Loading is fixed to cl-worker19.
* Benchmarking is fixed to cl-worker19.
* SUT is fixed to cl-worker36.
* Database uses ephemeral storage of size 50Gi.
* Loading is tested with [1] threads, split into [1] pods.
* Benchmarking is tested with [32] threads, split into [1, 2] pods.
* Pooling is done with [2] pods having [32] inbound and [32] outbound connections in total.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* PGBouncer-1-1-1-1 uses docker image postgres:18.3
* RAM:2164173213696
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:1091129
* cpu_list:0-223
* args:['-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'max_wal_size=32GB', '-c', 'shared_buffers=64GB', '-c', 'max_connections=2048', '-c', 'autovacuum_max_workers=10', '-c', 'autovacuum_vacuum_cost_limit=3000', '-c', 'vacuum_cost_limit=1000', '-c', 'checkpoint_completion_target=0.9', '-c', 'cpu_tuple_cost=0.03', '-c', 'effective_cache_size=64GB', '-c', 'maintenance_work_mem=2GB', '-c', 'wal_buffers=1GB', '-c', 'work_mem=32GB', '-c', 'temp_buffers=4GB', '-c', 'autovacuum_work_mem=-1', '-c', 'max_stack_depth=7MB', '-c', 'max_files_per_process=4000', '-c', 'effective_io_concurrency=32', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'synchronous_commit=off', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_warning=0', '-c', 'autovacuum=off', '-c', 'max_locks_per_transaction=64', '-c', 'max_pred_locks_per_transaction=64', '-c', 'default_statistics_target=1000', '-c', 'random_page_cost=60']
* requests_cpu:4
* requests_memory:16Gi
* pool 0
* node:cl-worker25
* args:['-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'max_wal_size=32GB', '-c', 'shared_buffers=64GB', '-c', 'max_connections=2048', '-c', 'autovacuum_max_workers=10', '-c', 'autovacuum_vacuum_cost_limit=3000', '-c', 'vacuum_cost_limit=1000', '-c', 'checkpoint_completion_target=0.9', '-c', 'cpu_tuple_cost=0.03', '-c', 'effective_cache_size=64GB', '-c', 'maintenance_work_mem=2GB', '-c', 'wal_buffers=1GB', '-c', 'work_mem=32GB', '-c', 'temp_buffers=4GB', '-c', 'autovacuum_work_mem=-1', '-c', 'max_stack_depth=7MB', '-c', 'max_files_per_process=4000', '-c', 'effective_io_concurrency=32', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'synchronous_commit=off', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_warning=0', '-c', 'autovacuum=off', '-c', 'max_locks_per_transaction=64', '-c', 'max_pred_locks_per_transaction=64', '-c', 'default_statistics_target=1000', '-c', 'random_page_cost=60']
* pool 1
* node:cl-worker28
* args:['-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'max_wal_size=32GB', '-c', 'shared_buffers=64GB', '-c', 'max_connections=2048', '-c', 'autovacuum_max_workers=10', '-c', 'autovacuum_vacuum_cost_limit=3000', '-c', 'vacuum_cost_limit=1000', '-c', 'checkpoint_completion_target=0.9', '-c', 'cpu_tuple_cost=0.03', '-c', 'effective_cache_size=64GB', '-c', 'maintenance_work_mem=2GB', '-c', 'wal_buffers=1GB', '-c', 'work_mem=32GB', '-c', 'temp_buffers=4GB', '-c', 'autovacuum_work_mem=-1', '-c', 'max_stack_depth=7MB', '-c', 'max_files_per_process=4000', '-c', 'effective_io_concurrency=32', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'synchronous_commit=off', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_warning=0', '-c', 'autovacuum=off', '-c', 'max_locks_per_transaction=64', '-c', 'max_pred_locks_per_transaction=64', '-c', 'default_statistics_target=1000', '-c', 'random_page_cost=60']
* eval_parameters
* code:1783896325
* PGBouncer-1-1-2-1 uses docker image postgres:18.3
* RAM:2164173213696
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-111-generic
* node:cl-worker36
* disk:1091414
* cpu_list:0-223
* args:['-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'max_wal_size=32GB', '-c', 'shared_buffers=64GB', '-c', 'max_connections=2048', '-c', 'autovacuum_max_workers=10', '-c', 'autovacuum_vacuum_cost_limit=3000', '-c', 'vacuum_cost_limit=1000', '-c', 'checkpoint_completion_target=0.9', '-c', 'cpu_tuple_cost=0.03', '-c', 'effective_cache_size=64GB', '-c', 'maintenance_work_mem=2GB', '-c', 'wal_buffers=1GB', '-c', 'work_mem=32GB', '-c', 'temp_buffers=4GB', '-c', 'autovacuum_work_mem=-1', '-c', 'max_stack_depth=7MB', '-c', 'max_files_per_process=4000', '-c', 'effective_io_concurrency=32', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'synchronous_commit=off', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_warning=0', '-c', 'autovacuum=off', '-c', 'max_locks_per_transaction=64', '-c', 'max_pred_locks_per_transaction=64', '-c', 'default_statistics_target=1000', '-c', 'random_page_cost=60']
* requests_cpu:4
* requests_memory:16Gi
* pool 0
* node:cl-worker25
* args:['-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'max_wal_size=32GB', '-c', 'shared_buffers=64GB', '-c', 'max_connections=2048', '-c', 'autovacuum_max_workers=10', '-c', 'autovacuum_vacuum_cost_limit=3000', '-c', 'vacuum_cost_limit=1000', '-c', 'checkpoint_completion_target=0.9', '-c', 'cpu_tuple_cost=0.03', '-c', 'effective_cache_size=64GB', '-c', 'maintenance_work_mem=2GB', '-c', 'wal_buffers=1GB', '-c', 'work_mem=32GB', '-c', 'temp_buffers=4GB', '-c', 'autovacuum_work_mem=-1', '-c', 'max_stack_depth=7MB', '-c', 'max_files_per_process=4000', '-c', 'effective_io_concurrency=32', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'synchronous_commit=off', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_warning=0', '-c', 'autovacuum=off', '-c', 'max_locks_per_transaction=64', '-c', 'max_pred_locks_per_transaction=64', '-c', 'default_statistics_target=1000', '-c', 'random_page_cost=60']
* pool 1
* node:cl-worker28
* args:['-c', 'max_worker_processes=64', '-c', 'max_parallel_workers=64', '-c', 'max_parallel_workers_per_gather=64', '-c', 'max_parallel_maintenance_workers=64', '-c', 'max_wal_size=32GB', '-c', 'shared_buffers=64GB', '-c', 'max_connections=2048', '-c', 'autovacuum_max_workers=10', '-c', 'autovacuum_vacuum_cost_limit=3000', '-c', 'vacuum_cost_limit=1000', '-c', 'checkpoint_completion_target=0.9', '-c', 'cpu_tuple_cost=0.03', '-c', 'effective_cache_size=64GB', '-c', 'maintenance_work_mem=2GB', '-c', 'wal_buffers=1GB', '-c', 'work_mem=32GB', '-c', 'temp_buffers=4GB', '-c', 'autovacuum_work_mem=-1', '-c', 'max_stack_depth=7MB', '-c', 'max_files_per_process=4000', '-c', 'effective_io_concurrency=32', '-c', 'wal_level=minimal', '-c', 'max_wal_senders=0', '-c', 'synchronous_commit=off', '-c', 'checkpoint_timeout=1h', '-c', 'checkpoint_warning=0', '-c', 'autovacuum=off', '-c', 'max_locks_per_transaction=64', '-c', 'max_pred_locks_per_transaction=64', '-c', 'default_statistics_target=1000', '-c', 'random_page_cost=60']
* eval_parameters
* code:1783896325
### SUT Container Restarts
* bexhoma-pool-pgbouncer-1-1783896325-5f88b97b5c-p7mrw: 0 0
* bexhoma-pool-pgbouncer-1-1783896325-5f88b97b5c-xc9cz: 0 0
* bexhoma-sut-pgbouncer-1-1783896325-56bf9dc46f-tsxhm: 0 0
### Workflow
#### Actual
* DBMS PGBouncer-1 - Experiment 1 Client 1: benchbase (1 pods)
* DBMS PGBouncer-1 - Experiment 1 Client 2: benchbase (2 pods)
#### Planned
* DBMS PGBouncer-1 - Experiment 1 Client 1: benchbase (1 pods)
* DBMS PGBouncer-1 - Experiment 1 Client 2: benchbase (2 pods)
### Loading
#### Per Run
| | experiment_run | SF | time_load | time_preload | time_generate | time_ingest | time_postload | loading_pods | terminals | tenant_id | type_tenants | num_tenants | vol_tenants | Throughput [SF/h] |
|:--------------|-----------------:|------:|------------:|---------------:|----------------:|--------------:|----------------:|---------------:|------------:|:------------|:---------------|--------------:|:--------------|--------------------:|
| PGBouncer-1-1 | 1 | 16.00 | 336.00 | 1.00 | 0.00 | 148.00 | 187.00 | 1 | 1 | | None | 0 | False | 171.43 |
### Execution
#### Per Connection
| DBMS | phase | job | experiment_run | terminals | target | client | benchmark_run | child | tenant_id | time | num_errors | Throughput (requests/second) | Goodput (requests/second) | efficiency | Latency Distribution.95th Percentile Latency (microseconds) | Latency Distribution.Average Latency (microseconds) |
|:--------------------|:----------------|:------------------|-----------------:|------------:|---------:|---------:|----------------:|--------:|------------:|-------:|-------------:|-------------------------------:|----------------------------:|-------------:|--------------------------------------------------------------:|------------------------------------------------------:|
| PGBouncer-1-1-1-1-1 | PGBouncer-1-1-1 | PGBouncer-1-1-1-1 | 1 | 32 | 16384 | 1 | 1 | 1 | -1 | 600.00 | 2 | 1350.97 | 463.77 | 0.00 | 48407.00 | 23681.00 |
| PGBouncer-1-1-2-1-1 | PGBouncer-1-1-2 | PGBouncer-1-1-2-1 | 1 | 16 | 8192 | 2 | 1 | 1 | -1 | 600.00 | 1 | 568.26 | 442.12 | 0.00 | 60777.00 | 28149.00 |
| PGBouncer-1-1-2-1-2 | PGBouncer-1-1-2 | PGBouncer-1-1-2-1 | 1 | 16 | 8192 | 2 | 1 | 2 | -1 | 600.00 | 2 | 573.52 | 441.96 | 0.00 | 60695.00 | 27891.00 |
#### Per Phase
| DBMS | phase | experiment_run | terminals | target | benchmark_run | pod_count | tenant_id | time | num_errors | Throughput (requests/second) | Goodput (requests/second) | efficiency | Latency Distribution.95th Percentile Latency (microseconds) | Latency Distribution.Average Latency (microseconds) |
|:----------------|:----------------|-----------------:|------------:|---------:|----------------:|------------:|------------:|-------:|-------------:|-------------------------------:|----------------------------:|-------------:|--------------------------------------------------------------:|------------------------------------------------------:|
| PGBouncer-1-1-1 | PGBouncer-1-1-1 | 1 | 32 | 16384 | 1 | 1 | -1 | 600.00 | 2 | 1350.97 | 463.77 | 0.00 | 48407.00 | 23681.00 |
| PGBouncer-1-1-2 | PGBouncer-1-1-2 | 1 | 32 | 16384 | 1 | 2 | -1 | 600.00 | 3 | 1141.78 | 884.08 | 0.00 | 60777.00 | 28020.00 |
### Monitoring
### Loading phase: component pool
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------|-------------:|----------:|---------------:|----------------------:|
| PGBouncer-1-1-1-1 | 0.52 | 0.00 | 0.04 | 0.04 |
| PGBouncer-1-1-2-1 | 0.52 | 0.00 | 0.04 | 0.04 |
### Loading phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------|-------------:|----------:|---------------:|----------------------:|
| PGBouncer-1-1-1-1 | 188.91 | 2.62 | 4.71 | 6.35 |
| PGBouncer-1-1-2-1 | 188.91 | 2.62 | 4.71 | 6.35 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------|-------------:|----------:|---------------:|----------------------:|
| PGBouncer-1-1-1-1 | 1907.89 | 15.32 | 0.25 | 0.25 |
| PGBouncer-1-1-2-1 | 1907.89 | 15.32 | 0.25 | 0.25 |
### Execution phase: component pool
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------|-------------:|----------:|---------------:|----------------------:|
| PGBouncer-1-1-1-1 | 1.61 | 0.00 | 0.05 | 0.05 |
| PGBouncer-1-1-2-1 | 1.71 | 0.00 | 0.05 | 0.05 |
### Execution phase: SUT deployment
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------|-------------:|----------:|---------------:|----------------------:|
| PGBouncer-1-1-1-1 | 832.83 | 3.13 | 5.08 | 6.97 |
| PGBouncer-1-1-2-1 | 1529.59 | 3.68 | 5.58 | 7.91 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------|-------------:|----------:|---------------:|----------------------:|
| PGBouncer-1-1-1-1 | 11840.07 | 30.18 | 0.47 | 0.47 |
| PGBouncer-1-1-2-1 | 11840.07 | 39.09 | 0.47 | 0.47 |
### Application Metrics
#### Loading phase: component pool
| DBMS | PgBouncer Query Throughput [queries/s] | PgBouncer Waiting Clients [s] | PgBouncer Waiting Clients [clients] | PgBouncer Idle Connections [connections] | PgBouncer Pool Load Pressure [%] |
|:------------------|-----------------------------------------:|--------------------------------:|--------------------------------------:|-------------------------------------------:|-----------------------------------:|
| PGBouncer-1-1-1-1 | 223.21 | 0.16 | 0.00 | 48.00 | 100.00 |
| PGBouncer-1-1-2-1 | 223.21 | 0.16 | 0.00 | 48.00 | 100.00 |
#### Loading phase: SUT deployment
| DBMS | Number of Idle Sessions | Number of Idle-in-transaction Sessions | Number of Idle-in-transaction Aborted Sessions | Number of Active Sessions | Number of Active Application Sessions |
|:------------------|--------------------------:|-----------------------------------------:|-------------------------------------------------:|----------------------------:|----------------------------------------:|
| PGBouncer-1-1-1-1 | 64.00 | 0.00 | 0.00 | 5.00 | 5.00 |
| PGBouncer-1-1-2-1 | 64.00 | 0.00 | 0.00 | 5.00 | 5.00 |
#### Execution phase: component pool
| DBMS | PgBouncer Query Throughput [queries/s] | PgBouncer Waiting Clients [s] | PgBouncer Waiting Clients [clients] | PgBouncer Idle Connections [connections] | PgBouncer Pool Load Pressure [%] |
|:------------------|-----------------------------------------:|--------------------------------:|--------------------------------------:|-------------------------------------------:|-----------------------------------:|
| PGBouncer-1-1-1-1 | 9627.75 | 0.50 | 0.00 | 63.00 | 100.00 |
| PGBouncer-1-1-2-1 | 18545.54 | 0.15 | 0.00 | 64.00 | 100.00 |
#### Execution phase: SUT deployment
| DBMS | Number of Idle Sessions | Number of Idle-in-transaction Sessions | Number of Idle-in-transaction Aborted Sessions | Number of Active Sessions | Number of Active Application Sessions |
|:------------------|--------------------------:|-----------------------------------------:|-------------------------------------------------:|----------------------------:|----------------------------------------:|
| PGBouncer-1-1-1-1 | 65.00 | 27.00 | 0.00 | 7.00 | 7.00 |
| PGBouncer-1-1-2-1 | 65.00 | 28.00 | 0.00 | 9.00 | 9.00 |
### Tests
* TEST passed: No SUT container restarts
* TEST passed: Loading phase: component pool contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component pool contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: SUT deployment contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Throughput (requests/second) contains no 0 or NaN
* TEST passed: Workflow as planned
YugabyteDB
Make sure to have YugabyteDB installed externally, because bexhoma does not manage it.
YCSB
Example:
bexhoma ycsb \
-dbms YugabyteDB \
-sf 1 \
-xwl a \
-xtb 16384 \
-xnbf 4 \
-xnlf 4 \
-nc 1 \
-ne 1 \
-nlp 8 \
-nlt 64 \
-nbp 1 \
-nbt 64 \
-xop 10 \
-m \
-ma \
-mc \
-ms $BEXHOMA_MS \
-tr \
run &>$LOG_DIR/docs_ycsb_yugabytedb_appmetrics.log
Show docs_ycsb_yugabytedb_appmetrics.log
## Show Summary
### Workload
YCSB SF=1
* Type: ycsb
* Duration: 950s
* Code: 1773432301
* YCSB driver runs the experiment.
* This experiment compares run time and resource consumption of YCSB queries.
* Workload is 'A'.
* Number of rows to insert is 1000000.
* Ordering of inserts is hashed.
* Number of operations is 10000000.
* Batch size is ''.
* Target is based on multiples of '16384'.
* Factors for loading are [4].
* Factors for benchmarking are [4].
* Experiment uses bexhoma version 0.9.4.
* System metrics are monitored by a cluster-wide installation.
* Application metrics are monitored by sidecar containers.
* Experiment is limited to DBMS ['YugabyteDB'].
* Import is handled by 8 processes (pods).
* Loading is tested with [64] threads, split into [8] pods.
* Benchmarking is tested with [64] threads, split into [1] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* YugabyteDB-64-8-65536-1 uses docker image postgres:15.0
* RAM:2164173209600
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-90-generic
* node:cl-worker36
* disk:1294649
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1773432301
### Loading
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [INSERT].Return=OK | [INSERT].99thPercentileLatency(us) |
|:----------------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|---------------------:|-------------------------------------:|
| YugabyteDB-64-8-65536 | 1 | 64 | 65536 | 8 | 0 | 19489.3 | 52544 | 1e+06 | 8961 |
### Execution
| DBMS | experiment_run | threads | target | pod_count | exceptions | [OVERALL].Throughput(ops/sec) | [OVERALL].RunTime(ms) | [READ].Return=OK | [READ].99thPercentileLatency(us) | [UPDATE].Return=OK | [UPDATE].99thPercentileLatency(us) |
|:------------------------|-----------------:|----------:|---------:|------------:|-------------:|--------------------------------:|------------------------:|-------------------:|-----------------------------------:|---------------------:|-------------------------------------:|
| YugabyteDB-64-8-65536-1 | 1 | 64 | 65536 | 1 | 0 | 34422.5 | 290508 | 5.00119e+06 | 7643 | 4.99881e+06 | 43583 |
### Workflow
#### Actual
* DBMS YugabyteDB-64-8-65536 - Pods [[1]]
#### Planned
* DBMS YugabyteDB-64-8-65536 - Pods [[1]]
### Monitoring
### Loading phase: component yb-tserver
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------------|-------------:|----------:|---------------:|----------------------:|
| YugabyteDB-64-8-65536-1 | 1234.82 | 13.47 | 5.6 | 13.6 |
### Loading phase: component yb-master
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------------|-------------:|----------:|---------------:|----------------------:|
| YugabyteDB-64-8-65536-1 | 6.87 | 0.04 | 0.24 | 0.26 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------------|-------------:|----------:|---------------:|----------------------:|
| YugabyteDB-64-8-65536-1 | 38.93 | 0 | 0.11 | 0.11 |
### Execution phase: component yb-tserver
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------------|-------------:|----------:|---------------:|----------------------:|
| YugabyteDB-64-8-65536-1 | 10378.5 | 39.73 | 9.51 | 20.76 |
### Execution phase: component yb-master
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------------|-------------:|----------:|---------------:|----------------------:|
| YugabyteDB-64-8-65536-1 | 10.69 | 0.06 | 0.26 | 0.29 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:------------------------|-------------:|----------:|---------------:|----------------------:|
| YugabyteDB-64-8-65536-1 | 319.68 | 1.25 | 0.14 | 0.14 |
### Application Metrics
#### Loading phase: component yb-tserver
| DBMS | YSQL Query Throughput [ops/s] | Tablet Read Latency [ms] | Tablet Write Latency [ms] | Active YSQL Connections | TServer Follower Lag [ms] |
|:------------------------|--------------------------------:|---------------------------:|----------------------------:|--------------------------:|----------------------------:|
| YugabyteDB-64-8-65536-1 | 3754.31 | 0.42 | 7.06 | 0 | 617 |
#### Loading phase: component yb-master
| DBMS | YB-Master RPC Throughput [ops/s] | YB-Master Avg RPC Latency [ms] | Table Metadata Operations [ops/s] | YB-Master Clock Error | Tablet Location Requests [ops/s] |
|:------------------------|-----------------------------------:|---------------------------------:|------------------------------------:|------------------------:|-----------------------------------:|
| YugabyteDB-64-8-65536-1 | 0.02 | 0.54 | 0 | 5.017e+08 | 0.02 |
#### Execution phase: component yb-tserver
| DBMS | YSQL Query Throughput [ops/s] | Tablet Read Latency [ms] | Tablet Write Latency [ms] | Active YSQL Connections | TServer Follower Lag [ms] |
|:------------------------|--------------------------------:|---------------------------:|----------------------------:|--------------------------:|----------------------------:|
| YugabyteDB-64-8-65536-1 | 31676.4 | 0.67 | 2.47 | 0 | 585 |
#### Execution phase: component yb-master
| DBMS | YB-Master RPC Throughput [ops/s] | YB-Master Avg RPC Latency [ms] | Table Metadata Operations [ops/s] | YB-Master Clock Error | Tablet Location Requests [ops/s] |
|:------------------------|-----------------------------------:|---------------------------------:|------------------------------------:|------------------------:|-----------------------------------:|
| YugabyteDB-64-8-65536-1 | 0.01 | 0.52 | 0 | 5.01948e+08 | 0.01 |
### Tests
* TEST passed: Loading Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Execution Phase: [OVERALL].Throughput(ops/sec) contains no 0 or NaN
* TEST passed: Loading phase: component yb-tserver contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component yb-master contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component yb-tserver contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component yb-master contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Workflow as planned
* TEST passed: Execution Phase: contains no FAILED column
Benchbase’s TPC-C
Example:
bexhoma benchbase \
-dbms YugabyteDB \
-sf 16 \
-xsd 5 \
-xtb 1024 \
-xnbf 16 \
-nbp 1,2 \
-nbt 16 \
-m \
-ma \
-mc \
-ms $BEXHOMA_MS \
-tr \
run &>$LOG_DIR/docs_benchbase_yugabytedb_appmetrics.log
Show docs_benchbase_yugabytedb_appmetrics.log
## Show Summary
### Workload
Benchbase Workload tpcc SF=16
* Type: benchbase
* Duration: 1265s
* Code: 1773430865
* Benchbase runs a TPC-C experiment.
* This experiment compares run time and resource consumption of Benchbase queries in different DBMS.
* Benchbase data is generated and loaded using several threads.
* Benchmark is 'tpcc'. Scaling factor is 16. Target is based on multiples of '1024'. Factors for benchmarking are [16]. Benchmarking runs for 5 minutes.
* Experiment uses bexhoma version 0.9.4.
* System metrics are monitored by a cluster-wide installation.
* Application metrics are monitored by sidecar containers.
* Experiment is limited to DBMS ['YugabyteDB'].
* Import is handled by 1 processes (pods).
* Loading is tested with [1] threads, split into [1] pods.
* Benchmarking is tested with [16] threads, split into [1, 2] pods.
* Benchmarking is run as [1] times the number of benchmarking pods.
* Experiment is run once.
### Connections
* YugabyteDB-1-1-1024-1 uses docker image postgres:15.0
* RAM:2164173209600
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-90-generic
* node:cl-worker36
* disk:1262462
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1773430865
* YugabyteDB-1-1-1024-2 uses docker image postgres:15.0
* RAM:2164173209600
* CPU:INTEL(R) XEON(R) PLATINUM 8570
* Cores:224
* host:6.8.0-90-generic
* node:cl-worker36
* disk:1263692
* cpu_list:0-223
* requests_cpu:4
* requests_memory:16Gi
* eval_parameters
* code:1773430865
### Execution
#### Per Pod
| DBMS | experiment_run | terminals | target | client | child | time | num_errors | Throughput (requests/second) | Goodput (requests/second) | efficiency | Latency Distribution.95th Percentile Latency (microseconds) | Latency Distribution.Average Latency (microseconds) |
|:------------------------|-----------------:|------------:|---------:|---------:|--------:|-------:|-------------:|-------------------------------:|----------------------------:|-------------:|--------------------------------------------------------------:|------------------------------------------------------:|
| YugabyteDB-1-1-1024-1-1 | 1 | 16 | 16384 | 1 | 1 | 300 | 0 | 475.947 | 473.847 | 0 | 81863 | 33609 |
| YugabyteDB-1-1-1024-2-1 | 1 | 8 | 8192 | 2 | 1 | 300 | 0 | 225.253 | 224.247 | 0 | 94011 | 35505 |
| YugabyteDB-1-1-1024-2-2 | 1 | 8 | 8192 | 2 | 2 | 300 | 0 | 214.683 | 213.59 | 0 | 97092 | 37254 |
#### Aggregated Parallel
| DBMS | experiment_run | terminals | target | pod_count | time | num_errors | Throughput (requests/second) | Goodput (requests/second) | efficiency | Latency Distribution.95th Percentile Latency (microseconds) | Latency Distribution.Average Latency (microseconds) |
|:----------------------|-----------------:|------------:|---------:|------------:|-------:|-------------:|-------------------------------:|----------------------------:|-------------:|--------------------------------------------------------------:|------------------------------------------------------:|
| YugabyteDB-1-1-1024-1 | 1 | 16 | 16384 | 1 | 300 | 0 | 475.95 | 473.85 | 0 | 81863 | 33609 |
| YugabyteDB-1-1-1024-2 | 1 | 16 | 16384 | 2 | 300 | 0 | 439.94 | 437.84 | 0 | 97092 | 36379.5 |
### Workflow
#### Actual
* DBMS YugabyteDB-1-1-1024 - Pods [[1, 2]]
#### Planned
* DBMS YugabyteDB-1-1-1024 - Pods [[1, 2]]
### Loading
| DBMS | time_load | terminals | pods | Throughput [SF/h] |
|:----------------------|------------:|------------:|-------:|--------------------:|
| YugabyteDB-1-1-1024-1 | 102 | 1 | 1 | 564.706 |
| YugabyteDB-1-1-1024-2 | 102 | 1 | 2 | 564.706 |
### Monitoring
### Loading phase: component yb-tserver
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------------|-------------:|----------:|---------------:|----------------------:|
| YugabyteDB-1-1-1024-1 | 2010.86 | 22.68 | 7.39 | 16.15 |
| YugabyteDB-1-1-1024-2 | 2508.42 | 22.68 | 7.39 | 16.15 |
### Loading phase: component yb-master
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------------|-------------:|----------:|---------------:|----------------------:|
| YugabyteDB-1-1-1024-1 | 17.84 | 0.21 | 0.34 | 0.38 |
| YugabyteDB-1-1-1024-2 | 24.18 | 0.21 | 0.36 | 0.4 |
### Loading phase: component loader
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------------|-------------:|----------:|---------------:|----------------------:|
| YugabyteDB-1-1-1024-1 | 75.68 | 0 | 0.25 | 0.26 |
| YugabyteDB-1-1-1024-2 | 75.68 | 0 | 0.25 | 0.26 |
### Execution phase: component yb-tserver
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------------|-------------:|----------:|---------------:|----------------------:|
| YugabyteDB-1-1-1024-1 | 10821.6 | 39.06 | 10.15 | 20.45 |
| YugabyteDB-1-1-1024-2 | 9532.47 | 36.81 | 11.15 | 22.77 |
### Execution phase: component yb-master
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------------|-------------:|----------:|---------------:|----------------------:|
| YugabyteDB-1-1-1024-1 | 13.01 | 0.07 | 0.39 | 0.43 |
| YugabyteDB-1-1-1024-2 | 17.1 | 0.08 | 0.45 | 0.49 |
### Execution phase: component benchmarker
| DBMS | CPU [CPUs] | Max CPU | Max RAM [Gb] | Max RAM Cached [Gb] |
|:----------------------|-------------:|----------:|---------------:|----------------------:|
| YugabyteDB-1-1-1024-1 | 102.58 | 0.52 | 0.3 | 0.3 |
| YugabyteDB-1-1-1024-2 | 92.67 | 0.98 | 0.3 | 0.3 |
### Application Metrics
#### Loading phase: component yb-tserver
| DBMS | YSQL Query Throughput [ops/s] | Tablet Read Latency [ms] | Tablet Write Latency [ms] | Active YSQL Connections | TServer Follower Lag [ms] |
|:----------------------|--------------------------------:|---------------------------:|----------------------------:|--------------------------:|----------------------------:|
| YugabyteDB-1-1-1024-1 | 243.83 | 0.44 | 26.87 | 0 | 856 |
| YugabyteDB-1-1-1024-2 | 3048.64 | 0.44 | 26.87 | 0 | 856 |
#### Loading phase: component yb-master
| DBMS | YB-Master RPC Throughput [ops/s] | YB-Master Avg RPC Latency [ms] | Table Metadata Operations [ops/s] | YB-Master Clock Error | Tablet Location Requests [ops/s] |
|:----------------------|-----------------------------------:|---------------------------------:|------------------------------------:|------------------------:|-----------------------------------:|
| YugabyteDB-1-1-1024-1 | 3.05 | 22.47 | 0.04 | 5.00068e+08 | 3.01 |
| YugabyteDB-1-1-1024-2 | 3.05 | 22.47 | 0.04 | 5.00234e+08 | 3.01 |
#### Execution phase: component yb-tserver
| DBMS | YSQL Query Throughput [ops/s] | Tablet Read Latency [ms] | Tablet Write Latency [ms] | Active YSQL Connections | TServer Follower Lag [ms] |
|:----------------------|--------------------------------:|---------------------------:|----------------------------:|--------------------------:|----------------------------:|
| YugabyteDB-1-1-1024-1 | 16629.5 | 0.28 | 11.22 | 0 | 599 |
| YugabyteDB-1-1-1024-2 | 7310.32 | 0.31 | 1.19 | 0 | 612 |
#### Execution phase: component yb-master
| DBMS | YB-Master RPC Throughput [ops/s] | YB-Master Avg RPC Latency [ms] | Table Metadata Operations [ops/s] | YB-Master Clock Error | Tablet Location Requests [ops/s] |
|:----------------------|-----------------------------------:|---------------------------------:|------------------------------------:|------------------------:|-----------------------------------:|
| YugabyteDB-1-1-1024-1 | 3.02 | 0.44 | 0.03 | 5.00262e+08 | 2.99 |
| YugabyteDB-1-1-1024-2 | 1.11 | 0.32 | 0 | 5.01112e+08 | 1.11 |
### Tests
* TEST passed: Throughput (requests/second) contains no 0 or NaN
* TEST passed: Loading phase: component yb-tserver contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component yb-master contains no 0 or NaN in CPU [CPUs]
* TEST passed: Loading phase: component loader contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component yb-tserver contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component yb-master contains no 0 or NaN in CPU [CPUs]
* TEST passed: Execution phase: component benchmarker contains no 0 or NaN in CPU [CPUs]
* TEST passed: Workflow as planned