... | ... | @@ -29,4 +29,314 @@ |
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
```
|
|
|
|
|
|
We can see that minimum latency is much lower for postgresT server across all calls. And average time is much better for get_transaction and get_account_history (at least on second run). Max times are also lower except for outlier of get_ops_in_block. |
|
|
\ No newline at end of file |
|
|
We can see that minimum latency is much lower for postgresT server across all calls. And average time is much better for get_transaction and get_account_history (at least on second run). Max times are also lower except for outlier of get_ops_in_block.
|
|
|
```
|
|
|
steem-11 testing 1GB huge pages (reindexed 36298s, 3hr55m for indexes)
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
| Endpoint | Max [ms] | Min [ms] | Average [ms] | Median [ms] |
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
| enum_virtual_ops | 3677 | 18 | 74 | 61 |
|
|
|
| get_transaction | 1049 | 12 | 24 | 18 |
|
|
|
| get_account_history | 1967 | 24 | 288 | 93 |
|
|
|
| get_ops_in_block | 3727 | 14 | 263 | 80 |
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
|
|
|
steem-11 dockerized HAF (reindexed 40637s
|
|
|
|
|
|
./haf/scripts/run_hived_img.sh --name=haf-full --data-dir=/blocklog/datadir-haf registry.gitlab.syncad.com/hive/haf/instance:f43d745c70c323a21ffbeff37186acd688953f04 --replay
|
|
|
|
|
|
|
|
|
|
|
|
cluster didn't help
|
|
|
|
|
|
SHED14 CONFIGURATION: 4xtpool lz4 noatime shared_buffers=128MB default arc (64GB) (READ)
|
|
|
postgres@shed14:~$ pgbench -S -j 32 -c 32 -M prepared -T 20 pgbench
|
|
|
pgbench (14.2 (Ubuntu 14.2-1ubuntu1))
|
|
|
starting vacuum...end.
|
|
|
transaction type: <builtin: select only>
|
|
|
scaling factor: 100
|
|
|
query mode: prepared
|
|
|
number of clients: 32
|
|
|
number of threads: 32
|
|
|
duration: 20 s
|
|
|
number of transactions actually processed: 12048387
|
|
|
latency average = 0.053 ms
|
|
|
initial connection time = 6.957 ms
|
|
|
tps = 602579.364950 (without initial connection time)
|
|
|
|
|
|
postgres@shed14:~$ pgbench -j 32 -c 32 -M prepared -T 10 pgbench (WRITING ALSO)
|
|
|
pgbench (14.2 (Ubuntu 14.2-1ubuntu1))
|
|
|
starting vacuum...end.
|
|
|
transaction type: <builtin: TPC-B (sort of)>
|
|
|
scaling factor: 100
|
|
|
query mode: prepared
|
|
|
number of clients: 32
|
|
|
number of threads: 32
|
|
|
duration: 10 s
|
|
|
number of transactions actually processed: 564471
|
|
|
latency average = 0.567 ms
|
|
|
initial connection time = 7.313 ms
|
|
|
tps = 56439.610464 (without initial connection time)
|
|
|
|
|
|
SHED14 CONFIGURATION: 4xtpool lz4 noatime shared_buffers=32GB default arc (64GB) (READ)
|
|
|
postgres@shed14:~$ pgbench -S -j 32 -c 32 -M prepared -T 20 pgbench
|
|
|
pgbench (14.2 (Ubuntu 14.2-1ubuntu1))
|
|
|
starting vacuum...end.
|
|
|
transaction type: <builtin: select only>
|
|
|
scaling factor: 100
|
|
|
query mode: prepared
|
|
|
number of clients: 32
|
|
|
number of threads: 32
|
|
|
duration: 20 s
|
|
|
number of transactions actually processed: 22057472
|
|
|
latency average = 0.029 ms
|
|
|
initial connection time = 8.032 ms
|
|
|
tps = 1,103,241.806953 (without initial connection time)
|
|
|
|
|
|
postgres@shed14:~$ pgbench -j 32 -c 32 -M prepared -T 10 pgbench
|
|
|
pgbench (14.2 (Ubuntu 14.2-1ubuntu1))
|
|
|
starting vacuum...end.
|
|
|
transaction type: <builtin: TPC-B (sort of)>
|
|
|
scaling factor: 100
|
|
|
query mode: prepared
|
|
|
number of clients: 32
|
|
|
number of threads: 32
|
|
|
duration: 10 s
|
|
|
number of transactions actually processed: 1007712
|
|
|
latency average = 0.318 ms
|
|
|
initial connection time = 7.165 ms
|
|
|
tps = 100778.556835 (without initial connection time)
|
|
|
|
|
|
SHED14 CONFIGURATION: 4xtpool lz4 noatime shared_buffers=64GB default arc (64GB)
|
|
|
postgres@shed14:~$ pgbench -S -j 32 -c 32 -M prepared -T 20 pgbench
|
|
|
pgbench (14.2 (Ubuntu 14.2-1ubuntu1))
|
|
|
starting vacuum...end.
|
|
|
transaction type: <builtin: select only>
|
|
|
scaling factor: 100
|
|
|
query mode: prepared
|
|
|
number of clients: 32
|
|
|
number of threads: 32
|
|
|
duration: 20 s
|
|
|
number of transactions actually processed: 21282999
|
|
|
latency average = 0.030 ms
|
|
|
initial connection time = 7.395 ms
|
|
|
tps = 1064477.542964 (without initial connection time)
|
|
|
postgres@shed14:~$ pgbench -j 32 -c 32 -M prepared -T 10 pgbench
|
|
|
pgbench (14.2 (Ubuntu 14.2-1ubuntu1))
|
|
|
starting vacuum...end.
|
|
|
transaction type: <builtin: TPC-B (sort of)>
|
|
|
scaling factor: 100
|
|
|
query mode: prepared
|
|
|
number of clients: 32
|
|
|
number of threads: 32
|
|
|
duration: 10 s
|
|
|
number of transactions actually processed: 1000628
|
|
|
latency average = 0.320 ms
|
|
|
initial connection time = 7.439 ms
|
|
|
tps = 99939.923864 (without initial connection time)
|
|
|
|
|
|
SHED14 CONFIGURATION: 4xtpool lz4 noatime shared_buffers=4GB default arc (64GB)
|
|
|
postgres@shed14:~$ pgbench -S -j 32 -c 32 -M prepared -T 20 pgbench
|
|
|
pgbench (14.2 (Ubuntu 14.2-1ubuntu1))
|
|
|
starting vacuum...end.
|
|
|
transaction type: <builtin: select only>
|
|
|
scaling factor: 100
|
|
|
query mode: prepared
|
|
|
number of clients: 32
|
|
|
number of threads: 32
|
|
|
duration: 20 s
|
|
|
number of transactions actually processed: 21712216
|
|
|
latency average = 0.029 ms
|
|
|
initial connection time = 7.372 ms
|
|
|
tps = 1085925.772770 (without initial connection time)
|
|
|
postgres@shed14:~$ pgbench -j 32 -c 32 -M prepared -T 10 pgbench
|
|
|
pgbench (14.2 (Ubuntu 14.2-1ubuntu1))
|
|
|
starting vacuum...end.
|
|
|
transaction type: <builtin: TPC-B (sort of)>
|
|
|
scaling factor: 100
|
|
|
query mode: prepared
|
|
|
number of clients: 32
|
|
|
number of threads: 32
|
|
|
duration: 10 s
|
|
|
number of transactions actually processed: 1000634
|
|
|
latency average = 0.320 ms
|
|
|
initial connection time = 6.853 ms
|
|
|
tps = 99978.408355 (without initial connection time)
|
|
|
summary = 20004 in 00:04:59 = 66.9/s Avg: 147 Min: 15 Max: 3925
|
|
|
|
|
|
steem-10 38143=10.60 8hr18m uncompressed WAL
|
|
|
steem-11 35543=9.87 4hr16m compressed WAL
|
|
|
|
|
|
steem-10 compressed WAL, max_wal_size=4GB, shared_buffers=4GB
|
|
|
|
|
|
syncad@steem-10:/hafpool/hafah$ ./tests/performance_test.py --postgres postgresql://hive:dandan@127.0.0.1:5432/haf_block_log
|
|
|
2022-05-01 00:15:39,103 - performance - INFO - postgres connection string in JMETER: jdbc:postgresql://127.0.0.1:5432/haf_block_log?user=hive&password=dandan
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
| Endpoint | Max [ms] | Min [ms] | Average [ms] | Median [ms] |
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
| enum_virtual_ops | 4374 | 20 | 72 | 44 |
|
|
|
| get_transaction | 1051 | 14 | 25 | 22 |
|
|
|
| get_account_history | 2085 | 24 | 396 | 260 |
|
|
|
| get_ops_in_block | 3790 | 16 | 278 | 84 |
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
|
|
|
|
|
|
default postgresql.conf
|
|
|
syncad@steem-11:/hafpool/HAfAH$ ./tests/performance_test.py --postgres postgresql://syncad:dandan@127.0.0.1:5432/haf_block_log
|
|
|
2022-05-01 04:41:07,228 - performance - INFO - postgres connection string in JMETER: jdbc:postgresql://127.0.0.1:5432/haf_block_log?user=syncad&password=dandan
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
| Endpoint | Max [ms] | Min [ms] | Average [ms] | Median [ms] |
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
| enum_virtual_ops | 4039 | 21 | 87 | 66 |
|
|
|
| get_transaction | 1055 | 14 | 28 | 21 |
|
|
|
| get_account_history | 2124 | 51 | 374 | 225 |
|
|
|
| get_ops_in_block | 3798 | 17 | 281 | 86 |
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
|
|
|
postgresql.conf: default_toast_compression=lz4, full_page_writes=off, wal_init_zero=off, wal_recycle=off, fsync=off, max_wal_size = 4GB, min_wal_size = 80MB
|
|
|
syncad@steem-11:/hafpool/HAfAH$ ./tests/performance_test.py
|
|
|
2022-05-01 05:33:38,413 - performance - INFO - postgres connection string in JMETER: jdbc:postgresql:haf_block_log
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
| Endpoint | Max [ms] | Min [ms] | Average [ms] | Median [ms] |
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
| enum_virtual_ops | 4017 | 17 | 80 | 61 |
|
|
|
| get_transaction | 1050 | 11 | 25 | 18 |
|
|
|
| get_account_history | 1973 | 20 | 299 | 199 |
|
|
|
| get_ops_in_block | 3943 | 13 | 273 | 83 |
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
|
|
|
postgresql.conf: default_toast_compression=lz4, full_page_writes=off, wal_init_zero=off, wal_recycle=off, fsync=off, max_wal_size = 4GB, min_wal_size = 80MB
|
|
|
+shared_buffers=4GB
|
|
|
syncad@steem-11:/hafpool/HAfAH$ ./tests/performance_test.py
|
|
|
2022-05-01 06:29:37,952 - performance - INFO - postgres connection string in JMETER: jdbc:postgresql:haf_block_log
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
| Endpoint | Max [ms] | Min [ms] | Average [ms] | Median [ms] |
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
| enum_virtual_ops | 4169 | 19 | 76 | 61 |
|
|
|
| get_transaction | 1052 | 11 | 25 | 19 |
|
|
|
| get_account_history | 1973 | 22 | 211 | 103 |
|
|
|
| get_ops_in_block | 3674 | 13 | 262 | 82 |
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
shared_buffers=16GB
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
| Endpoint | Max [ms] | Min [ms] | Average [ms] | Median [ms] |
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
| enum_virtual_ops | 1374 | 17 | 65 | 61 |
|
|
|
| get_transaction | 1052 | 12 | 25 | 19 |
|
|
|
| get_account_history | 1534 | 19 | 183 | 93 |
|
|
|
| get_ops_in_block | 3741 | 14 | 259 | 82 |
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
shared_buffers=32GB
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
| Endpoint | Max [ms] | Min [ms] | Average [ms] | Median [ms] |
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
| enum_virtual_ops | 1360 | 19 | 65 | 62 |
|
|
|
| get_transaction | 1052 | 11 | 26 | 19 |
|
|
|
| get_account_history | 1574 | 18 | 185 | 95 |
|
|
|
| get_ops_in_block | 3656 | 14 | 263 | 83 |
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
shared_buffers=2GB
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
| Endpoint | Max [ms] | Min [ms] | Average [ms] | Median [ms] |
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
| enum_virtual_ops | 1332 | 17 | 64 | 60 |
|
|
|
| get_transaction | 1056 | 11 | 24 | 18 |
|
|
|
| get_account_history | 1582 | 17 | 187 | 103 |
|
|
|
| get_ops_in_block | 3726 | 14 | 265 | 82 |
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
shared_buffers=64MB
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
| Endpoint | Max [ms] | Min [ms] | Average [ms] | Median [ms] |
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
| enum_virtual_ops | 2089 | 18 | 70 | 62 |
|
|
|
| get_transaction | 1052 | 12 | 24 | 19 |
|
|
|
| get_account_history | 2060 | 20 | 228 | 175 |
|
|
|
| get_ops_in_block | 3937 | 13 | 271 | 83 |
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
repeat shared_buffers=4GB
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
| Endpoint | Max [ms] | Min [ms] | Average [ms] | Median [ms] |
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
| enum_virtual_ops | 1419 | 18 | 63 | 61 |
|
|
|
| get_transaction | 1054 | 11 | 22 | 19 |
|
|
|
| get_account_history | 1566 | 17 | 184 | 100 |
|
|
|
| get_ops_in_block | 3718 | 13 | 261 | 81 |
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
|
|
|
shed14: default_toast_compression=lz4, full_page_writes=off, wal_init_zero=off, wal_recycle=off, fsync=off
|
|
|
shared_buffers=128MB
|
|
|
syncad@shed14:/xt-pool/HAfAH$ ./tests/performance_test.py --postgres postgresql://syncad:dandan@127.0.0.1:5432/haf_block_log
|
|
|
2022-05-01 05:37:04,843 - performance - INFO - postgres connection string in JMETER: jdbc:postgresql://127.0.0.1:5432/haf_block_log?user=syncad&password=dandan
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
| Endpoint | Max [ms] | Min [ms] | Average [ms] | Median [ms] |
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
| enum_virtual_ops | 1645 | 19 | 58 | 44 |
|
|
|
| get_transaction | 1056 | 14 | 26 | 21 |
|
|
|
| get_account_history | 1540 | 21 | 227 | 176 |
|
|
|
| get_ops_in_block | 3568 | 15 | 260 | 82 |
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
shared_buffers=4GB
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
| Endpoint | Max [ms] | Min [ms] | Average [ms] | Median [ms] |
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
| enum_virtual_ops | 1628 | 21 | 53 | 44 |
|
|
|
| get_transaction | 1056 | 14 | 26 | 21 |
|
|
|
| get_account_history | 1526 | 20 | 182 | 103 |
|
|
|
| get_ops_in_block | 3594 | 15 | 265 | 83 |
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
|
|
|
POSTGREST TEST
|
|
|
l/HAfAH/tests$ python3 ./performance_test.py -j /xt-pool/HAfAH/apache-jmeter-5.4.3/bin/jmeter --no-launch
|
|
|
2022-05-06 16:42:06,953 - performance - INFO - postgres connection string in JMETER: jdbc:postgresql:haf_block_log
|
|
|
2022-05-06 16:42:06,953 - performance - INFO - removed old datadir /xt-pool/HAfAH/tests/wdir
|
|
|
2022-05-06 16:42:06,953 - performance - INFO - starting performance testing
|
|
|
2022-05-06 16:47:32,766 - performance - INFO -
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
| Endpoint | Max [ms] | Min [ms] | Average [ms] | Median [ms] |
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
| enum_virtual_ops | 4487 | 3 | 63 | 42 |
|
|
|
| get_transaction | 189 | 0 | 8 | 4 |
|
|
|
| get_account_history | 1835 | 18 | 312 | 92 |
|
|
|
| get_ops_in_block | 3885 | 1 | 255 | 59 |
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
---------------------+----------+----------+--------------+-------------+
|
|
|
| Endpoint | Max [ms] | Min [ms] | Average [ms] | Median [ms] |
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
| enum_virtual_ops | 1334 | 3 | 50 | 40 |
|
|
|
| get_transaction | 67 | 0 | 9 | 5 |
|
|
|
| get_account_history | 1195 | 10 | 128 | 68 |
|
|
|
| get_ops_in_block | 4097 | 1 | 255 | 61 |
|
|
|
+---------------------+----------+----------+--------------+-------------+
|
|
|
|
|
|
|
|
|
|
|
|
without huge pages
|
|
|
syncad@steem-11:/hafpool/HAfAH$ pgbench -S -j 32 -c 32 -M prepared -T 20 pgbench (READ)
|
|
|
pgbench (14.2 (Ubuntu 14.2-1ubuntu1))
|
|
|
starting vacuum...end.
|
|
|
transaction type: <builtin: select only>
|
|
|
scaling factor: 100
|
|
|
query mode: prepared
|
|
|
number of clients: 32
|
|
|
number of threads: 32
|
|
|
duration: 20 s
|
|
|
number of transactions actually processed: 21663109
|
|
|
latency average = 0.030 ms
|
|
|
initial connection time = 7.053 ms
|
|
|
tps = 1,083,456.813513 (without initial connection time)
|
|
|
|
|
|
with huge pages (shared_files=4GB)
|
|
|
syncad@steem-11:/hafpool/HAfAH$ pgbench -S -j 32 -c 32 -M prepared -T 20 pgbench
|
|
|
pgbench (14.2 (Ubuntu 14.2-1ubuntu1))
|
|
|
starting vacuum...end.
|
|
|
transaction type: <builtin: select only>
|
|
|
scaling factor: 100
|
|
|
query mode: prepared
|
|
|
number of clients: 32
|
|
|
number of threads: 32
|
|
|
duration: 20 s
|
|
|
number of transactions actually processed: 22524270
|
|
|
latency average = 0.028 ms
|
|
|
initial connection time = 8.112 ms
|
|
|
tps = 1,126,587.583407 (without initial connection time)
|
|
|
``` |
|
|
\ No newline at end of file |