The new ATX node!
Finally revealing what I bought. And, for some, this might instantaneously be a disappointing thing, about me going to Intel, although I had a reason (double one actually) for this strategy.
First, Intel CPUs are still faster for some "front ends" (web stuff) and for mixed workloads, it might be a debate, but might also win in some groups of workloads. Power-wise, it's OK... most stuff in the real world is fine and efficient. Yes on games AMD wins mostly... but mostly, for things that need clock speed and rich architecture branches, Intel has a better domain of cover.
The second reason was the price! And I will explain below why...
For IO and storage, definitely AMD! IOps and overall latency vs power, AMD rocks... So many lanes and a very good connection to IO and on bigger enterprise models, the extra memory channels.
But I have bought this one with a vision already... and that's, buying later an AMD system for the IO part! and leaving this one in the future for the front end. As Intel CPUs can easily sustain longer periods of support for many codes without being tooo deprecated.
Anyhow, apart from this, and some other tests I can't reveal due to NDA's, I have decided to procure the Intel 14th gen CPU. Also, because this CPU in NZ was quite cheaper than AMD (due to high demand) - a problem I have to balance due to warranty, which is a pain in the ass when I get stuff outside the country... because anytime I have a problem, I incur more costs! Hence why do I default to in-country when I know it's not worth the pennies...
Either way, the CPU is quite good (I would probably buy this for games too, but probably buy AMD for anything else, at least while under AMD high demand, otherwise AMD first). And I have yet to reveal some power usages... but compared with my other AMD Zen2 6-core stuff, this one beats it quickly! Idle power is super nice... if you running stuff very low at the start... and it can quickly scale too.
Zen5 is going to be fricking crazy! Intel has some stuff under the sleeve tooo... things are changing with the introduction of HBM and NVIDIA coming into the play with ARM... there is so much activity this year, it's a clusterF... of news!
Either way, here is guy! Enjoy the benchmarks and let me know any questions...
Results Disclaimer
Be aware please, especially in the VM solution, that results may vary (due to other workloads, although I have attempted to minimize the impact as much as I could) due to specific details I might have missed or failed to detect their conditions/reasons for possible impact. As a result of this work, please take these AS IS.
The reason I do these, is to be sure what I will be counting on and to be able to find baselines for when troubleshooting is needed in the future. You don't need to do this yourself if you don't want to. For me, this is also a mix of "keeping up to date" and enjoyment, due to inherent professional skills.
Hive Compile
- 6-core (VM) Intel 10th Gen, DDR4 2933 MHz, NVMe 2TB PCIe 3.0
Ubuntu 23.10
time cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_HIVE_TESTNET=OFF -GNinja ../../hive/
# (3.7-seconds)
time ninja
# (real 18m9.183s) 6 threads
- 24-core Intel 14th Gen, DDR5 5600 MHz, NVMe 1TB Samsung 990 PRO PCIe 4.0
Ubuntu 22.04
time cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_HIVE_TESTNET=OFF -GNinja ../../hive/
# (1.3-seconds)
time ninja
# (real 2m53.608s) 32 threads (system consumption of 150-440 Watts)
Less than 3 minutes 😱
Hive Live Sync
Otherwise mentioned (if different), these were the seed nodes I used:
p2p-seed-node = api.hive.blog:2001 rpc.ecency.com:2001 seed.openhive.network:2001 anyx.io:2001 hiveseed-fin.privex.io:2001 hive-seed.arcange.eu:2001 seed.liondani.com:2016 hived.splinterlands.com:2001 node.mahdiyari.info:2001 seed.roelandp.nl:2001 hiveseed-se.privex.io:2001 hive-seed.lukestokes.info:2001
Partial
- 6-core (VM) Intel 10th Gen, DDR4 2933 MHz, NVMe 2TB PCIe 3.0
first 2M blocks live sync, Ubuntu 22.04
real 6m3.114s
user 9m59.928s
sys 3m2.908s
- 6-core (VM) Intel 10th Gen, DDR4 2933 MHz, NVMe 2TB PCIe 3.0
first 2M blocks live sync, Ubuntu 23.10
real 5m52.383s
user 10m12.387s
sys 3m11.303s
- 6-core (VM) Intel 10th Gen, DDR4 2933 MHz, NVMe 2TB PCIe 3.0
first 2M blocks live sync, Ubuntu 23.10
p2p-seed-node = localnode:2001
real 6m23.933s
user 10m37.310s
sys 3m29.787s
p2p-seed-node = localnode:2001 api.hive.blog:2001 rpc.ecency.com:2001 seed.openhive.network:2001 anyx.io:2001 hiveseed-fin.privex.io:2001 hive-seed.arcange.eu:2001 seed.liondani.com:2016 hived.splinterlands.com:2001 node.mahdiyari.info:2001 seed.roelandp.nl:2001 hiveseed-se.privex.io:2001 hive-seed.lukestokes.info:2001
real 6m33.510s
user 11m4.258s
sys 3m47.537s
- 24-core Intel 14th Gen, DDR5 5600 MHz, NVMe 1TB Samsung 990 PRO PCIe 4.0
first 2M blocks live sync, Ubuntu 22.04
real 2m6.406s
user 4m10.067s
sys 0m51.402s
Full (blocks indicated)
- 24-core Intel 14th Gen, DDR5 5600 MHz, NVMe 1TB Samsung 990 PRO PCIe 4.0
full live sync, Ubuntu 22.04
2023-12-24T00:04:31.632 application.cpp:174 load_logging_config ] Logging thread started
...
2023-12-24T19:22:59.666 chain_plugin.cpp:1176 accept_block ] Syncing Blockchain --- Got block: #81310000 time: 2023-12-24T16:40:12 producer: guiltyparties --- 979.81µs/block, avg. size 10553, avg. tx count 39
2023-12-24T19:23:03.017 chain_plugin.cpp:498 operator() ] entering live mode
2023-12-24T19:23:03.017 chain_plugin.cpp:519 operator() ] waiting for work: 0.01%, waiting for locks: 0.00%, processing transactions: 0.00%, processing blocks: 99.95%, unknown: 0.04%
2023-12-24T19:23:03.331 block_log_artifacts.cpp:443 flush_header ] Attempting to write header (pack_size: 56) containing: git rev: 0acce16829c0702ac09d9e51beb57b9fac157c22, format version: 1.1, head_block_num: 81313230, tail_block_num: 1, generating_interrupted_at_block: 0, dirty_close: 1
19h18m 😱 - 81313230 blocks
And I have observed the same performance with just a local node (p2p-seed-node = localnode:2001). Probably using more than one local node could have advantages, but likely very little to no improvement (as most of the time spent here is buffered/hidden, assuming enough resources obviously).
(after firmware update)
No difference, and I am guessing here, but possibly because there are too many hidden processing overlaps between access to IO and the time that takes to process things. Cumulative with the downloading time, adds no difference. And since most of the time (writing to disk) is cached, the impact is very little to none (or hidden as previously mentioned).
Hive Local Replay
Partial
- 6-core (VM) Intel 10th Gen, DDR4 2933 MHz, NVMe 2TB PCIe 3.0
first 10M blocks replay, Ubuntu 22.04
real 19m24.695s
user 23m27.612s
sys 2m21.661s
- 6-core (VM) Intel 10th Gen, DDR4 2933 MHz, NVMe 2TB PCIe 3.0
first 10M blocks replay, Ubuntu 23.10
real 19m36.713s
user 24m6.525s
sys 1m52.247s
- 24-core Intel 14th Gen, DDR5 5600 MHz, NVMe 1TB Samsung 990 PRO PCIe 4.0
first 10M blocks replay, Ubuntu 22.04
real 6m35.121s
user 7m25.173s
sys 0m36.348s
I tried overclocking the memory to 5800 and 6000 MHz and it gave worse timings... 6m41s and 6m40s respectively (voltage raised to stabilize, but controller frequency ratios might have needed more frequency to compensate). I didn't spend enough time with this to explore better conclusions. Timings were also calculated by the motherboard, so they might not have been ideal.
Somewhere in the process of many of these benchmarks, I managed to experience a problem with the 990 PRO NVMe, when this was running the 3B2QJXD7 firmware, so I have updated it to the at the time most recent (4B2QJXD7). I shared some notes on how you can do this here.
(after firmware update)
- 24-core Intel 14th Gen, DDR5 5600 MHz, NVMe 1TB Samsung 990 PRO PCIe 4.0
first 10M blocks replay, Ubuntu 22.04
real 5m51.730s
user 6m42.006s
sys 0m35.455s
Full (blocks indicated)
- 24-core Intel 14th Gen, DDR5 5600 MHz, NVMe 1TB Samsung 990 PRO PCIe 4.0
full replay, Ubuntu 22.04
2023-12-24T20:44:35.362 application.cpp:174 load_logging_config ] Logging thread started
...
2023-12-25T04:34:16.905 block_log.cpp:837 operator() ] Exiting the queue thread
2023-12-25T04:34:17.386 block_log.cpp:887 for_each_block ] Attempting to join queue_filler_thread...
2023-12-25T04:34:17.386 block_log.cpp:889 for_each_block ] queue_filler_thread joined.
2023-12-25T04:34:17.386 database.cpp:460 reindex ] Done reindexing, elapsed time: 28181.39933899999959976 sec
2023-12-25T04:34:17.386 chain_plugin.cpp:1101 plugin_startup ] P2P enabling after replaying...
2023-12-25T04:34:17.386 chain_plugin.cpp:768 work ] Started on blockchain with 81314881 blocks, LIB: 81314862
2023-12-25T04:34:17.386 chain_plugin.cpp:774 work ] Started on blockchain with 81314881 blocks
7h50m 😱 - 81314881 blocks
- 24-core Intel 14th Gen, DDR5 5600 MHz, NVMe 1TB Samsung 990 PRO PCIe 4.0
full replay, Ubuntu 23.10
2024-01-03T04:54:41.502 application.cpp:174 load_logging_config ]
...
2024-01-03T12:39:03.301 chain_plugin.cpp:498 operator() ] entering live mode
2024-01-03T12:39:03.305 block_log_artifacts.cpp:443 flush_header ] Attempting to write header (pack_size: 56) containing: git rev: 0acce16829c0702ac09d9e51beb57b9fac157c22, format version: 1.1, head_block_num: 81592963, tail_block_num: 1, generating_interrupted_at_block: 0, dirty_close: 1
The difference compared with Ubuntu 22.04
2024-01-03T05:00:27.502 10000000 of 81583660 blocks = 12.26% +5m46s 346 x1 -4s
2024-01-03T05:36:49.577 20000000 of 81583660 blocks = 24.51% +36m22s 2182 x6.31 -35s
2024-01-03T06:44:02.082 30000000 of 81583660 blocks = 36.77% +1h07m13s 4033 x11.66 -71s
2024-01-03T07:35:10.006 40000000 of 81583660 blocks = 49.03% +51m08s 3068 x8.87 -55s
2024-01-03T08:08:33.446 50000000 of 81583660 blocks = 61.29% +33m23s 2003 x5.79 -9s
2024-01-03T09:07:53.024 60000000 of 81583660 blocks = 73.54% +59h20s 3560 x10.29 -49s
2024-01-03T10:38:08.463 70000000 of 81583660 blocks = 85.80% +1h29m16s 5356 x15.48 -189s
2024-01-03T12:25:57.193 80000000 of 81583660 blocks = 98.06% +1h42m49s 6169 x17.83 -491s
2024-01-03T12:38:13.505 81500000 of 81583660 blocks = 99.90% +12m16s
7h45m22s 😱 - 81592963 blocks
Same system, filesystem, etc.
(after firmware update)
Not much noticeable difference! As expected because most of the code is dependent on rather lower latency things.
Hive-Engine
Light DB (restore)
- 6-core (VM) Intel 10th Gen, DDR4 2933 MHz, NVMe 2TB PCIe 3.0
mongo 7, Ubuntu 23.10, zfs recordsize=128k
time mongorestore --gzip --archive=./hsc_12-27-2023_b81403927.archive --drop
real 40m36.832s
user 9m0.805s
sys 3m9.786s
- 24-core Intel 14th Gen, DDR5 5600 MHz, NVMe 1TB Samsung 990 PRO PCIe 4.0
mongo 7, Ubuntu 22.04, zfs recordsize=128k
time mongorestore --gzip --archive=./hsc_12-27-2023_b81403927.archive --drop -j 8
real 19m17.922s
user 7m16.488s
sys 0m46.082s
Note: Using 8 (-j 8) numParallelCollections did not make any difference (against the 4 default) for this DB dump, in the same system.
- 24-core Intel 14th Gen, DDR5 5600 MHz, NVMe 1TB Samsung 990 PRO PCIe 4.0
mongo 7, Ubuntu 23.10, zfs recordsize=128k
time mongorestore --gzip --archive=./hsc_12-27-2023_b81403927.archive --drop -j 8
real 26m47.608s
user 8m47.959s
sys 0m58.855s
vs -f 4
real 26m42.191s
user 8m44.144s
sys 1m3.569s
All the above had ashift=9 (512 bytes), which makes a huge impact on NVMe wear...and it's not worth the performance, given the amount of work you will be doing. So make sure you increase the ashift if you are using ZFS, to at least 12 (4k bytes).
- 24-core Intel 14th Gen, DDR5 5600 MHz, NVMe 1TB Samsung 990 PRO PCIe 4.0
mongo 7.0.5, Ubuntu 22.04, zfs recordsize=4k, NVMe with ashift=12
time mongorestore --gzip --archive=./hsc_01-10-2024_b81806836.archive --drop -j 16
real 21m16.251s
user 7m25.942s
sys 0m42.511s
- 24-core Intel 14th Gen, DDR5 5600 MHz, NVMe 1TB Samsung 990 PRO PCIe 4.0
mongo 7.0.5, Ubuntu 22.04, zfs recordsize=128k, NVMe with ashift=12
time mongorestore --gzip --archive=./hsc_01-10-2024_b81806836.archive --drop -j 16
real 21m10.244s
user 7m37.555s
sys 0m42.051s
And it looks like 32k aligned IO, is still more performant.
- 24-core Intel 14th Gen, DDR5 5600 MHz, NVMe 1TB Samsung 990 PRO PCIe 4.0
mongo 7.0.5, Ubuntu 22.04, zfs recordsize=32k, NVMe with ashift=12
time mongorestore --gzip --archive=./hsc_01-10-2024_b81806836.archive --drop -j 16
real 20m24.926s
user 7m29.396s
sys 0m41.792s
Full DB (restore)
- 24-core Intel 14th Gen, DDR5 5600 MHz, NVMe 1TB Samsung 990 PRO PCIe 4.0
mongo 7.0.4, Ubuntu 23.10, zfs recordsize=128k
time mongorestore --gzip --archive=./hsc_12-27-2023_b81403927.archive
This one was cancelled due to some sort of performance bug between Mongo 7 and Ubuntu 23.10
2024-01-08T00:50:33.938+1300 hsc.chain 43.6GB
2024-01-08T00:50:33.938+1300 hsc.transactions 42.0GB
2024-01-08T00:50:33.938+1300
2024-01-08T00:50:34.122+1300 signal 'interrupt' received; attempting to shut down
2024-01-08T00:50:34.202+1300 terminating read on hsc.transactions
2024-01-08T00:50:34.239+1300 hsc.transactions 42.0GB
2024-01-08T00:50:34.239+1300 finished restoring hsc.transactions (539924017 documents, 0 failures)
2024-01-08T00:50:34.239+1300 Failed: hsc.transactions: error restoring from archive './hsc_01-05-2024_b81662950.archive': received termination signal
2024-01-08T00:50:34.239+1300 539924017 document(s) restored successfully. 0 document(s) failed to restore.
real 1583m42.338s
user 67m41.998s
sys 8m37.995s
And then because I have seen that there was an update to Mongo, I did another run...
- 24-core Intel 14th Gen, DDR5 5600 MHz, NVMe 1TB Samsung 990 PRO PCIe 4.0
mongo 7.0.5-rc0, Ubuntu 23.10, zfs recordsize=128k
time mongorestore --gzip --archive=./hsc_12-27-2023_b81403927.archive -j 16
2024-01-10T00:46:54.603+1300 727706502 document(s) restored successfully. 0 document(s) failed to restore.
real 1480m42.037s
user 124m20.456s
sys 14m11.413s
Slightly more than a full day... and I am not sure why it can't do more IO, but it likely needs lower latency than parallel IO. The next step is to increase the number of NVMes to see if this scales.
And because this one was done with -j 16 I am now thinking about the first run, and if I was actually going to finish, but there is a part of the restore where the speed of restore is super slow. In this run, I have also increased the cache of Mongo from 8 to 16GB, as I saw it being limited when using 8GB size.
- 24-core Intel 14th Gen, DDR5 5600 MHz, NVMe 1TB Samsung 990 PRO PCIe 4.0
mongo 7.0.5-rc0, Ubuntu 23.10, zfs recordsize=32k
time mongorestore --gzip --archive=./hsc_12-27-2023_b81403927.archive -j 16
2024-01-11T04:27:19.628+1300 727706502 document(s) restored successfully. 0 document(s) failed to restore.
real 905m53.218s
user 113m19.469s
sys 12m29.928s