This post will be a go-to reference for myself, and a resource for anyone interested in setting up an api node. It's very technical, so if you don't belong to one of those two groups you can skip it.
Introduction
My current setup is close to the default hivemind setup provided in the example jussi config. After talks with @gtg one of the nodes offers a few more apis for performance reasons, as the fat node is very slow. There surely are not-optimal or even misconfigurations, optimization will be an ongoing process. See the end of the post for more info, make sure to read and understand everything before you start. I will update this post with any important changes.
Hardware
First, we need hardware. The setup consists of 3 nodes, for which I selected the following specs:
- hivemind
32GB RAM
2x240GB SSD RAID0 - fat
64GB RAM
2x480GB SSD RAID0 - accounthistory
64GB RAM
2x512GB NVMe RAID0 (64GB SWAP)
All are set up with a clean install of Ubuntu 18.04
Setup
Common steps
Set up each server to log in securely and with a dedicated user. The user on all machines will be called "hive" during this process. Individual needs may differ so I don't go into details here and only provide the steps necessary to proceed.
sudo apt-get update && sudo apt-get upgrade
sudo apt-get install -y screen git nginx certbot python-certbot-nginx
Step 1: make everything sync
hivemind node
install software
cd ~
sudo apt-get install -y python3 python3-pip postgresql postgresql-contrib docker.io
git clone https://gitlab.syncad.com/hive/hivemind.git
cd hivemind
sudo pip3 install -e .[test]
setup database
sudo su postgres
createdb hive
Create db user hive and grant access to database
createuser --interactive
psql
GRANT ALL PRIVILEGES ON DATABASE hive TO hive;
\q
exit
optimize postgres
sudo nano /etc/postgresql/10/main/postgresql.conf
use https://pgtune.leopard.in.ua/ to find the optimal settings for your machine. I used the following:
# DB Version: 10
# OS Type: linux
# DB Type: web
# Total Memory (RAM): 32 GB
# Data Storage: ssd
max_connections = 200
shared_buffers = 8GB
effective_cache_size = 24GB
maintenance_work_mem = 2GB
checkpoint_completion_target = 0.7
wal_buffers = 16MB
default_statistics_target = 100
random_page_cost = 1.1
effective_io_concurrency = 200
work_mem = 20971kB
min_wal_size = 1GB
max_wal_size = 4GB
sudo service postgresql restart
The irredeemables list is a blacklist containing mass spammers mostly. It's recommended to use it if you serve browser based interfaces, because the amount of comments by these accounts creates a lot of traffic and is a burden on browsers. It's defined in /home/hive/hivemind/hive/conf.py under --muted-accounts-url. You can change it there, or add the environment variable MUTED_ACCOUNTS_URL in both scripts if you do not want to use the default. I offer an empty version if you don't want to filter the results.
Create sync script
nano sync.sh
Insert the following (the STEEMD_URL is temporary until your own fat node has synced, update and restart it in a few days to speed up the hivemind sync):
#!/bin/bash
export DATABASE_URL=postgresql://hive:pass@localhost:5432/hive
export STEEMD_URL='{"default": "https://fat.pharesim.me"}'
export HTTP_SERVER_PORT=28091
hive sync
chmod +x sync.sh
screen -S hivesync
./sync.sh
Use Ctrl-a d to detach screen, screen -r hivesync to reattach
The whole sync process takes about a week. Don't forget changing the STEEMD_URL when your fat node is finished. Unlike the steemd replays, you can interrupt this sync at any time and it picks up where you stopped.
The sync is finished when you see single blocks coming in. Keep it running, and set up the server:
cp sync.sh hivemind.sh
nano hivemind.sh
Change sync in the end for server
screen -S hivemind
./hivemind.sh
Use Ctrl-a d to detach screen, screen -r hivemind to reattach
steemd nodes
Both the fat and the accounthistory node will run an instance of steemd, these are the steps to prepare them:
sudo apt-get install -y autoconf automake cmake g++ git libbz2-dev libsnappy-dev libssl-dev libtool make pkg-config python3-jinja2 libboost-chrono-dev libboost-context-dev libboost-coroutine-dev libboost-date-time-dev libboost-filesystem-dev libboost-iostreams-dev libboost-locale-dev libboost-program-options-dev libboost-serialization-dev libboost-signals-dev libboost-system-dev libboost-test-dev libboost-thread-dev doxygen libncurses5-dev libreadline-dev perl ntp
cd
git clone https://github.com/openhive-network/hive
cd hive
git checkout v0.23.0
git submodule update --init --recursive
mkdir build
cd build
The build options differ for the two nodes
fat
cmake -DCMAKE_BUILD_TYPE=Release -DLOW_MEMORY_NODE=OFF -DCLEAR_VOTES=OFF -DSKIP_BY_TX_ID=OFF -DBUILD_STEEM_TESTNET=OFF -DENABLE_MIRA=ON -DSTEEM_STATIC_BUILD=ON ..
accounthistory
cmake -DCMAKE_BUILD_TYPE=Release -DLOW_MEMORY_NODE=ON -DCLEAR_VOTES=ON -DSKIP_BY_TX_ID=OFF -DBUILD_STEEM_TESTNET=OFF -DENABLE_MIRA=OFF -DSTEEM_STATIC_BUILD=ON ..
Again on both:
make -j$(nproc) steemd
cd
mkdir bin
cp /home/hive/hive/build/programs/steemd bin/v0.23.0
mkdir .steemd
nano .steemd/config.ini
And again, the configs differ for the two nodes
fat
log-appender = {"appender":"stderr","stream":"std_error"} {"appender":"p2p","file":"logs/p2p/p2p.log"}
log-logger = {"name":"default","level":"info","appender":"stderr"} {"name":"p2p","level":"warn","appender":"p2p"}
backtrace = yes
plugin = webserver p2p json_rpc witness account_by_key reputation market_history
plugin = database_api account_by_key_api network_broadcast_api reputation_api
plugin = market_history_api condenser_api block_api rc_api
history-disable-pruning = 0
account-history-rocksdb-path = "blockchain/account-history-rocksdb-storage"
block-data-export-file = NONE
block-log-info-print-interval-seconds = 86400
block-log-info-print-irreversible = 1
block-log-info-print-file = ILOG
sps-remove-threshold = 200
shared-file-dir = "blockchain"
shared-file-size = 360G
shared-file-full-threshold = 0
shared-file-scale-rate = 0
follow-max-feed-size = 500
follow-start-feeds = 0
market-history-bucket-size = [15,60,300,3600,86400]
market-history-buckets-per-size = 5760
p2p-seed-node = anyx.io:2001 gtg.steem.house:2001 seed.jesta.us:2001
rc-skip-reject-not-enough-rc = 0
rc-compute-historical-rc = 0
statsd-batchsize = 1
tags-start-promoted = 0
tags-skip-startup-update = 0
transaction-status-block-depth = 64000
transaction-status-track-after-block = 0
webserver-http-endpoint = 0.0.0.0:28091
webserver-ws-endpoint = 0.0.0.0:28090
webserver-thread-pool-size = 32
enable-stale-production = 0
required-participation = 33
witness-skip-enforce-bandwidth = 1
Not sure about the shared_file_size here, it's rocksdb? Better safe than sorry...
accounthistory
log-appender = {"appender":"stderr","stream":"std_error"} {"appender":"p2p","file":"logs/p2p/p2p.log"}
log-logger = {"name":"default","level":"info","appender":"stderr"} {"name":"p2p","level":"warn","appender":"p2p"}
backtrace = yes
plugin = webserver p2p json_rpc witness
plugin = rc market_history_account_history_rocksdb transaction_status account_by_key
plugin = database_api condenser_api market_history_api account_history_api transaction_status_api account_by_key_api
plugin = block_api network_broadcast_api rc_api
history-disable-pruning = 1
account-history-rocksdb-path = "blockchain/account-history-rocksdb-storage"
block-data-export-file = NONE
block-log-info-print-interval-seconds = 86400
block-log-info-print-irreversible = 1
block-log-info-print-file = ILOG
sps-remove-threshold = 200
shared-file-dir = "/run/hive"
shared-file-size = 120G
shared-file-full-threshold = 9500
shared-file-scale-rate = 1000
flush-state-interval = 0
follow-max-feed-size = 500
follow-start-feeds = 0
market-history-bucket-size = [15,60,300,3600,86400]
market-history-buckets-per-size = 5760
p2p-seed-node = anyx.io:2001 seed.jesta.us:2001
rc-skip-reject-not-enough-rc = 0
rc-compute-historical-rc = 0
statsd-batchsize = 1
tags-start-promoted = 0
tags-skip-startup-update = 0
transaction-status-block-depth = 64000
transaction-status-track-after-block = 42000000
webserver-http-endpoint = 0.0.0.0:28091
webserver-ws-endpoint = 0.0.0.0:28090
webserver-thread-pool-size = 32
The fat node also needs a database.cfg
nano .steemd/database.cfg
These settings are for 32GB of RAM. Adapt global.shared_cache.capacity, global.write_buffer_manager.write_buffer_size and global.object_count accordingly
{
"global": {
"shared_cache": {
"capacity": "21474836480"
},
"write_buffer_manager": {
"write_buffer_size": "4294967296"
},
"object_count": 250000,
"statistics": false
},
"base": {
"optimize_level_style_compaction": true,
"increase_parallelism": true,
"block_based_table_options": {
"block_size": 8192,
"cache_index_and_filter_blocks": true,
"bloom_filter_policy": {
"bits_per_key": 10,
"use_block_based_builder": false
}
}
}
}
As well as increased file limits (hive is the username, adapt to your memory again)
sudo nano /etc/security/limits.conf
Insert near the end of the file
hive soft nofile 262140
hive hard nofile 262140
sudo nano /etc/sysctl.conf
Insert near the end of the file
fs.file-max = 2097152
You need to log in to the server again for these to take effect.
The accounthistory requires a change of the size of /run
mount -o remount ,size=120G /run
and a directory /run/hive
sudo mkdir /run/hive
sudo chown hive:hive /run/hive
Then continue on both servers.
Download block_log.index and block_log
rsync -avh --progress --append rsync://files.privex.io/hive/block_log.index .steemd/blockchain/block_log.index
rsync -avh --progress --append rsync://files.privex.io/hive/block_log .steemd/blockchain/block_log
Go for a walk or have dinner.
Start up steemd and replay blockchain
screen -S hive
echo 75 | sudo tee /proc/sys/vm/dirty_background_ratio
echo 1000 | sudo tee /proc/sys/vm/dirty_expire_centisecs
echo 80 | sudo tee /proc/sys/vm/dirty_ratio
echo 30000 | sudo tee /proc/sys/vm/dirty_writeback_centisecs
~/bin/v0.23.0 --replay
Use Ctrl-d a to detach from screen, and screen -r hive to reattach.
The sync process takes a bit more than 2 days on the fat node, less on accounthistory. Do not interrupt them, or you will have to start over. If you are syncing a hivemind node, don't forget to switch to the fat node when that's finished.
Step 2: webserver + routing
all nodes
All requests will be proxied by nginx, so we need this on all machines. We will install SSL certificates, so all communication is encrypted and all nodes can be called individually.
sudo nano /etc/nginx/sites-enabled/hive
The config is the same for each node, only change the server_name
upstream hivesrvs {
# Dirty Hack. Causes nginx to retry node
server 127.0.0.1:28091;
server 127.0.0.1:28091;
server 127.0.0.1:28091;
server 127.0.0.1:28091;
keepalive 10;
}
server {
server_name hivemind/fat/acchist.you.tld;
root /var/www/html/;
location ~ ^(/|/ws) {
proxy_pass http://hivesrvs;
proxy_set_header Connection "";
include snippets/rpc.conf;
}
Add the rpc.conf to each server
sudo nano /etc/nginx/snippets/rpc.conf
Insert
access_log off;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
proxy_connect_timeout 10;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
keepalive_timeout 65;
keepalive_requests 100000;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
proxy_ssl_verify off;
Let certbot configure the domains for automatic redirect to https
sudo certbot --nginx
hivemind node
We need an additional file for nginx, for the general entry point
sudo cp /etc/nginx/sites-enabled/hive /etc/nginx/sites-enabled/api
sudo nano /etc/nginx/sites-enabled/api
Change both occurances of hivesrvs to jussisrv, the ports from 28091 to 9000, and the server_name to api., or whatever you want your node to be accessible on.
jussi
cd ~
git clone https://gitlab.syncad.com/hive/jussi.git
Create build script
nano build.sh
and insert
#!/bin/bash
cd /home/hive/jussi
sudo docker build -t="$USER/jussi:$(git rev-parse --abbrev-ref HEAD)" .
chmod +x build.sh
Create run script
nano jussi.sh
and insert
#!/bin/bash
cd /home/hive/jussi
sudo docker run -itp 9000:8080 --log-opt max-size=50m "$USER/jussi:$(git rev-parse --abbrev-ref HEAD)"
chmod +x jussi.sh
screen -S jussi
cd ~/jussi
nano DEV_config.json
Currently, my config looks like this:
{
"limits": { "accounts_blacklist": [ "accounttoblock" ] },
"upstreams": [
{
"name": "steemd",
"translate_to_appbase": true,
"urls": [["steemd", "https://fat1.pharesim.me" ]],
"ttls": [["steemd", 3]],
"timeouts": [["steemd",3]]
},
{
"name": "appbase",
"urls": [
["appbase", "https://fat1.pharesim.me"],
["appbase.account_history_api", "https://acchist1.pharesim.me"],
["appbase.condenser_api.get_account_history", "https://acchist1.pharesim.me"],
["appbase.condenser_api.get_ops_in_block", "https://acchist1.pharesim.me"],
["appbase.condenser_api.get_transaction", "https://acchist1.pharesim.me"],
["appbase.condenser_api.get_followers", "https://hivemind.pharesim.me"],
["appbase.condenser_api.get_following", "https://hivemind.pharesim.me"],
["appbase.condenser_api.get_follow_count", "https://hivemind.pharesim.me"],
["appbase.condenser_api.get_discussions_by_trending", "https://hivemind.pharesim.me"],
["appbase.condenser_api.get_discussions_by_hot", "https://hivemind.pharesim.me"],
["appbase.condenser_api.get_discussions_by_promoted", "https://hivemind.pharesim.me"],
["appbase.condenser_api.get_discussions_by_created", "https://hivemind.pharesim.me"],
["appbase.condenser_api.get_discussions_by_blog", "https://hivemind.pharesim.me"],
["appbase.condenser_api.get_discussions_by_feed", "https://hivemind.pharesim.me"],
["appbase.condenser_api.get_discussions_by_comments", "https://hivemind.pharesim.me"],
["appbase.condenser_api.get_reblogged_by", "https://hivemind.pharesim.me"],
["appbase.condenser_api.get_replies_by_last_update", "https://hivemind.pharesim.me"],
["appbase.condenser_api.get_trending_tags", "https://hivemind.pharesim.me"],
["appbase.condenser_api.get_discussions_by_author_before_date", "https://hivemind.pharesim.me"],
["appbase.condenser_api.get_post_discussions_by_payout", "https://hivemind.pharesim.me"],
["appbase.condenser_api.get_comment_discussions_by_payout", "https://hivemind.pharesim.me"],
["appbase.condenser_api.get_blog", "http://localhost:28091"],
["appbase.condenser_api.get_blog_entries", "https://hivemind.pharesim.me"],
["appbase.condenser_api.get_account_votes", "https://hivemind.pharesim.me"],
["appbase.condenser_api.get_state", "https://hivemind.pharesim.me"],
["appbase.condenser_api.get_state.params=['witnesses']", "https://acchist1.pharesim.me"],
["appbase.condenser_api.get_state.params=['/witnesses']", "https://acchist1.pharesim.me"],
["appbase.condenser_api.get_state.params=['/~witnesses']", "https://acchist1.pharesim.me"],
["appbase.condenser_api.get_state.params=['~witnesses']", "https://acchist1.pharesim.me"],
["appbase.follow_api", "https://hivemind.pharesim.me"],
["appbase.tags_api", "https://hivemind.pharesim.me"],
["appbase.market_history_api", "https://acchist1.pharesim.me"],
["appbase.transaction_status_api", "https://acchist1.pharesim.me"],
["appbase.account_by_key_api", "https://acchist1.pharesim.me"],
["appbase.block_api", "https://acchist1.pharesim.me"],
["appbase.network_broadcast_api", "https://acchist1.pharesim.me"],
["appbase.rc_api", "https://acchist1.pharesim.me"]
],
"ttls": [
["appbase", 3],
["appbase.login_api",-1],
["appbase.network_broadcast_api", -1],
["appbase.follow_api", 10],
["appbase.market_history_api", 1],
["appbase.condenser_api", 3],
["appbase.condenser_api.get_block", -2],
["appbase.condenser_api.get_block_header", -2],
["appbase.condenser_api.get_content", 1],
["appbase.condenser_api.get_state", 1],
["appbase.condenser_api.get_state.params=['/trending']", 30],
["appbase.condenser_api.get_state.params=['trending']", 30],
["appbase.condenser_api.get_state.params=['/hot']", 30],
["appbase.condenser_api.get_state.params=['/welcome']", 30],
["appbase.condenser_api.get_state.params=['/promoted']", 30],
["appbase.condenser_api.get_state.params=['/created']", 10],
["appbase.condenser_api.get_dynamic_global_properties", 3]
],
"timeouts": [
["appbase", 3],
["appbase.network_broadcast_api",0],
["appbase.chain_api.push_block", 0],
["appbase.chain_api.push_transaction", 0],
["appbase.condenser_api.broadcast_block", 0],
["appbase.condenser_api.broadcast_transaction", 0],
["appbase.condenser_api.broadcast_transaction_synchronous", 0],
["appbase.condenser_api.get_account_history", 20],
["appbase.condenser_api.get_account_votes", 20],
["appbase.condenser_api.get_ops_in_block.params=[2889020,false]", 20],
["appbase.account_history_api.get_account_history", 20],
["appbase.account_history_api.get_ops_in_block.params={\"block_num\":2889020,\"only_virtual\":false}", 20]
]
},
{
"name": "hive",
"urls": [["hive", "http://localhost:28091"]],
"ttls": [["hive", -1]],
"timeouts": [["hive", 30]]
},
{
"name": "bridge",
"translate_to_appbase": false,
"urls": [["bridge","http://localhost:28091"]],
"ttls": [["bridge",-1]],
"timeouts": [["bridge",30]]
}
]
}
cd
./build.sh
This takes a while, when finished
./run.sh
Use Ctrl-a d to detach screen, screen -r jussi to reattach
If you update your config, run build.sh outside of screen, then attach to screen and restart the run script. (There may be faster ways to update the docker with a new config, but I'm new to this).
Something about steemd apis
As you might have realized, there are some duplicates in the apis fat and accounthistory provide. That's because of what I mentioned above, fat is slow. I did not investigate which aren't needed on fat now to still work for hivemind, so I didn't change anything in that default configuration. I also just took the list for apis on accounthistory from @gtg without questioning. There is unnecessary redundancy for sure, instructions may change in the future to improve on this. Your perfect setup may differ completely, depending on which apis you need to serve (most).
Finishing words
That's it. After everything is synced, you should have a working public node ready to serve requests! If this guide has helped you and/or you want to support my work on Hive infrastructure, education, onboarding and retention, please help me secure my witness spot!
Stats
Current output of df -h on the three servers (April 14):
hivemind
/dev/md2 407G 272G 115G 71% /
fat
/dev/md2 815G 369G 406G 48% /
accounthistory
tmpfs 120G 58G 63G 48% /run
/dev/md2 874G 531G 299G 65% /