As we continue to build out our monitoring framework, we decided it would be a cool experiment to extend the checks to external BP services.
We use Diamond collectors to fetch stats on a wide array of system level stats, but the collectors are easy to extend and to integrate your own.
There are 4 checks that we thought would be useful to allow anyone to get a decent snapshot view of the health over the EOS network across the top 50 block producers:
- Ping: Send
ICMPECHO_REQUEST to the public BP host - HTTP: Send a request to the
/v1/chain/get_infoRPC API endpoint - P2P: Attempt a socket connection to the public P2P port
- Head Block: Determine how well the local chain is in sync via the
head_block_time
To ensure we have decent geographic distribution, we placed these 4 custom diamond collectors around the globe. They fetch stats locally, then transport them securely to a central influxDB instance which is then rendered through Grafana.
InfluxDB works well for this kind of immutable time series data, but it's really important to get your schema correct. Using the right combination of well considered key names with associated tags is imperative to ensure it scales well (don't forget to set a retention period too!).
You can interact with the monitoring at our EOS Node Tools microsite: https://eosnode.tools/monitor
It is possible to filter the monitoring by individual block producers and you can also filter through the 6 locations where we collect the stats.
As the EOS network continues to grow, it's really important that we are able to visualise the network behaviour as an alternative to trawling log files. We hope that conveying the data in this way will hopefully catch trends in the data that will allow us to spot issues before they manifest in a widespread manner and it should also help our fellow BP's understand their performance against others and within specific locations.
Block Matrix are currently a paid standby BP for the EOS network. We are really passionate about the EOS project and are focussed on creating robust infrastructure and open sourcing everything we build to support the network and the wider community.