Hey everyone,
My investigation into the high CPU usage on the "moon" server has been a process of refining the data to get a clear answer. After first identifying python3 processes as the general cause, I updated my metrics script to capture more detailed information, specifically the full command line of the top process.
This new data allows for a much more precise analysis. Instead of just guessing which Python scripts are the problem, I can now query the metrics database for exactly which commands are running when the server's total CPU usage spikes.
The Analysis
I ran a simple query to select the top CPU command line from any data point where the server's overall CPU usage was greater than 50%:
sqlite3 -header -csv server_metrics.db "SELECT top_cpu_cmdline FROM metrics WHERE cpu_percent == 100"
sqlite3 -header -csv server_metrics.db "SELECT top_cpu_cmdline FROM metrics WHERE cpu_percent > 50"
The output was a list of the commands that were at the top of the process list during these high-load moments. After tallying the results from the last 24 hours, two scripts are clearly responsible for the vast majority of the CPU spikes:
python3 sendNotifications.pypython3 monitorDeposits.py
These two background bots appeared with roughly equal frequency during the periods of high load. Interestingly, the metrics collection script itself showed up once, but its impact is negligible compared to the notification and deposit monitoring scripts.
Conclusion
This finally gives us a clear, actionable target. The high server load isn't coming from the web applications, as one might suspect, but from background daemons running constantly.
The next step is to dive into the code for sendNotifications.py and monitorDeposits.py. I'll be looking for any inefficiencies, such as loops that don't have a sleep interval, or resource-intensive calculations that can be optimized. This has been a great lesson in the importance of collecting the right data. It took a few tries, but we now have a precise answer.
As always,
Michael Garcia a.k.a. TheCrazyGM