Analysis : Posts created per day and Organic Ratio

Repository

Introduction

Hearing the call of @eastmael in Utopian discord channel, I started to prepare necessary scripts and extract data from Steem block-chain.

This analysis is searching for answers to the following questions :

How many posts are posted to Steemit in an average day?
How many of these posts are organic ( no bid-bot upvoted )

Scope of Analysis

Analysis date: 27 May 2018
Analysis timeframe: 2018-05-21T13:42:39 --> 2018-05-22T13:30:15
Post Quantity: 40 000 posts
Bots list: All the upvote bots registered to SteemBotTracker by @yabapmatt (99 bots)
bot list
All times are in UTC.

Tools

Finding the posts

A javascript code is written using steemjsAPI to find the posts that is created 5 days ago.
5 days is chosen because :

If the posts are paid, the script used doesn't show them, so they must be pending payout status posts
If the posts are recent, the author can still buy votes from bid-bots and this will give a false result
Mainly none of the major bid-bots accept posts older than 3.5 days, so 5 days is safe.

function search(){
     var query = {
        start_author:"adsactly", 
        start_permlink:"adsactly-tech-news-does-increased-access-to-technology-increase-societal-inequality",
        limit: 100
    };
    steem.api.getDiscussionsByCreated(query, function(err, result) {
        console.log(err, result);
        console.log(query);
        postarray=postarray.concat(result);
        calculate_rest();
    });
}

With above script, starting from any post that is 5 days old, following 100 posts are taken sorted accroding to their posting time.

It is observed that 100 posts are created in approximately 3.5 minutes.
So to cover a range of 24 hours, it is calculated that approximately 40 000 posts must be analyzed.

Below script is written to extend the search to 40 000 posts ( we can get posts in packs of 100's so 400 API request have been sent )

 function calculate_rest(){
        count=(i*100)-1;
        if(i<400){
            



            var query = {
                tag: "",
                start_author:postarray[count].author,
                start_permlink:postarray[count].permlink,           
                limit: 100
            };

            steem.api.getDiscussionsByCreatedAsync(query, function(err, result) {
                //console.log(err);
                if(!err){
                    postarray=postarray.concat(result);
                    i+=1;
                    calculate_rest();
                }else{
                    calculate_rest();
                }
                
                
            });   
            
    }else{calculate_bots();}
    console.log(postarray);     
        }

It has been some time to process this in javascript.
As the 40 000 posts are received it is observed that the calculation was valid, 40 000 posts covered exactly 24 hours time frame.
The first post is at 2018-05-21T13:42:39 and the last post is at 2018-05-22T13:30:15.
Missing only 12 minutes in 24 hours.

The result is put in the Google spreadsheet

Eliminate the posts that used bid-bots

Using the bot list all the posts are filtered if any of the voter is in the botlist with below code.

function calculate_bots(){
var author=[];
var permlink=[];
var time=[];
var activevotes=[];
var control;
    console.log(postarray);
      for (let i = 0; i < postarray.length; i++) { 
      control=0;
      author.push(postarray[i].author);
      permlink.push(postarray[i].permlink);
      time.push(postarray[i].created);
         for (let j = 0; j < postarray[i].active_votes.length; j++) {
         activevotes = postarray[i].active_votes; 
         if (bots.includes(activevotes[j].voter)) {
            control = 1;
          }
         
          }
         if (control==0){
            postarraybot.push(postarray[i]);
         }
      
    }
    console.log(author);
    console.log(permlink);
    console.log(time);
    console.log(postarraybot);
    
}

Now we have the organic posts list that hasn't been upvoted by any of the bid-bots.

Results

Analyzing the 40 000 posts, the following results are found

40 000 posts are posted in 24 hours (- 12 minutes)
40 000 posts are posted by 20 362 users ( average 2 posts per author )
The *rush-hours and slow-hours for posts are as follows

Rush-hours are from 15:30 to 19:30 UTC with 2000+ posts / hour
Slow-hours are 22:30 to 02:30 UTC with less than 1400 posts/hour
With an average of 2 post/day the distribution of posting frequency of uses are as follows.