Building an asyncsteem python web app with micro-transaction-based steem user authentication, Redis and Jinja2

Repository

https://github.com/pibara-utopian/asyncsteem

What Will I Learn?

• How to integrate micro-transaction based steemit user authentication into your asyncsteem/Twisted-based web app.
• How to chain multiple STEEM JSON-RPC operations asynchronously with asyncsteem
• Why and how to avoid synchronous operations when interacting with a database or template engine from a twisted web app.

Requirements

• Experience with Python
• A base understanding of the STEEM blockchain
• A base understanding of the HTTP protocol and cookies.
• Having read and understood Writing an asynchronous STEEM web-app in Python with asyncsteem

Difficulty Vs

• Advanced

Tutorial Contents

Where in the previous asyncsteem tutorial we learned how to build a web interface for visualizing what is happening on the STEEM blockchain in real time using asyncsteem, Twisted-web and VisJs, in this tutorial we are going to create a slightly more real-life web application with Twisted-web and Asyncsteem. We will use both the high-level blockchain event streaming API and the lower level rpcclient API that asyncsteem offers, and we are going to add in some more asynchonous operations by adding a bit of Redis to the mix. The application we will build in this tutorial is based on the ideas behind this script by @tcpolymath that finds inactive follows, combined with the core concept described in this post of using a cookie and a micro transaction as alternative for using SteemConnect for authentication of your steemit account with third-party services.

SteemConnect vs microtransactions

There is a diference between third-party services that require to know who you are and services that require to be able to act with part of your authority. The SteemConnect service provides facilities for the later. Think of steemconnect as a trusted third party between the user and the service provider when it comes to delegating part of the authority your account comes with. This can be very useful if for example if you write a service that votes or posts on behalf of the user. If there are twenty people like you running such services, the user doesn't need to trust his or her posting key to each of these services, but could simply delegate them to steemconnect, to then, using a delegate-by-proxy pattern, delegate a revocable and possibly attenuated part of the authority of his/her posting key to your service. Do this times ten, and it makes a lot of sense the user would rather delegate once to steemconnect than to ten different services.

This is all nice and dandy if the service you are running does indeed need delegated authority from its users. But what if it doesn't? What if the user has zero interest in services that require him or her to delegate authority to it, but would like to use services that merely wont to know for sure who he or she is?

In such a scenario, steemconnect is, I would argue, a horrible solution. The user must delegate authority to steemconnect that no service actually requires. To any security conscience steemit user, this should come as a complete horror. There are two relatively recent examples where users were asked to use steemconnect in situations where it would seem all the service needed was for the user to proof his.her identity yet ended up using requiring the user to delegate authority to steemconnect.

• The ByteBall airdrop
• The @steem-ua View your UA functionality.

Both services as it seems only need(ed) to know for sure that you as a user are in fact the owner of a given account and both services thought it justified to for that reason demand you as a user should delegate a large chunk of your authority to steemconnect.

Some people may think steemconnect, in fact, is the only option third party service providers have to authenticate steemit users, but that is incorrect. There is a relatively simple way to authenticate you as the owner of your account, and that is using a microtransaction. You are the only one who can initiate a payment from your account, and with some clever use of the memo field, it is almost trivially easy to set up authentication for a third party service using a 0.001 SBD or 0.001 STEEM microtransaction.

The four parts of the puzzle.

There are five parts to using microtransactions as a way to authenticate the users of your service:

1. Generating an unguessable random session id.
2. Using a private secret and a keyed hashing algorithm to create a session-id signature
3. Using HTTP Cookies to bind a signed session-id to a browser session
4. Using a microtransaction to bind the unsigned session-id to a steemit account.

Let's look at the first two steps, generating a session id:

#Blake2 keyed hashing for cookie signing
from pyblake2 import blake2b
#Random numbers for unique cookie id
from random import randint

class CookieUtil:
    def __init__(self,secret):
        #We use a secret for BLAKE2 signing of our cookies
        self.secret = secret.encode("ascii")
    def new_cookie(self):
        #Take a large random number
        random_buffer = hex(randint(0,1000000000000000000000000000000000000000000))[2:-1]
        #Create a signature for that number using our secret
        signature = blake2b(data=random_buffer,key=self.secret,digest_size=48).hexdigest()
        #Make a cookie string from the random number and its signature. Noone can make a valid cookie but us.
        random_string = str(random_buffer).encode("ascii")
        cookie = random_string + "-" + signature
        return cookie
    def valid_cookie(self,cookie):
        if cookie == None:
            return False
        #Split up the cookie into the original random number and its signature.
        [cookie_id,signature1] = cookie.split("-");
        cookie_buffer = cookie_id.encode("ascii");
        #Calculate the signuture of the random bit again.
        signature2 = blake2b(data=cookie_buffer,key=self.secret,digest_size=48).hexdigest()
        #The existing signature should be the same as the new one.
        ok = (signature1 == signature2)
        return ok

We use the randint function from the Python random module to generate a large random integer that we then convert to a hex string. As keyed hashing function we use BLAKE2B, the 64 bit variant of the fast and secure BLAKE2 hashing algoritm. We could alternatively have used HMAC with SHA2, but as BLAKE2 komes with keyed usage as a build in option, BLAKE2 seems like the perfect option for our goal. We than concattenate the random string and the signature together with a hyphen character as splitter.

In the valid_cookie method, we simply chop up our cookie value into its two parts again and calculate the keyes BLAKE2 hash once more. If the calculated hash equates the one we found in the cookie, then our cookie string is OK.

Now we move on to step three, the actual HTTP Cookie. To do this we make a trivial Twisted Web based mini web server.

A simple Twisted Web web server

#Jinja2 templates for HTML output.
from jinja2 import Environment, FileSystemLoader
#Some base twisted imports
from twisted.internet import reactor, endpoints
from twisted.logger import Logger, textFileLogObserver
from twisted.web import server,resource
from twisted.web.static import File
from twisted.web.util import redirectTo
#some basic imports
from os.path import join, dirname, realpath

#Simple twisted web resource that sets cookie if needed and redirects to functionality page.
class SetCookieIfNeeded(resource.Resource):
    isLeaf = True
    def __init__(self,cu):
        self.cu = cu
    def render_GET(self, request):
        oldcookie = request.getCookie("steemauth")
        #Only create a fresh new cookie if non was already set.
        if oldcookie == None:
            newcookie = self.cu.new_cookie()
            request.addCookie("steemauth",newcookie)
        #Redirect to service.
        return redirectTo("/steemauth",request)

#Our base resource and authentication check.
class SteemAuth(resource.Resource):
    isLeaf = True
    def __init__(self,cu, account,templates,reactor,log):
        self.cu = cu
        self.account = account
        self.templates = templates
        self.reactor = reactor
        self.log = log
    def render_GET(self, request):
        #If we are here, there should be a cookie
        oldcookie = request.getCookie("steemauth")
        #Check if the cookie is actually a valid one we created ourselv
        if cu.valid_cookie(oldcookie):
            #Take just the cookie id from the cookie.
            cookie_id = oldcookie.split("-")[0]
            rv = self.templates["askauth"].render(session=cookie_id)
            return rv.encode("ascii");
        else:
            newcookie = self.cu.new_cookie()
            request.addCookie("steemauth",newcookie)
            return redirectTo("/steemauth",request)

#Our mini authentication server with some basic helo world functionality.
class MiniAuthWebServer(resource.Resource):
    def __init__(self,cu, account,reactor,log):
        self.cu = cu
        self.children = ["/","/steemauth"]
        self.account = account
        self.reactor = reactor
        self.log = log
        thisdir = os.path.dirname(os.path.abspath(__file__))
        #We do our synchonous IO for loading Jinja2 templates at server start.
        j2_env = Environment(loader=FileSystemLoader(thisdir),trim_blocks=True)
        self.templates = dict();
        self.templates["askauth"] = j2_env.get_template('authenticate.tmpl')
        self.templates["helloworld"] = j2_env.get_template('hello-world.tmpl')
        self.templates["error"] = j2_env.get_template('error.tmpl')
    def getChild(self, name, request):
        if request.uri == '/steemauth':
            return SteemAuth(cu, self.account,self.templates,self.reactor, self.log)
        else:
            if request.uri == '/':
                return SetCookieIfNeeded(self.cu);
            else:
                return resource.NoResource()

#Get config with unique unguessable secret
mypath = os.path.dirname(os.path.abspath(__file__))
with open(mypath + "/mini-auth.json") as configjsonfile:
    conf = json.load(configjsonfile)
secret = conf["secret"]
#Set up logging
observer = textFileLogObserver(io.open(join(mypath,"mini-auth.log"), "a"))
logger = Logger(observer=observer,namespace="asyncsteem")
#Our CookieUtil is the main hub of our program
cu = CookieUtil(secret)
#Run our HTTP server on port 5080
root = MiniAuthWebServer(cu,account,reactor,logger)
factory = server.Site(root)
endpoint = endpoints.TCP4ServerEndpoint(reactor, 5080)
endpoint.listen(factory)
#Start twisted reactor.
reactor.run()

Most of this should look familiar from our previous tutorial, but let us focus on the new parts. We make some changes to what we have now to get things up and running with a simple service and we'll discuss the jinja2 template stuff afterwards, so lets focus on the actual cookie stuff for now. There are two resources we serve in order to make sure our user has a cookie:

• /: SetCookieIfNeeded, this is our root resource. This resource set the cookie if none is found and then redirects to the /steemauth resource. If there is no cookie, a new cookie is created and set.
• /steemauth : SteemAuth, this resource is assumed to be used with the cookie set. If there is no valid cookie, a new cookie is created overwriting any old cookie that may be there.

Adding Redis, asynchronously

Redis is an in-memory database that is very suitable for our current purpose of storing info on cookie usage and authenticated users. Even blocking synchronous operations in Redis are quite fast, it is important to resist our tendency to just intermix synchronous operations in our asynchronous event processing code. Remember, we don't know how many concurrent clients might be knocking on our door at any given time. It is important to know when you can call synchronous operations, for example at server construction time as we see with our Jinja2 template loading in the code above, and when not to, for example, while processing an asynchronous HTTP request. Fortunately, many databases can be approached asynchronously using Twisted (compatible) libraries in Python. For old-school relational databases, there is twisted.enterprise.adbapi and many other like Redis a bit of searching will usually yield an option to asynchronously interact with the database of your choice. For Redis the cyclone library provides us with the nonblocking asynchronous primitives we need.

#Asynchonous Redis library for storing some persistent data with.
from cyclone import redis

#This is our hub class that is used both from the blockchain streamer and from the web server.
class CookieUtil:
    def __init__(self,secret,logger):
        ...
        #Callback for setting the redis connection.
        def setDb(db):
            self.db = db
            self.logger.info("Connected to Redis")
        self.logger = logger
        #The local Redis is used to keep a persistent store of cookie to authenticated user mappings
        self.db = None
        d = redis.Connection()
        d.addCallback(setDb)
        #Try to keep a record of how far behind we are.
        self.last_blocktime = datetime.utcfromtimestamp(0)
    def new_cookie(self):
       ...
    def valid_cookie(self,cookie):
        ...
        if ok and self.db != None:
            #If the cookie is OK, keep a record of when this cookie was last used in our Redis server.
            k = cookie_id + "-lastvisit"
            v = str(time.time())
            d=self.db.set(k,v)
        return ok
    def authenticated_user(self,cookie_id):
        if self.db == None:
            return None
        #Look up if the cookie is linked to an authenticated user.
        k = cookie_id + "-steemauth"
        return self.db.get(k)
    def blocktime(self,tm):
        #Remember the last blocktime
        self.last_blocktime = tm
    def process_transfer(self,frm, memo):
        if self.db != None:
            #If we register a potentially authenticating transfer in the blochchain,
            #look in Redis to see if we gave the memo out as cookie id.
            k1 = memo[:40] + "-lastvisit"
            ltp = self.db.get(k1)
            def process_last_time(lasttime):
                #If the last time isn't None, this is a cookie id we know
                if lasttime != None:
                    self.logger.info("Known cookie id in transaction.")
                    #Check if this cookie id has already been registered to a steemit account
                    k2 = memo[:40] + "-steemauth"
                    acp = self.db.get(k2)
                    def process_account(account):
                        #If the memo holds a valid cookie id that has never been registered to any steem account before
                        if account == None:
                            #Store the cookie steem-account mapping.
                            d=self.db.set(k2,frm)
                            self.logger.info("Cookie has now been linked to a steemit user.")
                        else:
                            self.logger.info("This particular cookie id was already registered to a steem user before.")
                    #Above is asynchounous callback for result from Redis
                    acp.addCallback(process_account)
                else:
                    self.logger.info("Transaction without valid cookie id")
            #Above is asynchounous callback for result from Redis
            ltp.addCallback(process_last_time)
        else:
            self.logger.error("No Redis Db connection yet while processing transfer.")
    def behind_string(self):
        #Return how many time has passed since last block we saw.
        behind = datetime.utcnow() - self.last_blocktime
        return str(behind)

We start off making some changes to our CookieUtil class. Note that the set method is basically fire and forget so it is easy to overlook the fact these methods are asynchronous. Note also the authenticated_user method looks deceptively like something that might return a string. We will come back to that one below, now we look at the process_transfer method that we will bind to an asynchsteem bot class soon.

If we detect a transfer, we look up in our Redis database to see if the memo field matches a session id we know, and if it does, we check if the session id has not previously been claimed. Then if it hasn't, we can safely bind the user doing the transfer to the session id we found in the memo field.

Notice we have created a chain of callbacks. We do a get, an other get and then a set, all asynchonously. The last set is fire and forget as we saw before, but the gets all register a callback that gets called with the results.

asyncsteem transfers bot code

#Asyncsteem import for talking to JSON-RPC steem API nodes.
from asyncsteem import ActiveBlockChain, RpcClient

#This is the basic asyncsteem bot.
class TransferStream:
    def __init__(self,cu,account):
        self.cu = cu
        self.account = account
    #We listen only to transfers
    def transfer(self,tm,transfer_event,client):
        frm = transfer_event["from"]
        to = transfer_event["to"]
        memo = transfer_event["memo"]
        #For keeping track of if we are behind on the blockchain.
        self.cu.blocktime(tm)
        #Only process transfers to one single account, our owner.
        if to == self.account:
            self.cu.process_transfer(frm, memo)

...
account = conf["account"]
...
#Asyncsteem blochchain streamer
blockchain = ActiveBlockChain(reactor,log=logger,nodelist="default")
#Our CookieUtil is the main hub of our program
cu = CookieUtil(secret,logger)
#Instantiate our bot and link it to our blockchain
steembot = TransferStream(cu,account)
blockchain.register_bot(steembot,"flag_stream")
...

So now it is time to link in our main asyncsteem stuff, We define a blockchain streaming bot that processes transfer events and call the method we just discussed at every transfer to our configured owner account.

So now we have our code for authenticating users with microtransactions, let us add some hello-world code to show we mean business.

Some authentication using service.

We discussed @steem-ua before when talking about services that should benefit from using micro-transaction based authentication. Now, however, we look at @steem-ua from a more up note angle. In this post @tcpolymath described how, for @steem-ua users, it might be useful to find and unfollow inactive follows. In his script @tcpolymath used a 40-day threshold for the newest posts for finding inactive follows. We change our web server code and add in some extra asyncsteem code to get our authenticated service to do something useful. List the stale follows of our authenticated user. Again we do this in a way that chains a number of callbacks.

#Main function this server provides to authenticated users. List stale followed accounts.
#This functionality was borowed from a script by @tcpolymath.
def report_on_followers(request,steemaccount,cu,reactor,log,template,etemplate):
    def process_following(event,client):
        #Process the followed accounts response from the API node.
        def process_accounts(event2,client2):
            #Make a list of all accounts that didn't post for more than 40 days.
            stale_accounts = []
            for account in event2:
                name = account["name"]
                inactive = (datetime.utcnow() - dateutil.parser.parse(account["last_post"])).days
                if inactive > 40:
                    ias = str(inactive)
                    stale_accounts.append({"name": name , "inactive" : ias})
            #Use jinja2 to return a simple report
            html = template.render(authenticated = steemaccount, results=stale_accounts, behind=cu.behind_string())
            request.write(html.encode("ascii")) 
            request.finish()
        def process_accounts_err(errno,msg,client):
            html = etemplate.render(message=msg)
            request.write(html.encode("ascii"))
            request.finish()
        #Make a list with the accounts that we follow.
        accountlist = [];
        for entry in event:
            accountlist.append(entry["following"])
        #Request account info on all followed accounts.
        opp2 = client.get_accounts(accountlist)
        opp2.on_result(process_accounts)
        opp2.on_error(process_accounts_err)
    def process_following_err(errno,msg,client):
        html = etemplate.render(message=msg)
        request.write(html.encode("ascii"))
        request.finish()
    #STEEM JSON-RPC client.
    client = RpcClient(reactor,log,stop_when_empty=False,rpc_timeout=40)
    #Start by querying up to 1000 folowers of the authenticared account.
    opp = client.get_following(steemaccount,"","blog",1000)
    opp.on_result(process_following)
    opp.on_error(process_following_err)
    client()

#Our base resource and authentication check.
class SteemAuth(resource.Resource):
    isLeaf = True
    def __init__(self,cu, account,templates,reactor,log):
        self.cu = cu
        self.account = account
        self.templates = templates
        self.reactor = reactor
        self.log = log
    def render_GET(self, request):
        #If we are here, there should be a cookie
        oldcookie = request.getCookie("steemauth")
        #Check if the cookie is actually a valid one we created ourselv
        if cu.valid_cookie(oldcookie):
            #Take just the cookie id from the cookie.
            cookie_id = oldcookie.split("-")[0]
            #Get defered for a possible authenticated user from this cookie id.
            sa = self.cu.authenticated_user(cookie_id)
            def get_steemacount_from_redis(steemaccount):
                if steemaccount == None:
                    #If the user isn't authenticated yet, return authentication page.
                    self.log.info("Cookie not assigned to any steemit user")
                    rv = self.templates["askauth"].render(account=account,
                            session=cookie_id, 
                            behind=cu.behind_string()
                        )
                    request.write(rv.encode("ascii"))
                    request.finish()
                else:
                    #If the user is authenticated, run core business logic as borowed from @tcpolymaths script.
                    self.log.info("Cookie matches authenticated steemit user")
                    req = request
                    template = self.templates["following"]
                    etemplate = self.templates["error"]
                    report_on_followers(req,steemaccount,cu,self.reactor,self.log,template,etemplate)
                return
            #If there is a Redis problem, sa will ne None
            if sa == None :
                html = self.templates["error"].render(message="No connection to local Redis database")
                return html.encode("ascii")
            else :
                #Let the defered callback above handle the result from the Redis lookup
                sa.addCallback(get_steemacount_from_redis)
                return server.NOT_DONE_YET
        else:
            newcookie = self.cu.new_cookie()
            request.addCookie("steemauth",newcookie)
            return redirectTo("/steemauth",request)

#Our mini authentication server with some basic helo world functionality.
class MiniAuthWebServer(resource.Resource):
    def __init__(self,cu, account,reactor,log):
        ...
        self.templates["following"] = j2_env.get_template('following.tmpl')
        ...
    def getChild(self, name, request):
        ...

As we mentioned before, the return value from authenticated_user isn't a string. It is a deferred that we need to add a callback to. A callback that processes the actual return string, or None value if no authenticated user is found. If in our case an authenticated user is found, the report_on_followers function gets called that does the same thing @tcpolymath's script did.

This function uses asyncsteem in a different way. It spawns a new asynchonous RPC client and creates a chain of two asynchonous JSON-RPC calls to the STEEM API nodes:

• get_following : Getting a list of accounts the authenticated user is following.
• get_accounts: Get meta info, including last post date for the list of users returned from get_following.

Notice also how this chain is handled by the web resource. Instead of returning content from render-GET, the render_GET returns the special value NOT_DONE_YET. This leaves the responsibility for the HTTP response to the end of the callback chain using the write and finish methods on the request object.

jinja2

There is still one loose end to our code. Jinja2. Jinja2 is a HTML template engine that allows us to easily map data structures onto HTML content. Our stale follows teomate liiks something like this.

<!DOCTYPE html>
<html>
  <head>
    <title>Mini-Auth demo asyncsteem</title>
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <link href="http://netdna.bootstrapcdn.com/bootstrap/3.0.0/css/bootstrap.min.css" rel="stylesheet" media="screen">
    <style type="text/css">
      .container {
        max-width: 500px;
        padding-top: 150px;
      }
    </style>
    <meta http-equiv="refresh" content="90">
  </head>
  <body>
    <div class="container">
        <p>
      <h3>Inactive accounts followed by you ({{authenticated}})</h3>
          <ol>
          {% for result in results %}
             <li><A HREF="https://steemit.com/@{{ result.name }}">@{{ result.name }}</A>  {{ result.inactive }}  days since last post.
          {% endfor %}
          </ol>
        </p>
        <p>Currently running <b>{{behind}}</b> behind on the blockchain.p>
    </div>
    <script src="http://code.jquery.com/jquery-1.10.2.min.js"></script>
    <script src="http://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
  </body>
</html>

A detailed discussion of jinja2 falls outside of the context of this tutorial, but as the for loop in this example shows us, jinja2 allows us to map nested data structures to html structures.
The result of our script will now look something like this:

Curriculum

Writing a simple STEEM statistics-script in Python with asyncsteem
Writing an asynchronous STEEM web-app in Python with asyncsteem

Proof of Work Done
The mini-auth example script , that is part of the sample scripts in the asyncsteem repo, was written for this tutorial. Want to see this code in action, check this demo site.