If you haven't read any of our previous posts here, a litte bit of a recap first:

FlureeDB is a blockchain-based graph-database with a few unique properties for database technology. Apart from the fact that it's a database that uses blockchain technology as its foundation, implicitly storing full provenance for reliable and secure future auditing, it also has advanced access control built in that is configurable through a collection of built-in collections.

• _user : In flureeDB a user is basically a collection of ECDSA signing keys. Users are defined in the _auth collection. A user can have zero or more roles.
• _auth : An individual ECDSA signing key in FlureeDB ir represented by an _auth record. A signing key can have zero or more roles.
• _role : In FlureeDB, the concept of a role can be bound either to a user or to a signing-key. A role is a collection of access control rules.
• _rule: A rule in FlureeDB is an access control rule flowing from a role towards access to a collection or individual predicates. The validation of the rules is implemented by smart functions.
• _collection: Next to access controll flowing from roles, FlureeDB also allows for access controll flowing from collections. A collection can have a spec defined that binds it to smart functions for access control.
• _predicate: At a more granular level, FlureeDB also allows access controlls to flow from individual predicates with a predicate spec bound to smart functions.
• _fn: The _fn collection is a built in collection for smart functions. Smart functions are little snippets of code written in a subset of the Clojure language. These can be simple expressions comparing two strings, or these can be whole chunks of Clojure that query multiple collections and make decisions based on complex logic.

While access control in FlureeDB can be powerful and can help set up a highly secure database set-up, there is quite some modeling to keep up with and mistakes in schema updates could end up leading to a non-functional production systems if you don't take a lot of care. The Fluree Schema Scenario Tool (FSST), a community developed TDD tool for FlureeDB gives us the ability to throw some queries and transactions at a dockerized FlureeDB deployment, and use these as a test. Until yesterday, though, the way you could write tests was a bit archaic.

In our last post here, we discussed how to create your own Python domain API for aioflureedb using FlureeQL templates. In this post, we are going to look at how the latest release of the fsst TDD tool for FlureeDB leverages the aioflureedb domain-API into a more flexible TDD tool for python.

Extending the apimap for fsst

Let's revisit the apimap directory structure from our last post.

• roles
  • <role_1>.json
• query
  • <method_1>.json
  • <method_2>.json
  • <method_2>.xform
• transaction
  • <method_3>.json
• fsst
  • roles
    • root.json
    • test.json
  • query
    • <helper_1>.json
    • <helper_2>.json
    • <helper_2>.xform
  • transaction
    • <helper_3>.json
  • tests
    • <role_1>.py

The domain API is defined in terms of query and transaction templates bound together in groups by roles. YAs you see above, the role/query/transaction directory structure exists twice. Once at the top level where the public API is defined, and once in the fsst subdir where an extra API is defined that isn't part of the public API, but that is needed to implement testing modules in Python. There are two distinct role definitions at this level:

• test: for transactions and queries that ought to be run with auths of specific roles
• root: for transactions and queries that require priviledges beyond those of the roles being tested against.

root/role/user

Now for the tests and under what collection of privileges they run and what domain-API subset they have at their disposal. It is important to realize that the tests we'll be running will be testing a role bound sub-API, either from a role that is supposed to use that API, or from a role that isn't. It is quite likely that some operations when done in test contest are preparation operations that either need to be run with the specific role the API was defined for, or even need more privileges than that.

role	domain_api	test_api	description
user	role	test	User/auth with roles defined for the sub test, might contain API role, might not
role	role	test	User/auth with the API role (and only the API role) as role
root	role	root	The root-role auth

Keep this little table in mind when writing tests. Also realize that for many tests user and role auths will have the exact same role.

Writing a test module

The Python test modules for FSST define two types of tests, method tests that count towards test coverage and will be run in any test unless specifically disabled, and scenario tests what will only be run when role and user role are the same. And apart from these, ther is an optional prepare method, that isn't part of any actual test but can be used to run some preparing transactions and possibly run some queries to fill test-global object attributes.

So what does a test module look like. Let's start with the file name. A test module should have a file name equal to the name of the domain-API role being tested, with a .py file extension. The file should be located under apimap/fsst/tests. So if our API defines a role head_of_state, the python test module for head_of_state should be apimap/fsst/tests/head_of_state.py.

So what for the content? Let's start with a skeleton version:

class DomainApiTest:
    """Domain-API test class"""
    # pylint: disable=too-few-public-methods
    def __init__(self, test_env):
        """Constructor"""
        self.env = test_env

    async def prepare(self, domain_api, test_api):
        """Any prepare stuff we need to do as preparation with the propper role"""
        return None

    async def run_test_create_user(self, domain_api, test_api):
        """The test for create_user"""
        return None

    async def run_test_create_user_2(self, domain_api, test_api):
        """The test for create_user"""
        return None
   
    async def scenario1(self, domain_api, test_api):
        """Here we should be writing out whole scenarios for role to be involved in"""
        return True, "Not really OK, no scenario here"
  
    async def scenario1(self, domain_api, test_api):
        """Here we should be writing out whole scenarios for role to be involved in"""
        return True, "Not really OK, no scenario here"

You define a class named DomainApiTest that takes a single constructor parameter, and for every method the role sub-API defined you create zero or more methods. It's highly recomended you define at least one test method per API method. Just make it return None if there is no implementation yet. Defining it is a reminder that your coverage is lacking a test. You can define more than one test per API method by numbering, but make sure not to skip any numbers or fsst will prematurely think it's done.

You can optionally define numbered scenario methods in your test module. These are meant to test whole scenarios, what can be important. Scenario tests don't currently count to coverage in any way.

As you can see, except for the constructor, all test methods have the same function signature. You get a domain_api and a test_api method argument. These are both actually dict objects containing the actual domain API as a value in a way that coresponds to the table in the last section before this one, above.

So in order to use these, we need to pick using the string "user", "role" or "root".

An example:

async def prepare(self, domain_api, test_api):
        """Any prepare stuff we need to do as preparation with the propper role"""
        await test_api["root"].create_demo_role()()
        trans = domain_api["role"].create_user(
            fullname="Sheev Palpatine",
            email="the_sheev_man@naboo.gov",
            phone="0234230237",
            account_id="Tf999CRLL6FVagQVMFPEt7bzqSPh5Y00000",
            roles=["civil_servant","senator", "stealth_sith_lord"])
        await trans()
        return True

Now for tests, these work exactly the same. Make sure to use domain_api["user"] for the method actually being tested. Make sure to handle exceptions in your own code, but don't worry about exceptions from aioflureedb, these will get handled. On failure, you can return a boolean or optionally two values, a boolean and a descriptive string. Try not to keep any print statements in your test code when you are done.

Linking apimap test modules into an fsst run

Now that you have created your test module, you may think you are done, but you aren't yet. You need to link your test from your fsst schema source directory structure.

As you may remember from our first post, part of an ffst schema source directory structure could look something like this:

• fluree_parts
  • build.json
    • role_head_of_state
      • main.json
      • somesmartfunction.clj
      • test.json
      • test2
        • user.json
        • prepare.json
        • no.json
        • tno.json
        • yes.json
        • tyes.json

With the latest version of fsst, we have the possibility to replace part of this directory structure by linking it to our Python test module. This linking is done by placing a file named domain.json in the stage directory.

• fluree_parts
  • build.json
    • role_head_of_state
      • main.json
      • somesmartfunction.clj
      • domain.json

Note that both can coexist as well if you want them to. There could be good reasons for doing so. For now we'll assume domain.json replaces test.json and your python testing module depricates the test directory.

Now for the content of domain.json.

[
  {
      "doc": "Test if the role head_of_state can do everything defined in the domain API",
      "user": "president",
      "auth_roles": ["head_of_state"],
      "api_role": "head_of_state",
      "test": "head_of_state",
      "scenarios": true,
      "minimal_coverage": 80.0
  },
  {
      "doc": "Test if the role senator can't do any of the head_of_state transactions",
      "user": "test_senator",
      "auth_roles": ["senator"],
      "auth_roles_prepare": ["head_of_state"],
      "api_role": "head_of_state",
      "test": "head_of_state",
      "should_succeed": false,
      "should_succeed_exceptions":  ["give_speach_on_tv"],
      "warn_only": false,
      "warn_only_exceptions": ["do_crystal_meth"],
      "skip": ["impeach"],
      "minimal_coverage": 99.0
  }
]

Here we see an example of two tests linked to our head_of_state test module. For each test, fsst will create a new database, fill it using all of the stransactions for all fsst built stages of the current target up to and including the current stage directory, then it will create two user/auth combination with some of the above specs. One user/auth combination for fulfilling the "user" role and one for fulfilling the "role" role in the Python test module API as described above. In the example above, roles are bound to the _auth, but if needed you may define them bound to _user instead defining user_roles instead of auth_roles.

Let's walk throug the fields:

• user: Name for the primary user to create
• user_roles: List of roles this user should be assigned
• user_roles_prepare: Roles the secondary (unnamed) user should get.
• auth_roles: Roles that the _auth bound to the primary user should get
• auth_roles_prepare: Roles the _auth bound to the secondary (unnamed) user should get.
• api_role: The name of the sub domain-API
• test: The name of the test, usually the same as the api_role unless there are reasons to have multiple test modules for one api_role.
• scenarios: Boolean indicating if scenario tests should get run too.
• should_succeed: A boolean indicating the default for if individual tests are expected to succeed. True if undefined.
• should_succeed_exceptions: A list of API method names for what the default is inverted.
• warn_only: Boolean indicating the default for if individual tests giving non-expected test results should make the test run warn only (True) or fail hard (False). False if undefined.
• warn_only_exceptions: A list of API method names for what the default is inverted.
• skip: A list of API method names we'll skip for this test.
• minimal_coverage: The percentage of non-skipped methods that should have a defined and succeeding test. 100 if undefined.

So getting back to our example, our first test tests if head_of_state can succesfully do all the things a head_of_state is supposed to be able to do in terms of individual methods, and it also tests all defined scenario walkthroughs to test if these also work in context. We accept 20% of the thests not having been implemented, for now. The second test tests we test if a senator can succesfully give_speach_on_tv, but will fail to invoke any other of the head_of_state priviledges. Except for impeach, because, well, we don't care about that one. Finaly, for the do_crystal_meht method, that should fail, we opt for setting warn_only to true, meaning we will get a warning if it succeeds, but success won't fail the whole test set.

Generating an apimap artifact with fsst

Before we end this post, we need to talk a bit about the context within what these fsst tests would normally get run. In these days of test driven development, continuous integration and continous deployment, we want multi repository projects, if they end up failing, to fail early. If we make a change to our schema that would affect the correct working of a subsystem running with particular roles, we want to keep dependencies and deployments back at the old non-broken versions.

The tests we described so far can be done in a CICD pipeline, for example in GitLab, and can make deployment of the schema fail. But the api-map that untill now we considered part of the application should now be thought of part of a whole together with the schema. Both residing in the schema repo.

This however requires a way for us to get the relevant parts of the domain API to our sub-systems. For that purpose, two updates have been made:

• aioflureedb now supports a JSON artifact version of the apimap
• fsst has a new subcommand, apiartifact for generating such an artifact.

The later is invoked something like this:

./fsst apiartifact --roles head_of_state,senator apimap.json

The result is a single file JSON version od the apimap dir, containing the api map of just the named roles.

Use of apimap artefacts beyond aioflureedb and Python

For now the domain API implenemtation only exists in the Python FlureeDB library aioflureedb. But in theory nothing is stopping usage of the apimap artifact from getting used in other language FlureeDB libraries. The implementation might need to be partial on most languages though, unless these compile to JavaScript, but this is an issue we already discussed in the previous post for Python on non-Linux systems. The jsonata Python port runs only on Linux, making the usability of xform files in the API map currently constrained to that platform for Python. Without jsonata support though, portability should be trivial.

The apiartifact ffst subcommand currently comes with an experimental --js flag, embedding the json into a a javascript file instead of outputting the json directly.

./fsst apiartifact --js --roles head_of_state,senator apimap.js

Anyone reading this wanting to port the domain-API and the use of the apimap artifacts to other languages such as JavaScript or Clojure is invited to do so. Please drop a message though if you do so for JavaScript and/or TypeScript.

Concluding

We hope the adition of Python based schema and domain-API tests, that we created for our own work process will prove usefull to the FlureeDB community, and our sharing of our tooling for FlureeDB will incentify other FlureeDB users to do the same with their internal tools. FlureeDB is amazing technology combining blockchain and graph database technology with powerfull and flexible access contoll, but unlike many much older database technology it currently is a bit lacking in terms of tooling. We, as users of this technology, hope that sharing our own tooling as open source projects can help a tiny bit in improving the tooling situation and we hope others will follow.