In this blog post we want to discuss the use of the 'dockertest' subcommands of the Fluree-Schema-Scenario-Tool, that we shall further refer to as fsst. The ffst tool is a multi-functional tool for development with the blockchain-based graph-database FlureeDB.
As this is our very first blog post discussing either the Fluree-Schema-Scenario-Tool (fsst), or any FlureeDB related subject for that matter, before we get to the main topic of this post, we'll start of this blog post with a short introduction about the fsst tool, the aioflureedb python library, and the FlureeDB blockchain-based graph database.
FlureeDB
FlureeDB is a graph-database with very interesting features and an even more interesting design philosophy. It is a general purpose graph-database that runs on top of blockchain technology. You can query the database and run mutating transactions on the database using signed queries and transactions, and the database provides access control paradigms that fit to this signing regime that are both powerfull and flexible, but at the same time not always easy, especialy when you first start using the technology. The data-driven securityparadigm that FlureeDB allows for means that (with some work) you can express a wide range of access controls flowing from the ownership of private keys by users. To set this up, a flureeDB database is populated with a schema that includes smart functions. From a contract based testing perspective, you can look at the combinations of schema-creating transactions and fmart-function-creating transactions as a strict (or less strict depending on your needs) contract for your database. As we will see later on, it is the development and testing of this contract that fsst concerns itself with the most. Smart functions in FlureeDB are written in a subset of the Clojure language, so while most transactions for building up a database schema are in the realm of database experts, the addition of a full programming language woven into the very fabric of the database it's access control system makes FlureeDB a piece of technology that requires database experts and software developers to work closely together.
Docker
A second piece of technology that is relevant for fsst is docker. Docker is a virtualization system that allows you to run what is basically a small Linux operating system container with some pieces of custom software to perform a specific task. A container that runs on top of the actual operating system . In most cases, that OS would be Linux as well, but it's also possible to run Docker on top of other operating systems. Next to that, docker is at the foundation of multiple *-As-A-Service stacks populating the cloud-computing ecosystem, and is used as well in software development as important part of CI/CD solution stacks. One of those stacks relevant for fsst, as we will later see, is GitLab.
aioflureedb
The fsst tool is written in Python and uses the asynchonous FlureeDB API as provided by the aioflureedb library. While from a user perspective aioflureedb is currently pretty irrelevant to the usage scenarios of fsst (it's just there), in the near future, the underlying templates and transformations that make up the new aioflureedb domain-api interface, will make aioflureedb a more explicit part of the stack. As this post is partially an introductory post, talking about this aspect of the stack here and now should aid in understanding this and future blog posts in their context.
The Fluree-Schema-Scenario-Tool (fsst)
Thusfar the wider introduction. Time now to zoom in to the fsst tool. The fsst tool at this point in time is a single-file script that defines a wide range of sub-commands. Often the functional sub-commands come in three flavours. To clarify, let's put the sub-commands in a little table
| explicit server | docker-host | docker-container | use |
|---|---|---|---|
| artifact | artifact | artifact | Compile an artifact file but do not deploy |
| version | version | version | Get the version ofthe fsst tool |
| test | dockertest | guesttest | Run the schema/smart-function unit-tests |
| deploy | dockerdeploy | guestdeploy | Compile and deploy the FlureeDB schema |
| artifactdeploy | Deploy a previously build artifact | ||
| versioncheck | Check that the host version of fsst is compatible with the docker guest version | ||
| dockerparams | Retreive port and root-user privkey info from a running docker instance | ||
| dockerstart | Start fluree only, no schema deployment or test | ||
| dockerstop | Stop a previously started fsst/flureedb docker instance |
As for this blog post we only look at the dockertest command, we won't go into the rows of the above table, but we should discuss the columns.
The first column has the fsst sub-commands that you can use on a running FlureeDB instance, or for the first two commands, even without. This running FlureeDB could be a FlureeDB running on your local PC or on a server, or it could be a FlureeDB that was started earlier using the dockerstart command. If you use these commands, you will often need to have knowledge of the private root-key of the fluree instance running.
The second column are commands you will use if you are on a development system that has docker, and want to use one of the fsst docker images to run fluree and a guest fsst on.
As you can see in the image above, fsst on the host system communicates with fsst on the guest system, by starting fsst inside of the guest system, passing specific parameters transparently. The 3th column of commands contains the commands that can be run with the guest fsst running inside of the docker container. This are situations where the fsst commands on the host get mapped to fsst commands for the guest, but also situations where the docker image runs as part of a CI/CD pipeline.
Installing aioflureedb and other dependencies.
So far for introduction. Time to get our hands dirty and see how things actually work. We start by installing aioflureedb and some other dependencies. This will work best if you are on Linux, as all dependencies will be available. If you are on Windows, you likely won't be able to install bitcoinlib and as a result a few fsst commands will end up not working.
First, make sure your system is running docker, and if it isn't, install docker on your development system. Also make sure you have Python (>= version 3.7) running on your system.
Next, we install our Python dependencies.
We assume your python binary is named python3. As this post is about the dockertest sub command, our most essential dependency is the python docker module.
python3 -m pip install docker
Next, see if you aren't on MS-Windows, it is a good idea to also install the following three modules.
python3 -m pip install aioflureedb
python3 -m pip install base58
python3 -m pip install bitcoinlib
This will enable the explicit server sub-commands. On Windows, currently, bitcoinlib seems to have trouble installing on many configurations. For now fsst disables too many sub-commands when bitcoinlib is unavailable, we plan to fix this in a future version. For now, if any of the above three dependencies is missing the whole explicit server column of fsst subcommands will get disabled.
Getting and updating the fsst docker images
You can fetch docker images containing both fsst and flureedb from docker hub here. Before we go on, it is important to discuss an image picking policy, so you can choose the one most suitable for your working process. The fsst/fluree docker images are pushed with multiple tags. Some tags are unambiguous, and designate the same image no matter when you pull them, while other tags may or will be reused by newer images. Note that if you pull a tag like stable or beta and later want to make sure you have the latest stable or the latest beta, you will need to delete the old one first and fetch it once more. On the other hand, if you fetch an unambiguous tag name, the name of the tag will be long, and you may need to type this long name into your fsst commands more than just once, something that might become cumbersome. So pick your tag using policy and pick it carefully.
Let's assume you pick the unambiguous tag names and want to ge the latest stable docker image. You check out the dockerhub tags page for fsst and look for stable.
At time of writing, we see this is a 7-day-old image with digest dd2ec94ec3bc. Now scroll up to the first occurence of this hash.
This means, at this point in time, the unambiguous tag for stable is v0.2.7-fluree-0.16.1, Version 0.2.7 of fsst together with version 0.16.1 of FlureeDB. If we do the same for beta we get the v0.2.7-fluree-v1.0.0-beta16 for the latest beta.
So now we can fetch these two:
docker pull pibara/fsst:v0.2.7-fluree-0.16.1
docker pull pibara/fsst:v0.2.7-fluree-v1.0.0-beta16
Or in the case we rather have short names and don't mind deleting the old images first if we want to upgrade, we can use the short tags:
docker pull pibara/fsst:stable
docker pull pibara/fsst:beta
Installing and updating fsst
Now for the host fsst. To first fetch the latest version of fsst, you need to clone the fsst repository from github.
git clone https://github.com/pibara/fluree-schema-scenario-tool.git
Now if you want to update, you go to the fluree-schema-scenario-tool directory and run:
git pull origin master
That's it.
Now you have to make a choice again. You can either add the fluree-schema-scenario-tool to your binaries execution paths specification, so the fsst tool will be available everywhere, or you can create an alias. As the best way for doing this is OS and even distribution dependent, we'll won't be describing it in this blog post.
On linux, adding a line to your .bashrc or .profile is most likely the way to go.
Alternatively, you can just designate the fsst tool completely for invocations.
~/fluree-schema-scenario-tool/fsst --help
Using the fsst 'dockertest' subcommand for unit-testing a schema build.
fluree_parts directory tree structure
Now that we have both the host version and two different fluree versions with the guest version of fsst installed, it is time to use fsst. There is a very minimal example directory in the fsst repo named demo-schema-parts we will be using to show how things work. Let's walk through the content of this directory and see what will be going on.
The first file we see is the file build.json. The build file defines a single target named default, that consists of two stages.
{
"default": [
"roles",
"there_can_be_only_one"
]
}
The first stage doesn't have any tests, and thus is defined in a file (and not a directory)
[
[
{
"_id": "_role",
"id": "headofstate"
},
{
"_id": "_role",
"id": "minister"
}
]
]
The stage contains an array with inside a single FlureeQL transaction that creates two entries in the _role collection.
The second stage is defined in a directory, there_can_be_only_one. This directory contains a file main.json that just like the first transaction file contains transactions.
[
[{
"name" : "roleCount",
"code_from_query" : {
"select": ["(count ?auth)"],
"where":[
["?roles", "_role/id", "PARAM"],
["?auth", "_auth/roles", "?roles"]
]
},
"doc" : "Retrieve count for a specific role",
"_id" : "_fn$roleCount",
"params" : ["myRole"]
}],
[{
"name" : "roleAssignedHeadOfState?",
"code" : "(<= (nth (roleCount \"headofstate\") 0) 1)",
"_id" : "_fn$roleAssignedHeadOfState",
"doc" : "Is the role HeadOfState Assigned to 0 or 1 Auths?"
}],
[{
"specDoc" : "HeadOfState is already assigned to an Auth",
"spec" : [["_fn/name", "roleAssignedHeadOfState?"]],
"_id" : ["_predicate/name", "_auth/roles"]
}]
]
Note that the first transaction isn't exactly FlureeQL. The code_from_query key in the roleCount function will get expanded by fsst to a code field with a correctly escaped clojure string. More on that in a future blog post.
per-stage test definitions
Now we see there is also a file named test.json that contains an array of test names.
[
"test1"
]
This defines that the subdir test1 contains tests for this part of the schema. Let's look what is inside of the test1 dir.
The first relevant file we see in the test1 dir is the user.json file.
{
"keys" : [
{
"account-id" : "TeyTH5RdmweiWzrkEzwc5vYnPwqZDSh8Ng6",
"private" : "bb19048fbfc8f6a7f02b336048fa8e5aba1fb87f55893d40ff1ba375b90a8398"
},
{
"account-id" : "Tf5p2VyQgYJQ6p8suVkX633BD4kQgsp29FC",
"private" : "b11648b8f45a2cae682cbc549c35b065dff32794562b0765874e911e31163469"
}
],
"no" : 0,
"yes" : 0,
"tno": 0,
"tyes": 0
}
This file defines two keys to be used in tests with an index of 0 and 1 respectively. After that, it defines that all of the tests for all of the test types (no/yes/tno/tyes) will be run using the key at index 0.
The next relevant file, the prepare.json file, does a few extra transactions that will be needed to run the actual tests.
[
[
{
"_id": "_auth",
"id": "TeyTH5RdmweiWzrkEzwc5vYnPwqZDSh8Ng6",
"roles": [["_role/id","root"]]
}
],
[
{
"_id": "_auth",
"id": "Tf5p2VyQgYJQ6p8suVkX633BD4kQgsp29FC",
"roles": [["_role/id","root"]]
}
]
]
Two entries are created in the _auth collection. Two _auth's matching the two accounts in user.json.
The yes.json and the no.json files are simply empty lists. These could contain FlureeQL queries that should either result a non empty result (yes.json) or an empty result (no.json).
There is a file tyes.json though.
[
[
{
"_id": ["_auth/id", "TeyTH5RdmweiWzrkEzwc5vYnPwqZDSh8Ng6"],
"roles": [["_role/id","headofstate"]]
}
]
]
And a file tno.json
[
[
{
"_id": ["_auth/id","Tf5p2VyQgYJQ6p8suVkX633BD4kQgsp29FC"],
"roles": [["_role/id","headofstate"]]
}
]
]
Both contain a single transaction assigning the role headofstate to one of the _auth records.
This is our test, basically. If the smart functions defined in the there_can_be_only_one stage work correctly, the first assignment of the role should succeed but the second one should fail.
basic usage
Now let's test it.
./fsst dockertest --dir demo-schema-parts --target default --tag v0.2.7-fluree-0.16.1
In the above example we run fsst from within the fsst project directory. We specify three subcommand arguments:
- dir : The directory where our fluree_parts schema build files are.
- tag : The tag of the docker image we want to test against.
- target : The target we want to test.
The result from running this command will look something like this:
COMMAND: ./fsst guesttest --target default --network test12432
IMAGE: pibara/fsst:v0.2.7-fluree-0.16.1
WARNING: - docker python lib not found.
- docker related commands disabled.
# waiting for default-private-key.txt to appear
# waiting for default-private-key.txt to appear
Started FlureeDB and got createkey from newly created keyfile
- Database: test12432/there-can-be-only-one
- collecting transactions from build subdirs
Going through first 1 stages
- roles
- there_can_be_only_one
- processing schema transaction sub-set
- ok, completed 5 transactions on test12432/there-can-be-only-one
- running test scenarios
- SCENARIO: test1
- prepare.json
- Ran 2 database transactions
- yes.json
- Ran 0 database queries
- no.json
- Ran 0 database queries
- tyes.json
- Ran 1 database transactions
- tno.json
- NOTICE: Expected error in NO transaction
: 400 db/invalid-tx HeadOfState is already assigned to an Auth Value: 123145302311912
- Ran 1 database transactions
- clean
- file not found, skipping: clean
- Ran 0 database transactions
- 1 tests completed
So what are we seeing? The first line show us what command the guest fsst is given. The network argument that we didn't specify on our commandline is auto generated based on a modulus of a low granulatity version of the current time. It is possible to specify it explicitly if we want.
The next line shows us what docker image we are running with.
After that, two spurious warnings. from within the docker image. These are a bit confusing, and we'll try to remove them in a future version.
In the next lines, we see that fsst has started up FlureeDB, and waits for a bit untill the file default-private-key.txt appears. fsst will need this file in order to populate the databases and run the tests.
Then we get to the interesting bit. For every stage that defines tests, fsst creates a database, that it fills with the transactions from all the stages up to and including the one being tested. The name of the database will be the name of the stage or something verry similar.
We see the part of the schema (in this case our entire schema) consists of five transactions across two stages.
Now we move on to the line stating running test scenarios. We see in the logging that there are two transactions in the prepare phase, zero in both the query phases, and one in each of the transaction phases. And finaly zero in the cleanup phase.
We see both the tests succeeded, and we see the reason the tno transaction failed.
There are a few more arguments that we might give to the dockertest subcommand:
- network : FlureeDB network name.
- verbosefluree : Dont redirect flureedb stdout/stderr to /dev/null
- verboseerrors : Add verbose errors to rules without an error defined.
verbosefluree
One interesting one is the the --verbosefluree option. It will make it that the test results and fluree logging are combined on the console output. While the output can be a bit messy, it could provide extra and usefull info when trying to track down a problem.
Let's see what happens if we use it.
./fsst dockertest --dir demo-schema-parts --target default --tag v0.2.7-fluree-0.16.1 --network testresults --verbosefluree
The (messy) result:
fluree-0.16.1 --network testresults --verbosefluree
COMMAND: ./fsst guesttest --target default --network testresults --verbosefluree
IMAGE: pibara/fsst:v0.2.7-fluree-0.16.1
WARNING: - docker python lib not found.
- docker related commands disabled.
# waiting for default-private-key.txt to appear
/usr/bin/java
Java version 11.0.12.
No properties file specified. Using default properties file fluree_sample.properties.
FlureeDL starting with properties file: fluree_sample.properties
FlureeDL starting with java options: -Xms1g -Xmx1g -XX:+UseG1GC -XX:MaxGCPauseMillis=50 -Dfdb-api-port=8090
INFO f.db.server-settings - Properties file fluree_sample.properties successfully loaded.
INFO fluree.db.server - Starting FlureeDB in mode: dev
INFO fluree.db.server - Starting with config: {:fdb-pw-auth-jwt-secret nil, :fdb-ledger-private-keys nil, :fdb-group-catch-up-rounds 10, :fdb-pw-auth-enable true, :fdb-group-private-key nil, :fdb-api-port 8090, :fdb-join? false, :fdb-group-this-server myserver, :fdb-mode dev, :fdb-group-timeout 2000, :fdb-stats-report-frequency 1m, :fdb-pw-auth-signing-key nil, :fdb-api-open true, :fdb-settings nil, :fdb-group-private-key-file default-private-key.txt, :fdb-consensus-type raft, :fdb-ledger-port 9790, :fdb-storage-type file, :fdb-pw-auth-secret fluree, :fdb-storage-file-directory data/ledger/, :fdb-group-snapshot-threshold 200, :fdb-memory-reindex-max 2mb, :fdb-group-heartbeat nil, :fdb-pw-auth-jwt-max-exp 1y, :fdb-ledger-servers nil, :fdb-encryption-secret nil, :fdb-license-key nil, :fdb-group-log-history 5, :fdb-memory-reindex 1mb, :fdb-group-log-directory data/group/, :fdb-pw-auth-jwt-max-renewal 1y, :fdb-memory-cache 200mb, :fdb-group-servers myserver@localhost:9790}
INFO fluree.db.server - JVM arguments: {:jvm "OpenJDK 64-Bit Server VM", :input ["-Xmx1g" "-Xms1g" "-XX:+UseG1GC" "-XX:MaxGCPauseMillis=50" "-Dfdb-api-port=8090" "-Dfdb.properties.file=fluree_sample.properties" "-Dfdb.log.ansi" "-Dlogback.configurationFile=./logback.xml"]}
INFO fluree.db.server - Memory Info: {:used 0.2 GB, :committed 1.0 GB, :max 1.0 GB, :init 1.0 GB}
INFO fluree.db.peer.http-api - Starting web server on port: 8090 with an open API.
INFO fluree.db.peer.http-api -
INFO fluree.db.peer.http-api - http://localhost:8090
INFO fluree.db.peer.http-api -
INFO f.d.l.consensus.raft - Ledger group leader change: {:cause :become-leader, :server myserver, :new-leader myserver, :old-leader nil, :event :become-leader, :message This server, myserver, received the majority of the votes to become leader. New term: 1, latest index: 0.}
INFO fluree.db.server - This server just became leader of the raft group.
INFO f.d.l.consensus.raft - Brand new Fluree instance, establishing default shared private key.
Started FlureeDB and got createkey from newly created keyfile
- Database: testresults/there-can-be-only-one
- collecting transactions from build subdirs
Going through first 1 stages
- roles
- there_can_be_only_one
INFO fluree.db.peer.http-api - Request exception: {:status 400, :error :db/unavailable, :message Database testresults/there-can-be-only-one is not currently available. Status is: :initialize.}
INFO fluree.db.peer.http-api - Request exception: {:status 400, :error :db/unavailable, :message Database testresults/there-can-be-only-one is not currently available. Status is: :initialize.}
INFO fluree.db.peer.http-api - Request exception: {:status 400, :error :db/unavailable, :message Database testresults/there-can-be-only-one is not currently available. Status is: :initialize.}
INFO fluree.db.peer.http-api - Request exception: {:status 400, :error :db/unavailable, :message Database testresults/there-can-be-only-one is not currently available. Status is: :initialize.}
INFO fluree.db.peer.http-api - Request exception: {:status 400, :error :db/unavailable, :message Database testresults/there-can-be-only-one is not currently available. Status is: :initialize.}
INFO f.db.ledger.indexing - Index Update begin at: 2021-11-05T15:33:05.392810Z {:network testresults, :dbid there-can-be-only-one, :t -2, :block 1, :novelty-size 43666}
INFO fluree.db.peer.http-api - Request exception: {:status 400, :error :db/unavailable, :message Database testresults/there-can-be-only-one is not currently available. Status is: :initialize.}
INFO f.db.ledger.indexing - Index Update end at: 2021-11-05T15:33:05.486343Z {:network testresults, :dbid there-can-be-only-one, :block 1, :t -2, :idx-duration 94}
INFO fluree.db.util.log - "Transactor session closing for db: testresults/$there-can-be-only-one[there-can-be-only-one]"
INFO fluree.db.util.log - "Channel closed for session updates for: testresults/there-can-be-only-one."
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 30, :time 13.25ms, :block 1}
- processing schema transaction sub-set
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:command [200] 127.0.0.1 {:status 200, :fuel 0}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 1.18ms, :block 1}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 1.06ms, :block 1}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 1.19ms, :block 1}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 0.95ms, :block 1}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 0.83ms, :block 1}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 1.74ms, :block 1}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 1.17ms, :block 1}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 1.29ms, :block 1}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 0.78ms, :block 1}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 1.58ms, :block 1}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 0.85ms, :block 1}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 0.66ms, :block 1}
INFO f.d.l.consensus.raft - testresults/there-can-be-only-one new-block {:block 2, :txns ("d914c8ce4db37e6ec14837561f9cb5c0ac3aec56357008375d1b3d01d82df9ec"), :server "myserver", :network-queue 0}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 2.78ms, :block 2}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:command [200] 127.0.0.1 {:status 200, :fuel 0}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 0.66ms, :block 2}
INFO f.d.l.consensus.raft - testresults/there-can-be-only-one new-block {:block 3, :txns ("e998b0b183d8441b662a37a43f5b150288cc5075a1d5468beb1f26cb22503533"), :server "myserver", :network-queue 0}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 2.21ms, :block 3}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:command [200] 127.0.0.1 {:status 200, :fuel 0}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 0.87ms, :block 3}
INFO f.d.l.consensus.raft - testresults/there-can-be-only-one new-block {:block 4, :txns ("4bcc062f5fe324904f21f43ff265a2469ac491ebbc74da5045d94b40270f6cf0"), :server "myserver", :network-queue 0}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 1.95ms, :block 4}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:command [200] 127.0.0.1 {:status 200, :fuel 0}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 0.70ms, :block 4}
INFO f.d.l.consensus.raft - testresults/there-can-be-only-one new-block {:block 5, :txns ("352191acd35b174fb3882e4b24047513b4069af3fd5d22da2b955a6bd635d44f"), :server "myserver", :network-queue 0}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 2.09ms, :block 5}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:command [200] 127.0.0.1 {:status 200, :fuel 0}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 1.33ms, :block 5}
INFO f.d.l.consensus.raft - testresults/there-can-be-only-one new-block {:block 6, :txns ("1cf11e67bf4951661bea9a1eab551a2ea7475b64e2731ac35ab28a9c30d8204f"), :server "myserver", :network-queue 0}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 2.30ms, :block 6}
- ok, completed 5 transactions on testresults/there-can-be-only-one
- running test scenarios
- SCENARIO: test1
- prepare.json
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:command [200] 127.0.0.1 {:status 200, :fuel 0}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 0.85ms, :block 6}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 0.79ms, :block 6}
INFO f.d.l.consensus.raft - testresults/there-can-be-only-one new-block {:block 7, :txns ("f99aedbe7a27bc0e8a317bbfaf453e4d166e479b6dd455b431d840fa0f8c9129"), :server "myserver", :network-queue 0}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 2.07ms, :block 7}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 1.88ms, :block 7}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:command [200] 127.0.0.1 {:status 200, :fuel 0}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 0.75ms, :block 7}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 0.91ms, :block 7}
INFO f.d.l.consensus.raft - testresults/there-can-be-only-one new-block {:block 8, :txns ("a1975ca86a07e9d354edb07edf516722ff6c403bad4d46815b6d1d51064d5cae"), :server "myserver", :network-queue 0}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 1.84ms, :block 8}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 2.06ms, :block 8}
- Ran 2 database transactions
- yes.json
- Ran 0 database queries
- no.json
- Ran 0 database queries
- tyes.json
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:command [200] 127.0.0.1 {:status 200, :fuel 0}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 0.60ms, :block 8}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 0.74ms, :block 8}
INFO f.d.l.consensus.raft - testresults/there-can-be-only-one new-block {:block 9, :txns ("78d010c7c63869afb9be107b36633ad5dae5d5b9bdaa2b83a4a73e93334da1e5"), :server "myserver", :network-queue 0}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 1.87ms, :block 9}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 2.14ms, :block 9}
- Ran 1 database transactions
- tno.json
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:command [200] 127.0.0.1 {:status 200, :fuel 0}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 1.01ms, :block 9}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 0.82ms, :block 9}
INFO f.d.l.consensus.raft - testresults/there-can-be-only-one new-block {:block 10, :txns ("d02969ccf8d38c760dad45f7f1a57d4c03cf9c7548085263fb035d711341513e"), :server "myserver", :network-queue 0}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 2.11ms, :block 10}
INFO fluree.db.peer.http-api - :testresults/there-can-be-only-one:query [200] 127.0.0.1 {:status 200, :fuel 1, :time 2.69ms, :block 10}
- NOTICE: Expected error in NO transaction
: 400 db/invalid-tx HeadOfState is already assigned to an Auth Value: 123145302311912
- Ran 1 database transactions
- clean
- file not found, skipping: clean
- Ran 0 database transactions
- 1 tests completed
upgrade or not?
This last bit wasn't very usefull. It could have been usefull if there were errors we didn't quite understand. So far we used the docker image with the stable version of FlureeDB. Now imagine we have a complete schema with smart functions and tests (with solid test coverage) and on FlureeDB 0.16.1 everything runs just dandy and all tests succeed. But version 1.0 of fluree is just around the corner, and we would really like to know if, once it is released, we could upgrade our production systems to FlureeDB 1.0. We might want to take a peek at the latest beta and see if our tests still succeed. Doing so is trivially easy. We simple invoke the dockertest command with the appropriate tag for the latest beta, and Bob's your uncle.
./fsst dockertest --dir demo-schema-parts --target default --tag v0.2.7-fluree-v1.0.0-beta16
The result:
COMMAND: ./fsst guesttest --target default --network test12727
IMAGE: pibara/fsst:v0.2.7-fluree-v1.0.0-beta16
WARNING: - docker python lib not found.
- docker related commands disabled.
# waiting for default-private-key.txt to appear
# waiting for default-private-key.txt to appear
Started FlureeDB and got createkey from newly created keyfile
- Database: test12727/there-can-be-only-one
- collecting transactions from build subdirs
Going through first 1 stages
- roles
- there_can_be_only_one
- processing schema transaction sub-set
- ok, completed 5 transactions on test12727/there-can-be-only-one
- running test scenarios
- SCENARIO: test1
- prepare.json
- Ran 2 database transactions
- yes.json
- Ran 0 database queries
- no.json
- Ran 0 database queries
- tyes.json
- Ran 1 database transactions
- tno.json
- NOTICE: Expected error in NO transaction
: 400 db/predicate-spec Predicate spec failed for predicate: _auth/roles. HeadOfState is already assigned to an Auth
- Ran 1 database transactions
- clean
- file not found, skipping: clean
- Ran 0 database transactions
- 1 tests completed
As you can see, for our little demo schema everything is still fine and everything looks mostly the same except for the much better error message that the beta gives us.
Closing
This was our first blog post on our FlureeDB related Python projects. We hope to continue to contribute to the FlureeDB community by improving on the aioflureedb python library, the fsst tool, and ofcause by creating these (hopefully) helpfull blog posts. Remember, these projects are projects we started out of necesity for our own usage of FlureeDB, and outside of that they are not usually our main priority. As such, merge requests on either project are highly welcomed. As are feedback and bug reports.