Overview
Each node in the Lightning network use channel discovery to create and maintain a local view of the network's topology, so that it can discover routes to desired destinations. Source-routing is utilized in order to give nodes full control over the route their payment follows within the network. Routes are selected based on total cumulative fee and other criteria. Route selection algorithms is an area of active research.
Graph databases are optimized to represent a set of nodes connected to each other by relationships, and for queries that transverse the graph, i.e. start from a (set of) specific nodes and search for patterns across the graph, and in particular to find paths between two nodes, i.e. for route calculations. Neo4j is one of the prominent open source Graph database out there. See What is a Graph database? and From Relational to Neo4j for more details.
Here we describe how to export a node's local view of the network topology to a Neo4j database. The Lightning Network Daemon (LND) implementation is used. The branch is currently available here and includes three components:
A new command added to the command line tool lncli to create a snapshot of the channels database (the node's local view of the network's topology)
A new command line tool lnneo4j to import the snapshot to the Neo4j database
A tutorial LN guide that can be loaded on the Neo4j browser to investigate the Lightning Network snapshot.
This is work in progress, and hopefully some of this will be useful enough to be merged to the main LND code. Comments, corrections and contributions are welcomed.
Examples
These examples are taken from the LN guide (item 3 above) for the Neo4j browser.
Graph databases use the Cypher query language (similar to SQL in relational databases). Cypher is a declarative graph query language that allows for expressive and efficient querying and updating of the graph store. The first example uses the following query:
MATCH p = (s:LNNode)-[*1..3]-(n:LNNode) WHERE s.alias = "FederalReserve" AND ALL (c IN relationships(p) WHERE c.capacity > 10000000) RETURN DISTINCT n, s LIMIT 20
The query looks for all nodes connected to the 'FederalReserve' node with 'fat' channels with capacity larger than 0.1BTC up to 3 hops away. It selects all paths up to 3 hops from the node, and filters those paths without enough capacity. The Neo4j browser displays the results graphically below.
The browser allows you to modify the query and resubmit it, which is ideal for experimentation. Neo4j Cypher Refcard may be useful for that.
The second example uses the build-in shortest path query to calculate all the shortest paths between the the node LN.BLUETEGU.CC and the Blockstream Store node. The query used is:
MATCH (s:LNNode { alias: 'SLEEPYARK-6-11-21-2612-gf083a699' }),(t:LNNode { alias: 'LN.BLUETEGU.CC' }), p = allShortestPaths((s)-[:CHANNEL*]-(t)) WHERE ALL (c IN relationships(p) WHERE c.disabled = false) RETURN p
And the resulting graph (after some manual stretching) looks as follows:
Note that the browser has limitations; it is slow to present graphs with many nodes and relationships, and hence its always advised to add a LIMIT clause. It has a built in limit on the number of nodes it retrieves, and in order to present graphs, it 'complements' queries to get information about relationships between nodes, even if you don't include those in the RETURN clause.
Try it yourself
Installing Neo4j is straight forward, so it shouldn't be too hard to span an instance, take a snapshot of the channels database and import it to Neo4j, and run the guide through the browser. But at least for a limited period, you can try out the browser at my server at:
http://ln.bluetegu.cc:7474/browser/
And use user 'guest' and password 'guest'. Unfortunately the open source edition of Neo4j doesn't allow controlling access per user so the guest user can also modify or delete data, so please play nice, and understand that the server may be down or slow. Below is the login screen.
Once in, you have several guides you can run. If you press 'Jump into Code', you'll see two guides. You can run the 'Movie Graph' guide that walk you through Cypher basics. No need to do the 'Create' commands, as the database will probably be already populated with the movie graph nodes. Similarly, no need to do the 'Drop' command at end. Below is a screen capture of the available guides.
You can run the Lightning Network guide, by entering the following play command and pressing run (like any other command on the browser):
:play http://ln.bluetegu.cc:80/html/ln.html
To start the Lightning Networks guide. The first page of the guide looks as follows:
Technical Details
Creating snapshots
The command to create a snapshot of the channels database is:
$ lncli snapshotchannels
It creates, or overwrites the previous snapshot of the channels database at [lnddir]/snapshot/graph/[network]/channel.db. For example, ~/.lnd/snapshot/graph/mainnet/channel.db in Linux when the Bitcoin mainnet network is used.
Importing to Neo4j
The Neo4j command line tool first empties the Neo4j database from previous snapshot, and then imports the nodes and channels to the Neo4j database. The tool has the following configuration options:
ubuntu@ln:~$ lnneo4j --help
NAME:
lnneo4j - import a snapshot of your channel databse to neo4js
USAGE:
lnneo4j [global options] command [command options] [arguments...]
VERSION:
0.1.1 commit=
COMMANDS:
help, h Shows a list of commands or help for one command
GLOBAL OPTIONS:
--lnddir value path to lnd's base directory (default: "/home/ubuntu/.lnd")
--network value either mainnet, testnet, simnet, regtest, etc. (default: "mainnet")
--neouser value neo4j user for authentication (default: "neo4j")
--neopass value neo4j password for authentication. Required unless neo4j authentication is disabled, e.g. dbms.security.auth_enabled=false is set in neo4j.conf
--neohost value neo4j hostname (default: "localhost")
--neoport value neo4j port number (default: "7687")
--help, -h show help
--version, -v print the version
When run, it provides the following output. Note that it takes a while for the import process to complete:
ubuntu@ln:~$ lnneo4j -neopass <password>
[lnneo4j] 15615 nodes and channels deleted.
[lnneo4j] Added 2313 detailed nodes and 6 nodes known only by public key.
[lnneo4j] Added 6541 bi-directional channels, 314 uni-directional channels and 0 channels without edges dropped.
ubuntu@ln:~$
Nodes are assigned the LNNode label. A node for which an announcement was received is also labeled with Detailed label. The following properties are modelled per detailed node:
- pubKey
- alias
- addresses
- color
- lastUpdate
Only open channels are included in the database (not pending or waiting-to-close). Channels are modelled as CHANNEL relationships with the following properties:
- channelID
- capacity
- lastUpdate
- disabled
- minHTLC
- feeBaseMsat
- timeLockDelta
- feeRateMilliMsat
Neo4j Guide
The guide is written according to Create a Custom Neo4j Browser Guide. In particular, the guide is written in adoc format, and is compiled to html form. The minimal set of files required for the compilation of the guide from the neo4j-guides repository are copied to the project. The readme.doc at /lnd/cmd/lnneo4j/guide/readme.adoc includes all details on how to edit, compile and host the guide.
Installing
To install this version follow the instructions of the LND project, but use the neo4j branch from my repository. My detailed installation instructions here may be helpful.
ubuntu@x:~$ git clone https://github.com/bluetegu/lnd $GOPATH/src/github.com/lightningnetwork/lnd
ubuntu@x:~$ cd $GOPATH/src/github.com/lightningnetwork/lnd
ubuntu@x:~/go/src/github.com/lightningnetwork/lnd$ git checkout neo4j
Branch neo4j set up to track remote branch neo4j from origin.
Switched to a new branch 'neo4j'
ubuntu@x:~/go/src/github.com/lightningnetwork/lnd$
ubuntu@x:~/go/src/github.com/lightningnetwork/lnd$ dep ensure
ubuntu@x:~/go/src/github.com/lightningnetwork/lnd$ go install . ./cmd/...
ubuntu@x:~/go/src/github.com/lightningnetwork/lnd$