[Image 1]
Introduction
Hey it's a me again drifter1!
I'm lucky, I got another 2 weeks to play with! I asked the professor and the new final-final deadline is 16/9. But, I will still try to get a lot of work done this week, or more specifically I already did. :)
During the last days I started implementing the UTXO and its corresponding endpoints. After a lot of thought, I chose to store the UTXOs (unspent transaction outputs) for each address individually. Thus, a new folder will basically be created in the client's directory and inside of it multiple JSON files, which correspond to each address. The UTXO will consist of an array of UTXO outputs, which basically store the essential information needed for retrieving the transaction output using block endpoints. Endpoints for retrieving the UTXO of an address, the outputs only or even a specific output, as well as creating a utxo, and adding/removing an utxo output, have also been defined. For now, only the UTXO endpoints check for the validity of the output, by trying to retrieve the corresponding transaction output from the block specified in the UTXO output, in order to verify that it truly exists, before adding the output to the UTXO.
I did some more coding, but the rest is still in a draft implementation stage, and so I will commit the changes in a later point. The changes are basically about validating if the inputs referenced in the transactions that are posted or to-be-included in a block are defined as UTXO outputs. But, double-spending is still a problem as each transaction is checked one-by-one, and so a temporary UTXO or so, will be needed. Anyway, there is still some work to be done there, and I'm thinking of merging these "smaller" changes with removing the transactions that have been confirmed from the unconfirmed transaction structure (transactions.json).
So, without further ado, let's get more in-depth!
GitHub Repository
UTXO Structure
So, what exactly is the UTXO? Well, it's just a means of storing the "state" the blockchain is currently in. It helps track the balance that each wallet/address has available, is updated frequently (after each block) and can speed-up validation by quite some margin. Of course, going through the complete blockchain again-and-again each time a transaction input needs to be validated would be way to time-consuming. By storing the most essential information, which is the unspent transaction outputs (UTXOs), transaction posting and block creation time is kept low.
The structure that I chose to go with, tracks the UTXOs per wallet/address, as shown below.
So, inside of the client's "utxo" sub-directory, multiple files are created, which are updated independently.
The nessecary information that needs to be stored for a transaction output is:
- the block the transaction was included in (block_height)
- the index of the transaction within that block (transaction_index)
- the index of the transaction output (output_index)
UTXO Endpoints
A total of six new endpoints have been defined:
- GET /utxo/address/ : retrieve utxo by address
- GET /utxo/address/outputs/ : retrieve utxo outputs by address
- GET /utxo/address/outputs/transaction_hash : retrieve utxo output by address-transaction_hash pair
- POST /utxo/address/ : create new utxo of address
- POST /utxo/address/outputs/ : add utxo output to utxo of address
- DELETE /utxo/address/outputs/ : remove utxo output from utxo of address
I think most of them are self-explanatory!
I would like to note that when creating a new utxo, it's possible to optionally add a complete utxo in JSON format. In this case (as well as when a new utxo output is added) each utxo output is checked for validity, by contacting the block endpoints. More specifically, the corresponding block transaction output is retrieved, and if it exists the output is valid.
Knowing if the output is spent is easy, as the corresponding utxo output will be removed when it is spent at a later point in time, and so it will not be retrieved when the transaction posting or block creation callback tries to access it. Also, don't worry too much about outside interference at this point in time, as the utxo is updated locally for now.
Testing Thread Example Run
I will skip the code this time around, and only talk about the structure changes...
The test thread now first creates a block with a single coinbase transaction (or reward transaction). In JSON format this looks like this:
{
"timestamp": 1630318151,
"height": 0,
"creator": "0x3037f04f76620e094e56de685654d4ff0f0a2af0",
"reward": 2.5,
"fees": 0,
"nonce": "abcdef",
"transactions": [
{
"timestamp": 1630318151,
"inputs": [
{
"transaction_hash": "0000000000000000000000000000000000000000000000000000000000000000",
"output_index": 0,
"output_address": "0x0000000000000000000000000000000000000000",
"output_value": 2.5,
"output_hash": "0000000000000000000000000000000000000000000000000000000000000000",
"signature": "00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
}
],
"outputs": [
{
"index": 0,
"address": "0x3037f04f76620e094e56de685654d4ff0f0a2af0",
"value": 2.5,
"hash": "43df22c0ede8fedb4c8a174a7e167ccfba2dcfcdf465eb3e2e3446b32a3ed2fe"
}
],
"total_input": 2.5,
"total_output": 2.5,
"fee": 0,
"hash": "2b7d0f3d50de8ebe0169c050dd03c749b63eeaf34dbba366324508fbf614a2d9"
}
],
"hash": "3b2d0583018e859bc68fa2678ce1154f6874a7e798c03dae102ef025ab3d8faa",
"prev_hash": "0000000000000000000000000000000000000000000000000000000000000000"
}For the corresponding reward address, this leads to the following UTXO:
{
"utxo_outputs": [
{
"block_height": 0,
"transaction_hash": "2b7d0f3d50de8ebe0169c050dd03c749b63eeaf34dbba366324508fbf614a2d9",
"transaction_index": 0,
"output_index": 0
}
],
"balance": 0,
"last_block_included": 0
}
So, what do we see? Well, because the address is mentioned in the coinbase transaction, a registry in the utxo of that address is created that referenced that specific output. The transaction output is in block 0, the transaction index is 0 and the output index is also 0. The transaction_hash is also included, for the sake of retrieving the utxo output faster later on.
Next up, a new block is formed, which again has a reward transaction referencing the same address. But, now there is also another transaction which accesses the previous output as an input. In JSON format, block 1 looks like this:
{
"timestamp": 1630318166,
"height": 1,
"creator": "0x3037f04f76620e094e56de685654d4ff0f0a2af0",
"reward": 2.5,
"fees": 0.001,
"nonce": "123456",
"transactions": [
{
"timestamp": 1630318166,
"inputs": [
{
"transaction_hash": "0000000000000000000000000000000000000000000000000000000000000000",
"output_index": 0,
"output_address": "0x0000000000000000000000000000000000000000",
"output_value": 2.5,
"output_hash": "0000000000000000000000000000000000000000000000000000000000000000",
"signature": "00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
}
],
"outputs": [
{
"index": 0,
"address": "0x3037f04f76620e094e56de685654d4ff0f0a2af0",
"value": 2.5,
"hash": "43df22c0ede8fedb4c8a174a7e167ccfba2dcfcdf465eb3e2e3446b32a3ed2fe"
}
],
"total_input": 2.5,
"total_output": 2.5,
"fee": 0,
"hash": "390bbefffb8e2d4800e80c0a2f1fddead89852f9961d5a9eeb77d822ff5099d9"
},
{
"timestamp": 1630318166,
"inputs": [
{
"transaction_hash": "2b7d0f3d50de8ebe0169c050dd03c749b63eeaf34dbba366324508fbf614a2d9",
"output_index": 0,
"output_address": "0x3037f04f76620e094e56de685654d4ff0f0a2af0",
"output_value": 2.5,
"output_hash": "43df22c0ede8fedb4c8a174a7e167ccfba2dcfcdf465eb3e2e3446b32a3ed2fe",
"signature": "305e67ad23618b03a63ce35e4df089314aaf28dea8dc962cc6f33a15559409ea2345d01990e4b5c7968bd5fc98e6c9394806ab59e5ff3799a34bac3cc7eefe0e"
}
],
"outputs": [
{
"index": 0,
"address": "0x60a192daca0804e113d6e6d41852c611be5de0bf",
"value": 1.2,
"hash": "0d111fcab3dd5a13945224fa286a8bb9b027f47ffca21991072a8ae3b087b667"
},
{
"index": 1,
"address": "0xe23fc383c7007002ef08be610cef95a86ce44e26",
"value": 0.3,
"hash": "f52a8e1a890b992350089e7081d404855f007eff2054fb8cdbc766267b57df19"
},
{
"index": 2,
"address": "0x3037f04f76620e094e56de685654d4ff0f0a2af0",
"value": 0.999,
"hash": "f9fa3c3566971ef8b4184378f51a5de8d24e8acf16eeac938adbda9c19c7f5d5"
}
],
"total_input": 2.5,
"total_output": 2.499,
"fee": 0.001,
"hash": "b4a51095f52edefb4824fc25624741f76a383f954535583e132dd470746a76b5"
}
],
"hash": "e9df0b4a8df38adad216dd03e3fbd2be9386eb7b3999baca52df3a1ee8104f15",
"prev_hash": "0000000000000000000000000000000000000000000000000000000000000000"
}The UTXO of the address, needs to be updated accordingly! So, what exactly happens this time around?
Well, first of all, a new registry needs to be added, as the reward transaction (block 1, transaction index 0) again mentions the same address. For the second transaction now, the first entry that has been added to the UTXO is now "spent", which means that it will be removed from the structure. This transaction also has three outputs, where one of the outputs again references the same address (let's ignore the other addresses). So, the new UTXO file for the address, now looks like this:
{
"utxo_outputs": [
{
"block_height": 1,
"transaction_hash": "390bbefffb8e2d4800e80c0a2f1fddead89852f9961d5a9eeb77d822ff5099d9",
"transaction_index": 0,
"output_index": 0
},
{
"block_height": 1,
"transaction_hash": "b4a51095f52edefb4824fc25624741f76a383f954535583e132dd470746a76b5",
"transaction_index": 1,
"output_index": 2
}
],
"balance": 0,
"last_block_included": 1
}And that's it! As simple as that, the unspent transaction outputs can be tracked. Using simple requests on the utxo endpoints it will now also be quite easy to check if the outputs referenced as inputs are unspent.
RESOURCES:
References
- https://www.python.org/
- https://flask.palletsprojects.com/en/2.0.x/
- https://docs.python-requests.org/
- https://pypi.org/project/mnemonic/0.20/
- https://pypi.org/project/ecdsa/
- https://docs.microsoft.com/en-us/windows/wsl/install-win10
- https://www.docker.com/
- https://code.visualstudio.com/
- https://insomnia.rest/
Images
The rest is screenshots or made using draw.ioPrevious dev blogs of the series
Final words | Next up
And this is actually it for today's post!
Next up will be checking the validity of the transaction inputs using the UTXO endpoints, as well as some other minor topics...
I will keep you posted on my journey! Expect such articles to come out weekly, or even twice a week, until I'm finished!
Keep on drifting!