Block locator feature (transaction list -x, deck init -s, deck cache, address balance -i)

[d5000]

The block locator feature is finally ready for testing.

Basically, this allows to scan the blockchain once for relevant transactions for an address or deck, stores the block heights of these transactions in a file called blocklocator.json, and then retrieve the values fastly (like a "caching" of the transactions).

In the case of addresses, it stores all blockheights where a transaction was recorded where this address was used to send or receive coins or tokens.

In the case of decks, it stores all block heights where the following transactions occur:

• all transactions where tokens were transfered,
• all burn transactions in the case of PoB tokens,
• all gateway transactions in the case of AT tokens,
• all signalling/locking/donation/voting/proposal transactions in the case of dPoD tokens.

This allows two things:

• retrieve these transactions with pacli transaction list -x much faster (almost like using a database), and potentially with other commands, without having to store the P2TH addresses in the wallet, having thus a direct access to the blockchain, without the potentially buggy slimcoind RPC commands or corrupted wallet files,
• realize integrity tests. This is the main reason why I implemented this, because some of the RPC commands are unfortunately not completely reliable.

The commands implemented for this are at this moment:

deck init -n -s BLOCKS. This scans BLOCKS blocks (default: 50000) for transactions relevant to a deck DECK or for the standard PoB and dPoD decks and stores the blockheights of the transactions and all p2th addresses in blocklocator.json. The last block which was checked is stored, so everytime this command is called it will continue from where it ended the last time.

The idea is that the user could call this once he has installed the software, let it run e.g. overnight and then being able to test te integrity or double check if there's something strange.

deck cache [DECKID] -b BLOCKS does the same than deck init -n -s BLOCKS, but without initializing the deck(s).

(Edit: this command was originally called "storeblocks").

transaction list ADDRESS -x [-e ENDBLOCK/END_DATE]. If the command now finds the address in blocklocator.json, then it uses the data instead of scanning again. It will then scan the blocks which still were not scanned before (and meanwhile store the locators), so it's still recommended to use it with -e for an end block or date.

There will be another command which will realize an integrity test which is still not ready. It will be called per address or for all P2TH addresses of a deck. The idea is: if you find an address where the balance shown with address balance or address list seems incorrect, you can perform this command to get a "second opinion" about what really happened at blockchain level.

As it would be quite long to scan the 4 million blocks of mainnet, the blocklocator.json file could be distributed like a blockchain snapshot once the token is launched.

---

This will be the last major feature I'm implementing. From now on, I'll concentrate on bugfixes and unittests, and small usability improvements for good reasons. The only exceptions may be the DEX and, if existing, features which really fix a big usability problem relevant for a lot of average users.

[buhtignew]

Great!
I'll begin testing this as well as soon as I can.

[buhtignew]

This is something very new I think I'll need some time to understand how it work and what tests can be performed on this new implementation.
In the mean time I have some question about this issue text:

Basically, this allows to scan the blockchain once for relevant transactions for an address or deck, stores the block heights of these transactions in a file called blocklocator.json, and then retrieve the values fastly...

Does it mean that the blocklocator.json merges the result of different scans both if they are address scans and deck scans and also the scans of different addresses/decks?

This scans BLOCKS blocks (default: 50000)...

What 5000 means? It's a depth or it's the blockheight?

transaction list ADDRESS -x [-e ENDBLOCK/END_DATE]. If the command now finds the address in blocklocator.json, then it uses the data instead of scanning again. It will then scan the blocks which still were not scanned before (and meanwhile store the locators), so it's still recommended to use it with -e for an end block or date.

My expectation here was that it would be better to store as much as possible once the command is being used, since the scanning requires a lot of time in any case. Do you mean that by limiting the scanning by an END_DATE the process would be quicker?

[d5000]

Does it mean that the blocklocator.json merges the result of different scans both if they are address scans and deck scans and also the scans of different addresses/decks?

Correct. blocklocator.json contains a dictionary with addresses and block heights. In the case a deck is scanned, all P2TH addresses of the deck are added to this list.

BLOCKS is an absolute number of blocks. The end block height (when the scan stops) is calculated in the following way: It starts with either:

• block 0 (in a scan of a normal address which never has been scanned),
• the block the deck was spawned (in the case of deck scans, which never have been scanned)
• the last known block which was stored in blocklocator.json.

To this number BLOCKS blocks are added. For example, if you don't specify blocks and scan an address which never has been scanned, it scans from the genesis block to block 50000. If it's a deck launched at block 100000, it will scan until 150000, and so on.

I guessed that an absolute block height would be too complicated in this case, as the user then would have to be aware of the block height. So he can select the number of blocks and then scan, for example, some hours per day.

Do you mean that by limiting the scanning by an END_DATE the process would be quicker?

This recommendation is mainly to avoid the command being running for days on mainnet when trying to scan millions of blocks at once. In most cases it will store the block heights even if it's stopped with a KeyboardInterrupt (e.g. CTRL-C), but in the case the computer freezes (or the command is killed due to memory usage, for example), it won't store the data and thus the whole scan was in vain.

It would be perhaps a good idea to add a guide to the manual how to setup a cronjob with the storeblocks command, so it scans every day or week either 50000 blocks or the missing blocks until the tip of the blockchain.

In general the best strategy is to scan all decks you will be using at once, this will take as much time as scanning a single deck, and only occupy a low amount of space (the whole idea of the feature was to make it as lightweight as possible, this is for example why it stores block heights and not transactions or txids).

[buhtignew]

In general the best strategy is to scan all decks you will be using at once

To do so one should run deck storeblocks -b BLOCKS, i.e. without [DECKID], right?

[d5000]

To do so one should run deck storeblocks -b BLOCKS, i.e. without [DECKID], right?

No, this will store only the block heights for the two main decks (standard PoB + standard dPoD). You have to add the -a flag. I'm unsure if I should change this, making the -a option the main option. See the help at the docstring.

[d5000]

I have implemented a rudimentary integrity test in commit 9323b14, with the command address balance -i.

It is however still not properly tested from my side, and still misses some functions, although it seems to work so far. I uploaded the commit mainly because it fixes also other issues, for example the ex-storeblocks command (for now called deck cache) was not getting the start block properly, and the forgotten print statements in transaction list.

[buhtignew]

I've just tested address balance -i and got the following output:

Getting RPC txes ...
Reading locator file ...
File does not exist.
Last blockheight checked 0
Reading locator file ...
File does not exist.
Addresses ['mx5MdsenFDZufFuwT9ND7BKQBzS4Hy9YUP'] were not checked. Storing locator data now.
Processing block: 0
TX ce86aa96a71e5c74ea535ed5f23d5b1b6ca279ad16cac3cb95e123d80027f014 Error: Transaction does not exist or is corrupted.
Reading locator file ...
File does not exist.
Storing new locator block heights.
New Locator dict: {'mx5MdsenFDZufFuwT9ND7BKQBzS4Hy9YUP': {'heights': [], 'lastblock': '00000d7e8a80fec4057cb6d560822705596040bf41f0ebb2465dcdf46e4c517e'}}
UTXO check not possible, as balances changed after the given blockheight 0.
Address: mx5MdsenFDZufFuwT9ND7BKQBzS4Hy9YUP
RPC balances: {'balance': Decimal('0'), 'observed': False}
Blockchain balances: {'balance': Decimal('0'), 'observed': False}
PASSED (partly)

My address balance mx5MdsenFDZufFuwT9ND7BKQBzS4Hy9YUP output is : {'balance': 33707.108257}

[d5000]

I should have written more explicitly that this is not ready for testing. I'll later announce when it's ready.

Anyway, that's not a bug.
The integrity test is always done for a balance at a specific block height - the last block stored in the blocklocator.json or a block you give behind -i. And if there is nothing stored in blocklocator.json, it looks for block 0. I will add a warning message for this case.

The problem seems to be that in your case these addresses have still not be added to the blocklocator.json. If you want to test this feature I suggest first to do a deck cache (ex deck storeblocks) scanning at least to block 150000. Best would be to do this with deck cache -a so you have all your decks stored. Once this is done, the output will be more meaningful.

The error message in the output after TX ce86aa96a71e5c74ea535ed5f23d5b1b6ca279ad16cac3cb95e123d80027f014 appears because SLM doesn't interpret the transaction in the genesis block. This is something I have already noticed before and it seems nothing can be done about that.

[buhtignew]

I should have written more explicitly that this is not ready for testing. I'll later announce when it's ready.

I have seen that but since it was working fine for you I've tested it as well just out of curiosity.

[d5000]

I added a new command address cache. This does the same thing than deck cache (storing block locators without showing transactions) but only for a single address. Commit: 1426a6b

This command can already be tested, I have pre-tested it.

I have also worked on the integrity test (address balance -i) but it reports lots of inconsistencies to me (differences between the values of the different balances), so I want to test it a bit further until it should be tested publicly to be sure there are no bugs hidden deeply in the new code.