[CSL-124] Optimise serialisation

[jagajaga]

@neongreen

Bitcoin tries to be smart about serialising its data (blocks, transactions, even integers). Should we do the same? This is a catch-all issue for listing and discussing optimisations.

I'll list the optimisations I can think of:

1. The transactions in the block are stored as a tree. Instead they could be stored as a list, with the tree being reconstructed during deserialisation. This would save us 2×tx_count bytes per block.

2. The numbers could be optimised better (Bitcoin uses a special “compact int” encoding for that purpose). Our ChainDifficulty is a 8-byte int; our SlotId is 8+2=10 bytes; our transaction inputs contain a 4-byte int for index, when in most cases it would fit into one byte; our Coin is 8 bytes but I'm not sure whether anything can be done here (other than assuming that multiples of 10^8 or 10^7 satoshi would be used more often than fine-grained amounts and optimising for them).

UPDATE: I have implemented varint and it's used for most things now, so #2 is obsolete. It might still be useful to use some kind of floating-point encoding for Coin, though (like Bitcoin does).

[jagajaga]

@Gromak

Should we do the same?

Yes, of course. This issue is similar to CSL-114.

[jagajaga]

@volhovm

TLDR: https://github.com/bitcoin/bitcoin/blob/master/src/serialize.h

• Variable-length number serialization (used everywhere -- in set/map/vector length as well).
• Map and Set format is just sequential elements.

I think for now it's enough to abstract binary format to a separate module. Problems with libraries who require Serializable may appear if we declare our own typeclass for that. TBD

[jagajaga]

@georgeee

• commit 969e35c