DSN spec

[dariolina]

Based on the docs below, but needs a deep review and update:

• https://www.notion.so/subspacelabs/DSN-v3-Overview-c50971c4a4f34ca7b0d911a3c202d166
• https://www.notion.so/subspacelabs/DSN-Data-Retrieval-46903999ed65467e9498403a2d210eb5
• https://www.notion.so/subspacelabs/Auto-Drive-Infra-Description-1bc0bae47e9580848b69cedad102b27a
• https://github.com/jfrank-summit/subspace/tree/dsn-specs/specs

High-level docs:

• path of data from sys.remark through archiving to object retrieval

[teor2345]

Some Q&As we'd like to cover:

I had misinterpreted this one, I had assumed that object mappings were emitted on archival

The object mapping is (usually) emitted when the block is archived:
https://github.com/autonomys/subspace/blob/24d036cc6efd3f6372db275f1dfd4fc590ba6849/crates/sc-consensus-subspace/src/archiver.rs#L891

But the piece(s) containing that object, and the segment commitment for that object, are only available when the segment is full.

The previous segment header is only put into a piece when the next segment is full (it is at the start of the first piece of the next segment), but we usually don't have to wait that long.

If you're running an archival node, it will generate the segment header at the end of the last segment. That's the fastest and most reliable way to do it. Other nodes will pick it up via gossip or from the runtime (it is included in the first block after the segment is complete):
https://github.com/autonomys/subspace/blob/24d036cc6efd3f6372db275f1dfd4fc590ba6849/crates/sc-consensus-subspace/src/archiver.rs#L867

There's also a delay of 100 blocks between block creation and archiving, because they have to be confirmed first:
https://github.com/autonomys/subspace/blob/24d036cc6efd3f6372db275f1dfd4fc590ba6849/crates/sc-consensus-subspace/src/archiver.rs#L10
https://github.com/autonomys/subspace/blob/24d036cc6efd3f6372db275f1dfd4fc590ba6849/crates/pallet-runtime-configs/src/lib.rs#L72

Avoid running a farmer on the same node, as it can slow down archiving:
https://github.com/autonomys/subspace/blob/24d036cc6efd3f6372db275f1dfd4fc590ba6849/crates/sc-consensus-subspace/src/archiver.rs#L16

Then, the best solution would be to keep track of the currentSegmentIndex and add to the object mapping their segment index, and only propagate from the indexer to the subscribed services (auto-drive) whenever the segment gets archived.

You might also want to add a short delay for piece propagation to farmers and through the network. The segment is created on the node, but doesn't become available to DSN clients (like the gateway) until it is stored in the piece cache on enough farmers (which are the only DSN servers).

This is usually quick, because the node attached to the farmer is doing the archiving. But I'd give farming clusters and the rest of the network 30 seconds to fully propagate and register the new segment. It takes 6 seconds to produce a block containing the segment header, and 6 seconds to propagate it. On a 100 Mbps farming cluster, it could take around 25 seconds for a 256 MB segment to be transferred from node to farmer and written to the piece cache. (Most farmers will be on the same machine or a gigabit network, and much faster than this.)

Is this formula correct for calculating the number of pieces within a segment? And therefore segmentIndex(object) = pieceIndex(object) / 256?

Yes, that's the formula we use from genesis onwards. Note that the division is integer truncating division.

It's simple enough that we don't provide the segment index with the mapping, because it would be redundant.

assuming neither of these params were updated since genesis block

It hasn't been changed, and it can't be without a lot of special-casing in the code, which you'd be able to see in the code you got this formula from.

The object mapping is (usually) emitted when the block is archived:

I thought that blocks within the same segment were archived at the same time. But what I understand is that the archiving lifecycle is the following:

Block created -> Block archived (is encoded within a piece + object mapping emissions) -> Segment completes (starts propagation + within a short delay the pieces within that segment are available)

We used to wait for the whole segment to emit mappings. But we changed it and now sometimes they are emitted as soon as that part of the block is archived.

When your node is near the tip, that's usually straight away, but when a block splits over a segment, the mappings are delayed until the next block is archived (which triggers the archiving of the previous partial block).

When your node is not near the tip, blocks are archived in bulk, so mappings might be delayed until the end of the segment (or the end of the block batch).

autonomys/subspace#3108

We don't want to make too many promises here, we might change the implementation to improve performance. So assume mappings can be produced as early as the block import, and as late as the next segment being complete.

[teor2345]

Jeremy and Carlos have also asked for a high-level object data flow:

Object submission:

• system.remark[_with_event] (or history for older mainnet blocks) -> client RPC -> transaction -> node tx pool -> block

Any time from block confirmation to bulk sync on a new node:

Either:
During sync near tip:

• block gossip between nodes, or
  During bulk block sync:
• farmer pieces -> node DSN client -> archiver segment reconstruction -> segment blocks
  Then:
• block(s) -> archiver -> runtime (for object mappings) -> partial (or complete) segment

Any time from block archiving to the completion of the next segment (for blocks split between segments):

• archiver object mappings -> internal node mapping queue -> node RPC subscriptions -> mapping RPC clients (indexer etc)

After the completion of the segment:

• archiver pieces -> internal node piece queue -> node RPC subscriptions -> farmers -> DSN clients (node, gateway, other farmers)

This also happens during bulk block sync. The pieces already exist on the network, but the segments and blocks are still reconstructed, then turned into segments again. This allows nodes to generate all historical mappings, not just the ones near the tip.

After mappings and pieces are available on the network:

• autodrive request -> mapping indexer -> gateway RPC -> gateway DSN client -> piece(s) (from farmers) -> objects -> gateway RPC response -> autodrive response

Note: there is no segment reconstruction on the gateway. To avoid segment reconstruction, we manually parse and discard segment headers, and guess end of segment padding. This is secure and reliable because mappings have the object hash.