Improve uptime

[frol]

During the weekend we had two instances of Indexer for Explorer getting stuck (unable to save some specific transactions to the database) for hours. It is impossible to prevent all the bugs, so in this post, I am going to lay out the plan of what we can do better to reduce the downtime in a face of unknown unknowns.

• Improve alerting:

  • catch individual nodes "downtime" (there are scenarios when the indexing process cannot progress)
  • catch indexing process lagging behind the network (compare the latest block from RPC with the latest block from Indexer for Explorer database, and allow 30 blocks lag)
    • feat: API indexer-status endpoint near-explorer#732
    • integrate it into alerting system

• Define the incidents communication channels

  • initiate a single public feed of incident communication (Zulip subtopic?)
  • craft a list of channels (Telegram Dev group, Discord support channels) where we should post the link to the incident communication channel

• Enable flexible and fast restarts of indexer from any block:

  • fix --allow-missing-relations-in-first-blocks (currently, account_changes contraints cast this feature to a useless one) - it is unfixable, so we implemented non-strict-mode: refactor: Strict/Non-strict mode improvements #164
  • make sure we can quickly restart indexer-for-explorer even on testnet (where the genesis file is huge), maybe at a cost of adding --UNSAFE-fast-start flag which would disable genesis validation in nearcore somehow (I believe that is the main contributor to the start time)
  • consider implementing various helper commands to store a single block or transaction, skip certain transactions, etc

• Write a handbook on:

  • how Indexer for Explorer works on the high-level (TL;DR: we embed nearcore node inside through Indexer Framework and handle the stream of blocks by storing blocks, chunks, ... [list all and provide references to the source code] into PostgreSQL in an idempotent way with foreign key relations to ensure consistency ...)
  • how to troubleshoot/fix Indexer for Explorer if the maintainers are sleeping

• Improve performance of catching up the network (currently a single indexer node is only capable of catching up at a speed of 2 blocks per second and seems to be blocked on a single thread capacity somewhere since it uses only a single core out of 8 available on the server we currently run) -- we postpone this until we decide on a general problem of where to store the growing amount of indexed data

[bowenwang1996]

Thanks for taking the lead on this @frol. What doesn't destroy us will make us stronger 💪

Improve performance of catching up the network

Does this specifically refer to the indexer part? Block sync is known to be slow, so maybe it is an inherent issue with nearcore.

[frol]

Does this specifically refer to the indexer part? Block sync is known to be slow, so maybe it is an inherent issue with nearcore.

Yes, it is indexing performance specific, not syncing. Even when the node is completely synced, we observe that we cannot store more than 2 blocks per second unless those are already stored in the DB.

[bowenwang1996]

I see. I am curious to know why that is the case :)

[frol]

Most probably the issue is the latency, we should come up with a strategy on how to parallelize INSERTs even further. We insert data into 15 tables that have some foreign key relations, so you cannot insert a receipt before you insert a transaction. This creates certain constraints in the parallelization during a single block processing and also across blocks.

[maxzaver]

initiate a single public feed of incident communication (Zulip subtopic?)

How about we combine https://status.nearprotocol.com/ with https://twitter.com/NEARChainStatus ?

[bowenwang1996]

How about we combine https://status.nearprotocol.com/ with https://twitter.com/NEARChainStatus ?

How should we combine them exactly?