Tool for generating proofs for delayed validator exits in the Lido ecosystem. This bot monitors validator exit requests and generates cryptographic proofs when validators fail to exit by their required deadline, enabling penalty enforcement on node operators.

The Late Proof Verifier Bot is a critical component of Lido's validator exit delay monitoring system. It:

• Monitors beacon chain for validator exit requests
• Detects delayed exits when validators haven't exited by their deadline
• Generates proofs using Merkle tree cryptography for delayed validator exits
• Submits verification to the ValidatorExitDelayVerifier contract
• Enables penalty enforcement on node operators whose validators are delayed

The bot runs as a daemon that continuously processes beacon chain roots:

• Node.js 16+
• Access to Ethereum execution layer RPC
• Access to Ethereum consensus layer API
• Private key for transaction signing (if not in dry-run mode)

1. Clone and install dependencies

   git clone https://github.com/lidofinance/late-prover-bot.git
   cd late-prover-bot
   yarn install

2. Generate contract types

   yarn run typechain

3. Build the project

   yarn build

1. Create environment file

   cp .env.example .env

2. Fill in the required variables (see Environment Variables section below)

# Start daemon with monitoring stack
docker-compose up -d daemon prometheus

# View logs
docker-compose logs -f daemon

# Access metrics
curl http://localhost:8081/metrics

# Development mode
NODE_OPTIONS=--max-old-space-size=8192 yarn run start:dev

# Production mode
yarn run start:prod

Before running the bot, you can estimate current gas costs on mainnet using the standalone gas estimation script:

# Basic usage (uses default public RPC)
yarn estimate-gas

# With your own RPC
EL_RPC_URLS=https://eth-mainnet.g.alchemy.com/v2/YOUR-KEY yarn estimate-gas

# With custom configuration
TX_GAS_FEE_HISTORY_DAYS=3 TX_GAS_FEE_HISTORY_PERCENTILE=75 yarn estimate-gas

What it does:

• ✅ Fetches current mainnet base fee
• ✅ Analyzes historical gas data (1 day by default)
• ✅ Calculates recommended gas fee using percentile
• ✅ Estimates costs for small/typical/large batches
• ✅ Shows if bot would send transactions now
• ✅ Displays costs in Gwei, ETH, and USD

Example output:

Current base fee: 25.3 Gwei
Recommended (50th percentile): 28.5 Gwei
Status: ✅ ACCEPTABLE

Typical Batch (~2.2M gas):
  Gas Cost: 55,660,000 Gwei
  ETH Cost: 0.055660 ETH
  USD Cost: $138.35

✅ DECISION: Bot would send transactions now.

See scripts/README.md for full documentation.

curl http://localhost:8081/health

The bot uses dynamic gas estimation with hard limits:

1. Estimates gas required for each transaction using eth_estimateGas
2. Adds 20% buffer to account for gas variation
3. Enforces hard limit: TX_GAS_LIMIT is a hard upper bound
4. Rejects transaction if estimated gas (with buffer) exceeds TX_GAS_LIMIT

Example behavior:

• Configured: TX_GAS_LIMIT=2000000
• Estimated: 1,500,000 gas
• Used: 1,800,000 gas (1.5M × 1.2 buffer) ✅
• If estimated: 2,500,000 gas (with buffer: 3,000,000)
  • ❌ Transaction rejected: Exceeds hard limit of 2,000,000
  • Error message shows required minimum: TX_GAS_LIMIT=3000000

If you encounter UNPREDICTABLE_GAS_LIMIT errors:

1. Check logs: Look for the estimated gas amount before the error
2. Increase gas limit: Set TX_GAS_LIMIT to estimated × 1.5 for safety
3. Check contract state: Ensure your validators and exit requests are valid
4. Network issues: Verify your RPC endpoints are stable and responsive

If you see "Transaction rejected: Estimated gas exceeds hard limit":

1. Check the error message: It will show the required gas limit

   Estimated gas (with buffer): 2,500,000
   Configured hard limit (TX_GAS_LIMIT): 2,000,000
   Required: TX_GAS_LIMIT must be at least 2,500,000
   

2. Increase TX_GAS_LIMIT: Set it to at least the required amount

   TX_GAS_LIMIT=2500000

3. Or reduce batch size: Process fewer validators per transaction

   VALIDATOR_BATCH_SIZE=25  # Reduce from default 50

If you see "intrinsic gas too low: gas X, minimum needed Y":

1. Set higher gas limit: Use the "minimum needed" value + 20% buffer

   # If error shows "minimum needed 1588836"
   TX_GAS_LIMIT=1906600  # 1588836 * 1.2

2. Check batch size: Larger batches need more gas - consider reducing VALIDATOR_BATCH_SIZE

• 1-10 validators: TX_GAS_LIMIT=1500000 (with safety margin)
• 11-25 validators: TX_GAS_LIMIT=2000000 (with safety margin)
• 26-50 validators: TX_GAS_LIMIT=3000000 (with safety margin)
• 51+ validators: TX_GAS_LIMIT=4000000+ or reduce batch size

Note: TX_GAS_LIMIT is a hard upper limit. Transactions will be rejected if estimated gas (with 20% buffer) exceeds this value.

If you encounter "oversized data" or "transaction size limit exceeded" errors:

1. Reduce batch size: Set VALIDATOR_BATCH_SIZE=25 (or lower)
2. Set size limit: Adjust MAX_TRANSACTION_SIZE_BYTES=50000 for stricter limits
3. Monitor logs: Check how many validators are being processed per batch
4. RPC provider limits: Some providers have stricter size limits (Alchemy: 128KB, Infura: varies)

Example configuration for large validator sets:

VALIDATOR_BATCH_SIZE=20
MAX_TRANSACTION_SIZE_BYTES=80000
TX_GAS_LIMIT=2500000

The application automatically splits large validator groups into smaller batches to prevent oversized transactions and provides detailed batch processing logs.

The bot exposes comprehensive metrics at /metrics endpoint:

# View all metrics
curl http://localhost:8081/metrics

# View custom metrics
curl http://localhost:8081/metrics | grep late_prover_bot

• Proof Generation: Duration and success rates
• Validator Processing: Processed, skipped, and eligible validators
• Contract Interactions: Call duration and verification counts
• Block Processing: Range processing and batch operations
• Memory Usage: Heap usage and RSS memory
• Daemon Operations: Cycle duration and sleep tracking
• Error Tracking: Various error counters

The project includes a complete monitoring setup:

# Start with Prometheus
docker-compose up -d daemon prometheus

# Access Prometheus UI
open http://localhost:9090

# Uncomment Grafana in docker-compose.yml for dashboards
# open http://localhost:8082 (admin/MYPASSWORT)

# Build for production
yarn build

# Build and watch for changes
yarn run start:dev

# Unit tests
yarn test

# E2E daemon tests  
yarn run test-daemon

# Test coverage
yarn run test:cov

# Check code style
yarn run lint

# Fix code style issues
yarn run lint:fix

# Format code
yarn run format

• DaemonService: Main orchestrator running the processing loop
• RootsProcessor: Processes beacon chain roots and block ranges
• RootsProvider: Provides next roots to process with crash recovery
• ProverService: Generates Merkle proofs for delayed validator exits
• Contract Services: Interact with Lido contracts (ValidatorExitDelayVerifier, StakingRouter, etc.)
• Consensus/Execution Providers: Interface with beacon chain and execution layer

1. Root Discovery: RootsProvider determines next beacon chain root to process
2. Block Range Processing: RootsProcessor handles execution layer blocks
3. Exit Request Detection: ProverService discovers exit requests from events
4. Validator Analysis: Groups validators by deadline and checks exit status
5. Proof Generation: Creates Merkle proofs for delayed validators
6. Contract Submission: Submits proofs to ValidatorExitDelayVerifier contract
7. Progress Tracking: Stores last processed root and updates metrics

GPL-3.0

For issues and questions, please open an issue on the GitHub repository.