[bug]: unable to forward, wrong "insufficient bandwidth"

[C-Otto]

Background

I have more than enough liquidity in one of my channels and should be able to serve outgoing forward requests. However, this doesn't work. Instead, lnd reports "insufficient bandwidth". Details below.

Your environment

• lnd v0.15.4-beta
• Linux server 5.10.0-13-amd64 #1 SMP Debian 5.10.106-1 (2022-03-17) x86_64 GNU/Linux
• bitcoind v23

Steps to reproduce

Get forwarding request out via channel X.

Expected behaviour

Forwarding request is served, fees are earned, local balance is reduced.

Actual behaviour

"insufficient bandwidth to route htlc: x is larger than y" (canSendHtlc in link.go)

First observed failure:

• Date: 2022-10-25 18:31:17
• Requested amount: 188,242,405 msat
• Available (according to lnd): 3,005,999 msat

Balance at 2022-10-25 18:29:25+02:

• local: 9,897,025 sat
• local reserve: 100,000 sat
• remote: 100,078 sat
• remote reserve: 100,000 sat

Most recent failure:

• Date: 2022-11-03 13:18:31
• Requested amount: 50,024,602 msat
• Available (according to lnd): 2,843,999 msat

Balance at 2022-11-03 13:14:47+01:

• local: 9,896,568 sat
• local reserve: 100,000 sat
• remote: 100,712 sat
• remote reserve: 100,000 sat

lncli listchannels:

{
            "active": true,
            "remote_pubkey": "03627ebe50fc6eb80b0caab0c3714958c701eda735e3c29588e83150d6d4a93976",
            "channel_point": "db8a9a4d483e0fd0c84e5a004d51dd51ecbb113329729c18db6a5112bab61a4f:0",
            "chan_id": "834535922707529728",
            "capacity": "10000000",
            "local_balance": "9891564",
            "remote_balance": "108241",
            "commit_fee": "195",
            "commit_weight": "724",
            "fee_per_kw": "269",
            "unsettled_balance": "0",
            "total_satoshis_sent": "2036708",
            "total_satoshis_received": "11928272",
            "num_updates": "9950",
            "pending_htlcs": [
            ],
            "csv_delay": 144,
            "private": false,
            "initiator": false,
            "chan_status_flags": "ChanStatusDefault",
            "local_chan_reserve_sat": "100000",
            "remote_chan_reserve_sat": "100000",
            "static_remote_key": true,
            "commitment_type": "STATIC_REMOTE_KEY",
            "lifetime": "174093",
            "uptime": "174093",
            "close_address": "",
            "push_amount_sat": "0",
            "thaw_height": 0,
            "local_constraints": {
                "csv_delay": 144,
                "chan_reserve_sat": "100000",
                "dust_limit_sat": "354",
                "max_pending_amt_msat": "18446744073709551615",
                "min_htlc_msat": "0",
                "max_accepted_htlcs": 30
            },
            "remote_constraints": {
                "csv_delay": 1201,
                "chan_reserve_sat": "100000",
                "dust_limit_sat": "546",
                "max_pending_amt_msat": "9900000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 483
            },
            "alias_scids": [
            ],
            "zero_conf": false,
            "zero_conf_confirmed_scid": "0"
        }

lncli getchaninfo:

{
    "channel_id": "834535922707529728",
    "chan_point": "db8a9a4d483e0fd0c84e5a004d51dd51ecbb113329729c18db6a5112bab61a4f:0",
    "last_update": 1667403066,
    "node1_pub": "027ce055380348d7812d2ae7745701c9f93e70c1adeb2657f053f91df4f2843c71",
    "node2_pub": "03627ebe50fc6eb80b0caab0c3714958c701eda735e3c29588e83150d6d4a93976",
    "capacity": "10000000",
    "node1_policy": {
        "time_lock_delta": 99,
        "min_htlc": "1",
        "fee_base_msat": "0",
        "fee_rate_milli_msat": "1",
        "disabled": false,
        "max_htlc_msat": "10000000000",
        "last_update": 1667403066
    },
    "node2_policy": {
        "time_lock_delta": 34,
        "min_htlc": "1",
        "fee_base_msat": "0",
        "fee_rate_milli_msat": "135",
        "disabled": false,
        "max_htlc_msat": "104096134",
        "last_update": 1667143395
    }
}

[C-Otto]

The available local balance (msat), as parsed from lnd's log messages:

 count |  available   
-------+--------------
     1 |    2,603,999
     1 |    2,607,999
     1 |    2,701,999
     1 |    2,702,999
     1 |    2,703,999
     1 |    2,705,999
     4 |    2,706,999
     1 |    2,843,999
     2 |    2,976,999
     1 |    2,980,999
     2 |    2,991,999
     1 |    2,993,999
     2 |    2,994,999
     1 |    2,995,999
    69 |    3,005,999

[C-Otto]

Note that I was able to forward 200,104 sat after the first failure (a few days later, 2022-10-29 10:54:17+02). As far as I can tell, the channel didn't move any sats between the first error and this outgoing forward.

[positiveblue]

@C-Otto I guess not but is there is any htlc in flight blocking the channel liquidity?

[C-Otto]

No, not according to the logs, and not right now (see lncli listchannels output above). Considering that I had more than 50 failures, this also seems very unlikely.

[C-Otto]

Note that all "available local balance" values end with 999msat, so I guess it's this code:
https://github.com/lightningnetwork/lnd/blob/master/lnwallet/channel.go#L7034

[C-Otto]

Note that I'm able to send payments out via the channel, which could be a related bug.

[Roasbeef]

FWIW, that's a static key channel (legacy), so any HTLC needs to be able to pay fees for itself at the given fee rate, this can cause you to be unable to send even though it looks like you can.

Anchor channels allow the second level HTLC to be zero fees, so you don't need to commit as much since you just care about the impact of the output and not also the second level transaction itself.

Not sure how many HTLCs you had at the time (or prior instance), but this also jumps out:

                "max_accepted_htlcs": 30

AFAICT, you're not the initaitor in this channel, and in one of the instance the remote party had a very low balance:

remote: 100,078 sat
remote reserve: 100,000 sat

If they can't pay for the fees of an HTLC added, then we won't allow that to proceed as we can get into a situation where they only have dust or no amount left over in the channel. The reserve also comes into play as well: even if they can pay the fee, we can't let them dip below the reserve.

[C-Otto]

I don't think that max_accepted_htlcs is an issue, as it only limits the number of in-flight HTLCs. I'm pretty sure I didn't have more than one pending (or failed) HTLC in virtually all cases. Correct me if I'm wrong.

Yes, the remote party has a very low balance, especially considering the reserve. However, I don't understand why this stops me from sending funds to their side, i.e. creating a more balanced channel. It feels like the channel drove itself into a dead end?

Assuming that there's a good reason for lnd to reject forwards, why am I able to use the channel for outgoing payments, initiated by me? Was this luck? Did I pick a suitable amount by chance? Is it a bug, i.e. should lnd also not allow these payments?

All in all I'm rather confused and don't understand how the protection idea ("HTLC needs to be able to pay fees for itself") is supposed to work out in this situation if my node doesn't allow the remote balance to increase.

[Roasbeef]

Yes, the remote party has a very low balance, especially considering the reserve. However, I don't understand why this stops me from sending funds to their side, i.e. creating a more balanced channel. It feels like the channel drove itself into a dead end?

So the initiator pays fees for a commitment transaction the entire time. They need to have enough fees to pay for a force close assigned, and also they pay all the fees for a co-op close.

Each time you add an HTLC, they need to have enough funds to pay for that HTLC (it increases the size of the commitment transaction). This is the case even if they aren't the one adding an HTLC. In the case above, the initiator only has 100,078 sats on their side. The reserve itself is just 78 sats below that at 100k. As as a result, the channel can't be utilized unless the fee rate (for the commitment) transaction is reduced to the point that it can support another HTLC, then things can be rebalanced somewhat.

The relationship between the fees paid, HTLC weight and the reserve mostly works on the spec level. However, there're some degenerate cases when a channel isn't able to be used due to the initiator not being able to pay fees. I think what happened is that they tried to send everything out of the channel, realized the reserve was there, then played around to send the most they could out, leaving that amount.

Returning to the line you linked above: https://github.com/lightningnetwork/lnd/blob/master/lnwallet/channel.go#L7030-L7035

Since theirBalance < htlcCommitFee is true, they don't have enough fees to pay for the HTLC. The function only gets down here if you aren't the initiator. In order to ensure we don't route across this (only hard fails are possible in the protocol when a constraint is violated, you can't "undo" the HTLC). Then ourBalance >= nonDustHtlcAmt is also true, but we don't want to allow this to happen (the channel would basically force close), so we say we have one less than the HTLC amount. This then prevents the HTLC from being routed in general.

There is a way to get around this though: send a small amount continually back to yourself as a rebalance. If the amount is below dust, then no HTLC is added, so no extra fees are required.

Prior spec discussion related to this issue can be found here: lightning/bolts#728.

[Roasbeef]

We added that logic to fix issues like this: #3787. In this case, the router would keep trying to send across that link, but it can't support it due to fees. Now we'll just ignore that and try another link. There were also prior cases of things like the reserve being violated, an output taking too many fees and becoming unspendable ("negative" balance).

[U-got-IT]

If I may chime in. I am running the node on the other side and initiated the channel a few weeks ago.
I am running Core Lightning 0.12.1, not LND. When opening the channel I used default values for everything and didn't do anything manually ever since other than automatic fee adjustment depending on imbalance via a plugin. Over time all the liquidity shifted over to C-Otto's side through normal, unattended routing activity. If this can lead to the situation at hand this seems to be a rather severe incompatibility. Please correct me if I'm wrong.

[C-Otto]

Thanks a lot for the explanation. I think the discussion in lightning/bolts#728 is really helpful, and @Roasbeef's comment above puts this into the context of lnd's code. Personally, I think that this issue could be closed (see below), as it's a weird edge case without an obvious solution.

The only two things missing:

• Why can I send out payments myself, if routing is forbidden? I'm using SendToRoute where the first hop (the problematic one to @U-got-IT) is fixed. I just sent out 10,000sat, which worked out just fine. Yesterday it worked with 50,000sat. I'd be happy to raise a follow-up issue.
• I'll add an INFO log message to describe this edge case, so that future me doesn't need to raise yet another issue: lnwallet: add log message for edge case #7115

@U-got-IT please switch to anchor channels, if possible. If not, please campaign for it to be supported.