Return-Path: Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 2FC46C0032; Tue, 17 Oct 2023 18:48:15 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id ED9ED61400; Tue, 17 Oct 2023 18:48:14 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org ED9ED61400 Authentication-Results: smtp3.osuosl.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=mTbNFFfT X-Virus-Scanned: amavisd-new at osuosl.org X-Spam-Flag: NO X-Spam-Score: -2.098 X-Spam-Level: X-Spam-Status: No, score=-2.098 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wHCTp80FXOxY; Tue, 17 Oct 2023 18:48:12 +0000 (UTC) Received: from mail-il1-x12a.google.com (mail-il1-x12a.google.com [IPv6:2607:f8b0:4864:20::12a]) by smtp3.osuosl.org (Postfix) with ESMTPS id BC1C060AC2; Tue, 17 Oct 2023 18:48:11 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org BC1C060AC2 Received: by mail-il1-x12a.google.com with SMTP id e9e14a558f8ab-352a1a0348fso23473275ab.1; Tue, 17 Oct 2023 11:48:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697568491; x=1698173291; darn=lists.linuxfoundation.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=FHFVypTpwHkCPuykqZIoZks1nhouYL0FqFFysODiX2s=; b=mTbNFFfTtc9lVi8LUrMiNQwAbRgB90xEhN4wxC2OlJ++gzqg1HBm/mYy9E098wkH/I Jsu+fMHcEiwSKwqLxIWy0S/Vxn/OdlJQiQv20r0f63tOigk1lbJHMiD+5boBqfPGf/CU 2PA+URp4/PXGebA0DqRq6MX89K5HN9ldeLPZrl/6/94QOZO18QOjqbZGQP/dPbhUhD+m mgSd/M33GvntlHCYj2wygazd0xZHahDISPcoPB/MVO6FkP4U7kieGajRcyccSqdxDq4T GjaavzuPgewox6rbenFsPmB3q9hmxMtkknbZnKHg5ExOTBDVSs0a7Ae9HE0wEKNfE+BE M4KA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697568491; x=1698173291; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=FHFVypTpwHkCPuykqZIoZks1nhouYL0FqFFysODiX2s=; b=bj0VMNTwcID/W/TljyL2VAiVjFow810POoIbuf3aplWF0LqPd28Ct/XlidhCzTMRU5 zQOWDUSLkjGtGOIsCHOeSOGYy0Bh341zJgOhCJ9O58JIpnZVQRn8uR8tGFrxJUTlVZGG PoDt4QGmXDeB32QzfGIDnV8qt0lNCFVi1OUzuQPYyY98lHMe5wKd6CfroDWeP/Min+np RhfcO3AsvmMVz4odKjqtaPixEVkmxTKiPJcNZQ1HXGz3f/ESL3WP2n4RGhM8fONnstlD WSTAaamgpwjxo1ZEV7ZnUno+dtrMxst4uRrn9SGA0S+hF7gcngx4qOWaeFG9NebM1PMK uiKQ== X-Gm-Message-State: AOJu0Yzhlruo7SE4sfD9t5X+HfZk0t9Grw8n5dRG8xiBLUMmh9u6INbN 0k/dQ41CnW5l+0r0+J0R4ZtBplTIEHf4RD/32Pc= X-Google-Smtp-Source: AGHT+IGf/XtwrTAsIvteviXiJxAMOTuiVShRpEY7wm4hxxq99XdEYgi/Ve9QAxrWJCxrC18DKI7IZGlOMX87ef2MHD0= X-Received: by 2002:a05:6e02:20e8:b0:352:a306:9ad1 with SMTP id q8-20020a056e0220e800b00352a3069ad1mr3953939ilv.0.1697568490761; Tue, 17 Oct 2023 11:48:10 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Antoine Riard Date: Tue, 17 Oct 2023 19:47:59 +0100 Message-ID: To: ziggie1984 Content-Type: multipart/alternative; boundary="000000000000dfb6740607edf671" X-Mailman-Approved-At: Wed, 18 Oct 2023 00:07:44 +0000 Cc: Bitcoin Protocol Discussion , security@ariard.me, "lightning-dev\\\\\\\\@lists.linuxfoundation.org" Subject: Re: [bitcoin-dev] [Lightning-dev] Full Disclosure: CVE-2023-40231 / CVE-2023-40232 / CVE-2023-40233 / CVE-2023-40234 "All your mempool are belong to us" X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Bitcoin Protocol Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Oct 2023 18:48:15 -0000 --000000000000dfb6740607edf671 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable > We have conducted one so far, multiple scenarios to look at. _*none*_ so far. typo of mine - apologize english is not my native language= . We discussed conducting experiments pre-disclosure in an e-mail of the 11th August 2023. "If someone is down to setup a "black box" Lightning infra on mainet, I'm game on to exercise the vulnerabilities and mitigations during the coming months and revisit the disclosure date dependent on the learnings." However as the number of Lightning worldwide experts who have level of knowledge and understandings to take part to experiments is I think restrained to people listed on the disclosure mails _and_ we had other pendings non-disclosed security issues at the time like the ones revealed "fake channel DoS vector" the 23th August 2023, we didn't conduct them. Le mar. 17 oct. 2023 =C3=A0 18:47, Antoine Riard = a =C3=A9crit : > Hi Ziggie, > > > thanks for this detailed explanation. This class of pinning attacks > sound not too unlikely especially if the attacker targets channels with > high capacity and very loose channel policies (allowing the full htlc > > amount to be the channel capacity). Could you add more details about th= e > attack you observed on mainnet ? How did you monitor the chain, are the > some tools available I can run in parallel to my > > lightning software to record this kind of suspicious behaviour (which > did you use)? > > Just to give a clarification no _such_ attack has been observed on > mainnet, since I and other LN maintainers have been aware of this issue. = If > there is a confusion on the disclosure mail thanks to point to it, I'll > correct it. > > We discussed privately to experiment a demo attack in restrained dev > circles, like we have done in the past for some LN sec issues. We have > conducted one so far, multiple scenarios to look at. > > I confirm the risk of exposure if an attacker targets channels with high > capacity and loose channel policies. Note there is no way to configure th= e > cap for the total value of outbound HTLC in-flight, which is the flow > affected. > > If you would like to observe the existence of such an attack happening, > look at your mempool logs and the amount of HTLC output being > systematically conflicted out with the following sequence (HTLC-timeout - > HTLC-preimage - HTLC-timeout - ...). > > As an observation note, this is not akin to a pinning attack, as there is > no "honest" or "malicious" transaction pinned in network mempools. And it > can happen without network mempools congestion. > > > What's also worth mentioning here is that you do not really have to > control 2 neighbouring nodes to target your victim. If you can cycle the > attack on the tail side and delay the confirmation of the htlc- timeout > covenant the peer at the front (incoming link) of the victim will > force-close the channel and claim his timeout-path in the same way > (canceling back the initial htlc amount to the attackers initial node). > > I think this is a behavior worthy of testing. > > > Apart from that I think one can even introduce some kind of feebumping > race between the victim and the attacker on the tail side of the attack > making the attack even more costly. I think > > currently when lightning nodes see the preimage in the mempool (during > the time where they already can spend the same output with the > timeout-covenant) we are honest and just extract > > the preimage and don't try to race this tx output. > > Local-mempool preimage monitoring has been implemented by Eclair for year= s > as a mitigation against old school pinning attacks on second-stage HTLC > transactions. > > This mechanism has been implemented by LND in the last months, following > the report of replacement cycling attacks. As of today this is not > implemented by Core-Lightning or LDK. > > > So we maybe should start feebumping this output if we end up in this > scenario? If we see the preimage and can also claim this output via the > htlc-timeout path, we should aggressively fee-bump (racing this output) o= ur > htlc-output in addition to grabbing the preimage and claiming it on the > incoming. This is only feasible with anchor channels where we can add fee= s > to the htlc-covenant. This would make the attack more costly for a peer > when he knows that we use fees up to 50% of the htlc value. When you cycl= e > this 144 times you will be at a heavy loss trying to steal this htlc. > > This is the "defensive fee mitigation" proposed in the paper. Coming with > some unknown. > > > I would add another mitigation to the list for node runners to restrict > the amount and number of HTLCs for big channels to unknown peers. It > quickly comes with a loss when the HTLCs the attacker tries to steal are > small. > > See the point above on the lack of way at the spec-level to negotiate cap > on the total value of outbound HTLC in-flight. > > Le mar. 17 oct. 2023 =C3=A0 08:21, ziggie1984 = a > =C3=A9crit : > >> ## Deployed LN mitigations >> >> Aggressive rebroadcasting: As the replacement cycling attacker benefits >> from the HTLC-timeout being usually broadcast by lightning nodes only on= ce >> every block, or less the replacement cycling malicious transactions paid >> only equal the sum of the absolute fees paid by the HTLC, adjusted with = the >> replacement penalty. Rebroadcasting randomly and multiple times before t= he >> next block increases the absolute fee cost for the attacker. >> >> Implemented and deployed by Eclair, Core-Lightning, LND and LDK . >> >> Local-mempool preimage monitoring: As the replacement cycling attacker i= n >> a simple setup broadcast the HTLC-preimage to all the network mempools, = the >> honest lightning node is able to catch on the flight the unconfirmed >> HTLC-preimage, before its subsequent mempool replacement. The preimage c= an >> be extracted from the second-stage HTLC-preimage and used to fetch the >> off-chain inbound HTLC with a cooperative message or go on-chain with it= to >> claim the accepted HTLC output. >> >> >> Hi Antoine, >> >> thanks for this detailed explanation. This class of pinning attacks soun= d >> not too unlikely especially if the attacker targets channels with high >> capacity and very loose channel policies (allowing the full htlc amount = to >> be the channel capacity). Could you add more details about the attack yo= u >> observed on mainnet ? How did you monitor the chain, are the some tools >> available I can run in parallel to my lightning software to record this >> kind of suspicious behaviour (which did you use)? >> What's also worth mentioning here is that you do not really have to >> control 2 neighbouring nodes to target your victim. If you can cycle the >> attack on the tail side and delay the confirmation of the htlc-timeout >> covenant the peer at the front (incoming link) of the victim will >> force-close the channel and claim his timeout-path in the same way >> (canceling back the initial htlc amount to the attackers initial node). >> >> Apart from that I think one can even introduce some kind of feebumping >> race between the victim and the attacker on the tail side of the attack >> making the attack even more costly. I think currently when lightning nod= es >> see the preimage in the mempool (during the time where they already can >> spend the same output with the timeout-covenant) we are honest and just >> extract the preimage and don't try to race this tx output. So we maybe >> should start feebumping this output if we end up in this scenario? If we >> see the preimage and can also claim this output via the htlc-timeout pat= h, >> we should aggressively fee-bump (racing this output) our htlc-output in >> addition to grabbing the preimage and claiming it on the incoming. This = is >> only feasible with anchor channels where we can add fees to the >> htlc-covenant. This would make the attack more costly for a peer when he >> knows that we use fees up to 50% of the htlc value. When you cycle this = 144 >> times you will be at a heavy loss trying to steal this htlc. >> >> I would add another mitigation to the list for node runners to restrict >> the amount and number of HTLCs for big channels to unknown peers. It >> quickly comes with a loss when the HTLCs the attacker tries to steal are >> small. >> >> >> Kind regards, >> >> ziggie >> >> >> ------- Original Message ------- >> On Monday, October 16th, 2023 at 18:57, Antoine Riard < >> antoine.riard@gmail.com> wrote: >> >> (cross-posting mempool issues identified are exposing lightning chan to >> loss of funds risks, other multi-party bitcoin apps might be affected) >> >> Hi, >> >> End of last year (December 2022), amid technical discussions on eltoo >> payment channels and incentives compatibility of the mempool anti-DoS >> rules, a new transaction-relay jamming attack affecting lightning channe= ls >> was discovered. >> >> After careful analysis, it turns out this attack is practical and >> immediately exposed lightning routing hops carrying HTLC traffic to loss= of >> funds security risks, both legacy and anchor output channels. A potentia= l >> exploitation plausibly happening even without network mempools congestio= n. >> >> Mitigations have been designed, implemented and deployed by all major >> lightning implementations during the last months. >> >> Please find attached the release numbers, where the mitigations should b= e >> present: >> - LDK: v0.0.118 - CVE-2023 -40231 >> - Eclair: v0.9.0 - CVE-2023-40232 >> - LND: v.0.17.0-beta - CVE-2023-40233 >> - Core-Lightning: v.23.08.01 - CVE-2023-40234 >> >> While neither replacement cycling attacks have been observed or reported >> in the wild since the last ~10 months or experimented in real-world >> conditions on bitcoin mainet, functional test is available exercising th= e >> affected lightning channel against bitcoin core mempool (26.0 release >> cycle). >> >> It is understood that a simple replacement cycling attack does not deman= d >> privileged capabilities from an attacker (e.g no low-hashrate power) and >> only access to basic bitcoin and lightning software. Yet I still think >> executing such an attack successfully requests a fair amount of bitcoin >> technical know-how and decent preparation. >> >> From my understanding of those issues, it is yet to be determined if the >> mitigations deployed are robust enough in face of advanced replacement >> cycling attackers, especially ones able to combine different classes of >> transaction-relay jamming such as pinnings or vetted with more privilege= d >> capabilities. >> >> Please find a list of potential affected bitcoin applications in this >> full disclosure report using bitcoin script timelocks or multi-party >> transactions, albeit no immediate security risk exposure as severe as th= e >> ones affecting lightning has been identified. Only cursory review of >> non-lightning applications has been conducted so far. >> >> There is a paper published summarizing replacement cycling attacks on th= e >> lightning network: >> >> https://github.com/ariard/mempool-research/blob/2023-10-replacement-pape= r/replacement-cycling.pdf >> >> ## Problem >> >> A lightning node allows HTLCs forwarding (in bolt3's parlance accepted >> HTLC on incoming link and offered HTLC on outgoing link) should settle t= he >> outgoing state with either a success or timeout before the incoming stat= e >> timelock becomes final and an asymmetric defavorable settlement might >> happen (cf "Flood & Loot: A Systematic Attack on The Lightning Network" >> section 2.3 for a classical exposition of this lightning security proper= ty). >> >> Failure to satisfy this settlement requirement exposes a forwarding hop >> to a loss of fund risk where the offered HTLC is spent by the outgoing l= ink >> counterparty's HTLC-preimage and the accepted HTLC is spent by the incom= ing >> link counterparty's HTLC-timeout. >> >> The specification mandates the incoming HTLC expiration timelock to be >> spaced out by an interval of `cltv_expiry_delta` from the outgoing HTLC >> expiration timelock, this exact interval value being an implementation a= nd >> node policy setting. As a minimal value, the specification recommends 34 >> blocks of interval. If the timelock expiration I of the inbound HTLC is >> equal to 100 from chain tip, the timelock expiration O of the outbound H= TLC >> must be equal to 66 blocks from chain tip, giving a reasonable buffer of >> reaction to the lightning forwarding node. >> >> In the lack of cooperative off-chain settlement of the HTLC on the >> outgoing link negotiated with the counterparty (either >> `update_fulfill_htlc` or `update_fail_htlc`) when O is reached, the >> lightning node should broadcast its commitment transaction. Once the >> commitment is confirmed (if anchor and the 1 CSV encumbrance is present)= , >> the lightning node broadcasts and confirms its HTLC-timeout before I hei= ght >> is reached. >> >> Here enter a replacement cycling attack. A malicious channel counterpart= y >> can broadcast its HTLC-preimage transaction with a higher absolute fee a= nd >> higher feerate than the honest HTLC-timeout of the victim lightning node >> and triggers a replacement. Both for legacy and anchor output channels, = a >> HTLC-preimage on a counterparty commitment transaction is malleable, i.e >> additional inputs or outputs can be added. The HTLC-preimage spends an >> unconfirmed and unrelated to the channel parent transaction M and confli= cts >> its child. >> >> As the HTLC-preimage spends an unconfirmed input that was already >> included in the unconfirmed and unrelated child transaction (rule 2), pa= ys >> an absolute higher fee of at least the sum paid by the HTLC-timeout and >> child transaction (rule 3) and the HTLC-preimage feerate is greater than >> all directly conflicting transactions (rule 6), the replacement is >> accepted. The honest HTLC-timeout is evicted out of the mempool. >> >> In an ulterior move, the malicious counterparty can replace the parent >> transaction itself with another candidate N satisfying the replacement >> rules, triggering the eviction of the malicious HTLC-preimage from the >> mempool as it was a child of the parent T. >> >> There is no spending candidate of the offered HTLC output for the curren= t >> block laying in network mempools. >> >> This replacement cycling tricks can be repeated for each rebroadcast >> attempt of the HTLC-timeout by the honest lightning node until expiratio= n >> of the inbound HTLC timelock I. Once this height is reached a HTLC-timeo= ut >> is broadcast by the counterparty's on the incoming link in collusion wit= h >> the one on the outgoing link broadcasting its own HTLC-preimage. >> >> The honest Lightning node has been "double-spent" in its HTLC forwarding= . >> >> As a notable factor impacting the success of the attack, a lightning >> node's honest HTLC-timeout might be included in the block template of th= e >> miner winning the block race and therefore realizes a spent of the offer= ed >> output. In practice, a replacement cycling attack might over-connect to >> miners' mempools and public reachable nodes to succeed in a fast evictio= n >> of the HTLC-timeout by its HTLC-preimage. As this latter transaction can >> come with a better ancestor-score, it should be picked up on the flight = by >> economically competitive miners. >> >> A functional test exercising a simple replacement cycling of a HTLC >> transaction on bitcoin core mempool is available: >> https://github.com/ariard/bitcoin/commits/2023-test-mempool >> >> ## Deployed LN mitigations >> >> Aggressive rebroadcasting: As the replacement cycling attacker benefits >> from the HTLC-timeout being usually broadcast by lightning nodes only on= ce >> every block, or less the replacement cycling malicious transactions paid >> only equal the sum of the absolute fees paid by the HTLC, adjusted with = the >> replacement penalty. Rebroadcasting randomly and multiple times before t= he >> next block increases the absolute fee cost for the attacker. >> >> Implemented and deployed by Eclair, Core-Lightning, LND and LDK . >> >> Local-mempool preimage monitoring: As the replacement cycling attacker i= n >> a simple setup broadcast the HTLC-preimage to all the network mempools, = the >> honest lightning node is able to catch on the flight the unconfirmed >> HTLC-preimage, before its subsequent mempool replacement. The preimage c= an >> be extracted from the second-stage HTLC-preimage and used to fetch the >> off-chain inbound HTLC with a cooperative message or go on-chain with it= to >> claim the accepted HTLC output. >> >> Implemented and deployed by Eclair and LND. >> >> CLTV Expiry Delta: With every jammed block comes an absolute fee cost >> paid by the attacker, a risk of the HTLC-preimage being detected or >> discovered by the honest lightning node, or the HTLC-timeout to slip in = a >> winning block template. Bumping the default CLTV delta hardens the odds = of >> success of a simple replacement cycling attack. >> >> Default setting: Eclair 144, Core-Lightning 34, LND 80 and LDK 72. >> >> ## Affected Bitcoin Protocols and Applications >> >> From my understanding the following list of Bitcoin protocols and >> applications could be affected by new denial-of-service vectors under so= me >> level of network mempools congestion. Neither tests or advanced review o= f >> specifications (when available) has been conducted for each of them: >> - on-chain DLCs >> - coinjoins >> - payjoins >> - wallets with time-sensitive paths >> - peerswap and submarine swaps >> - batch payouts >> - transaction "accelerators" >> >> Inviting their developers, maintainers and operators to investigate how >> replacement cycling attacks might disrupt their in-mempool chain of >> transactions, or fee-bumping flows at the shortest delay. Simple flows a= nd >> non-multi-party transactions should not be affected to the best of my >> understanding. >> >> ## Open Problems: Package Malleability >> >> Pinning attacks have been known for years as a practical vector to >> compromise lightning channels funds safety, under different scenarios (c= f. >> current bip331's motivation section). Mitigations at the mempool level h= ave >> been designed, discussed and are under implementation by the community >> (ancestor package relay + nverrsion=3D3 policy). Ideally, they should >> constraint a pinning attacker to always attach a high feerate package >> (commitment + CPFP) to replace the honest package, or allow a honest >> lightning node to overbid a malicious pinning package and get its >> time-sensitive transaction optimistically included in the chain. >> >> Replacement cycling attack seem to offer a new way to neutralize the >> design goals of package relay and its companion nversion=3D3 policy, whe= re an >> attacker package RBF a honest package out of the mempool to subsequently >> double-spend its own high-fee child with a transaction unrelated to the >> channel. As the remaining commitment transaction is pre-signed with a >> minimal relay fee, it can be evicted out of the mempool. >> >> A functional test exercising a simple replacement cycling of a lightning >> channel commitment transaction on top of the nversion=3D3 code branch is >> available: >> https://github.com/ariard/bitcoin/commits/2023-10-test-mempool-2 >> >> ## Discovery >> >> In 2018, the issue of static fees for pre-signed lightning transactions >> is made more widely known, the carve-out exemption in mempool rules to >> mitigate in-mempool package limits pinning and the anchor output pattern >> are proposed. >> >> In 2019, bitcoin core 0.19 is released with carve-out support. Continued >> discussion of the anchor output pattern as a dynamic fee-bumping method. >> >> In 2020, draft of anchor output submitted to the bolts. Initial finding >> of economic pinning against lightning commitment and second-stage HTLC >> transactions. Subsequent discussions of a preimage-overlay network or >> package-relay as mitigations. Public call made to inquiry more on potent= ial >> other transaction-relay jamming attacks affecting lightning. >> >> In 2021, initial work in bitcoin core 22.0 of package acceptance. >> Continued discussion of the pinning attacks and shortcomings of current >> mempool rules during community-wide online workshops. Later the year, in >> light of all issues for bitcoin second-layers, a proposal is made about >> killing the mempool. >> >> In 2022, bip proposed for package relay and new proposed v3 policy desig= n >> proposed for a review and implementation. Mempoolfullrbf is supported in >> bitcoin core 24.0 and conceptual questions about alignment of mempool ru= les >> w.r.t miners incentives are investigated. >> >> Along this year 2022, eltoo lightning channels design are discussed, >> implemented and reviewed. In this context and after discussions on mempo= ol >> anti-DoS rules, I discovered this new replacement cycling attack was >> affecting deployed lightning channels and immediately reported the findi= ng >> to some bitcoin core developers and lightning maintainers. >> >> ## Timeline >> >> - 2022-12-16: Report of the finding to Suhas Daftuar, Anthony Towns, Gre= g >> Sanders and Gloria Zhao >> - 2022-12-16: Report to LN maintainers: Rusty Russell, Bastien >> Teinturier, Matt Corallo and Olaoluwa Osuntunkun >> - 2022-12-23: Sharing to Eugene Siegel (LND) >> - 2022-12-24: Sharing to James O'Beirne and Antoine Poinsot >> (non-lightning potential affected projects) >> - 2022-01-14: Sharing to Gleb Naumenko (miners incentives and >> cross-layers issuers) and initial proposal of an early public disclosure >> - 2022-01-19: Collection of analysis if other second-layers and >> multi-party applications affected. LN mitigations development starts. >> - 2023-05-04: Sharing to Wilmer Paulino (LDK) >> - 2023-06-20: LN mitigations implemented and progressively released. Wee= k >> of the 16 october proposed for full disclosure. >> - 2023-08-10: CVEs assigned by MITRE >> - 2023-10-05: Pre-disclosure of LN CVEs numbers and replacement cycling >> attack existence to security@bitcoincore.org. >> - 2023-10-16: Full disclosure of CVE-2023-40231 / CVE-2023-40232 / >> CVE-2023-40233 / CVE-2023-40234 and replacement cycling attacks >> >> ## Conclusion >> >> Despite the line of mitigations adopted and deployed by current major >> lightning implementations, I believe replacement cycling attacks are sti= ll >> practical for advanced attackers. Beyond this new attack might come as a >> way to partially or completely defeat some of the pinning mitigations wh= ich >> have been working for years as a community. >> >> As of today, it is uncertain to me if lightning is not affected by a mor= e >> severe long-term package malleability critical security issue under curr= ent >> consensus rules, and if any other time-sensitive multi-party protocol, >> designed or deployed isn't de facto affected too (loss of funds or denia= l >> of service). >> >> Assuming analysis on package malleability is correct, it is unclear to m= e >> if it can be corrected by changes in replacement / eviction rules or >> mempool chain of transactions processing strategy. Inviting my technical >> peers and the bitcoin community to look more on this issue, including to >> dissent. I'll be the first one pleased if I'm fundamentally wrong on tho= se >> issues, or if any element has not been weighted with the adequate techni= cal >> accuracy it deserves. >> >> Do not trust, verify. All mistakes and opinions are my own. >> >> Antoine >> >> "meet with Triumph and Disaster. And treat those two impostors just the >> same" - K. >> >> >> --000000000000dfb6740607edf671 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
>=C2=A0We = have conducted one so far, multiple=C2=A0scenarios=C2=A0to look at.<= div>
_*none*_ so far. typo of mine - apologize engl= ish is not my native language.

We discussed c= onducting experiments pre-disclosure in an e-mail of the 11th August 2023.<= /font>

"If someone is down to setup a &q= uot;black box" Lightning infra on mainet, I'm game on to exercise = the vulnerabilities and mitigations during the coming months and revisit th= e disclosure date dependent on the learnings."

However as the number of Lightning worldwide experts who have level of kn= owledge and understandings to take part to experiments is I think restraine= d to people listed on the disclosure mails _and_ we had other pendings non-= disclosed security issues at the time like the ones revealed "fake cha= nnel DoS vector" the 23th August 2023, we didn't conduct them.
Le=C2=A0mar. 17 oct. 2023 =C3=A0=C2=A018:47, Antoine Riard <antoine.riard@gmail.com<= /a>> a =C3=A9crit=C2=A0:
Hi Z= iggie,

>= thanks for this detailed explanation. This class of pinning attacks sound = not too unlikely especially if the attacker=C2=A0targets=C2=A0channels with= high capacity=C2=A0and very loose channel policies (allowing the full htlc=
> amount = to be the channel capacity). Could you add more details about the attack yo= u observed=C2=A0on mainnet ? How did you monitor=C2=A0the chain, are the so= me tools available=C2=A0I can run in parallel to my
> lightning software to record th= is kind of suspicious behaviour (which did you use)?
<= span style=3D"font-family:Arial,sans-serif">
Just to give a clarification no _such_ = attack has been observed on mainnet, since I and other LN maintainers have = been aware of this issue. If there is a confusion on the disclosure mail th= anks to point to it, I'll correct it.

We discussed privately to experiment a demo attack in = restrained dev circles, like we have done in the past for some LN sec issue= s. We have conducted one so far, multiple=C2=A0scenarios=C2=A0to look at.

I confirm the risk of = exposure if an attacker targets channels with high capacity and loose chann= el policies. Note there is no way to configure the cap for the total value = of outbound HTLC in-flight, which is the flow affected.

If you would like to observe the existen= ce of such an attack happening, look at your mempool logs and the amount of= HTLC output being systematically conflicted out with the following sequenc= e (HTLC-timeout - HTLC-preimage - HTLC-timeout - ...).

As an observation note, this is not akin = to a pinning attack, as there is no "honest" or "malicious&q= uot; transaction pinned in network mempools. And it can happen without netw= ork mempools congestion.


I think this is a behavior worthy of testing.

> Apart from that I thi= nk one can even introduce some kind of feebumping race between the victim a= nd the attacker on the tail side of the attack making the attack even more = costly. I think
> currently when lightning nodes see the preimage in t= he mempool (during the time where they already can spend the same output wi= th the timeout-covenant) we are honest and just extract
> the preimage= and don't try to race this tx output.


<= /div>
This mechanism has been implemen= ted by LND in the last months, following the report of replacement cycling = attacks. As of today this is not implemented by Core-Lightning or LDK.

> So we maybe shoul= d start feebumping this output if we end up in this scenario? If we see the= preimage and can also claim this output via the htlc-timeout path, we shou= ld aggressively fee-bump (racing this output) our htlc-output in addition t= o grabbing the preimage and claiming it on the incoming. This is only feasi= ble with anchor channels where we can add fees to the htlc-covenant. This w= ould make the attack more costly for a peer when he knows that we use fees = up to 50% of the htlc value. When you cycle this 144 times you will be at a= heavy loss trying to steal this htlc.=C2=A0

This is the "defensive fee mitig= ation" proposed in the paper. Coming with some unknown.

=
>= I would add another mitigation to the list for node runners to restrict th= e amount and number of HTLCs=C2=A0 for big channels to unknown peers. It qu= ickly comes with a loss when the HTLCs the attacker tries to steal are smal= l.

S= ee the point above on the lack of way at the spec-level to negotiate cap on= the total value of outbound HTLC in-flight.

Le=C2=A0mar. 17 oct. 2023= =C3=A0=C2=A008:21, ziggie1984 <ziggie1984@protonmail.com> a =C3=A9crit=C2=A0= :
## Deployed LN miti= gations

Aggressive rebroadcasting: As the replacement cycling attacker benefits f= rom the HTLC-timeout being usually broadcast by lightning nodes only once e= very block, or less the replacement cycling malicious transactions paid onl= y equal the sum of the absolute fees paid by the HTLC, adjusted with the re= placement penalty. Rebroadcasting randomly and multiple times before the ne= xt block increases the absolute fee cost for the attacker.

Implemented and deplo= yed by Eclair, Core-Lightning, LND and LDK .

Local-mempool preimage monitori= ng: As the replacement cycling attacker in a simple setup broadcast the HTL= C-preimage to all the network mempools, the honest lightning node is able t= o catch on the flight the unconfirmed HTLC-preimage, before its subsequent = mempool replacement. The preimage can be extracted from the second-stage HT= LC-preimage and used to fetch the off-chain inbound HTLC with a cooperative= message or go on-chain with it to claim the accepted HTLC output.

Hi Antoine,

thanks for this detailed explanation. This class of pinning att= acks sound not too unlikely especially if the attacker=C2=A0targets=C2=A0ch= annels with high capacity=C2=A0and very loose channel policies (allowing th= e full htlc amount to be the channel capacity). Could you add more details = about the attack you observed=C2=A0on mainnet ? How did you monitor=C2=A0th= e chain, are the some tools available=C2=A0I can run in parallel to my ligh= tning software to record this kind of suspicious behaviour (which did you u= se)?
What's also wort= h mentioning here is that you do not really have to control 2 neighbouring = nodes to target your victim. If you can cycle the attack on the tail side a= nd delay the confirmation of the htlc-timeout covenant=C2=A0the peer at the= front (incoming link) of the victim will force-close the channel and claim= his timeout-path in the same way (canceling back the initial htlc amount t= o the attackers initial node).

Apart from that I think one can even introduce s= ome kind of feebumping race between the victim and the attacker on the tail= side of the attack making the attack even more costly. I think currently w= hen lightning nodes see the preimage in the mempool (during the time where = they already can spend the same output with the timeout-covenant) we are ho= nest and just extract the preimage and don't try to race this tx output= . So we maybe should start feebumping this output if we end up in this scen= ario? If we see the preimage and can also claim this output via the htlc-ti= meout path, we should aggressively fee-bump (racing this output) our htlc-o= utput in addition to grabbing the preimage and claiming it on the incoming.= This is only feasible with anchor channels where we can add fees to the ht= lc-covenant. This would make the attack more costly for a peer when he know= s that we use fees up to 50% of the htlc value. When you cycle this 144 tim= es you will be at a heavy loss trying to steal this htlc.=C2=A0

I would add another mitiga= tion to the list for node runners to restrict the amount and number of HTLC= s=C2=A0 for big channels to unknown peers. It quickly comes with a loss whe= n the HTLCs the attacker tries to steal are small.


Kind regards,

ziggie


------- Original Message -------
On Monday, October 16th, 2023 at 18:57, Antoine Riard <antoine.riard@gmail.c= om> wrote:

(cross-posting mempool issues identified = are exposing lightning chan to loss of funds risks, other multi-party bitco= in apps might be affected)

Hi,

End o= f last year (December 2022), amid technical discussions on eltoo payment ch= annels and incentives compatibility of the mempool anti-DoS rules, a new tr= ansaction-relay jamming attack affecting lightning channels was discovered.=

After careful analysis, it turns out this attack = is practical and immediately exposed lightning routing hops carrying HTLC t= raffic to loss of funds security risks, both legacy and anchor output chann= els. A potential exploitation plausibly happening even without network memp= ools congestion.

Mitigations have been designed, i= mplemented and deployed by all major lightning implementations during the l= ast months.

Please find attached the release numbe= rs, where the mitigations should be present:
- LDK: v0.0.118 - CV= E-2023 -40231
- Eclair: v0.9.0 - CVE-2023-40232
- LND: = v.0.17.0-beta - CVE-2023-40233
- Core-Lightning: v.23.08.01 - CVE= -2023-40234

While neither replacement cycling atta= cks have been observed or reported in the wild since the last ~10 months or= experimented in real-world conditions on bitcoin mainet, functional test i= s available exercising the affected lightning channel against bitcoin core = mempool (26.0 release cycle).

It is understood tha= t a simple replacement cycling attack does not demand privileged capabiliti= es from an attacker (e.g no low-hashrate power) and only access to basic bi= tcoin and lightning software. Yet I still think executing such an attack su= ccessfully requests a fair amount of bitcoin technical know-how and decent = preparation.

From my understanding of those issues= , it is yet to be determined if the mitigations deployed are robust enough = in face of advanced replacement cycling attackers, especially ones able to = combine different classes of transaction-relay jamming such as pinnings or = vetted with more privileged capabilities.

Please f= ind a list of potential affected bitcoin applications in this full disclosu= re report using bitcoin script timelocks or multi-party transactions, albei= t no immediate security risk exposure as severe as the ones affecting light= ning has been identified. Only cursory review of non-lightning applications= has been conducted so far.

There is a paper publi= shed summarizing replacement cycling attacks on the lightning network:

#= # Problem

A lightning node allows HTLCs forwarding= (in bolt3's parlance accepted HTLC on incoming link and offered HTLC o= n outgoing link) should settle the outgoing state with either a success or = timeout before the incoming state timelock becomes final and an asymmetric = defavorable settlement might happen (cf "Flood & Loot: A Systemati= c Attack on The Lightning Network" section 2.3 for a classical exposit= ion of this lightning security property).

Failure = to satisfy this settlement requirement exposes a forwarding hop to a loss o= f fund risk where the offered HTLC is spent by the outgoing link counterpar= ty's HTLC-preimage and the accepted HTLC is spent by the incoming link = counterparty's HTLC-timeout.

The specification= mandates the incoming HTLC expiration timelock to be spaced out by an inte= rval of `cltv_expiry_delta` from the outgoing HTLC expiration timelock, thi= s exact interval value being an implementation and node policy setting. As = a minimal value, the specification recommends 34 blocks of interval. If the= timelock expiration I of the inbound HTLC is equal to 100 from chain tip, = the timelock expiration O of the outbound HTLC must be equal to 66 blocks f= rom chain tip, giving a reasonable buffer of reaction to the lightning forw= arding node.

In the lack of cooperative off-chain = settlement of the HTLC on the outgoing link negotiated with the counterpart= y (either `update_fulfill_htlc` or `update_fail_htlc`) when O is reached, t= he lightning node should broadcast its commitment transaction. Once the com= mitment is confirmed (if anchor and the 1 CSV encumbrance is present), the = lightning node broadcasts and confirms its HTLC-timeout before I height is = reached.

Here enter a replacement cycling attack. = A malicious channel counterparty can broadcast its HTLC-preimage transactio= n with a higher absolute fee and higher feerate than the honest HTLC-timeou= t of the victim lightning node and triggers a replacement. Both for legacy = and anchor output channels, a HTLC-preimage on a counterparty commitment tr= ansaction is malleable, i.e additional inputs or outputs can be added. The = HTLC-preimage spends an unconfirmed and unrelated to the channel parent tra= nsaction M and conflicts its child.

As the HTLC-pr= eimage spends an unconfirmed input that was already included in the unconfi= rmed and unrelated child transaction (rule 2), pays an absolute higher fee = of at least the sum paid by the HTLC-timeout and child transaction (rule 3)= and the HTLC-preimage feerate is greater than all directly conflicting tra= nsactions (rule 6), the replacement is accepted. The honest HTLC-timeout is= evicted out of the mempool.

In an ulterior move, = the malicious counterparty can replace the parent transaction itself with a= nother candidate N satisfying the replacement rules, triggering the evictio= n of the malicious HTLC-preimage from the mempool as it was a child of the = parent T.

There is no spending candidate of the of= fered HTLC output for the current block laying in network mempools.

This replacement cycling tricks can be repeated for each = rebroadcast attempt of the HTLC-timeout by the honest lightning node until = expiration of the inbound HTLC timelock I. Once this height is reached a HT= LC-timeout is broadcast by the counterparty's on the incoming link in c= ollusion with the one on the outgoing link broadcasting its own HTLC-preima= ge.

The honest Lightning node has been "doubl= e-spent" in its HTLC forwarding.

As a notable= factor impacting the success of the attack, a lightning node's honest = HTLC-timeout might be included in the block template of the miner winning t= he block race and therefore realizes a spent of the offered output. In prac= tice, a replacement cycling attack might over-connect to miners' mempoo= ls and public reachable nodes to succeed in a fast eviction of the HTLC-tim= eout by its HTLC-preimage. As this latter transaction can come with a bette= r ancestor-score, it should be picked up on the flight by economically comp= etitive miners.

A functional test exercising a sim= ple replacement cycling of a HTLC transaction on bitcoin core mempool is av= ailable:
<= br>
## Deployed LN mitigations

Aggressiv= e rebroadcasting: As the replacement cycling attacker benefits from the HTL= C-timeout being usually broadcast by lightning nodes only once every block,= or less the replacement cycling malicious transactions paid only equal the= sum of the absolute fees paid by the HTLC, adjusted with the replacement p= enalty. Rebroadcasting randomly and multiple times before the next block in= creases the absolute fee cost for the attacker.

Im= plemented and deployed by Eclair, Core-Lightning, LND and LDK .
<= br>
Local-mempool preimage monitoring: As the replacement cycling= attacker in a simple setup broadcast the HTLC-preimage to all the network = mempools, the honest lightning node is able to catch on the flight the unco= nfirmed HTLC-preimage, before its subsequent mempool replacement. The preim= age can be extracted from the second-stage HTLC-preimage and used to fetch = the off-chain inbound HTLC with a cooperative message or go on-chain with i= t to claim the accepted HTLC output.

Implemented a= nd deployed by Eclair and LND.

CLTV Expiry Del= ta: With every jammed block comes an absolute fee cost paid by the attacker= , a risk of the HTLC-preimage being detected or discovered by the honest li= ghtning node, or the HTLC-timeout to slip in a winning block template. Bump= ing the default CLTV delta hardens the odds of success of a simple replacem= ent cycling attack.

Default setting: Eclair 144, C= ore-Lightning 34, LND 80 and LDK 72.

## Affected B= itcoin Protocols and Applications

From my understa= nding the following list of Bitcoin protocols and applications could be aff= ected by new denial-of-service vectors under some level of network mempools= congestion. Neither tests or advanced review of specifications (when avail= able) has been conducted for each of them:
- on-chain DLCs
<= div>- coinjoins
- payjoins
- wallets with time-sensitiv= e paths
- peerswap and submarine swaps
- batch payouts<= /div>
- transaction "accelerators"

I= nviting their developers, maintainers and operators to investigate how repl= acement cycling attacks might disrupt their in-mempool chain of transaction= s, or fee-bumping flows at the shortest delay. Simple flows and non-multi-p= arty transactions should not be affected to the best of my understanding.

## Open Problems: Package Malleability
Pinning attacks have been known for years as a practical vecto= r to compromise lightning channels funds safety, under different scenarios = (cf. current bip331's motivation section). Mitigations at the mempool l= evel have been designed, discussed and are under implementation by the comm= unity (ancestor package relay + nverrsion=3D3 policy). Ideally, they should= constraint a pinning attacker to always attach a high feerate package (com= mitment + CPFP) to replace the honest package, or allow a honest lightning = node to overbid a malicious pinning package and get its time-sensitive tran= saction optimistically included in the chain.

Repl= acement cycling attack seem to offer a new way to neutralize the design goa= ls of package relay and its companion nversion=3D3 policy, where an attacke= r package RBF a honest package out of the mempool to subsequently double-sp= end its own high-fee child with a transaction unrelated to the channel. As = the remaining commitment transaction is pre-signed with a minimal relay fee= , it can be evicted out of the mempool.

A function= al test exercising a simple replacement cycling of a lightning channel comm= itment transaction on top of the nversion=3D3 code branch is available:
## Discovery

In 2018, the issue of sta= tic fees for pre-signed lightning transactions is made more widely known, t= he carve-out exemption in mempool rules to mitigate in-mempool package limi= ts pinning and the anchor output pattern are proposed.

=
In 2019, bitcoin core 0.19 is released with carve-out support. Continu= ed discussion of the anchor output pattern as a dynamic fee-bumping method.=

In 2020, draft of anchor output submitted to the = bolts. Initial finding of economic pinning against lightning commitment and= second-stage HTLC transactions. Subsequent discussions of a preimage-overl= ay network or package-relay as mitigations. Public call made to inquiry mor= e on potential other transaction-relay jamming attacks affecting lightning.=

In 2021, initial work in bitcoin core 22.0 of pac= kage acceptance. Continued discussion of the pinning attacks and shortcomin= gs of current mempool rules during community-wide online workshops. Later t= he year, in light of all issues for bitcoin second-layers, a proposal is ma= de about killing the mempool.

In 2022, bip propose= d for package relay and new proposed v3 policy design proposed for a review= and implementation. Mempoolfullrbf is supported in bitcoin core 24.0 and c= onceptual questions about alignment of mempool rules w.r.t miners incentive= s are investigated.

Along this year 2022, eltoo li= ghtning channels design are discussed, implemented and reviewed. In this co= ntext and after discussions on mempool anti-DoS rules, I discovered this ne= w replacement cycling attack was affecting deployed lightning channels and = immediately reported the finding to some bitcoin core developers and lightn= ing maintainers.

## Timeline

<= div>- 2022-12-16: Report of the finding to Suhas Daftuar, Anthony Towns, Gr= eg Sanders and Gloria Zhao
- 2022-12-16: Report to LN maintainers= : Rusty Russell, Bastien Teinturier, Matt Corallo and Olaoluwa Osuntunkun
- 2022-12-23: Sharing to Eugene Siegel (LND)
- 2022-12-2= 4: Sharing to James O'Beirne and Antoine Poinsot (non-lightning potenti= al affected projects)
- 2022-01-14: Sharing to Gleb Naumenko (min= ers incentives and cross-layers issuers) and initial proposal of an early p= ublic disclosure
- 2022-01-19: Collection of analysis if other s= econd-layers and multi-party applications affected. LN mitigations developm= ent starts.
- 2023-05-04: Sharing to Wilmer Paulino (LDK)
- 2023-06-20: LN mitigations implemented and progressively released. Wee= k of the 16 october proposed for full disclosure.
- 2023-08-10: C= VEs assigned by MITRE
- 2023-10-05: Pre-disclosure of LN CVEs num= bers and replacement cycling attack existence to se= curity@bitcoincore.org.
- 2023-10-16: Full disclosure of CVE-= 2023-40231 / CVE-2023-40232 / CVE-2023-40233 / CVE-2023-40234 and replaceme= nt cycling attacks

## Conclusion

Despite the line of mitigations adopted and deployed by current ma= jor lightning implementations, I believe replacement cycling attacks are st= ill practical for advanced attackers. Beyond this new attack might come as = a way to partially or completely defeat some of the pinning mitigations whi= ch have been working for years as a community.

As = of today, it is uncertain to me if lightning is not affected by a more seve= re long-term package malleability critical security issue under current con= sensus rules, and if any other time-sensitive multi-party protocol, designe= d or deployed isn't de facto affected too (loss of funds or denial of s= ervice).

Assuming analysis on package malleability= is correct, it is unclear to me if it can be corrected by changes in repla= cement / eviction rules or mempool chain of transactions processing strateg= y. Inviting my technical peers and the bitcoin community to look more on th= is issue, including to dissent. I'll be the first one pleased if I'= m fundamentally wrong on those issues, or if any element has not been weigh= ted with the adequate technical accuracy it deserves.

<= div>Do not trust, verify. All mistakes and opinions are my own.
<= br>
Antoine

"meet with Triumph and = Disaster. And treat those two impostors just the same" - K.

--000000000000dfb6740607edf671--