diff options
| author | Antoine Riard <antoine.riard@gmail.com> | 2021-10-26 19:44:45 -0400 |
|---|---|---|
| committer | bitcoindev <bitcoindev@gnusha.org> | 2021-10-26 23:45:00 +0000 |
| commit | 68f9c8f234da2b9c4d43125edaa93267f53a4a17 (patch) | |
| tree | 55a548d814875a6f421423ab9814a28b438cdb9f | |
| parent | ffe0450c59ed2d643be0a320ee9473977de7d45c (diff) | |
| download | pi-bitcoindev-68f9c8f234da2b9c4d43125edaa93267f53a4a17.tar.gz pi-bitcoindev-68f9c8f234da2b9c4d43125edaa93267f53a4a17.zip | |
Re: [bitcoin-dev] death to the mempool, long live the mempool
| -rw-r--r-- | be/12a18276aad203bce633d5222b3b604b042339 | 322 |
1 files changed, 322 insertions, 0 deletions
diff --git a/be/12a18276aad203bce633d5222b3b604b042339 b/be/12a18276aad203bce633d5222b3b604b042339 new file mode 100644 index 000000000..18e6f942b --- /dev/null +++ b/be/12a18276aad203bce633d5222b3b604b042339 @@ -0,0 +1,322 @@ +Return-Path: <antoine.riard@gmail.com> +Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) + by lists.linuxfoundation.org (Postfix) with ESMTP id AB4C9C000E + for <bitcoin-dev@lists.linuxfoundation.org>; + Tue, 26 Oct 2021 23:45:00 +0000 (UTC) +Received: from localhost (localhost [127.0.0.1]) + by smtp4.osuosl.org (Postfix) with ESMTP id 879F9403E6 + for <bitcoin-dev@lists.linuxfoundation.org>; + Tue, 26 Oct 2021 23:45:00 +0000 (UTC) +X-Virus-Scanned: amavisd-new at osuosl.org +X-Spam-Flag: NO +X-Spam-Score: -2.098 +X-Spam-Level: +X-Spam-Status: No, score=-2.098 tagged_above=-999 required=5 + tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, + DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, + HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, + SPF_PASS=-0.001] autolearn=ham autolearn_force=no +Authentication-Results: smtp4.osuosl.org (amavisd-new); + dkim=pass (2048-bit key) header.d=gmail.com +Received: from smtp4.osuosl.org ([127.0.0.1]) + by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id guC9uAnaco6X + for <bitcoin-dev@lists.linuxfoundation.org>; + Tue, 26 Oct 2021 23:44:59 +0000 (UTC) +X-Greylist: whitelisted by SQLgrey-1.8.0 +Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com + [IPv6:2a00:1450:4864:20::429]) + by smtp4.osuosl.org (Postfix) with ESMTPS id 22F6740235 + for <bitcoin-dev@lists.linuxfoundation.org>; + Tue, 26 Oct 2021 23:44:59 +0000 (UTC) +Received: by mail-wr1-x429.google.com with SMTP id d3so967169wrh.8 + for <bitcoin-dev@lists.linuxfoundation.org>; + Tue, 26 Oct 2021 16:44:58 -0700 (PDT) +DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; + h=mime-version:references:in-reply-to:from:date:message-id:subject:to; + bh=vGm8PaQupt1LhBLMa+SVxKlWGB1tU0IBKUjd2w/Xc3U=; + b=oTuH2Kb0ldx8jYGQlE5kHQIbz7oTkkvDAFAqqPfSe9gcsZf0ijQp2M1Fyj64yrex+D + k5L6dTAy/gDUCc7Nd/iHyUMhy8AXWdg/QsXRyWfDrWXujFZTFVrqBIuAAs0R9tZDFxB4 + El1d22kNCNIvlagT40qHzBD5DMnFvORcKBJh7DF0fOjtMm8kLt33R8forYdja10gia+9 + ipIpm44kPvwM3R3VPPcYCqWP89hSPcAByJB6IFQLC1E8GjJG9bjbKrO8iYUNX/vvsBrg + SJfuX4pXYiSCrstB4OY8qSYysPwVtbG5BrbXOJ47ara0Ii7k6HchbSa16f8aIogey4/A + nq9Q== +X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; + d=1e100.net; s=20210112; + h=x-gm-message-state:mime-version:references:in-reply-to:from:date + :message-id:subject:to; + bh=vGm8PaQupt1LhBLMa+SVxKlWGB1tU0IBKUjd2w/Xc3U=; + b=EJnlBil7abrcW/1FEuRkcmxHQSnKYNeUCaNAfiXGtvJtz1IftkWmf0DLQLF1bfyyb0 + N/m6//dgu80jEgbKu/8ZcyaGKBrQzL3TMQwvhadV8MAznpJ9ugjjHfkqv9orLwi/LRAK + ZbQcmsAD5aWalQXQ7q7+t7aRbGSI+psFwn1ms+vlPgv3Upk1l7x7sTjyxGBb4QMNr7C9 + zQ2/hKUQdibOnwPTWJNdMmx2RaiB1aedPFHVkRef5bUJyBhhfrHhJPSd7wyf3gvDtxJD + 0zVhvjYaoEwtYXTOkYViiQlSdYbwFTzuBJagVEMW4l/m2uYKALCBhZB3tVSsWnwvAVSF + Q1lQ== +X-Gm-Message-State: AOAM533GS9PX6weMvdtonYegC9dalnjX5CWm3IMZsepVU+AImZiDcu2X + CHGn6C6AzaX9H8QX1T7D8Gs/DY/8kyJk9MKScZNMkJMdkynjqQ== +X-Google-Smtp-Source: ABdhPJxcLWUtA162Ap7REdouitYq1p2yvTmwjFaP/eC+Sqr/w3TYaotoyzeFgQMYi7trXl6fitTyzxy+eOIXgcyBfik= +X-Received: by 2002:a5d:5274:: with SMTP id l20mr9042439wrc.328.1635291897361; + Tue, 26 Oct 2021 16:44:57 -0700 (PDT) +MIME-Version: 1.0 +References: <CAM1a7P04apCwwmqNp4VLRam5_uk59tVRWv74UVD_g0-DUGNghw@mail.gmail.com> +In-Reply-To: <CAM1a7P04apCwwmqNp4VLRam5_uk59tVRWv74UVD_g0-DUGNghw@mail.gmail.com> +From: Antoine Riard <antoine.riard@gmail.com> +Date: Tue, 26 Oct 2021 19:44:45 -0400 +Message-ID: <CALZpt+E_47wmDg3Z2N5=z5x3cca5vSxP6fgYzMK_pybwp1161g@mail.gmail.com> +To: lisa neigut <niftynei@gmail.com>, + Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org> +Content-Type: multipart/alternative; boundary="000000000000a577a705cf4a1065" +X-Mailman-Approved-At: Wed, 27 Oct 2021 07:37:33 +0000 +Subject: Re: [bitcoin-dev] death to the mempool, long live the mempool +X-BeenThere: bitcoin-dev@lists.linuxfoundation.org +X-Mailman-Version: 2.1.15 +Precedence: list +List-Id: Bitcoin Protocol Discussion <bitcoin-dev.lists.linuxfoundation.org> +List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>, + <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe> +List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/> +List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org> +List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help> +List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>, + <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe> +X-List-Received-Date: Tue, 26 Oct 2021 23:45:00 -0000 + +--000000000000a577a705cf4a1065 +Content-Type: text/plain; charset="UTF-8" +Content-Transfer-Encoding: quoted-printable + +Hi Lisa, + +Network mempools constitute a blockspace marketplace where block demand +meets the offer in real-time. Block producers are acting to discover the +best feerate bids compensating for their operational costs and transaction +proposers are acting to offer the best feerate in function of their +confirmation preferences. + +Of course in a distributed system like bitcoin, we can't guarantee perfect +information from the market participants. But moving away from this model +by decreasing the ability of the non-mining nodes to observe the current +demand is softening the requirements for potential attackers. + +As transaction proposers are competing with each other to publish, they +have an interest to "front-run" each other by querying the pending +transactions to the block producers instead of observing only the published +blocks. Therefore good connections to +the block producers are now critical and censorship-resistance of the +mining endpoints must be guaranteed. + +Such a list of endpoints couldn't be static otherwise it's an artificial +barrier to enter in the mining competition, and as such a centralization +vector. Dynamic, trust-minimized discovery of the mining endpoints assumes +an address-relay network, of which the robustness must be high enough +against sophisticated sybil attacks. One current defense mechanism in core +to achieve that is selecting outbound peers based in different /16 subnets +as it's harder for an attacker to obtain IP addresses. Replicating this +mechanism for the mining endpoints binds the mining topology to the +Internet one, which is downgrading the mining competition. + +Relying on tor to guarantee the confidentiality of the transaction +announcement is raising its own issues. Flowing by default all the bitcoin +traffic over tor will change the incentive structure of tor attackers, +potentially attracting a new class of attackers able to do deanonymization +attacks, not that expensive in practice [0]. Tor bridges are another +censorship vector as the fingerprint of the bitcoin traffic (a block every +10 min, etc) make it possible to drop or delay the tor channel, in the lack +of high-bandwidth consuming "synthetic" traffic. + +Further, identified mining endpoints make it easier to launch partition +attacks, where mining mempools are sent low-feerate clusters of +transactions, to prevent the replacement by a better feerate offer. This is +especially concerning for L2 nodes with time-sensitive requirements [1] + +Lastly, removing the mempool won't solve the current issues inherent with +pre-signed transactions under the mempool min fee as ultimately miner's +mempools are also finite in memory and a dynamic lower bound must exist to +prevent spam. These lower bounds potentially increase after the signature +exchange of the time-sensitive transactions. + +Antoine + +[0] https://www.usenix.org/system/files/sec19-jansen.pdf +[1] See "The Ugly" +https://lists.linuxfoundation.org/pipermail/lightning-dev/2020-June/002758.= +html + +Le mar. 26 oct. 2021 =C3=A0 03:37, lisa neigut via bitcoin-dev < +bitcoin-dev@lists.linuxfoundation.org> a =C3=A9crit : + +> Hi all, +> +> In a recent conversation with @glozow, I had the realization that the +> mempool is obsolete and should be eliminated. +> +> Instead, users should submit their transactions directly to mining pools, +> preferably over an anonymous communication network such as tor. This can +> easily be achieved by mining pools running a tor onion node for this +> express purpose (or via a lightning network extension etc) +> +> Mempools make sense in a world where mining is done by a large number of +> participating nodes, eg where the block template is constructed by a +> majority of the participants on the network. In this case, it is necessar= +y +> to socialize pending transaction data to all participants, as you don=E2= +=80=99t +> know which participant will be constructing the winning block template. +> +> In reality however, mempool relay is unnecessary where the majority of +> hashpower and thus block template creation is concentrated in a +> semi-restricted set. +> +> Removing the mempool would greatly reduce the bandwidth requirement for +> running a node, keep intentionality of transactions private until +> confirmed/irrevocable, and naturally resolve all current issues inherent = +in +> package relay and rbf rules. It also resolves the recent minimum relay +> questions, as relay is no longer a concern for unmined transactions. +> +> Provided the number of block template producing actors remains beneath, +> say 1000, it=E2=80=99d be quite feasible to publish a list of tor endpoin= +ts that +> nodes can independently + directly submit their transactions to. In fact= +, +> merely allowing users to select their own list of endpoints to use +> alternatively to the mempool would be a low effort starting point for the +> eventual replacement. +> +> On the other hand, removing the mempool would greatly complicate solo +> mining and would also make BetterHash proposals, which move the block +> template construction away from a centralized mining pool back to the +> individual miner, much more difficult. It also makes explicit the target +> for DoS attacks. +> +> A direct communication channel between block template construction venues +> and transaction proposers also provides a venue for direct feedback wrt +> acceptable feerates at the time, which both makes transaction confirmatio= +n +> timelines less variable as well as provides block producers a mechanism f= +or +> (independently) enforcing their own minimum security budget. In other +> words, expressing a minimum acceptable feerate for continued operation. +> +> Initial feerate estimation would need to be based on published blocks, no= +t +> pending transactions (as this information would no longer be available), = +or +> from direct interactions with block producers. +> +> +> ~niftynei +> _______________________________________________ +> bitcoin-dev mailing list +> bitcoin-dev@lists.linuxfoundation.org +> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev +> + +--000000000000a577a705cf4a1065 +Content-Type: text/html; charset="UTF-8" +Content-Transfer-Encoding: quoted-printable + +<div dir=3D"ltr"><div>Hi Lisa,<br><br>Network mempools constitute a blocksp= +ace marketplace where block demand meets the offer in real-time. Block prod= +ucers are acting to discover the best feerate bids compensating for their o= +perational costs and transaction proposers are acting to offer the best fee= +rate in function of their confirmation preferences.<br><br>Of course in a d= +istributed system like bitcoin, we can't guarantee perfect information = +from the market participants. But moving away from this model by decreasing= + the ability of the non-mining nodes to observe the current demand is softe= +ning the requirements for potential attackers.<br><br>As transaction propos= +ers are competing with each other to publish, they have an interest to &quo= +t;front-run" each other by querying the pending transactions to the bl= +ock producers instead of observing only the published blocks. Therefore goo= +d connections to<br>the block producers are now critical and censorship-res= +istance of the mining endpoints must be guaranteed.<br><br>Such a list of e= +ndpoints couldn't be static otherwise it's an artificial barrier to= + enter in the mining competition, and as such a centralization vector. Dyna= +mic, trust-minimized discovery of the mining endpoints assumes an address-r= +elay network, of which the robustness must be high enough against sophistic= +ated sybil attacks. One current defense mechanism in core to achieve that i= +s selecting outbound peers based in different /16 subnets as it's harde= +r for an attacker to obtain IP addresses. Replicating this mechanism for th= +e mining endpoints binds the mining topology to the Internet one, which is = +downgrading the mining competition.<br><br>Relying on tor to guarantee the = +confidentiality of the transaction announcement is raising its own issues. = +Flowing by default all the bitcoin traffic over tor will change the incenti= +ve structure of tor attackers, potentially attracting a new class of attack= +ers able to do deanonymization attacks, not that expensive in practice [0].= + Tor bridges are another censorship vector as the fingerprint of the bitcoi= +n traffic (a block every 10 min, etc) make it possible to drop or delay the= + tor channel, in the lack of high-bandwidth consuming "synthetic"= + traffic.<br><br>Further, identified mining endpoints make it easier to lau= +nch partition attacks, where mining mempools are sent low-feerate clusters = +of transactions, to prevent the replacement by a better feerate offer. This= + is especially concerning for L2 nodes with time-sensitive requirements [1]= +<br><br>Lastly, removing the mempool won't solve the current issues inh= +erent with pre-signed transactions under the mempool min fee as ultimately = +miner's mempools are also finite in memory and a dynamic lower bound mu= +st exist to prevent spam. These lower bounds potentially increase after the= + signature exchange of the time-sensitive transactions.<br><br></div>Antoin= +e<br><div><br>[0] <a href=3D"https://www.usenix.org/system/files/sec19-jans= +en.pdf">https://www.usenix.org/system/files/sec19-jansen.pdf</a><br>[1] See= + "The Ugly" <a href=3D"https://lists.linuxfoundation.org/pipermai= +l/lightning-dev/2020-June/002758.html">https://lists.linuxfoundation.org/pi= +permail/lightning-dev/2020-June/002758.html</a><br></div></div><br><div cla= +ss=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">Le=C2=A0mar. 26 oc= +t. 2021 =C3=A0=C2=A003:37, lisa neigut via bitcoin-dev <<a href=3D"mailt= +o:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.linuxfoundation.= +org</a>> a =C3=A9crit=C2=A0:<br></div><blockquote class=3D"gmail_quote" = +style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);pa= +dding-left:1ex"><div><div dir=3D"auto">Hi all,</div><div dir=3D"auto"><br><= +/div><div dir=3D"auto">In a recent conversation with @glozow, I had the rea= +lization that the mempool is obsolete and should be eliminated.</div><div d= +ir=3D"auto"><br></div><div dir=3D"auto">Instead, users should submit their = +transactions directly to mining pools, preferably over an anonymous communi= +cation network such as tor. This can easily be achieved by mining pools run= +ning a tor onion node for this express purpose (or via a lightning network = +extension etc)</div><div dir=3D"auto"><br></div><div dir=3D"auto">Mempools = +make sense in a world where mining is done by a large number of participati= +ng nodes, eg where the block template is constructed by a majority of the p= +articipants on the network. In this case, it is necessary to socialize pend= +ing transaction data to all participants, as you don=E2=80=99t know which p= +articipant will be constructing the winning block template.</div><div dir= +=3D"auto"><br></div><div dir=3D"auto">In reality however, mempool relay is = +unnecessary where the majority of hashpower and thus block template creatio= +n is concentrated in a semi-restricted set.=C2=A0</div><div dir=3D"auto"><b= +r></div><div dir=3D"auto">Removing the mempool would greatly reduce the ban= +dwidth requirement for running a node, keep intentionality of transactions = +private until confirmed/irrevocable, and naturally resolve all current issu= +es inherent in package relay and rbf rules. It also resolves the recent min= +imum relay questions, as relay is no longer a concern for unmined transacti= +ons.</div><div dir=3D"auto"><br></div><div dir=3D"auto">Provided the number= + of block template producing actors remains beneath, say 1000, it=E2=80=99d= + be quite feasible to publish a list of tor endpoints that nodes can indepe= +ndently =C2=A0+ directly submit their transactions to. In fact, merely allo= +wing users to select their own list of endpoints to use alternatively to th= +e mempool would be a low effort starting point for the eventual replacement= +.</div><div dir=3D"auto"><br></div><div dir=3D"auto">On the other hand, rem= +oving the mempool would greatly complicate solo mining and would also make = +BetterHash proposals, which move the block template construction away from = +a centralized mining pool back to the individual miner, much more difficult= +. It also makes explicit the target for DoS attacks.</div><div dir=3D"auto"= +><br></div><div dir=3D"auto">A direct communication channel between block t= +emplate construction venues and transaction proposers also provides a venue= + for direct feedback wrt acceptable feerates at the time, which both makes = +transaction confirmation timelines less variable as well as provides block = +producers a mechanism for (independently) enforcing their own minimum secur= +ity budget. In other words, expressing a minimum acceptable feerate for con= +tinued operation.</div><div dir=3D"auto"><br></div><div dir=3D"auto">Initia= +l feerate estimation would need to be based on published blocks, not pendin= +g transactions (as this information would no longer be available), or from = +direct interactions with block producers.</div><div dir=3D"auto"><br></div>= +<div dir=3D"auto"><br></div><div dir=3D"auto">~niftynei</div> +</div> +_______________________________________________<br> +bitcoin-dev mailing list<br> +<a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org" target=3D"_blank">= +bitcoin-dev@lists.linuxfoundation.org</a><br> +<a href=3D"https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev" = +rel=3D"noreferrer" target=3D"_blank">https://lists.linuxfoundation.org/mail= +man/listinfo/bitcoin-dev</a><br> +</blockquote></div> + +--000000000000a577a705cf4a1065-- + |