diff options
| author | Keagan McClelland <keagan.mcclelland@gmail.com> | 2021-03-01 13:55:10 -0700 |
|---|---|---|
| committer | bitcoindev <bitcoindev@gnusha.org> | 2021-03-01 20:55:24 +0000 |
| commit | 3784da79dc18a2f9a303f5ed554feb1aa6f5a063 (patch) | |
| tree | 5cb29eb4f67451ff68cbee69b48a176d02b9baac | |
| parent | 7a0b1b54467d9fbfeb01c9fa3a00bf72b0731c4f (diff) | |
| download | pi-bitcoindev-3784da79dc18a2f9a303f5ed554feb1aa6f5a063.tar.gz pi-bitcoindev-3784da79dc18a2f9a303f5ed554feb1aa6f5a063.zip | |
Re: [bitcoin-dev] A design for Probabilistic Partial Pruning
| -rw-r--r-- | 9e/1ec16fede91e0c90010a16c3e7cf4674a83f7d | 248 |
1 files changed, 248 insertions, 0 deletions
diff --git a/9e/1ec16fede91e0c90010a16c3e7cf4674a83f7d b/9e/1ec16fede91e0c90010a16c3e7cf4674a83f7d new file mode 100644 index 000000000..4c2f903cb --- /dev/null +++ b/9e/1ec16fede91e0c90010a16c3e7cf4674a83f7d @@ -0,0 +1,248 @@ +Return-Path: <keagan.mcclelland@gmail.com> +Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) + by lists.linuxfoundation.org (Postfix) with ESMTP id B05BFC0001 + for <bitcoin-dev@lists.linuxfoundation.org>; + Mon, 1 Mar 2021 20:55:24 +0000 (UTC) +Received: from localhost (localhost [127.0.0.1]) + by smtp4.osuosl.org (Postfix) with ESMTP id 9E2984EC14 + for <bitcoin-dev@lists.linuxfoundation.org>; + Mon, 1 Mar 2021 20:55:24 +0000 (UTC) +X-Virus-Scanned: amavisd-new at osuosl.org +X-Spam-Flag: NO +X-Spam-Score: -0.2 +X-Spam-Level: +X-Spam-Status: No, score=-0.2 tagged_above=-999 required=5 + tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, + DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, + RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] + autolearn=ham autolearn_force=no +Authentication-Results: smtp4.osuosl.org (amavisd-new); + dkim=pass (2048-bit key) header.d=gmail.com +Received: from smtp4.osuosl.org ([127.0.0.1]) + by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id MB09ykEDiqyt + for <bitcoin-dev@lists.linuxfoundation.org>; + Mon, 1 Mar 2021 20:55:23 +0000 (UTC) +X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 +Received: from mail-wr1-f52.google.com (mail-wr1-f52.google.com + [209.85.221.52]) + by smtp4.osuosl.org (Postfix) with ESMTPS id 114414EC0D + for <bitcoin-dev@lists.linuxfoundation.org>; + Mon, 1 Mar 2021 20:55:22 +0000 (UTC) +Received: by mail-wr1-f52.google.com with SMTP id h98so17578446wrh.11 + for <bitcoin-dev@lists.linuxfoundation.org>; + Mon, 01 Mar 2021 12:55:22 -0800 (PST) +DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; + h=mime-version:references:in-reply-to:from:date:message-id:subject:to; + bh=W8CHiQv7S2RR8jRdJAIAF73fpdaGNdLrkVTCvpnT8lk=; + b=fX3NDwT1/WqhBXGH/kwz7v+XmuezAwgousr5Z/AHmXJKOV+ADij30stGQ1THvUTKhh + dBnn/M4RfQD64WfWPjdNUG0yt3LrJKJgT2pcMGPUh9bbusCIbu5pFdxVxz2vB/qZM7Nr + P1/oCM+BX/eJ6ZtSkeRh6vrITReukGbol5e0k6mbxS/ddd92wJxo4VJ8IxxytukaZwJj + X+QobceAR6kUgbngvbLwyPU25znTOUgb+plqnMigxti3vSRe+hgN7XYSMQEfXlCdGWpt + 1vCWuYma/b1Cy5pkMAC3zlC3ZLldcrPW2rNWFYn3G5N83ItaPhmfeDET3PrBmFVbzmxR + jguQ== +X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; + d=1e100.net; s=20161025; + h=x-gm-message-state:mime-version:references:in-reply-to:from:date + :message-id:subject:to; + bh=W8CHiQv7S2RR8jRdJAIAF73fpdaGNdLrkVTCvpnT8lk=; + b=ZDoXWoKywMS1ruweQAhHCnVicdDclRnRjwOHQCkbwPQq111BrpNO+ZQB26FR99TlPA + fpIVw1EtbCJWpf7KI7BOXzQwSPFkYdz16yikIytV3nnob8czAwJe2M+EY0sWHHptymxE + axklR4v+pTDS3JY0mAbG3VNLwH5MAP95HJwXGL7tVefKk5UA+mkzxtjAjDK5Sz/QrLqv + HxUWl9dMPrLf9cfpZKHL7PewYWC0x83lfxON1dia754f08TofOrYoRx6Z9AnJbICcpaA + G+zlUYIffPHG4rV0gDmrO88A9+dptX2dXS3EBK197QfJbNi64W3akLJpxFaXJxgPFThx + SD0A== +X-Gm-Message-State: AOAM530tAp8KWhFz9P7B73wmg2K3ydszLOM73HOPQPS7zVK/c+GB50yG + Q2xv/bTvb2S7Yn7rmP2fPRYW9vTu67OfzHUD9vUPMHonUnNgJQ== +X-Google-Smtp-Source: ABdhPJw660+u4kMrJZAQEDx6v52aZeTBSdoR/sYxWp3xhjpBvxneu9biH4EKb6VmmChZ1agDSOmDv05d0NbV3j5baQI= +X-Received: by 2002:a5d:4443:: with SMTP id x3mr3220394wrr.49.1614632121231; + Mon, 01 Mar 2021 12:55:21 -0800 (PST) +MIME-Version: 1.0 +References: <CALeFGL1WSSA69ARvJW3di-UC_gz7NV9q7=6zd7s=CHnmttdQFg@mail.gmail.com> + <CAJx8jdz3uOCpwed3MZkf1ghkvaZMfy-+vvOCVZdvhz2KAn38DQ@mail.gmail.com> + <b895f2e4-513f-0c0d-91ac-52af055f332c@LeoWandersleb.de> + <CAPR5oBND9DS1Ea=RMqrhnMBjFqU6wpCpv-sMeX9mxqgyLJGYiQ@mail.gmail.com> + <00b001d70e7e$7204e2f0$560ea8d0$@voskuil.org> +In-Reply-To: <00b001d70e7e$7204e2f0$560ea8d0$@voskuil.org> +From: Keagan McClelland <keagan.mcclelland@gmail.com> +Date: Mon, 1 Mar 2021 13:55:10 -0700 +Message-ID: <CALeFGL2=JLo4SZ4eeDWLbRRMhNV7T58-Pe2jYa6aC6XtPWmC9A@mail.gmail.com> +To: eric@voskuil.org, + Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org> +Content-Type: multipart/alternative; boundary="000000000000076db405bc7fd653" +X-Mailman-Approved-At: Mon, 01 Mar 2021 21:06:49 +0000 +Subject: Re: [bitcoin-dev] A design for Probabilistic Partial Pruning +X-BeenThere: bitcoin-dev@lists.linuxfoundation.org +X-Mailman-Version: 2.1.15 +Precedence: list +List-Id: Bitcoin Protocol Discussion <bitcoin-dev.lists.linuxfoundation.org> +List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>, + <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe> +List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/> +List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org> +List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help> +List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>, + <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe> +X-List-Received-Date: Mon, 01 Mar 2021 20:55:24 -0000 + +--000000000000076db405bc7fd653 +Content-Type: text/plain; charset="UTF-8" +Content-Transfer-Encoding: quoted-printable + +> Personally I consider this counterproductive. Apart from the complexity, +it=E2=80=99s not healthy. And the chain grows linearly with storage cost fa= +lling +exponentially, leading to a straightforward conclusion. + +The motivation for this change is not to encourage full archival nodes to +prune, but to make it possible for pruned nodes to beef up what kind of +archive they retain. Personally I think using the falling storage costs as +a means of providing access to more users is more important than using it +to justify requiring higher node requirements. + +> Something to consider adding to this proposal is to keep the idea of +pruning - i.e. retain a sequentially uninterrupted number of the most +recent blocks. +> +> Many users do not run a node for entirely altruistic reasons - they do +so, at least in part, because it allows them to use their wallets +privately. Without this ability, I think the number of users willing to run +their node in this configuration might be reduced. +> +> Another related thought is to have a decreasing density over blocks over +time as you go backwards towards genesis, in order for the data density of +the storage to match the actual usage of the network, in which (I would +imagine) more recent blocks are more heavily requested than early ones. + +Per my above comments, this change is actually capitalizing primarily upon +those who wish to do it for more altruistic reasons. Furthermore, doing +linear block scans when you need to query blocks that you don't keep does +not leak privacy details in the same way that bloom filters do. You are not +signaling to the peer that there is something specific in that block that +you care about, because you don't actually know. You are signalling only +that you do not have that block right now, which from the other parts of +the design you are already leaking. In light of this, I don't think that it +is necessary for the blocks to be in sequential sets at all. If there is no +requirement on them being sequential, uniform randomness will take care of +the density problem automatically. + +Keagan + + +On Mon, Mar 1, 2021 at 4:20 AM Eric Voskuil via bitcoin-dev < +bitcoin-dev@lists.linuxfoundation.org> wrote: + +> On Sun, Feb 28, 2021 at 10:18 AM Leo Wandersleb via bitcoin-dev < +> bitcoin-dev@lists.linuxfoundation.org> wrote: +> +> +> +> > Only headers need to be downloaded sequentially so downloading relevant +> blocks from one node is totally possible with gaps in between. +> +> +> +> In fact this is exactly how libbitcoin v4 works. We download and store +> blocks in parallel. In the case of a restart block gaps are repopulated. +> Given that headers are validated, we go after the most responsive nodes. +> Based on standard deviation, we drop the slowest peers and rebalance load +> to new/empty channels. We make ordered but not necessarily sequential +> requests. There is no distinction between =E2=80=9Cinitial=E2=80=9D block= + download, a +> restart, or a single or few blocks at the top. So it=E2=80=99s referred t= +o as +> continuous parallel block download. +> +> +> +> But we don=E2=80=99t prune. Personally I consider this counterproductive.= + Apart +> from the complexity, it=E2=80=99s not healthy. And the chain grows linear= +ly with +> storage cost falling exponentially, leading to a straightforward conclusi= +on. +> +> +> +> e +> _______________________________________________ +> bitcoin-dev mailing list +> bitcoin-dev@lists.linuxfoundation.org +> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev +> + +--000000000000076db405bc7fd653 +Content-Type: text/html; charset="UTF-8" +Content-Transfer-Encoding: quoted-printable + +<div dir=3D"ltr">> Personally I consider this counterproductive. Apart f= +rom the complexity, it=E2=80=99s not healthy. And the chain grows linearly = +with storage cost falling exponentially, leading to a straightforward concl= +usion.<div><br></div><div>The motivation for this change is not to encourag= +e full archival nodes to prune, but to make it possible for pruned nodes to= + beef up what kind of archive they retain. Personally I think using the fal= +ling storage costs as a means of providing access to more users is more imp= +ortant than using it to justify requiring higher node requirements.</div><d= +iv><br></div><div>> Something to consider adding to this proposal is to = +keep the idea of pruning - i.e. retain a sequentially uninterrupted number = +of the most recent blocks.<div>></div><div>> Many users do not run a = +node for entirely altruistic reasons - they do so, at least in part, becaus= +e it allows them to use their wallets privately. Without this ability, I th= +ink the number of users willing to run their node in this configuration mig= +ht be reduced.</div><div>></div><div>> Another related thought is to = +have a decreasing density over blocks over time as you go backwards towards= + genesis, in order for the data density of the storage to match the actual = +usage of the network, in which (I would imagine) more recent blocks are mor= +e heavily requested than early ones.</div><div style=3D"color:rgb(136,136,1= +36)"><br></div><div style=3D""><font color=3D"#000000">Per my above comment= +s, this change is actually capitalizing primarily upon those who wish to do= + it for more altruistic reasons. Furthermore, doing linear block scans when= + you need to query blocks that you don't keep does not leak privacy det= +ails in the same way that bloom filters do. You are not signaling to the pe= +er that there is something specific in that block that you care about, beca= +use you don't actually know. You are signalling only that you do not ha= +ve that block right now, which from the other parts of the design you are a= +lready leaking. In light of this, I don't think that it is necessary fo= +r the blocks to be in sequential sets at all. If there is no requirement on= + them being sequential, uniform randomness will take care of the density pr= +oblem automatically.</font></div><div style=3D""><font color=3D"#000000"><b= +r></font></div><div style=3D""><font color=3D"#000000">Keagan</font></div><= +div style=3D"color:rgb(136,136,136)"><br></div></div></div><br><div class= +=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Mon, Mar 1, 2021 = +at 4:20 AM Eric Voskuil via bitcoin-dev <<a href=3D"mailto:bitcoin-dev@l= +ists.linuxfoundation.org">bitcoin-dev@lists.linuxfoundation.org</a>> wro= +te:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px = +0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang=3D= +"EN-US" style=3D"overflow-wrap: break-word;"><div class=3D"gmail-m_17503728= +01160849895WordSection1"><div><div><p class=3D"MsoNormal">On Sun, Feb 28, 2= +021 at 10:18 AM Leo Wandersleb via bitcoin-dev <<a href=3D"mailto:bitcoi= +n-dev@lists.linuxfoundation.org" target=3D"_blank">bitcoin-dev@lists.linuxf= +oundation.org</a>> wrote:<u></u><u></u></p><p class=3D"MsoNormal"><u></u= +>=C2=A0<u></u></p><p class=3D"MsoNormal">> Only headers need to be downl= +oaded sequentially so downloading relevant blocks from one node is totally = +possible with gaps in between.<u></u><u></u></p><p class=3D"MsoNormal"><u><= +/u>=C2=A0<u></u></p><p class=3D"MsoNormal">In fact this is exactly how libb= +itcoin v4 works. We download and store blocks in parallel. In the case of a= + restart block gaps are repopulated. Given that headers are validated, we g= +o after the most responsive nodes. Based on standard deviation, we drop the= + slowest peers and rebalance load to new/empty channels. We make ordered bu= +t not necessarily sequential requests. There is no distinction between =E2= +=80=9Cinitial=E2=80=9D block download, a restart, or a single or few blocks= + at the top. So it=E2=80=99s referred to as continuous parallel block downl= +oad.<u></u><u></u></p><p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p><p cla= +ss=3D"MsoNormal">But we don=E2=80=99t prune. Personally I consider this cou= +nterproductive. Apart from the complexity, it=E2=80=99s not healthy. And th= +e chain grows linearly with storage cost falling exponentially, leading to = +a straightforward conclusion.<u></u><u></u></p><p class=3D"MsoNormal"><u></= +u>=C2=A0<u></u></p><p class=3D"MsoNormal">e<u></u><u></u></p></div></div></= +div></div>_______________________________________________<br> +bitcoin-dev mailing list<br> +<a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org" target=3D"_blank">= +bitcoin-dev@lists.linuxfoundation.org</a><br> +<a href=3D"https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev" = +rel=3D"noreferrer" target=3D"_blank">https://lists.linuxfoundation.org/mail= +man/listinfo/bitcoin-dev</a><br> +</blockquote></div> + +--000000000000076db405bc7fd653-- + |