Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id E19309C for ; Tue, 18 Apr 2017 07:44:00 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from server3 (server3.include7.ch [144.76.194.38]) by smtp1.linuxfoundation.org (Postfix) with ESMTP id 704E01D3 for ; Tue, 18 Apr 2017 07:43:59 +0000 (UTC) Received: by server3 (Postfix, from userid 115) id 4CDC12D006CE; Tue, 18 Apr 2017 09:43:58 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00, FSL_HELO_NON_FQDN_1, HTML_MESSAGE autolearn=ham version=3.3.1 Received: from [192.168.1.198] (public-docking-pat-bs-mapped-0017.ethz.ch [195.176.111.114]) by server3 (Postfix) with ESMTPSA id 8B7822D00117; Tue, 18 Apr 2017 09:43:57 +0200 (CEST) From: Jonas Schnelli Message-Id: Content-Type: multipart/signed; boundary="Apple-Mail=_4C08EC54-E5AA-4866-A99C-65DA54F156B2"; protocol="application/pgp-signature"; micalg=pgp-sha256 Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Date: Tue, 18 Apr 2017 09:43:52 +0200 In-Reply-To: To: David Vorick References: X-Mailer: Apple Mail (2.3273) Cc: Bitcoin Dev Subject: Re: [bitcoin-dev] Small Nodes: A Better Alternative to Pruned Nodes X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Protocol Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Apr 2017 07:44:01 -0000 --Apple-Mail=_4C08EC54-E5AA-4866-A99C-65DA54F156B2 Content-Type: multipart/alternative; boundary="Apple-Mail=_4B1B5E29-A7B1-4A90-A621-FCA62E77278D" --Apple-Mail=_4B1B5E29-A7B1-4A90-A621-FCA62E77278D Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi Dave > A node that stores the full blockchain (I will use the term archival = node) requires over 100GB of disk space, which I believe is one of the = most significant barriers to more people running full nodes. And I = believe the ecosystem would benefit substantially if more users were = running full nodes. Thanks for your proposal. I agree that 100GB of data may be cumbersome for some systems, = especially if you target end user systems (Laptops/Desktops). Though, in = my opinion, for those systems, CPU consumption is the biggest UX = blocker. Bootstrapping a full node on a decent consumer system with default = parameters takes days, and, during this period, you probably run at full = CPU capacity and you will be disturbed by constant fan noise. Standard = tasks may be impossible because your system will be slowed down to a = point where even word processing may get difficult. This is because Core (with its default settings) is made to sync as fast = as possible. Once you have verified the chain and you reach the chain tip, indeed, it = will be much better (until you shutdown for a couple of days/hours and = have to re-sync/catch-up). 1. I agree that we need to have a way for pruned nodes to partially = serve historical blocks. My personal measurements told me that around ~80% of historical block = serving are between tip and -1=E2=80=99000 blocks. Currently, Core nodes have only two modes of operations, =E2=80=9Eserver = all historical blocks=E2=80=9C or =E2=80=9Enone=E2=80=9C. This makes little sense especially if you prune to a target size of, = lets say, 80GB (~80% of the chain). Ideally, there would be a mode where your full node can signal a third = mode =E2=80=9EI keep the last 1000 blocks=E2=80=9C (or make this more = dynamic). 2. Bootstrapping new peers I=E2=80=99m not sure if full nodes must be the single point of = historical data storage. Full nodes provide a valuable service = (verification, relay, filtering, etc.). I=E2=80=99m not sure if serving = historical blocks is one of them. Historical blocks could be made = available on CDN=E2=80=99s or other file storage networks. You are going = to verify them anyways,... the serving part is pure data storage. I=E2=80=99m also pretty sure that some users have stopping running full = nodes because their upstream bandwidth consumption (because of serving = historical blocks) was getting intolerable. Especially =E2=80=9Econsumer=E2=80=9C peers must have been hit by this = (little experience in how to reduce traffic, upstream in general is bad = for consumers-connections, little resources in general). Having a second option built into full nodes (or as an external = bootstrap service/app) how to download historical blocks during = bootstrapping could probably be a relieve for "small nodes=E2=80=9C. It could be a little daemon that downloads historical blocks from = CDN=E2=80=99s, etc. and feeds them into your full node over p2p/8333 and = kickstarts your bootstrapping without bothering valuable peers. Or, the alternative download, could be built into the full nodes main = logic. And, if it wasn=E2=80=99t obvious, this must not bypass the = verification! I=E2=80=99m also aware of the downsides of this. This can eventually = reduce decentralisation of the storage of historical bitcoin blockchain = data and =E2=80=93 eventually =E2=80=93 increase the upstream bandwidth = of peers willing to serve historical blocks (especially in a transition = phase to a second =E2=80=9Edownload=E2=80=9C-option). Maybe it=E2=80=99s a tradeoff between reducing decentralisation by = killing low resource nodes because serving historical blocks is getting = too resource-intense _or_ reducing decentralisation by moving some = percentage of the historical data storage away from the bitcoin p2p = network. The later seems more promising to me. To your proposal: - Isn=E2=80=99t there a tiny finger-printing element if peers have to = pick an segmentation index? - SPV bloom filter clients can=E2=80=99t use fragmented blocks to filter = txns? Right? How could they avoid connecting to those peers? --Apple-Mail=_4B1B5E29-A7B1-4A90-A621-FCA62E77278D Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Hi Dave

A node = that stores the full blockchain (I will use the term archival node) = requires over 100GB of disk space, which I believe is one of the most = significant barriers to more people running full nodes. And I = believe the ecosystem would benefit substantially if more users were = running full nodes.


Thanks for your proposal.

I = agree that 100GB of data may be cumbersome for some systems, especially = if you target end user systems (Laptops/Desktops). Though, in my = opinion, for those systems, CPU consumption is the biggest UX = blocker.
Bootstrapping a full node on a decent consumer = system with default parameters takes days, and, during this period, you = probably run at full CPU capacity and you will be disturbed by = constant fan noise. Standard tasks may be impossible because your = system will be slowed down to a point where even word processing may get = difficult.
This is because Core (with its default = settings) is made to sync as fast as possible.

Once you have verified the chain and you reach the chain tip, = indeed, it will be much better (until you shutdown for a couple of = days/hours and have to re-sync/catch-up).

1. I agree that we need to have a way for pruned nodes to = partially serve historical blocks.
My = personal measurements told me that around ~80% of historical block = serving are between tip and -1=E2=80=99000 blocks.
Currently, Core nodes have only two modes of operations, = =E2=80=9Eserver all historical blocks=E2=80=9C or =E2=80=9Enone=E2=80=9C.<= br class=3D"">This makes little sense especially if you prune to a = target size of, lets say, 80GB (~80% of the chain).
Ideally,= there would be a mode where your full node can signal a third mode =E2=80= =9EI keep the last 1000 blocks=E2=80=9C (or make this more dynamic).

2. Bootstrapping new = peers
I=E2=80=99m not sure if full nodes must be the = single point of historical data storage. Full nodes provide a valuable = service (verification, relay, filtering, etc.). I=E2=80=99m not sure if = serving historical blocks is one of them. Historical blocks could = be made available on CDN=E2=80=99s or other file storage networks. You = are going to verify them anyways,... the serving part is pure data = storage.
I=E2=80=99m also pretty sure that some users have = stopping running full nodes because their upstream bandwidth consumption = (because of serving historical blocks) was getting intolerable.
Especially =E2=80=9Econsumer=E2=80=9C peers must have been = hit by this (little experience in how to reduce traffic, upstream in = general is bad for consumers-connections, little resources in = general).

Having= a second option built into full nodes (or as an external bootstrap = service/app) how to download historical blocks during bootstrapping = could probably be a relieve for "small nodes=E2=80=9C.
It could be a little daemon that downloads historical blocks = from CDN=E2=80=99s, etc. and feeds them into your full node over = p2p/8333 and kickstarts your bootstrapping without bothering valuable = peers.
Or, the alternative download, could be built = into the full nodes main logic.
And, if it wasn=E2=80= =99t obvious, this must not bypass the verification!

I=E2=80=99m also aware = of the downsides of this. This can eventually reduce decentralisation of = the storage of historical bitcoin blockchain data and =E2=80=93 = eventually =E2=80=93 increase the upstream bandwidth of peers willing to = serve historical blocks (especially in a transition phase to a second = =E2=80=9Edownload=E2=80=9C-option).
Maybe it=E2=80=99= s a tradeoff between reducing decentralisation by killing low resource = nodes because serving historical blocks is getting too resource-intense = _or_ reducing decentralisation by moving some percentage of the = historical data storage away from the bitcoin p2p network.
The later seems more promising to me.


To your proposal:
- Isn=E2=80=99t there a tiny finger-printing element if peers = have to pick an segmentation index?
- SPV bloom filter = clients can=E2=80=99t use fragmented blocks to filter txns? Right? How = could they avoid connecting to those peers?

</jonas>
= --Apple-Mail=_4B1B5E29-A7B1-4A90-A621-FCA62E77278D-- --Apple-Mail=_4C08EC54-E5AA-4866-A99C-65DA54F156B2 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEyhopCNzi8TB0xizeHrd2uwPHki0FAlj1w7gACgkQHrd2uwPH ki1E3A//SjwjLa1JcT0iB3sA9JMtBcO5+UxIaMvTdYV5Ccn28y/MHIL3Q+GlFRUH /E/zTOIXt+pqV7/26Z04b4uXrSwJuL7rBVQMQouCdYgDux+tUxPvcvUfYhPJg3/B otGEDXHgBGkYqPJXgbBuwxL1PLJ/56eXX+hD7Wm+BjiyQh6ioXQzozZGPNWxBIl3 IdrsZ3JqTq6dtqgJV4Nht8cJSLrXpF0841pTa/PiUW4HybY7LSZ1EMw52Gy8Ht6X ZAWFe7C2gnjt6S2e7nvcOrG5fqM4wAXrBRx6mtuz14W+W9UpgMYuoIf4liigQwtH b1o3jniJ647p94LftEV8bvhJUIkfFekscmQmuNQGsdAVWWmQq75/8TXew/OYWFiF ef0JkqtQdsxopBIr2nBoHBaXXGXCao89tN7oE5c6WjkSWR79LEocAM7wT4tJWOU0 yrcq7pTfwYdoCLpoNWF8daoFVnK9Tbqd7csE3UIwrHpLRsLqH9FMRPmnqo5itwlu 1Vi6R3y/kAaM/bHCVP5bKov2+kmV33uaCQDhelb97feNLM4L8PCswG8cfgw0yoJv c3JI9fOurQQj2IWY/3J6E2VgwUD7FkMHuiRbXvtTVRCjhoXU+/D0T1sq69CCnFEk W4vrrtUOpZUd5n4nscQLg516yILBCfSPrKz2RhjzoOpKEnqIXXw= =sUS0 -----END PGP SIGNATURE----- --Apple-Mail=_4C08EC54-E5AA-4866-A99C-65DA54F156B2--