path: root/78/e98613d0fb42d13ee00d91c813b7891d602c15

Received: from sog-mx-3.v43.ch3.sourceforge.com ([172.29.43.193]
	helo=mx.sourceforge.net)
	by sfs-ml-2.v29.ch3.sourceforge.com with esmtp (Exim 4.76)
	(envelope-from <21xe14@gmail.com>) id 1XdoL9-0004Ts-3J
	for bitcoin-development@lists.sourceforge.net;
	Mon, 13 Oct 2014 22:43:43 +0000
Received-SPF: pass (sog-mx-3.v43.ch3.sourceforge.com: domain of gmail.com
	designates 74.125.82.68 as permitted sender)
	client-ip=74.125.82.68; envelope-from=21xe14@gmail.com;
	helo=mail-wg0-f68.google.com; 
Received: from mail-wg0-f68.google.com ([74.125.82.68])
	by sog-mx-3.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128)
	(Exim 4.76) id 1XdoL7-0001w3-8e
	for bitcoin-development@lists.sourceforge.net;
	Mon, 13 Oct 2014 22:43:43 +0000
Received: by mail-wg0-f68.google.com with SMTP id a1so2850470wgh.3
	for <bitcoin-development@lists.sourceforge.net>;
	Mon, 13 Oct 2014 15:43:35 -0700 (PDT)
MIME-Version: 1.0
X-Received: by 10.194.236.200 with SMTP id uw8mr1304928wjc.50.1413240215069;
	Mon, 13 Oct 2014 15:43:35 -0700 (PDT)
Received: by 10.194.79.67 with HTTP; Mon, 13 Oct 2014 15:43:35 -0700 (PDT)
In-Reply-To: <543AD2C2.3010309@gmail.com>
References: <CAPg+sBjG86ZYKZZXA4VCMiFhz0sdxr_so41Zr+DATOXyRa9_Jw@mail.gmail.com>
	<543AD2C2.3010309@gmail.com>
Date: Mon, 13 Oct 2014 22:43:35 +0000
Message-ID: <CAFZQHkGDjxFW3yCZe4PauicgV-ic7X3_GcKn+uQCkNQgAsw8Xw@mail.gmail.com>
From: 21E14 <21xe14@gmail.com>
To: Jameson Lopp <jameson.lopp@gmail.com>
Content-Type: multipart/alternative; boundary=089e01493f60006808050555a11d
X-Spam-Score: 1.3 (+)
X-Spam-Report: Spam Filtering performed by mx.sourceforge.net.
	See http://spamassassin.org/tag/ for more details.
	1.7 URIBL_WS_SURBL Contains an URL listed in the WS SURBL blocklist
	[URIs: sipa.be]
	-1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for
	sender-domain
	0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
	(21xe14[at]gmail.com)
	-0.0 SPF_PASS               SPF: sender matches SPF record
	0.2 FREEMAIL_ENVFROM_END_DIGIT Envelope-from freemail username ends in
	digit (21xe14[at]gmail.com)
	1.0 HTML_MESSAGE           BODY: HTML included in message
	-0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from
	author's domain
	0.1 DKIM_SIGNED            Message has a DKIM or DK signature,
	not necessarily valid
	-0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
X-Headers-End: 1XdoL7-0001w3-8e
Cc: bitcoin-development@lists.sourceforge.net
Subject: Re: [Bitcoin-development] Request for review/testing: headers-first
 synchronization in Bitcoin Core
X-BeenThere: bitcoin-development@lists.sourceforge.net
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: <bitcoin-development.lists.sourceforge.net>
List-Unsubscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
	<mailto:bitcoin-development-request@lists.sourceforge.net?subject=unsubscribe>
List-Archive: <http://sourceforge.net/mailarchive/forum.php?forum_name=bitcoin-development>
List-Post: <mailto:bitcoin-development@lists.sourceforge.net>
List-Help: <mailto:bitcoin-development-request@lists.sourceforge.net?subject=help>
List-Subscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
	<mailto:bitcoin-development-request@lists.sourceforge.net?subject=subscribe>
X-List-Received-Date: Mon, 13 Oct 2014 22:43:43 -0000

--089e01493f60006808050555a11d
Content-Type: text/plain; charset=UTF-8

When forgoing bootstrapping due to disk space constraints, you, and the
network, are likely better off -reindex-ing from current blk000??.dat files.

Which brings up an interesting point: The improvements related to the
headers first approach are likely to increase, how ever marginally, the
percentage of block exchange-related traffic, as it is less painful now to
be catching up.

It'd be interesting to see the statistics, not from a single node
perspective, but from the viewpoint of an Internet backbone provider, say
through the cables coming ashore in Cornwall. For the incurred bandwidth
expense would invariably trickle down to transaction fees in an equilibrium
model.

There is an opportunity somewhere in this.


On Sun, Oct 12, 2014 at 7:13 PM, Jameson Lopp <jameson.lopp@gmail.com>
wrote:

> Great work, Pieter. I've been spooling up several nodes per week lately
> and can testify that stalled downloads during initial syncing are a pain. I
> usually forgo bootstrapping on VPSes because I don't want to have to adjust
> the disk space allocation.
>
> With headers-first I'm saturating my home cable connection with download
> rates of 4 MB/s until block 295,000 at which point CPU becomes the
> bottleneck and it settles down in the 1 MB/s range.
>
> It took 6 minutes for my node to sync to block height 100,000
> 22 minutes to reach height 200,000
> 62 minutes to reach height 250,000
> 125 minutes to reach height 295,000
> 144 minutes to reach height 300,000
> 248 minutes to reach height 325,000
>
> - Jameson
>
> On 10/11/2014 07:34 PM, Pieter Wuille wrote:
> > Hi all,
> >
> > I believe that a large change that I've been working on for Bitcoin
> > Core is ready for review and testing: headers-first synchronization.
> > In short, it changes the way the best chain is discovered, downloaded
> > and verified, with several advantages:
> > * Parallel block downloading (much faster sync on typical network
> connections).
> > * No more stalled downloads.
> > * Much more robust against unresponsive or slow peers.
> > * Removes a class of DoS attacks related to peers feeding you
> > low-difficulty valid large blocks on a side branch.
> > * Reduces the need for checkpoints in the code.
> > * No orphan blocks stored in memory anymore (reducing memory usage
> during sync).
> > * A major step step towards an SPV mode using the reference codebase.
> >
> > Historically, this mode of operation has been known for years (Greg
> > Maxwell wrote up a description of a very similar method in
> > https://en.bitcoin.it/wiki/User:Gmaxwell/Reverse_header-fetching_sync
> > in early 2012, but it was known before that), but it took a long time
> > to refactor these code enough to support it.
> >
> > Technically, it works by replacing the single-peer blocks download by
> > a single-peer headers download (which typically takes seconds/minutes)
> > and verification, and simultaneously fetching blocks along the best
> > known headers chain from all peers that are known to have the relevant
> > blocks. Downloading is constrained to a moving window to avoid
> > unbounded unordering of blocks on disk (which would interfere with
> > pruning later).
> >
> > At the protocol level, it increases the minimally supported version
> > for peers to 31800 (corresponding to bitcoin v3.18, released in
> > december 2010), as earlier versions did not support the getheaders P2P
> > message.
> >
> > So, the code is available as a github pull request
> > (https://github.com/bitcoin/bitcoin/pull/4468), or packaged on
> > http://bitcoin.sipa.be/builds/headersfirst, where you can also find
> > binaries to test with.
> >
> > Known issues:
> > * At the very start of the sync, especially before all headers are
> > processed, downloading is very slow due to a limited number of blocks
> > that are requested per peer simultaneously. The policies around this
> > will need some experimentation can certainly be improved.
> > * Blocks will be stored on disk out of order (in the order they are
> > received, really), which makes it incompatible with some tools or
> > other programs. Reindexing using earlier versions will also not work
> > anymore as a result of this.
> > * The block index database will now hold headers for which no block is
> > stored on disk, which earlier versions won't support. If you are fully
> > synced, it may still be possible to go back to an earlier version.
> >
> > Unknown issues:
> > * Who knows, maybe it will replace your familiy pictures with Nyan
> > Cat? Use at your own risk.
> >
> > TL;DR: Review/test https://github.com/bitcoin/bitcoin/pull/4468 or
> > http://bitcoin.sipa.be/builds/headersfirst.
> >
>
>
> ------------------------------------------------------------------------------
> Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
> Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
> Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
> Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
> http://p.sf.net/sfu/Zoho
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>

--089e01493f60006808050555a11d
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">When forgoing bootstrapping due to disk space constraints,=
 you, and the network, are likely better off -reindex-ing from current blk0=
00??.dat files.<br><br>Which brings up an interesting point: The improvemen=
ts related to the headers first approach are likely to increase, how ever m=
arginally, the percentage of block exchange-related traffic, as it is less =
painful now to be catching up.<br><br>It&#39;d be interesting to see the st=
atistics, not from a single node perspective, but from the viewpoint of an =
Internet backbone provider, say through the cables coming ashore in Cornwal=
l. For the incurred bandwidth expense would invariably trickle down to tran=
saction fees in an equilibrium model.<br><br>There is an opportunity somewh=
ere in this.<br><br><div class=3D"gmail_extra"><br><div class=3D"gmail_quot=
e">On Sun, Oct 12, 2014 at 7:13 PM, Jameson Lopp <span dir=3D"ltr">&lt;<a h=
ref=3D"mailto:jameson.lopp@gmail.com" target=3D"_blank">jameson.lopp@gmail.=
com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"mar=
gin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1=
ex">Great work, Pieter. I&#39;ve been spooling up several nodes per week la=
tely and can testify that stalled downloads during initial syncing are a pa=
in. I usually forgo bootstrapping on VPSes because I don&#39;t want to have=
 to adjust the disk space allocation.<br>
<br>
With headers-first I&#39;m saturating my home cable connection with downloa=
d rates of 4 MB/s until block 295,000 at which point CPU becomes the bottle=
neck and it settles down in the 1 MB/s range.<br>
<br>
It took 6 minutes for my node to sync to block height 100,000<br>
22 minutes to reach height 200,000<br>
62 minutes to reach height 250,000<br>
125 minutes to reach height 295,000<br>
144 minutes to reach height 300,000<br>
248 minutes to reach height 325,000<br>
<span class=3D""><font color=3D"#888888"><br>
- Jameson<br>
</font></span><div class=3D""><div class=3D"h5"><br>
On 10/11/2014 07:34 PM, Pieter Wuille wrote:<br>
&gt; Hi all,<br>
&gt;<br>
&gt; I believe that a large change that I&#39;ve been working on for Bitcoi=
n<br>
&gt; Core is ready for review and testing: headers-first synchronization.<b=
r>
&gt; In short, it changes the way the best chain is discovered, downloaded<=
br>
&gt; and verified, with several advantages:<br>
&gt; * Parallel block downloading (much faster sync on typical network conn=
ections).<br>
&gt; * No more stalled downloads.<br>
&gt; * Much more robust against unresponsive or slow peers.<br>
&gt; * Removes a class of DoS attacks related to peers feeding you<br>
&gt; low-difficulty valid large blocks on a side branch.<br>
&gt; * Reduces the need for checkpoints in the code.<br>
&gt; * No orphan blocks stored in memory anymore (reducing memory usage dur=
ing sync).<br>
&gt; * A major step step towards an SPV mode using the reference codebase.<=
br>
&gt;<br>
&gt; Historically, this mode of operation has been known for years (Greg<br=
>
&gt; Maxwell wrote up a description of a very similar method in<br>
&gt; <a href=3D"https://en.bitcoin.it/wiki/User:Gmaxwell/Reverse_header-fet=
ching_sync" target=3D"_blank">https://en.bitcoin.it/wiki/User:Gmaxwell/Reve=
rse_header-fetching_sync</a><br>
&gt; in early 2012, but it was known before that), but it took a long time<=
br>
&gt; to refactor these code enough to support it.<br>
&gt;<br>
&gt; Technically, it works by replacing the single-peer blocks download by<=
br>
&gt; a single-peer headers download (which typically takes seconds/minutes)=
<br>
&gt; and verification, and simultaneously fetching blocks along the best<br=
>
&gt; known headers chain from all peers that are known to have the relevant=
<br>
&gt; blocks. Downloading is constrained to a moving window to avoid<br>
&gt; unbounded unordering of blocks on disk (which would interfere with<br>
&gt; pruning later).<br>
&gt;<br>
&gt; At the protocol level, it increases the minimally supported version<br=
>
&gt; for peers to 31800 (corresponding to bitcoin v3.18, released in<br>
&gt; december 2010), as earlier versions did not support the getheaders P2P=
<br>
&gt; message.<br>
&gt;<br>
&gt; So, the code is available as a github pull request<br>
&gt; (<a href=3D"https://github.com/bitcoin/bitcoin/pull/4468" target=3D"_b=
lank">https://github.com/bitcoin/bitcoin/pull/4468</a>), or packaged on<br>
&gt; <a href=3D"http://bitcoin.sipa.be/builds/headersfirst" target=3D"_blan=
k">http://bitcoin.sipa.be/builds/headersfirst</a>, where you can also find<=
br>
&gt; binaries to test with.<br>
&gt;<br>
&gt; Known issues:<br>
&gt; * At the very start of the sync, especially before all headers are<br>
&gt; processed, downloading is very slow due to a limited number of blocks<=
br>
&gt; that are requested per peer simultaneously. The policies around this<b=
r>
&gt; will need some experimentation can certainly be improved.<br>
&gt; * Blocks will be stored on disk out of order (in the order they are<br=
>
&gt; received, really), which makes it incompatible with some tools or<br>
&gt; other programs. Reindexing using earlier versions will also not work<b=
r>
&gt; anymore as a result of this.<br>
&gt; * The block index database will now hold headers for which no block is=
<br>
&gt; stored on disk, which earlier versions won&#39;t support. If you are f=
ully<br>
&gt; synced, it may still be possible to go back to an earlier version.<br>
&gt;<br>
&gt; Unknown issues:<br>
&gt; * Who knows, maybe it will replace your familiy pictures with Nyan<br>
&gt; Cat? Use at your own risk.<br>
&gt;<br>
&gt; TL;DR: Review/test <a href=3D"https://github.com/bitcoin/bitcoin/pull/=
4468" target=3D"_blank">https://github.com/bitcoin/bitcoin/pull/4468</a> or=
<br>
&gt; <a href=3D"http://bitcoin.sipa.be/builds/headersfirst" target=3D"_blan=
k">http://bitcoin.sipa.be/builds/headersfirst</a>.<br>
&gt;<br>
<br>
</div></div><div class=3D""><div class=3D"h5">-----------------------------=
-------------------------------------------------<br>
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer<br>
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports<br=
>
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper<br>
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer<br>
<a href=3D"http://p.sf.net/sfu/Zoho" target=3D"_blank">http://p.sf.net/sfu/=
Zoho</a><br>
_______________________________________________<br>
Bitcoin-development mailing list<br>
<a href=3D"mailto:Bitcoin-development@lists.sourceforge.net">Bitcoin-develo=
pment@lists.sourceforge.net</a><br>
<a href=3D"https://lists.sourceforge.net/lists/listinfo/bitcoin-development=
" target=3D"_blank">https://lists.sourceforge.net/lists/listinfo/bitcoin-de=
velopment</a><br>
</div></div></blockquote></div><br></div></div>

--089e01493f60006808050555a11d--