path: root/a6/72e672b5832745dadb719fda44fc574d437777

Received: from sog-mx-1.v43.ch3.sourceforge.com ([172.29.43.191]
	helo=mx.sourceforge.net)
	by sfs-ml-1.v29.ch3.sourceforge.com with esmtp (Exim 4.76)
	(envelope-from <rebroad@gmail.com>) id 1XedGc-0003Lz-8C
	for bitcoin-development@lists.sourceforge.net;
	Thu, 16 Oct 2014 05:06:26 +0000
Received-SPF: pass (sog-mx-1.v43.ch3.sourceforge.com: domain of gmail.com
	designates 209.85.218.46 as permitted sender)
	client-ip=209.85.218.46; envelope-from=rebroad@gmail.com;
	helo=mail-oi0-f46.google.com; 
Received: from mail-oi0-f46.google.com ([209.85.218.46])
	by sog-mx-1.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128)
	(Exim 4.76) id 1XedGa-000602-It
	for bitcoin-development@lists.sourceforge.net;
	Thu, 16 Oct 2014 05:06:26 +0000
Received: by mail-oi0-f46.google.com with SMTP id h136so2106903oig.19
	for <bitcoin-development@lists.sourceforge.net>;
	Wed, 15 Oct 2014 22:06:19 -0700 (PDT)
X-Received: by 10.202.77.75 with SMTP id a72mr14330801oib.8.1413435979059;
	Wed, 15 Oct 2014 22:06:19 -0700 (PDT)
MIME-Version: 1.0
Sender: rebroad@gmail.com
Received: by 10.76.2.113 with HTTP; Wed, 15 Oct 2014 22:05:58 -0700 (PDT)
In-Reply-To: <CAPg+sBjG86ZYKZZXA4VCMiFhz0sdxr_so41Zr+DATOXyRa9_Jw@mail.gmail.com>
References: <CAPg+sBjG86ZYKZZXA4VCMiFhz0sdxr_so41Zr+DATOXyRa9_Jw@mail.gmail.com>
From: "Rebroad (sourceforge)" <rebroad+sourceforge.net@gmail.com>
Date: Thu, 16 Oct 2014 13:05:58 +0800
X-Google-Sender-Auth: I9ho5cUpYUEe0yeGZMVt1ASAWNI
Message-ID: <CAFBxzABOWgy=5NszM+yTFOPVzLegd5h7R+s_YbKG_teKryhEgA@mail.gmail.com>
Cc: Bitcoin Dev <bitcoin-development@lists.sourceforge.net>
Content-Type: multipart/alternative; boundary=001a1135308271d70505058335a3
X-Spam-Score: 3.1 (+++)
X-Spam-Report: Spam Filtering performed by mx.sourceforge.net.
	See http://spamassassin.org/tag/ for more details.
	-1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for
	sender-domain
	0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
	(rebroad[at]gmail.com)
	-0.0 SPF_PASS               SPF: sender matches SPF record
	1.2 MISSING_HEADERS        Missing To: header
	1.0 HTML_MESSAGE           BODY: HTML included in message
	-0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from
	author's domain
	0.1 DKIM_SIGNED            Message has a DKIM or DK signature,
	not necessarily valid
	-0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
	2.5 MALFORMED_FREEMAIL Bad headers on message from free email service
X-Headers-End: 1XedGa-000602-It
Subject: Re: [Bitcoin-development] Request for review/testing: headers-first
 synchronization in Bitcoin Core
X-BeenThere: bitcoin-development@lists.sourceforge.net
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: <bitcoin-development.lists.sourceforge.net>
List-Unsubscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
	<mailto:bitcoin-development-request@lists.sourceforge.net?subject=unsubscribe>
List-Archive: <http://sourceforge.net/mailarchive/forum.php?forum_name=bitcoin-development>
List-Post: <mailto:bitcoin-development@lists.sourceforge.net>
List-Help: <mailto:bitcoin-development-request@lists.sourceforge.net?subject=help>
List-Subscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
	<mailto:bitcoin-development-request@lists.sourceforge.net?subject=subscribe>
X-List-Received-Date: Thu, 16 Oct 2014 05:06:26 -0000

--001a1135308271d70505058335a3
Content-Type: text/plain; charset=UTF-8

Hi all,

I've also been spending a few months coding upon the change's Pieter has
been making with the headersfirst8 pull request.

My code updates are also ready to test, and are available on github at
https://github.com/rebroad/bitcoin/  and the branch is
"sipa-headersfirst8-patches".

I've made a number of improvement. Namely that it tracks the block as it
downloads and won't disconnect if the block is downloading at a reasonable
speed. The current stall logic of Pieter's is broken in that it will
continue to disconnect a peer that is providing a block - particularly the
next block needed to advance the current tip. I've raised this issue, but
so far haven't been able to communicate the problem in a way that's been
understood.

I've also added logic to avoid the node stalling due to many blocks being
added to the ActiveTip (which would cause timeouts both from our node, and
nodes we are connected to). It will also log and determine bandwidth per
node, and the next changes I will be adding will be to make it prefer to
download from the faster nodes (coming shortly).

I have also added code ready to adapt the window size for the download.
Currently the start setting for blocks in flight is 3 per node, but for
early on on the block chain this will be too small, so once it realises
this after a few downloads and determines the average block size and speed,
it will automatically adjust the number of blocks to request per node and
revise this each minute.

Please do take a look at my code, and feel free to test it. It also
improves some of the debug.log output to make it easier to read and provide
useful information about concurrent downloads, etc.

Edmund

On Sun, Oct 12, 2014 at 7:34 AM, Pieter Wuille <pieter.wuille@gmail.com>
wrote:

> Hi all,
>
> I believe that a large change that I've been working on for Bitcoin
> Core is ready for review and testing: headers-first synchronization.
> In short, it changes the way the best chain is discovered, downloaded
> and verified, with several advantages:
> * Parallel block downloading (much faster sync on typical network
> connections).
> * No more stalled downloads.
> * Much more robust against unresponsive or slow peers.
> * Removes a class of DoS attacks related to peers feeding you
> low-difficulty valid large blocks on a side branch.
> * Reduces the need for checkpoints in the code.
> * No orphan blocks stored in memory anymore (reducing memory usage during
> sync).
> * A major step step towards an SPV mode using the reference codebase.
>
> Historically, this mode of operation has been known for years (Greg
> Maxwell wrote up a description of a very similar method in
> https://en.bitcoin.it/wiki/User:Gmaxwell/Reverse_header-fetching_sync
> in early 2012, but it was known before that), but it took a long time
> to refactor these code enough to support it.
>
> Technically, it works by replacing the single-peer blocks download by
> a single-peer headers download (which typically takes seconds/minutes)
> and verification, and simultaneously fetching blocks along the best
> known headers chain from all peers that are known to have the relevant
> blocks. Downloading is constrained to a moving window to avoid
> unbounded unordering of blocks on disk (which would interfere with
> pruning later).
>
> At the protocol level, it increases the minimally supported version
> for peers to 31800 (corresponding to bitcoin v3.18, released in
> december 2010), as earlier versions did not support the getheaders P2P
> message.
>
> So, the code is available as a github pull request
> (https://github.com/bitcoin/bitcoin/pull/4468), or packaged on
> http://bitcoin.sipa.be/builds/headersfirst, where you can also find
> binaries to test with.
>
> Known issues:
> * At the very start of the sync, especially before all headers are
> processed, downloading is very slow due to a limited number of blocks
> that are requested per peer simultaneously. The policies around this
> will need some experimentation can certainly be improved.
> * Blocks will be stored on disk out of order (in the order they are
> received, really), which makes it incompatible with some tools or
> other programs. Reindexing using earlier versions will also not work
> anymore as a result of this.
> * The block index database will now hold headers for which no block is
> stored on disk, which earlier versions won't support. If you are fully
> synced, it may still be possible to go back to an earlier version.
>
> Unknown issues:
> * Who knows, maybe it will replace your familiy pictures with Nyan
> Cat? Use at your own risk.
>
> TL;DR: Review/test https://github.com/bitcoin/bitcoin/pull/4468 or
> http://bitcoin.sipa.be/builds/headersfirst.
>
> --
> Pieter
>
>
> ------------------------------------------------------------------------------
> Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
> Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
> Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
> Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
> http://p.sf.net/sfu/Zoho
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>

--001a1135308271d70505058335a3
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi all,<div><br></div><div>I&#39;ve also been spending a f=
ew months coding upon the change&#39;s Pieter has been making with the head=
ersfirst8 pull request.</div><div><br></div><div>My code updates are also r=
eady to test, and are available on github at=C2=A0<a href=3D"https://github=
.com/rebroad/bitcoin/">https://github.com/rebroad/bitcoin/</a> =C2=A0and th=
e branch is &quot;sipa-headersfirst8-patches&quot;.</div><div><br></div><di=
v>I&#39;ve made a number of improvement. Namely that it tracks the block as=
 it downloads and won&#39;t disconnect if the block is downloading at a rea=
sonable speed. The current stall logic of Pieter&#39;s is broken in that it=
 will continue to disconnect a peer that is providing a block - particularl=
y the next block needed to advance the current tip. I&#39;ve raised this is=
sue, but so far haven&#39;t been able to communicate the problem in a way t=
hat&#39;s been understood.</div><div><br></div><div>I&#39;ve also added log=
ic to avoid the node stalling due to many blocks being added to the ActiveT=
ip (which would cause timeouts both from our node, and nodes we are connect=
ed to). It will also log and determine bandwidth per node, and the next cha=
nges I will be adding will be to make it prefer to download from the faster=
 nodes (coming shortly).</div><div><br></div><div>I have also added code re=
ady to adapt the window size for the download. Currently the start setting =
for blocks in flight is 3 per node, but for early on on the block chain thi=
s will be too small, so once it realises this after a few downloads and det=
ermines the average block size and speed, it will automatically adjust the =
number of blocks to request per node and revise this each minute.</div><div=
><br></div><div>Please do take a look at my code, and feel free to test it.=
 It also improves some of the debug.log output to make it easier to read an=
d provide useful information about concurrent downloads, etc.</div><div><br=
></div><div>Edmund</div></div><div class=3D"gmail_extra"><br><div class=3D"=
gmail_quote">On Sun, Oct 12, 2014 at 7:34 AM, Pieter Wuille <span dir=3D"lt=
r">&lt;<a href=3D"mailto:pieter.wuille@gmail.com" target=3D"_blank">pieter.=
wuille@gmail.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote"=
 style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi=
 all,<br>
<br>
I believe that a large change that I&#39;ve been working on for Bitcoin<br>
Core is ready for review and testing: headers-first synchronization.<br>
In short, it changes the way the best chain is discovered, downloaded<br>
and verified, with several advantages:<br>
* Parallel block downloading (much faster sync on typical network connectio=
ns).<br>
* No more stalled downloads.<br>
* Much more robust against unresponsive or slow peers.<br>
* Removes a class of DoS attacks related to peers feeding you<br>
low-difficulty valid large blocks on a side branch.<br>
* Reduces the need for checkpoints in the code.<br>
* No orphan blocks stored in memory anymore (reducing memory usage during s=
ync).<br>
* A major step step towards an SPV mode using the reference codebase.<br>
<br>
Historically, this mode of operation has been known for years (Greg<br>
Maxwell wrote up a description of a very similar method in<br>
<a href=3D"https://en.bitcoin.it/wiki/User:Gmaxwell/Reverse_header-fetching=
_sync" target=3D"_blank">https://en.bitcoin.it/wiki/User:Gmaxwell/Reverse_h=
eader-fetching_sync</a><br>
in early 2012, but it was known before that), but it took a long time<br>
to refactor these code enough to support it.<br>
<br>
Technically, it works by replacing the single-peer blocks download by<br>
a single-peer headers download (which typically takes seconds/minutes)<br>
and verification, and simultaneously fetching blocks along the best<br>
known headers chain from all peers that are known to have the relevant<br>
blocks. Downloading is constrained to a moving window to avoid<br>
unbounded unordering of blocks on disk (which would interfere with<br>
pruning later).<br>
<br>
At the protocol level, it increases the minimally supported version<br>
for peers to 31800 (corresponding to bitcoin v3.18, released in<br>
december 2010), as earlier versions did not support the getheaders P2P<br>
message.<br>
<br>
So, the code is available as a github pull request<br>
(<a href=3D"https://github.com/bitcoin/bitcoin/pull/4468" target=3D"_blank"=
>https://github.com/bitcoin/bitcoin/pull/4468</a>), or packaged on<br>
<a href=3D"http://bitcoin.sipa.be/builds/headersfirst" target=3D"_blank">ht=
tp://bitcoin.sipa.be/builds/headersfirst</a>, where you can also find<br>
binaries to test with.<br>
<br>
Known issues:<br>
* At the very start of the sync, especially before all headers are<br>
processed, downloading is very slow due to a limited number of blocks<br>
that are requested per peer simultaneously. The policies around this<br>
will need some experimentation can certainly be improved.<br>
* Blocks will be stored on disk out of order (in the order they are<br>
received, really), which makes it incompatible with some tools or<br>
other programs. Reindexing using earlier versions will also not work<br>
anymore as a result of this.<br>
* The block index database will now hold headers for which no block is<br>
stored on disk, which earlier versions won&#39;t support. If you are fully<=
br>
synced, it may still be possible to go back to an earlier version.<br>
<br>
Unknown issues:<br>
* Who knows, maybe it will replace your familiy pictures with Nyan<br>
Cat? Use at your own risk.<br>
<br>
TL;DR: Review/test <a href=3D"https://github.com/bitcoin/bitcoin/pull/4468"=
 target=3D"_blank">https://github.com/bitcoin/bitcoin/pull/4468</a> or<br>
<a href=3D"http://bitcoin.sipa.be/builds/headersfirst" target=3D"_blank">ht=
tp://bitcoin.sipa.be/builds/headersfirst</a>.<br>
<br>
--<br>
Pieter<br>
<br>
---------------------------------------------------------------------------=
---<br>
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer<br>
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports<br=
>
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper<br>
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer<br>
<a href=3D"http://p.sf.net/sfu/Zoho" target=3D"_blank">http://p.sf.net/sfu/=
Zoho</a><br>
_______________________________________________<br>
Bitcoin-development mailing list<br>
<a href=3D"mailto:Bitcoin-development@lists.sourceforge.net">Bitcoin-develo=
pment@lists.sourceforge.net</a><br>
<a href=3D"https://lists.sourceforge.net/lists/listinfo/bitcoin-development=
" target=3D"_blank">https://lists.sourceforge.net/lists/listinfo/bitcoin-de=
velopment</a><br>
</blockquote></div><br></div>

--001a1135308271d70505058335a3--