path: root/2d/6ae2c9b35f4a9878e3b43b0a8e3ff856cffb61

Return-Path: <laolu32@gmail.com>
Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org
	[172.17.192.35])
	by mail.linuxfoundation.org (Postfix) with ESMTPS id BABFA892
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Thu,  6 Aug 2015 17:34:01 +0000 (UTC)
X-Greylist: whitelisted by SQLgrey-1.7.6
Received: from mail-qg0-f49.google.com (mail-qg0-f49.google.com
	[209.85.192.49])
	by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 8B5D716A
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Thu,  6 Aug 2015 17:34:00 +0000 (UTC)
Received: by qgj62 with SMTP id 62so32706951qgj.2
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Thu, 06 Aug 2015 10:33:59 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
	h=mime-version:references:in-reply-to:from:date:message-id:subject:to
	:content-type; bh=2rMdE9kR/9IV3I0hwoPYDNF012YiIi826qyAyagvQ/Y=;
	b=OXiql6hiOFZNuCVp8e7jHiAZ6bRsb9IyH9ZhXALoq1ZsXgOVJCKnxyo+sDTF2n9JhP
	QfYF3X52rv6p73rgfgnhJCUeuJw+xC2Z3d/Hy2RE/BBYQ9gK5qvG4qtKPpeonErYsigW
	BO8k6lyPtscjUMRUOS3yDqIkgyz94b88FxGmXSw1bDVNTEg4gyZatMf8zL6taRLAEeBU
	YJA1ud8EyxLsIb1gaw3Sy5Dv4a1sGyPn7/rq9y4DHRzo37mMN34+AsPl5tlbHRGhFvx+
	fQEFMjW1GvTeUrZFdEnhQhlJwdiljLliEfPiZtQ9u1dZMvGJIa3Gm9Dp89oSzwnLZAtU
	Kecw==
X-Received: by 10.140.237.67 with SMTP id i64mr5430332qhc.5.1438882439524;
	Thu, 06 Aug 2015 10:33:59 -0700 (PDT)
MIME-Version: 1.0
References: <CALwsPgnnkbfUBhL=Qyspz13pnZ-6RHdaZOGvfLG34JjJRgt2Dw@mail.gmail.com>
	<B3546CB9-6A24-474C-8B56-9B1E2D33B470@mattcorallo.com>
	<CALwsPgm6xcBfLXZTNTZ40R_s3oUawE0ANZycDWpSo0cXZ+=-Vg@mail.gmail.com>
	<CAAS2fgQLNH+rivNNTFz6xt_9SxO3fFj7-3z7A-_B_2X2x-6M5w@mail.gmail.com>
	<CALwsPgm9S3UNd3bEuWreyGS7bcvSD+cXxueoD+F_D9fC=xLz2Q@mail.gmail.com>
	<CAAS2fgSXWzCv=4cF=0bwL9+udzBHSPR7goL3U_c1NjS22dpWzQ@mail.gmail.com>
	<CAKzdR-qPtPsxAXsmUX=vTkq-ro=EAmH7M8nL_Px_b4D4Z0WAXg@mail.gmail.com>
In-Reply-To: <CAKzdR-qPtPsxAXsmUX=vTkq-ro=EAmH7M8nL_Px_b4D4Z0WAXg@mail.gmail.com>
From: Olaoluwa Osuntokun <laolu32@gmail.com>
Date: Thu, 06 Aug 2015 17:33:49 +0000
Message-ID: <CAO3Pvs-knJ5DyJou2YuB1nAQH+UXV4mcwn42KKh0Wb-YpeOm_w@mail.gmail.com>
To: Sergio Demian Lerner <sergio.d.lerner@gmail.com>, 
	Arnoud Kouwenhoven - Pukaki Corp via bitcoin-dev
	<bitcoin-dev@lists.linuxfoundation.org>
Content-Type: multipart/alternative; boundary=001a1135c5feae7896051ca7ecfb
X-Spam-Status: No, score=-2.4 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,
	HTML_MESSAGE,RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.1
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
	smtp1.linux-foundation.org
Subject: Re: [bitcoin-dev] Idea: Efficient bitcoin block propagation
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Bitcoin Development Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Thu, 06 Aug 2015 17:34:01 -0000

--001a1135c5feae7896051ca7ecfb
Content-Type: text/plain; charset=UTF-8

Other than the source code, the best documentation I've come across is a few
lines on IRC explaining the high-level design of the protocol:
https://botbot.me/freenode/bitcoin-wizards/2015-07-10/?msg=44146764&page=2

On Thu, Aug 6, 2015 at 10:18 AM Sergio Demian Lerner via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Is there any up to date documentation about TheBlueMatt relay network
> including what kind of block compression it is currently doing? (apart from
> the source code)
>
> Regards, Sergio.
>
> On Wed, Aug 5, 2015 at 7:14 PM, Gregory Maxwell via bitcoin-dev <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
>
>> On Wed, Aug 5, 2015 at 9:19 PM, Arnoud Kouwenhoven - Pukaki Corp
>> <arnoud@pukaki.bz> wrote:
>> > Thanks for this (direct) feedback. It would make sense that if blocks
>> can be
>> > submitted using ~5kb packets, that no further optimizations would be
>> needed
>> > at this point. I will look into the relay network transmission protocol
>> to
>> > understand how it works!
>> >
>> > I hear that you are saying that this network solves speed of
>> transmission
>> > and thereby (technical) block size issues. Presumably it would solve
>> speed
>> > of block validation too by prevalidating transactions.
>>
>>
>> Correct. Bitcoin Core has cached validation for many years now... if
>> not for that and other optimizations, things would be really broken
>> right now. :)
>>
>> > Assuming this is all
>> > true, and I have no reason to doubt that at this point, I do not
>> understand
>> > why there is any discussion at all about the (technical) impact of large
>> > blocks, why there are large numbers of miners building on invalid blocks
>> > (SPV mining, https://bitcoin.org/en/alert/2015-07-04-spv-mining), or
>> why
>> > there is any discussion about the speed of block validation (cpu
>> processing
>> > time to verify blocks and transactions in blocks being a limitation).
>>
>> I'm also mystified by a lot of the large block discussion, much of it
>> is completely divorced from the technology as deployed; much less what
>> we-- in industry-- know to be possible. I don't blame you or anyone in
>> particular on this; it's a new area and we don't yet know what we need
>> to know to know what we need to know; or to the extent that we do it
>> hasn't had time to get effectively communicated.
>>
>> The technical/security implications of larger blocks are related to
>> other things than propagation time, if you assume people are using the
>> available efficient relay protocol (or better).
>>
>> SPV mining is a bit of a misnomer (If I coined the term, I'm sorry).
>> What these parties are actually doing is blinding mining on top of
>> other pools' stratum work. You can think of it as sub-pooling with
>> hopping onto whatever pool has the highest block (I'll call it VFSSP
>> in this post-- validation free stratum subpooling).  It's very easy to
>> implement, and there are other considerations.
>>
>> It was initially deployed at a time when a single pool in Europe has
>> amassed more than half of the hashrate. This pool had propagation
>> problems and a very high orphan rate, it may have (perhaps
>> unintentionally) been performing a selfish mining attack; mining off
>> their stratum work was an easy fix which massively cut down the orphan
>> rates for anyone who did it.  This was before the relay network
>> protocol existed (the fact that all the hashpower was consolidating on
>> a single pool was a major motivation for creating it).
>>
>> VFSSP also cuts through a number of practical issues miners have had:
>> Miners that run their own bitcoin nodes in far away colocation
>> (>100ms) due to local bandwidth or connectivity issues (censored
>> internet); relay network hubs not being anywhere near by due to
>> strange internet routing (e.g. japan to china going via the US for ...
>> reasons...); the CreateNewBlock() function being very slow and
>> unoptimized, etc.   There are many other things like this-- and VFSSP
>> avoids them causing delays even when you don't understand them or know
>> about them. So even when they're easily fixed the VFSSP is a more
>> general workaround.
>>
>> Mining operations are also usually operated in a largely fire and
>> forget manner. There is a long history in (esp pooled) mining where
>> someone sets up an operation and then hardly maintains it after the
>> fact... so some of the use of VFSSP appears to just be inertia-- we
>> have better solutions now, but they they work to deploy and changing
>> things involves risk (which is heightened by a lack of good
>> monitoring-- participants learn they are too latent by observing
>> orphaned blocks at a cost of 25 BTC each).
>>
>> One of the frustrating things about incentives in this space is that
>> bad outcomes are possible even when they're not necessary. E.g. if a
>> miner can lower their orphan rate by deploying a new protocol (or
>> simply fixing some faulty hardware in their infrastructure, like
>> Bitcoin nodes running on cheap VPSes with remote storage)  OR they can
>> lower their orphan rate by pointing their hashpower at a free
>> centeralized pool, they're likely to do the latter because it takes
>> less effort.
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

--001a1135c5feae7896051ca7ecfb
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Other than the source code, the best documentation I&#39;v=
e come across is a few<div>lines on=C2=A0<span style=3D"line-height:1.5;fon=
t-size:13.1999998092651px">IRC explaining the high-level design of the prot=
ocol:=C2=A0</span><div><a href=3D"https://botbot.me/freenode/bitcoin-wizard=
s/2015-07-10/?msg=3D44146764&amp;page=3D2">https://botbot.me/freenode/bitco=
in-wizards/2015-07-10/?msg=3D44146764&amp;page=3D2</a></div></div><br><div =
class=3D"gmail_quote"><div dir=3D"ltr">On Thu, Aug 6, 2015 at 10:18 AM Serg=
io Demian Lerner via bitcoin-dev &lt;<a href=3D"mailto:bitcoin-dev@lists.li=
nuxfoundation.org">bitcoin-dev@lists.linuxfoundation.org</a>&gt; wrote:<br>=
</div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-l=
eft:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr">Is there any up to da=
te documentation about TheBlueMatt relay network including what kind of blo=
ck compression it is currently doing? (apart from the source code)<div><br>=
<div>Regards, Sergio.</div></div></div><div class=3D"gmail_extra"><br><div =
class=3D"gmail_quote">On Wed, Aug 5, 2015 at 7:14 PM, Gregory Maxwell via b=
itcoin-dev <span dir=3D"ltr">&lt;<a href=3D"mailto:bitcoin-dev@lists.linuxf=
oundation.org" target=3D"_blank">bitcoin-dev@lists.linuxfoundation.org</a>&=
gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 =
0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>On Wed, Aug 5, 20=
15 at 9:19 PM, Arnoud Kouwenhoven - Pukaki Corp<br>
&lt;<a href=3D"mailto:arnoud@pukaki.bz" target=3D"_blank">arnoud@pukaki.bz<=
/a>&gt; wrote:<br>
&gt; Thanks for this (direct) feedback. It would make sense that if blocks =
can be<br>
&gt; submitted using ~5kb packets, that no further optimizations would be n=
eeded<br>
&gt; at this point. I will look into the relay network transmission protoco=
l to<br>
&gt; understand how it works!<br>
&gt;<br>
&gt; I hear that you are saying that this network solves speed of transmiss=
ion<br>
&gt; and thereby (technical) block size issues. Presumably it would solve s=
peed<br>
&gt; of block validation too by prevalidating transactions.<br>
<br>
<br>
</span>Correct. Bitcoin Core has cached validation for many years now... if=
<br>
not for that and other optimizations, things would be really broken<br>
right now. :)<br>
<span><br>
&gt; Assuming this is all<br>
&gt; true, and I have no reason to doubt that at this point, I do not under=
stand<br>
&gt; why there is any discussion at all about the (technical) impact of lar=
ge<br>
</span>&gt; blocks, why there are large numbers of miners building on inval=
id blocks<br>
<span>&gt; (SPV mining, <a href=3D"https://bitcoin.org/en/alert/2015-07-04-=
spv-mining" rel=3D"noreferrer" target=3D"_blank">https://bitcoin.org/en/ale=
rt/2015-07-04-spv-mining</a>), or why<br>
&gt; there is any discussion about the speed of block validation (cpu proce=
ssing<br>
&gt; time to verify blocks and transactions in blocks being a limitation).<=
br>
<br>
</span>I&#39;m also mystified by a lot of the large block discussion, much =
of it<br>
is completely divorced from the technology as deployed; much less what<br>
we-- in industry-- know to be possible. I don&#39;t blame you or anyone in<=
br>
particular on this; it&#39;s a new area and we don&#39;t yet know what we n=
eed<br>
to know to know what we need to know; or to the extent that we do it<br>
hasn&#39;t had time to get effectively communicated.<br>
<br>
The technical/security implications of larger blocks are related to<br>
other things than propagation time, if you assume people are using the<br>
available efficient relay protocol (or better).<br>
<br>
SPV mining is a bit of a misnomer (If I coined the term, I&#39;m sorry).<br=
>
What these parties are actually doing is blinding mining on top of<br>
other pools&#39; stratum work. You can think of it as sub-pooling with<br>
hopping onto whatever pool has the highest block (I&#39;ll call it VFSSP<br=
>
in this post-- validation free stratum subpooling).=C2=A0 It&#39;s very eas=
y to<br>
implement, and there are other considerations.<br>
<br>
It was initially deployed at a time when a single pool in Europe has<br>
amassed more than half of the hashrate. This pool had propagation<br>
problems and a very high orphan rate, it may have (perhaps<br>
unintentionally) been performing a selfish mining attack; mining off<br>
their stratum work was an easy fix which massively cut down the orphan<br>
rates for anyone who did it.=C2=A0 This was before the relay network<br>
protocol existed (the fact that all the hashpower was consolidating on<br>
a single pool was a major motivation for creating it).<br>
<br>
VFSSP also cuts through a number of practical issues miners have had:<br>
Miners that run their own bitcoin nodes in far away colocation<br>
(&gt;100ms) due to local bandwidth or connectivity issues (censored<br>
internet); relay network hubs not being anywhere near by due to<br>
strange internet routing (e.g. japan to china going via the US for ...<br>
reasons...); the CreateNewBlock() function being very slow and<br>
unoptimized, etc.=C2=A0 =C2=A0There are many other things like this-- and V=
FSSP<br>
avoids them causing delays even when you don&#39;t understand them or know<=
br>
about them. So even when they&#39;re easily fixed the VFSSP is a more<br>
general workaround.<br>
<br>
Mining operations are also usually operated in a largely fire and<br>
forget manner. There is a long history in (esp pooled) mining where<br>
someone sets up an operation and then hardly maintains it after the<br>
fact... so some of the use of VFSSP appears to just be inertia-- we<br>
have better solutions now, but they they work to deploy and changing<br>
things involves risk (which is heightened by a lack of good<br>
monitoring-- participants learn they are too latent by observing<br>
orphaned blocks at a cost of 25 BTC each).<br>
<br>
One of the frustrating things about incentives in this space is that<br>
bad outcomes are possible even when they&#39;re not necessary. E.g. if a<br=
>
miner can lower their orphan rate by deploying a new protocol (or<br>
simply fixing some faulty hardware in their infrastructure, like<br>
Bitcoin nodes running on cheap VPSes with remote storage)=C2=A0 OR they can=
<br>
lower their orphan rate by pointing their hashpower at a free<br>
centeralized pool, they&#39;re likely to do the latter because it takes<br>
less effort.<br>
<div><div>_______________________________________________<br>
bitcoin-dev mailing list<br>
<a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org" target=3D"_blank">=
bitcoin-dev@lists.linuxfoundation.org</a><br>
<a href=3D"https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev" =
rel=3D"noreferrer" target=3D"_blank">https://lists.linuxfoundation.org/mail=
man/listinfo/bitcoin-dev</a><br>
</div></div></blockquote></div><br></div>
_______________________________________________<br>
bitcoin-dev mailing list<br>
<a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org" target=3D"_blank">=
bitcoin-dev@lists.linuxfoundation.org</a><br>
<a href=3D"https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev" =
rel=3D"noreferrer" target=3D"_blank">https://lists.linuxfoundation.org/mail=
man/listinfo/bitcoin-dev</a><br>
</blockquote></div></div>

--001a1135c5feae7896051ca7ecfb--