MIME-Version: 1.0
In-Reply-To: <CAAS2fgROykenGH43_FXun+q4u+=94tKqnRW=kQQ2AQFeXKdW=Q@mail.gmail.com>
References: <CADZtCShywq_s9ZK3oBBTUdCgjrTxbyeb-p7QrhJJ3mwRAECCMA@mail.gmail.com>
	<CAAS2fgROykenGH43_FXun+q4u+=94tKqnRW=kQQ2AQFeXKdW=Q@mail.gmail.com>
From: Jim Posen <jim.posen@gmail.com>
Date: Mon, 11 Dec 2017 13:56:08 -0800
Message-ID: <CADZtCSi1vimZ88kqG=HQHa3sHeLtvMjdJrZuqS2=iYr-=EHaLw@mail.gmail.com>
To: Gregory Maxwell <greg@xiph.org>
Content-Type: multipart/alternative; boundary="94eb2c08b2380ecd1e0560179ae5"
Cc: Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org>
Subject: Re: [bitcoin-dev] "Compressed" headers stream
Precedence: list

--94eb2c08b2380ecd1e0560179ae5
Content-Type: text/plain; charset="UTF-8"

On Mon, Dec 11, 2017 at 1:04 PM, Gregory Maxwell <greg@xiph.org> wrote:

> On Mon, Dec 11, 2017 at 8:40 PM, Jim Posen via bitcoin-dev
> <bitcoin-dev@lists.linuxfoundation.org> wrote:
> > Firstly, I don't like the idea of making the net header encoding
> dependent
> > on the specific header validation rules that Bitcoin uses (eg. the fact
> that
> > difficulty is only recalculated every 2016 blocks). This would be
> coupling
>
> In the last proposal I recall writing up, there was a one byte flag on
> each header to indicate what was included.
>
>
Is there a link somewhere to that proposal? The only thing I could find was
your forwarded email
<https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-September/014904.html>
on
this thread.


> Nbits _never_ needs to be sent even with other consensus rules because
> its more or less necessarily a strict function of the prior headers.
> This still holds in every clone of Bitcoin I'm aware of; sending it
> with the first header in a group probably makes sense so it can be
> checked independently.
>
> > with insufficient benefit.
>
> another >18% reduction in size beyond the removal of prev. is not
> insubstantial by any means.  I don't think it should lightly be
> ignored.
>
>
Omitting nBits entirely seems reasonable, I wrote up a possible
implementation here
<https://github.com/bitcoin/bitcoin/compare/master...jimpo:compact-headers-difficulty>.
The downside is that it is more complex because it leaks into the
validation code. The extra 4 byte savings is certainly nice though.


> Prev omission itself is not, sadly, magically compatible:  I am quite
> confident that if there is a bitcoin hardfork it would recover the
> nbits/4-guarenteed always-zero bits of prev to use as extra nonce for
> miners. This has been proposed many times, implemented at least once,
> and the current requirement for mining infrastructure to reach inside
> the coinbase txn to increment a nonce has been a reliable source of
> failures.  So I think we'd want to have the encoding able to encode
> leading prev bits.
>
> Many altcoins also change the header structures. If the better thing
> is altcoin incompatible, we should still do it. Doing otherwise would
> competitively hobble Bitcoin especially considering the frequent
> recklessly incompetent moves made by various altcoins and the near
> total lack of useful novel development we've seen come out of the
> clones.
>
> Probably the most important change in a new header message wouldn't be
> the encoding, but it would be changing the fetching mechanism so that
> header sets could be pulled in parallel, etc.
>
> I would rather not change the serialization of existing messages,
> nodes are going to have to support speaking both messages for a long
> time, and I think we already want a different protocol flow for
> headers fetching in any case.
>

Can you elaborate on how parallel header fetching might work? getheaders
requests could probably already be pipelined, where the node requests the
next 2,000 headers before processing the current batch (though would make
sense to check that they are all above min difficulty first).

I'm open to more ideas on how to optimize the header download or design the
serialization format to be more flexible, but I'm concerned that we forgo a
40-45% bandwidth savings on the current protocol for a long time because
something better might be possible later on or there might be a hard fork
that at some point requires another upgrade. I do recognize that supporting
multiple serialization formats simultaneously adds code complexity, but in
this case the change seems simple enough to me that the tradeoff is worth
it.

--94eb2c08b2380ecd1e0560179ae5
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On M=
on, Dec 11, 2017 at 1:04 PM, Gregory Maxwell <span dir=3D"ltr">&lt;<a href=
=3D"mailto:greg@xiph.org" target=3D"_blank">greg@xiph.org</a>&gt;</span> wr=
ote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border=
-left:1px #ccc solid;padding-left:1ex"><span class=3D"">On Mon, Dec 11, 201=
7 at 8:40 PM, Jim Posen via bitcoin-dev<br>
&lt;<a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@li=
sts.<wbr>linuxfoundation.org</a>&gt; wrote:<br>
&gt; Firstly, I don&#39;t like the idea of making the net header encoding d=
ependent<br>
&gt; on the specific header validation rules that Bitcoin uses (eg. the fac=
t that<br>
&gt; difficulty is only recalculated every 2016 blocks). This would be coup=
ling<br>
<br>
</span>In the last proposal I recall writing up, there was a one byte flag =
on<br>
each header to indicate what was included.<br>
<br></blockquote><div><br></div><div>Is there a link somewhere to that prop=
osal? The only thing I could find was your <a href=3D"https://lists.linuxfo=
undation.org/pipermail/bitcoin-dev/2017-September/014904.html">forwarded em=
ail</a>=C2=A0on this thread.</div><div>=C2=A0</div><blockquote class=3D"gma=
il_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-lef=
t:1ex">
Nbits _never_ needs to be sent even with other consensus rules because<br>
its more or less necessarily a strict function of the prior headers.<br>
This still holds in every clone of Bitcoin I&#39;m aware of; sending it<br>
with the first header in a group probably makes sense so it can be<br>
checked independently.<br>
<br>
&gt; with insufficient benefit.<br>
<br>
another &gt;18% reduction in size beyond the removal of prev. is not<br>
insubstantial by any means.=C2=A0 I don&#39;t think it should lightly be<br=
>
ignored.<br>
<br></blockquote><div><br></div><div>Omitting nBits entirely seems reasonab=
le, I wrote up a possible implementation <a href=3D"https://github.com/bitc=
oin/bitcoin/compare/master...jimpo:compact-headers-difficulty">here</a>. Th=
e downside is that it is more complex because it leaks into the validation =
code. The extra 4 byte savings is certainly nice though.</div><div>=C2=A0</=
div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-lef=
t:1px #ccc solid;padding-left:1ex">
Prev omission itself is not, sadly, magically compatible:=C2=A0 I am quite<=
br>
confident that if there is a bitcoin hardfork it would recover the<br>
nbits/4-guarenteed always-zero bits of prev to use as extra nonce for<br>
miners. This has been proposed many times, implemented at least once,<br>
and the current requirement for mining infrastructure to reach inside<br>
the coinbase txn to increment a nonce has been a reliable source of<br>
failures.=C2=A0 So I think we&#39;d want to have the encoding able to encod=
e<br>
leading prev bits.<br>
<br>
Many altcoins also change the header structures. If the better thing<br>
is altcoin incompatible, we should still do it. Doing otherwise would<br>
competitively hobble Bitcoin especially considering the frequent<br>
recklessly incompetent moves made by various altcoins and the near<br>
total lack of useful novel development we&#39;ve seen come out of the<br>
clones.<br>
<br>
Probably the most important change in a new header message wouldn&#39;t be<=
br>
the encoding, but it would be changing the fetching mechanism so that<br>
header sets could be pulled in parallel, etc.<br>
<br>
I would rather not change the serialization of existing messages,<br>
nodes are going to have to support speaking both messages for a long<br>
time, and I think we already want a different protocol flow for<br>
headers fetching in any case.<br>
</blockquote></div><br></div><div class=3D"gmail_extra">Can you elaborate o=
n how parallel header fetching might work? getheaders requests could probab=
ly already be pipelined, where the node requests the next 2,000 headers bef=
ore processing the current batch (though would make sense to check that the=
y are all above min difficulty first).</div><div class=3D"gmail_extra"><br>=
</div><div class=3D"gmail_extra">I&#39;m open to more ideas on how to optim=
ize the header download or design the serialization format to be more flexi=
ble, but I&#39;m concerned that we forgo a 40-45% bandwidth savings on the =
current protocol for a long time because something better might be possible=
 later on or there might be a hard fork that at some point requires another=
 upgrade. I do recognize that supporting multiple serialization formats sim=
ultaneously adds code complexity, but in this case the change seems simple =
enough to me that the tradeoff is worth it.=C2=A0</div></div>

--94eb2c08b2380ecd1e0560179ae5--