1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
|
Return-Path: <riccardo.casatta@gmail.com>
Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org
[172.17.192.35])
by mail.linuxfoundation.org (Postfix) with ESMTPS id 19AB196F
for <bitcoin-dev@lists.linuxfoundation.org>;
Mon, 28 Aug 2017 15:50:46 +0000 (UTC)
X-Greylist: whitelisted by SQLgrey-1.7.6
Received: from mail-it0-f66.google.com (mail-it0-f66.google.com
[209.85.214.66])
by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 6EED23F5
for <bitcoin-dev@lists.linuxfoundation.org>;
Mon, 28 Aug 2017 15:50:45 +0000 (UTC)
Received: by mail-it0-f66.google.com with SMTP id w204so573443ita.1
for <bitcoin-dev@lists.linuxfoundation.org>;
Mon, 28 Aug 2017 08:50:45 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
h=mime-version:from:date:message-id:subject:to;
bh=0IbS5gRc7wQ+0T5bl8y9wXQZyUtU5w3SHbj5ixowDT0=;
b=jgKa9p7rPOYwuEntxILMTfLJybhWyrRJpyafL2PeZMcVyftvqP2e4z/+ptsZAQJM+9
PYNeFhh6TWD5M5TTDxnt9AgBRh4aIWBxX/eZlv334evq/+qWPBBujNZyqNEBqOGfMX9l
jWWV4giQTF2ZvLfUeHl8SZxv1kXyNmMk6KYT+/Cpm0IR3c5l0SdooUCbBoHcAAqsLMIv
N2iWAkO14QEaK+8fZxRWUzAIbMNha8dx+26bqLArJaJwHoKh9fTgadaX4sp04+VQxuJf
XGPO9wvgi+PLfQIyOsFcPTmiTUVlSnKomz17JxzSB3Xqwm7gpb8GjToLyzXqDrVMMEeZ
bWnA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20161025;
h=x-gm-message-state:mime-version:from:date:message-id:subject:to;
bh=0IbS5gRc7wQ+0T5bl8y9wXQZyUtU5w3SHbj5ixowDT0=;
b=Wv7JomM9CHdjB1t1CfKnHQjxLBMitbHh+4lVoeSgi7CrMC7LWxHlEAoFEMcClF2hzD
VGykZSVx0MLqEMB7G02mog9K+wD3Wfq+kaMfpYKN3G8ZNhhlY+8MebhSgJILgpSSPGrf
0lvtqQ65JSVGYMBxk147R+qBuzWYKOKR8XAjLj9q79SFe21lZdIn+udP7Z0Yx+uU+dUV
r6tpiIdNhRZaArOuScno8aMUvJxL85xEKBYaW5sHEzsZEZS4Q6/GS/rEYlSwuO3qrNI3
IY3yjUHDxCgK4M3aLOn66scf0qbM+r/RYOHuH00AvLmBkPFXolp//w+sdSBg7py2kVcu
bvow==
X-Gm-Message-State: AHYfb5hAf01X9e6VxcMOfm2ZyG6Y2R2rOCAQQwEW0O4jlXOVn2MbNm8S
/Bja2wie/m8dIJzHU42TmYvdrZOOhLZOTMU=
X-Received: by 10.36.52.74 with SMTP id z71mr1052913itz.142.1503935444275;
Mon, 28 Aug 2017 08:50:44 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.2.120.29 with HTTP; Mon, 28 Aug 2017 08:50:23 -0700 (PDT)
From: Riccardo Casatta <riccardo.casatta@gmail.com>
Date: Mon, 28 Aug 2017 17:50:23 +0200
Message-ID: <CADabwBBrrPM2f9h_sgxY12tg=FUvKKCcnCC8ixnct93YL9uEFQ@mail.gmail.com>
To: bitcoin-dev@lists.linuxfoundation.org
Content-Type: multipart/alternative; boundary="001a11473df6ebd7600557d241a0"
X-Spam-Status: No, score=0.4 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
DKIM_VALID_AU,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,
RCVD_IN_SORBS_SPAM autolearn=disabled version=3.3.1
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
smtp1.linux-foundation.org
X-Mailman-Approved-At: Mon, 28 Aug 2017 16:05:38 +0000
Subject: [bitcoin-dev] "Compressed" headers stream
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Bitcoin Protocol Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Mon, 28 Aug 2017 15:50:46 -0000
--001a11473df6ebd7600557d241a0
Content-Type: text/plain; charset="UTF-8"
Hi everyone,
the Bitcoin headers are probably the most condensed and important piece of
data in the world, their demand is expected to grow.
When sending a stream of continuous block headers, a common case in IBD and
in disconnected clients, I think there is a possible optimization of the
transmitted data:
The headers after the first could avoid transmitting the previous hash
cause the receiver could compute it by double hashing the previous header
(an operation he needs to do anyway to verify PoW).
In a long stream, for example 2016 headers, the savings in bandwidth are
about 32/80 ~= 40%
without compressed headers 2016*80=161280 bytes
with compressed headers 80+2015*48=96800 bytes
What do you think?
In OpenTimestamps calendars we are going to use this compression to give
lite-client a reasonable secure proofs (a full node give higher security
but isn't feasible in all situations, for example for in-browser
verification)
To speed up sync of a new client Electrum starts with the download of a file
<https://headers.electrum.org/blockchain_headers> ~36MB containing the
first 477637 headers.
For this kind of clients could be useful a common http API with fixed
position chunks to leverage http caching. For example /headers/2016/0
returns the headers from the genesis to the 2015 header included while
/headers/2016/1 gives the headers from the 2016th to the 4031.
Other endpoints could have chunks of 20160 blocks or 201600 such that with
about 10 http requests a client could fast sync the headers
--
Riccardo Casatta - @RCasatta <https://twitter.com/RCasatta>
--001a11473df6ebd7600557d241a0
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div>Hi everyone,</div><div><br></div><div>the Bitcoin hea=
ders are probably the most condensed and important piece of data in the wor=
ld, their demand is expected to grow.</div><div><br></div><div>When sending=
a stream of continuous block headers, a common case in IBD and in disconne=
cted clients, I think there is a possible optimization of the transmitted d=
ata:</div><div>The headers after the first could avoid transmitting the pre=
vious hash cause the receiver could compute it by double hashing the previo=
us header (an operation he needs to do anyway to verify PoW).</div><div>In =
a long stream, for example 2016 headers, the savings in bandwidth are about=
32/80 ~=3D 40%</div><div>without compressed headers 2016*80=3D161280 bytes=
</div><div>with compressed headers 80+2015*48=3D96800 bytes</div><div><br><=
/div><div>What do you think?</div><div><br></div><div><br></div><div>In Ope=
nTimestamps calendars we are going to use this compression to give lite-cli=
ent a reasonable secure proofs (a full node give higher security but isn=
9;t feasible in all situations, for example for in-browser verification)</d=
iv><div>To speed up sync of a new client Electrum starts with the download =
of a <a href=3D"https://headers.electrum.org/blockchain_headers">file</a>=
=C2=A0~36MB containing the first 477637 headers.=C2=A0</div><div>For this k=
ind of clients could be useful a common http API with fixed position chunks=
to leverage http caching. For example /headers/2016/0 returns the headers =
from the genesis to the 2015 header included while /headers/2016/1 gives th=
e headers from the 2016th to the 4031.</div><div>Other endpoints could have=
chunks of 20160 blocks or 201600 such that with about 10 http requests a c=
lient could fast sync the headers</div><br clear=3D"all"><div><br></div>-- =
<br><div class=3D"gmail_signature"><div dir=3D"ltr">Riccardo Casatta - <a h=
ref=3D"https://twitter.com/RCasatta" target=3D"_blank">@RCasatta</a></div><=
/div>
</div>
--001a11473df6ebd7600557d241a0--
|
