1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
|
Return-Path: <peter.tschipper@gmail.com>
Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org
[172.17.192.35])
by mail.linuxfoundation.org (Postfix) with ESMTPS id 1D9D6952
for <bitcoin-dev@lists.linuxfoundation.org>;
Tue, 10 Nov 2015 16:46:57 +0000 (UTC)
X-Greylist: whitelisted by SQLgrey-1.7.6
Received: from mail-pa0-f48.google.com (mail-pa0-f48.google.com
[209.85.220.48])
by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 41CAC1A7
for <bitcoin-dev@lists.linuxfoundation.org>;
Tue, 10 Nov 2015 16:46:56 +0000 (UTC)
Received: by pasz6 with SMTP id z6so1875636pas.2
for <bitcoin-dev@lists.linuxfoundation.org>;
Tue, 10 Nov 2015 08:46:56 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
h=subject:to:references:from:message-id:date:user-agent:mime-version
:in-reply-to:content-type;
bh=5SFTzjBHhhfqBSufXk6NZjOuWq9G+uAyF3X/SOGRsXA=;
b=X6K9Izp++LvMsW7ftcj7TMuJ2hJS4aXO190eMsyPO8uIFug9pBICwwc9yLuAlctnan
6E+X55nOgLLXXxd7kIHk3qghN0XtQd+ZpUtWkRULzigIhdpLMKxU1eGvIoRNnVmoV8wA
UVnfboH3N0VQA953mOMaJ9Q/1oLMs4dSGsa3Wut10glYYnbhvtP0lOuOiVhUrcZ/ZTdr
WKE09mPeWLN0MWGeOkxal/aEOTMZwnWw+VMz+YjjwUhUb9PVdL+ZeAbue8YNnX2JZdei
yJ39ihYvMl1UMg9hs0NpqOwuSa1bHflteBQem8avxIULAtg/OSXDzOPUrMPAvon6ub3f
KewA==
X-Received: by 10.68.163.97 with SMTP id yh1mr7093186pbb.36.1447174015911;
Tue, 10 Nov 2015 08:46:55 -0800 (PST)
Received: from [192.168.0.132] (S0106bcd165303d84.cc.shawcable.net.
[96.54.102.88]) by smtp.googlemail.com with ESMTPSA id
fp2sm5024717pbb.34.2015.11.10.08.46.54
for <bitcoin-dev@lists.linuxfoundation.org>
(version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
Tue, 10 Nov 2015 08:46:55 -0800 (PST)
To: Bitcoin Dev <bitcoin-dev@lists.linuxfoundation.org>
References: <5640F172.3010004@gmail.com> <20151109210449.GE5886@mcelrath.org>
<CAL7-sS0Apm4O_Qi0FmY7=H580rEVD6DYjk2y+ACpZmKqUJTQwA@mail.gmail.com>
<CALOxbZtTUrZwDfy_jTbs60n=K8RKDGg5X0gkLsh-OX3ikLf1FQ@mail.gmail.com>
<CAE-z3OUB-se_HUvW2NLjWt=0d5sgMiPEciu0hLzr_HQN0m9fqQ@mail.gmail.com>
<5642172C.701@gmail.com>
<CAE-z3OXgWCHL_3CDR-ACc7ojbLi7EavyObNa3s7hPUMGj_V2+A@mail.gmail.com>
<56421F1E.4050302@gmail.com>
From: Peter Tschipper <peter.tschipper@gmail.com>
Message-ID: <56421F7E.8070305@gmail.com>
Date: Tue, 10 Nov 2015 08:46:54 -0800
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101
Thunderbird/38.3.0
MIME-Version: 1.0
In-Reply-To: <56421F1E.4050302@gmail.com>
Content-Type: multipart/alternative;
boundary="------------060604090906010105020205"
X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_SIGNED,
DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW
autolearn=ham version=3.3.1
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
smtp1.linux-foundation.org
Subject: Re: [bitcoin-dev] request BIP number for: "Support for Datastream
Compression"
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Bitcoin Development Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Tue, 10 Nov 2015 16:46:57 -0000
This is a multi-part message in MIME format.
--------------060604090906010105020205
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
On 10/11/2015 8:45 AM, Peter Tschipper wrote:
> On 10/11/2015 8:30 AM, Tier Nolan via bitcoin-dev wrote:
>>
>>
>> On Tue, Nov 10, 2015 at 4:11 PM, Peter Tschipper
>> <peter.tschipper@gmail.com> wrote:
>>
>> There are better ways of sending new blocks, that's certainly
>> true but for sending historical blocks and seding transactions I
>> don't think so. This PR is really designed to save bandwidth and
>> not intended to be a huge performance improvement in terms of
>> time spent sending.
>>
>>
>> If the main point is for historical data, then sticking to just
>> blocks is the best plan.
>>
> at the beginning yes.
>> Since small blocks don't compress well, you could define a "cblocks"
>> message that handles multiple blocks (just concatenate the block
>> messages as payload before compression).
>>
> Small block are rare these days (but plenty of historical block), but
> still they get a 10% compression, not bad and I think worthwhile and
> the time it takes to compress small blocks is less that a millisecond
> so no loss there in time. But still you have a good point and
> something worthy of doing after getting compression to work. I think
> it's wise to keep it simple at first and build on the success later.
>> The sending peer could combine blocks so that each cblock is
>> compressing at least 10kB of block data (or whatever is optimal). It
>> is probably worth specifying a maximum size for network buffer
>> reasons (either 1MB or 1 block maximum).
> Good idea. Same answer as above.
>> Similarly, transactions could be combined together and compressed
>> "ctxs". The inv messages could be modified so that you can request
>> groups of 10-20 transactions. That would depend on how much of an
>> improvement compressed transactions would represent.
>>
> Good idea. Same answer as above.
>> More generally, you could define a message which is a compressed
>> message holder. That is probably to complex to be worth the effort
>> though.
> That's actually pretty easy to do and part of the plan. Sending a
> cmp_block rather than a block makes it all easier to implement. It's
> just a matter of doing pnode->pushmessage("cmp_block",
> compressed_block); and handling the "cmp_block" command string at the
> other end.
>>
>>
>>
>>>
>>> On Tue, Nov 10, 2015 at 5:40 AM, Johnathan Corgan via
>>> bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org
>>> <mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote:
>>>
>>> On Mon, Nov 9, 2015 at 5:58 PM, gladoscc via bitcoin-dev
>>> <bitcoin-dev@lists.linuxfoundation.org> wrote:
>>>
>>>
>>> I think 25% bandwidth savings is certainly considerable,
>>> especially for people running full nodes in countries
>>> like Australia where internet bandwidth is lower and
>>> there are data caps.
>>>
>>>
>>> This reinforces the idea that such trade-off decisions
>>> should be be local and negotiated between peers, not a
>>> required feature of the network P2P.
>>>
>>>
>>> --
>>> Johnathan Corgan
>>> Corgan Labs - SDR Training and Development Services
>>> http://corganlabs.com
>>>
>>> _______________________________________________
>>> bitcoin-dev mailing list
>>> bitcoin-dev@lists.linuxfoundation.org
>>> <mailto:bitcoin-dev@lists.linuxfoundation.org>
>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> bitcoin-dev mailing list
>>> bitcoin-dev@lists.linuxfoundation.org
>>> <mailto:bitcoin-dev@lists.linuxfoundation.org>
>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>>
>>
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
--------------060604090906010105020205
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 10/11/2015 8:45 AM, Peter Tschipper
wrote:<br>
</div>
<blockquote cite="mid:56421F1E.4050302@gmail.com" type="cite">
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
<div class="moz-cite-prefix">On 10/11/2015 8:30 AM, Tier Nolan via
bitcoin-dev wrote:<br>
</div>
<blockquote
cite="mid:CAE-z3OXgWCHL_3CDR-ACc7ojbLi7EavyObNa3s7hPUMGj_V2+A@mail.gmail.com"
type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Nov 10, 2015 at 4:11 PM,
Peter Tschipper <span dir="ltr"><<a
moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:peter.tschipper@gmail.com"><a class="moz-txt-link-abbreviated" href="mailto:peter.tschipper@gmail.com">peter.tschipper@gmail.com</a></a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex"> <span
class=""></span><span class=""></span>There are better
ways of sending new blocks, that's certainly true but
for sending historical blocks and seding transactions I
don't think so. This PR is really designed to save
bandwidth and not intended to be a huge performance
improvement in terms of time spent sending.<span
class=""><br>
</span></blockquote>
<div><br>
</div>
<div>If the main point is for historical data, then
sticking to just blocks is the best plan.<br>
<br>
</div>
</div>
</div>
</div>
</blockquote>
at the beginning yes.<br>
<blockquote
cite="mid:CAE-z3OXgWCHL_3CDR-ACc7ojbLi7EavyObNa3s7hPUMGj_V2+A@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>Since small blocks don't compress well, you could
define a "cblocks" message that handles multiple blocks
(just concatenate the block messages as payload before
compression). <br>
<br>
</div>
</div>
</div>
</div>
</blockquote>
Small block are rare these days (but plenty of historical block),
but still they get a 10% compression, not bad and I think
worthwhile and the time it takes to compress small blocks is less
that a millisecond so no loss there in time. But still you have
a good point and something worthy of doing after getting
compression to work. I think it's wise to keep it simple at first
and build on the success later.<br>
<blockquote
cite="mid:CAE-z3OXgWCHL_3CDR-ACc7ojbLi7EavyObNa3s7hPUMGj_V2+A@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>The sending peer could combine blocks so that each
cblock is compressing at least 10kB of block data (or
whatever is optimal). It is probably worth specifying a
maximum size for network buffer reasons (either 1MB or 1
block maximum).<br>
</div>
</div>
</div>
</div>
</blockquote>
Good idea. Same answer as above.<br>
<blockquote
cite="mid:CAE-z3OXgWCHL_3CDR-ACc7ojbLi7EavyObNa3s7hPUMGj_V2+A@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>Similarly, transactions could be combined together
and compressed "ctxs". The inv messages could be
modified so that you can request groups of 10-20
transactions. That would depend on how much of an
improvement compressed transactions would represent. <br>
<br>
</div>
</div>
</div>
</div>
</blockquote>
Good idea. Same answer as above.<br>
<blockquote
cite="mid:CAE-z3OXgWCHL_3CDR-ACc7ojbLi7EavyObNa3s7hPUMGj_V2+A@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>More generally, you could define a message which is a
compressed message holder. That is probably to complex
to be worth the effort though.<br>
</div>
</div>
</div>
</div>
</blockquote>
That's actually pretty easy to do and part of the plan. Sending a
cmp_block rather than a block makes it all easier to implement.
It's just a matter of doing pnode->pushmessage("cmp_block",
compressed_block); and handling the "cmp_block" command string at
the other end.<br>
<blockquote
cite="mid:CAE-z3OXgWCHL_3CDR-ACc7ojbLi7EavyObNa3s7hPUMGj_V2+A@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><span class="">
<blockquote type="cite">
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Nov 10, 2015 at
5:40 AM, Johnathan Corgan via bitcoin-dev <span
dir="ltr"><<a moz-do-not-send="true"
href="mailto:bitcoin-dev@lists.linuxfoundation.org"
target="_blank">bitcoin-dev@lists.linuxfoundation.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
<div dir="ltr"><span>
<div style="font-size:small">On Mon, Nov
9, 2015 at 5:58 PM, gladoscc via
bitcoin-dev <span dir="ltr"><<a
moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:bitcoin-dev@lists.linuxfoundation.org"><a class="moz-txt-link-abbreviated" href="mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.linuxfoundation.org</a></a>></span>
wrote:<br>
</div>
</span>
<div class="gmail_extra">
<div class="gmail_quote"><span>
<div> </div>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir="ltr">I think 25%
bandwidth savings is certainly
considerable, especially for
people running full nodes in
countries like Australia where
internet bandwidth is lower and
there are data caps.</div>
</blockquote>
<div><br>
</div>
</span>
<div>
<div
style="font-size:small;display:inline">
This reinforces the idea that such
trade-off decisions should be be
local and negotiated between
peers, not a required feature of
the network P2P.</div>
</div>
</div>
<span>
<div><br>
</div>
-- <br>
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>Johnathan Corgan<br>
Corgan Labs - SDR
Training and
Development Services</div>
<div><a
moz-do-not-send="true"
class="moz-txt-link-freetext" href="http://corganlabs.com"><a class="moz-txt-link-freetext" href="http://corganlabs.com">http://corganlabs.com</a></a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</span></div>
</div>
<br>
_______________________________________________<br>
bitcoin-dev mailing list<br>
<a moz-do-not-send="true"
href="mailto:bitcoin-dev@lists.linuxfoundation.org"
target="_blank">bitcoin-dev@lists.linuxfoundation.org</a><br>
<a moz-do-not-send="true"
href="https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev"
rel="noreferrer" target="_blank">https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
bitcoin-dev mailing list
<a moz-do-not-send="true" href="mailto:bitcoin-dev@lists.linuxfoundation.org" target="_blank">bitcoin-dev@lists.linuxfoundation.org</a>
<a moz-do-not-send="true" href="https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev" target="_blank">https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev</a>
</pre>
</blockquote>
<br>
</span></div>
</blockquote>
</div>
<br>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
bitcoin-dev mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.linuxfoundation.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev">https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev</a>
</pre>
</blockquote>
<br>
</blockquote>
<br>
</body>
</html>
--------------060604090906010105020205--
|
