1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
|
Received: from sog-mx-4.v43.ch3.sourceforge.com ([172.29.43.194]
helo=mx.sourceforge.net)
by sfs-ml-2.v29.ch3.sourceforge.com with esmtp (Exim 4.76)
(envelope-from
<SRS0=nh6BJu=HM=godofgod.co.uk=matthewmitchell@eigbox.net>)
id 1TCFxv-00085p-Ji for bitcoin-development@lists.sourceforge.net;
Thu, 13 Sep 2012 20:24:47 +0000
Received-SPF: pass (sog-mx-4.v43.ch3.sourceforge.com: domain of eigbox.net
designates 66.96.188.16 as permitted sender)
client-ip=66.96.188.16;
envelope-from=SRS0=nh6BJu=HM=godofgod.co.uk=matthewmitchell@eigbox.net;
helo=bosmailout16.eigbox.net;
Received: from bosmailout16.eigbox.net ([66.96.188.16])
by sog-mx-4.v43.ch3.sourceforge.com with esmtp (Exim 4.76)
id 1TCFxu-0007zf-Px for bitcoin-development@lists.sourceforge.net;
Thu, 13 Sep 2012 20:24:47 +0000
Received: from bosmailscan20.eigbox.net ([10.20.15.20])
by bosmailout16.eigbox.net with esmtp (Exim) id 1TCFxp-0008Jb-Lv
for bitcoin-development@lists.sourceforge.net;
Thu, 13 Sep 2012 16:24:41 -0400
Received: from bosimpout02.eigbox.net ([10.20.55.2])
by bosmailscan20.eigbox.net with esmtp (Exim)
id 1TCFxo-0002en-US; Thu, 13 Sep 2012 16:24:40 -0400
Received: from bosauthsmtp06.eigbox.net ([10.20.18.6])
by bosimpout02.eigbox.net with NO UCE
id ykQg1j00C07rX7u01kQgwH; Thu, 13 Sep 2012 16:24:41 -0400
X-Authority-Analysis: v=2.0 cv=V8rKJ5bi c=1 sm=1
a=Vnyysazsc9gF4v5jbSvB8A==:17 a=Goz4v7xpImgA:10 a=JhU9KSwl1l8A:10
a=RmqW3wxksLsA:10 a=N659UExz7-8A:10 a=eGitJVp2AAAA:8 a=-aUnnjAAwhoA:10
a=pGLkceISAAAA:8 a=FYJQRDSw8x8RavvxkmoA:9 a=pILNOxqGKmIA:10
a=MSl-tDqOz04A:10
a=auv_4ZWqOX5hJEMy:21 a=i7WL7bnb9U27SoyJ:21
a=fIc3/5IyPUehxkj7BpkQ7Q==:117
X-EN-OrigOutIP: 10.20.18.6
X-EN-IMPSID: ykQg1j00C07rX7u01kQgwH
Received: from [109.175.173.27]
by bosauthsmtp06.eigbox.net with esmtpsa (TLSv1:AES128-SHA:128)
(Exim) id 1TCFxo-0004tV-9S; Thu, 13 Sep 2012 16:24:40 -0400
Content-Type: text/plain; charset=windows-1252
Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1486\))
From: Matthew Mitchell <matthewmitchell@godofgod.co.uk>
In-Reply-To: <20120913185900.GA393@vps7135.xlshosting.net>
Date: Thu, 13 Sep 2012 21:24:35 +0100
Content-Transfer-Encoding: quoted-printable
Message-Id: <9F7EFD92-B922-45E9-97A8-03FFC811502D@godofgod.co.uk>
References: <2104FC7F-0AE0-4C55-9987-B20EFCE19FC3@godofgod.co.uk>
<CAAS2fgSymOJ=cQnNwK9+nvRYszHHH4vtUpoQYWNNYoVaYB5Gpg@mail.gmail.com>
<19ED4257-0BCA-41C5-A533-B0AB9B500181@godofgod.co.uk>
<CAAS2fgRfXBrsFm_zdNFe7Z4FN7uQ5zSg++scng=hNrHZZV93VQ@mail.gmail.com>
<CANEZrP0HHhSXyN9pWODtTxHMfRB4B0w-eSdvNHH13WwLVgECrw@mail.gmail.com>
<FC0AF926-2CBD-49BA-A3B7-AF9D70A83CE4@godofgod.co.uk>
<CAAS2fgSd6t038Yzb-Vy34J61xob+kVqA8pK+w1+ZwJ6XtQRJww@mail.gmail.com>
<2B95CF41-4304-4D2A-9ABF-198D97B7449B@godofgod.co.uk>
<CAAS2fgQi8QFwU2M=wLiDodt3SmO48vUV5Sp3YCb1OmGJ5m=E7Q@mail.gmail.com>
<A1DC7DE8-F355-4B61-AF18-94F07DF6421E@godofgod.co.uk>
<20120913185900.GA393@vps7135.xlshosting.net>
To: Pieter Wuille <pieter.wuille@gmail.com>,
"bitcoin-development@lists.sourceforge.net"
<bitcoin-development@lists.sourceforge.net>
X-Mailer: Apple Mail (2.1486)
X-EN-UserInfo: c68a83c59c94ef03b40bb4bc312c51e4:dffc0a9b4c8a0435ad832ff5852cab82
X-EN-AuthUser: godofgod@godofgod.co.uk
Sender: Matthew Mitchell <matthewmitchell@godofgod.co.uk>
X-EN-OrigIP: 109.175.173.27
X-EN-OrigHost: unknown
X-Spam-Score: -1.3 (-)
X-Spam-Report: Spam Filtering performed by mx.sourceforge.net.
See http://spamassassin.org/tag/ for more details.
-1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for
sender-domain
-0.0 SPF_PASS SPF: sender matches SPF record
-0.5 RP_MATCHES_RCVD Envelope sender domain matches handover relay
domain 0.7 AWL AWL: From: address is in the auto white-list
X-Headers-End: 1TCFxu-0007zf-Px
Subject: Re: [Bitcoin-development] Segmented Block Relaying BIP draft.
X-BeenThere: bitcoin-development@lists.sourceforge.net
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: <bitcoin-development.lists.sourceforge.net>
List-Unsubscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
<mailto:bitcoin-development-request@lists.sourceforge.net?subject=unsubscribe>
List-Archive: <http://sourceforge.net/mailarchive/forum.php?forum_name=bitcoin-development>
List-Post: <mailto:bitcoin-development@lists.sourceforge.net>
List-Help: <mailto:bitcoin-development-request@lists.sourceforge.net?subject=help>
List-Subscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
<mailto:bitcoin-development-request@lists.sourceforge.net?subject=subscribe>
X-List-Received-Date: Thu, 13 Sep 2012 20:24:47 -0000
On 13 Sep 2012, at 19:59, Pieter Wuille <pieter.wuille@gmail.com> wrote:
> You want to parallellize block downloads, while at the same time =
preventing re-download of transactions that are already known.
> To do so, a requesting node would first request (for example) the 8 =
level-3 hashes, then start 8 parallel threads to download the
> transactions from presumably 8 different peers. Each thread then =
fetches the transaction id's that correspond to its own 1/8th of
> the block, and requests the transactions whose txid is not yet known.
> Comparing this with Gregory's own suggestion (just fetch the entire =
txid list at first, and then (again as parallellized as needed)
> fetch the unknown transactions from several peers), this does indeed =
have an advantage: you need to download (relatively) far less
> data before the threaded part can start. If this is what you propose, =
it is certainly meaningful, but the gains aren't very large,
> in my opinion.
This is not fully correct. I propose downloading the roots of the =
segments when you are not looking to remove redundant transaction =
downloads. This would be the case when the node is not up-to-date and is =
unlikely to have transactions in the required blocks. When a node is =
up-to-date and can benefit from removing redundant transaction downloads =
it can switch to downloading all the transactions hashes by specifying a =
high level number. Also I wouldn't recommend using one thread per =
connection, I'd recommend using one thread for all connections.
> However, my impression while reading your proposal was at times that =
you intend to process the different layers of the
> Merkle tree iteratively after starting the initial parallel segments. =
I don't think that is useful, as you'll need the actual
> txids anyway before deciding whether they need to be downloaded at =
all, it adds several round-trips, and once you have downloaded
> the intermediate merkle hashes for about 8 levels, you've already =
transferred more data than the transactions themselves. I think
> Gregory also assumed something like this, making him question why it's =
useful. Anyway, this whole paragraph is one assumption, so
> if it's not the case, ignore me.
This isn't what I was suggesting. I was suggesting you only need to =
download one level. Once you have done that you verify everything for =
the hashes on that level.
>=20
> Can you clarify what you mean? Preferably by giving a concrete =
sequence of exchanged messages, with a given purpose?
Well you will need to ask for the headers first. Once you do that you =
can start downloading the full blocks for the headers. The node should =
use "get blocks" to find nodes with segments of the blocks they need. =
Now for each block:
1. Send getsiginv to a number of peers to know the segments of the =
blocks they have.=20
2. Send gettreelevel requesting a level of the merkle tree from a peer =
that can provide it. When up-to-date use a high level to get the =
transaction hashes to find redundant data.
3. Validate the treelevel response
4. Send getsegment for each segment wanted (at the same time where =
possible) to the peers that have these segments. Skip transactions =
already known.
5. Validate the transactions in each segment received. Stop if the block =
is invalid and disconnect peers that give transactions which do not fit =
the merkle tree.
6. Revert to getdata if peers using the protocol cannot satisfy the =
block download.
When a valid block segment is received, include the block in inv and =
headers messages for other peers using the protocol. Thus relaying can =
begin before the entire block is downloaded.
I'm thinking about improvements to this proposal. I'll get to that =
tomorrow perhaps=85
Thank you everyone for the replies.=
|
