1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
|
Received: from sog-mx-1.v43.ch3.sourceforge.com ([172.29.43.191]
helo=mx.sourceforge.net)
by sfs-ml-4.v29.ch3.sourceforge.com with esmtp (Exim 4.76)
(envelope-from <shadders.del@gmail.com>) id 1R1U5k-0001VU-US
for bitcoin-development@lists.sourceforge.net;
Thu, 08 Sep 2011 02:11:48 +0000
Received-SPF: pass (sog-mx-1.v43.ch3.sourceforge.com: domain of gmail.com
designates 209.85.160.175 as permitted sender)
client-ip=209.85.160.175; envelope-from=shadders.del@gmail.com;
helo=mail-gy0-f175.google.com;
Received: from mail-gy0-f175.google.com ([209.85.160.175])
by sog-mx-1.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128)
(Exim 4.76) id 1R1U5j-000310-W3
for bitcoin-development@lists.sourceforge.net;
Thu, 08 Sep 2011 02:11:48 +0000
Received: by gyg8 with SMTP id 8so94633gyg.34
for <bitcoin-development@lists.sourceforge.net>;
Wed, 07 Sep 2011 19:11:42 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.8.32 with SMTP id f32mr92172ibf.65.1315447902344; Wed, 07
Sep 2011 19:11:42 -0700 (PDT)
Sender: shadders.del@gmail.com
Received: by 10.231.15.67 with HTTP; Wed, 7 Sep 2011 19:11:42 -0700 (PDT)
Date: Thu, 8 Sep 2011 12:11:42 +1000
X-Google-Sender-Auth: mElmtGZFZdx-clf1WbLXx80W2xA
Message-ID: <CAOPxoMshHGy00d5c2XC2OV9Jy5MAT4P6ySQ5+2gsFQrvhH30Xg@mail.gmail.com>
From: Steve diy-eco-life <info@diy-eco-life.com>
To: Gavin Andresen <gavinandresen@gmail.com>,
Bitcoin Dev <bitcoin-development@lists.sourceforge.net>
Content-Type: text/plain; charset=ISO-8859-1
X-Spam-Score: 1.2 (+)
X-Spam-Report: Spam Filtering performed by mx.sourceforge.net.
See http://spamassassin.org/tag/ for more details.
-1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for
sender-domain
0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
(shadders.del[at]gmail.com)
1.2 RCVD_IN_BL_SPAMCOP_NET RBL: Received via a relay in bl.spamcop.net
[Blocked - see <http://www.spamcop.net/bl.shtml?209.85.160.175>]
-0.0 SPF_PASS SPF: sender matches SPF record
0.1 DKIM_SIGNED Message has a DKIM or DK signature,
not necessarily valid
-0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
1.5 SF_NO_SPF_SPAM SF_NO_SPF_SPAM
X-Headers-End: 1R1U5j-000310-W3
Subject: [Bitcoin-development] bitcoind multiplexing proxy -
request/response routing problem
X-BeenThere: bitcoin-development@lists.sourceforge.net
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: <bitcoin-development.lists.sourceforge.net>
List-Unsubscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
<mailto:bitcoin-development-request@lists.sourceforge.net?subject=unsubscribe>
List-Archive: <http://sourceforge.net/mailarchive/forum.php?forum_name=bitcoin-development>
List-Post: <mailto:bitcoin-development@lists.sourceforge.net>
List-Help: <mailto:bitcoin-development-request@lists.sourceforge.net?subject=help>
List-Subscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
<mailto:bitcoin-development-request@lists.sourceforge.net?subject=subscribe>
X-List-Received-Date: Thu, 08 Sep 2011 02:11:49 -0000
This a reworking of a post I made on the bitcoinj list under a
different topic but it's something I'd like to throw out there for
input.
I'm going to build a proof of concept this weekend of the multiplexing
proxy that Gavin and Mike were talking about here:
http://sourceforge.net/mailarchive/message.php?msg_id=28049519
Initially just a dumb as doornails proxy between one local bitcoind
and one remote node. Once I've got that far the next step is to work
out how deal with request-response exchanges from multiple remote
nodes. I discussed this tatcm on IRC last night. The problem is
after relaying several requests from different remote nodes to the
local daemon you expect multiple responses to come back. How identify
which response matches which client's request. Bitcoind can implicitly
identify the recipient based on which connection made the request. By
piling all the requests onto one channel we lose this identifier. I
can think of 3 approaches to dealing with this:
1/ Generate a unique key from the request and can also be generated
from the response. e.g. getheaders key could be "headers" +
hash_start. We locally store a mapping to client (or clients) that
requested it and pass it to bitcoind. When we get a headers packet
back unique key = "headers" + hash_of_first_header, so we can lookup
the clients who requested it and send it back.
The unique key should have the following properties:
- can be reliably generated from both the request and the response.
- identical requests from different clients should generate the same
unique key (this allows us to recycle responses)
Problems:
This is dependent on each pair of request/response messages being
guaranteed to contain information needed to create an identical unique
key. I haven't looked in detail at every request/response pair yet to
confirm this. If it is the case then this is an onerous obligation to
place on the protocol to fulfil this condition for all future protocol
changes.
To obtain guaranteed uniqueness may require large keys. Is
'almost-unique' an option? e.g. generating a key off a getblocks
request using the first n bytes of each block_locator_hash would be
much smaller/faster and very likely to be unique. Are the risks of
sending back the wrong response to a request acceptable?
2/ Modify bitcoind to accept sequence numbers for request/response
type messages, similar to 'id' field in json-rpc. This is more
reliable but potentially quite invasive to the bitcoin protocol. It
also loses the inherent de-duplication of requests that we get with
the previous solution. If it were to be implemented I'd suggest
something like a separate sequence number message. i.e. proxy sends
seq message containing a unique ID. The contract is that the seq
message refers to the next message that comes over the wire. When the
response is ready the bitcoind sends a seq message with matching id
then sends the response message. A separate seq message at least
leaves the existing protocol untouched as the handling will remain
unchanged unless a request type message is preceded by a seq message.
This approach allows the proxy remain a lot thinner and dumber but we
lose a lot of the de-duplicating efficiencies from the first approach.
If we want to add this capability as well we essentially need to
implement option 1 as well although in this case we have a reliable
fallback. i.e. If we can't generate a unique key then we just use
single request/response matching via the seq id.
3/ Make the proxy intelligent enough to handle these requests itself.
Using getheaders as an example again. Proxy maintains it's own local
cache of headers. when a getheaders message comes in the proxy checks
if it has all requested. If not it requests the missing ones from the
local daemon, adds to it's cache and builds a headers response itself.
In this case the proxy definately has to be protocol version aware...
Advantages:
- This probably achieves the best combination of request/response
matching reliability and de-duplication of work.
Disadvantages:
- Complexity. The proxy needs to be far more protocol aware which
creates a maintenance dependency for future protocol changes.
Having spent the last couple of days studying the protocol I'm
inclined toward the first approach as an initial easy implementation
with a view to moving to the 3rd approach. It appears that most
response type messages could be relatively easily constructed from a
local cache. Before I looked at the protocol I would have said no way
to the 3rd but the depth of protocol awareness for 1 or 3 is not
really much different. Option 2 allows for a much dumber and thinner
proxy but loses a lot of potential efficiencies and if those were to
be regained it would require the same level of protocol awareness
inherent in 1 and 3 anyway. It would also require someone on the
bitcoind side to put their hand up to add the seq message
functionality as I don't have any c skills to speak of.
Ultimately option 3 is where I was seeing the proxy progressing to at
some point far in the future but the routing problem needs to be
solved right from the beginning as I see it.
I hope I'm not over complicating it. If anyone can think of a simpler
approach to the request/response routing problem I'm all ears.
|
