1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
|
Received: from sog-mx-2.v43.ch3.sourceforge.com ([172.29.43.192]
helo=mx.sourceforge.net)
by sfs-ml-3.v29.ch3.sourceforge.com with esmtp (Exim 4.76)
(envelope-from <runesvend@gmail.com>) id 1UiNxG-0005vB-Ro
for bitcoin-development@lists.sourceforge.net;
Fri, 31 May 2013 11:57:10 +0000
Received-SPF: pass (sog-mx-2.v43.ch3.sourceforge.com: domain of gmail.com
designates 74.125.82.169 as permitted sender)
client-ip=74.125.82.169; envelope-from=runesvend@gmail.com;
helo=mail-we0-f169.google.com;
Received: from mail-we0-f169.google.com ([74.125.82.169])
by sog-mx-2.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128)
(Exim 4.76) id 1UiNxF-0000DH-PS
for bitcoin-development@lists.sourceforge.net;
Fri, 31 May 2013 11:57:10 +0000
Received: by mail-we0-f169.google.com with SMTP id q55so1171152wes.28
for <bitcoin-development@lists.sourceforge.net>;
Fri, 31 May 2013 04:57:03 -0700 (PDT)
X-Received: by 10.194.95.167 with SMTP id dl7mr9053104wjb.46.1370001423551;
Fri, 31 May 2013 04:57:03 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.194.161.227 with HTTP; Fri, 31 May 2013 04:56:43 -0700 (PDT)
From: =?ISO-8859-1?Q?Rune_Kj=E6r_Svendsen?= <runesvend@gmail.com>
Date: Fri, 31 May 2013 13:56:43 +0200
Message-ID: <CAH2=CKzW41TYbX6c1F8oknA_LttOaA8vmDPmojuowXgEADY61g@mail.gmail.com>
To: bitcoin-development@lists.sourceforge.net
Content-Type: multipart/alternative; boundary=047d7bb042f231519704de025082
X-Spam-Score: -0.6 (/)
X-Spam-Report: Spam Filtering performed by mx.sourceforge.net.
See http://spamassassin.org/tag/ for more details.
-1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for
sender-domain
0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
(runesvend[at]gmail.com)
-0.0 SPF_PASS SPF: sender matches SPF record
1.0 HTML_MESSAGE BODY: HTML included in message
-0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from
author's domain
0.1 DKIM_SIGNED Message has a DKIM or DK signature,
not necessarily valid
-0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
X-Headers-End: 1UiNxF-0000DH-PS
Subject: [Bitcoin-development] Implementing batch processing for -blocknotify
X-BeenThere: bitcoin-development@lists.sourceforge.net
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: <bitcoin-development.lists.sourceforge.net>
List-Unsubscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
<mailto:bitcoin-development-request@lists.sourceforge.net?subject=unsubscribe>
List-Archive: <http://sourceforge.net/mailarchive/forum.php?forum_name=bitcoin-development>
List-Post: <mailto:bitcoin-development@lists.sourceforge.net>
List-Help: <mailto:bitcoin-development-request@lists.sourceforge.net?subject=help>
List-Subscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
<mailto:bitcoin-development-request@lists.sourceforge.net?subject=subscribe>
X-List-Received-Date: Fri, 31 May 2013 11:57:11 -0000
--047d7bb042f231519704de025082
Content-Type: text/plain; charset=ISO-8859-1
Hello dear list
I have an application that wants to keep up with new blocks as they come
in. For that I can use the -blocknotify option with bitcoind, which will
execute my application for each new block.
The problem is that my app isn't necessarily quick enough to finish its
work before a new block comes in and the app is executed again. This means
the that bitcoind might keep executing my application even though the
previous instance hasn't finished, and that's fairly inefficient
resource-wise, as many instances of the application will be running
simultaneously.
I've discussed this with wumpus on bitcoin-dev, and we figured out a
solution that might be better. It could replace -blocknotify or we could
put it in a new function called -batchblocknotify
The idea is that when bitcoind is executed with the -batchblocknotify
option, and it's missing a lot of blocks, upon the first incoming block,
the command specified by -batchblocknotify is executed, and if additional
blocks come in while this command is still running, we add the block hashes
to a list instead of executing the command again. When the previous command
finishes, we execute it again and pass two parameters to it: 1. the first
block hash in the list of queued blocks, and 2. the number of blocks that
have come in while the last command was executing.
This prevents bitcoind from "fork bombing" the system, and allows the
command to handle incoming blocks in batches.
Would this make sense as an approach?
I've been looking at the code and I'm not sure how to implement it.
As far as I can see, I need to pass an object - whose state is retained
between calls - to the thread function (runCommand) that runs the command,
which contains a variable that keeps track of whether a previously executed
command is still running, and that contains a list of block hashes that
haven't been processed. And I'm not sure how to do this.
The runCommand thread is started in SetBestChain() in
main.cpp. SetBestChain() is executed by ConnectBestBlock() in main.cpp.
ConnectBestBlock() is executed by CBlock::AddToBlockIndex() in main.cpp.
CBlock::AddToBlockIndex() is executed by CBlock::AcceptBlock() in main.cpp.
CBlock::AcceptBlock() is executed by ProcessBlock() in main.cpp.
ProcessBlock() is executed by ProcessMessage() in main.cpp. And so on, and
so forth.
What's the right way to create an object that can be passed to the
runCommand thread, whose state is retained, so it can hold information
about whether the -batchblocknotify command is still executing, and contain
a list of blocks that are waiting to be passed to the -batchblocknotify
command?
I assume I shouldn't add a new parameter to the ProcessMessage() function,
which passes it to ProcessBlock(), which passes it to AcceptBlock() which
passes it to AddToBlockIndex()... and so on.
Would it be appropriate to store this object inside the CValidationState
class that is passed to SetBestChain()?
I'm not quite so how to go about this.
/Rune
--047d7bb042f231519704de025082
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr">Hello dear list
<div><br></div><div>I have an application that wants to keep up with new bl=
ocks as they come in. For that I can use the -blocknotify option with bitco=
ind, which will execute my application for each new block.</div><div><br>
</div><div>The problem is that my app isn't necessarily quick enough to=
finish its work before a new block comes in and the app is executed again.=
This means the that bitcoind might keep executing my application even thou=
gh the previous instance hasn't finished, and that's fairly ineffic=
ient resource-wise, as many instances of the application will be running si=
multaneously.</div>
<div><br></div><div>I've discussed this with=A0wumpus on bitcoin-dev, a=
nd we figured out a solution that might be better. It could replace -blockn=
otify or we could put it in a new function called -batchblocknotify</div>
<div><br></div><div>The idea is that when bitcoind is executed with the -ba=
tchblocknotify option, and it's missing a lot of blocks, upon the first=
incoming block, the command specified by -batchblocknotify is executed, an=
d if additional blocks come in while this command is still running, we add =
the block hashes to a list instead of executing the command again. When the=
previous command finishes, we execute it again and pass two parameters to =
it: 1. the first block hash in the list of queued blocks, and 2. the number=
of blocks that have come in while the last command was executing.</div>
<div><br></div><div>This prevents bitcoind from "fork bombing" th=
e system, and allows the command to handle incoming blocks in batches.</div=
><div><br></div><div>Would this make sense as an approach?</div><div><br>
</div><div>I've been looking at the code and I'm not sure how to im=
plement it.</div><div><br></div><div>As far as I can see, I need to pass an=
object - whose state is retained between calls - to the thread function (r=
unCommand) that runs the command, which contains a variable that keeps trac=
k of whether a previously executed command is still running, and that conta=
ins a list of block hashes that haven't been processed. And I'm not=
sure how to do this.</div>
<div><br></div><div>The runCommand thread is started in=A0SetBestChain() in=
main.cpp.=A0SetBestChain() is executed by=A0ConnectBestBlock() in main.cpp=
. ConnectBestBlock() is executed by=A0CBlock::AddToBlockIndex() in main.cpp=
. CBlock::AddToBlockIndex() is executed by=A0CBlock::AcceptBlock() in main.=
cpp. CBlock::AcceptBlock() is executed by=A0ProcessBlock() in main.cpp. Pro=
cessBlock() is executed by=A0ProcessMessage() in main.cpp. And so on, and s=
o forth.</div>
<div><br></div><div>What's the right way to create an object that can b=
e passed to the runCommand thread, whose state is retained, so it can hold =
information about whether the -batchblocknotify command is still executing,=
and contain a list of blocks that are waiting to be passed to the -batchbl=
ocknotify command?</div>
<div><br></div><div>I assume I shouldn't add a new parameter to the Pro=
cessMessage() function, which passes it to ProcessBlock(), which passes it =
to AcceptBlock() which passes it to AddToBlockIndex()... and so on.</div>
<div><br></div><div>Would it be appropriate to store this object inside the=
=A0CValidationState class that is passed to=A0SetBestChain()?</div><div><br=
></div><div>I'm not quite so how to go about this.</div><div><br></div>
<div>/Rune</div></div>
--047d7bb042f231519704de025082--
|
