1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
|
Return-Path: <tomas@tomasvdw.nl>
Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org
[172.17.192.35])
by mail.linuxfoundation.org (Postfix) with ESMTPS id D7321949
for <bitcoin-dev@lists.linuxfoundation.org>;
Tue, 11 Apr 2017 08:43:33 +0000 (UTC)
X-Greylist: from auto-whitelisted by SQLgrey-1.7.6
Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com
[66.111.4.25])
by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 9D21816E
for <bitcoin-dev@lists.linuxfoundation.org>;
Tue, 11 Apr 2017 08:43:32 +0000 (UTC)
Received: from compute2.internal (compute2.nyi.internal [10.202.2.42])
by mailout.nyi.internal (Postfix) with ESMTP id 8778120CD6;
Tue, 11 Apr 2017 04:43:30 -0400 (EDT)
Received: from web3 ([10.202.2.213])
by compute2.internal (MEProxy); Tue, 11 Apr 2017 04:43:30 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
messagingengine.com; h=content-transfer-encoding:content-type
:date:from:in-reply-to:message-id:mime-version:references
:subject:to:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; bh=vMTC5d
1NyvF5BB1ow4UC/B0bFmH6LiO7G7Za4+PJEtc=; b=pKZ29BdYN1f6ziW6wwyi1j
ACq+B5oyrKQB/zS43yf+FtuAFgPnz59WitTq4zv212+CMqhQyh1065Nye6xTrlAq
vc0zgCmDKqqoU2l6wbUD78azj7Wq3BdLtAiaylMjMygTIkZvhPrFNucW8I7Rpb1l
QyK4Up0fsSrpnGNhgMG6qNaJPzWTn00YqCOl0lmwm8fHCVhiVDreXCWFKwRrD4nb
4u1U7rgbbXmwjvPckYIU4vMYqfzXQIYTMJzxsJQGwbiAKVKR7JkhI7GRWpdVja4d
sd4WcvWQ/I7Td00Lbv0TxD5J+5j+ZaKrKvF94Zq6vEGXJz0BTCOp78DcViWnuoVw
==
X-ME-Sender: <xms:MpfsWNj0PkUL3MA3aGp-l9DxovPLI8ksPYFo1sY3u93KOPFxWNeuUA>
Received: by mailuser.nyi.internal (Postfix, from userid 99)
id 585829EBFD; Tue, 11 Apr 2017 04:43:30 -0400 (EDT)
Message-Id: <1491900210.2802950.940953480.4C391C68@webmail.messagingengine.com>
From: Tomas <tomas@tomasvdw.nl>
To: Eric Voskuil <eric@voskuil.org>,
Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org>,
Libbitcoin Development <libbitcoin@lists.dyne.org>
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset="utf-8"
X-Mailer: MessagingEngine.com Webmail Interface - ajax-0b509d77
Date: Tue, 11 Apr 2017 10:43:30 +0200
In-Reply-To: <1b6b300d-4b24-2a64-12a3-4e654174c132@voskuil.org>
References: <1491516747.3791700.936828232.69F82904@webmail.messagingengine.com>
<eebc06a4-5ab8-46a8-2f50-a472cb57a775@voskuil.org>
<1491524267.715789.936916864.1156D8CB@webmail.messagingengine.com>
<83618cca-c6a3-301c-af70-ab7807474f30@voskuil.org>
<1491695882.3440363.938700256.78C37AC3@webmail.messagingengine.com>
<1b6b300d-4b24-2a64-12a3-4e654174c132@voskuil.org>
X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,DKIM_SIGNED,
DKIM_VALID,RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.1
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
smtp1.linux-foundation.org
X-Mailman-Approved-At: Tue, 11 Apr 2017 12:50:19 +0000
Subject: Re: [bitcoin-dev] Using a storage engine without UTXO-index
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Bitcoin Protocol Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Tue, 11 Apr 2017 08:43:33 -0000
On Tue, Apr 11, 2017, at 03:44, Eric Voskuil wrote:
> As I understand it you would split tx inputs and outputs and send them
> independently, and that you intend this to be a P2P network
> optimization - not a consensus rule change. So my comments are based
> on those inferences. If we are talking about consensus changes this
> conversation will end up in an entirely different place.
> I don't agree with the input/output relevance statements above. When a
> tx is announced the entire tx is relevant. It cannot be validated as
> outputs only. If it cannot be validated it cannot be stored by the
> node. Validating the outputs only would require the node store invalid
> transactions.
Splitting transactions only happens *on storage* and is just a minor
optimization compared to storing them in full. (actually a very recent
change with only marginally better results). This is simply because the
output scripts are read on script validation, and storing the outputs of
the transaction separately ensures better spatial locality of reference
(the inputs are just "in the way"). This is not relevant when using a
UTXO-index, because the outputs are then directly stored in the index,
where bitcrust has to read them from the transaction data.
It is not my intention to send them independently.
> I do accept that a double-spend detection is not an optimal criteria
> by which to discard a tx. One also needs fee information. But without
> double-spend knowledge the node has no rational way to defend itself
> against an infinity of transactions that spend the minimal fee but
> also have conflicting inputs (i.e. risking the fee only once). So tx
> (pool) validation requires double-spend knowledge and at least a
> summary from outputs.
Double spent information is still available to the network node and
could still be used for DoS protection, although I do believe
alternatives may exist.
>
> A reorg is conceptual and cannot be engineered out. What you are
> referring to is a restructuring of stored information as a consequence
> of a reorg. I don't see this as related to the above. The ability to
> perform reorganization via a branch pointer swap is based not on the
> order or factoring of validation but instead on the amount of
> information stored. It requires more information to maintain multiple
> branches.
>
> Transactions have confirmation states, validation contexts and spender
> heights for potentially each branch of an unbounded number of
> branches. It is this requirement to maintain that state for each
> branch that makes this design goal a very costly trade-off of space
> and complexity for reorg speed. As I mentioned earlier, it's the
> optimization for this scenario that I find questionable.
Sure, we can still call switching tips a "reorg". And it is indeed a
trade off as orphan blocks are stored, but a block in the spend tree
takes only ~12kb and contains the required state information.
I believe this trade off reduced complexity. For the earlier tree this
could be pruned.
> Because choosing the lesser amount of work is non-consensus behavior.
> Under the same circumstances (i.e. having seen the same set of blocks)
> two nodes will disagree on whether there is one confirmation or no
> confirmations for a given tx. This disagreement will persist (i.e. why
> take the weaker block only to turn around and replace it with the
> stronger block that arrives a few seconds or minutes later). It stands
> to reason that if one rejects a stronger block under a race condition,
> one would reorg out a stronger block when a weaker block arrives a
> little after the stronger block. Does this "optimization" then apply
> to chains of blocks too?
The blockchain is - by design - only eventually consistent across nodes.
Even if nodes would use the same "tip-selection" rules, you cannot rely
on all blocks being propagated and hence each transaction having the
same number of confirmations across all nodes.
As a simpler example, if two miners both mine a block at approximately
the same time and send it to each other, then surely they would want to
continue mining on their own block. Otherwise they would be throwing
away their own reward.
And yes, this can also happen over multiple blocks, but the chances of
consistency are vastly increased with each confirmation.
> Accepting a block that all previous implementations would have
> rejected under the same circumstance could be considered a hard fork,
> but you may be right.
I am not talking about rejecting blocks, I am only talking choosing on
which tip to mine.
> > Frankly, I think this is a bit of an exaggeration. Soft forks are
> > counted on a hand, and I don't think there are many - if any -
> > transactions in the current chain that have changed compliance
> > based on height.
>
> Hope is a bug.
>
> If you intend this to be useful it has to help build the chain, not
> just rely on hardwiring checkpoints once rule changes are presumed to
> be buried deeply enough to do so (as the result of other implementations
> ).
>
> I understand this approach, it was ours at one time. There is a
> significant difference, and your design is to some degree based on a
> failure to fully consider this. I encourage you to not assume any
> consensus-related detail is too small.
I am not failing to consider this, and I don't consider this too small .
But ensuring contextual transaction validity by "validate => valid with
rules X,Y,Z" and then checking the active rules (softfork activation) on
order validation, will give logically the same results as "validate with
X,Y,Z => valid". This is not "hardwiring checkpoints" at all.
> You cannot have a useful performance measure without full compliance.
I agree that the results are preliminary and I will post more if the
product reaches later stages.
> It's worth noting that many of your stated objectives, including
> modularity, developer platform, store isolation, consensus rule
> isolation (including optional use of libbitcoinconsensus) are implemente
> d.
>
> It seems like you are doing some good work and it's not my intent to
> discourage that. Libbitcoin is open source, I don't get paid and I'm
> not selling anything. But if you are going down this path you should
> be aware of it and may benefit from our successes as well as some of
> the other stuff :). And hopefully we can get the benefit of your
> insights as well.
Thank you, I will definitely further dive into libbitcoin, and see what
insights I can use for Bitcrust.
Tomas
|
