1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
|
Return-Path: <tomas@tomasvdw.nl>
Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org
[172.17.192.35])
by mail.linuxfoundation.org (Postfix) with ESMTPS id 519E83EE
for <bitcoin-dev@lists.linuxfoundation.org>;
Fri, 7 Apr 2017 21:14:55 +0000 (UTC)
X-Greylist: from auto-whitelisted by SQLgrey-1.7.6
Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com
[66.111.4.25])
by smtp1.linuxfoundation.org (Postfix) with ESMTPS id E841422B
for <bitcoin-dev@lists.linuxfoundation.org>;
Fri, 7 Apr 2017 21:14:52 +0000 (UTC)
Received: from compute2.internal (compute2.nyi.internal [10.202.2.42])
by mailout.nyi.internal (Postfix) with ESMTP id 405F520B1A;
Fri, 7 Apr 2017 17:14:52 -0400 (EDT)
Received: from web3 ([10.202.2.213])
by compute2.internal (MEProxy); Fri, 07 Apr 2017 17:14:52 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
messagingengine.com; h=content-transfer-encoding:content-type
:date:from:in-reply-to:message-id:mime-version:references
:subject:to:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; bh=77Jyat
DyzgsuGrQP/MJnVPTGCiFAAPC6x5eLEN792NA=; b=k1lI5dSU8samJbcpI7TaLb
mjqoXOBy7E6useh3M2OuUT6dAq82hcC4snw5VkBNvftaojQqAJRkUlmfBo+xEd1S
mFQmtqnAOU0phCB+1j1avu10t2cqDgYds9nyBw/lMJavQqAc8l1dusVaGChU5jiC
af9QmHjphAXfqhlPg88q1fv+bcMMb7MB3bd2yTOrGNU2mq9BM7qebcQ9tH4Yyc3W
mjCPIdYwPKfhcd3ojMXon0aUy+XK9ZRe9Eqi5GPn/D/CIOAkp/CpYFpXxrY3a65j
Aiv4/4T0ws20x30G9PD0wgzcqXV0dcmIXNI//o9/MqrScgtSmW+iV50i3nrWUNMg
==
X-ME-Sender: <xms:TAHoWGgzPNL3jD_zsZ10n8J_4S6J0174iJ4Hlm8Bik518xx0qBWcvg>
Received: by mailuser.nyi.internal (Postfix, from userid 99)
id 1252C9EC4C; Fri, 7 Apr 2017 17:14:52 -0400 (EDT)
Message-Id: <1491599691.1245876.937920664.6EBA20DC@webmail.messagingengine.com>
From: Tomas <tomas@tomasvdw.nl>
To: Bram Cohen <bram@bittorrent.com>, Gregory Maxwell <greg@xiph.org>,
Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org>
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: multipart/alternative; boundary="_----------=_149159969112458761"
X-Mailer: MessagingEngine.com Webmail Interface - ajax-7c174d5d
In-Reply-To: <CA+KqGko0cDY29bhznMxJJ7yAUTuB6GaDDNGBRwzssJUxM_53xQ@mail.gmail.com>
References: <1491516747.3791700.936828232.69F82904@webmail.messagingengine.com>
<CAAS2fgTJ8xOj8zCmBq1LN9OdMV-tDfSjVUPhLpO98cR1w-QAoA@mail.gmail.com>
<CA+KqGko0cDY29bhznMxJJ7yAUTuB6GaDDNGBRwzssJUxM_53xQ@mail.gmail.com>
Date: Fri, 07 Apr 2017 23:14:51 +0200
X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,DKIM_SIGNED,
DKIM_VALID,HTML_MESSAGE,RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.1
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
smtp1.linux-foundation.org
X-Mailman-Approved-At: Fri, 07 Apr 2017 21:15:41 +0000
Subject: Re: [bitcoin-dev] Using a storage engine without UTXO-index
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Bitcoin Protocol Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Fri, 07 Apr 2017 21:14:55 -0000
This is a multi-part message in MIME format.
--_----------=_149159969112458761
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset="utf-8"
Answering both,
On Fri, Apr 7, 2017 at 11:18 AM, Gregory Maxwell via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:
>>
>> I'm still lost on this-- AFAICT your proposals long term resource
>> requirements are directly proportional to the amount of
>> unspent output
>> data, which grows over time at some fraction of the total transaction
>> volume (plus the rate of spending which is more or less a constant).
>>
>> Can you help out my understanding here?
>>
On Fri, Apr 7, 2017, at 20:39, Bram Cohen wrote:
> Expanding on this question a bit, it's optimized for parallel access,
> but hard drive access isn't parallel and memory accesses are very
> fast, so shouldn't the target of optimization be about cramming as
> much as possible in memory and minimizing disk accesses?
The long term *minimal disk storage* requirement, can obviously not be
less then all the unspent outputs. Minimal disk requirements is not
something bitcrust attempts to address.
The storage that is accessed during peak load (block validation with
pre-synced transactions), is minimized as this only needs the
transaction index (to lookup ptrs from hashes), the tip of the spend-
tree and the tip of the spend-index (together to check double
spents/spending non-existing outputs). These not only easily fit in
RAM, but are accessed in a cache efficient way. *These* only grow with
inputs as the spend tree contains one record per input referencing the
output being spent.
Script validation is also not something bitcrust *directly* addresses;
it uses libbitcoinconsensus for the actual validation and lookups to
outputs are mostly similar. They are kept fast by trusting the OS on MRU
caching of transaction-outputs; I don't think that for this part the
UTXO index has much drawbacks,. Bitcrust seems to have a small advantage
due to the awesomeness of Rayon's parallelization and the lock-free data
structures, but a disadvantage in that keeping all spent outputs
decreases spatial locality of reference. Script validation is not the
innovative part.
Tomas
--_----------=_149159969112458761
Content-Transfer-Encoding: 7bit
Content-Type: text/html; charset="utf-8"
<!DOCTYPE html>
<html>
<head>
<title></title>
</head>
<body><div>Answering both,<br></div>
<div><br></div>
<div>On Fri, Apr 7, 2017 at 11:18 AM, Gregory Maxwell via bitcoin-dev <span dir="ltr"><<a href="mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.linuxfoundation.org</a>></span> wrote:<br></div>
<blockquote type="cite"><div><div defang_data-gmailquote="yes"><blockquote style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204, 204, 204);padding-left:1ex;" defang_data-gmailquote="yes"><div><br></div>
<div>I'm still lost on this-- AFAICT your proposals long term resource<br></div>
<div>requirements are directly proportional to the amount of unspent output<br></div>
<div>data, which grows over time at some fraction of the total transaction<br></div>
<div>volume (plus the rate of spending which is more or less a constant).<br></div>
<div><div><br></div>
</div>
<div>Can you help out my understanding here?<br></div>
<div><div><br></div>
</div>
</blockquote></div>
</div>
</blockquote><div><br></div>
<div>On Fri, Apr 7, 2017, at 20:39, Bram Cohen wrote:<br></div>
<blockquote type="cite"><div dir="ltr">Expanding on this question a bit, it's optimized for parallel access, but hard drive access isn't parallel and memory accesses are very fast, so shouldn't the target of optimization be about cramming as much as possible in memory and minimizing disk accesses?<br></div>
</blockquote><div><br></div>
<div>The long term *minimal disk storage* requirement, can obviously not be less then all the unspent outputs. Minimal disk requirements is not something bitcrust attempts to address.<br></div>
<div><br></div>
<div> The storage that is accessed during peak load (block validation with pre-synced transactions), is minimized as this only needs the transaction index (to lookup ptrs from hashes), the tip of the spend-tree and the tip of the spend-index (together to check double spents/spending non-existing outputs). These not only easily fit in RAM, but are accessed in a cache efficient way. *These* only grow with inputs as the spend tree contains one record per input referencing the output being spent.<br></div>
<div><br></div>
<div>Script validation is also not something bitcrust *directly* addresses; it uses libbitcoinconsensus for the actual validation and lookups to outputs are mostly similar. They are kept fast by trusting the OS on MRU caching of transaction-outputs; I don't think that for this part the UTXO index has much drawbacks,. Bitcrust seems to have a small advantage due to the awesomeness of Rayon's parallelization and the lock-free data structures, but a disadvantage in that keeping all spent outputs decreases spatial locality of reference. Script validation is not the innovative part.<br></div>
<div><br></div>
<div>Tomas<br></div>
</body>
</html>
--_----------=_149159969112458761--
|
