path: root/e0/1d182731546cef670d8cba5e4c8d7675c2e89f

Return-Path: <tomas@tomasvdw.nl>
Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org
	[172.17.192.35])
	by mail.linuxfoundation.org (Postfix) with ESMTPS id 411FFB90
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Fri,  7 Apr 2017 16:02:37 +0000 (UTC)
X-Greylist: from auto-whitelisted by SQLgrey-1.7.6
Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com
	[66.111.4.25])
	by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 4EB4E151
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Fri,  7 Apr 2017 16:02:36 +0000 (UTC)
Received: from compute2.internal (compute2.nyi.internal [10.202.2.42])
	by mailout.nyi.internal (Postfix) with ESMTP id 97F8720AC1;
	Fri,  7 Apr 2017 12:02:35 -0400 (EDT)
Received: from web3 ([10.202.2.213])
	by compute2.internal (MEProxy); Fri, 07 Apr 2017 12:02:35 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
	messagingengine.com; h=content-transfer-encoding:content-type
	:date:from:in-reply-to:message-id:mime-version:references
	:subject:to:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; bh=BnVY9p
	lJBVd+tQwwdJHcIIzjgRHS78W4ClxqmRyiz54=; b=agoTBvsjlu5wP4k3dVA2om
	TNnMTp2bzbEORsgapedPlJ7ydnWHPFigXAaYZgf0UuPSPamUURGelgof6689PIFC
	wOizS45TafsiSqTEnSzK5gSp7uPU6CfmDLst+MgvLJiqNzmkAK7Hj19y4m08cqsi
	lbFDp8uq4DP0GagC+PvKZ/q8rGpXkVJdxeH6NAXA7nWxq1pIP1n1pZIbiQSdAwMp
	sacnv4aeezX793N+22OfroBcfAC01o90cXWxu3yLbeMLVpd7OubNeFnHFwRm5yAZ
	EcVhzR1mjMtRgW89dmxsNG2oC/+p8ZK5sAp5FeCUCX9Qi1l4cKYT4wElWiIUfsyg
	==
X-ME-Sender: <xms:G7jnWOPq840q-S7L_VsRxYJ_WvF6WRsYyOygx1SRk8TMbK3yROEU1w>
Received: by mailuser.nyi.internal (Postfix, from userid 99)
	id 7642B9EC4C; Fri,  7 Apr 2017 12:02:35 -0400 (EDT)
Message-Id: <1491580955.73830.937628696.4EEA507C@webmail.messagingengine.com>
From: Tomas <tomas@tomasvdw.nl>
To: Greg Sanders <gsanders87@gmail.com>,
	Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org>
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: multipart/alternative; boundary="_----------=_1491580955738300"
X-Mailer: MessagingEngine.com Webmail Interface - ajax-7c174d5d
In-Reply-To: <CAB3F3DvG4NSBEk1vS-KWQjn3PXWOwF3xNP7_txUpwTdDMHettg@mail.gmail.com>
Date: Fri, 07 Apr 2017 18:02:35 +0200
References: <1491516747.3791700.936828232.69F82904@webmail.messagingengine.com>
	<f55cdaa01e5b37036a674df6eefbfebc.squirrel@mail.fairluck.net>
	<1491554876.1963053.937226528.7010832E@webmail.messagingengine.com>
	<CAB3F3DvG4NSBEk1vS-KWQjn3PXWOwF3xNP7_txUpwTdDMHettg@mail.gmail.com>
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,URIBL_RHS_DOB autolearn=ham
	version=3.3.1
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
	smtp1.linux-foundation.org
X-Mailman-Approved-At: Fri, 07 Apr 2017 16:04:16 +0000
Subject: Re: [bitcoin-dev] Using a storage engine without UTXO-index
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Bitcoin Protocol Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Fri, 07 Apr 2017 16:02:37 -0000

This is a multi-part message in MIME format.

--_----------=_1491580955738300
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset="utf-8"

Thank you,



The benches are running in Google Cloud Engine; currently on 8 vCPU
32gb, but I tend to switch hardware regularly.


Roughly, the results are better for Bitcrust with high end hardware and
the difference for total block validations is mostly diminished at 2
vCPU, 7,5 gb.


Note that the spend-tree optimization primarily aims to improve peak
load order validation; when a block with pre-synced transactions comes
in, but this is tricky to accurately bench with Core using this simple
method of comparison by logs.


I will upgrade to, and show the results against 0.14 in the next weeks.


Best,

Tomas





On Fri, Apr 7, 2017, at 16:14, Greg Sanders wrote:

> Interesting work.

> 

> I was wondering if you could tellank  us what specs for the machine
> being used as preliminary benchmark is here:
> https://bitcrust.org/results ?
> 

> I'd be interested to also see comparisons with 0.14 which has some
> improvements for script validation with more cores.
> 

> On Fri, Apr 7, 2017 at 4:47 AM, Tomas via bitcoin-dev <bitcoin-
> dev@lists.linuxfoundation.org> wrote:
>> Thank you Marcos,

>> 

>>  Though written in Rust, bitcrust-db is definitely usable as
>>  pluggable
>>  module as its interface will be roughly some queries, add_tx and

>>  add_block with blobs and flags. (Bitcrust internally uses a

>>  deserialize-only model, keeping references to the blobs with the
>>  parsed
>>  data).

>> 

>>  However, from Core's side I believe network and storage are
>>  currently
>>  rather tightly coupled, which will make this far from trivial.

>> 

>>  Regardless, I am also hoping (with funding & a team) to build a
>>  Bitcrust
>>  networking component as well to bring a strong competitor to the
>>  market.
>> 

>>  best,

>>  Tomas

>> 

>> 

>> 

>> 

>> On Fri, Apr 7, 2017, at 09:55, Marcos mayorga wrote:

>>  > Hi Tomas,

>>  >

>>  > I've read it and think it is an excellent work, I'd like to see it
>>  > integrated into bitcoin-core as a 'kernel module'.

>>  >

>>  > I see there are a lot of proof of concepts out there, IMO
>>  > every one
>>  > deserve a room in the bitcoin client as a selectable feature, to
>>  > make the
>>  > software more flexible and less dictatorial, an user could easily
>>  > select
>>  > which features she wants to run.

>>  >

>>  > Best regards,

>>  > Marcos

>>  >

>>  > > I have been working on a bitcoin implementation that uses a
>>  > > different
>>  > > approach to indexing for verifying the order of transactions.
>>  > > Instead of
>>  > > using an index of unspent outputs, double spends are verified by
>>  > > using a
>>  > > spend-tree where spends are scanned against spent outputs
>>  > > instead of
>>  > > unspent outputs.

>>  > >

>>  > > This allows for much better concurrency, as not only blocks, but
>>  > > also
>>  > > individual inputs can be verified fully in parallel.

>>  > >

>>  > > I explain the approach at https://bitcrust.org, source code is
>>  > > available
>>  > > at https://github.com/tomasvdw/bitcrust

>>  > >

>>  > > I am sharing this not only to ask for your feedback, but also to
>>  > > call
>>  > > for a clear separation of protocol and implementations: As this
>>  > > solution, reversing the costs of outputs and inputs, seems to
>>  > > have
>>  > > excellent performance characteristics (as shown in the test
>>  > > results),
>>  > > updates to the protocol addressing the UTXO growth, might not be
>>  > > worth
>>  > > considering *protocol improvements* and it might be best to
>>  > > address
>>  > > these concerns as implementation details.

>>  > >

>>  > > Kind regards,

>>  > > Tomas van der Wansem

>>  > > tomas@bitcrust.org

>>  > > Bitcrust

>>  > > _______________________________________________

>>  > > bitcoin-dev mailing list

>>  > > bitcoin-dev@lists.linuxfoundation.org

>>  > > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

>>  > >

>>  >

>>  >

>>  _______________________________________________

>>  bitcoin-dev mailing list

>> bitcoin-dev@lists.linuxfoundation.org

>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev



--_----------=_1491580955738300
Content-Transfer-Encoding: 7bit
Content-Type: text/html; charset="utf-8"

<!DOCTYPE html>
<html>
<head>
<title></title>
</head>
<body><div>Thank you,<br></div>
<div><br></div>
<div>The benches are running in Google Cloud Engine; currently on 8 vCPU 32gb, but I tend to switch hardware regularly.<br></div>
<div><br></div>
<div>Roughly, the results are better for Bitcrust with high end hardware and the difference for total block validations is mostly diminished at <span class="">2 vCPU, 7,5 gb. </span><br></div>
<div><br></div>
<div><span class="">Note that the spend-tree optimization primarily aims to improve peak load order validation; when a block with pre-synced transactions comes in, but this is tricky to accurately bench with Core using this simple method of comparison by logs.</span><br></div>
<div><br></div>
<div><span class="">I will upgrade to, and show the results against 0.14 in the next weeks.</span><br></div>
<div><br></div>
<div><span class="">Best,</span><br></div>
<div><span class="">Tomas</span><br></div>
<div><br></div>
<div><br></div>
<div>On Fri, Apr 7, 2017, at 16:14, Greg Sanders wrote:<br></div>
<blockquote type="cite"><div dir="ltr"><div>Interesting work.<br></div>
<div><br></div>
<div>I was wondering if you could tellank&nbsp; us what specs for the machine being used as preliminary benchmark is here:&nbsp;<a href="https://bitcrust.org/results">https://bitcrust.org/results</a> ?<br></div>
<div><br></div>
<div>I'd be interested to also see comparisons with 0.14 which has some improvements for script validation with more cores.<br></div>
</div>
<div><div><br></div>
<div defang_data-gmailquote="yes"><div>On Fri, Apr 7, 2017 at 4:47 AM, Tomas via bitcoin-dev <span dir="ltr">&lt;<a href="mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.linuxfoundation.org</a>&gt;</span> wrote:<br></div>
<blockquote defang_data-gmailquote="yes" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204, 204, 204);padding-left:1ex;"><div>Thank you Marcos,<br></div>
<div> <br></div>
<div> Though written in Rust, bitcrust-db is definitely usable as pluggable<br></div>
<div> module as its interface will be roughly some queries, add_tx and<br></div>
<div> add_block with blobs and flags. (Bitcrust internally uses a<br></div>
<div> deserialize-only model, keeping references to the blobs with the parsed<br></div>
<div> data).<br></div>
<div> <br></div>
<div> However, from Core's side I believe network and storage are currently<br></div>
<div> rather tightly coupled, which will make this far from trivial.<br></div>
<div> <br></div>
<div> Regardless, I am also hoping (with funding &amp; a team) to build a Bitcrust<br></div>
<div> networking component as well to bring a strong competitor to the market.<br></div>
<div> <br></div>
<div> best,<br></div>
<div> Tomas<br></div>
<div> <br></div>
<div><div><div><br></div>
<div><br></div>
<div><br></div>
<div>On Fri, Apr 7, 2017, at 09:55, Marcos mayorga wrote:<br></div>
<div> &gt; Hi Tomas,<br></div>
<div> &gt;<br></div>
<div> &gt; I've read it and think it is an excellent work, I'd like to see it<br></div>
<div> &gt; integrated into bitcoin-core as a 'kernel module'.<br></div>
<div> &gt;<br></div>
<div> &gt; I see there are a lot of proof of concepts out there, IMO every one<br></div>
<div> &gt; deserve a room in the bitcoin client as a selectable feature, to make the<br></div>
<div> &gt; software more flexible and less dictatorial, an user could easily select<br></div>
<div> &gt; which features she wants to run.<br></div>
<div> &gt;<br></div>
<div> &gt; Best regards,<br></div>
<div> &gt; Marcos<br></div>
<div> &gt;<br></div>
<div> &gt; &gt; I have been working on a bitcoin implementation that uses a different<br></div>
<div> &gt; &gt; approach to indexing for verifying the order of transactions. Instead of<br></div>
<div> &gt; &gt; using an index of unspent outputs, double spends are verified by using a<br></div>
<div> &gt; &gt; spend-tree where spends are scanned against spent outputs instead of<br></div>
<div> &gt; &gt; unspent outputs.<br></div>
<div> &gt; &gt;<br></div>
<div> &gt; &gt; This allows for much better concurrency, as not only blocks, but also<br></div>
<div> &gt; &gt; individual inputs can be verified fully in parallel.<br></div>
<div> &gt; &gt;<br></div>
<div> &gt; &gt; I explain the approach at <a href="https://bitcrust.org">https://bitcrust.org</a>, source code is available<br></div>
<div> &gt; &gt; at <a href="https://github.com/tomasvdw/bitcrust">https://github.com/tomasvdw/<wbr>bitcrust</a><br></div>
<div> &gt; &gt;<br></div>
<div> &gt; &gt; I am sharing this not only to ask for your feedback, but also to call<br></div>
<div> &gt; &gt; for a clear separation of protocol and implementations: As this<br></div>
<div> &gt; &gt; solution, reversing the costs of outputs and inputs, seems to have<br></div>
<div> &gt; &gt; excellent performance characteristics (as shown in the test results),<br></div>
<div> &gt; &gt; updates to the protocol addressing the UTXO growth, might not be worth<br></div>
<div> &gt; &gt; considering *protocol improvements* and it might be best to address<br></div>
<div> &gt; &gt; these concerns as implementation details.<br></div>
<div> &gt; &gt;<br></div>
<div> &gt; &gt; Kind regards,<br></div>
<div> &gt; &gt; Tomas van der Wansem<br></div>
<div> &gt; &gt; <a href="mailto:tomas@bitcrust.org">tomas@bitcrust.org</a><br></div>
<div> &gt; &gt; Bitcrust<br></div>
<div> &gt; &gt; ______________________________<wbr>_________________<br></div>
<div> &gt; &gt; bitcoin-dev mailing list<br></div>
<div> &gt; &gt; <a href="mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.<wbr>linuxfoundation.org</a><br></div>
<div> &gt; &gt; <a href="https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev">https://lists.linuxfoundation.<wbr>org/mailman/listinfo/bitcoin-<wbr>dev</a><br></div>
<div> &gt; &gt;<br></div>
<div> &gt;<br></div>
<div> &gt;<br></div>
<div> ______________________________<wbr>_________________<br></div>
<div> bitcoin-dev mailing list<br></div>
<div> <a href="mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.<wbr>linuxfoundation.org</a><br></div>
<div> <a href="https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev">https://lists.linuxfoundation.<wbr>org/mailman/listinfo/bitcoin-<wbr>dev</a><br></div>
</div>
</div>
</blockquote></div>
</div>
</blockquote><div><br></div>
</body>
</html>

--_----------=_1491580955738300--