path: root/1b/b65c18ccf220c0488456aee7dfde87211e761f

Return-Path: <james.obeirne@gmail.com>
Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org
	[172.17.192.35])
	by mail.linuxfoundation.org (Postfix) with ESMTPS id 789F822
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Tue, 23 Apr 2019 14:17:20 +0000 (UTC)
X-Greylist: whitelisted by SQLgrey-1.7.6
Received: from mail-vs1-f51.google.com (mail-vs1-f51.google.com
	[209.85.217.51])
	by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 456E1829
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Tue, 23 Apr 2019 14:17:19 +0000 (UTC)
Received: by mail-vs1-f51.google.com with SMTP id s2so8339289vsi.5
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Tue, 23 Apr 2019 07:17:19 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
	h=mime-version:references:in-reply-to:from:date:message-id:subject:to; 
	bh=zyQdGB5GRSHUCyBcXjkxem4CzzQK41U9vFVRN6CwZYg=;
	b=XLik7HbydLcekmiTgzaSkR1qwR/7UBtsE4Zv6tot1GUN1MBFEMqVrpEl+WJo03Upw8
	VDteZxRPZvB4rFfPpML9RdCi5Bm/rqEiaoaUaJ+/kjBV+dRJrNW1glDENN8zVUsouxN4
	dDgxptettLwUt0QM4oo7CmnsHFRGBFMHp8L7gGmM7uliv1wMWHUmgqxtNonNNqaCnX2W
	fNZMv11uMuG6Q1eQIhVvDV1xm9512GdA+7RlVRN4e6Q+kNCpD87FvMhxAgtV8Xc98yDr
	pPccJnq3NTWGt2y4x/d9UtJq9khSpwfaB2S7rAUAtCiIYmZAXDvytsuBZ8dXBqzCTef1
	cuPA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
	d=1e100.net; s=20161025;
	h=x-gm-message-state:mime-version:references:in-reply-to:from:date
	:message-id:subject:to;
	bh=zyQdGB5GRSHUCyBcXjkxem4CzzQK41U9vFVRN6CwZYg=;
	b=I886gQd0w7moMOZqDc15vKLOD0S9axEKQlDdGkt2yH0PMNrLhOqxfWP5nbTFcx7Smm
	sFWmoHqcOyvdGj3+Q7PjQFuBbKaDV4YYsgvq6zOzPI6nDkywSBwmeauf+X9NEFqP7MhB
	DTRE4YSWVwRcCtHW9XDhj/z02n2lbU2SLIko83pWZDkOIcNWVMQpZgcbG0DS5Fkn/2pj
	Y515kpFThjQpC7S5RO19O35C7S9KgI/g3d5QTNuIIKaRI8votmLV1sC7SuktSHqwhePv
	ceDzqaeWZHczp0iQs9cWxLW2oR6PyppXXnMLN+V2K5E5xhADmRF5lEL0bHKm6IxwHomg
	ZW0Q==
X-Gm-Message-State: APjAAAXcYvxAKc4pakN9A6UJjDWm5dkwqvoFieDS+xV9abbjE04y2zGw
	dWn5HzVsAAICweRxxMWDd/0KSyymaS8fyP3UL93BuEwlW28=
X-Google-Smtp-Source: APXvYqza7WzcRwp9lfuoTugJIK9kLMD1DAsKNcC/w5Z7h3FpNq4jDF34nb/Bah2VWfNo3yFS196ZuB0McB28KuIW3FU=
X-Received: by 2002:a67:ee04:: with SMTP id f4mr13905867vsp.34.1556029037656; 
	Tue, 23 Apr 2019 07:17:17 -0700 (PDT)
MIME-Version: 1.0
References: <CAPfvXf+JS6ZhXUieWVxiaNa4uhhWwafCk3odMKy5F_yi=XwngA@mail.gmail.com>
In-Reply-To: <CAPfvXf+JS6ZhXUieWVxiaNa4uhhWwafCk3odMKy5F_yi=XwngA@mail.gmail.com>
From: "James O'Beirne" <james.obeirne@gmail.com>
Date: Tue, 23 Apr 2019 10:17:06 -0400
Message-ID: <CAPfvXfJkCqSpB9r-nF0vNh-GpP1RAhaEzxDJ0jit3JkUEeJtog@mail.gmail.com>
To: Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org>
Content-Type: multipart/alternative; boundary="0000000000000cbe160587333ede"
X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, HTML_MESSAGE,
	RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
	smtp1.linux-foundation.org
X-Mailman-Approved-At: Tue, 23 Apr 2019 14:47:04 +0000
Subject: Re: [bitcoin-dev] assumeutxo and UTXO snapshots
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Bitcoin Protocol Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Tue, 23 Apr 2019 14:17:20 -0000

--0000000000000cbe160587333ede
Content-Type: text/plain; charset="UTF-8"

Good morning all,

Over the past weeks I've had a number of conversations with a few frequent
contributors about this idea. I've condensed these discussions into a
proposal document which you can view here:
https://github.com/jamesob/assumeutxo-docs/tree/2019-04-proposal/proposal

The document is structured as an FAQ, and so hopefully it addresses some of
the common questions that would come up in this thread. If you'd like to
comment, there's an associated pull request here:
https://github.com/jamesob/assumeutxo-docs/pull/1

Regards,
James


On Tue, Apr 2, 2019 at 4:43 PM James O'Beirne <james.obeirne@gmail.com>
wrote:

> Hi,
>
> I'd like to discuss assumeutxo, which is an appealing and simple
> optimization in the spirit of assumevalid[0].
>
> # Motivation
>
> To start a fully validating bitcoin client from scratch, that client
> currently
> needs to perform an initial block download. To the surprise of no one, IBD
> takes a linear amount time based on the length of the chain's history. For
> clients running on modest hardware under limited bandwidth constraints,
> say a mobile device, completing IBD takes a considerable amount of time
> and thus poses serious usability challenges.
>
> As a result, having fully validating clients run on such hardware is rare
> and
> basically unrealistic. Clients with even moderate resource constraints
> are encouraged to rely on the SPV trust model. Though we have promising
> improvements to existing SPV modes pending deployment[1], it's worth
> thinking about a mechanism that would allow such clients to use trust
> models closer to full validation.
>
> The subject of this mail is a proposal for a complementary alternative to
> SPV
> modes, and which is in the spirit of an existing default, `assumevalid`.
> It may
> help modest clients transact under a security model that closely resembles
> full validation within minutes instead of hours or days.
>
> # assumeutxo
>
> The basic idea is to allow nodes to initialize using a serialized version
> of the
> UTXO set rendered by another node at some predetermined height. The
> initializing node syncs the headers chain from the network, then obtains
> and
> loads one of these UTXO snapshots (i.e. a serialized version of the UTXO
> set
> bundled with the block header indicating its "base" and some other
> metadata).
>
> Based upon the snapshot, the node is able to quickly reconstruct its
> chainstate,
> and compares a hash of the resulting UTXO set to a preordained hash
> hard-coded
> in the software a la assumevalid. This all takes ~23 minutes, not
> accounting for
> download of the 3.2GB snapshot[2].
>
> The node then syncs to the network tip and afterwards begins a simultaneous
> background validation (i.e., a conventional IBD) up to the base height of
> the
> snapshot in order to achieve full validation. Crucially, even while the
> background validation is happening the node can validate incoming blocks
> and
> transact with the benefit of the full (assumed-valid) UTXO set.
>
> Snapshots could be obtained from multiple separate peers in the same
> manner as
> block download, but I haven't put much thought into this. In concept it
> doesn't
> matter too much where the snapshots come from since their validity is
> determined via content hash.
>
> # Security
>
> Obviously there are some security implications due consideration. While
> this
> proposal is in the spirit of assumevalid, practical attacks may become
> easier.
> Under assumevalid, a user can be tricked into transacting under a false
> history
> if an attacker convinces them to start bitcoind with a malicious
> `-assumevalid`
> parameter, sybils their node, and then feeds them a bogus chain
> encompassing
> all of the hard-coded checkpoints[3].
>
> The same attack is made easier in assumeutxo because, unlike in
> assumevalid,
> the attacker need not construct a valid PoW chain to get the victim's node
> into
> a false state; they simply need to get the user to accept a bad
> `-assumeutxo`
> parameter and then supply them an easily made UTXO snapshot containing,
> say, a
> false coin assignment.
>
> For this reason, I recommend that if we were to implement assumeutxo, we
> not
> allow its specification via commandline argument[4].
>
> Beyond this risk, I can't think of material differences in security
> relative to
> assumevalid, though I appeal to the list for help with this.
>
> # More fully validating clients
>
> A particularly exciting use-case for assumeutxo is the possibility of
> mobile
> devices functioning as fully validating nodes with access to the complete
> UTXO
> set (as an alternative to SPV models). The total resource burden needed to
> start a node
> from scratch based on a snapshot is, at time of writing, a ~(3.2GB
> + blocks_to_tip * 4MB) download and a few minutes of processing time,
> which sounds
> manageable for many mobile devices currently in use.
>
> A mobile user could initialize an assumed-valid bitcoin node within an
> hour,
> transact immediately, and complete a pruned full validation of their
> assumed-valid chain over the next few days, perhaps only doing the
> background
> IBD when their device has access to suitable high-bandwidth connections.
>
> If we end up implementing an accumulator-based UTXO scaling design[5][6]
> down
> the road, it's easy to imagine an analogous process that would allow very
> fast
> startup using an accumulator of a few kilobytes in lieu of a multi-GB
> snapshot.
>
> ---
>
> I've created a related issue at our Github repository here:
>   https://github.com/bitcoin/bitcoin/issues/15605
>
> and have submitted a draft implementation of snapshot usage via RPC here:
>   https://github.com/bitcoin/bitcoin/pull/15606
>
> I'd like to discuss here whether this is a good fit for Bitcoin
> conceptually. Concrete
> plans for deployment steps should be discussed in the Github issue, and
> after all
> that my implementation may be reviewed as a sketch of the specific software
> changes necessary.
>
> Regards,
> James
>
>
> [0]:
> https://bitcoincore.org/en/2017/03/08/release-0.14.0/#assumed-valid-blocks
> [1]: https://github.com/bitcoin/bips/blob/master/bip-0157.mediawiki
> [2]: as tested at height 569895, on a 12 core Intel Xeon Silver 4116 CPU @
> 2.10GHz
> [3]:
> https://github.com/bitcoin/bitcoin/blob/84d0fdc/src/chainparams.cpp#L145-L161
> [4]: Marco Falke is due credit for this point
> [5]: utreexo: https://www.youtube.com/watch?v=edRun-6ubCc
> [6]: Boneh, Bunz, Fisch on accumulators: https://eprint.iacr.org/2018/1188
>
>

--0000000000000cbe160587333ede
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr"><div dir=3D"ltr"><div dir=3D"ltr">Good mo=
rning all,<div><br></div><div>Over the past weeks I&#39;ve had a number of =
conversations with a few frequent contributors about this idea. I&#39;ve co=
ndensed these discussions into a proposal document which you can view here:=
=C2=A0<a href=3D"https://github.com/jamesob/assumeutxo-docs/tree/2019-04-pr=
oposal/proposal">https://github.com/jamesob/assumeutxo-docs/tree/2019-04-pr=
oposal/proposal</a></div><div><br></div><div>The document is structured as =
an FAQ, and so hopefully it addresses some of the common questions that wou=
ld come up in this thread. If you&#39;d like to comment, there&#39;s an ass=
ociated pull request here:=C2=A0<a href=3D"https://github.com/jamesob/assum=
eutxo-docs/pull/1">https://github.com/jamesob/assumeutxo-docs/pull/1</a></d=
iv><div><br></div><div>Regards,</div><div>James</div><div><br></div></div><=
/div></div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"g=
mail_attr">On Tue, Apr 2, 2019 at 4:43 PM James O&#39;Beirne &lt;<a href=3D=
"mailto:james.obeirne@gmail.com">james.obeirne@gmail.com</a>&gt; wrote:<br>=
</div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;b=
order-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr"><d=
iv dir=3D"ltr"><div>Hi,<br></div><div><br></div><div>I&#39;d like to discus=
s assumeutxo, which is an appealing and simple=C2=A0</div><div>optimization=
 in the spirit of assumevalid[0].</div><div><br></div><div># Motivation</di=
v><div><br></div><div>To start a fully validating bitcoin client from scrat=
ch, that client currently</div><div>needs to perform an initial block downl=
oad. To the surprise of no one, IBD=C2=A0</div><div>takes a linear amount t=
ime based on the length of the chain&#39;s history. For=C2=A0</div><div>cli=
ents running on modest hardware under limited bandwidth constraints,=C2=A0<=
/div><div>say a mobile device, completing IBD takes a considerable amount o=
f time=C2=A0</div><div>and thus poses serious usability challenges.</div><d=
iv><br></div><div>As a result, having fully validating clients run on such =
hardware is rare and</div><div>basically unrealistic. Clients with even mod=
erate resource constraints</div><div>are encouraged to rely on the SPV trus=
t model. Though we have promising</div><div>improvements to existing SPV mo=
des pending deployment[1], it&#39;s worth</div><div>thinking about a mechan=
ism that would allow such clients to use trust</div><div>models closer to f=
ull validation.</div><div><br></div><div>The subject of this mail is a prop=
osal for a complementary alternative to SPV</div><div>modes, and which is i=
n the spirit of an existing default, `assumevalid`. It may</div><div>help m=
odest clients transact under a security model that closely resembles</div><=
div>full validation within minutes instead of hours or days.</div><div><br>=
</div><div># assumeutxo</div><div><br></div><div>The basic idea is to allow=
 nodes to initialize using a serialized version of the</div><div>UTXO set r=
endered by another node at some predetermined height. The</div><div>initial=
izing node syncs the headers chain from the network, then obtains and</div>=
<div>loads one of these UTXO snapshots (i.e. a serialized version of the UT=
XO set</div><div>bundled with the block header indicating its &quot;base&qu=
ot; and some other metadata).</div><div><br></div><div>Based upon the snaps=
hot, the node is able to quickly reconstruct its chainstate,</div><div>and =
compares a hash of the resulting UTXO set to a preordained hash hard-coded<=
/div><div>in the software a la assumevalid. This all takes ~23 minutes, not=
 accounting for</div><div>download of the 3.2GB snapshot[2].=C2=A0</div><di=
v><br></div><div>The node then syncs to the network tip and afterwards begi=
ns a simultaneous</div><div>background validation (i.e., a conventional IBD=
) up to the base height of the</div><div>snapshot in order to achieve full =
validation. Crucially, even while the</div><div>background validation is ha=
ppening the node can validate incoming blocks and</div><div>transact with t=
he benefit of the full (assumed-valid) UTXO set.</div><div><br></div><div>S=
napshots could be obtained from multiple separate peers in the same manner =
as</div><div>block download, but I haven&#39;t put much thought into this. =
In concept it doesn&#39;t</div><div>matter too much where the snapshots com=
e from since their validity is</div><div>determined via content hash.</div>=
<div><br></div><div># Security</div><div><br></div><div>Obviously there are=
 some security implications due consideration. While this</div><div>proposa=
l is in the spirit of assumevalid, practical attacks may become easier.</di=
v><div>Under assumevalid, a user can be tricked into transacting under a fa=
lse history</div><div>if an attacker convinces them to start bitcoind with =
a malicious `-assumevalid`</div><div>parameter, sybils their node, and then=
 feeds them a bogus chain encompassing</div><div>all of the hard-coded chec=
kpoints[3].=C2=A0</div><div><br></div><div>The same attack is made easier i=
n assumeutxo because, unlike in assumevalid,</div><div>the attacker need no=
t construct a valid PoW chain to get the victim&#39;s node into</div><div>a=
 false state; they simply need to get the user to accept a bad `-assumeutxo=
`</div><div>parameter and then supply them an easily made UTXO snapshot con=
taining, say, a</div><div>false coin assignment.</div><div><br></div><div>F=
or this reason, I recommend that if we were to implement assumeutxo, we not=
</div><div>allow its specification via commandline argument[4].</div><div><=
br></div><div>Beyond this risk, I can&#39;t think of material differences i=
n security relative to</div><div>assumevalid, though I appeal to the list f=
or help with this.</div><div><br></div><div># More fully validating clients=
</div><div><br></div><div>A particularly exciting use-case for assumeutxo i=
s the possibility of mobile</div><div>devices functioning as fully validati=
ng nodes with access to the complete UTXO</div><div>set (as an alternative =
to SPV models). The total resource burden needed to start a node</div><div>=
from scratch based on a snapshot is, at time of writing, a ~(3.2GB</div><di=
v>+ blocks_to_tip * 4MB) download and a few minutes of processing time, whi=
ch sounds</div><div>manageable for many mobile devices currently in use.</d=
iv><div>=C2=A0=C2=A0</div><div>A mobile user could initialize an assumed-va=
lid bitcoin node within an hour,</div><div>transact immediately, and comple=
te a pruned full validation of their</div><div>assumed-valid chain over the=
 next few days, perhaps only doing the background</div><div>IBD when their =
device has access to suitable high-bandwidth connections.</div><div><br></d=
iv><div>If we end up implementing an accumulator-based UTXO scaling design[=
5][6] down</div><div>the road, it&#39;s easy to imagine an analogous proces=
s that would allow very fast</div><div>startup using an accumulator of a fe=
w kilobytes in lieu of a multi-GB snapshot.</div><div><br></div><div>---</d=
iv><div><br></div><div>I&#39;ve created a related issue at our Github repos=
itory here:</div><div>=C2=A0 <a href=3D"https://github.com/bitcoin/bitcoin/=
issues/15605" target=3D"_blank">https://github.com/bitcoin/bitcoin/issues/1=
5605</a></div><div><br></div><div>and have submitted a draft implementation=
 of snapshot usage via RPC here:</div><div>=C2=A0 <a href=3D"https://github=
.com/bitcoin/bitcoin/pull/15606" target=3D"_blank">https://github.com/bitco=
in/bitcoin/pull/15606</a></div><div><br></div><div>I&#39;d like to discuss =
here whether this is a good fit for Bitcoin conceptually. Concrete</div><di=
v>plans for deployment steps should be discussed in the Github issue, and a=
fter all=C2=A0</div><div>that my implementation may be reviewed as a sketch=
 of the specific software</div><div>changes necessary.</div><div><br></div>=
<div>Regards,</div><div>James</div><div><br></div><div><br></div><div>[0]: =
<a href=3D"https://bitcoincore.org/en/2017/03/08/release-0.14.0/#assumed-va=
lid-blocks" target=3D"_blank">https://bitcoincore.org/en/2017/03/08/release=
-0.14.0/#assumed-valid-blocks</a></div><div>[1]: <a href=3D"https://github.=
com/bitcoin/bips/blob/master/bip-0157.mediawiki" target=3D"_blank">https://=
github.com/bitcoin/bips/blob/master/bip-0157.mediawiki</a></div><div>[2]: a=
s tested at height 569895, on a 12 core Intel Xeon Silver 4116 CPU @ 2.10GH=
z</div><div>[3]: <a href=3D"https://github.com/bitcoin/bitcoin/blob/84d0fdc=
/src/chainparams.cpp#L145-L161" target=3D"_blank">https://github.com/bitcoi=
n/bitcoin/blob/84d0fdc/src/chainparams.cpp#L145-L161</a></div><div>[4]: Mar=
co Falke is due credit for this point</div><div>[5]: utreexo: <a href=3D"ht=
tps://www.youtube.com/watch?v=3DedRun-6ubCc" target=3D"_blank">https://www.=
youtube.com/watch?v=3DedRun-6ubCc</a></div><div>[6]: Boneh, Bunz, Fisch on =
accumulators: <a href=3D"https://eprint.iacr.org/2018/1188" target=3D"_blan=
k">https://eprint.iacr.org/2018/1188</a></div><div><br></div></div></div>
</blockquote></div>

--0000000000000cbe160587333ede--