path: root/5c/b27fd8dd8848af6f979a394a2f3085f780b085

Return-Path: <jan.matejek@satoshilabs.com>
Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org
	[172.17.192.35])
	by mail.linuxfoundation.org (Postfix) with ESMTPS id DE991CCC
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Tue, 26 Jun 2018 15:33:19 +0000 (UTC)
X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6
Received: from mail.sldev.cz (mail.sldev.cz [88.208.115.66])
	by smtp1.linuxfoundation.org (Postfix) with ESMTPS id EC37A17E
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Tue, 26 Jun 2018 15:33:18 +0000 (UTC)
Received: from localhost (localhost [127.0.0.1])
	by mail.sldev.cz (Postfix) with ESMTP id D247FE1046;
	Tue, 26 Jun 2018 15:33:16 +0000 (UTC)
X-Virus-Scanned: Debian amavisd-new at mail.sldev.cz
Received: from mail.sldev.cz ([127.0.0.1])
	by localhost (mail.sldev.cz [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id OyN6STrVwhTC; Tue, 26 Jun 2018 15:33:15 +0000 (UTC)
Received: from [10.8.0.37] (unknown [10.8.0.37])
	by mail.sldev.cz (Postfix) with ESMTPSA id 22E3AE1040;
	Tue, 26 Jun 2018 15:33:15 +0000 (UTC)
To: bitcoin-dev@lists.linuxfoundation.org
References: <CAPg+sBhGMxXatsyCAqeboQKH8ASSFAfiXzxyXR9UrNFnah5PPw@mail.gmail.com>
	<CHCiA27GTRiVfkF1DoHdroJL1rQS77ocB42nWxIIhqi_fY3VbB3jsMQveRJOtsJiA4RaCAVe3VZmLZsXVYS3A5wVLNP2OgKQiHE0T27P2qc=@achow101.com>
	<21a616f5-7a17-35b9-85ea-f779f20a6a2d@satoshilabs.com>
	<20180621195654.GC99379@coinkite.com>
	<CAPg+sBgdQqZ8sRSn=dd9EkavYJA6GBiCu6-v5k9ca-9WLPp72Q@mail.gmail.com>
	<ljk5Z_a3KK6DHfmPJxI8o9W2CkwszkUG34h0i1MTGU4ss8r3BTQ3GnTtDTfWF6J7ZqcSAmejzrr11muWqYN-_wnWw_0NFn5_lggNnjI0_Rc=@achow101.com>
	<f8f5b1e3-692a-fc1e-2ad3-c4ad4464957f@satoshilabs.com>
	<TGyS7Azu3inMQFv9QFn8USr9v2m5QbhDRmiOI-4FWwscUeuIB9rA7mCmZA4-kwCJOMAx92fO7XICHtE7ES_QmIYLDy6RHof1WLALskGUYAc=@achow101.com>
From: matejcik <jan.matejek@satoshilabs.com>
Openpgp: preference=signencrypt
Autocrypt: addr=jan.matejek@satoshilabs.com; prefer-encrypt=mutual; keydata=
	xsFNBFqFmMgBEADPJ8NULpuu0nwox/tIfo+slGfcXZLUEZstNoaY9QgNuILJRtoJ6xZy8rQf
	S7iQlkaZcrpMJYdZtkRHvndkceBxesCG8io6tsU+t2SK6AvaW0FG95a9shFM/U9/JVO/QmBi
	IuQzbiE2XTZ/JStyEp4zpuyJqX1o9gzS/4MBXwj7Rzk8u+fHI28h96HILC2a0mC+c2gJ7f5t
	o/w+vxFZmk06COK08W5+odb9I8mjs0uf7jgTUEFrfwi6oCoTFmSon7cOy/WTieClwF/vUKuJ
	DBAtsMh2rxh8IHyH8xpR+Ay/K6jUWqeb3P2csQqMXmquYG/qdaHjQgxyuoJFbn+nT6jNGVQZ
	MjpZkMrGnjLccecaXlgx/rZK6ElCZ1PDHKOTW7A1YY1/eG7TWYnVv1ehQLueAoqyyfiEutsK
	E5jGbR0AmNjCahpeK7dxj+8g8TXpVsH207xJ+mqOm5RYqlX4OzfVvcnoHhlRIOu85i4I9rWm
	1u/pP6uJFnBCKtuhhbmXCxM6wF7W5U6EVW3yymsPmSoVoaR024vffE3L5jZSsDMRxY6fDXNm
	ljRnOpT3l3d+kMVdAM3CdDCgmV87fdo4PAaGDfnmufGue/Gp0RiLCe/Wsm4DgIIa5UK6DmzD
	q0B6i9y/GJSPUChzZ8y7fYzuyXdpk/13gV2NRsskg9oXJVd1vQARAQABzSZtYXRlamNpayA8
	amFuLm1hdGVqZWtAc2F0b3NoaWxhYnMuY29tPsLBfQQTAQgAJwUCWoWYyAIbIwUJCWYBgAUL
	CQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRDGf7EG5O0XHoU0D/4+fTbt4KELEtnpkirDH4mQ
	Vt3KtKJrI/gp/3u+r6jUWMv2V9iRFMs09GAVBmE2DkXXIlfaT1P0QfwVSpHC4k5lwKwSCSyS
	MUgBbQGPOiYMCgMQ+in4vjlqWWcx6jjlgxQctQHRrVG5jyi7BSb0jwG8rcYtx8SAYkN4joG/
	oy2zMbq6qu+Vsl+xR5WwWF2mcUUyiVo7dSwNy+1PaeygOR9xAWkM8J42ckLfJgvyLSviBKnU
	9rgg94ryEDAMNUL5yJUygQmUM/jdpyBpBycRbWMB+zIYDPVGnFj4vN8Hs9DyGUHVb2OqSW+q
	VPxD7U9m9z6J3NnY9HpaFX1DD8leK3TebpyYaeODY5jyk7retuLrMq+W4kJU0290xzlWa9sU
	wa7lTWw63pelfPUKZ+mjhSFQSZBqiuNv67CBd/UmoqMWSDrCWj+3IFQxReFbh47Wl4MUX2cK
	cLocYkBzDck7hH4YfK6jJ++teN6RKXr7P1y6EI25WEfJxWK9say7x/FRkNW0s98MxtOuwEsm
	/vHqHQQanAT4R5l+Rr7XfU7fpmH0As98qD81lc3RHbrxEXgA0ks2VuCxBWsPpzaHUFPOcE9H
	hsg1jSEDi/Mo6D4e2ap7FYXDgZiKye9WnSdPlVBqJxqinDDgSBv5wzKaEGQS0MKrF9myS7d0
	pBSy1Dr6IWOegM7BTQRahZjIARAAwwT6h4IFvs/hmY9KHiX/GIbvybQUU71ZWYRE2KKo5E2c
	ZXBJj7SiDtU80bS+NCSeF2c0i4xOYgZlIYMqlgS8k1zfdBt/JHmG3tm1JgohVj+pm42RfBAF
	d0y05zz5wysQOw1M4WlWKZH0ameM+0/AGqspeZushWay8Q4yx1dO/6MeyPy/NwE/MKEsCOPV
	aN28DndN3iKOyriCQt/IhG/n6ORPRGyei3JYqxsnpW36BOmSPWJ7Qj2pFw53p5coPOEDL8mN
	Ique0LJZ3zVFVMa4i7HtqIEnYO+ZnKx2G8aLsHEir2pzBv6tMwlgETcUTVfK1ePN7OzhYy4q
	a38hMWzk0db2V+gOlAu6SuAi1ANkcPhCPUWxPIvXiNdd9iwe5gOzFy0FoZxj22rFwgUX8wcc
	cfWStgoE1MGE9G5zrqc01R0x7by8BOFkImAwTyJ9vq4jG+w7Npky3PhoHPgCT5knV7Q91U2I
	TqPOQBcMda0B+4LOaElb1sXqe44dHwcg4dMVngaea5xL7winSqU2Gtm6pqFAGut5F7JiYhPb
	dGUHJPMS67ONkKe5ARu/Z/r9XoFe2TxpkvNJ/+QJQ3PCiJ6ya31ij6HOIfFbZr3xlTyU/DvG
	SejIvDK/SnJMw+/x60bYAshYBp0uQgih1ugtoZh7cnKj3KfhlpXT0mL8rsl1QHsAEQEAAcLB
	ZQQYAQgADwUCWoWYyAIbDAUJCWYBgAAKCRDGf7EG5O0XHs2xD/92sa5L6gafP/rRKfo9u3/w
	s+7E/kKPgG4VGDeirLo8hbinCjPr0cfZ7OgDDvp0zy6lTdZc2tcHsEbiPqblzaSZimV5Y3EQ
	eIzz0UhY6YdDELr8pvdnB8qnOJHXgWmZTRYkRgxFOWI3v4STmOYZQ7MFv0kHBfV3htCjYTHS
	Qx2jQO4CTbcSEbkVwNv56OiZroabrHRf0WUSyzElf13P/MRFjUJFYYZDqc0iOWUh4QeXbFiY
	fLYpOCtm0nqaDdG1VD4jMpKq1FKBvTw4id1i7pONENd4BB7ytnDvKGdVI6oDnGUBsc5VUrEa
	h1PbbshNMbRtFigeMe8998jWhK4jQzeuDr0FSBlhxbluGfyMUgk7s6aBC9BOsdDkgtJk1Fd/
	j9sWOj8Pxzc4lMQRfygm+QxxLdqa36Qh3oK+jfK7362CXlqBfb9ryerjfFGY4VqMBzQ+BFtj
	lYZSdVzGWlmLD9D88wzeByIZMScQPvrXSFwPO2/TuOQNCo0VHcgHpNFzeMRK2eT8bhry+dlq
	U+0Kxy2gQijw9j/EZlqR3w053EwUrfAAmHHeYPimXK4pc8oSw0s1A6hQO7Vc0SgblF8taFTM
	UhRR7xZg+l5vybAgrDYVL75b9CDscZqd7WVmZx+xU23sUG6SaxXI7PV6bPuMug0fD3SAsieu
	+vypQ3jCcUKGrA==
Message-ID: <c32dc90d-9919-354b-932c-f93fe329760b@satoshilabs.com>
Date: Tue, 26 Jun 2018 17:33:14 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
	Thunderbird/52.8.0
MIME-Version: 1.0
In-Reply-To: <TGyS7Azu3inMQFv9QFn8USr9v2m5QbhDRmiOI-4FWwscUeuIB9rA7mCmZA4-kwCJOMAx92fO7XICHtE7ES_QmIYLDy6RHof1WLALskGUYAc=@achow101.com>
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature";
	boundary="74aBNEZiIFs9NuoqOguzkKgVgn9KqMPKP"
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=ham
	version=3.3.1
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
	smtp1.linux-foundation.org
X-Mailman-Approved-At: Tue, 26 Jun 2018 15:36:54 +0000
Subject: Re: [bitcoin-dev] BIP 174 thoughts
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Bitcoin Protocol Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Tue, 26 Jun 2018 15:33:20 -0000

This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--74aBNEZiIFs9NuoqOguzkKgVgn9KqMPKP
Content-Type: multipart/mixed; boundary="HN3AdUEsOchTlvdQeDk1FNoAuphj1qawm";
 protected-headers="v1"
From: matejcik <jan.matejek@satoshilabs.com>
To: bitcoin-dev@lists.linuxfoundation.org
Cc: tomas.susanka@satoshilabs.com
Message-ID: <c32dc90d-9919-354b-932c-f93fe329760b@satoshilabs.com>
Subject: Re: [bitcoin-dev] BIP 174 thoughts
References: <CAPg+sBhGMxXatsyCAqeboQKH8ASSFAfiXzxyXR9UrNFnah5PPw@mail.gmail.com>
 <CHCiA27GTRiVfkF1DoHdroJL1rQS77ocB42nWxIIhqi_fY3VbB3jsMQveRJOtsJiA4RaCAVe3VZmLZsXVYS3A5wVLNP2OgKQiHE0T27P2qc=@achow101.com>
 <21a616f5-7a17-35b9-85ea-f779f20a6a2d@satoshilabs.com>
 <20180621195654.GC99379@coinkite.com>
 <CAPg+sBgdQqZ8sRSn=dd9EkavYJA6GBiCu6-v5k9ca-9WLPp72Q@mail.gmail.com>
 <ljk5Z_a3KK6DHfmPJxI8o9W2CkwszkUG34h0i1MTGU4ss8r3BTQ3GnTtDTfWF6J7ZqcSAmejzrr11muWqYN-_wnWw_0NFn5_lggNnjI0_Rc=@achow101.com>
 <f8f5b1e3-692a-fc1e-2ad3-c4ad4464957f@satoshilabs.com>
 <TGyS7Azu3inMQFv9QFn8USr9v2m5QbhDRmiOI-4FWwscUeuIB9rA7mCmZA4-kwCJOMAx92fO7XICHtE7ES_QmIYLDy6RHof1WLALskGUYAc=@achow101.com>
In-Reply-To: <TGyS7Azu3inMQFv9QFn8USr9v2m5QbhDRmiOI-4FWwscUeuIB9rA7mCmZA4-kwCJOMAx92fO7XICHtE7ES_QmIYLDy6RHof1WLALskGUYAc=@achow101.com>

--HN3AdUEsOchTlvdQeDk1FNoAuphj1qawm
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable

hello,

in general, I agree with my colleague Tomas, the proposed changes are
good and achieve the most important things that we wanted. We'll review
the proposal in more detail later.

For now a couple minor things I have run into:

- valid test vector 2 ("one P2PKH input and one P2SH-P2WPKH input")
seems broken, at least its hex version; a delimiter seems to be missing,
misplaced or corrupted
- at least the first signing vector is not updated, but you probably
know that
- BIP32 derivation fields don't specify size of the "master public key",
which would make it un-parsable :)
- "Transaction format is specified as follows" and its table need to be
updated.


I'm still going to argue against the key-value model though.

It's true that this is not significant in terms of space. But I'm more
concerned about human readability, i.e., confusing future implementers.
At this point, the key-value model is there "for historical reasons",
except these aren't valid even before finalizing the format. The
original rationale for using key-values seems to be gone (no key-based
lookups are necessary). As for combining and deduplication, whether key
data is present or not is now purely a stand-in for a "repeatable" flag.
We could just as easily say, e.g., that the high bit of "type" specifies
whether this record can be repeated.

(Moreover, as I wrote previously, the Combiner seems like a weirdly
placed role. I still don't see its significance and why is it important
to correctly combine PSBTs by agents that don't understand them. If you
have a usecase in mind, please explain.
ISTM a Combiner could just as well combine based on whole-record
uniqueness, and leave the duplicate detection to the Finalizer. In case
the incoming PSBTs have incompatible unique fields, the Combiner would
have to fail anyway, so the Finalizer might as well do it. Perhaps it
would be good to leave out the Combiner role entirely?)

There's two remaining types where key data is used: BIP32 derivations
and partial signatures. In case of BIP32 derivation, the key data is
redundant ( pubkey =3D derive(value) ), so I'd argue we should leave that=

out and save space. In case of partial signatures, it's simple enough to
make the pubkey part of the value.

Re breaking change, we are proposing breaking changes anyway, existing
code *will* need to be touched, and given that this is a hand-parsed
format, changing `parse_keyvalue` to `parse_record` seems like a small
matter?

---

At this point I'm obliged to again argue for using protobuf.

Thing is: BIP174 *is basically protobuf* (v2) as it stands. If I'm
succesful in convincing you to switch to a record set model, it's going
to be "protobuf with different varint".

I mean this very seriously: (the relevant subset of) protobuf is a set
of records in the following format:
<record type><varint field length><field data>
Record types can repeat, the schema specifies whether a field is
repeatable or not - if it's not, the last parsed value is used.

BIP174 is a ad-hoc format, simple to parse by hand; but that results in
_having to_ parse it by hand. In contrast, protobuf has a huge
collection of implementations that will do the job of sorting record
types into relevant struct fields, proper delimiting of records, etc.
=2E..while at the same time, implementing "protobuf-based-BIP174" by hand=

is roughly equally difficult as implementing the current BIP174.

N.B., it's possible to write a parser for protobuf-BIP174 without
needing a general protobuf library. Protobuf formats are designed with
forwards- and backwards- compatibility in mind, so having a hand-written
implementation should not lead to incompatibilities.

I did an experiment with this and other variants of the BIP174 format.
You can see them here:
[1] https://github.com/matejcik/bip174-playground
see in particular:
[2] https://github.com/matejcik/bip174-playground/blob/master/bip174.prot=
o

The tool at [1] does size comparisons. On the test vectors, protobuf is
slightly smaller than key-value, and roughly equal to record-set, losing
out a little when BIP32 fields are used.
(I'm also leaving out key-data for BIP32 fields.)

There's some technical points to consider about protobuf, too:

- I decided to structure the message as a single "PSBT" type, where
"InputType" and "OutputType" are repeatable embedded fields. This seemed
better in terms of extensibility - we could add more sections in the
future. But the drawback is that you need to know the size of
Input/OutputType record in advance.
The other option is sending the messages separated by NUL bytes, same as
now, in which case you don't need to specify size.

- in InputType, i'm using "uint32" for sighash. This type is not
length-delimited, so non-protobuf consumers would have to understand it
specially. We could also declare that all fields must be
length-delimited[1] and implement sighash as a separate message type
with one field.

- non-protobuf consumers will also need to understand both protobuf
varint and bitcoin compact uint, which is a little ugly

best regards
matejcik


[1] https://developers.google.com/protocol-buffers/docs/encoding#structur=
e


--HN3AdUEsOchTlvdQeDk1FNoAuphj1qawm--

--74aBNEZiIFs9NuoqOguzkKgVgn9KqMPKP
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCAAGBQJbMly6AAoJEMZ/sQbk7RcePQoQAIRHSplzTDWvd8xmeOn7iUKc
YJhXQ9bZaKXa9Gq51QFXfzL0Q2IdSeMwcBYxqxX73dQNy4keWZ/JHCu2gz8ULpcD
KF9bfzKM4yq/vAcc+xwCJPtJADa5na5XrG3jcqR5YpfjaH3YTpmzKchFnGNShu61
aco8VygvOwaT6NKuJQ37q3XbWrNga1EBSc8KEfob1xbzxoQ1v4Y8obZ4PzUMmtyn
6KXAcOSGTJLJRHqquUhp7Cw6l9zwKa+ueJ1vSA7EHwxoj+RncAjUBcKskJzW2Aa1
IAcrZK/zn2vc69Y9pkotiNvvYDL1Vr+R4OqCxLaV/eJN6cr1zfRjvtyxVAm36aFC
DHageXHofqI3XcD1o8C2UsbYPGJJZxSdUMo0thPDN59SLMoxJmyYSirO5Eam50eC
rbHk21ToF8NsRNtFzGX39h/HQ40pLpqQoaLwXyn/ReWbgDBKFwTTu5OTy8BHMjGN
iScIvVpoMBS/JHbals39e05sVZUzfjpSJdCN52gSPu/lrW7I/ku4OMeiKp+fZy1u
qVPvxJ9pKLq6rcUvc54S5XEnpzDSBGrLzSZxx/VhkAAyBgsKnvac/YZPVtC2sYrT
4pp+AyLSmp392qA8aM3MBeEFzrMctri7PhBDNcXx76WZkyztlyPFw4uc6ZV4c6gZ
1D0fNidkTWcQh49Ih3C0
=bzDw
-----END PGP SIGNATURE-----

--74aBNEZiIFs9NuoqOguzkKgVgn9KqMPKP--