Delivery-date: Tue, 02 Jul 2024 04:27:12 -0700
Received-SPF: pass (google.com: domain of darosior@protonmail.com designates 185.70.40.141 as permitted sender) client-ip=185.70.40.141;
Date: Tue, 02 Jul 2024 10:23:51 +0000
To: Eric Voskuil <eric@voskuil.org>
From: "'Antoine Poinsot' via Bitcoin Development Mailing List" <bitcoindev@googlegroups.com>
Cc: Bitcoin Development Mailing List <bitcoindev@googlegroups.com>
Subject: Re: [bitcoindev] Re: Great Consensus Cleanup Revival
Message-ID: <wg_er0zMhAF9ERoYXmxI6aB7rc97Cum6PQj4UOELapsHVBBVWktFeOZT7sHDlyrXwJ5o5s9iMb2LW2Od-qacywsh-86p5Q7dP3XjWASXcMw=@protonmail.com>
In-Reply-To: <9a4c4151-36ed-425a-a535-aa2837919a04n@googlegroups.com>
References: <gnM89sIQ7MhDgI62JciQEGy63DassEv7YZAMhj0IEuIo0EdnafykF6RH4OqjTTHIHsIoZvC2MnTUzJI7EfET4o-UQoD-XAQRDcct994VarE=@protonmail.com> <72e83c31-408f-4c13-bff5-bf0789302e23n@googlegroups.com> <heKH68GFJr4Zuf6lBozPJrb-StyBJPMNvmZL0xvKFBnBGVA3fVSgTLdWc-_8igYWX8z3zCGvzflH-CsRv0QCJQcfwizNyYXlBJa_Kteb2zg=@protonmail.com> <5b0331a5-4e94-465d-a51d-02166e2c1937n@googlegroups.com> <yt1O1F7NiVj-WkmnYeta1fSqCYNFx8h6OiJaTBmwhmJ2MWAZkmmjPlUST6FM7t6_-2NwWKdglWh77vcnEKA8swiAnQCZJY2SSCAh4DOKt2I=@protonmail.com> <be78e733-6e9f-4f4e-8dc2-67b79ddbf677n@googlegroups.com> <jJLDrYTXvTgoslhl1n7Fk9-pL1mMC-0k6gtoniQINmioJpzgtqrJ_WqyFZkLltsCUusnQ4jZ6HbvRC-mGuaUlDi3kcqcFHALd10-JQl-FMY=@protonmail.com> <9a4c4151-36ed-425a-a535-aa2837919a04n@googlegroups.com>
Feedback-ID: 7060259:user:proton
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="b1_Imppqfpa05uoQJeuGY4iXaW2SkKgtL17uJOzJMt8"
Reply-To: Antoine Poinsot <darosior@protonmail.com>
Precedence: list
Mailing-list: list bitcoindev@googlegroups.com; contact bitcoindev+owners@googlegroups.com

This is a multi-part message in MIME format.

--b1_Imppqfpa05uoQJeuGY4iXaW2SkKgtL17uJOzJMt8
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

>>> This does not produce unmalleable block hashes. Duplicate tx hash malle=
ation remains in either case, to the same effect. Without a resolution to b=
oth issues this is an empty promise.
>> Duplicate txids have been invalid since 2012 (CVE-2012-2459).
>
> I think again here you may have misunderstood me. I was not making a poin=
t pertaining to BIP30.

No, in fact you did. CVE-2012-2459 is unrelated to BIP30, it's the duplicat=
e txids malleability found by forrestv in 2012. It's the one you are talkin=
g about thereafter and the one relevant for the purpose of this discussion.

For future reference, the full disclosure of CVE-2012-2459 can be found the=
re: https://bitcointalk.org/?topic=3D102395.

> The proposal does not enable that objective, it is already the case. No m=
alleated block is a valid block.

You are right. The advantage i initially mentioned about how making 64-byte=
s transactions invalid could help caching block failures at an earlier stag=
e is incorrect.

Best,
Antoine Poinsot
On Friday, June 28th, 2024 at 7:14 PM, Eric Voskuil <eric@voskuil.org> wrot=
e:

>>> It is not clear to me how determining the coinbase size can be done at =
an earlier stage of validation than detection of the non-null coinbase.
>> My point wasn't about checking the coinbase size, it was about being abl=
e to cache the hash of a (non-malleated) invalid block as permanently inval=
id to avoid re-downloading and re-validating it.
>
> This I understood, but I think you misunderstood me. Your point was speci=
fically that, "it would let node implementations cache block failures at an=
 earlier stage of validation." Since you have not addressed that aspect I a=
ssume you agree with my assertion above that the proposed rule does not act=
ually achieve this.
>
> Regarding the question of checking coinbase size, the issue is of detecti=
ng (or preventing) hashes mallied via the 64 byte tx technique. A rule agai=
nst 64 byte txs would allow this determination by checking the coinbase alo=
ne. If the coinbase is 64 bytes the block is invalid, if it is not the bloc=
k hash cannot have been mallied (all txs must have been 64 bytes, see previ=
ous reference).
>
> In that case if the block is invalid the invalidity can be cached. But bl=
ock invalidity cannot actually be cached until the block is fully validated=
. A rule to prohibit *all* 64 byte txs is counterproductive as it only adds=
 additional checks on typically thousands of txs per block, serving no purp=
ose.
>
>>> It seems to me that introducing an arbitrary tx size validity may creat=
e more potential implementation bugs than it resolves.
>> The potential for implementation bugs is a fair point to raise, but in t=
his case i don't think it's a big concern. Verifying no transaction in a bl=
ock is 64 bytes is as simple a check as you can get.
>
> You appear to be making the assumption that the check is performed after =
the block is fully parsed (contrary to your "earlier" criterion above). The=
 only way to determine the tx sizes is to parse each tx for witness marker,=
 input count, output count, input script sizes, output script sizes, witnes=
s sizes, and skipping over the header, several constants, and associated bu=
ffers. Doing this "early" to detect malleation is an extraordinarily comple=
x and costly process. On the other hand, as I pointed out, a rational imple=
mentation would only do this early check for the coinbase.
>
> Yet even determining the size of the coinbase is significantly more compl=
ex and costly than checking its first input point against null. That check =
(which is already necessary for validation) resolves the malleation questio=
n, can be performed on the raw unparsed block buffer by simply skipping hea=
der, version, reading input count and witness marker as necessary, offsetti=
ng to the 36 byte point buffer, and performing a byte comparison against [0=
000000000000000000000000000000000000000000000000000000000000000ffffffff].
>
> This is:
>
> (1) earlier
> (2) faster
> (3) simpler
> (4) already consensus
>
>>> And certainly anyone implementing such a verifier must know many intric=
acies of the protocol.
>> They need to know some, but i don't think it's reasonable to expect them=
 to realize the merkle tree construction is such that an inner node may be =
confused with a 64 bytes transaction.
>
> A protocol developer needs to understand that the hash of an invalid bloc=
k cannot be cached unless at least the coinbase has been restricted in size=
 (under the proposal) -or- that the coinbase is a null point (presently or =
under the proposal). In the latter case the check is already performed in v=
alidation, so there is no way a block would presently be cached as invalid =
without checking it. The proposal adds a redundant check, even if limited t=
o just the coinbase. [He must also understand the second type of malleabili=
ty, discussed below.]
>
> If this proposed rule was to activate we would implement it in a late sta=
ge tx.check, after txs/blocks had been fully deserialized. We would not che=
ck it an all in the case where the block is under checkpoint or milestone (=
"assume valid"). In this case we would retain the early null point malleati=
on check (along with the hash duplication malleation check) that we present=
ly have, would validate tx commitments, and commit the block. In other word=
s, the proposal adds unnecessary late stage checks only. Implementing it ot=
herwise would just add complexity and hurt performance.
>
>>> I do not see this. I see a very ugly perpetual seam which will likely r=
esult in unexpected complexities over time.
>> What makes you think making 64 bytes transactions invalid could result i=
n unexpected complexities? And why do you think it's likely?
>
> As described above, it's later, slower, more complex, unnecessarily broad=
, and a consensus change. Beyond that it creates an arbitrary size limit - =
not a lower or upper bound, but a slice out of the domain. Discontinuities =
are inherent complexities in computing. The "unexpected" part speaks for it=
self.
>
>>> This does not produce unmalleable block hashes. Duplicate tx hash malle=
ation remains in either case, to the same effect. Without a resolution to b=
oth issues this is an empty promise.
>> Duplicate txids have been invalid since 2012 (CVE-2012-2459).
>
> I think again here you may have misunderstood me. I was not making a poin=
t pertaining to BIP30. I was referring to the other form of block hash mall=
eability, which results from duplicating sets of trailing txs in a single b=
lock (see previous reference). This malleation vector remains, even with in=
valid 64 byte txs. As I pointed out, this has the "same effect" as the 64 b=
yte tx issue. Merkle hashing the set of txs is insufficient to determine id=
entity. In one case the coinbase must be checked (null point or size) and i=
n the other case the set of tx hashes must be checked for trailing duplicat=
ed sets. [Core performs this second check within the Merkle hashing algorit=
hm (with far more comparisons than necessary), though this can be performed=
 earlier and independently to avoid any hashing in the malleation case.]
>
> I would also point out in the interest of correctness that Core reverted =
its BIP30 soft fork implementation as a consequence of the BIP90 hard fork,=
 following and requiring the BIP34 soft fork that presumably precluded it b=
ut didn't, so it is no longer the case that duplicate tx hashes are invalid=
 in implementation. As you have proposed in this rollup, this requires fixi=
ng again.
>
>> If 64 bytes transactions are also made invalid, this would make it impos=
sible for two valid blocks to have the same hash.
>
> Aside from the BIP30/34/90 issue addressed above, it is already "impossib=
le" (cannot be stronger than computationally infeasible) for two *valid* bl=
ocks to have the same hash. The proposal does not enable that objective, it=
 is already the case. No malleated block is a valid block.
>
> The proposal aims only to make it earlier or easier or faster to check fo=
r block hash malleation. And as I've pointed out above, it doesn't achieve =
those objectives. Possibly the perception that this would be the case is a =
consequence of implementation details, but as I have shown above, it is not=
 in fact the case.
>
> Given either type of malleation, the malleated block can be determined to=
 be invalid by a context free check. But this knowledge cannot ever be cach=
ed against the block hash, since the same hash may be valid. Invalidity can=
 only be cached once a non-mallied block is validated and determined to be =
invalid. Block hash malleations are and will remain invalid blocks with or =
without the proposal, and it will continue to be necessary to avoid caching=
 invalid against the malleation. As you said:
>
>> it was about being able to cache the hash of a (non-malleated) invalid b=
lock as permanently invalid to avoid re-downloading and re-validating it.
>
> This is already the case, and requires validating the full non-malleated =
block. Adding a redundant invalidity check doesn't improve this in any way.
>
> Best,
> Eric
>
> --
> You received this message because you are subscribed to the Google Groups=
 "Bitcoin Development Mailing List" group.
> To unsubscribe from this group and stop receiving emails from it, send an=
 email to bitcoindev+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgi=
d/bitcoindev/9a4c4151-36ed-425a-a535-aa2837919a04n%40googlegroups.com.

--=20
You received this message because you are subscribed to the Google Groups "=
Bitcoin Development Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to bitcoindev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/=
bitcoindev/wg_er0zMhAF9ERoYXmxI6aB7rc97Cum6PQj4UOELapsHVBBVWktFeOZT7sHDlyrX=
wJ5o5s9iMb2LW2Od-qacywsh-86p5Q7dP3XjWASXcMw%3D%40protonmail.com.

--b1_Imppqfpa05uoQJeuGY4iXaW2SkKgtL17uJOzJMt8
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<blockquote style=3D"border-color: rgb(200, 200, 200); border-left: 3px sol=
id rgb(200, 200, 200); padding-left: 10px; color: rgb(102, 102, 102);"><div=
>&gt;&gt; This does not produce unmalleable block hashes. Duplicate tx hash=
=20
malleation remains in either case, to the same effect. Without a=20
resolution to both issues this is an empty promise.<br></div><div>&gt; Dupl=
icate txids have been invalid since 2012 (CVE-2012-2459).<br></div><div><br=
>I think again here you may have misunderstood me. I was not making a point=
 pertaining to BIP30.<br></div></blockquote><div style=3D"font-family: Aria=
l, sans-serif; font-size: 14px; color: rgb(0, 0, 0); background-color: rgb(=
255, 255, 255);"><br></div><div style=3D"font-family: Arial, sans-serif; fo=
nt-size: 14px; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">=
No, in fact you did. CVE-2012-2459 is unrelated to BIP30, it's the duplicat=
e txids malleability found by forrestv in 2012. It's the one you are talkin=
g about thereafter and the one relevant for the purpose of this discussion.=
<br></div><div style=3D"font-family: Arial, sans-serif; font-size: 14px; co=
lor: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><br></div><div st=
yle=3D"font-family: Arial, sans-serif; font-size: 14px; color: rgb(0, 0, 0)=
; background-color: rgb(255, 255, 255);">For future reference, the full dis=
closure of&nbsp;CVE-2012-2459 can be found there: <span><a target=3D"_blank=
" rel=3D"noreferrer nofollow noopener" href=3D"https://bitcointalk.org/?top=
ic=3D102395">https://bitcointalk.org/?topic=3D102395</a></span>.</div><div =
style=3D"font-family: Arial, sans-serif; font-size: 14px; color: rgb(0, 0, =
0); background-color: rgb(255, 255, 255);"><br></div><blockquote style=3D"b=
order-color: rgb(200, 200, 200); border-left: 3px solid rgb(200, 200, 200);=
 padding-left: 10px; color: rgb(102, 102, 102);"><div style=3D"font-family:=
 Arial, sans-serif; font-size: 14px; color: rgb(0, 0, 0); background-color:=
 rgb(255, 255, 255);">The proposal does not enable that objective, it is al=
ready the case. No malleated block is a valid block.</div></blockquote><div=
 style=3D""><br></div><div style=3D"">You are right. The advantage i initia=
lly mentioned about how making 64-bytes transactions invalid could help cac=
hing block failures at an earlier stage is incorrect.</div><div style=3D"">=
<br></div><div style=3D"">Best,<br>Antoine Poinsot<br></div><div class=3D"p=
rotonmail_quote">
        On Friday, June 28th, 2024 at 7:14 PM, Eric Voskuil &lt;eric@voskui=
l.org&gt; wrote:<br>
        <blockquote class=3D"protonmail_quote" type=3D"cite">
            &gt;&gt; It is not clear to me how determining the coinbase siz=
e can be done at an earlier stage of validation than detection of the non-n=
ull coinbase.<br>&gt; My point wasn't about checking the coinbase size, it =
was about being able to cache the hash of a (non-malleated) invalid block a=
s permanently invalid to avoid re-downloading and re-validating it.<br><br>=
This I understood, but I think you misunderstood me. Your point was specifi=
cally that, "it would let node implementations cache block failures at an e=
arlier stage of validation." Since you have not addressed that aspect I ass=
ume you agree with my assertion above that the proposed rule does not actua=
lly achieve this.<br><br>Regarding the question of checking coinbase size, =
the issue is of detecting (or preventing) hashes mallied via the 64 byte tx=
 technique. A rule against 64 byte txs would allow this determination by ch=
ecking the coinbase alone. If the coinbase is 64 bytes the block is invalid=
, if it is not the block hash cannot have been mallied (all txs must have b=
een 64 bytes, see previous reference).<br><br>In that case if the block is =
invalid the invalidity can be cached. But block invalidity cannot actually =
be cached until the block is fully validated. A rule to prohibit *all* 64 b=
yte txs is counterproductive as it only adds additional checks on typically=
 thousands of txs per block, serving no purpose.<br><br>&gt;&gt; It seems t=
o me that introducing an arbitrary tx size validity may create more potenti=
al implementation bugs than it resolves.<br>&gt; The potential for implemen=
tation bugs is a fair point to raise, but in this case i don't think it's a=
 big concern. Verifying no transaction in a block is 64 bytes is as simple =
a check as you can get.<br><br>You appear to be making the assumption that =
the check is performed after the block is fully parsed (contrary to your "e=
arlier" criterion above). The only way to determine the tx sizes is to pars=
e each tx for witness marker, input count, output count, input script sizes=
, output script sizes, witness sizes, and skipping over the header, several=
 constants, and associated buffers. Doing this "early" to detect malleation=
 is an extraordinarily complex and costly process. On the other hand, as I =
pointed out, a rational implementation would only do this early check for t=
he coinbase.<br><br>Yet even determining the size of the coinbase is signif=
icantly more complex and costly than checking its first input point against=
 null. That check (which is already necessary for validation) resolves the =
malleation question, can be performed on the raw unparsed block buffer by s=
imply skipping header, version, reading input count and witness marker as n=
ecessary, offsetting to the 36 byte point buffer, and performing a byte com=
parison against [0000000000000000000000000000000000000000000000000000000000=
000000ffffffff].<br><br>This is:<br><br>(1) earlier<br>(2) faster<br>(3) si=
mpler<br>(4) already consensus<br><br>&gt;&gt; And certainly anyone impleme=
nting such a verifier must know many intricacies of the protocol.<br>&gt; T=
hey need to know some, but i don't think it's reasonable to expect them to =
realize the merkle tree construction is such that an inner node may be conf=
used with a 64 bytes transaction.<br><br>A protocol developer needs to unde=
rstand that the hash of an invalid block cannot be cached unless at least t=
he coinbase has been restricted in size (under the proposal) -or- that the =
coinbase is a null point (presently or under the proposal). In the latter c=
ase the check is already performed in validation, so there is no way a bloc=
k would presently be cached as invalid without checking it. The proposal ad=
ds a redundant check, even if limited to just the coinbase. [He must also u=
nderstand the second type of malleability, discussed below.]<br><br>If this=
 proposed rule was to activate we would implement it in a late stage tx.che=
ck, after txs/blocks had been fully deserialized. We would not check it an =
all in the case where the block is under checkpoint or milestone ("assume v=
alid"). In this case we would retain the early null point malleation check =
(along with the hash duplication malleation check) that we presently have, =
would validate tx commitments, and commit the block. In other words, the pr=
oposal adds unnecessary late stage checks only. Implementing it otherwise w=
ould just add complexity and hurt performance.<br><br>&gt;&gt; I do not see=
 this. I see a very ugly perpetual seam which will likely result in unexpec=
ted complexities over time.<br>&gt; What makes you think making 64 bytes tr=
ansactions invalid could result in unexpected complexities? And why do you =
think it's likely?<br><br>As described above, it's later, slower, more comp=
lex, unnecessarily broad, and a consensus change. Beyond that it creates an=
 arbitrary size limit - not a lower or upper bound, but a slice out of the =
domain. Discontinuities are inherent complexities in computing. The "unexpe=
cted" part speaks for itself.<br><br>&gt;&gt; This does not produce unmalle=
able block hashes. Duplicate tx hash malleation remains in either case, to =
the same effect. Without a resolution to both issues this is an empty promi=
se.<br>&gt; Duplicate txids have been invalid since 2012 (CVE-2012-2459).<b=
r><br>I think again here you may have misunderstood me. I was not making a =
point pertaining to BIP30. I was referring to the other form of block hash =
malleability, which results from duplicating sets of trailing txs in a sing=
le block (see previous reference). This malleation vector remains, even wit=
h invalid 64 byte txs. As I pointed out, this has the "same effect" as the =
64 byte tx issue. Merkle hashing the set of txs is insufficient to determin=
e identity. In one case the coinbase must be checked (null point or size) a=
nd in the other case the set of tx hashes must be checked for trailing dupl=
icated sets. [Core performs this second check within the Merkle hashing alg=
orithm (with far more comparisons than necessary), though this can be perfo=
rmed earlier and independently to avoid any hashing in the malleation case.=
]<br><br>I would also point out in the interest of correctness that Core re=
verted its BIP30 soft fork implementation as a consequence of the BIP90 har=
d fork, following and requiring the BIP34 soft fork that presumably preclud=
ed it but didn't, so it is no longer the case that duplicate tx hashes are =
invalid in implementation. As you have proposed in this rollup, this requir=
es fixing again.<br><br>&gt; If 64 bytes transactions are also made invalid=
, this would make it impossible for two valid blocks to have the same hash.=
<br><br>Aside from the BIP30/34/90 issue addressed above, it is already "im=
possible" (cannot be stronger than computationally infeasible) for two *val=
id* blocks to have the same hash. The proposal does not enable that objecti=
ve, it is already the case. No malleated block is a valid block.<br><br>The=
 proposal aims only to make it earlier or easier or faster to check for blo=
ck hash malleation. And as I've pointed out above, it doesn't achieve those=
 objectives. Possibly the perception that this would be the case is a conse=
quence of implementation details, but as I have shown above, it is not in f=
act the case.<br><br>Given either type of malleation, the malleated block c=
an be determined to be invalid by a context free check. But this knowledge =
cannot ever be cached against the block hash, since the same hash may be va=
lid. Invalidity can only be cached once a non-mallied block is validated an=
d determined to be invalid. Block hash malleations are and will remain inva=
lid blocks with or without the proposal, and it will continue to be necessa=
ry to avoid caching invalid against the malleation. As you said:<br><br>&gt=
; it was about being able to cache the hash of a (non-malleated) invalid bl=
ock as permanently invalid to avoid re-downloading and re-validating it.<br=
><br>This is already the case, and requires validating the full non-malleat=
ed block. Adding a redundant invalidity check doesn't improve this in any w=
ay.<br><br>Best,<br>Eric

<p></p>

-- <br>
You received this message because you are subscribed to the Google Groups "=
Bitcoin Development Mailing List" group.<br>
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to <a href=3D"mailto:bitcoindev+unsubscribe@googlegroups.com" rel=3D"n=
oreferrer nofollow noopener" target=3D"_blank">bitcoindev+unsubscribe@googl=
egroups.com</a>.<br>
To view this discussion on the web visit <a href=3D"https://groups.google.c=
om/d/msgid/bitcoindev/9a4c4151-36ed-425a-a535-aa2837919a04n%40googlegroups.=
com" rel=3D"noreferrer nofollow noopener" target=3D"_blank">https://groups.=
google.com/d/msgid/bitcoindev/9a4c4151-36ed-425a-a535-aa2837919a04n%40googl=
egroups.com</a>.<br>

        </blockquote><br>
    </div>

<p></p>

-- <br />
You received this message because you are subscribed to the Google Groups &=
quot;Bitcoin Development Mailing List&quot; group.<br />
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to <a href=3D"mailto:bitcoindev+unsubscribe@googlegroups.com">bitcoind=
ev+unsubscribe@googlegroups.com</a>.<br />
To view this discussion on the web visit <a href=3D"https://groups.google.c=
om/d/msgid/bitcoindev/wg_er0zMhAF9ERoYXmxI6aB7rc97Cum6PQj4UOELapsHVBBVWktFe=
OZT7sHDlyrXwJ5o5s9iMb2LW2Od-qacywsh-86p5Q7dP3XjWASXcMw%3D%40protonmail.com?=
utm_medium=3Demail&utm_source=3Dfooter">https://groups.google.com/d/msgid/b=
itcoindev/wg_er0zMhAF9ERoYXmxI6aB7rc97Cum6PQj4UOELapsHVBBVWktFeOZT7sHDlyrXw=
J5o5s9iMb2LW2Od-qacywsh-86p5Q7dP3XjWASXcMw%3D%40protonmail.com</a>.<br />

--b1_Imppqfpa05uoQJeuGY4iXaW2SkKgtL17uJOzJMt8--