MIME-Version: 1.0
From: Olaoluwa Osuntokun <laolu32@gmail.com>
Date: Thu, 4 Nov 2021 15:01:13 -0700
Message-ID: <CAO3Pvs-Jazo27Vmi3Rforke++T5rkboS=2PK9CSSTtCk4enF2w@mail.gmail.com>
To: lnd@lightning.engineering, 
 Arnoud Kouwenhoven - Pukaki Corp via bitcoin-dev
 <bitcoin-dev@lists.linuxfoundation.org>
Content-Type: multipart/alternative; boundary="000000000000e77fd905cffdaa33"
Subject: [bitcoin-dev] Neutrino, Taproot, and The Evolution of BiPs 157/158
Precedence: list

--000000000000e77fd905cffdaa33
Content-Type: text/plain; charset="UTF-8"

Hi y'all,

If you're an active user of neutrino [8], then you probably heard about what
went down over the past week or so on testnet as related to Taproot. First,
i just wanted to reassure everyone that nothing is fundamentally broken with
BIP 157/158 as it relates to taproot, and we already have a mitigation patch
in place for the issue we encountered.

The rest of this mail is structured in a FAQ style to make it easy to skim
and extract the information that may be relevant to the reader.

## What happened on testnet?

Neutrino nodes on testnet rejected a filter (thinking it was invalid) due
to this transaction spending a taproot input [1]. This was due to a faulty
heuristics in the neutrino _client code_ (not part of the protocol) that
attempted to verify the contents of a filter more completely.

In retrospect, the heuristic in question wasn't full proof, as it attempted
to derive the _pk script_ of a transaction based on its input
witness/sigScript. This worked pretty well in the context of segwit v0, but
it isn't possible to exhaustively do as we don't know what future spends
will look like.

## Is neutrino broken?

No, the client side is fine, and the protocol is fine.

The problematic heuristic has been removed in this PR [2], which will be
included in lnd 0.14, and has been tagged with neutrino 0.13 [3].

To dig into _why_ we attempted to use such a heuristic, we'll need to
revisit how BIP 158 works briefly. For each block, we insert the `pkScript`s
of all the outputs, and also the prev out's pkScript (the script being
spent) as well. This lets the filter compress script re-use in both inputs
and outputs, and also makes it possible to implement some protocols in a
more light-client friendly manner (for example Loop uses this to has the
client watch HTLC _scripts_ being spent, as it doesn't necessarily know the
txid/outpoint).

The one issue with this, is that while clients can ensure that all the
`pkScripts` of outputs have been inserted, they can't do the same for the
inputs (which is why we added that heuristic in the client code). Luckily we
know how to properly fix this at the protocol level, more on that below.

## How can I make sure my neutrino clients handle the Taproot upgrade on
mainnet smoothly?

Upgrade to 0.14 (assuming it's out in time), or apply this small patch [4].
The patch just demotes an error case to a warning message, so anyone running
a fork should be able to easily apply the fix.

Alongside, optionally extend these filter header guides [7].

We're looking into some intermediate ground where we can verify the scripts
that we know are relevant to the node.

## How will BIP 158/157 evolve post taproot?

In terms of adding more taproot specific functionality, I've had a number of
items in my laundry list such as:

  * creating new segwit-only filters with re-parameterized fp rates (also
    examine other filter types such as pure outpoints, etc)

  * creating filters that include witness data to allow matching on
    internal/external keys, the control block, merkle root, annex, etc

  * add a new protocol extension to btcd (with a corresponding BIP) to
    allow notes to fetch block undo data (as described here [5]) to fully
    verify fetched filters or a node needs to reconcile conflicting filters

  * new filters that span across multiple blocks as previously worked on by
    Kalle Alm (couldn't find a link to his PR when typing this...)

Make further progress towards a proposal that allows filters to be committed
either as a soft-fork, or a "velvet fork" [6] where miners optionally
commit to
the past filter header chain.


-- Laolu

[1]:
https://mempool.space/testnet/tx/4b425a1f5c0fcf4794c48b810c53078773fb768acd2be1398e3f561cc3f19fb8
[2]: https://github.com/lightninglabs/neutrino/pull/234
[3]: https://github.com/lightninglabs/neutrino/releases/tag/v0.13.0
[4]: https://github.com/lightninglabs/neutrino/pull/234/files
[5]:
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2019-February/016649.html
[6]: https://eprint.iacr.org/2018/087
[7]:
https://github.com/lightninglabs/neutrino/blob/5e09bd9b5d65e90c6ff07aa11b3b9d80d42afb86/chainsync/filtercontrol.go#L15
[8]: https://github.com/lightninglabs/neutrino

--000000000000e77fd905cffdaa33
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi y&#39;all,<br><br>If you&#39;re an active user of neutr=
ino [8], then you probably heard about what<br>went down over the past week=
 or so on testnet as related to Taproot. First,<br>i just wanted to reassur=
e everyone that nothing is fundamentally broken with<br>BIP 157/158 as it r=
elates to taproot, and we already have a mitigation patch<br>in place for t=
he issue we encountered.<br><br>The rest of this mail is structured in a FA=
Q style to make it easy to skim<br>and extract the information that may be =
relevant to the reader.<br><br>## What happened on testnet?<br><br>Neutrino=
 nodes on testnet rejected a filter (thinking it was invalid) due<br>to thi=
s transaction spending a taproot input [1]. This was due to a faulty<br>heu=
ristics in the neutrino _client code_ (not part of the protocol) that<br>at=
tempted to verify the contents of a filter more completely. <br><br>In retr=
ospect, the heuristic in question wasn&#39;t full proof, as it attempted<br=
>to derive the _pk script_ of a transaction based on its input<br>witness/s=
igScript. This worked pretty well in the context of segwit v0, but<br>it is=
n&#39;t possible to exhaustively do as we don&#39;t know what future spends=
<br>will look like.<br><br>## Is neutrino broken?<br><br>No, the client sid=
e is fine, and the protocol is fine.<br><br>The problematic heuristic has b=
een removed in this PR [2], which will be<br>included in lnd 0.14, and has =
been tagged with neutrino 0.13 [3].<br><br>To dig into _why_ we attempted t=
o use such a heuristic, we&#39;ll need to<br>revisit how BIP 158 works brie=
fly. For each block, we insert the `pkScript`s<br>of all the outputs, and a=
lso the prev out&#39;s pkScript (the script being<br>spent) as well. This l=
ets the filter compress script re-use in both inputs<br>and outputs, and al=
so makes it possible to implement some protocols in a<br>more light-client =
friendly manner (for example Loop uses this to has the<br>client watch HTLC=
 _scripts_ being spent, as it doesn&#39;t necessarily know the<br>txid/outp=
oint).<br><br>The one issue with this, is that while clients can ensure tha=
t all the<br>`pkScripts` of outputs have been inserted, they can&#39;t do t=
he same for the<br>inputs (which is why we added that heuristic in the clie=
nt code). Luckily we<br>know how to properly fix this at the protocol level=
, more on that below.<br><br>## How can I make sure my neutrino clients han=
dle the Taproot upgrade on mainnet smoothly?<br><br>Upgrade to 0.14 (assumi=
ng it&#39;s out in time), or apply this small patch [4].<br>The patch just =
demotes an error case to a warning message, so anyone running<br>a fork sho=
uld be able to easily apply the fix.<br><br>Alongside, optionally extend th=
ese filter header guides [7].<br><br>We&#39;re looking into some intermedia=
te ground where we can verify the scripts<br>that we know are relevant to t=
he node.<br><br>## How will BIP 158/157 evolve post taproot?<br><br>In term=
s of adding more taproot specific functionality, I&#39;ve had a number of<b=
r>items in my laundry list such as:<br><br>=C2=A0 * creating new segwit-onl=
y filters with re-parameterized fp rates (also<br>=C2=A0 =C2=A0 examine oth=
er filter types such as pure outpoints, etc)<br><br>=C2=A0 * creating filte=
rs that include witness data to allow matching on<br>=C2=A0 =C2=A0 internal=
/external keys, the control block, merkle root, annex, etc<br><br>=C2=A0 * =
add a new protocol extension to btcd (with a corresponding BIP) to<br>=C2=
=A0 =C2=A0 allow notes to fetch block undo data (as described here [5]) to =
fully<br>=C2=A0 =C2=A0 verify fetched filters or a node needs to reconcile =
conflicting filters<br><br>=C2=A0 * new filters that span across multiple b=
locks as previously worked on by<br>=C2=A0 =C2=A0 Kalle Alm (couldn&#39;t f=
ind a link to his PR when typing this...)<br><br>Make further progress towa=
rds a proposal that allows filters to be committed<br>either as a soft-fork=
, or a &quot;velvet fork&quot; [6] where miners optionally commit to<br>the=
 past filter header chain.<br><br><br>-- Laolu<br><br>[1]: <a href=3D"https=
://mempool.space/testnet/tx/4b425a1f5c0fcf4794c48b810c53078773fb768acd2be13=
98e3f561cc3f19fb8">https://mempool.space/testnet/tx/4b425a1f5c0fcf4794c48b8=
10c53078773fb768acd2be1398e3f561cc3f19fb8</a><br>[2]: <a href=3D"https://gi=
thub.com/lightninglabs/neutrino/pull/234">https://github.com/lightninglabs/=
neutrino/pull/234</a><br>[3]: <a href=3D"https://github.com/lightninglabs/n=
eutrino/releases/tag/v0.13.0">https://github.com/lightninglabs/neutrino/rel=
eases/tag/v0.13.0</a><br>[4]: <a href=3D"https://github.com/lightninglabs/n=
eutrino/pull/234/files">https://github.com/lightninglabs/neutrino/pull/234/=
files</a><br>[5]: <a href=3D"https://lists.linuxfoundation.org/pipermail/bi=
tcoin-dev/2019-February/016649.html">https://lists.linuxfoundation.org/pipe=
rmail/bitcoin-dev/2019-February/016649.html</a><br>[6]: <a href=3D"https://=
eprint.iacr.org/2018/087">https://eprint.iacr.org/2018/087</a><br>[7]: <a h=
ref=3D"https://github.com/lightninglabs/neutrino/blob/5e09bd9b5d65e90c6ff07=
aa11b3b9d80d42afb86/chainsync/filtercontrol.go#L15">https://github.com/ligh=
tninglabs/neutrino/blob/5e09bd9b5d65e90c6ff07aa11b3b9d80d42afb86/chainsync/=
filtercontrol.go#L15</a><br>[8]: <a href=3D"https://github.com/lightninglab=
s/neutrino">https://github.com/lightninglabs/neutrino</a><br></div>

--000000000000e77fd905cffdaa33--