path: root/transcripts/bitcoin-core-dev-tech/2017-09-05.mdwn

2017-09-05 - 2017-09-07

coredev.tech september 2017

<https://twitter.com/kanzure/status/907233490919464960>

((As always, any errors are most likely my own. etc.))

<div id="intro" />
# Introduction

There is significant concern regarding whether BlueMatt has become a misnomer.

Monday night presentation: <http://diyhpl.us/wiki/transcripts/sf-bitcoin-meetup/2017-09-04-jonas-schenlli-bip150-bip151/>

I think we should continue to use #bitcoin-core-dev for anything about changing Bitcoin Core and try to keep things open even though we're together here today and tomorrow and the next.

<div id="rescans" />
# Wallets and block pruning and rescans

Nobody produces P2SH change when they have a native output. And producing a mixed set of outputs will tell you which one it is. So this is not ideal from a privacy standpoint.

If you scan a transaction... so, there's still this question of for do you have... P2SH embedded or not. And we will need separate ones, for other script versions too. ....

At some point, you can't expect all nodes to have the full blockchain. So we should have wallet management practices which will not be gratuitously requiring as much, and such rescans. You could have a Bloom filter, scan it, and then determine if you are going to fetch the block or not, like if you know you have correlated things that you want to fetch from that block. You can add noise, sure, that will help.

I suspect that the way that rescan will work a year from now is to <a href="https://www.youtube.com/watch?v=7FWKc8lM4Ek">implement Laolu's proposal</a>. So we'd say that the ... Bloom filter for every block, and then a rescan will scan the filters, and then go for it. We should not limit ourselves to... there, just because we might... This is basically like when we generate keys for P2SH embedded.

Do I have the pubkey? Well, if not, then never consider it mine. Right now you won't accept a P2SH that is paying.. key hash. You will. If you have the P2SH, yes. Ah I think I have a solution. If you have a redeemScript then it will try to recurse into it. There are more possibilities-- the pubkeyhash versus the scripthash, ... maybe two chains.

You should only accept things you issued. It's important to have this discussion. Bryan had previously raised a concern that it looked like we would accept IsMine() people converting your... in order to accept anything witness, the equivalent redeemScript must be in your key store, meaning you must have explicitly added it.

There's all kinds of other scripts that you can have that use the same keys. You only want to accept payments to those if you added that script. That template as applies to that key. The problem with that rule is that the situation is not robust... backup. This goes back to an earlier concern... every time you use the wallet, you have to backup now. I'd like to think about, as we move forward, that the logic for determining is this a payment to me becomes a completely separate logic from the logic of how do I sign for this. Right now they are intertwined in multiple ways. If we have a private key for something, then we accept the ... for the way this was... but not for a 1-of-2 multisig. We accept P2SH stuff-- the backup--- P2SH is ... supported in Bitcoin Core, it's like a shim for testing.

If you see a payment to a multisig address-- you have to remember... redo the add multisig and rescan the relevant. How are you going to .... your rescan plan should be stored in the wallet. So do you receive it when you do the backup, on the blockchain or when you rescan? We should not continue the logic of trying to automatically enable IsMine things.

IsMine is basically a half-assed copy of the signing logic that tries to determine can I sign for this, and if I can sign for it, IsMine is true. This is backup-hostile if you add more ways to sign. On the signing side, this is fine, your code should be able to sign for as much as it can, but that does not mean it should be treated as payments. What if we add to the keystore, a set of scripts, or scripthashes or even, it's not a map of scripts you know about, but these are the things I consider mine, and if it's there, it's mine. If it's not there, we'll still do the legacy lookup for pay to pubkey, but we don't try anything with the P2SH or whatever. And this means that if you have a witness-enabled chain, then the automatic rescan and add things to that map. You solve the legacy the same way by having the legacy chain add the two things to the map. This is true. As far as reserving the redeemScripts, the coinjoin bounty fund when it was announced-- to go recover these coins, you just have to go to the announcement and the current multisig is listed.

When we have these things like createmultisig RPC calls, it should come with a warning that you need to backup. createmultisig[address] should be create multisig wallet, it should be creating a chain for that, and you backup that. So if we're talking about the generalizing the chain thing, then for a particular chain you have some metadata that says how do the keys from this chain translate to addresses. And then you need some new english to describe this to a user. It's more complex because if you're talking about multisig it might be 2-of-2 together with another chain. I think all of this is out-of-scope for v0.15.1... I think we don't have to support all those things, but we should setup the design so that it evolves into what we think is a good direction.

I like the idea of adding to the keystore a set of addresses or scripthashes... they should be scriptpubkeys. Yeah. Variable size data. siphash of the... well, for bits? Well, alright. The point is that there should not be a secondary check anymore. That is the thing that decides... it could be a 256-bit hash of the... or a 128-bit is probably enough. It's effectively a hash table of your scriptpubkeys. It's not a hashtable, it's just a set. It doesn't map to anything. Independently you have the logic of... don't we already have that as the watchonly stuff?

We do need a map. .... which you need for signing things that have redeemScripts. That is used in the signing logic and that is also currently used to recurse in the IsMine logic, and I'm arguing that shouldn't be done. I think this already exists though, in watchonlys stuff. This seems conceptually to make the most sense. I understand why the wallet works like this and what its goal is. If I don't look at the wallet for a year, I start thinking it does... there's not an IsMine set, currently.

We have a set of CScripts in the watchonly set. That is conceptually what we need. I think it should be full scripts and also hashes of it. Let's ignore that distinction for now. I think that's easy to fix. Let's imagine full scripts for a second. When you create a... let's say a coinjoin... let's say you want to create a new 1-off address that you want to accept payment to, your backup.. you don't have to invalidate the private keys backup, you could have an additional scripts backup and maybe it references the path or location of the other thing. But this way you can from a user standpoint you can instead of invalidating the backup... it's less less. So there's two backups, there's one that is key material, which might be 32 bytes.

We should rewrite IsMine stuff. Key material can be separate, which would match the separation for signing versus matching anyway.

Nested vs native choices. Does this discussion have any bearing? Native versus nested. Supporting previously addwitnessaddress things would be hard in this model. They are imports. No. The script is in the wallet... but that's what I want to get rid of. If it's still there, then add it to the set. IsMine will just look at the set, when the wallet starts up, it will go through the wallet and build the set, the set is not saved in the wallet but it's built, it walks through the keypool, walks... No, it needs to be in the wallet. Why? You haven't solved anything otherwise- you're determining IsMine based on what you can sign for. You get the performance benefits of adding it to a set, but you don't get more. We should leave the existing logic in place which uses signability as a proxy for is it ours. Missed opportunity to clean up the IsMine code. Well, maybe IsMine cleanup can be done later.

You can have a one-time conversion operation where you start up, you don't have a set yet, you're going through all the keys, saving them, done, then going forward I'll add things to the set.

So, we're talking about the ideas of adding to keystore a set of watched outputs, scripts or scripthashes and then eventually stopping to use the ability to sign as a proxy for whether you consider it part of your wallet. The observation is that you want to be able to sign for as much as possible whatever weird combination your wallet is technically able to sign for, but that doesn't mean that anything you can sign for is going to be payments. There are I guess sort of ideological reasons for why you shouldn't try to accept for things that you haven't give out, but there are other reasons too like if you give out a bech32 address then you don't want to accept to a P2SH version because it's more expensive to you to spend from a P2SH and so on. So I think this makes sense as an evolution, but I don't think we're capable of doing all of that in v0.15.1. We've got 3 days, so maybe. Uh it was 5 days a moment ago?

Then the question becomes, how do we... assume this is how we do it: we just use the existing watchonly mac as the only thing for determining whether an output is yours. I think that would be an awesome simplification of the code. You drop the entire IsMine... we can't do it for backwards incompatibility reasons.. but this is just an idea that we are thinking about; you see an output you check is it in the set, yes it's yours, else not yours. If it's yours, then we can use a DummySign operation to determine whether it's signable, the sign logic goes away; it just goes to lookup in the set, and if so can I sign for it?

Can we do it now? Let's forget about addwitness for now-- we just, there are separate chains in the wallet, IsMine doesn't look at them for some reason... and then those get added to this new map. So you check IsMine and check this set. It can't just be a set, by the way. Why not? You need to advance the pool for top-up. That is a wallet... the set is in the keystore and it's the only thing that IsMine cares about. But once a wallet detects it, you still have to resolve it.

You have to make sure that either we don't care or there's a case where the fork is o.... by themselves they're not IsMine legacy in a sense. Yes that's fine. The multisig stuff is that, in the future, and even today 1-of-1, you would add a multisig template to the wallet, they generate chains, and then they generate the addresses. This is sort of heavy weight. This is an import address. Try to sign it, try to sign it, but we shouldn't do that. If you insert a thing, run the command, insert it into that set, no try to sign stuff.

Could this set only be for SegWit stuff? Yes that's what I would hope. But I'm not sure. I think that it will imply that we leave all the existing logic in place, but there's a front-up check is it in my set, and then we stop supporting the recursion thing for witnesses so all the witness logic is removed from the existing IsMine and replaced with this new stuff... We break compatibility... You can leave that historical business. But then you will keep accepting ... no, you don't look at new chains in that logic. No, there's no concept of chains in the keystore. But you could add a concept of chains. But you shouldn't. It could be a flag instead. This is a new key, you shouldn't treat it as something that is part of your old lookup code. Okay, yeah. This should be implementable. Great, but is it testable?

We could get rid of the recursion stuff, and then during load we add the p2pkh and p2pk version, but we lose... 1-of-1 multisig? You would need to be able to... all of the current watch scripts would need to be added as well, but that could be done later. I think that's trivial. I think it's trivial. You get the wallet to tell you the entire set of things you consider to be IsMine. It's going to be big. Have an upgrade path that blows out IsMine logic... no, this is only in memory. He wants to save the set. I think that the root of all these problems is that we used signability as a proxy for IsMine. If you're doing something in memory then you're not changing that. We do IsMine for watchonly... we should save that... .... old historical counterexamples that you need to save to disk. The new stuff, generated from the new chain is deterministic. I'm saying it's a optimization to say that when we're loading the wallet, if we're seeing a private key that is marked as SegWit use, then we add it to the set where we know it is added. It's a different way to store on disk. Using chains in the wallet as a means to determine how to store on disk. We decided to use separate chains previously.

I think that's what people want to do, but once implementing it, maybe it turns out to be a bad idea. What's the impetus? In half an hour it will be minus 2 days. Flag logic so you will have to accept whatever send at you, 1-of-1 bare multisig, if you have 1 chain you can never distinguish between... there's weird downgrade issues from v0.15.1... so you have this key... and then there's privacy issues, accepting transactions to a weird address, ... it's a good idea to try to support downgrading by one version. v0.15 wallet, and v0.15.1 code, and not upgrade the wallet. And then if you upgrade the wallet then sorry you need to use the new code. Make a backup of your wallet before you upgrade anyway, which is what you should be doing in the first place.

Logically, we want to be able to support people using bech32 from day 1.  The other question is, from the IRC meeting, the upgrade wallet method to switch to SegWit. There's add flags or something similar, or use the upgrade wallet thing that we have. I tend to lean towards the second but it means we have to support upgrading to HD as part of it. The next version is the HD stuff and we don't have good upgrade things. There's a pull request from dooglus about that. Has anyone reviewed that? The problem with adding a new flag is that we have to do it so that... support with non-HD wallets. I don't have great desire for us to do that... They are basically the same thing, HD wallet is a whole bunch of imported keys. The question is that if you have a wallet and you want to generate a new key.... The real question is do we want to take the delay on v0.15.1 to upgradehd? But if you want to do the upgrade for SegWit... we need an explicit upgrade. A new upgrade mechanism? I don't know. The question is do we need a new upgrade mechanism. Take an old wallet and start using SegWit, you need to upgrade. Does this require an upgrade to HD? Yes. It's a question of security because suddenly you have a mixed wallet. Any time you import, you have a mixed wallet. Maybe people expect to have... a single seed. They probably ignore the fact that they... well we should address this with a new warning at any time we do backups and say your wallet has imported keys in a warning.

The upgrade itself requires a new backup. That's fine, it doesn't break any kind of compatibility. There are users that will use dumpwallet and then write out the master seed and I think that's all they need... but in that text output, there should be warnings, and in the GUI for backup wallet, to say that your wallet has imported keys. With a pure hd wallet where you only do that, that would be nice.

This is a potential significant delay for v0.15.1... we already have this problem, the warnings? We already need that. It would be very nice if we had the RPC to rotate the HD master key. We're not doing that now. We have only 1 day-- 5 hours? The problem already exists, you can make mixed wallets already, guys online tell people to do this, ... the rotation stuff, you probably want to continue before you... at least for the... and encrypting the wallet doesn't do for you. As long as you have a long enough keypool. Well, make the keypool bigger ((laughter)). The default wallet from an install now is like a megabyte now. In the wallet we use a kilobyte entry... we use smaller records in the encrypted wallet. So we save an address in the wallet and we save it in this DER private key format, including the curve parameters, it's 270 bytes. It's 270 bytes for one key. And we start with 2000 keys now. So that's the default wallet. This is when you're not using encryption. So gzip it up when you're done ((laughter)). You use a more efficient encoding in the encrypted wallet case. Although we're also encoding nonce.

When you encrypt your wallet right now, what happens? With HD you do... for ... and then you have a backup key, it gets encrypted, it's still there, it won't continue to scan that chain. It will continue to generate new future... and encrypting a pre-HD wallet, just decrypts it? It dumps the... and any key in it marks as used... If you start with an encrypted wallet, does the HD restore check the old key you had before encryption for lookups? That's not needed because your new backup will include those keys already. It doesn't continue to explore that anymore. You have to rescan, encrypt, rescan, etc. But the wallet needs to get into a case where.. and we're so close.. we're so close to being able to not generate the wallet until you run a command that needs a wallet. There was a corner case that prevented this. But if we had this, then we could have the wallet be born encrypted. ... until you hit receive, in the GUI, it could even prompt you to encrypt it, prompt you to back it up, etc. I have encountered users that generated a wallet in 0.9 but are only just now using it. achow tried to do this and there was some corner case. The implementation is easy but there was a corner case... something else that made it hard.

Someone asks what about importprivkey with respect to supporting SegWit. Is this going to get added into 3 different copies in the... enumerated into all of them? We can add a argument to importprivkey where you say it's a privkey for this kind of address. So new rpc commands? importmulti. Can we upgrade to bech32? Not possible for v0.15.1... well that would have to bypass peer review, so no.

To make sure everyone is on the same page, we talked about switching to Pieter's proposal for a bag of keys and solver, we would switch entirely in v0.15.1 to using a set of scripts and on-disk storage would allow you to efficiently convert to a set of scripts. We talked about using the upgrade wallet mechanism for upgrading to SegWit which means we should support in v0.15.1 for upgrade to hd, which is hopefully a small patch, and use the same mechanism for upgrade to SegWit which uses a separate upgrade path? It doesn't need to wipe the wallet or rewrite the wallet. It's a strict subset of what ... like in terms of generating a new master key and keeps all the old keys around. It seems like a lot of work, especially the keypool, what about splitting the keypool. It will become a list of set anyway... list of sets. 3 sets? Internal, external, no 4 sets. IsMine only needs to know... just the keypool? Keypool is a separate concept. But now it's going to be 4 sets. 6 in fact, because of bech32. Are we doing a separate derivation path for bech32? Yes, it should be. I'm convinced. It's needed to among other things ....

For SegWit address versions later, how many... keypools are going to be. Multi-index? There will be one set for the IsMine check. But then you have separate keypools for each of the chains you're using. If you make a new wallet, it will only have the... To say that the old wallets... I'm not using these anymore, take them out of memory or anything, expire chain. Send funds to a new wallet, and archive. Making a new wallet should never be a requirement.

Here's a crazy idea. How much simpler would adding SegWit support be if we had a wallet that only did SegWit and didn't support any of the legacy stuff. Which we're kind of doing with multiwallet stuff. But it might be no simpler-- we need to support imported keys. We don't support spending coins from multiple wallets... you can't select coins across wallets. Okay, fair enough. And maybe that's the point of multiwallet, to not select coins across wallet. It reduces your accounting issue. It's a coinjoin.

I think we should take the time to go down this path even if it means we're going to take a while to do this, maybe we have to call it v0.16 or something. They are simple patches but they are invasive. Getting rid of the IsMine stuff is going to be months, here. Nah. The bigger patch is going to be the GUI changes. We could do it twice. Making it a priority is important, but putting a deadline on it... well, we're not going to put a deadline on it. People need to not lose funds. It's damaging to the ecosystem if people can't use SegWit. We don't want to proliferate a situation where people use addwitness. We're getting pull requests like "add accounts support to addwitness". "Add deprecated feature to deprecated feature", in other words. This is the bech32 stuff, IsMine stuff, hd upgrade, list of sets, .... This risks spanning into the wallet redo that we need. There are a lot of other things in the wallet that need to be redone.

I think it's just the things we just listed. Well that might be months. Can we break up the work into sections? What's the ugliest fastest way to do this, not separate changes. We should know what this is. What's our next option? If we want this by tomorrow, my approach would be to not distinguish bech32 from P2SH embedded witness, you always support p2pkh for anything you have given out a SegWit address for, we remove the requirement of having the redeemScript in your wallet for witnesses and instead replace it with a check for just do I have the pubkey and is it uncompressed and you keep the other one too, and you just relax it, and then you have a flag in your wallet and say this is how SegWit is enabled here, and then you have witness address for anything you add to the wallet. And this will work. This is probably 100 lines of patch. And how does this interfere with future changes? How much compatibility do we want in the future? Well maybe future wallets are incompatible with old stuff.

If we want to have support for the case where someone in v0.14.2 to use addwitnessaddress and then upgrades to whatever we think it's going to do, I think this makes it no harder, it just makes it slightly more bloated when you do it. If you say we're not going to forward support addwitness, I don't know how, some things become a lot easier here though. In the above summary, there is no new chains in there. If you have P2SH witness address given out, you automatically also support the native one, and if you have the native witness then you also automatically support p2pkh. You have the key, this is you have the key and the redeemScript and they're even equivalent.

How does addwitnessaddress work? It looks up the address in your wallet, tries to resolve it, figures out whether it's a P2PKH, fetches the pubkey, is it a P2SH then fetches the redeemScript, then if it is a pubkey then it creates a P2SH-P2WPKH... then tries to sign with it, with a dummy key pool that is a change that you made very recently, it does that for both, and then converts it to a P2SH address and then returns it, and then adds a redeemScript for that P2SH address as a CScript for the wallet, and then IsMine and the signing logic can recurse into it. So this implies that, if we don't get rid of the recursive IsMine logic, this is implication that P2SH witness and non-P2SH witness are equivalent because both require redeemScript to be present in the wallet, and then it implies that you are watching those keys as normal keys.

So I have two questions... you had a ?? in your wallet... and then you have to rescan from the point from which SegWit is activated, which is impossible for pruned nodes. If we do what you just said, does it make doing what someone laid out a minute ago any harder for say v0.16, and no I don't think so. We just need to support for all the needs for those things anyway. We need to support importing those. But we haven't locked ourselves into always.. pay the P2SH or the regular one.. addwitness is already doing that to ourselves. Let's make sure we don't lock ourselves into that. At this point I wish we never had addwitness. At this point people are telling others to use addwitness. We have to support those wallets anyway. In 6 months if we have a way to...you haven't locked yourself in a paradigm that if you give out bare witness you need to support the P2SH embedded, no I don't think we will get locked into that. We can lower the odds by communicating that. Probably many other wallets are going to have a from now SegWit-only approach. Ledger has a 'separate chain' they probably don't accept payments to the non-.... Well it's native versus non-native. They are not equivalent because of the fees. I don't know what's going to happen.

If people start doing this, it's not the end of the world because they are almost equivalent. We should address this with communication. What we do doesn't decide that outcome, it's what other people do too. To support both native and SegWit is to add them in parallel; it's what's easiest for us to do.

What was the requirement that they were relaxing to looking up the redeemScript? We don't need to do that for a minimum viable product. Actually maybe that is something to change when we introduce the new chain... it's almost better to not relax that. The change is now down to 12 lines of code. It's the entry to when we should start adding this... the MVP change also includes the change for sending to bech32. Well, that's separate.

Give me a new address and it spits out a native SegWit but you store both, the native and the ... are you talking about the minimum viable approach or the long-term right way approach? Uh, minimum. You don't store both because you don't need to. We already treat assuming that as soon as you add a redeemScript for a p2pk, that's currently the requirement for watching or treating as IsMine a witness output is that you have the redeemScript in your wallet. This is the only state change that addwitness causes-- it adds the redeemScript for the P2PK version of it to your wallet. Well okay, the reason I asked is because it would be nice at the GUI or client level that .. I've received these funds but not to the address where I've expected them.... Well..... ((groans)). It can't tell which one you gave out. We already have that problem, so the MVP doesn't address that problem. If we want to say that we're deprecating the addwitnessaddress going forward, or we want people to not use it, it would be nice to point this out to people. We can solve it to the key metadata that says which one we kept or generated the address as. That seems uncommon because nobody will do addwitness .....

It would be nice to create a Bech32 address that does not support the corresponding P2SH. We can relax this requirement. The P2SH version will always require knowing the redeemScript. For native witness output, this is not a requirement. This relaxation will be 5-10 lines of code. And none of these things have tests. If we make a relaxation which if we wanted it and it was the right thing to do, then you automatically have the ability to create a native witness output which is only accepted as a native witness output or a p2pk but not as a P2SH ... If we do the multichain thing eventually, then this is busywork, I agree.

Users explicitly asked for Bech32 getnewaddress. Or a command line flag that tells you what your change is. Maybe for MVP you can spend to bech32 but not give out your own address. I think we have to do that, I don't think the other thing is going to get done. Creating bech32... No he meant the bigger re-design. It's important for us to have something. I don't think it makes it worse, but we should clean it up for v0.16. It would need wallet upgrade option. It needs to store or hides... the height is when. The problem is that if you mark the height when SegWit activates. You don't need that either. You need to know where to rescan. If it's going to do it automatically then maybe you need to rescan for it. It might be a separate field.

Your wallet or node could be a different node.. move the wallet. It's not the activation of SegWit, it's activation of SegWit in the wallet. Why not use the activation ahead of SegWit. Pruned nodes can't upgrade then. It's past that point. It's already broken in this respect. Addwitness is broken in the same way that addmultisig is broken. So at least you implicitly triggered it each time. If you don't have this flag, then you could ... I don't know how that messes you up though. You need a flag, but it does't need to interact with the wallet upgrade. When you first run it, it adds that field if it's not there already.

I am beginning to doubt this again. You do an explicit upgrade operation to support SegWit operations... your wallet says you need new backup now. What's the problem? We don't have height for... well the only thing that might be a problem is if you were trying to recover from some earlier backup, but then you rescan, and you lost the height already anyway. It has the height for up to which it rescanned. If it is required to be in the backup, then you're good. So now we only need two hours to implement this. So we're doing the MVP? Pull request in 30 minutes.

I think someone should sketch this out. I think the point here is that.. if we don't do the MVP then we will be under pressure to get SegWit out. The nice thing about doing a MVP is that we can also tell that ... didn't Harding ask this morning can't we tell people to use addwitnessaddress, we can now tell people that's cool? ((laughter)) All the add- something RPC should have a warning that it needs a new wallet backup, including addwitnessaddress, if we do this then we can say it's supported but you probably don't want to. Well, never assume that users read the helptext. When SegWit activated, adding all of the... every time you get an address it just adds the thing. Add to wallet, add key to wallet function just does the thing. It calls addwitness address, including at load time, yeah. But there's no danger of having someone with pay to witness address of yours before SegWit is activated. We know this because Suhas went and checked to see... but there might be payments, but at least they are yours now. They have not been spent. There is an off-by-6 bug in my code where I accidentally activated SegWit 6 blocks after... but even if it's some fantastic screwup, the worst case is that there would be a payment to the wallet that it didn't make, it would cause confusion at worst. But if during the rescan we do this from the start not from SegWit activation, we would catch such cases. But the point we're saying is that we know... ignoring off-by-6, we know there have been no SegWit spends prior to activation.

How do you communicate to users that you can send to native SegWit but not receive? We're not going to generate Bech32 address? So to a user you can't receive SegWit. From a user perspective, this is confusing. You can send to this address format but not generate for yourself. No, we will probably support that. A user will see a P2SH address come out and say oh I don't support SegWit because I'm getting P2SH address. I think they will see a P2SH address and go "oh I support SegWit". If someone else gives you Bech32, you can pay that, but in the meantime you don't know if there's widespread support. There are crazy people who are giving them out now, like Armory(laughter).

In the bitcoin payment protocol, add SegWit to that? can we make a fallback? Well, then people will need extra code in URI handling to support this. Well we already have code in there. Give someone an address, and if they can't pay, then you give them another one. Coinbase is giving you an address by qr code or a link... let's set an example for other wallets. Support for Bech32 will probably go quite so fast, and we should plan for success, assume it goes well, and if it doesn't go fast then we should try to help it, and then we have learned a lesson which we need to know for other future upgrades.

Well lightning is going to use the BIP70 URI scheme, and the client can optionally choose to use the lightning stuff which is extra data in the URI. But it can also just fallback to that. The lightning rusty bolts should really be BIPs. 

You sign a transaction, it's witness malleated, and the malleated version gets mined, you might not notice that, a few weeks later you're looking at your wallet, you can't correlate that to what's in the blockchain. The txid will be confirmed. The transaction in the GUI will be like the wrong size.... The wallet behaves correctly, but what does the user expect? It's a good thing to fix, but it might not be critical. gettxoutproof might not... outputs not affected by witnesses. Does gettxoproof work with witness, period? It gives a Merkle branch between the block and transaction. You have arbitrary user, and verifytxout which will show you the witness, and that might be garbage. That's not good. BlueMatt will open issue. Also the issue for require some message about all the add RPC requires backup.

So in terms of breaking wallet, we just not.... I think we're too late. Plus 0.12 is about to be end of lifed anyway.

salvagewallet will take a ... on very first... you lose a ton of others. Including this is an HD wallet... Can we fix this? Chain split is basically version number. We could say salvagewallet is outlawed. No, we can rename it to savagewallet. Well, that's going to be hit by typos... yes actually that happened to me once, I thought hey that might be a better name.

In terms of parallelism, so, some of these things can, I think it, gonna be tiny, gonna take a bunch of tests. I have two PRs open. One to add support for.. one to get rid of.. those can be reviewed. We need an issue or milestone for v0.15.1. We need those 2 PRs, and this thing we just discussed, anything else we need? There's like whether your change goes to the... so maybe we should bring up that discussion again because we're a wider group now.

So the question of change-- should it be P2PKH, should it be P2SH-P2WPKH, or should it be native. One idea that was suggested earlier is you use the same thing as what you would use for a new address yourself which is a very simple... I'd prefer a defaults .... what you're making for yourself is a defaults question. What's the setting? If you do get an address and a Bech32 address, maybe that's what you do every time. It seems like to me like the way you do this is you do the change by default be this.. I'm really fine with anything. Aside from privacy... No, privacy is the only reason. Privacy in Core is kind of trash with change.. and maybe not in the future. It's not always trash, it's just mostly trash. Look at what you're paying to, and if we're paying to Bech32 then it needs to be Bech32. sendmany- it doesn't really generalize. If you any of them are Bech32, then make your change Bech32. So this might be a temporary privacy improvement. Well maybe it will increase the "privacy project" score.

Similar to the way that regular addresses will work, will you generate all 3 things for all the... change things. Yeah. Otherwise it would be too weird. This is something to fix at the same time that we switch to separate chains. This is straightforward- have a coin control option that says for my change use this one, use this type. What would be the default? I think the default should be P2SH witness at least for a while. Because of privacy. You want to match what you're sending to-- well not necessarily. If you're only the person spending to Bech32, then you don't want your change to also be Bech32, I think.

You shouldn't discriminate against less sophisticated users: say your transaction can become cheaper in exchange for some privacy; we have this massive performance improvement with dbcache that only sophisticated users know about. This is an argument against having settings. Well, there's a GUI option for dbcache size, but it's hidden in menus.

So are we going to add a GUI option for what type of change address?

P2SH .. witness. Default for change as of one is bare witness but that's still open for debate. If nobody debates it, then it doesn't change. Well it's up to whoever implements it. Joinmarket wants to use Bech32, openbazaar is using it, ... Alright, lunch.

<div id="p2sh-adoption" />
# P2SH adoption

How long did it take with P2SH before you could generally assume you could send it? Well, quite a while. You can't... it took years, but blockchain.info has P2SH support now, yes. They still have it. So right now if you send to a... they don't index it, you cannot see transactions by a certain address, it shows up as ... so everyone is using them because they are "private"? It took years-- perhaps faster around this time, but still some time.

blockchain.info didn't maintain their site for like 2 years. Armory didn't implement it for a long time, either. There's also special cases, like say you have a Bitcoin Core wallet. Coinbase was... bech32.. say I have an exchange ...  the particular case of sending to yourself from a service is a case that people will use quickly. Nobody wants to make an extra round trip. If I'm paying myself, I'll take it. I did have someone ask can we have the bip URI.. and I said, the problem is that it's even more things to worry about.

<div id="testnet" />
# testnet difficulty adjustment algorithm

Testnet difficulty adjustment algorithm is why the blockheight is high on testnet3.

<div id="bumpfee" />
# bumpfee and rebroadcasting

Do wallets commonly re-broadcast if the transaction is not in the blockchain after a few days? Anyonecanspend covenant utxo that it needs to create a new one of itself in its output. If you had covenants, you could somehow .... oh we could do it that other way, but it's not as cool as covenants. Replay protection with covenants would be a good use of covenants instead of some other uses of covenants. Segwit2x should add strong two-way replay protection.

<div id="libevent" />
# libevent

<https://github.com/bitcoin/bitcoin/pull/11227>

There are three layers in libevent. There's the base that deals with socket read/write events, then a layer on top of that which is their arrays or their buffers, whatever they call them, which have callbacks, and then buffer events on top of that like when I get data then do this etc. I started out at the most abstract level because I assumed it would be nice and clean. I started with buffer events because they were the most abstract and I thought it would lead to the cleanest code. There are a few little annoying things that don't work quite how we wanted it to, so I added a ton of hacks that ruined the abstraction anyway. So I went down a level and found the same problem. And then I went down to the most basic events, and it  ended up being the least amount of changes to the code. After rewriting it so many times, I was never happy with the diff and the readability. It's something that was straightforward, and now it's sort of-- it's going back. When it's callback based, the code is hard to read. I am satisfied now that it is reasonably easy to follow and see what's going on. "If this is set, set it to zero". I didn't do those because I pushed this Sunday night and I wanted to discuss it here.

The previous implementation was a single thread for networking and peers were connected to sequentially with a pause in between. But now with the event loop we can do multiple simultaneous connection attempts and do it all at once in an asynchronous manner.

New plan for Cory is we'll merge his work immediately, then we will just have Cory fix the problems as they show up, and we'll rename Bitcoin Core to Bitcoin Cory.

<div id="toolchain-builder" />
# Tool chain builder

How many turtles should we go down? Should this system attempt to be self-deterministic, or do you still run it inside of gitian to do builds? You have something that looks like depends, builds the toolchain for your system, builds whatever utilities you need, should that thing itself attempt to be produce a deterministic build between people? It would be nice, but might not be realistic. It would only be realistic if you kind of real blow-up what you build it. Maybe build your own shell, your own... It should build a compiler, a linker, and binutils. Not all the unix utilities.

Filesystem can impact build results, although we think we have fixed that. There's bit rot, and deterministic work done in upstreams, but a lot of it we can't use it because we're.... but if we were to assume we were using a way newer version of these tools, then we can eliminate a bunch of those tools, but we're relying on those tools. Doing a build of your own shell and busybox and building that stuff is kind of over the top. What's the minimal requirements to build? Maybe it's not that many. It needs a shell. It needs the build utilities. Busybox would be the easiest way to do it.

Compiler, linker, assembler are absolutely required. We should assume a linux system for builds. From linux, build host environment for OSX. Have a deterministic build from gitian that is the same, even if you build on OSX. It's pushing it, I recognize. Most of the time you assume build and host are the same. But maybe you build the toolchain on linux, and then run it on osx. It's about the same effort as supporting a new platform. wumpus does (non-deterministic) builds on openbsd and freebsd.

# other

If you change all the transactions in history to use indexes, that's an extra 30% reduction in chain size. So you get a 20% reduction using... but if you turn all the ... it's another 30% reduction. It's really good savings. You have to have an index of...

# other other

<http://bitcoin.sipa.be/bech32/demo/demo.html>

<div id="signature-aggregation" />
# Signature aggregation

Moved to <http://diyhpl.us/wiki/transcripts/bitcoin-core-dev-tech/2017-09-06-signature-aggregation/>

<div id="coin-selection" />
# Coin selection

First time around uses branch-and-bound. And then it falls back to the other method if it fails. It's one-shot. The branch-and-bound, first time it runs, it sees if it can construct... it uses effective values for the inputs. The argument I had for not using effective values is that it causes you to grind-down change more often. So the concern about effective value is that.. [bad things but since BnB creates no change it’s not a problem] 

Use the discard rate for deciding change. Calculating the cost of change and using discard rate, instead of the 1008 block estimate Say you have a 1k sat input and it would cost you 999 sat to spend it. You're happy to pick it, even though the fee rate you're paying is high. It's sorted by... given a list of inputs, we sort it first, in largest to smallest, and then we pick going down that. The window it has to hit is based on the discard rate. The discard rate is small compared to the total fee rate. The cost of creating a change is our current fee rate, so 34x. That's several thousand. Unlikely to pick anything smaller than that. Picks larger effective rates first. So it doesn't get into the window.

The effective value is ...  serialized or virtual size of the input (vsize), ... 148 bytes for non-SegWit, with your current fee rate, is used to compute the effective fee rate. Maybe exclude things that are smaller than 50% of their effective value. It's already going past zero and negative value. Might take multiple passes... done experiments where the smaller the window is... Instead of stopping at the first one, find as many that fit as possible, from the same branch down, and then use some other restrict to find the 'best'. If you take the first solution, go off and find the one with the highest discard rate but has the most number of inputs.

Change * fee rate is what we save if we don't create an output. We're using discard rate to be based on long-term estimate unless user overrides. If fee estimates are high, it will never create anything higher than the discard rate. Maybe be willing to discard a lot more, but even without discard a lot more, it would still produce changeless outputs pretty often.

branch-and-bound was implemented for a trezor bitcoinjs wallet. He reduced it to just the change output plus optimum I think was a 10th of the input cost.

This is a subset sum problem: there should be solutions once the number of possible subsets you can select is similar to the size of the value you are targeting. Once the number of solutions is much larger, then you could have many solutions to the size of the number you're targeting.

discardrate is a parameter that is how much you are willing to throw away extra fee to avoid making change output. It's also now used in branch-and-bound. The range is between the dust amount and the discard rate, somewhere within that range.

Avoiding a change output improves privacy. No change is better for privacy. So you have a cost that you are willing to pay for additional privacy. Adding this other waste metric.. and minimize that. Which will mean that we, instead of discarding more, instead of having less hits, have same number, and just wasting less money.

With branch-and-bound, the first solution is unlikely to be the optimum even with the same number of inputs. I just systematically combine and at some point I find something that is somewhere in the range of the amounts, because it is somewhere less than whatever the waste metric is. If we just continue searching, even if we were to allow the same number of inputs, we were almost certainly going to find the same number of inputs that has a smaller amount to be dropped. In branch-and-bound, this is self-regulating. If you have a larger fee, you can get few inputs to find an input set that will hit this allowance area. In the high fee situation, you have a small input set, which people should like because spending inputs increases cost of the transaction. It's likely going to take more inputs in combination to hit the exact target when the fees are low. We could modify the size of the window willing to tolerate based on our view of ratio of the current fee rate and long term fee rate. When we think the fee rate is low, we're willing to have an effective value that is a smaller percentage of the initial value. We could do with this with a preprocessing input, what percent of its value for each input is its effective value, we could do that. If the ratio between current and long-term, per input, and sort by that. We could also be strict and say never consider any input that is spending more than 10% of itself on fees, and then fallback to old algorithm.

Falling back... well, this sounds like it's optimizing only for some users. What do we need to do to do branch-and-bound in the subset sum approximation? It's not a bad algorithm. What are the objectives? What are we trying to optimize? If we had an objective function, the kind of knapsack solver we are currently using is not a bad approach. There aren't really good generic algorithms for solving these kinds of subset sum problems.

Small change can be bad. But why is big change bad? If you have too many big changes, you lose privacy. Optimizing for lots of small amounts of change is dumb. Optimizing for many inputs, that's good.

Say you can't find the exact match. Just randomly pick inputs until you find the amount until you're done. Well, that's branch-and-bound. Don't want to create dust, ... anything above a cent for change is fine. Maybe a directory of objectives. You want to spend all the inputs in the wallet and leave them with one output that they are stuck chaining off of, and now that output is unconfirmed, but with malleability fixed it's better now. Say they have a few big inputs you try to pay 20 BTC, you got like, some medium sized inputs and some big inputs, it picks the medium sized inputs and gets all of them. You should generally be greedy about spending inputs, because fees are going to be higher in the future. Ignoring what we know about high vs low fee periods, etc. But fees in bitcoin should probably be smaller in the future, so if the user is not going out to purchase more bitcoin in equal to their fee amounts, then you should minimize the fees they are currently paying. Say bitcoin is going to be $1M, then in bitcoin fees should go down-- or not-- in bitcoin fees?

Can a new algorithm use effective value? Sure, absolutely. But the concern is that if we drop in effective value into the current subset sum approximation, is that with a current algorithm we have, it would tend to not spend dust, and will grind up the outputs into little things and make a mess. I'm not certain of that, but I think my response of that is show it doesn't do this by running a simulation. Randomly picking seems less likely to grind it up into dust. As long as the algorithm isn't super-biased against picking low fee inputs, that's probably fine. And it can change based on what we think our relative fee rate is. We don't want to be in a situation where the wallet--- don't create outputs that we would be biased against choosing to spend later. This is what branch-and-bound does, it avoids creating change that ew would not spend. The existing algorithm just cleans up stuff at economic cost to you, sometimes. It tries to produce a small amount of change over the change target, which is unrelated to anything.

If you have a lot of change outputs, then you can split it into two change outputs, either spend your change into 50/50 half which is not great for branch-and-bound, or if your coin comes up heads, then you make one that is equal to the payment you're making, and the other is the remainder. So you have a transaction with three outputs, two that are equal, and one might be a payment or might be change. And half the time this will create an output that is going to be a payment the size of the payment you're making anyway, so future transactions will be cheaper.   If any services are freaking out about number of outputs, ....   bitpay will break if you pay them with two outputs, but I mean only change.

What are the objectives? If you pay a large amount you pay a larger amount, that might be a viable scheme. When the change is large, need to create multiple outputs. Never create outputs greater than 50 BTC, there's no gain to anyone in making any large outputs. You save yourself a couple bucks in fees in the future, and you stand out like crazy, adn you put all your eggs in one basket.

We can pick things by what's socially favorable to the network in terms of either overall privacy, overall utilization, utxo set size.   And then secondly there's different profiles or modes that a user can be using for users.

Constant fee rates make it easy to compare algorithms. But we have fee variation in reality, and this makes coin selection more interesting. Companies are consolidating utxos on the weekend. They do largest first, and then consolidate on the weekend. We see this behavior already.

Actually, a certain company (not one you might expect) is really interested in having us help them figure out coin selection for them. They are willing to setup an API for bitcoin developers to submit coin selection code and they will return the results, but not give the transaction data. So they are actively asking for help.

We should not be detecting based on the fee rate whether to aggregate inputs. Looking at fee estimates and trying to figure out what ... with random selection, it's biased towards aggregating.

Sort by effective value, keep adding them top down, and if we haven't at that point then we literally can't make the payment. Litecoin added 800 MB to their utxo set early on because someone sent 1 sat to everyone. And you had like a million small utxos and the coin selection algorithm failed because it could never find your real outputs.

If you go above the target, you cut off that branch. This gives you a small corridor in the binary tree where you actually have to explore in order to find all possible solutions. You would usually probably be broadly in range of that amounts maybe at the top of it, and you're throwing away a lot of money that you don't want to do that-- we want to minimize the waste. What could be the metric to measure the waste? The idea of this right now just take the first and we would continue searching and then depending on how many inputs our smallest solution has, we would calculate how much fees we pay for the inputs that we would not necessarily need compared to the long-term expectation of what an input would cost, plus the allowance. So we had two inputs, we find a solution that hits the allowance window in the lower area, so it's less give to the fees, but if we add two more inputs at 100 sat/byte, then the user is wasting money because he could send more to the fees but later when it's cheaper. Well we don't know it's going to be cheaper.

We have a lot of different things in the metric. Right now we just strictly prioritize things. We already have 6 confirm stuff, then 1 confirm stuff, then 0 confirm stuff, and maybe we want to treat every address as one unit, then you go on.... I'm not sure that discardfee. Maybe minimize discardfee. No, just whatever. We could minimize number of inputs, minimize amount of fee we discard....

<div id="release-validation" />
# Release validation

I have a pubkey in the binary. A download tool to check the binary that you've downloaded, as an alternative to asking the user to manually check that the release was signed by the bitcoin developers. verifysignature is not good for this. sha256. I am just saying minimum work. That would be a small number of lines.

<div id="jenkins" />
# jenkins

jenkins has standard IRC integration and we could use it to trigger builds. It also has some interaction with github pull requests. The advantage of running things on not travis-ci is that we could run a full set of tests because we wouldn't be limited to the same machines that travis-ci provides. You could also guarantee that there's nothing else running concurrently. You can build in parallel, still.

<div id="conflicting-transactions" />
# Conflicting transactions for withdrawals

We're trying to prevent transactions from expiring. We treat our change outputs as being spendable immediately. With SegWit we can avoid malleability. We can immediately spend the output from the last round in our transaction. And then we have a long-chain of unconfirmed transaction. Then we need to compute the fee rate to bump the entire chain.

You take all the unconfirmed transactions in the graph and you compute their size, you look at the fees, and then you say what's the size of the whole package, and what's the fee for the whole package that needs to be added. So for every withdrawal, update to the current fee target. In Core, it'd be nice to do replacement stuff.

Treat your own unconfirmed outputs as trustworthy, and deposits keep the confirmation depth requirement. Other people can chain off your stuff and mess it up, instead....

Do child pays for parent instead of replacement. CPFP for chains of 25 transactions. At some point you have to stop doing withdraws. An exchange was anchoring his replacements, which caused problems. Replacement is a lot more efficient.

200 byte transaction that pays 30 sat/byte. It gets spent by a 1 sat/byte 100 kb transaction, but my bump to replace it has to pay an enormous amount of sat/byte in order to pay the thing it bumped out. It doesn't have to be the fee rate, it has to be the thing it bumped out.

<div id="mast" />
# Merkle trees and MASTs

Moved to <http://diyhpl.us/wiki/transcripts/bitcoin-core-dev-tech/2017-09-07-merkleized-abstract-syntax-trees/>