Name: Christian Decker

Topic: Back the f\*\*\* up

Location: Potzblitz (online)

Date: May 3rd 2020

Video: https://www.youtube.com/watch?v=kGQF3wtzr04

# Intro (Jeff Gallas)

We’re live. Welcome to the fourth episode of Potzblitz, the weekly Lightning talk. Today we have a very special guest, Christian Decker of Blockstream. They also call him Dr Bitcoin. Our co-host today is Michael Folkson from the London BitDevs meetup. He is doing a bunch of Socratic Seminars and has a deep technical knowledge. He will be the perfect person to speak to Christian and has already prepared a bunch of questions. We announced this on [Twitter](https://twitter.com/michaelfolkson/status/1256204870597541888?s=20) a couple days ago. He has been collecting questions. If you do have questions during the stream put them in the YouTube comments or ask them on Twitter using the hashtag \#potzblitz. We also have a Mattermost channel mm.fulmo.org, the channel is called Potzblitz. You can find Christian there, he will be checking it out later. As always once we are done streaming you can join the Mattermost channel and the Jitsi room and hang out and ask some more questions. I expect that we will not be able answer all of your questions today. Join us in the Jitsi room afterwards. We are using Jitsi to stream this. It is an open source video conferencing tool. It usually works quite well. Sometimes it doesn’t but most of the time it does. If there are any glitches please forgive us. It is permissionless without collecting your data. This is a pretty good option. If you like this give a thumbs up, subscribe to the channel so we can do this another time. 

# Back the f\*\*\* up (Christian Decker)	

Thank you so much for having me. I hope people will excuse my swear word in the title. It fit in quite nicely with my feelings about backups or the lack thereof in Lightning. The title today is back the f\*\*\* up. How to backup and secure your Lightning node in a way that you cannot end up losing money by operating your node. Like Jeff said I am Christian Decker, I work for Blockstream hence the small logo up there. I work mainly on the c-lightning implementation of the Lightning protocol of which I also help out a bit. You might notice that later on because I tend to emphasize what c-lightning does correctly. More on that later.

# Why Backup?

First of all why do we want to backup in the first place? I think everybody will be shouting right now that we are actually handling money and so the rationale for having a backup of what is handling our money is pretty fundamental. The way to do backups in onchain systems like your Ledger wallets or your Bitcoin Core nodes or systems where you perform onchain payments is basically just write down 24 words. If you ever destroy or lose your device you can recover from those 24 words which seed a random number generator which is used to generate addresses. This is a common thing in the onchain world where you have your 24 words, you write them down on paper, you send half of them to your parents or you engrave them on metal. All in the name of making this as disaster resilient as possible and ensuring that you can recover. That is quite an easy option. We can do that in Lightning as well. However it turns out backups in Lightning are quite a bit harder. They can even be dangerous. To see why they are dangerous let’s quickly go through how not to backup a Lightning node.

# How not to backup

If you are anything like me you probably end up with a couple of backups like this where we have the original `.lightning` folder. Then at some point we decided we might want to take a backup of that. We rename that directory or copy it over to something with an extension. Let’s call it `.lightning.orig`. Then we do some more changes and we perform another backup. We call it `.lightning.v1`. But we are computer scientists so we start by counting at zero. Even our naming scheme becomes very inconsistent over time. That is not the whole issue. The whole issue is that even if we were able to take backups of our Lightning node the act of restoring it can put some of our funds in danger. Simply because of the mechanism that we use in the Lightning Network to update state. To see that we need to look at what the update mechanism in Lightning does.

# How the update mechanism in LN works

In Lightning we use a mechanism that is called a penalty based mechanism in which each of these square boxes here represents a state. Initially we have the initial state where person A has 10 Bitcoin and the other participant in the channel has zero. We transfer 1 Bitcoin over. Suddenly we have two states that are active. Once we have agreed on this state we poison the old state. If any participant ends up publishing this transaction representing our state this poison would be activated and the misbehaving node that has misbehaved by publishing the old state would lose all of its funds. We do that a couple of times. Transferring 1 from A to B, then we transfer 4 more from A to B. We then poison the old state. This state is currently active. Then we perform another update. This is a bit special. We don’t actually transfer money but these state transitions can also occur because we change the fee that we need to pay onchain. Therefore even non-user triggered actions can result in a state change that would end up agreeing on a new state and poisoning the old state. Finally we take this last state and transfer from B to A one more Bitcoin. This final state which does not have the biohazard symbol, the only state that can be enacted without incurring a penalty. All of the prior states are poisoned. By poisoned I mean the counterparty always has a matching reaction that could be used to punish if I were to publish this old transaction. If we were to take a backup anywhere here we might end up losing some information.

# So what’s the problem?

The problem is exactly that this is a copy of the previous image. Both A and B agreed on this state being the last state. However, because I took a backup of this state and then had to restore this state for me this appears this is actually the last valid state and I should be able to send that state onchain. Get my 5 Bitcoin. However, that would be cheating from the point of view of my counterparty because they have a matching penalty transaction for this state. If I were to try to enact this state then I would lose my whole state in this channel. Not only is backing up and restoring a Lightning node rather hard but it is also quite dangerous because if you end up restoring a state that isn’t the last one you might accidentally end up cheating. Over the time we the Lightning developers came up with a range of possible solutions to mitigate this situation or even prevent it altogether. Sadly none of these are complete or gives you a 100 percent protection but we are getting closer. What I’d like to do is step through the various mechanisms that have been used previously. What we are working towards right now when it comes to securing the data that you have in the channel.

# Backup to Google Drive (eclair)

The first one is something that eclair did right at the start when we started rolling out these Lightning nodes. The developers of eclair were getting some feedback that people were losing their devices with the Lightning node on them. They were unable to recover the funds on those phones. It is quite natural if you have a phone. Phones have a tendency to drop into water or break or just get lost. If you have money on the phone that is doubly sad. What the eclair team did was they added a trivial backup mechanism where you take the state of the Lightning node after each change and push that to a Google Drive folder from where they could restore the entire state. I’m not too familiar with how this was done. Whether it was done incrementally or whether it was done in a batch. However this mechanism seems not to be working anymore. Simply because people soon figured out that if I start my eclair wallet on Phone A and I restore on Phone B I can run this node from both phones at the same time. They share data withe each other. If you end up restoring a node that is not the latest state and you end up cheating, that same thing happens if you have two nodes running against the same dataset pretending to have the same channels and therefore contradicting each other while they proceed in the Lightning protocol. This may end up looking like a cheat attempt and the node being punished. This first attempt was mainly used for recovery but people abused it a bit to share a single node across multiple devices. So don’t do this.

# Static channel backups (lnd)

A better solution are the static channel backups that are implemented in lnd. As you can see on the left side this is basically the structure that a single channel static backup looks like. It has a bunch of information that relates to how the channel was created, who the counterparty is and all of the information that is static in the channel and doesn’t change over time. What this allows you to do is start the lnd node back up again with this backup file and restore what they call a channel shell. Not a true channel but enough information about the channel so that you can reconnect to your peer and ask for information for you to get your funds back. This is rather neat because it allows you to take a static channel backup, store it somewhere offline, it is probably not small enough so that you can write it on a piece of paper and write it back in when you want to restore it. But you can store it on a USB stick or a file server somewhere. That alone should allow you to at least get your funds back. It is minimal in that sense. It doesn’t change while you are renegotiating the state of the channel itself. And so for each channel you take one backup and you are safe until you open the next channel. The downside is that it requires you to be able to contact the counterparty node once you attempt to recover simply because this information that we track in the Lightning protocol cannot be recomputed solely from this structure. But you need to reach out to you peer and ask it “What is the current state? Please give me all the information to recognize and to retrieve my funds after closure. Please close the channel.” This reconnects and tells it to close the channel. The downside is of course that on relying on the peer to do something for you might be get you into a weird situation because you are telling the peer “I lost my data. Can you please help me recover my funds?” The counterparty could not give you that information or it can refuse to close the channel on your behalf that results in them holding your funds hostage. All of the current implementations implement this correctly and will cooperate with lnd to recover their funds. But a malicious actor could of course change the code so that lnd nodes cannot close successfully or recover successfully. While this is not a perfect backup it is an excellent emergency recovery mechanism to claw back as much of your initial funds as possible.

# Synchronous Database Log (c-lightning)

The third mechanism that we have, this goes more into what c-lightning is viewing as the correct version of doing this, is the ability for plugins that you can attach to c-lightning to keep a synchronous database log in the background on whatever medium you want to do. The reason why we made it into a plugin is because we don’t want to tell you how to manage these backups. We want you to have the ability of incorporating them into your own environment. This just tells the plugins every database transaction that would result in a modification of the state before we actually commit to this transaction. It is a write ahead log of the database transactions that we track before committing too changes. Depending on your infrastructure you might want to have an append-only log. You can have compressible logs or you can even a replica of the original database being managed concurrently to the main database. If you do that then you can failover immediately if your node dies. What you do is have a replica that tracks changes along the main database on a network mount and if your node dies you can spin up another node and connect to your replica. You can continue where you left off. We don’t rely on external help, we don’t need to interact with our peer which may not even be available even more. But the downside is of course that these are synchronous backups that require us to write to the backup every time that we have a state change. This is one mechanism.

# Database Replication and Failover (c-lightning)

The other mechanism that c-lightning is the database replication and failover. We have abstracted the database interface in such a way that we can take c-lightning and have it talk to a Postgres database for example. Whereas the default database is a sqlite3 database. Postgres does require a bit more set up. It allows you to also have a synchronous replication of your node and even transparent failover from one database instance to the next one. This is something that enterprise users are very much interested in because they can rely on what they already know about replicating and mirroring and failover on Postgres databases which is very well known in the wider software engineering world. Whereas replicating and scaling a single Lightning node that is very niche knowledge. We can reconnect to what institutional knowledge there already is and people feel pretty comfortable with the way that databases can get replicated and secured. I mentioned this is more of an enterprise set up. It also comes with a bit of upfront investment cost to set up all of these nodes, set up the database and have replication set up. But gabridome has written an excellent [guide](https://github.com/gabridome/docs/blob/master/c-lightning_with_postgresql_reliability.md) on how to set up Postgres in combination with c-lightning. I can only recommend reading that and seeing if you can get it working as well. The upside is obviously that restore is immediate because we have replicas that have the up to date information. If the node dies we just spin up a new node and if it connect to the database. If the database dies we switch over to a new master database and the node doesn’t even learn anything about the database having failed. This is another form of synchronous replication.

# The future (eltoo)

Of course this wouldn’t be a talk by me if I weren’t to somehow get eltoo into the picture. Eltoo is a [paper](https://blockstream.com/eltoo.pdf) that we wrote two years ago now about an alternative update mechanism in which we don’t penalize people anymore for publishing an old state. Instead what we do is override the effects that your misbehavior would have and enact the state that we actually had. What we have here is the set up transaction that creates a shared address between the blue user and the green user. We create a refund transaction that gives the original owner back the funds after a certain timeout expires. The way we perform updates is by having an Update 1 that takes these ten Bitcoin here, ratchets them forward and attaches this Settle 1 in which both parties have 5 Bitcoin each after this timeout expires. Then we update again by ratcheting these funds forward and attach this settlement with the newer state. This time the green user sends 1 Bitcoin over to the blue user and this update overrides whatever would have happened in Settle 1. Instead of punishing the counterparty for misbehaving what we do is override the effects that they would have had. Let’s say in this case the green user would like to enact this settlement. It sends out Update 1 and now has to wait for this timeout to expire. In the meantime the blue user publishes Update 2 overriding this Settle 1 and starting the timer on Settle 2. We no longer punish our counterparty for misbehaving but we override the effect and get to a state that was agreed upon in the end. If I were to backup and restore, restore at the time that Update 1 and Settle 1 were the last state then all that would happen is that my counterparty would send out Update 2 and correct my mistake instead of hitting me in my face by stealing all my funds. This is something that would allow us to have a much more robust mechanism of backup and restore. I think ultimately this might become the way that we end up doing stable nodes sometime in the future. Many people have argued that this loses the penalties. However as far as I can see the penalty mechanism is intrinsically linked with our inability to create backups and restore because whenever we backup and restore we might end up in a situation where we can be penalized. I think we might want to distance ourselves from this satisfaction of punishing people and go to more a collaborative solution which in this case would be eltoo.

# How about a unified backup standard?

I have just given five different backup standards. How about creating a unified one? Everybody seems to be pulling in a different direction and everybody is experimenting with different ways of backing up and restoring. Wouldn’t it be nice to have a backup that can be created by c-lightning and then later be imported into lnd and restore from that? The answer is of course. That would be really nice if in the end we have this convergence of different ideas and different implementations where we agree on a unified standard for these kind of operations. However, currently at least the internal structure of the data that we track is not in a form that we can share these backups between implementations. Maybe we will eventually get there but currently that is not the case. I would argue that this experimental phase which we are currently in is good because it allows us to experiment with different trade-offs and come up with better ideas in the long run that are then capable of getting a majority of the ecosystem behind it. Actually make this a better standard overall because this has always been the way that the Lightning development has worked. We have always gone off into different directions and then emerged back and shared experiences from the way that some of our approaches worked and others didn’t. My hope is that we will end up with a unified standard.

# Resources

This is my last slide which is the resources starting with the cloud backup by ACINQ, the excellent article on recovering funds from lnd using static channel backups. We also have a couple of backup plugins and documentation on the `db_write` hook. And of course how you can wire c-lightning to run with a PostgresQL database. Gabriele’s excellent tutorial on how to set up replication with PostgresQL so you can have the real enterprise feeling for you node. Read the eltoo paper. I think it is neat but of course I am biased. That is it from me.

# Q&A

Questions were collected [here](https://twitter.com/michaelfolkson/status/1256204870597541888?s=20) on Twitter.

Q - On backups, I think you alluded some of the trade-offs. There is ease of use, there is trust in terms of requesting certain information from your remote party, there is security, there is cost. Are there any other trade-offs and which of those align towards a technical user, a non-technical user and an enterprise?

A - One very important one that I failed to mention before is that these backups come with some latency. Your node, depending on which backup you choose, you might end up sacrificing some of the throughput of your node simply by having to reach out to a backup plugin or to a remote storage or some other operations that you need to do in order to make this safe. The only option that doesn’t incur this cost is the static channel backups because they are done once at the beginning of node set up and every time you open a channel. There is no additional latency cost that you incur at each update. Whereas the full backup solutions like pushing onto a cloud server or having some plugin, that will obviously incur throughput cost. I think the static channel backups are probably the easiest ones to set up. We are trying to make it as easy as possible for the plugins as well. We do have a plugin repository that is community driven. We do have people working on Dropbox backends where each single update gets written to Dropbox and then we have… working in the background. It should end up being a plugin that you drop into a folder and everything is taken care of from thereon. Currently there is some work that you need to invest to get these working. Of course the extreme example is the replicated database backend which really takes some digging to get right. Not for the faint hearted I guess but we are getting there.

Q - I imagine if either the lnd set up or another implementation comes up with a backup strategy that takes those trade-offs in a different way and is popular there could be a c-lightning plugin that replicates it?

A - That is how our development works as well. I wanted to point that out in the second to last slide which is we do learn from what other teams do. We do bring some of these innovations back to the specification to be discussed. If there is a clear winner when it comes to usability and other trade-offs then there is absolutely no hurdle from having that become part of the specification itself. At which point it becomes available to all implementations.

Q - You alluded to this. What exactly does need to be backed up? Chris Stewart did a very good [presentation](https://diyhpl.us/wiki/transcripts/lightning-conference/2019/2019-10-19-chris-stewart-private-key-management/) at the Lightning Conference last year on the six different private keys that you need to think about with Lightning. Some need to be hot, some can be cold. What exactly needs to be backed up in these different scenarios?

A - Obviously the first one is the seed key that you use to generate all of your addresses. That’s pretty much identical to onchain. It is what creates all of these addresses that are used throughout the lifetime of your funds. When they enter or leave a channel they will end up on addresses that are generated from that. That is the central piece of private information that you need to backup. We have a variety of parts where keys are used that are not directly derived from this seed key. Namely whenever you and your peer with which you have a channel open need to generate a shared address. That will not be part of your derivation tree that is rooted in this seed key. Those keys need to be backed up along the way. This includes HTLC secrets. Whenever a payment goes through your channel you had better remember what the key was that this payment was contingent on. The indices of the channel in the database because that is the index of the derivation path that is going to be used. And revocation secrets, those are important because that is the poison we use to keep each other honest. My absolute favorite is the per commitment points which is a ECDSA point that is added to your direct output if I close the channel. Without that point you wouldn’t even recognize that these funds are supposed to go to your node. That is a really weird use of these point tweaks because they are in there to make sure that each state commitment is unique. It just makes it so hard to create an onchain wallet for Lightning. When I stumbled over those and I had to implement the wallet I just swore.

Q - What about watchtowers? Is there a role for watchtowers with backups? They are predominantly there so that you don’t have to be online the whole time so they can stop someone from cheating you. Is there a role for them to store some state on your behalf? Do you see the role of a watchtower expanding and different services being provided around a watchtower?

A - Absolutely. The role of a watchtower is definitely not clear cut. I usually tend to think of a watchtower as a third party that I put in charge of reacting when something nefarious happens onchain. But of course this definition can be expanded to also include arbitrary data storage on behalf of a node in exchange maybe for a small fee. I wouldn’t say there is a clearcut boundary between a watchtower and a backup service. I guess the name implies what is the primary use for watchtowers, watching your channels to react if your counterparty dies or does something nefarious. Whereas backups are there to store your database changes so that you can recover the state later on. If we add a way for nodes to retrieve data from a watchtower it basically becomes a backup server as well. A little known secret is that the punishment transactions that we send over to a watchtower are actually encrypted. The encrypted data could be anything. It could be your backup. If you ever end up needing an arbitrary data storage just pay a watchtower to store it for your and hope they give it back to you if you need it.

Q - Are you planning on implementing watchtowers in c-lightning?

A - Absolutely. For a long time we’ve seen c-lightning as more of a hosted solution where you run your node at home or you run it in some data center and then you remotely connect to it. Our nodes would be online 24/7. But there has been quite a bit of demand for watchtower functionality as well. What I did, it is not yet released, there is a hook for plugins that pushed the penalty transaction for each of the prior states to a plugin. A hook is a mechanism for us to tell the plugin some piece of information in a synchronous manner so that the plugin can take this information and move it off to somewhere else, your watchtower backend for example, before we continue with the process of the Lightning channel itself. That way we make sure that the watchtower has the information that needs to be stored before we can make progress on the channel itself. By pushing that into a plugin we can enable our watchtower producers to create their own connection or protocols to talk to watchtowers and we don’t have to meld that into the Lightning node itself. It is very much on our roadmap now and I think I have three variants of the watchtower hook. It should hopefully be there in the next release.

Q - With end users or people with private channels on the edge of the network versus routing nodes in the middle, do you think the backup strategy or backup setup will be different for those two participants on the network?

A - I do think that as the routing nodes inside of the network start to professionalize they will probably start professionalizing their data backend as well. I foresee mostly businesses and routing nodes starting to use more and more the replicated database backends. For end users I do think that the backup plugins provide a very nice trade-off between having the functionality of a backup but not having to have the big setup or upfront cost of setting up a replicated database and automated failover and all of that stuff. Those are the two rails that we have chosen and cover these two use cases. Professional node operators and your geeky son that sets up a node for the rest of the family.

Q - If you are the end user on the edge of the network you don’t have to worry so much going down at the time that you are routing the payment. That is the big challenge here. You need to have that latest state. If there is some bizarre timing that is where it becomes technically difficult.

A - I have recently done a restore of one of my nodes that actually does quite a bit of probing in the network. I think it had a week’s backlog of changes and it recovered in 7 or 8 minutes. That might be too much for a professional operator but for me sitting at home not doing my grocery shopping or not buying anything on Amazon, 7 minutes of downtime while I restore my node is perfectly fine. Even then we do have trade-offs. We can have backup plugins that replicate the database in the background. Then it takes a couple of seconds to failover.

Q - You mentioned eltoo. One of the questions was when SIGHASH_NOINPUT? Obviously you don’t know when but is there any chance of getting it into the next soft fork with Taproot? Or has that ship sailed? If that ship has sailed what is viable for getting it into Bitcoin if we ever get it in?

A - It would be awesome if we could get SIGHASH_NOINPUT or ANYPREVOUT which is AJ (Town)’s proposal into the soft fork that is rolling out Taproot and Schnorr. The problem being that all of the reviewers currently are focused mainly on Taproot. It is really hard to grab people and get them to discuss yet another proposal that we might want to have in the same soft fork. While I think it is still possible I’m pretty sure that it will be a secondary, maybe lighter soft fork at a later point. Maybe bundled with some more lightweight proposals that require less changes to the structure itself so it is easier to review as well. See what kind of parts of the codebase are being touched and what the side effects are. Because Taproot and Schnorr are a big change all as well. That being said AJ has taken SIGHASH_NOINPUT and with his ANYPREVOUT he has formulated all of the details that need to change in SIGHASH_NOINPUT for it to nicely mesh with Schnorr and Taproot. With his proposal we could end up with a very nice bundle of changes that can be deployed at a later point in time independently of Taproot and Schnorr. I remain optimistic that we can get it in but I can’t promise anything.

Q - Do you have any thoughts on whether planning for the next soft fork can really happen seriously until we have what is likely to be, no one knows, the Schnorr, Taproot soft fork first? Can there be a parallel process or do we really need to get all of those Core reviewers and brains on the one soft fork before we even get serious about any other soft fork?

A - I think it is probably best if we keep the distractions as low as possible while we do an important update like Taproot and Schnorr is. I personally don’t feel comfortable creating more noise by trying to push hard on SIGHASH_NOINPUT. I think Schnorr and Taproot will get us many, many nice features and I think we shouldn’t hold up the process by trying to push in more features while we are at it. Everybody wants their favorite feature in and I don’t see this particular feature being so life changing that it should jump the queue. I would like to have it in because I think eltoo is a really nice proposal. Self congratulating again. Hopefully we will get it at some point but I don’t think we need to stop the entire machinery just for this one proposal. Who knows? We might come up with better solutions. I am under no impression that this proposal is perfect. The more time we can spend on improving and analyzing the better the outcome in the end.

Q - Have you been following the conversation on activation? Do you have any thoughts? Is there any way to speed up the process or is this going to be a long drawn out discussion?

A - Activation is a topic that many people like to talk about and it is a very crowded space. I have a tendency to stay out of these discussions where you already have too many cooks in a kitchen. I think whatever the activation mechanism is it is going to be perfectly fine if it works. I don’t really have a preference.

Q - For the researchers and developers in the space where should one put their efforts for the maximum positive impact on the Lightning Network? Obviously you do a lot of stuff. You are a researcher, you contribute to papers, you contribute to c-lightning, you do a lot of work on the protocol and the BOLT specifications. How do you prioritize your time? Do you follow the Pieter Wuille work on whatever is most fun or do you try to prioritize the things that you think are the most important? 

A - Definitely choose topics that interest you the most. I did my PhD on this and it was amazing because you could jump around the whole space. It was a big green field where you could start building stuff. It was all novel and new. I get the feeling that Lightning is the same way. All of a sudden you have this playground which you can explore. Don’t limit yourself to something that might be profitable or the cool thing. Do whatever interests you. For me personally I enjoy breaking stuff. I will always try to find out things that can be broken and see if I can get away with it within limits. Lightning is cool to explore and see what you can do. If you want to increase security through watchtowers that is a good thing. If you want to increase privacy by figuring out ways to break privacy that is another good thing. Whatever you like the most I guess.

Q - Where is the most impact you can make? We only have limited time and resources so how do you think one person can have the biggest impact?

A - I do think that there are quite a few statements that need to be proven or disproven. In particular one thing that I like doing is attacking the privacy of the network. Of course we now have completely different trade-offs when it comes to privacy from onchain to offchain. We don’t leave eternal traces of our actions like we do on a blockchain. We do leave those traces on Lightning but we do talk to peers. What might these peers infer from our actions? What information could they extract from that? There is definitely the privacy aspect that is worth looking at. Then there is network formation games. If you have ever done game theory, finding a good way to create resilient networks but also networks that are efficient is a huge question. How can we create a network where each node individually takes some decision and we don’t end up with one big node in the middle becoming the lynchpin, the single point of failure where if that one node goes down everybody starts crying. That is definitely an open research question. Also more fundamental stuff like how can we improve the protocol itself to be more efficient? How can we gossip better? How can we create an update mechanism that is more efficient, that is quicker, that needs less round trips? We do have one engineer, Rusty who is in Australia and he is always the guy who will try to get that half round trip shaved off of your protocol. That is his speciality. He has a focus on that. I wouldn’t say that there is an official list of priorities for the Lightning Network and associated research. It is you making your own priorities. People tend to congregate on certain tracks.

Q - What are your thoughts on the privacy papers? I believe you were the co-author of at least one. Could you give a high level summary of those conclusions and thoughts?

A - I started collecting data about the Lightning Network the moment we started. I do have a backlog on the evolution of the Lightning Network. I have been approached by researchers over time who want to analyze this data. How did the network grow? What the structure is, what the success probabilities are. That is how I usually end up in these research papers. So far all of the analyses are very much on point. Purely the analysis of centrality in the network and the upsides and downsides. The efficiency that we get through more centralized networks. The resilience we get from distributing the network more. All of these are pretty nicely laid out in these papers. In some papers there is some attempt to extrapolate from that information. That is usually not something that I encourage because these extrapolations are based on the bootstrapping phase of the Lightning Network. It is not clear that these patterns and behaviors will continue to exist going forward. It is mostly these extrapolations that people jump on when they say “The Lightning Network is getting increasingly more centralized and will continue to become so.” That is something that I don’t like too much. The other one is that people usually fail to see that the decentralization in the Lightning Network is fundamentally different from the decentralization in the Bitcoin network. In the Bitcoin network we have a broadcast medium where everybody exchanges transactions and by glancing at when I learn information about which transaction I can infer who the original sender is and so on. In Lightning the decentralization of the network is not so important because we do have quite robust mechanisms of preserving privacy even though we are now involving others in our transactions. We do have onion routing, we do have timing countermeasures, we do have mechanisms to add shadow routes which are appendices to the actual route pretending we send further than our destination. We fuzz the amounts so that the amounts are never round. There is never a 5 dollar amount in Bitcoin being transferred exactly over the Lightning Network. We have recently added multipart payments where payments of a certain size are split. Unless you can correlate all of the parts you will not even learn the exact amount being transferred. All of these things are there to make the network more privacy preserving. Is it perfect? Of course not. But in order for us to improve the situation we have to learn about what is working and what is not working. That is my motivation behind all of these research papers. To see whether our mitigations have an effect. If yes how much more can we improve them? Or should we just drop our mitigation altogether because it might have a cost associated with it. While I think we are doing good I think we could do better. That is why we do this research. We do need to talk publicly about the trade-offs of these systems because if we promise a perfect system to everybody that is a promise that we have already broken. Being upfront with the upsides but also with the downsides of a system is important I think.

Q - Going back to the backup discussion where we were talking about hobbyists and enterprises. Do you think it is important that there is a route for a hobbyist to set up a routing node? And that there are a lot of hobbyists running routing nodes just so that people don’t have to go through the enterprises. Or maybe we just need a lot more enterprises. Maybe both?

A - Does a hobbyist become an enterprise once we professionalizes? There must always be the option for users to professionalize and become more proficient and more professional in their operations because that is something that we’ve gotten from Bitcoin. It is not that everybody must run a node, it is not that we suddenly have a new currency. But now suddenly everybody has the option of taking on their own responsibility and becoming their own custodian and not having to rely on other parties. We shouldn’t shame people into doing that but the option must be there for those interested. I think there is a wide spectrum of options that we can offer from custodial wallets up to having fully self sovereign nodes that run the full software stack at home and you connect to your own realm in Bitcoin. Of course this is a spectrum. Depending on your interest you will land somewhere in there. We would like to have more people on the educated and the knowledgeable side of things where they run as much as possible but if you are somebody who just wants to accept the payment without having to read for months on how Bitcoin and Lightning works and have to understand everything. I think there should be an option for that as well. The important part is that it must be an option. We shouldn’t be forcing people into one direction or another.

Q - Is real time database replication already in? You just merged your plugin some weeks ago allowing back end customization. Can we get comments on real time replicas made with Postgres?

A - The replication of a Postgres database is something that we as c-lightning do not include in our code. It is just a backend that you write to. It is up to the operator to set up a replicated set of Postgres servers. Then just pointing c-lightning towards it. That is definitely in there. I think it has been there since 0.7.2. That is almost a year now. All you actually need to do is add `—wallet=postgres` and then the username, password and URL of where your database node lives. It will then start talking to Postgres instead of a sqlite database on your local machine. Again Gabriele Domenichini has an excellent tutorial on that which I have linked to in the presentation. 

Q - What do you think about custom channels or custom HTLC types? Is that possible and feasible? Would integration of Miniscript allow for such customization? Do you see Miniscript playing a role?

A - I don’t see where Miniscript comes in. It might be useful when it comes to having a fully flexible implementation where we can have external applications deciding on what kind of outputs we add. One of the proposals for example once we have eltoo, we can have multiparty channels where we can have any number of participants. The only thing that these multiparty channels is whether to add or remove or adjust the amounts on individual outputs. The way we can describe those outputs could be in the form of Miniscript. That way each participant in this ensemble of eltoo participants is deciding on whether or not to add an output to the state of the channel. They wouldn’t need to know the exact application that is sitting behind it in order for them to decide on whether an output makes sense or not because they would get a Miniscript descriptor. Besides that if we go back to the Lightning Network it absolutely makes sense to have custom protocols between nodes. These can include custom channels or custom assets being transferred on those channels. Or even different constructions of HTLCs. The only thing that is important is that the two nodes that decide to use that certain custom protocol agree on what this custom protocol is and they implement it correctly. The second thing is that if we want to maintain the cohesion of the Lightning Network and the ability to transfer funds from point A to point B we need to have a HTLC construction that is compatible with the preimage trick that we are currently using. Or the point contingency that PTLCs would bring. That means if I receive an incoming HTLC from left I can use whatever protocol I want to forward it to the right person if my right person knows that protocol. But we need to switch back to normal HTLCs once we’ve left this channel. We can have mixes of different forwarding mechanisms and different update mechanisms and custom channel constructions as long as from the outside it all looks compatible. That’s also one thing that I pointed out in the eltoo paper is that eltoo is a drop in replacement for the update mechanism in Lightning. Since we don’t change the HTLCs I could receive an HTLC over Lightning penalty from Jeff and I could forward it as a HTLC over eltoo to you. We can exchange individual parts as long as on the multihop part we do agree on certain standards that are interoperable.

Q - If we have a payment channel open between me and you we can do anything. It is completely down to what we want. 

A - We could be pretending to transfer. If we trust each other and we have the same operator, we can settle outside of the network. Jeff could send me a HTLC, I could tell you using a HTTP request that it is ok and to forward this payment to wherever it needs to go. We will settle with a beer later. Even these constructions are possible where we don’t even have to have a channel between us to forward a payment if we trust each other. Or we could change the transport mechanism and transfer the Lightning packets over SMTP or transfer them over ham radio like some people have done. We can take parts of this stack of protocols and replace them with other parts that resemble or have the same functionality but have different trade-offs. 

Q - If me and you have a channel in the middle of the route from A to B there are some restrictions. We have to be BOLT compatible if we are going to be routing onion routed payments?

A - There is one restriction that we will only accept channel announcements if they correspond to an outpoint on the blockchain. This is done as an anti-spam measure. We would have to create something that looks like a Lightning channel if we were to pretend there is a channel in the end. Other than that you can do whatever.

Q - You don’t think there is a role for Miniscript in terms of custom scripts or making changes to those standardized Lightning scripts that most of us will be using?

A - There might be a point to it. Miniscript is a mechanism where we can express some output that we haven’t agreed upon ahead of time. I could tell you “Please add this output here” and I wouldn’t have to tell you before we start a channel how it needs to look. I could just send you a Miniscript. Miniscript is a tool that allows us to talk about scripts. Currently in the Lightning protocol all of the scripts are defined in the specification itself. For us there is no currently no need to talk about different structures of outputs. There might be a use where we can add this flexibility and decide on the fly what a certain output should like. There has never been this need to meta talk about outputs because we already know what this looks like.

Q - So instead of Miniscript and Script perhaps we are using scriptless scripts and PTLCs. What are the biggest obstacles and pain points to implementing protocol changes in a Lightning node? If we wanted stuckless payments or PTLCs?

A - It varies a lot. Depending on where this change is being made it might require changes to just a single line of code which is always a nice thing because you get a change log entry for ten seconds of work. Others require reworking the entire state machine of a Lightning channel. Those often require months of work if not longer. For PTLCs it would not be too hard. It just changes the way we grab the preimage on the downstream HTLC and hand it over to the upstream HTLC. It would be an additional computational step in there instead of taking a preimage here and applying it here, it would be take a preimage or signature here, modify it slightly and that gives you whatever is needed to be plugged in on the other side. When it comes to deeper changes to the state machine we have the anchor outputs proposal currently which would require us to rework quite a lot of our onchain transaction handling. We try to be very slow and deliberate about which changes of this caliber we add to the specification because they usually bring a lot of work with them. Of course the extreme case of all of this is if we were to add eltoo, that would be reworking the entire state machine of the entire protocol. That is some major work. Eltoo has its downsides too.

Q - Nadav (Kohen) has been working on PTLCs. I think he implemented with Jonas Nick and a few others PTLCs with ECDSA. Do you think there should be a lot more work on things like channel factories before eltoo because eltoo is probably going to be a while?

A - I am a bit hesitant when it comes to channel factories because depending on what day it is they are either brilliant or they are just stupid. Being one of the authors of that paper I don’t know. The main problem with channel factories is we require multiparty channels first. What channel factories do is they take some shared funds that are managed collaboratively between a number of participants. They take part of that and move it into a separate subchannel in that construction. That has the advantage that since the entire group no longer needs to sign off changes, just the two of us that need to agree on what happens to these funds in the subchannel it is way quicker to collect two signatures rather than fifteen. That is the main upside. The downside of course that first we need to have this group of fifteen. The way we implement this group of fifteen, the multiparty channel, needs to either be a duplex micropayment channel which is a very old paper of mine which never really took off because its blockchain footprint is rather large or we use eltoo that allows us to set up these very lightweight 15-of-15 or 60-of-60 channels. Then we can add channel factories on top for efficiency. The reason why I am saying depending on the day the channel factories sound weird is that we already have an offchain construction where we can immediately sign off on changes without having to have yet another level of indirection. Then there is the efficiency gain, still undecided. I don’t know.

Q - What can we do to encourage better modularity? Is this important an approaching Taproot world?

A - I think the modularity of the protocol and the modularity of the implementations pretty much go hand in hand. If the specification has very nice modular boundaries where you have separation of concerns, one thing manages updates of state and one thing manages how we communicate with the blockchain and one thing manages how we do multihop security. That automatically leads to a structure which is very modular. The issue that we have currently have is that the Lightning penalty mechanism namely the fact that whatever output we create in our state must be penalizable makes it so that this update mechanism leaks into the rest of the protocol stack. I showed before how we punish the commitment transaction if I were ever to publish an old commitment. But if we had a HTLC attached to that, this HTLC too would have to have the facility for me to punish you if you published this old state with this HTLC that had been resolved correctly or incorrectly or timed out. It is really hard in the penalty mechanism to have a clear cut separation between the update mechanism and the multihop mechanism and whatever else we build on top of the update mechanism. It leaks into each other. That is something that I really like about eltoo. We have this clear separation of this is the update mechanism and this is the multihop mechanism and there is no interference between the two of them. I think by clearing up the protocol stack we will end up with more modular implementations. Of course at c-lightning we try to expose as much as possible from the internals to plugins so that plugins are first class citizens in the Lightning nodes themselves. They have the same power as most of our pre-shipped tools have. One little known fact is that the pay command which is used to pay a BOLT11 invoice is also implemented as a plugin. The plugin takes care of decoding the invoice of initiating a payment, retrying if a payment fails, splitting a payment if it is too large or adding a shadow route or adding fuzzing or all of this. It is all implemented in a plugin and the bare bones implementation of c-lighting is very light. It doesn’t come with a lot of bells and whistles but we make it so that you have the power of customizing it and so on. There we do try to keep a modular aspect to c-lightning despite the protocol not being a perfectly modular system itself.

Q - There was a [mailing list post](https://gnusha.org/url/https://lists.linuxfoundation.org/pipermail/lightning-dev/2020-February/002547.html) from Joost (Jager). What are the issues with upfront payments? There was some discussion about it then it seems to have stopped. Why haven’t we seen more development in that direction yet?

A - Upfront payments is a proposal that came up when we first started probing the network. Probing involves sending a payment that can never terminate correctly. By looking at the error code we receive back we learn about the network. It is for free because the payments never actually terminate. That brought up the question of aren’t we using somebody else’s resources by creating HTLCs with their funds as well but not paying them for it? The idea came up of having upfront payments which means that if I try to route a payment I will definitely leave a fee even if that payment fails. That is neat but the balance between working and not working is hard to get right. The main issue is that if we pay upfront for them receiving a payment and not forwarding a payment then they may be happy to take the upfront fee and just fail without any stake in the system. If I were to receive an incoming HTLC from Jeff and I need to forward it to you Michael and Jeff is paying me 10 millisatoshis for the privilege of talking to me I might not actually take my half a Bitcoin and lock it up in a HTLC to you. I might be happy taking those 10 millisatoshis and say “I’m ok with this. You try another route.” It is an issue of incentivizing good behavior versus incentivizing abusing the system to maximize your outcome. A mix of upfront payments and fees contingent on the success of the actual payment is probably the right way. We need to discuss a bit more and people’s time is tight when it comes to these proposals. There has been too much movement I guess.

Q - I’ve been reading a bit about Simplicity which your colleagues Adam Back and Russell O’Connor have been working on at Blockstream. They talk about being able to do new sighash flags, SIGHASH_NOINPUT, ANYPREVOUT without a soft fork. In a world where we had Simplicity perhaps the arguments against NOINPUT or ANYPREVOUT namely they being dangerous for users no longer apply if Simplicity was in Bitcoin and people could use it anyway. Any thoughts on that?

A - That is an interesting question. How could I not get fired for discussing this in public? I do think Simplicity is an awesome proposal. It is something that I would love to have because so far during my PhD and during my work at Blockstream the number one issue that we had was stuff that we wanted to do was blocked by them being available in Bitcoin itself. It can be frustrating at times to come up with a really neat solution and not being able to enact them. As far as the criticism to SIGHASH_NOINPUT and ANYPREVOUT we shouldn’t take them lightly and I do see points where people bring up good points about there being some uncertainty and there being insecurity when it comes to double spending and securing funds. How do we clamp down this proposal as much as possible so people can’t inadvertently abuse it. But I do think that with all of the existing systems that we already have in Bitcoin we are currently trying to save a sandcastle while the dyke is breaking behind us. It is a disproportionate amount of caution when we already have some really dangerous tools in the Bitcoin protocol itself. First and foremost who invented SIGHASH_NONE? You can have a signature that does not cover anything but you can still spend funds with it. While I do take criticism seriously I don’t think we need to spend too much time on that. Indeed if we get Simplicity a lot more flexibility could be added but of course with great power comes great responsibility. We need to make sure that people who want to use those features do know the trade-offs really well and don’t put user funds at risk. That has always been something that we’ve pointed towards. We need to have tech savvy people doing these custom protocols. Otherwise you should stick with what is tested and proven.

Q - And obviously Simplicity is still far off and won’t be in Bitcoin anytime soon.

A - We recently had a Simplicity based transaction on Elements. We do have a network where we test these experimental features to showcase that these are possible and what possible implementations could look like. That is our testing ground. Russell has used his Simplicity implementation on some of these transactions. We have some cool stuff in there like quines.