From: Damian Williamson <willtech@live.com.au>
To: Chris Riley <criley@gmail.com>
Thread-Topic: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction
	Priority For Ordering Transactions In Blocks
Thread-Index: AQHTb5z9LKaNJrfjBkOvllhZtrqBGqNEL2DvgAB3wACAAD9wXoAELFgAgAFHmoQ=
Date: Tue, 19 Dec 2017 07:48:37 +0000
Message-ID: <PS2P216MB01798757147723AF5B7B7CA49D0F0@PS2P216MB0179.KORP216.PROD.OUTLOOK.COM>
References: <PS2P216MB01794ABD544248B27BF0DFD99D330@PS2P216MB0179.KORP216.PROD.OUTLOOK.COM>
	<PS2P216MB01795BFC05612E021CCEDD7C9D0B0@PS2P216MB0179.KORP216.PROD.OUTLOOK.COM>
	<5upGmF0IhXUWhcikhdV-e9Pqg-kXfUuXe0kOpGxumie_TO2jLvCgTZ5c6vgBRtaqkL6DmOJb1YftK0osufB5RkhW7YhAhhCI0zBTH3YcORY=@protonmail.com>
	<PS2P216MB0179544A6503C2992190CEB69D0B0@PS2P216MB0179.KORP216.PROD.OUTLOOK.COM>,
	<CAL5BAw0pQJg8MjopRH5ReBVHmJR0bYud=E6fkFuY=-3hMAumEw@mail.gmail.com>
In-Reply-To: <CAL5BAw0pQJg8MjopRH5ReBVHmJR0bYud=E6fkFuY=-3hMAumEw@mail.gmail.com>
Accept-Language: en-AU, en-US
Content-Language: en-AU
x-ms-exchange-messagesentrepresentingtype: 1
x-tmn: [CDd+p1u6QTQtmmJfBPlegzm3NW+nZgE6]
x-ms-publictraffictype: Email
x-microsoft-exchange-diagnostics: 1; HK2APC01HT055;
	6:0GPa72gnOMdEApk9guZA43aq7wtEvO+Lr8ObjJHTonIhuwoZkMVDyVAf47/arK4N3CiitnPNLIsS380pcE/lx4KnZIUeY/D5Vl+3U84XvtTBKINRSkbQx0AiV6qE8GT72IUGwcanyZJgXKL5lP+3ISvUqdYIN5scUE+i8YKJ89exUBCOzjydkwD86FaxPtDKyLKWFIKe6LVRgr5/Vm5iShA/1RC/O+qc64OGnYzFMR0Mjd2IrZ4k/sLbE7AXaVjTRt7kBJSLZmw1JGFN7vBZDzA1Js0j4zO9muu0ZVyS3NvahvGqTKnopPLqhuR4fJOpj+77osaOPklN5Se7U3Kz0gsryeT/w8I0OtrcRPgPTyw=;
	5:4PzqCZCMBwQoXjSOo7z5yFiQ9pCjt6zSRbeY6rxuOsPB9xIhrr9g6nFK0PPaQWYYdHgBNSLxp7VOgJ0fq90onXaPzlWJ01TF7UWcV8mUQD+p5p50JgDlTpZ+I8+1L/fu0C5ZCtrbqtmPW4VZoulCUP3AxmkCIvrD/R+q1qSFNd4=;
	24:NU5LJuV7EyIKo6N9uRgANjXs41NWdkHWOj/YT02l/IkRPuOo66boO8gjsFPMTCbqF1vnOLvP2NvxG7td9BHrxW8mtb7xcW32H1VIn+p8rp0=;
	7:KuarIZn84zFpqBqflmOf06jgmwHte3A7ZiJkzMqEghRtGoUZpTjBXUrZqhT4nmELBQidcB+oKoGHtgKzm1JuHEsjBbx+Srq8Wmoo7xUgLPIoFfoBIEycQSS0iRxDD2biCDKPZp4K0/wxDQ0OkUZnc1uE9JYufKoBMQqWGvxUa43xQBGVwNJ1hKHF3KhRvLbJwI2/Ii/l1WhN2jtty9KMZFrsEndeNuapV+oLqmtGjGJZICAMtZWlS3F4TXauzaSn
x-incomingheadercount: 47
x-eopattributedmessage: 0
x-microsoft-antispam: UriScan:; BCL:0; PCL:0;
	RULEID:(201702061074)(5061506573)(5061507331)(1603103135)(2017031320274)(2017031324274)(2017031323274)(2017031322404)(1601125374)(1603101448)(1701031045);
	SRVR:HK2APC01HT055; 
x-ms-traffictypediagnostic: HK2APC01HT055:
x-ms-office365-filtering-correlation-id: 5fb8ff63-6adb-452c-b186-08d546b4e9ee
x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(444000031);
	SRVR:HK2APC01HT055; BCL:0; PCL:0;
	RULEID:(100000803101)(100110400095); SRVR:HK2APC01HT055; 
x-forefront-prvs: 052670E5A4
x-forefront-antispam-report: SFV:NSPM; SFS:(7070007)(98901004); DIR:OUT;
	SFP:1901; SCL:1; SRVR:HK2APC01HT055;
	H:PS2P216MB0179.KORP216.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:;
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: multipart/alternative;
	boundary="_000_PS2P216MB01798757147723AF5B7B7CA49D0F0PS2P216MB0179KORP_"
MIME-Version: 1.0
X-OriginatorOrg: outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 5fb8ff63-6adb-452c-b186-08d546b4e9ee
X-MS-Exchange-CrossTenant-originalarrivaltime: 19 Dec 2017 07:48:37.2460 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Internet
X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa
X-MS-Exchange-Transport-CrossTenantHeadersStamped: HK2APC01HT055
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID, HTML_MESSAGE, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
	smtp1.linux-foundation.org
X-Mailman-Approved-At: Tue, 19 Dec 2017 12:49:21 +0000
Cc: "bitcoin-dev@lists.linuxfoundation.org"
	<bitcoin-dev@lists.linuxfoundation.org>
Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction
 Priority For Ordering Transactions In Blocks
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Bitcoin Protocol Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Dec 2017 07:48:43 -0000

--_000_PS2P216MB01798757147723AF5B7B7CA49D0F0PS2P216MB0179KORP_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Thank you for your constructive feedback. I now see that the proposal intro=
duces a potential issue.


It is difficult to define then, what is a valid transaction? Clearly, my de=
finition was insufficient.


Regards,

Damian Williamson


________________________________
From: Chris Riley <criley@gmail.com>
Sent: Monday, 18 December 2017 11:09 PM
To: Damian Williamson; Bitcoin Protocol Discussion
Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transactio=
n Priority For Ordering Transactions In Blocks

Regarding "problem" #2 where you say "How do we ensure that all valid trans=
actions are eventually included in the blockchain?":  I do not believe that=
 all people would (a) agree this is a problem or (b) that we do want to *EN=
SURE* that *ALL* valid transactions are eventually included in the blockcha=
in.  There are many *valid* transactions that oftentimes miners do not (and=
 should not) wish to require be confirmed and included in the blockchain.  =
Spam transactions for example can be valid, but used to attack bitcoin by u=
sing no or low fee.  Any valid transaction MAY be included by a miner, but =
requiring it in some fashion at this point would open the network to other =
attack vectors.  Perhaps you meant it a different way.


On Fri, Dec 15, 2017 at 3:59 PM, Damian Williamson via bitcoin-dev <bitcoin=
-dev@lists.linuxfoundation.org<mailto:bitcoin-dev@lists.linuxfoundation.org=
>> wrote:
>
> There are really two separate problems to solve.
>
>
> How does Bitcoin scale with fixed block size?
> How do we ensure that all valid transactions are eventually included in t=
he blockchain?
>
>
> Those are the two issues that the proposal attempts to address. It makes =
sense to resolve these two problems together. Using the proposed system for=
 variable block sizes would solve the first problem but there would still b=
e a whole bunch of never confirming transactions. I am not sure how to reli=
ably solve the second problem at scale without first solving the first.
>
>
> >* Every node has a (potentially) different mempool, you can't use it to =
decide consensus values like the max block size.
>
>
> I do not suggest a consensus. Depending on which node solves a block the =
value for next block size will be different. The consensus would be that bl=
ocks will adhere to the next block size value transmitted with the current =
block. It is easy to verify that the consensus is being adhered to once in =
place.
>
> >* Increasing the entropy in a block to make it more unpredictable doesn'=
t really make sense.
>
> Not a necessary function, just an effect of using a probability-based dis=
tribution.
>
> >* Bitcoin should be roughly incentive compatible. Your proposal explicit=
s asks miners to ignore their best interests, and confirm transactions by "=
priority".  What are you going to do if a "malicious" miner decides to go a=
fter their profits and order by what makes them the most money. Add "ordere=
d by priority" as a consensus requirement? And even if you miners can still=
 sort their mempool by fee, and then order the top 1MB by priority.
>
> I entirely agree with your sentiment that Bitcoin must be incentive compa=
tible. It is necessary.
>
> It is in only miners immediate interest to make the most profitable block=
 from the available transaction pool. As with so many other things, it is n=
ecessary to partially ignore short-term gain for long-term benefit. It is i=
n miners and everybody's long-term interest to have a reliable transaction =
service. A busy transaction service that confirms lots of transactions per =
hour will become more profitable as demand increases and more users are pre=
pared to pay for priority. As it is there is currently no way to fully scal=
e because of the transaction bandwidth limit and that is problematic. If al=
l valid transactions must eventually confirm then there must be a way to re=
solve that problem.
>
> Bitcoin deliberately removes traditional scale by ensuring blocks take te=
n minutes on average to solve, an ingenious idea and, incentive compatible =
but, fixed block sizes leaves us with a problem to solve when we want to sc=
ale.
>
> >If you could find a good solution that would allow you to know if miners=
 were following your rule or not (and thus ignore it if it doesn't) then yo=
u wouldn't even need bitcoin in the first place.
>
> I am confident that the math to verify blocks based on the proposal can b=
e developed (and I think it will not be too complex for a mathematician wit=
h the relevant experience), however, I am nowhere near experienced enough w=
ith probability and statistical analysis to do it. Yes, if Bitcoin doesn't =
then it might make another great opportunity for an altcoin but I am not ev=
en nearly interested in promoting any altcoins.
>
>
> If not the proposal that I have put forward, then, hopefully, someone can=
 come up with a better solution. The important thing is that the issues are=
 resolved.
>
>
> Regards,
>
> Damian Williamson
>
>
>
> ________________________________
> From: Rhavar <rhavar@protonmail.com<mailto:rhavar@protonmail.com>>
> Sent: Saturday, 16 December 2017 3:38 AM
> To: Damian Williamson
> Cc: Bitcoin Protocol Discussion
> Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transact=
ion Priority For Ordering Transactions In Blocks
>
> > I understand that there would be technical issues to resolve in impleme=
ntation, but, are there no fundamental errors?
>
> Unfortunately your proposal is really fundamentally broken, on a few leve=
ls. I think you might need to do a bit more research into how bitcoin works=
 before coming up with such improvements =3D)
>
> But just some quick notes:
>
> * Every node has a (potentially) different mempool, you can't use it to d=
ecide consensus values like the max block size.
>
> * Increasing the entropy in a block to make it more unpredictable doesn't=
 really make sense.
>
> * Bitcoin should be roughly incentive compatible. Your proposal explicits=
 asks miners to ignore their best interests, and confirm transactions by "p=
riority".  What are you going to do if a "malicious" miner decides to go af=
ter their profits and order by what makes them the most money. Add "ordered=
 by priority" as a consensus requirement? And even if you miners can still =
sort their mempool by fee, and then order the top 1MB by priority.
>
> If you could find a good solution that would allow you to know if miners =
were following your rule or not (and thus ignore it if it doesn't) then you=
 wouldn't even need bitcoin in the first place.
>
>
>
>
> -Ryan
>
>
> -------- Original Message --------
> Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction =
Priority For Ordering Transactions In Blocks
> Local Time: December 15, 2017 3:42 AM
> UTC Time: December 15, 2017 9:42 AM
> From: bitcoin-dev@lists.linuxfoundation.org<mailto:bitcoin-dev@lists.linu=
xfoundation.org>
> To: Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org<ma=
ilto:bitcoin-dev@lists.linuxfoundation.org>>
>
>
>
> I should not take it that the lack of critical feedback to this revised p=
roposal is a glowing endorsement. I understand that there would be technica=
l issues to resolve in implementation, but, are there no fundamental errors=
?
>
> I suppose that it if is difficult to determine how long a transaction has=
 been waiting in the pool then, each node could simply keep track of when a=
 transaction was first seen. This may have implications for a verify routin=
e, however, for example, if a node was offline, how should it differentiate=
 how long each transaction was waiting in that case? If a node was restarte=
d daily would it always think that all transactions had been waiting in the=
 pool less than one day If each node keeps the current transaction pool in =
a file and updates it, as transactions are included in blocks and, as new t=
ransactions appear in the pool, then that would go some way to alleviate th=
e issue, apart from entirely new nodes. There should be no reason the conte=
nts of a transaction pool files cannot be shared without agreement as to th=
e transaction pool between nodes, just as nodes transmit new transactions f=
reely.
>
> It has been questioned why miners could not cheat. For the question of ho=
w many transactions to include in a block, I say it is a standoff and miner=
s will conform to the proposal, not wanting to leave transactions with vali=
d fees standing, and, not wanting to shrink the transaction pool. In any ca=
se, if miners shrink the transaction pool then I am not immediately concern=
ed since it provides a more efficient service. For the question of includin=
g transactions according to the proposal, I say if it is possible to keep t=
rack of how long transactions are waiting in the pool so that they can be i=
ncluded on a probability curve then it is possible to verify that blocks co=
nform to the proposal, since the input is a probability, the output should =
conform to a probability curve.
>
>
> If someone has the necessary skill, would anyone be willing to develop th=
e math necessary for the proposal?
>
> Regards,
> Damian Williamson
>
>
> ________________________________
>
> From: bitcoin-dev-bounces@lists.linuxfoundation.org<mailto:bitcoin-dev-bo=
unces@lists.linuxfoundation.org> <bitcoin-dev-bounces@lists.linuxfoundation=
.org<mailto:bitcoin-dev-bounces@lists.linuxfoundation.org>> on behalf of Da=
mian Williamson via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org<mail=
to:bitcoin-dev@lists.linuxfoundation.org>>
> Sent: Friday, 8 December 2017 8:01 AM
> To: bitcoin-dev@lists.linuxfoundation.org<mailto:bitcoin-dev@lists.linuxf=
oundation.org>
> Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction =
Priority For Ordering Transactions In Blocks
>
>
>
> Good afternoon,
>
> The need for this proposal:
>
> We all must learn to admit that transaction bandwidth is still lurking as=
 a serious issue for the operation, reliability, safety, consumer acceptanc=
e, uptake and, for the value of Bitcoin.
>
> I recently sent a payment which was not urgent so; I chose three-day targ=
et confirmation from the fee recommendation. That transaction has still not=
 confirmed after now more than six days - even waiting twice as long seems =
quite reasonable to me. That transaction is a valid transaction; it is not =
rubbish, junk or, spam. Under the current model with transaction bandwidth =
limitation, the longer a transaction waits, the less likely it is ever to c=
onfirm due to rising transaction numbers and being pushed back by transacti=
ons with rising fees.
>
> I argue that no transactions are rubbish or junk, only some zero fee tran=
sactions might be spam. Having an ever-increasing number of valid transacti=
ons that do not confirm as more new transactions with higher fees are creat=
ed is the opposite of operating a robust, reliable transaction system.
>
> Business cannot operate with a model where transactions may or may not co=
nfirm. Even a business choosing a modest fee has no guarantee that their va=
lid transaction will not be shuffled down by new transactions to the realm =
of never confirming after it is created. Consumers also will not accept thi=
s model as Bitcoin expands. If Bitcoin cannot be a reliable payment system =
for confirmed transactions then consumers, by and large, will simply not ac=
cept the model once they understand. Bitcoin will be a dirty payment system=
, and this will kill the value of Bitcoin.
>
> Under the current system, a minority of transactions will eventually be t=
he lucky few who have fees high enough to escape being pushed down the list=
.
>
> Once there are more than x transactions (transaction bandwidth limit) eve=
ry ten minutes, only those choosing twenty-minute confirmation (2 blocks) w=
ill have initially at most a fifty percent chance of ever having their paym=
ent confirm. Presently, not even using fee recommendations can ensure a suf=
ficiently high fee is paid to ensure transaction confirmation.
>
> I also argue that the current auction model for limited transaction bandw=
idth is wrong, is not suitable for a reliable transaction system and, is wr=
ong for Bitcoin. All transactions must confirm in due time. Currently, Bitc=
oin is not a safe way to send payments.
>
> I do not believe that consumers and business are against paying fees, eve=
n high fees. What is required is operational reliability.
>
> This great issue needs to be resolved for the safety and reliability of B=
itcoin. The time to resolve issues in commerce is before they become great =
big issues. The time to resolve this issue is now. We must have the foresig=
ht to identify and resolve problems before they trip us over.  Simply doubl=
ing block sizes every so often is reactionary and is not a reliable permane=
nt solution. I have written a BIP proposal for a technical solution but, ne=
ed your help to write it up to an acceptable standard to be a full BIP.
>
> I have formatted the following with markdown which is human readable so, =
I hope nobody minds. I have done as much with this proposal as I feel that =
I am able so far but continue to take your feedback.
>
> # BIP Proposal: UTPFOTIB - Use Transaction Priority For Ordering Transact=
ions In Blocks
>
> ## The problem:
> Everybody wants value. Miners want to maximize revenue from fees (and we =
presume, to minimize block size). Consumers need transaction reliability an=
d, (we presume) want low fees.
>
> The current transaction bandwidth limit is a limiting factor for both. As=
 the operational safety of transactions is limited, so is consumer confiden=
ce as they realize the issue and, accordingly, uptake is limited. Fees are =
artificially inflated due to bandwidth limitations while failing to provide=
 a full confirmation service for all transactions.
>
> Current fee recommendations provide no satisfaction for transaction relia=
bility and, as Bitcoin scales, this will worsen.
>
> Bitcoin must be a fully scalable and reliable service, providing full tra=
nsaction confirmation for every valid transaction.
>
> The possibility to send a transaction with a fee lower than one that is a=
cceptable to allow eventual transaction confirmation should be removed from=
 the protocol and also from the user interface.
>
> ## Solution summary:
> Provide each transaction with an individual transaction priority each tim=
e before choosing transactions to include in the current block, the priorit=
y being a function of the fee paid (on a curve), and the time waiting in th=
e transaction pool (also on a curve) out to n days (n=3D60 ?). The transact=
ion priority to serve as the likelihood of a transaction being included in =
the current block, and for determining the order in which transactions are =
tried to see if they will be included.
>
> Use a target block size. Determine the target block size using; current t=
ransaction pool size x ( 1 / (144 x n days ) ) =3D number of transactions t=
o be included in the current block. Broadcast the next target block size wi=
th the current block when it is solved so that nodes know the next target b=
lock size for the block that they are building on.
>
> The curves used for the priority of transactions would have to be appropr=
iate. Perhaps a mathematician with experience in probability can develop th=
e right formulae. My thinking is a steep curve. I suppose that the probabil=
ity of all transactions should probably account for a sufficient number of =
inclusions that the target block size is met although, it may not always be=
. As a suggestion, consider including some zero fee transactions to pad, hi=
ghest BTC value first?
>
> **Explanation of the operation of priority:**
> > If transaction priority is, for example, a number between one (low) and=
 one-hundred (high) it can be directly understood as the percentage chance =
in one-hundred of a transaction being included in the block. Using probabil=
ity or likelihood infers that there is some function of random. If random (=
100) < transaction priority then the transaction is included.
>
> >To break it down further, if both the fee on a curve value and the time =
waiting on a curve value are each a number between one and one-hundred, a r=
udimentary method may be to simply multiply those two numbers, to find the =
priority number. For example, a middle fee transaction waiting thirty days =
(if n =3D 60 days) may have a value of five for each part  (yes, just five,=
 the values are on a curve). When multiplied that will give a priority valu=
e of twenty-five, or,  a twenty-five percent chance at that moment of being=
 included in the block; it will likely be included in one of the next four =
blocks, getting more likely each chance. If it is still not included then t=
he value of time waiting will be higher, making for more probability. A ver=
y low fee transaction would have a value for the fee of one. It would not b=
e until near sixty-days that the particular low fee transaction has a high =
likelihood of being included in the block.
>
> I am not concerned with low (or high) transaction fees, the primary reaso=
n for addressing the issue is to ensure transactional reliability and scala=
bility while having each transaction confirm in due time.
>
> ## Pros:
> * Maximizes transaction reliability.
> * Fully scalable.
> * Maximizes possibility for consumer and business uptake.
> * Maximizes total fees paid per block without reducing reliability; becau=
se of reliability, in time confidence and overall uptake are greater; there=
fore, more transactions.
> * Market determines fee paid for transaction priority.
> * Fee recommendations work all the way out to 30 days or greater.
> * Provides additional block entropy; greater security since there is less=
 probability of predicting the next block.
>
> ## Cons:
> * Could initially lower total transaction fees per block.
> * Must be first be programmed.
>
> ## Solution operation:
> This is a simplistic view of the operation. The actual operation will nee=
d to be determined in a spec for the programmer.
>
> 1. Determine the target block size for the current block.
> 2. Assign a transaction priority to each transaction in the pool.
> 3. Select transactions to include in the current block using probability =
in transaction priority order until the target block size is met.
> 5. Solve block.
> 6. Broadcast the next target block size with the current block when it is=
 solved.
> 7. Block is received.
> 8. Block verification process.
> 9. Accept/reject block based on verification result.
> 10. Repeat.
>
> ## Closing comments:
> It may be possible to verify blocks conform to the proposal by showing th=
at the probability for all transactions included in the block statistically=
 conforms to a probability distribution curve, *if* the individual transact=
ion priority can be recreated. I am not that deep into the mathematics; how=
ever, it may also be possible to use a similar method to do this just based=
 on the fee, that statistically, the blocks conform to a fee distribution. =
Any zero fee transactions would have to be ignored. This solution needs a c=
lever mathematician.
>
> I implore, at the very least, that we use some method that validates full=
 transaction reliability and enables scalability of block sizes. If not thi=
s proposal, an alternative.
>
> Regards,
> Damian Williamson
>
>
>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org<mailto:bitcoin-dev@lists.linuxfound=
ation.org>
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

--_000_PS2P216MB01798757147723AF5B7B7CA49D0F0PS2P216MB0179KORP_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-=
1">
<style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;margi=
n-bottom:0;} --></style>
</head>
<body dir=3D"ltr">
<div id=3D"divtagdefaultwrapper" style=3D"font-size: 12pt; color: rgb(0, 0,=
 0); font-family: Calibri,Helvetica,sans-serif,&quot;EmojiFont&quot;,&quot;=
Apple Color Emoji&quot;,&quot;Segoe UI Emoji&quot;,NotoColorEmoji,&quot;Seg=
oe UI Symbol&quot;,&quot;Android Emoji&quot;,EmojiSymbols;" dir=3D"ltr">
<p style=3D"margin-top:0;margin-bottom:0">Thank you for your constructive f=
eedback. I now see that the proposal introduces a potential issue.</p>
<p style=3D"margin-top:0;margin-bottom:0"><br>
</p>
<p style=3D"margin-top:0;margin-bottom:0">It is difficult to define then, w=
hat is a valid transaction? Clearly, my definition was insufficient.<br>
</p>
<p style=3D"margin-top:0;margin-bottom:0"><br>
</p>
<p style=3D"margin-top:0;margin-bottom:0">Regards,</p>
<p style=3D"margin-top:0;margin-bottom:0">Damian Williamson<br>
</p>
<br>
<br>
<div style=3D"color: rgb(0, 0, 0);">
<hr style=3D"display:inline-block;width:98%" tabindex=3D"-1">
<div id=3D"divRplyFwdMsg" dir=3D"ltr"><font style=3D"font-size:11pt" face=
=3D"Calibri, sans-serif" color=3D"#000000"><b>From:</b> Chris Riley &lt;cri=
ley@gmail.com&gt;<br>
<b>Sent:</b> Monday, 18 December 2017 11:09 PM<br>
<b>To:</b> Damian Williamson; Bitcoin Protocol Discussion<br>
<b>Subject:</b> Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Tra=
nsaction Priority For Ordering Transactions In Blocks</font>
<div>&nbsp;</div>
</div>
<div>
<div dir=3D"ltr">Regarding &quot;problem&quot; #2 where you say &quot;How d=
o we ensure that all valid transactions are eventually included in the bloc=
kchain?&quot;: &nbsp;I do not believe that all people would (a) agree this =
is a problem or (b) that we do want to *ENSURE* that *ALL*
 valid transactions are eventually included in the blockchain.&nbsp; There =
are many *valid* transactions that oftentimes miners do not (and should not=
) wish to require be confirmed and included in the blockchain.&nbsp; Spam t=
ransactions for example can be valid, but
 used to attack bitcoin by using no or low fee.&nbsp; Any valid transaction=
 MAY be included by a miner, but requiring it in some fashion at this point=
 would open the network to other attack vectors.&nbsp; Perhaps you meant it=
 a different way.
<div><br>
<br>
On Fri, Dec 15, 2017 at 3:59 PM, Damian Williamson via bitcoin-dev &lt;<a h=
ref=3D"mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.linu=
xfoundation.org</a>&gt; wrote:<br>
&gt;<br>
&gt; There are really two separate problems to solve.<br>
&gt;<br>
&gt;<br>
&gt; How does Bitcoin scale with fixed block size?<br>
&gt; How do we ensure that all valid transactions are eventually included i=
n the blockchain?<br>
&gt;<br>
&gt;<br>
&gt; Those are the two issues that the proposal attempts to address. It mak=
es sense to resolve these two problems together. Using the proposed system =
for variable block sizes would solve the first problem but there would stil=
l be a whole bunch of never confirming
 transactions. I am not sure how to reliably solve the second problem at sc=
ale without first solving the first.<br>
&gt;<br>
&gt;<br>
&gt; &gt;* Every node has a (potentially) different mempool, you can't use =
it to decide consensus values like the max block size.
<br>
&gt;<br>
&gt;<br>
&gt; I do not suggest a consensus. Depending on which node solves a block t=
he value for next block size will be different. The consensus would be that=
 blocks will adhere to the next block size value transmitted with the curre=
nt block. It is easy to verify that
 the consensus is being adhered to once in place.<br>
&gt; &nbsp;<br>
&gt; &gt;* Increasing the entropy in a block to make it more unpredictable =
doesn't really make sense.
<br>
&gt;<br>
&gt; Not a necessary function, just an effect of using a probability-based =
distribution.<br>
&gt;<br>
&gt; &gt;* Bitcoin should be roughly incentive compatible. Your proposal ex=
plicits asks miners to ignore their best interests, and confirm transaction=
s by &quot;priority&quot;.&nbsp; What are you going to do if a &quot;malici=
ous&quot; miner decides to go after their profits and order by
 what makes them the most money. Add &quot;ordered by priority&quot; as a c=
onsensus requirement? And even if you miners can still sort their mempool b=
y fee, and then order the top 1MB by priority.<br>
&gt;<br>
&gt; I entirely agree with your sentiment that Bitcoin must be incentive co=
mpatible. It is necessary.<br>
&gt;<br>
&gt; It is in only miners immediate interest to make the most profitable bl=
ock from the available transaction pool. As with so many other things, it i=
s necessary to partially ignore short-term gain for long-term benefit. It i=
s in miners and everybody's long-term
 interest to have a reliable transaction service. A busy transaction servic=
e that confirms lots of transactions per hour will become more profitable a=
s demand increases and more users are prepared to pay for priority. As it i=
s there is currently no way to fully
 scale because of the transaction bandwidth limit and that is problematic. =
If all valid transactions must eventually confirm then there must be a way =
to resolve that problem.<br>
&gt;<br>
&gt; Bitcoin deliberately removes traditional scale by ensuring blocks take=
 ten minutes on average to solve, an ingenious idea and, incentive compatib=
le but, fixed block sizes leaves us with a problem to solve when we want to=
 scale.<br>
&gt;<br>
&gt; &gt;If you could find a good solution that would allow you to know if =
miners were following your rule or not (and thus ignore it if it doesn't) t=
hen you wouldn't even need bitcoin in the first place.<br>
&gt;<br>
&gt; I am confident that the math to verify blocks based on the proposal ca=
n be developed (and I think it will not be too complex for a mathematician =
with the relevant experience), however, I am nowhere near experienced enoug=
h with probability and statistical
 analysis to do it. Yes, if Bitcoin doesn't then it might make another grea=
t opportunity for an altcoin but I am not even nearly interested in promoti=
ng any altcoins.<br>
&gt;<br>
&gt;<br>
&gt; If not the proposal that I have put forward, then, hopefully, someone =
can come up with a better solution. The important thing is that the issues =
are resolved.<br>
&gt;<br>
&gt;<br>
&gt; Regards,<br>
&gt;<br>
&gt; Damian Williamson<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt; ________________________________<br>
&gt; From: Rhavar &lt;<a href=3D"mailto:rhavar@protonmail.com">rhavar@proto=
nmail.com</a>&gt;<br>
&gt; Sent: Saturday, 16 December 2017 3:38 AM<br>
&gt; To: Damian Williamson<br>
&gt; Cc: Bitcoin Protocol Discussion<br>
&gt; Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Trans=
action Priority For Ordering Transactions In Blocks<br>
&gt; &nbsp;<br>
&gt; &gt; I understand that there would be technical issues to resolve in i=
mplementation, but, are there no fundamental errors?<br>
&gt;<br>
&gt; Unfortunately your proposal is really fundamentally broken, on a few l=
evels. I think you might need to do a bit more research into how bitcoin wo=
rks before coming up with such improvements =3D)<br>
&gt;<br>
&gt; But just some quick notes:<br>
&gt;<br>
&gt; * Every node has a (potentially) different mempool, you can't use it t=
o decide consensus values like the max block size.
<br>
&gt;<br>
&gt; * Increasing the entropy in a block to make it more unpredictable does=
n't really make sense.
<br>
&gt;<br>
&gt; * Bitcoin should be roughly incentive compatible. Your proposal explic=
its asks miners to ignore their best interests, and confirm transactions by=
 &quot;priority&quot;.&nbsp; What are you going to do if a &quot;malicious&=
quot; miner decides to go after their profits and order by what
 makes them the most money. Add &quot;ordered by priority&quot; as a consen=
sus requirement? And even if you miners can still sort their mempool by fee=
, and then order the top 1MB by priority.<br>
&gt;<br>
&gt; If you could find a good solution that would allow you to know if mine=
rs were following your rule or not (and thus ignore it if it doesn't) then =
you wouldn't even need bitcoin in the first place.<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt; -Ryan<br>
&gt;<br>
&gt;<br>
&gt; -------- Original Message --------<br>
&gt; Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transacti=
on Priority For Ordering Transactions In Blocks<br>
&gt; Local Time: December 15, 2017 3:42 AM<br>
&gt; UTC Time: December 15, 2017 9:42 AM<br>
&gt; From: <a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin=
-dev@lists.linuxfoundation.org</a><br>
&gt; To: Bitcoin Protocol Discussion &lt;<a href=3D"mailto:bitcoin-dev@list=
s.linuxfoundation.org">bitcoin-dev@lists.linuxfoundation.org</a>&gt;<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt; I should not take it that the lack of critical feedback to this revise=
d proposal is a glowing endorsement. I understand that there would be techn=
ical issues to resolve in implementation, but, are there no fundamental err=
ors?<br>
&gt;<br>
&gt; I suppose that it if is difficult to determine how long a transaction =
has been waiting in the pool then, each node could simply keep track of whe=
n a transaction was first seen. This may have implications for a verify rou=
tine, however, for example, if a node
 was offline, how should it differentiate how long each transaction was wai=
ting in that case? If a node was restarted daily would it always think that=
 all transactions had been waiting in the pool less than one day If each no=
de keeps the current transaction
 pool in a file and updates it, as transactions are included in blocks and,=
 as new transactions appear in the pool, then that would go some way to all=
eviate the issue, apart from entirely new nodes. There should be no reason =
the contents of a transaction pool
 files cannot be shared without agreement as to the transaction pool betwee=
n nodes, just as nodes transmit new transactions freely.<br>
&gt;<br>
&gt; It has been questioned why miners could not cheat. For the question of=
 how many transactions to include in a block, I say it is a standoff and mi=
ners will conform to the proposal, not wanting to leave transactions with v=
alid fees standing, and, not wanting
 to shrink the transaction pool. In any case, if miners shrink the transact=
ion pool then I am not immediately concerned since it provides a more effic=
ient service. For the question of including transactions according to the p=
roposal, I say if it is possible
 to keep track of how long transactions are waiting in the pool so that the=
y can be included on a probability curve then it is possible to verify that=
 blocks conform to the proposal, since the input is a probability, the outp=
ut should conform to a probability
 curve.<br>
&gt;<br>
&gt;<br>
&gt; If someone has the necessary skill, would anyone be willing to develop=
 the math necessary for the proposal?<br>
&gt;<br>
&gt; Regards,<br>
&gt; Damian Williamson<br>
&gt;<br>
&gt;<br>
&gt; ________________________________<br>
&gt;<br>
&gt; From: <a href=3D"mailto:bitcoin-dev-bounces@lists.linuxfoundation.org"=
>bitcoin-dev-bounces@lists.linuxfoundation.org</a> &lt;<a href=3D"mailto:bi=
tcoin-dev-bounces@lists.linuxfoundation.org">bitcoin-dev-bounces@lists.linu=
xfoundation.org</a>&gt; on behalf of Damian Williamson
 via bitcoin-dev &lt;<a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.or=
g">bitcoin-dev@lists.linuxfoundation.org</a>&gt;<br>
&gt; Sent: Friday, 8 December 2017 8:01 AM<br>
&gt; To: <a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-d=
ev@lists.linuxfoundation.org</a><br>
&gt; Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transacti=
on Priority For Ordering Transactions In Blocks<br>
&gt; &nbsp;<br>
&gt;<br>
&gt;<br>
&gt; Good afternoon,<br>
&gt;<br>
&gt; The need for this proposal:<br>
&gt;<br>
&gt; We all must learn to admit that transaction bandwidth is still lurking=
 as a serious issue for the operation, reliability, safety, consumer accept=
ance, uptake and, for the value of Bitcoin.<br>
&gt;<br>
&gt; I recently sent a payment which was not urgent so; I chose three-day t=
arget confirmation from the fee recommendation. That transaction has still =
not confirmed after now more than six days - even waiting twice as long see=
ms quite reasonable to me. That transaction
 is a valid transaction; it is not rubbish, junk or, spam. Under the curren=
t model with transaction bandwidth limitation, the longer a transaction wai=
ts, the less likely it is ever to confirm due to rising transaction numbers=
 and being pushed back by transactions
 with rising fees.<br>
&gt;<br>
&gt; I argue that no transactions are rubbish or junk, only some zero fee t=
ransactions might be spam. Having an ever-increasing number of valid transa=
ctions that do not confirm as more new transactions with higher fees are cr=
eated is the opposite of operating
 a robust, reliable transaction system.<br>
&gt;<br>
&gt; Business cannot operate with a model where transactions may or may not=
 confirm. Even a business choosing a modest fee has no guarantee that their=
 valid transaction will not be shuffled down by new transactions to the rea=
lm of never confirming after it is
 created. Consumers also will not accept this model as Bitcoin expands. If =
Bitcoin cannot be a reliable payment system for confirmed transactions then=
 consumers, by and large, will simply not accept the model once they unders=
tand. Bitcoin will be a dirty payment
 system, and this will kill the value of Bitcoin.<br>
&gt;<br>
&gt; Under the current system, a minority of transactions will eventually b=
e the lucky few who have fees high enough to escape being pushed down the l=
ist.<br>
&gt;<br>
&gt; Once there are more than x transactions (transaction bandwidth limit) =
every ten minutes, only those choosing twenty-minute confirmation (2 blocks=
) will have initially at most a fifty percent chance of ever having their p=
ayment confirm. Presently, not even
 using fee recommendations can ensure a sufficiently high fee is paid to en=
sure transaction confirmation.<br>
&gt;<br>
&gt; I also argue that the current auction model for limited transaction ba=
ndwidth is wrong, is not suitable for a reliable transaction system and, is=
 wrong for Bitcoin. All transactions must confirm in due time. Currently, B=
itcoin is not a safe way to send payments.<br>
&gt;<br>
&gt; I do not believe that consumers and business are against paying fees, =
even high fees. What is required is operational reliability.<br>
&gt;<br>
&gt; This great issue needs to be resolved for the safety and reliability o=
f Bitcoin. The time to resolve issues in commerce is before they become gre=
at big issues. The time to resolve this issue is now. We must have the fore=
sight to identify and resolve problems
 before they trip us over.&nbsp; Simply doubling block sizes every so often=
 is reactionary and is not a reliable permanent solution. I have written a =
BIP proposal for a technical solution but, need your help to write it up to=
 an acceptable standard to be a full
 BIP.<br>
&gt;<br>
&gt; I have formatted the following with markdown which is human readable s=
o, I hope nobody minds. I have done as much with this proposal as I feel th=
at I am able so far but continue to take your feedback.<br>
&gt;<br>
&gt; # BIP Proposal: UTPFOTIB - Use Transaction Priority For Ordering Trans=
actions In Blocks<br>
&gt;<br>
&gt; ## The problem:<br>
&gt; Everybody wants value. Miners want to maximize revenue from fees (and =
we presume, to minimize block size). Consumers need transaction reliability=
 and, (we presume) want low fees.<br>
&gt;<br>
&gt; The current transaction bandwidth limit is a limiting factor for both.=
 As the operational safety of transactions is limited, so is consumer confi=
dence as they realize the issue and, accordingly, uptake is limited. Fees a=
re artificially inflated due to bandwidth
 limitations while failing to provide a full confirmation service for all t=
ransactions.<br>
&gt;<br>
&gt; Current fee recommendations provide no satisfaction for transaction re=
liability and, as Bitcoin scales, this will worsen.<br>
&gt;<br>
&gt; Bitcoin must be a fully scalable and reliable service, providing full =
transaction confirmation for every valid transaction.<br>
&gt;<br>
&gt; The possibility to send a transaction with a fee lower than one that i=
s acceptable to allow eventual transaction confirmation should be removed f=
rom the protocol and also from the user interface.<br>
&gt;<br>
&gt; ## Solution summary:<br>
&gt; Provide each transaction with an individual transaction priority each =
time before choosing transactions to include in the current block, the prio=
rity being a function of the fee paid (on a curve), and the time waiting in=
 the transaction pool (also on a curve)
 out to n days (n=3D60 ?). The transaction priority to serve as the likelih=
ood of a transaction being included in the current block, and for determini=
ng the order in which transactions are tried to see if they will be include=
d.<br>
&gt;<br>
&gt; Use a target block size. Determine the target block size using; curren=
t transaction pool size x ( 1 / (144 x n days ) ) =3D number of transaction=
s to be included in the current block. Broadcast the next target block size=
 with the current block when it is solved
 so that nodes know the next target block size for the block that they are =
building on.<br>
&gt;<br>
&gt; The curves used for the priority of transactions would have to be appr=
opriate. Perhaps a mathematician with experience in probability can develop=
 the right formulae. My thinking is a steep curve. I suppose that the proba=
bility of all transactions should probably
 account for a sufficient number of inclusions that the target block size i=
s met although, it may not always be. As a suggestion, consider including s=
ome zero fee transactions to pad, highest BTC value first?<br>
&gt;<br>
&gt; **Explanation of the operation of priority:**<br>
&gt; &gt; If transaction priority is, for example, a number between one (lo=
w) and one-hundred (high) it can be directly understood as the percentage c=
hance in one-hundred of a transaction being included in the block. Using pr=
obability or likelihood infers that there
 is some function of random. If random (100) &lt; transaction priority then=
 the transaction is included.<br>
&gt;<br>
&gt; &gt;To break it down further, if both the fee on a curve value and the=
 time waiting on a curve value are each a number between one and one-hundre=
d, a rudimentary method may be to simply multiply those two numbers, to fin=
d the priority number. For example, a
 middle fee transaction waiting thirty days (if n =3D 60 days) may have a v=
alue of five for each part &nbsp;(yes, just five, the values are on a curve=
). When multiplied that will give a priority value of twenty-five, or, &nbs=
p;a twenty-five percent chance at that moment
 of being included in the block; it will likely be included in one of the n=
ext four blocks, getting more likely each chance. If it is still not includ=
ed then the value of time waiting will be higher, making for more probabili=
ty. A very low fee transaction would
 have a value for the fee of one. It would not be until near sixty-days tha=
t the particular low fee transaction has a high likelihood of being include=
d in the block.<br>
&gt;<br>
&gt; I am not concerned with low (or high) transaction fees, the primary re=
ason for addressing the issue is to ensure transactional reliability and sc=
alability while having each transaction confirm in due time.<br>
&gt;<br>
&gt; ## Pros:<br>
&gt; * Maximizes transaction reliability.<br>
&gt; * Fully scalable.<br>
&gt; * Maximizes possibility for consumer and business uptake.<br>
&gt; * Maximizes total fees paid per block without reducing reliability; be=
cause of reliability, in time confidence and overall uptake are greater; th=
erefore, more transactions.<br>
&gt; * Market determines fee paid for transaction priority.<br>
&gt; * Fee recommendations work all the way out to 30 days or greater.<br>
&gt; * Provides additional block entropy; greater security since there is l=
ess probability of predicting the next block.<br>
&gt;<br>
&gt; ## Cons:<br>
&gt; * Could initially lower total transaction fees per block.<br>
&gt; * Must be first be programmed.<br>
&gt;<br>
&gt; ## Solution operation:<br>
&gt; This is a simplistic view of the operation. The actual operation will =
need to be determined in a spec for the programmer.<br>
&gt;<br>
&gt; 1. Determine the target block size for the current block.<br>
&gt; 2. Assign a transaction priority to each transaction in the pool.<br>
&gt; 3. Select transactions to include in the current block using probabili=
ty in transaction priority order until the target block size is met.<br>
&gt; 5. Solve block.<br>
&gt; 6. Broadcast the next target block size with the current block when it=
 is solved.<br>
&gt; 7. Block is received.<br>
&gt; 8. Block verification process.<br>
&gt; 9. Accept/reject block based on verification result.<br>
&gt; 10. Repeat.<br>
&gt;<br>
&gt; ## Closing comments:<br>
&gt; It may be possible to verify blocks conform to the proposal by showing=
 that the probability for all transactions included in the block statistica=
lly conforms to a probability distribution curve, *if* the individual trans=
action priority can be recreated. I
 am not that deep into the mathematics; however, it may also be possible to=
 use a similar method to do this just based on the fee, that statistically,=
 the blocks conform to a fee distribution. Any zero fee transactions would =
have to be ignored. This solution
 needs a clever mathematician.<br>
&gt;<br>
&gt; I implore, at the very least, that we use some method that validates f=
ull transaction reliability and enables scalability of block sizes. If not =
this proposal, an alternative.<br>
&gt;<br>
&gt; Regards,<br>
&gt; Damian Williamson<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt; _______________________________________________<br>
&gt; bitcoin-dev mailing list<br>
&gt; <a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@l=
ists.linuxfoundation.org</a><br>
&gt; <a href=3D"https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-=
dev">https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev</a><br>
&gt;<br>
</div>
</div>
</div>
</div>
</div>
</body>
</html>

--_000_PS2P216MB01798757147723AF5B7B7CA49D0F0PS2P216MB0179KORP_--