path: root/0c/833bf73d8524fe779f0c5ac8cad32a94f8e430

Return-Path: <hearn@vinumeris.com>
Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org
	[172.17.192.35])
	by mail.linuxfoundation.org (Postfix) with ESMTPS id E803C3EE
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Tue, 20 Oct 2015 10:12:13 +0000 (UTC)
X-Greylist: whitelisted by SQLgrey-1.7.6
Received: from mail-ig0-f172.google.com (mail-ig0-f172.google.com
	[209.85.213.172])
	by smtp1.linuxfoundation.org (Postfix) with ESMTPS id CD9CBA9
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Tue, 20 Oct 2015 10:12:12 +0000 (UTC)
Received: by igbni9 with SMTP id ni9so68320648igb.1
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Tue, 20 Oct 2015 03:12:12 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
	d=vinumeris.com; s=google;
	h=mime-version:in-reply-to:references:date:message-id:subject:from:to
	:cc:content-type;
	bh=+eJf9QL3JcrGdODQ0lESHLUb1tOScVfzsP5rJK9J9F0=;
	b=M/3tuJZcGpo9synulAf1wv1MiqfjFxkIXTXq2hEhC33mMtJ7q9khXLb5wbFBzkc0wg
	umRmGnq6ucKZQXCfRVo7IEI16yBRfJ5IW7B2MKMIowujUzKPDgu7Ne4wTazlJUQ8FGZj
	/G8MRKvOJIcVleNxIF/jyhHda2uJbhd9+m1so=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
	d=1e100.net; s=20130820;
	h=x-gm-message-state:mime-version:in-reply-to:references:date
	:message-id:subject:from:to:cc:content-type;
	bh=+eJf9QL3JcrGdODQ0lESHLUb1tOScVfzsP5rJK9J9F0=;
	b=jL4mbhXTeyJ2UYUpTXksMdkRj8S7MJQ7hGjLaMjZj1x3Lw+J8p6hY293M/V12xk7eZ
	yE6/aIUguhMQ87qptDzs3P+KjXqvlba+yN6P/HDpNtYxZm6Xx9BdYgF8N1//uz9l7eTo
	I6i1Gv3eYW7RCV7Ufn+b7nU6QOpxoDCGlJmD5rQY1vRN8BfzEYmflOWjsQZmdU6XTsfg
	nSkkbMRej9Tcytlt8mV6v5YZnoiNRr8G0e0RhEuLk1fHiZ/pb+wUt7NDMcBCSIApCsHS
	ckk1FjQlEMnoXqVzk6YiR35c7SxmVCJAJ0o6nsXs9DOC6NCEQaiJABvOgq7xN7aNGb9f
	biQA==
X-Gm-Message-State: ALoCoQnXEHVtmiKyryK2zW4E0uJLrm7wRybc0opsPSTSE0pAa+weETJfbX7lEl8SEh9rkoTWlRIu
MIME-Version: 1.0
X-Received: by 10.50.103.99 with SMTP id fv3mr23349471igb.69.1445335932201;
	Tue, 20 Oct 2015 03:12:12 -0700 (PDT)
Received: by 10.50.123.166 with HTTP; Tue, 20 Oct 2015 03:12:12 -0700 (PDT)
In-Reply-To: <CAP3QyGJNBdsBtxYjOprRJ=YW2v-N_CopVQeSgDs6J4J8LMWuxA@mail.gmail.com>
References: <99C42DE7-814A-48F8-AB28-A5ADD77A9FD9@toom.im>
	<20151014093913.GB19607@amethyst.visucore.com>
	<F938BD02-3D80-4E99-BD1C-490543187895@toom.im>
	<CAP3QyGJNBdsBtxYjOprRJ=YW2v-N_CopVQeSgDs6J4J8LMWuxA@mail.gmail.com>
Date: Tue, 20 Oct 2015 12:12:12 +0200
Message-ID: <CA+w+GKTU6C7KKFx9dDd_--s1DQCO15n=034Lku2-kTYKf96XYw@mail.gmail.com>
From: Mike Hearn <hearn@vinumeris.com>
To: bitcoin-xt <bitcoin-xt@googlegroups.com>
Content-Type: multipart/alternative; boundary=047d7b2e08abd2095e0522867e14
X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HTML_MESSAGE,RCVD_IN_DNSWL_LOW autolearn=ham
	version=3.3.1
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
	smtp1.linux-foundation.org
Cc: Bitcoin Dev <bitcoin-dev@lists.linuxfoundation.org>
Subject: Re: [bitcoin-dev] Memory leaks?
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Bitcoin Development Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Tue, 20 Oct 2015 10:12:14 -0000

--047d7b2e08abd2095e0522867e14
Content-Type: text/plain; charset=UTF-8

OK, then running under Valgrind whilst sending gbt RPCs would be the next
step.

On Mon, Oct 19, 2015 at 9:17 PM, Multipool Admin <admin@multipool.us> wrote:

> My nodes are continuously running getblocktemplate and getinfo, and I also
> suspected the issue is in either gbt or the rpc server.
>
> The instance only takes a few hours to get up to that memory usage.
> On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
>
>> On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan <laanwj@gmail.com>
>> wrote:
>>
>> This is *most likely* the mempool, but is just not reported correctly.
>>
>>
>> I did some testing with PR #6410's better mempool reporting. The improved
>> reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is
>> about 2.5x to 3x higher than the serialized transaction sizes ("bytes":).
>> The excess memory usage that I'm seeing is on the order of 100x higher than
>> the mempool "bytes": value. As such, I think it's unlikely that this is the
>> mempool, or at least not normal/correct mempool behavior.
>>
>> Another user (admin@multipool.us) reported 35 GB of RSS usage. I'm
>> guessing his bitcoind has been running longer than any of mine. His server
>> definitely has more RAM. I don't know which email list he is subscribed to
>> (probably XT), so I'm sharing it with both lists to make sure you're all
>> aware of how big an issue this can be.
>>
>> In the meantime you can mitigate the mempool growth by setting
>> `-mintxfee`, see
>>
>> https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding
>>
>>
>> I have mintxfee and minrelaytxfee set to about 0.00003, which is high
>> enough to exclude essentially all of the of the 14700-14800 byte flood
>> transactions. My nodes' mempools only contain about one or two blocks'
>> worth of transactions. So I don't think this is correct either.
>>
>>
>>
>> Some additional notes on this issue:
>>
>> 1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a
>> Core bitcoind process (commit d78a880) overnight with no mining connected
>> to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around
>> 400 MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I
>> ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around
>> 800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes,
>> and RSS climbed to 1180 MB. An hour after that with more getblocktemplates,
>> and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.]
>> getmempoolinfo is still showing "usage" around 25MB or less.
>>
>> I'll do some more testing with this and see if I can make it repeatable,
>> and record the results more carefully. Expect a follow-up from me in a day
>> or two.
>>
>> 2. valgrind did not show anything super promising. It did report this:
>>
>> ==6880== LEAK SUMMARY:
>> ==6880==    definitely lost: 0 bytes in 0 blocks
>> ==6880==    indirectly lost: 0 bytes in 0 blocks
>> ==6880==      possibly lost: 288 bytes in 1 blocks
>> ==6880==    still reachable: 10,552 bytes in 39 blocks
>> ==6880==         suppressed: 0 bytes in 0 blocks
>> (Bitcoin Core commit d78a880)
>>
>> and this:
>> ==6778== LEAK SUMMARY:
>> ==6778==    definitely lost: 0 bytes in 0 blocks
>> ==6778==    indirectly lost: 0 bytes in 0 blocks
>> ==6778==      possibly lost: 320 bytes in 1 blocks
>> ==6778==    still reachable: 10,080 bytes in 32 blocks
>> ==6778==         suppressed: 0 bytes in 0 blocks
>> (Bitcoin XT commit fe446d)
>>
>> I haven't found anything in there yet that I think would produce the
>> multi-GB memory usage after running for a few days, but I could be missing
>> it. Email me if you want the full log.
>>
>> I did not try running getblocktemplate while valgrind was running. I'll
>> have to try that. I also have not let valgrind run for more than an hour.
>>
>>
>>
>> P.S.: Sorry for all the cross-post confusion and consequent flamewar
>> fallout. While it's probably too late for this thread, I'll make sure to
>> post in a manner that keeps the threads clearly separate in the future
>> (e.g. different subject lines).
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "bitcoin-xt" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to bitcoin-xt+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

--047d7b2e08abd2095e0522867e14
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">OK, then running under Valgrind whilst sending gbt RPCs wo=
uld be the next step.</div><div class=3D"gmail_extra"><br><div class=3D"gma=
il_quote">On Mon, Oct 19, 2015 at 9:17 PM, Multipool Admin <span dir=3D"ltr=
">&lt;<a href=3D"mailto:admin@multipool.us" target=3D"_blank">admin@multipo=
ol.us</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"m=
argin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir=3D"ltr=
">My nodes are continuously running getblocktemplate and getinfo, and I als=
o suspected the issue is in either gbt or the rpc server.</p>
<p dir=3D"ltr">The instance only takes a few hours to get up to that memory=
 usage.</p>
<div class=3D"gmail_quote"><div><div class=3D"h5">On Oct 18, 2015 8:59 AM, =
&quot;Jonathan Toomim via bitcoin-dev&quot; &lt;<a href=3D"mailto:bitcoin-d=
ev@lists.linuxfoundation.org" target=3D"_blank">bitcoin-dev@lists.linuxfoun=
dation.org</a>&gt; wrote:<br type=3D"attribution"></div></div><blockquote c=
lass=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;=
padding-left:1ex"><div><div class=3D"h5"><div style=3D"word-wrap:break-word=
"><div><div>On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan &lt;<a hr=
ef=3D"mailto:laanwj@gmail.com" target=3D"_blank">laanwj@gmail.com</a>&gt; w=
rote:</div><blockquote type=3D"cite">This is *most likely* the mempool, but=
 is just not reported correctly.<br></blockquote><div><br></div><div>I did =
some testing with PR #6410&#39;s better mempool reporting. The improved rep=
orting suggests that actual in-memory usage (&quot;usage&quot;:) by CTxMemP=
ool is about 2.5x to 3x higher than the serialized transaction sizes (&quot=
;bytes&quot;:). The excess memory usage that I&#39;m seeing is on the order=
 of 100x higher than the mempool &quot;bytes&quot;: value. As such, I think=
 it&#39;s unlikely that this is the mempool, or at least not normal/correct=
 mempool behavior.=C2=A0</div><div><br></div><div>Another user (<a href=3D"=
mailto:admin@multipool.us" target=3D"_blank">admin@multipool.us</a>) report=
ed 35 GB of RSS usage. I&#39;m guessing his bitcoind has been running longe=
r than any of mine. His server definitely has more RAM. I don&#39;t know wh=
ich email list he is subscribed to (probably XT), so I&#39;m sharing it wit=
h both lists to make sure you&#39;re all aware of how big an issue this can=
 be.</div><br><blockquote type=3D"cite">In the meantime you can mitigate th=
e mempool growth by setting `-mintxfee`, see<br><a href=3D"https://github.c=
om/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding" =
target=3D"_blank">https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/relea=
se-notes.md#transaction-flooding</a><br></blockquote><div><br></div>I have =
mintxfee and minrelaytxfee set to about=C2=A00.00003, which is high enough =
to exclude essentially all of the of the 14700-14800 byte flood transaction=
s. My nodes&#39; mempools only contain about one or two blocks&#39; worth o=
f transactions. So I don&#39;t think this is correct either.</div><div><br>=
</div><div><br></div><div><br></div><div>Some additional notes on this issu=
e:</div><div><br></div><div>1. I think it&#39;s related to CreateNewBlock()=
 and getblocktemplate. I ran a Core bitcoind process (commit d78a880) overn=
ight with no mining connected to it, and (IIRC -- my memory is fuzzy) when =
I woke up it was using around 400 MB of RSS and the mempool was at around &=
quot;bytes&quot;:10MB, &quot;usage&quot;: 25MB. I ran ./bitcoin-cli getbloc=
ktemplate once, and IIRC the RSS shot up to around 800 MB. I then ran getbl=
ocktemplate every 5 seconds for about 30 minutes, and RSS climbed to 1180 M=
B. An hour after that with more getblocktemplates, and now RSS is at 1350 M=
B. [Edit: 1490 MB about 30 minutes later.] getmempoolinfo is still showing =
&quot;usage&quot; around 25MB or less.</div><div><br></div><div>I&#39;ll do=
 some more testing with this and see if I can make it repeatable, and recor=
d the results more carefully. Expect a follow-up from me in a day or two.</=
div><div><br></div><div>2. valgrind did not show anything super promising. =
It did report this:</div><div><br></div><div><div style=3D"margin:0px;font-=
family:&#39;Andale Mono&#39;;color:rgb(41,249,20);background-color:rgb(0,0,=
0)">=3D=3D6880=3D=3D LEAK SUMMARY:</div><div style=3D"margin:0px;font-famil=
y:&#39;Andale Mono&#39;;color:rgb(41,249,20);background-color:rgb(0,0,0)">=
=3D=3D6880=3D=3D=C2=A0 =C2=A0 definitely lost: 0 bytes in 0 blocks</div><di=
v style=3D"margin:0px;font-family:&#39;Andale Mono&#39;;color:rgb(41,249,20=
);background-color:rgb(0,0,0)">=3D=3D6880=3D=3D=C2=A0 =C2=A0 indirectly los=
t: 0 bytes in 0 blocks</div><div style=3D"margin:0px;font-family:&#39;Andal=
e Mono&#39;;color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6880=3D=
=3D=C2=A0 =C2=A0 =C2=A0 possibly lost: 288 bytes in 1 blocks</div><div styl=
e=3D"margin:0px;font-family:&#39;Andale Mono&#39;;color:rgb(41,249,20);back=
ground-color:rgb(0,0,0)">=3D=3D6880=3D=3D=C2=A0 =C2=A0 still reachable: 10,=
552 bytes in 39 blocks</div><div style=3D"margin:0px;font-family:&#39;Andal=
e Mono&#39;;color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6880=3D=
=3D =C2=A0 =C2=A0 =C2=A0 =C2=A0 suppressed: 0 bytes in 0 blocks</div></div>=
<div>(Bitcoin Core commit d78a880)</div><div><br></div><div>and this:</div>=
<div><div style=3D"margin:0px;font-family:&#39;Andale Mono&#39;;color:rgb(4=
1,249,20);background-color:rgb(0,0,0)">=3D=3D6778=3D=3D LEAK SUMMARY:</div>=
<div style=3D"margin:0px;font-family:&#39;Andale Mono&#39;;color:rgb(41,249=
,20);background-color:rgb(0,0,0)">=3D=3D6778=3D=3D=C2=A0 =C2=A0 definitely =
lost: 0 bytes in 0 blocks</div><div style=3D"margin:0px;font-family:&#39;An=
dale Mono&#39;;color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6778=
=3D=3D=C2=A0 =C2=A0 indirectly lost: 0 bytes in 0 blocks</div><div style=3D=
"margin:0px;font-family:&#39;Andale Mono&#39;;color:rgb(41,249,20);backgrou=
nd-color:rgb(0,0,0)">=3D=3D6778=3D=3D=C2=A0 =C2=A0 =C2=A0 possibly lost: 32=
0 bytes in 1 blocks</div><div style=3D"margin:0px;font-family:&#39;Andale M=
ono&#39;;color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6778=3D=3D=
=C2=A0 =C2=A0 still reachable: 10,080 bytes in 32 blocks</div><div style=3D=
"margin:0px;font-family:&#39;Andale Mono&#39;;color:rgb(41,249,20);backgrou=
nd-color:rgb(0,0,0)">=3D=3D6778=3D=3D =C2=A0 =C2=A0 =C2=A0 =C2=A0 suppresse=
d: 0 bytes in 0 blocks</div></div><div>(Bitcoin XT commit=C2=A0fe446d)</div=
><div><br></div><div>I haven&#39;t found anything in there yet that I think=
 would produce the multi-GB memory usage after running for a few days, but =
I could be missing it. Email me if you want the full log.</div><div><br></d=
iv><div>I did not try running getblocktemplate while valgrind was running. =
I&#39;ll have to try that. I also have not let valgrind run for more than a=
n hour.=C2=A0</div><div><br></div><div><br></div><div><br></div><div>P.S.: =
Sorry for all the cross-post confusion and consequent flamewar fallout. Whi=
le it&#39;s probably too late for this thread, I&#39;ll make sure to post i=
n a manner that keeps the threads clearly separate in the future (e.g. diff=
erent subject lines).</div></div><br></div></div><span class=3D"">_________=
______________________________________<br>
bitcoin-dev mailing list<br>
<a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org" target=3D"_blank">=
bitcoin-dev@lists.linuxfoundation.org</a><br>
<a href=3D"https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev" =
rel=3D"noreferrer" target=3D"_blank">https://lists.linuxfoundation.org/mail=
man/listinfo/bitcoin-dev</a><br>
<br></span></blockquote></div><div class=3D"HOEnZb"><div class=3D"h5">

<p></p>

-- <br>
You received this message because you are subscribed to the Google Groups &=
quot;bitcoin-xt&quot; group.<br>
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to <a href=3D"mailto:bitcoin-xt+unsubscribe@googlegroups.com" target=
=3D"_blank">bitcoin-xt+unsubscribe@googlegroups.com</a>.<br>
For more options, visit <a href=3D"https://groups.google.com/d/optout" targ=
et=3D"_blank">https://groups.google.com/d/optout</a>.<br>
</div></div></blockquote></div><br></div>

--047d7b2e08abd2095e0522867e14--