Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id D0FC64D3 for ; Wed, 21 Oct 2015 03:01:24 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from c.mail.sonic.net (c.mail.sonic.net [64.142.111.80]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 30707E1 for ; Wed, 21 Oct 2015 03:01:22 +0000 (UTC) Received: from [192.168.1.190] (63.135.62.197.nwinternet.com [63.135.62.197] (may be forged)) (authenticated bits=0) by c.mail.sonic.net (8.15.1/8.15.1) with ESMTPSA id t9L31H9j002198 (version=TLSv1 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Tue, 20 Oct 2015 20:01:18 -0700 Content-Type: multipart/signed; boundary="Apple-Mail=_83BDB11D-1F73-4B97-9508-FF4D7B4BDA07"; protocol="application/pgp-signature"; micalg=pgp-sha512 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) X-Pgp-Agent: GPGMail 2.5.2 From: Jonathan Toomim In-Reply-To: <1FE17DEB-8F77-4A60-A644-46A4F97D0E24@toom.im> Date: Tue, 20 Oct 2015 20:01:16 -0700 Message-Id: <984D5FD5-9871-43FC-BD44-5F2E6EFD0671@toom.im> References: <99C42DE7-814A-48F8-AB28-A5ADD77A9FD9@toom.im> <20151014093913.GB19607@amethyst.visucore.com> <1FE17DEB-8F77-4A60-A644-46A4F97D0E24@toom.im> To: Bitcoin Dev , bitcoin-xt X-Mailer: Apple Mail (2.1878.6) X-Sonic-CAuth: UmFuZG9tSVYREh/4W0hJpIYP7Lo7CrnE7dvhQXDd/nuaPuwSqQR46/Qy1DDPgBnlC8yq37Yfku+Yn0uWu0sHYBNBdYJjHfaR X-Sonic-ID: C;Yja0/5935RGSWL0U9jFv0A== M;vAM3AKB35RGSWL0U9jFv0A== X-Sonic-Spam-Details: 0.0/5.0 by cerberusd X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,HTML_MESSAGE, RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: Re: [bitcoin-dev] Memory leaks? X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Development Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Oct 2015 03:01:24 -0000 --Apple-Mail=_83BDB11D-1F73-4B97-9508-FF4D7B4BDA07 Content-Type: multipart/alternative; boundary="Apple-Mail=_AD29A490-0C81-493D-A9F4-B7389A1D22F4" --Apple-Mail=_AD29A490-0C81-493D-A9F4-B7389A1D22F4 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii More notes: 1. I ran a side-by-side comparison with two bitcoind processes (Core, = same recent git commit as before) on the same computer with the same = settings running on different ports. With both processes, I logged RSS = (via /proc/$pid/status) every 6 seconds. With one of those processes, I = also ran bitcoin-cli getblocktemplate > /dev/null every 6 seconds. I let = that run for about 30 hours. A graph and links to the CSVs of raw data = are below. Results seem pretty clear: the getblocktemplate RPC is = implicated in this issue. http://toom.im/files/memlog8518.csv http://toom.im/files/memlog-nogbt-8503.csv http://toom.im/files/bitcoind_memory_usage_gbt.png 2. I ran valgrind twice, for about 6 hours each, on bitcoind while = hitting it with getblocktemplate every 6 hours. Full valgrind output can = be found at these two URLs: http://toom.im/files/valgrind-gbt-1.log http://toom.im/files/valgrind-gbt-2.log The summary: =3D=3D4064=3D=3D LEAK SUMMARY: =3D=3D4064=3D=3D definitely lost: 0 bytes in 0 blocks =3D=3D4064=3D=3D indirectly lost: 0 bytes in 0 blocks =3D=3D4064=3D=3D possibly lost: 288 bytes in 1 blocks =3D=3D4064=3D=3D still reachable: 527,594 bytes in 4,367 blocks =3D=3D4064=3D=3D suppressed: 0 bytes in 0 blocks The main components of that still reachable section seem to just be one = output of CreateNewBlock that's cached in case another getblocktemplate = request is received before any new transactions come in: =3D=3D4064=3D=3D 98,304 bytes in 1 blocks are still reachable in loss = record 39 of 40 =3D=3D4064=3D=3D at 0x4C29180: operator new(unsigned long) = (vg_replace_malloc.c:324) =3D=3D4064=3D=3D by 0x28EAA1: = __gnu_cxx::new_allocator::allocate(unsigned long, void = const*) (new_allocator.h:104) =3D=3D4064=3D=3D by 0x27EE44: = __gnu_cxx::__alloc_traits = >::allocate(std::allocator&, unsigned long) = (alloc_traits.h:182) =3D=3D4064=3D=3D by 0x26DFB0: std::_Vector_base >::_M_allocate(unsigned long) = (stl_vector.h:170) =3D=3D4064=3D=3D by 0x2D5BDE: std::vector = >::_M_insert_aux(__gnu_cxx::__normal_iterator > >, CTransaction = const&) (vector.tcc:353) =3D=3D4064=3D=3D by 0x2D3FF8: std::vector >::push_back(CTransaction const&) = (stl_vector.h:925) =3D=3D4064=3D=3D by 0x2D113E: CreateNewBlock(CScript const&) = (miner.cpp:298) =3D=3D4064=3D=3D by 0x442D78: getblocktemplate(UniValue const&, bool) = (rpcmining.cpp:513) =3D=3D4064=3D=3D by 0x390CEB: CRPCTable::execute(std::string const&, = UniValue const&) const (rpcserver.cpp:526) =3D=3D4064=3D=3D by 0x41C5AB: HTTPReq_JSONRPC(HTTPRequest*, = std::string const&) (httprpc.cpp:125) =3D=3D4064=3D=3D by 0x3559BD: = boost::detail::function::void_function_invoker2::invoke(boost::detail::function::function_buffer&, HTTPRequest*, = std::string const&) (function_template.hpp:112) =3D=3D4064=3D=3D by 0x422520: boost::function2::operator()(HTTPRequest*, std::string const&) const = (function_template.hpp:767) There are a few other similar loss records (mostly referring to pblock = or pblocktemplate in CreateNewBlock(...), but I see nothing that can = explain the multi-GB memory consumption. 3. One user on the bitcointalk p2pool thread = (https://bitcointalk.org/index.php?topic=3D18313.msg12733791#msg12733791) = claimed that he had this memory usage issue on Linux, but not on Mac OS = X, under a GBT workload in both situations. If this is true, that would = suggest this might be a fragmentation issue due to poor memory = allocation. The other likely hypothesis is bloated caches. Looking into = those two possibilities will be my next steps. On Oct 20, 2015, at 5:39 AM, Jonathan Toomim wrote: > I did that Sunday twice. I'll report the results soon. Short version = is that it looks like valgrind is just finding 200 kB to 600 kB of = pblocktemplate, which is declared as a static pointer. Not exactly the = multi-GB leak I'm looking for, but possibly related. >=20 > I've also got two bitcoind processes running on the same machine that = I started at the same time, running on different ports, all with the = same settings, but one of which is serving getblocktemplate every 5-6 = seconds and the other is not, while logging RSS on both every 6 seconds. = RSS for the non-serving node is now 734 MB, and for the serving node = 1997 MB. Graphs coming soon. >=20 >=20 > On Oct 20, 2015, at 3:12 AM, Mike Hearn wrote: >=20 >> OK, then running under Valgrind whilst sending gbt RPCs would be the = next step. >>=20 >> On Mon, Oct 19, 2015 at 9:17 PM, Multipool Admin = wrote: >> My nodes are continuously running getblocktemplate and getinfo, and I = also suspected the issue is in either gbt or the rpc server. >>=20 >> The instance only takes a few hours to get up to that memory usage. >>=20 >> On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" = wrote: >> On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan = wrote: >>> This is *most likely* the mempool, but is just not reported = correctly. >>=20 >> I did some testing with PR #6410's better mempool reporting. The = improved reporting suggests that actual in-memory usage ("usage":) by = CTxMemPool is about 2.5x to 3x higher than the serialized transaction = sizes ("bytes":). The excess memory usage that I'm seeing is on the = order of 100x higher than the mempool "bytes": value. As such, I think = it's unlikely that this is the mempool, or at least not normal/correct = mempool behavior. >>=20 >> Another user (admin@multipool.us) reported 35 GB of RSS usage. I'm = guessing his bitcoind has been running longer than any of mine. His = server definitely has more RAM. I don't know which email list he is = subscribed to (probably XT), so I'm sharing it with both lists to make = sure you're all aware of how big an issue this can be. >>=20 >>> In the meantime you can mitigate the mempool growth by setting = `-mintxfee`, see >>> = https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#trans= action-flooding >>=20 >> I have mintxfee and minrelaytxfee set to about 0.00003, which is high = enough to exclude essentially all of the of the 14700-14800 byte flood = transactions. My nodes' mempools only contain about one or two blocks' = worth of transactions. So I don't think this is correct either. >>=20 >>=20 >>=20 >> Some additional notes on this issue: >>=20 >> 1. I think it's related to CreateNewBlock() and getblocktemplate. I = ran a Core bitcoind process (commit d78a880) overnight with no mining = connected to it, and (IIRC -- my memory is fuzzy) when I woke up it was = using around 400 MB of RSS and the mempool was at around "bytes":10MB, = "usage": 25MB. I ran ./bitcoin-cli getblocktemplate once, and IIRC the = RSS shot up to around 800 MB. I then ran getblocktemplate every 5 = seconds for about 30 minutes, and RSS climbed to 1180 MB. An hour after = that with more getblocktemplates, and now RSS is at 1350 MB. [Edit: 1490 = MB about 30 minutes later.] getmempoolinfo is still showing "usage" = around 25MB or less. >>=20 >> I'll do some more testing with this and see if I can make it = repeatable, and record the results more carefully. Expect a follow-up = from me in a day or two. >>=20 >> 2. valgrind did not show anything super promising. It did report = this: >>=20 >> =3D=3D6880=3D=3D LEAK SUMMARY: >> =3D=3D6880=3D=3D definitely lost: 0 bytes in 0 blocks >> =3D=3D6880=3D=3D indirectly lost: 0 bytes in 0 blocks >> =3D=3D6880=3D=3D possibly lost: 288 bytes in 1 blocks >> =3D=3D6880=3D=3D still reachable: 10,552 bytes in 39 blocks >> =3D=3D6880=3D=3D suppressed: 0 bytes in 0 blocks >> (Bitcoin Core commit d78a880) >>=20 >> and this: >> =3D=3D6778=3D=3D LEAK SUMMARY: >> =3D=3D6778=3D=3D definitely lost: 0 bytes in 0 blocks >> =3D=3D6778=3D=3D indirectly lost: 0 bytes in 0 blocks >> =3D=3D6778=3D=3D possibly lost: 320 bytes in 1 blocks >> =3D=3D6778=3D=3D still reachable: 10,080 bytes in 32 blocks >> =3D=3D6778=3D=3D suppressed: 0 bytes in 0 blocks >> (Bitcoin XT commit fe446d) >>=20 >> I haven't found anything in there yet that I think would produce the = multi-GB memory usage after running for a few days, but I could be = missing it. Email me if you want the full log. >>=20 >> I did not try running getblocktemplate while valgrind was running. = I'll have to try that. I also have not let valgrind run for more than an = hour. >>=20 >>=20 >>=20 >> P.S.: Sorry for all the cross-post confusion and consequent flamewar = fallout. While it's probably too late for this thread, I'll make sure to = post in a manner that keeps the threads clearly separate in the future = (e.g. different subject lines). >>=20 >> _______________________________________________ >> bitcoin-dev mailing list >> bitcoin-dev@lists.linuxfoundation.org >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev >>=20 >>=20 >> -- >> You received this message because you are subscribed to the Google = Groups "bitcoin-xt" group. >> To unsubscribe from this group and stop receiving emails from it, = send an email to bitcoin-xt+unsubscribe@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >>=20 >=20 --Apple-Mail=_AD29A490-0C81-493D-A9F4-B7389A1D22F4 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii
More notes:

1. I ran a = side-by-side comparison with two bitcoind processes (Core, same recent = git commit as before) on the same computer with the same settings = running on different ports. With both processes, I logged RSS (via = /proc/$pid/status) every 6 seconds. With one of those processes, I also = ran bitcoin-cli getblocktemplate > /dev/null every 6 seconds. I let = that run for about 30 hours. A graph and links to the CSVs of raw data = are below. Results seem pretty clear: the getblocktemplate RPC is = implicated in this issue.

<= div>http://toom.im/files/memlog85= 18.csv

=

2. I ran valgrind twice, for about 6 hours each, on = bitcoind while hitting it with getblocktemplate every 6 hours. Full = valgrind output can be found at these two = URLs:


The = summary:

=3D=3D4064=3D=3D LEAK =
SUMMARY:
=3D=3D4064=3D=3D=
    definitely lost: 0 bytes in 0 blocks
=3D=3D4064=3D=3D    =
indirectly lost: 0 bytes in 0 blocks
=3D=3D4064=3D=3D      possibly lost: 288 =
bytes in 1 blocks
=3D=3D4064=3D=3D    still reachable: 527,594 bytes in =
4,367 blocks
=3D=3D4064=3D=3D=
         suppressed: 0 bytes in 0 =
blocks
The main components of that still = reachable section seem to just be one output of CreateNewBlock that's = cached in case another getblocktemplate request is received before any = new transactions come in:

=3D=3D4064=3D=3D 98,304 bytes in 1 blocks are =
still reachable in loss record 39 of 40
=3D=3D4064=3D=3D    at 0x4C29180: operator new(unsigned =
long) (vg_replace_malloc.c:324)
=3D=3D4064=3D=3D    by 0x28EAA1: =
__gnu_cxx::new_allocator<CTransaction>::allocate(unsigned long, =
void const*) (new_allocator.h:104)
=3D=3D4064=3D=3D    by 0x27EE44: =
__gnu_cxx::__alloc_traits<std::allocator<CTransaction> =
>::allocate(std::allocator<CTransaction>&, unsigned long) =
(alloc_traits.h:182)
=3D=3D4064=3D=3D&nbs=
p;   by 0x26DFB0: std::_Vector_base<CTransaction, =
std::allocator<CTransaction> >::_M_allocate(unsigned long) =
(stl_vector.h:170)
=3D=3D4064=3D=3D =
   by 0x2D5BDE: std::vector<CTransaction, =
std::allocator<CTransaction> =
>::_M_insert_aux(__gnu_cxx::__normal_iterator<CTransaction*, =
std::vector<CTransaction, std::allocator<CTransaction> > =
>, CTransaction const&) (vector.tcc:353)
=3D=3D4064=3D=3D    by 0x2D3FF8: =
std::vector<CTransaction, std::allocator<CTransaction> =
>::push_back(CTransaction const&) (stl_vector.h:925)
=3D=3D4064=3D=3D    by 0x2D113E: =
CreateNewBlock(CScript const&) (miner.cpp:298)
=3D=3D4064=3D=3D    by 0x442D78: =
getblocktemplate(UniValue const&, bool) =
(rpcmining.cpp:513)
=3D=3D4064=3D=3D =
;   by 0x390CEB: CRPCTable::execute(std::string const&, =
UniValue const&) const (rpcserver.cpp:526)
=3D=3D4064=3D=3D    by 0x41C5AB: =
HTTPReq_JSONRPC(HTTPRequest*, std::string const&) =
(httprpc.cpp:125)
=3D=3D4064=3D=3D  =
  by 0x3559BD: =
boost::detail::function::void_function_invoker2<bool =
(*)(HTTPRequest*, std::string const&), void, HTTPRequest*, =
std::string =
const&>::invoke(boost::detail::function::function_buffer&, =
HTTPRequest*, std::string const&) =
(function_template.hpp:112)
=3D=3D4064=3D=3D    by 0x422520: =
boost::function2<void, HTTPRequest*, std::string =
const&>::operator()(HTTPRequest*, std::string const&) const =
(function_template.hpp:767)
There are a few other similar loss records = (mostly referring to pblock or pblocktemplate in CreateNewBlock(...), = but I see nothing that can explain the multi-GB memory = consumption. 

3. One user on the = bitcointalk p2pool thread (https://bitcointalk.org/index.php?topic=3D18313.msg12733791#msg1273= 3791) claimed that he had this memory usage issue on Linux, but not = on Mac OS X, under a GBT workload in both situations. If this is true, = that would suggest this might be a fragmentation issue due to poor = memory allocation. The other likely hypothesis is bloated caches. = Looking into those two possibilities will be my next = steps.



On Oct 20, 2015, = at 5:39 AM, Jonathan Toomim <j@toom.im> wrote:

I did that Sunday twice. I'll = report the results soon. Short version is that it looks like valgrind is = just finding 200 kB to 600 kB of pblocktemplate, which is declared as a = static pointer. Not exactly the multi-GB leak I'm looking for, but = possibly related.

I've also got two bitcoind = processes running on the same machine that I started at the same time, = running on different ports, all with the same settings, but one of which = is serving getblocktemplate every 5-6 seconds and the other is not, = while logging RSS on both every 6 seconds. RSS for the non-serving node = is now 734 MB, and for the serving node 1997 MB. Graphs coming = soon.


On Oct 20, 2015, at = 3:12 AM, Mike Hearn <hearn@vinumeris.com> = wrote:

OK, then running under Valgrind whilst = sending gbt RPCs would be the next step.

On Mon, Oct 19, = 2015 at 9:17 PM, Multipool Admin <admin@multipool.us> = wrote:

My = nodes are continuously running getblocktemplate and getinfo, and I also = suspected the issue is in either gbt or the rpc server.

The instance only takes a few hours to get up to that memory = usage.

On Oct 18, 2015 8:59 = AM, "Jonathan Toomim via bitcoin-dev" <bitcoin-dev@lists.linuxfoundation.org> = wrote:
On Oct 14, 2015, at 2:39 AM, = Wladimir J. van der Laan <laanwj@gmail.com> wrote:
This is *most likely* the mempool, but is just not = reported correctly.

I did some = testing with PR #6410's better mempool reporting. The improved reporting = suggests that actual in-memory usage ("usage":) by CTxMemPool is about = 2.5x to 3x higher than the serialized transaction sizes ("bytes":). The = excess memory usage that I'm seeing is on the order of 100x higher than = the mempool "bytes": value. As such, I think it's unlikely that this is = the mempool, or at least not normal/correct mempool = behavior. 

Another user (admin@multipool.us) reported 35 GB of RSS usage. = I'm guessing his bitcoind has been running longer than any of mine. His = server definitely has more RAM. I don't know which email list he is = subscribed to (probably XT), so I'm sharing it with both lists to make = sure you're all aware of how big an issue this can = be.

In the meantime you can mitigate = the mempool growth by setting `-mintxfee`, see
https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/rele= ase-notes.md#transaction-flooding

I = have mintxfee and minrelaytxfee set to about 0.00003, which is high = enough to exclude essentially all of the of the 14700-14800 byte flood = transactions. My nodes' mempools only contain about one or two blocks' = worth of transactions. So I don't think this is correct = either.



Some = additional notes on this issue:

1. I think it's = related to CreateNewBlock() and getblocktemplate. I ran a Core bitcoind = process (commit d78a880) overnight with no mining connected to it, and = (IIRC -- my memory is fuzzy) when I woke up it was using around 400 MB = of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I ran = ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around = 800 MB. I then ran getblocktemplate every 5 seconds for about 30 = minutes, and RSS climbed to 1180 MB. An hour after that with more = getblocktemplates, and now RSS is at 1350 MB. [Edit: 1490 MB about 30 = minutes later.] getmempoolinfo is still showing "usage" around 25MB or = less.

I'll do some more testing with this and = see if I can make it repeatable, and record the results more carefully. = Expect a follow-up from me in a day or two.

2. = valgrind did not show anything super promising. It did report = this:

=3D=3D6880=3D=3D = LEAK SUMMARY:
=3D=3D6880=3D=3D&n= bsp;   definitely lost: 0 bytes in 0 blocks
=3D=3D6880=3D=3D&n= bsp;   indirectly lost: 0 bytes in 0 blocks
=3D=3D6880=3D=3D&n= bsp;     possibly lost: 288 bytes in 1 blocks
=3D=3D6880=3D=3D&n= bsp;   still reachable: 10,552 bytes in 39 blocks
=3D=3D6880=3D=3D =         suppressed: 0 bytes in 0 = blocks
(Bitcoin Core commit = d78a880)

and this:
=3D=3D6778=3D=3D = LEAK SUMMARY:
=3D=3D6778=3D=3D&n= bsp;   definitely lost: 0 bytes in 0 blocks
=3D=3D6778=3D=3D&n= bsp;   indirectly lost: 0 bytes in 0 blocks
=3D=3D6778=3D=3D&n= bsp;     possibly lost: 320 bytes in 1 blocks
=3D=3D6778=3D=3D&n= bsp;   still reachable: 10,080 bytes in 32 blocks
=3D=3D6778=3D=3D =         suppressed: 0 bytes in 0 = blocks
(Bitcoin XT = commit fe446d)

I haven't found anything in = there yet that I think would produce the multi-GB memory usage after = running for a few days, but I could be missing it. Email me if you want = the full log.

I did not try running = getblocktemplate while valgrind was running. I'll have to try that. I = also have not let valgrind run for more than an = hour. 



P.S.: = Sorry for all the cross-post confusion and consequent flamewar fallout. = While it's probably too late for this thread, I'll make sure to post in = a manner that keeps the threads clearly separate in the future (e.g. = different subject lines).

_______________________________________________
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitco= in-dev


--
You received this message because you are subscribed to the Google = Groups "bitcoin-xt" group.
To unsubscribe from this group and stop receiving emails from it, send = an email to bitcoin-xt+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

=


= --Apple-Mail=_AD29A490-0C81-493D-A9F4-B7389A1D22F4-- --Apple-Mail=_83BDB11D-1F73-4B97-9508-FF4D7B4BDA07 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJWJv/8AAoJEIEuMk4MG0P1Pl8IAKQjDuS1NPXKCxJ87HH2hGtI GC7Ei31/bYE0lxtKMYsN/AG3gI8JYpXRXEH1+A6Xn/H5lcfh1sl5iOibrxqCsyst 46Zps6/9ibdsSWuX1RQCARUf21jDpUytPssQm5AiOptlSLcsDdiGENvzuNYvIAT6 favryjktnHtQIJM4SD5BVS52lym5rt/9BKLSXH5BaXYyWPbrBy0CDMQAholxomiO Rj2xSwNwwe+yEn/kwIqQSQGOtjsPUTQBZPVcZPUP23gxk0O56GqTAjYt6wvOyKX3 jMoSF4yGSSggcFFGmIQc6pm/Cfr5g+uSmzNW7eI48R2irJaa3NN+SSDHns0ck14= =ZNB0 -----END PGP SIGNATURE----- --Apple-Mail=_83BDB11D-1F73-4B97-9508-FF4D7B4BDA07--