Received-SPF: pass (sog-mx-4.v43.ch3.sourceforge.com: domain of gmail.com
	designates 209.85.223.175 as permitted sender)
	client-ip=209.85.223.175; envelope-from=gmaxwell@gmail.com;
	helo=mail-ie0-f175.google.com; 
MIME-Version: 1.0
In-Reply-To: <1348707206.1193.16.camel@localhost.localdomain>
References: <5061F8CC.9070906@mistfpga.net>
	<1348605677.2284.2.camel@localhost.localdomain>
	<5062F4F8.6040504@mistfpga.net>
	<1348707206.1193.16.camel@localhost.localdomain>
Date: Wed, 26 Sep 2012 22:29:30 -0400
Message-ID: <CAAS2fgRbco9Qaq8AvkZFL9FhDe6C-dTH9KgyxFg-GXAOhYCEWA@mail.gmail.com>
From: Gregory Maxwell <gmaxwell@gmail.com>
To: Matt Corallo <bitcoin-list@bluematt.me>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Cc: Bitcoin Development List <bitcoin-development@lists.sourceforge.net>,
	Bill Hees <billhees@gmail.com>
Subject: Re: [Bitcoin-development] Bitcoin Testing Project
Precedence: list

On Wed, Sep 26, 2012 at 8:53 PM, Matt Corallo <bitcoin-list@bluematt.me> wr=
ote:
> Jenkins currently just runs the test script after each new commit to
> bitcoin (and provides binaries to anyone who wants them), so its pretty
> basic (though jenkins has way more features than we use).  The bitcoin
> one lives at http://jenkins.bluematt.me/

Jenkins is excellent at cycling through tests,  while additional
external tools may bring some value they're not required.  It's also
essential to automate all tests that we really care are run=E2=80=94 with o=
ur
small active development group and volunteer contributors the only
tests we can count on being run are the automated ones. Automated
tests included with the software=E2=80=94 or at least the source=E2=80=94 a=
re also the
only way to have a good chance of catching gnarly platform
interactions.

I think more than talking about testing I think we need is actual
testing. Code coverage from the current tests (e.g. bitcoin-test and a
testnet sync) is very unimpressive, and while coverage isn't some
magical silver bullet and does not, by itself, mean the tests are good
flaws in uncovered code can't be detected by the tests.  We also lack
simple testing cycle documentation for people interested in testing
manually to walk through, etc. I think all the meta discussion is not
very useful until we actually have more substance to put into it.
Otherwise I fear we're just building an airport by painting stripes
and waiting for the planes to land...

If someone wants to help and would like a list of some of the testing
I think would be useful, ping me off-list and I can blast some
suggestions. But I assume that anyone who actually wants to work on
this isn't short of ideas, and at this point "work on what interests
you, report what interesting thing you accomplish or discover" is
probably a perfectly fine level of coordination.