MIME-Version: 1.0
In-Reply-To: <5631C363.5060705@neomailbox.net>
References: <5631C363.5060705@neomailbox.net>
Date: Thu, 29 Oct 2015 08:17:27 +0000
Message-ID: <CAAS2fgQ=+5b4tht9SeGVJPXDbqKan=Y8x7Q5F=uT5iiq+BnHJA@mail.gmail.com>
From: Gregory Maxwell <gmaxwell@gmail.com>
To: telemaco <telemaco@neomailbox.net>
Content-Type: text/plain; charset=UTF-8
Cc: Bitcoin Dev <bitcoin-dev@lists.linuxfoundation.org>
Subject: Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db
Precedence: list

On Thu, Oct 29, 2015 at 6:57 AM, telemaco via bitcoin-dev
<bitcoin-dev@lists.linuxfoundation.org> wrote:
> Why not "outsource" totally that data management part to the already
> existing with decades of experience database world. People would be able to
> create incredibly easy bitcoin statistics/graphs/analisys with existing
> software packages (hey even excel or libreoffice like) or connect bitcoin
> data to their own sources and if so they chose analyze bitcoin data on a
> datawarehouse or any imaginable approach. Of course every transaction would
> be have to do through the bitcoin node and only the data management would be
> on rdbms side.

The word "database" is likely confusing people here.  This is not a
database in an ordinary sense.

The bitcoin core consensus engine requires a highly optimized ultra
compact data structure to perform the lookups for coin existence. The
data stored is highly compressed and very specialized, it would not be
useful to other applications.  Right now, on boring laptop hardware,
during network synchronization updates to this database run at over
10,000 records per second, while the system is also busy doing the
other validation chores of a node. This is backended by a high
performance transactional key value store.  The need for performance
here is essential to even keeping up with the network, it's not about
enabling any kind of fancy querying (bitcoin core does not offer fancy
querying), it's about the base load that every node must handle to
usably sync up and keep up with the Bitcoin network.

The backend can be swapped out for something else that provides the
same properties, but doing so does not give you any of the
inspection/analytics that you're looking for.  Systems that do that
exist, and they require databases taking hundreds of gigabytes of
storage and take days to weeks to import the network data.  They're
great for what they're for, but they're not suitable for consensus use
in the system for space efficiency, performance, and consensus
consistency reasons.