path: root/47/d90e2c95868d42eadb60c22e0b95b294994e7c

Return-Path: <nadav@shesek.info>
Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137])
 by lists.linuxfoundation.org (Postfix) with ESMTP id CC066C0176
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Mon,  1 Jun 2020 08:11:06 +0000 (UTC)
Received: from localhost (localhost [127.0.0.1])
 by fraxinus.osuosl.org (Postfix) with ESMTP id B4397884B0
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Mon,  1 Jun 2020 08:11:06 +0000 (UTC)
X-Virus-Scanned: amavisd-new at osuosl.org
Received: from fraxinus.osuosl.org ([127.0.0.1])
 by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id HGP1iOjcgCUa
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Mon,  1 Jun 2020 08:11:05 +0000 (UTC)
X-Greylist: from auto-whitelisted by SQLgrey-1.7.6
Received: from mail-lj1-f194.google.com (mail-lj1-f194.google.com
 [209.85.208.194])
 by fraxinus.osuosl.org (Postfix) with ESMTPS id 82249884AD
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Mon,  1 Jun 2020 08:11:05 +0000 (UTC)
Received: by mail-lj1-f194.google.com with SMTP id c17so1239860lji.11
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Mon, 01 Jun 2020 01:11:05 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shesek.info; s=shesek;
 h=mime-version:references:in-reply-to:from:date:message-id:subject:to
 :cc:content-transfer-encoding;
 bh=0hvwDUCuqLHSSYB81p040dFY7iYp3Vm9mhGhSR4K8/E=;
 b=lxcpG2/BEuBPOSHdPnnBi6BPLqcMWDUNGhlpNfl0Z+FSY8sKr1hXmHiHD/pjM+7sEb
 jVH3uOWi+Fr3yGuLC57F9t8s9y6yh/5BXLSwKXJ9hUbQhinp8AwcTTbWGXivZ00itixo
 wASyaHMsl2a9EWCXQKQl4i7BiWHVXp2myjP54=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:references:in-reply-to:from:date
 :message-id:subject:to:cc:content-transfer-encoding;
 bh=0hvwDUCuqLHSSYB81p040dFY7iYp3Vm9mhGhSR4K8/E=;
 b=R9RStKvjIjhyjhG2rdXnnL9Qu302NJsjC6ZcybXonY0t71Yz2NYvJhaxbsQ/hEFokH
 OWzmFjz4C/yubisPkD8pnXQlHpieHKaLOUHe9J0uUFsJigT2nIwDHteAXnZaKT+3wEPt
 mu2W+AnoeOANNeVTc9h2YGm0TVDoUQbZZFLJ7K5Pv3uTqXfTHa01fGkgvHP8JXcwabYn
 lZrT0IAjWZsnt7JB9U1uw3ifnEnS5lbnvNpzJF+NgHlxb8iqYiAr30ELhGRPIAEvJHMZ
 X/umT1LE628b7MewYVUjASBFtVDEkX6Uqh56+7fk4GgXEAKMyxbtMvgtaND5+vh08397
 M9gw==
X-Gm-Message-State: AOAM532kE0nxV+1HtU+yjVZNIt3346ROER3yLxzbaFWjE2xIapeeCT4K
 29daYzo0OvppqC/EMOr5iyHtZH4dkbfX6DHiCJtckg==
X-Google-Smtp-Source: ABdhPJxjzFLs3zVKPKi8RjJkiaePhMNlJWzppg0+1lAhIhvzJzFCrKSO66P67f2GTH0fiVXQAaJMeoKRvE1uHq3dtoc=
X-Received: by 2002:a2e:9e04:: with SMTP id e4mr9566545ljk.4.1590999063309;
 Mon, 01 Jun 2020 01:11:03 -0700 (PDT)
MIME-Version: 1.0
References: <CAGXD5f16OFOTb3fobOPNjrhFkmXk6w=V7cW1LijP+OumEh_3NQ@mail.gmail.com>
 <h-iXaWQ7GCW2m7hWhPPdBalMXR_yzNL6AbK-YV4o-gaHyTxIOu8SwfdVp2hsYL8XrbkV_pry20fKQPSfc2g0uk-r5IrMXX1Ci0A4R1fLr4A=@protonmail.com>
In-Reply-To: <h-iXaWQ7GCW2m7hWhPPdBalMXR_yzNL6AbK-YV4o-gaHyTxIOu8SwfdVp2hsYL8XrbkV_pry20fKQPSfc2g0uk-r5IrMXX1Ci0A4R1fLr4A=@protonmail.com>
From: Nadav Ivgi <nadav@shesek.info>
Date: Mon, 1 Jun 2020 11:10:50 +0300
Message-ID: <CAGXD5f2wZ+x1Mc7kak0+Qp5DmBp1ZoED4bu1xw9J7LE6sJxhmw@mail.gmail.com>
To: darosior <darosior@protonmail.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Mailman-Approved-At: Mon, 01 Jun 2020 10:23:13 +0000
Cc: Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org>
Subject: Re: [bitcoin-dev] Announcing Bitcoin Wallet Tracker
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: Bitcoin Protocol Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>, 
 <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>, 
 <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Mon, 01 Jun 2020 08:11:07 -0000

Hi Antoine,

I designed both APIs, so they definitely do share some similarities.

It's difficult to compare their performance directly, since
esplora-electrs keeps a full index of everything, while bwt tracks
your wallet addresses only.

If you're only interested in your wallet addresses and don't have a
*really* huge number of them, bwt will definitely perform better, as
it can avoid a lot of unnecessary indexing work. If you do have a lot
of addresses, esplora-electrs will be better suited for the job, as
its designed to deal with high volumes of data and does not rely on
the bitcoind wallet functionality, which was not designed for this.

I'm not sure where the line for "really huge" crosses exactly though,
I have not put this to the test. Definitely if you're tracking
millions of addresses, probably also for hundreds of thousands,
possibly even less?

API design wise, the main differences between the two are:

- Esplora provides every bit of information one might want to know
about transactions/blocks, while bwt intentionally tries to reduce
this to the subset useful in the context of app development.

- bwt provides wallet-contextual information, like key origins next to
addresses and the net change inflicted on the wallet's balance by
transactions.

- Esplora doesn't provide real-time updates (yet), while bwt provides
them using two different mechanisms (SSE and Web Hooks).

Nadav



On Sun, May 31, 2020 at 5:56 PM darosior <darosior@protonmail.com> wrote:
>
> Hi,
>
> I gave a quick look to the http API, and it seems very similar to Esplora=
's. So I wonder : how does
> bwt compares to Esplora, performance-wise ?
>
> Thanks!
> Antoine
>
>
> =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 Original =
Message =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90
> Le samedi, mai 30, 2020 4:16 PM, Nadav Ivgi via bitcoin-dev <bitcoin-dev@=
lists.linuxfoundation.org> a =C3=A9crit :
>
> Hi all,
>
> I recently released bwt [0], an HD wallet indexer implemented in Rust, us=
ing
> a model similar to that of Electrum Personal Server.
>
> It uses the bitcoind wallet functionality to do the heavy lifting and bui=
lds
> additional indexes on top of that, which can be queried using the Electru=
m
> RPC protocol, as well as a more modern, developer-friendly HTTP REST API.
>
> The electrum server can also be used as an electrum plugin [1], which
> integrates the server straight into the electrum client. From the user's
> perspective, this allows connecting electrum directly to a full node.
>
> The HTTP API is my take on a modern design for a wallet tracking API aime=
d
> at app developers. Some use-cases include using it as a backend for walle=
ts
> (similarly to Samuari's Dojo) or to track deposits to a watch-only xpub
> (similarly to BTCPay's NBXplorer).
>
> Compared to using the bitcoind RPC directly, bwt provides:
>
> - The ability to track an xpub and automatically have new addresses deriv=
ed
>   and imported as needed, according to the gap limit.
>
> - Two additional indexes, one for looking up the transaction history of
>   addresses, and another one for looking up txo spends (a map of
>   funding_txid:vout =3D> spending_txid:vin).
>
> - Real-time updates using Server-Sent Events [2] (a long-lived streaming =
HTTP
>   connection) or Web Hooks [3] (an HTTP request sent to a configured URL)=
.
>   The updates being sent [4] directly provide information about the funde=
d
>   and spent wallet txos, instead of the client figuring it out from the t=
x.
>
> - Some API conveniences and simplifications, like including key origin
>   information directly alongside inputs/outputs [5], the ability to speci=
fy
>   key origins in place of addresses (eg. GET /hd/15cb9edc/8/utxos), a com=
pact
>   history format [6], and an easy way to catch-up with missed events [7].
>   Unless explicitly asked for, the API omits information about non-wallet
>   inputs/outputs and protocol-level details like scriptsig and witnesses,
>   which are typically not needed for higher-level app development.
>
> The indexer is designed in a way that minimizes RPC requests to bitcoind.=
 By
> using labels to store key origin information, it is able to index incomin=
g
> transactions using the information available from `listtransactions` alon=
e
> (plus 3 extra rpc requests that don't grow with the number of transaction=
s),
> but requires 1 additional rpc call per outgoing transaction (to learn whi=
ch
> prevouts were spent). It can index 10k incoming txs in under a second, or=
 a
> mixture of 5k/5k in under 5 seconds. The index is currently entirely in-
> memory and does not get persisted. The indexer logic can be seen in [8].
>
> One major item on the roadmap that I'm hoping to tackle soon is support f=
or
> output script descriptors.
>
> If anyone is interested in contributing, the README has some useful devel=
oper
> resources [9] and a handy script for setting up a development environment=
.
>
> This is an early alpha release, recommended for use with testnet/regtest.
>
> All feedback welcome!
>
> Cheers,
> Nadav
>
> [0] https://github.com/shesek/bwt
> [1] https://github.com/shesek/bwt#electrum-plugin
> [2] https://github.com/shesek/bwt#server-sent-events
> [3] https://github.com/shesek/bwt#web-hooks
> [4] https://github.com/shesek/bwt#event-categories
> [5] https://github.com/shesek/bwt#wallet-transaction-format
> [6] https://github.com/shesek/bwt#get-txssinceblock-heightcompact
> [7] https://github.com/shesek/bwt#catching-up-with-missed-events--re-org-=
detection
> [8] https://github.com/shesek/bwt/blob/master/src/indexer.rs (sync_transa=
ctions and load_transactions_since)
> [9] https://github.com/shesek/bwt#developing
>
>