share-append-only-databases-with-hyperbee.md

How to share Append-Only Databases with Hyperbee

Hyperbee is an append-only B-tree based on Hypercore. It provides a key/value-store API with methods to insert and get key/value pairs, perform atomic batch insertions, and create sorted iterators.

This How-to consists of three applications: bee-writer-app , bee-reader-app and core-reader-app.

The bee-writer-app stores 100k entries from a given dictionary file into a Hyperbee instance. The Corestore instance used to create the Hyperbee instance is replicated using Hyperswarm. This enables other peers to replicate their Corestore instance and sparsely (on-demand) download the dictionary data into their local Hyperbee instances.

Start the bee-writer-app project with the following commands:

mkdir bee-writer-app
cd bee-writer-app
pear init -y -t terminal
npm install corestore hyperswarm hyperbee b4a bare-fs

Click here to save dict.json.

Save it into bee-writer-app directory. The dict.json file contains 100K dictionary words.

Alter the generated bee-writer-app/index.js file to the following

import fsp from 'bare-fs/promises'
import Hyperswarm from 'hyperswarm'
import Corestore from 'corestore'
import Hyperbee from 'hyperbee'
import b4a from 'b4a'
// create a corestore instance with the given location
const store = new Corestore(Pear.config.storage)

const swarm = new Hyperswarm()
Pear.teardown(() => swarm.destroy())

// replication of corestore instance
swarm.on('connection', conn => store.replicate(conn))

// creation of Hypercore instance (if not already created)
const core = store.get({ name: 'my-bee-core' })

// creation of Hyperbee instance using the core instance 
const bee = new Hyperbee(core, {
  keyEncoding: 'utf-8',
  valueEncoding: 'utf-8'
})

// wait till all the properties of the hypercore are initialized
await core.ready()

// join a topic
const discovery = swarm.join(core.discoveryKey)

// Only display the key once the Hyperbee has been announced to the DHT
discovery.flushed().then(() => {
  console.log('bee key:', b4a.toString(core.key, 'hex'))
})

// Only import the dictionary the first time this script is executed
// The first block will always be the Hyperbee header block
if (core.length <= 1) {
  console.log('importing dictionary...')
  const dict = JSON.parse(await fsp.readFile('./dict.json'))
  const batch = bee.batch()
  for (const { key, value } of dict) {
    await batch.put(key, value)
  }
  await batch.flush()
} else {
  // Otherwise just seed the previously-imported dictionary
  console.log('seeding dictionary...')
}

Open the app with pear run --dev .:

cd bee-writer-app
pear run --dev .

Start the bee-reader-app project with the following commands:

mkdir bee-reader-app
cd bee-reader-app
pear init -y -t terminal
npm install corestore hyperswarm hyperbee b4a bare-pipe

The bee-reader-app creates a Corestore instance and replicates it using the Hyperswarm instance to the same topic as bee-writer-app. On every word entered in the command line, it will download the respective data to the local Hyperbee instance.

Alter the generated bee-reader-app/index.js file to the following

import Hyperswarm from 'hyperswarm'
import Corestore from 'corestore'
import Hyperbee from 'hyperbee'
import Pipe from 'bare-pipe'
import b4a from 'b4a'

const key = Pear.config.args[0]

if (!key) throw new Error('provide a key')

// creation of a corestore instance 
const store = new Corestore(Pear.config.storage)

const swarm = new Hyperswarm()
Pear.teardown(() => swarm.destroy())

// replication of the corestore instance on connection with other peers
swarm.on('connection', (conn) => store.replicate(conn))

// create or get the hypercore using the public key supplied as command-line argument
const core = store.get({ key: b4a.from(key, 'hex') })

// create a hyperbee instance using the hypercore instance
const bee = new Hyperbee(core, {
  keyEncoding: 'utf-8',
  valueEncoding: 'utf-8'
})

// wait till the hypercore properties to be initialized
await core.ready()

// logging the public key of the hypercore instance
console.log('core key here is:', core.key.toString('hex'))

// Attempt to connect to peers
swarm.join(core.discoveryKey)

const stdin = new Pipe(0)

stdin.on('data', (data) => {
  const word = data.toString().trim()
  if (!word.length) return
  bee.get(word).then(node => {
    if (!node || !node.value) console.log(`No dictionary entry for ${word}`)
    else console.log(`${word} -> ${node.value}`)
    setImmediate(console.log) // flush hack
  }, console.error)
})

Open the bee-reader-app and pass it the core key:

cd bee-reader-app
pear run --dev . <SUPPLY KEY HERE>

Query the database by entering a key to lookup into the bee-reader-app terminal and hitting return.

Each application has dedicated storage at Pear.config.storage. Try logging out Pear.config.storage for the bee-reader-app and then look at the disk space for that storage path after each query. Notice that it's significantly smaller than bee-writer-app! This is because Hyperbee only downloads the Hypercore blocks it needs to satisfy each query, a feature we call sparse downloading.

Importantly, a Hyperbee is just a Hypercore, where the tree nodes are stored as Hypercore blocks.

Finally create a core-reader-app project:

mkdir core-reader-app
cd core-reader-app
pear init -y -t terminal
npm install corestore hyperswarm hyperbee b4a

Alter the generated core-reader-app/index.js file to the following

import Hyperswarm from 'hyperswarm'
import Corestore from 'corestore'
import b4a from 'b4a'

import { Node } from 'hyperbee/lib/messages.js'

const key = Pear.config.args[0]
if (!key) throw new Error('provide a key')

// creation of a corestore instance 
const store = new Corestore('./reader-storage')

const swarm = new Hyperswarm()
Pear.teardown(() => swarm.destroy())

// replication of the corestore instance on connection with other peers
swarm.on('connection', conn => store.replicate(conn))

// create or get the hypercore using the public key supplied as command-line argument
const core = store.get({ key: b4a.from(key, 'hex') })
// wait till the properties of the hypercore instance are initialized
await core.ready()

const foundPeers = store.findingPeers()
// join a topic
swarm.join(core.discoveryKey)
swarm.flush().then(() => foundPeers())

// update the meta-data information of the hypercore instance
await core.update()

const seq = core.length - 1
const lastBlock = await core.get(core.length - 1)

// print the information about the last block or the latest block of the hypercore instance
console.log(`Raw Block ${seq}:`, lastBlock)
console.log(`Decoded Block ${seq}`, Node.decode(lastBlock))

Open the core-reader-app with pear run --dev ., passing the core key to it:

cd core-reader-app
pear run --dev . <SUPPLY KEY HERE>

Now we can examine the Hyperbee as if it were just a Hypercore.

The core-reader-app will continually download and log the last block of the Hypercore containing the Hyperbee data. Note that these blocks are encoded using Hyperbee's Node encoding, which has been imported directly from Hyperbee here for the purposes of explanation.