Updates on my experiments with Fireproof

[mschoch]

Hey everyone, I wanted to share an update since the fireside chat last week. If you recall, my goal was to get multi-device sync working at level where I could really understand the important moving parts. An ideal end state would be to have my own "pluggable connector" for both storage and metadata, but I was willing to take some shortcuts along the way. I have been mostly successful, with a few minor todo items left, and here are my notes.

• Some of my research indicated that there was an existing REST connector. In the source, I found this, however it appears non-functional:

  fireproof/packages/encrypted-blockstore/src/connect-rest.ts

  Lines 37 to 79 in 6639199

  	async metaUpload(bytes: Uint8Array, params: UploadMetaFnParams) {
  	// const event = await this.createEventBlock(bytes)
  	// const base64String = Base64.fromUint8Array(bytes)
  	// const crdtEntry = {
  	// cid: event.cid.toString(),
  	// data: base64String,
  	// parents: this.parents.map(p => p.toString())
  	// }
  	// const fetchUploadUrl = new URL(
  	// `?${new URLSearchParams({ type: 'meta', ...params }).toString()}`,
  	// this.uploadUrl
  	// )
  	// const done = await fetch(fetchUploadUrl, {
  	// method: 'PUT',
  	// body: JSON.stringify(crdtEntry)
  	// })
  	// const result = await done.json()
  	// if (result.status != 201) {
  	// throw new Error('failed to upload data ' + JSON.parse(result.body).message)
  	// }
  	// this.parents = [event.cid]
  	return null
  	}
  	async metaDownload(params: DownloadMetaFnParams) {
  	// const { name, branch } = params
  	// const fetchUploadUrl = new URL(`?${new URLSearchParams({ type: "meta", ...params }).toString()}`,this.uploadUrl)
  	// const data = await fetch(fetchUploadUrl)
  	// let response = await data.json()
  	// if (response.status != 200) throw new Error("Failed to download data")
  	// response = JSON.parse(response.body).items
  	// const events = await Promise.all(
  	// response.map(async (element: any) => {
  	// const base64String = element.data
  	// const bytes = Base64.toUint8Array(base64String)
  	// return { cid: element.cid, bytes }
  	// })
  	// )
  	// const cids = events.map((e) => e.cid)
  	// const uniqueParentsMap = new Map([...this.parents, ...cids].map((p) => [p.toString(), p]))
  	// this.parents = Array.from(uniqueParentsMap.values())
  	// return events.map((e) => e.bytes)
  	return []

• Rather than try to write my own, I pivoted to trying to reuse one of the existing connectors. Both the AWS/s3 and netlify connectors take configurable URLs, so they were good candidates.

• It's time to get concrete, we need an application to work with. I chose fireproof-storage/react-typescript-starter-kit. I plan to build this and serve the static content out of a minimal Go application, which will also include custom REST endpoints for the storage and metadata.

• On the react app side of things, my modifications are confined to src/components/Topics.tsx. The plan is to import some additional things, and after the component starts using the database, wire up the sync.

• I eventually found my way to importing @fireproof/aws directly and then using connect.aws(db, conf), but I went through many wrong paths first

• One problem was that more than one example I looked at used an older version of fireproof, and that was still using @fireproof/connect. As I understand it now, that package is deprecated, and with 0.18 and later it is no longer needed/used. I think this issue actually touches on the same thing without quite closing the loop: (@fireproof/encrypted-blockstore): npm release build refresh? #110 At least jchris's comment in this thread made me aware there was some reorg going on, and older examples may not be helpful.

• At this point I tried to use npm to update my app to use v0.18 of all the @fireproof packages

• At some point I learned that fireproof wasn't going to work properly without TLS, more on this another day

• On the Go server side of things, I've emulated the request/response behaviors expected enough to keep the connector happy.

• I plan to release this code, but I think it could be useful beyond this experiment, so I'm cleaning it up further.

Some questions and thoughts:

• It would be great to bring back and/or introduce a pure REST connector. The s3 and netlify ones each have some platform specific considerations which make them not a good example to understand what is really required. In particular with the aws/s3 connector, signficant logic also exists in the lambda function, making the connector source code a bit deceiving.

• What is branch used for? In the process of reverse engineering the interactions, I could see a value for "branch" coming through. However while the s3 adapter makes reference to it, it doesn't seem to really use it. And the netlify adapter doesn't even mention it.

  fireproof/packages/connect-aws/src/connect-s3.ts

  Line 148 in 6639199

  const { name, branch } = params

• Do I understand correctly that the meta-data endpoint can delete a record once it sees a new record pointing to it as the parent? I'd really like to better understand the technical requirements of an implementation of the metadata connector. Again I feel that the netlify and s3 implementations are interesting, and well adapted to their platforms, yet not necessarily good general purpose examples.

• One missing piece is that I don't yet have a websocket hooked up to notify you of changes from other clients, but the sync does work when refreshed.

In conclusion, I'm pretty excited to have gotten all this working over the weekend.

[vmx]

Thanks for the post @mschoch! I currently go down a similar route. My plan is to create a simple RESTful connector (I already have a simple server in Rust that can take GET and PUT requests and stores things as files on disk. My plan was to get this working in order to get a better understanding on how things work.

I started a new connector as a copy of the netlify one. One problem I was hitting that somehow vite wasn't happy (which I used in order to play with Fireproof. My own example app is based on Sveltekit). Just yesterday I found out that tsup seems to build things fine for my use case (which is also used to build @fireproof/core). So I'll try to get my own connector working with that.

[jchris]

Thanks for digging. B/c the database has simple forking and merging, branch (default main) is reserved in the namespace to allow tagging multiple forks during operation. I think this is too complicated for normal use cases, so maybe should be removed.

[jchris]

This rest server works https://github.com/fireproof-storage/fireproof/blob/main/packages/fireproof/scripts/server.js

with this application config:

fireproof/packages/connect-partykit/src/index.ts

Line 50 in 20a7753

const restConn = new ConnectREST('http://localhost:8000/')

[mschoch]

When you say it works with that application config, I think that is only partially true. The rest connection is only used for data storage. The meta-data portion of that connector is commented out (see:

fireproof/packages/encrypted-blockstore/src/connect-rest.ts

Lines 37 to 81 in 20a7753

	async metaUpload(bytes: Uint8Array, params: UploadMetaFnParams) {
	// const event = await this.createEventBlock(bytes)
	// const base64String = Base64.fromUint8Array(bytes)
	// const crdtEntry = {
	// cid: event.cid.toString(),
	// data: base64String,
	// parents: this.parents.map(p => p.toString())
	// }
	// const fetchUploadUrl = new URL(
	// `?${new URLSearchParams({ type: 'meta', ...params }).toString()}`,
	// this.uploadUrl
	// )
	// const done = await fetch(fetchUploadUrl, {
	// method: 'PUT',
	// body: JSON.stringify(crdtEntry)
	// })
	// const result = await done.json()
	// if (result.status != 201) {
	// throw new Error('failed to upload data ' + JSON.parse(result.body).message)
	// }
	// this.parents = [event.cid]
	return null
	}
	async metaDownload(params: DownloadMetaFnParams) {
	// const { name, branch } = params
	// const fetchUploadUrl = new URL(`?${new URLSearchParams({ type: "meta", ...params }).toString()}`,this.uploadUrl)
	// const data = await fetch(fetchUploadUrl)
	// let response = await data.json()
	// if (response.status != 200) throw new Error("Failed to download data")
	// response = JSON.parse(response.body).items
	// const events = await Promise.all(
	// response.map(async (element: any) => {
	// const base64String = element.data
	// const bytes = Base64.toUint8Array(base64String)
	// return { cid: element.cid, bytes }
	// })
	// )
	// const cids = events.map((e) => e.cid)
	// const uniqueParentsMap = new Map([...this.parents, ...cids].map((p) => [p.toString(), p]))
	// this.parents = Array.from(uniqueParentsMap.values())
	// return events.map((e) => e.bytes)
	return []
	}
	}

). And that party-kit example you linked to uses the party-kit mechanism for sharing the meta-data.

[jchris]

yes! we need that local metadata realtime action. currently miniflare / partykit is the easiest way to get it. can't wait to see a clean one