Create example on how to create a custom y-websocket persistence module

[dmonad]

If possible, try to show how to store the document in a MySQL database

[vwall]

Hey @dmonad. I got this working with Prosemirror via TipTap, y-websocket and Postgres. Just looking to find the best way to randomly save the doc. Seems bindState only runs on first connect and writeState runs when everyone connected to the doc disconnects. Maybe just a simple timer? Would bindState be the proper place for this?

[bumi]

@vwall are you willing to share how you managed to do it/your solution?

[StefKors]

I would also love a better example on the y-websocket and persistence. It feels like the client-side is super simple but the server is not. The websocket repo has an example (https://github.com/yjs/y-websocket) but I don't think using npx is really the way to go for a node/express backend for example.

What would best practices be for saving documents, in a stripped down but complete client + server example?

[dmonad]

Hi @StefKors , I realize that the documentation and the example sections are pretty poor. It's getting pretty hard to keep the examples up to date and include helpful resources.

I'm currently working on a new documentation site that explains more internals so the backend examples make more sense. Because when you implement your own database adapter, you need to partially understand the complexity that you are dealing with. I don't want to lose any data because of a poor backend implementation. There have been various discussions about different approaches on how to store the document to a database.

The easiest approach is to store the Yjs document in some database (e.g. MySQL, postgres, but better yet a fast NoSQL database).

You can define your own persistence in y-websocket server:

setPersistence({
  bindState: async (roomName,ydoc) => {
    // this is where you initialize the ydoc
    const initialContent = await getServerContent(roomName) // retrieve the original Yjs doc here
    if (initalContent) {
      Y.applyUpdate(initialContent)
    }
  },
  writeState:(roomName,ydoc) => {
    // This is where you store the Yjs document to your database. This is called when all clients disconnect
    await saveContent(roomName, Y.encodeDocumentAsUpdate(ydoc))
    // optionally store the text data / rich text data to your database
    await saveText(roomName, ydoc.getText('').toString())
  }
})

If you are wondering why to store the Yjs document in the database, then have a look at the various discussions around this topic. I'm not saying that this is the only solution. But in my opinion the cleanest.

[StefKors]

Thanks for the quick response @dmonad. Awesome to hear that it's still developing. If YJS keeps improving I can imagine keeping the demos and examples up to date it's quite the task.

My preference is MongoDB but just like the examples in this repo, they are a lot of different options there. I understand there is some complexity involved, however this writeState snippet looks super simple. thanks @dmonad

From reading through the client-side examples I was expecting a websever/persistence example would be as simple as npm i yjs and this on the node-server (admittedly this probably omits a bunch of important functionality that should be handled and I just don't understand yet):

// pseudo code based on your snippet
import * as Y from 'yjs'
import { WebsocketProvider } from 'yjs/provider/websocket.js'

const wsProvider = new WebsocketProvider('ws://localhost:1234').setPersistence({
  bindState: async (roomName,ydoc) => {
    // this is where you initialize the ydoc
    const initialContent = await getServerContent(roomName) // retrieve the original Yjs doc here
    if (initalContent) {
      Y.applyUpdate(initialContent)
    }
  },
  writeState:(roomName,ydoc) => {
    // This is where you store the Yjs document to your database. This is called when all clients disconnect
    await saveContent(roomName, Y.encodeDocumentAsUpdate(ydoc))
    // optionally store the text data / rich text data to your database
    await saveText(roomName, ydoc.getText('').toString())
  }
})

or something like:

// pseudo code
import * as Y from 'yjs'
import { WebsocketProvider } from 'yjs/provider/websocket.js'

const wsProvider = new WebsocketProvider('ws://localhost:1234')

wsProvider.on('status', event => {
  console.log(event.status) // logs "connected" or "disconnected"
})

// On client new connection
wsProvider.on('connect', (roomName) => {
  // Find document in the DB and return it
  return await Documents.find({ name: roomName })
});

// On websocket event named "writeState"
wsProvider.on('writeState', (roomName, ydoc) => {

  // Upsert the data into the database Documents collection
  await Documents.findOneAndUpdate({name: roomName }, Y.encodeDocumentAsUpdate(ydoc))
});

Is this something that's already possible and does this fit with how yjs should work?

Note: I'm also just learning and reading more and more about yjs so if I'm asking things that are coming in updated docs or examples I'm happy to wait!

[dmonad]

Eventually, I'd like to provide an extensible y-websocket server alternative. The current y-websocket server is by design very minimal. Because it only consists of a few hundred lines of code, it is easy to add new functionality. The protocol (in y-protocols) is stable and will allow you to keep compatibility with future upgrades. Writing a generic, scalable server that works for everyone and is extensible is quite a task. There is an alternative that you could look into: https://www.hocuspocus.dev/

If your company is building a product that would like to have such a server, please PM me. I'm currently assembling funding to make that happen.