Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add changes queue support #18

Open
benoitc opened this issue Jun 19, 2013 · 2 comments
Open

add changes queue support #18

benoitc opened this issue Jun 19, 2013 · 2 comments
Assignees
Labels
Milestone

Comments

@benoitc
Copy link
Member

benoitc commented Jun 19, 2013

Changes queues are the core of blobs synchronisations and can be used by indexers or other agents, They can are created on the source and are semi persistent.

Key mechanism is the following:

  • anonymous changes queue are created by a POST to /<storage>/_queue. An id is returned allowing the listener to watch changes on it.
  • named queue are created by a PUT to /storage/_queue/<name>. If the queue exists, a conflicts is returned (409) .
  • Multiple listeners can watch a queue, the queue items are balanced between them. An option can be passed to only accept one listener for a queue.
  • Queue can be watched using websockets or webrtc (data channels). A detection mechanism is provided to detect on the source the transport used.
  • A watcher can continuously watch queue
  • Items are received in bulk operation:
    1. During watching operations a controller ask for N blobs
    2. if no blobs are available a NOP is sent to the listener
    3. Once the N blobs are received by the listener an ACK is sent to the queue
    4. If an ACK is received on the queue, blobs received are deleted from the queue
    5. If a NOP operation is received on the queue, nothing is done. NOP operations are used to maintain the connection alive
  • Queues are persistent.
  • Queues can be automatically deleted if they has been active since a while. Optionnal.
@benoitc
Copy link
Member Author

benoitc commented Jun 19, 2013

Queue backend can be pluggable as well imo. With a default to ETS saved from time to time to the disk. Thoughts?

@nrdufour
Copy link
Member

It would be interesting to have a backend for queues or lists. And yes starting with an ETS or DETS ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants