Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

REST listener improvements for liteserv #2117

Draft
wants to merge 50 commits into
base: master
Choose a base branch
from
Draft

Conversation

snej
Copy link
Collaborator

@snej snej commented Aug 23, 2024

This branch contains all the LiteCore enhancements & fixes needed to implement the liteserv server.

  • Implemented the _changes endpoint of the REST API, all 3 modes (normal, longpoll, continuous.) The latter two modes often have to block waiting for new changes; the implementation using Listener::Task ensures these idle connections aren't blocking a thread or locking a database, so it should be very scalable.
  • Implemented a new _query endpoint for running queries.
  • Lots of changes to the _replicate endpoint:
    • collection support
    • channel & docID filter support
    • doesn't block until the replication completes (that way the client gets the session_id and can cancel it)
    • simplified the cancel API to take the session_id
    • added a C4Listener method to start a replication task
  • I made a bunch of improvements to the RequestResponse class, like chunked-encoding support, and the ability to flush output to the socket before finishing the request (so long responses don't allocate a lot of memory.)

@cbl-bot
Copy link

cbl-bot commented Aug 23, 2024

Code Coverage Results:

Type Percentage
branches 66.42
functions 79.0
instantiations 33.64
lines 77.93
regions 73.76

@snej snej force-pushed the feature/changes-feed branch 5 times, most recently from a93a5cd to fce880d Compare August 28, 2024 00:27
@snej snej force-pushed the feature/changes-feed branch 2 times, most recently from 9f1c40f to 3e0870a Compare August 29, 2024 20:57
@snej snej changed the title Implemented changes feed for REST listener REST listener improvements for liteserv Sep 5, 2024
- Fixed a bug where the existing `split` method would report an
  extra empty string, if the input string ends with the delimiter.
- Added a variant of `split` that returns a vector instead of using
  a callback.
If a response had neither a Content-Length, Transfer-Encoding nor
Connection header, the error would be "Unsupported 'Connection'
response header", which is misleading.
Use Encoder::writeFormatted to simplify creating JSON.
Also used the not-at-all-new `enc["key"] = value` form.
Server was locking its mutex while calling a handler. This made it
impossible to handle more than one HTTP request at once. Especially
bad when handling `/db/_replicate`, whose handler blocks until the
replication completes!
Body::bodyAsJSON() would silently return nullptr if the response is
supposed to be JSON but fails to parse. This made some test failures
hard to track down today! It now logs a warning.
- Added "Server:" response header
- Added "ETag:" response header to document response
- Honor "If-None-Match" request header getting document
- Added "total_rows" and "update_seq" to _all_docs response
- Suppress response body on HEAD request
- _changes feed improvements:
  - Flush output before waiting
  - No heartbeat if ?heartbeat=0
  - Longest possible timeout if ?timeout=0
  - Stop on socket error
  - Make logging verbose not info
Instead of writing them immediately, save them in a Headers object.
That gives client code more flexibility about when it can set
headers -- it doesn't have to wait until after the status line is
sent.
Discovered this scenario while testing replications in liteserv:
1. Create an empty database
2. Close it
3. Reopen it without giving the kC4DB_Create flag
4. Access the BlobStore, i.e. call c4Database::getBlobStore()

Result: getBlobStore fails with a NotFound error.
The reason is that step 1 didn't instantiate the BlobStore or create
the Attachments directory; and then in step 4 the db's flags don't
include kC4DB_Create so the BlobStore instance won't create the
directory.

Fixed by passing the kC4DB_Create flag to the C4BlobStore constructor
whenever the database is writeable.
- Moved all the request parsing out of the Task class.
- One-shot replication does not block response until finished.
- Returns 202 Accepted status, not 200.
- To cancel a replication you now set the `cancel` property to the
  `session_id` of the task.
- Preliminary support for channel/docID filters, but they're not
  per-collection yet.
Client specifies the version in a header, e.g:
  API-Version: 1
  API-Version: 3.5
If missing, it defaults to 1.0.
Handlers support a specific major version, and an incompatible
version in a request triggers a 400 response.
CBL will probably never use this, but liteserv needs it.
They're slow, and we only need very simple pattern matching.
Rather, get it when accepting the socket connection and reuse it.
Yes, this was actually a significant time-sink, roughly 6% of the
time to handle a simple request.
There were issues making it work with the DatabasePool.
Added `serverName`, `serverVersion` to C4ListenerConfig.
By default these should be nullslice.
The default error messages in the Network domain call the peer
"server" and the local process "client", which is backwards.
This results in misleading error logs when a TLS handshake fails,
like "server rejected the client TLS certificate" when actually it's
the other way around!
Fixed this by simply swapping the strings "client" and "server" when
logging errors in the Network domain when establishing a connection.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants