Partial autoload for postgres-persister

[jakubriedl]

Summary

Implements partial update in autoload from postgres to save significant resources and prevent full load on every change. Related issue here #236

How did you test this change?

All postgres tests are passing and I've tested it manually as well. But I'm open to suggestions what are best practices to test with the partial loads

[jamesgpearce]

My main questions are:

1. I wonder if we can wire up something similar for sqlite (even though the implementation will be quite different). I can look into this.

2a) We are sending both rows over the channel and out to the client if it's an update. Is it possible (or more efficient) to calculate the changes in the SQL function and just send the diff?

2b) We are sending the old row if it's a delete. We don't need it!

...which make me wonder if we should have three separate SQL functions (we already have three triggers) to create different payloads for each operation - and then the JS function can just apply whatever it's been sent.

2a may be tricky for SQL, I admit. But if possible, let's try.

[jakubriedl]

1. I'll look into that
   2a) It seems to be possible
   2b) we still need a row.id to know what to delete, we could theoretically send just that though

So theoretically we could make the payload look like { rowId: "xy", operation: "insert/update/delete", changes: "{}:json" }

For both of the 2) items it feels like bit of tradeoff between simplicity and ease to understand vs squeezing tiny fractions of performance. In which case I'd probably pick simplicity, but I do understand if you want to go the other direction.

[jamesgpearce]

Yes it might be overkill. But theoretically anything going over the channel will be over a network connection, whereas between trigger and function will be in-memory. If you had a row that looked like |id|booleanFlag|largeSlabOfText| and only the booleanFlag changed, it would be a shame to send (two!) largeSlabOfText fields in JSON for no reason.

But don't sweat it too much. We can always optimize later if it gets too tricky.

[jamesgpearce]

Crude, but something like this in the pgsql should allow you to detect new and changed cells:

SELECT * FROM
  (SELECT key, value FROM json_each(row_to_json(NEW))) a
    LEFT OUTER JOIN
  (SELECT key, value FROM json_each(row_to_json(OLD))) b
    ON a.key = b.key

(maybe a full outer join to get deletions too!)

[jakubriedl]

yeah I've saw similar in the SO and keen to try that

[jamesgpearce]

(Do we want to keep these now you have your dedicated connector?)

[jakubriedl]

let's close this, to make the postgres partial autoload work well it needed much more work than just this and is solved by the custom persister.