Add partitioned tables support

[magnetised]

Allows for subscribing to partitions of a partitioned table (which you could already do) but also subscribing to the root, partitioned, table and receiving all updates across all partitions.

Fixes #2118

Major changes:

• Add support for partitioned table types in the various inspector queries
• Include partition hierarchy information in relation information from pg and in inspector relation info
• Force reload of table information when receiving a new relation that is a partition
• Add partition information to Filter so that it can correctly pass on a change on a partition table to a shape on the partition root.
• For a shape subscribed to the partition root, changes coming in on partitions will be re-written so that their relation matches the partition root. The change keys are not being re-written - I don't think this is an issue as the keys are guaranteed to be unique and that's enough.
• Relation messages notifying of a new partition table are forwarded onto root-partition consumers so the shape can be updated with knowledge of the new partition (this does not cause the shape to be deleted)
• I've added a full-stack test to verify this stuff so that we're not making assumptions about pg's handling of partitions

Minor:

• support querying the inspector based on the {schema, name} relation tuple, as well as the table name string because I needed to be able to look up table info based
• Filter instances now have an inspector instance so they can do table info lookups to get partition information
• The filter functions used by the dispatcher now return {filter, shape_ids} so that the filter's state can be updated
• I've tweaked StubInspector so we can return richer info from the load_relation call
• Added missing serialisation of a shape's replica setting

[netlify]

✅ Deploy Preview for electric-next ready!

Name	Link
🔨 Latest commit	e7c5ccb
🔍 Latest deploy log	https://app.netlify.com/sites/electric-next/deploys/6788e7214568d100085fb5ba
😎 Deploy Preview	https://deploy-preview-2197--electric-next.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[magnetised]

Think i have a more up-to-date version of this on my laptop which I seem to have forgotten to push or have over-written with my old work-machine version. @robacourt please hold off review until I can recover the situation

[magnetised]

@robacourt situation recovered. review away. great start to the year!

[balegas]

Nice work! We should document the two possible ways to subscribe to partitioned tables.

[magnetised]

Nice work! We should document the two possible ways to subscribe to partitioned tables.

@balegas I've added basic docs highlighting the behaviour of shapes on partitioned tables - pls review

[robacourt]

Nice work! I've left a few comments.

I've also been thinking there might be an alternative implementation, whereby the replication steam is processed (by a PartitionTransformer for want of a better name) before it gets to the filter, and if it sees a change for a partition it inserts a change into the stream for the logical table. That way, all the partition logic can go in the PartitionTransformer. Shape, Filter, Consumer etc. would remain unchanged. Sure it will create some changes that might not be used, but that seems a good tradeoff for the simplicity. Is there a reason why this is not possible? Or undesirable?

[robacourt]

Surely the ShapeLogCollector should crash if there's an error? An error could be something as simple as loss of network connectivity to the database, which if we just swallow the error we won't clean the relation leading to stale cache and an inconsistent state, or have I missed something?

[robacourt]

When would this happen?

[magnetised]

some of our mocks don't return the full table info, including a parent value. rather than update all our existing tests and add the burden of adding all these extra keys to all mock results in perpetuity i just added a clause that handles an implicit parent: nil clause in the inspector result.

[balegas]

👌

[magnetised]

I've also been thinking there might be an alternative implementation, whereby the replication steam is processed (by a PartitionTransformer for want of a better name) before it gets to the filter, and if it sees a change for a partition it inserts a change into the stream for the logical table. That way, all the partition logic can go in the PartitionTransformer. Shape, Filter, Consumer etc. would remain unchanged. Sure it will create some changes that might not be used, but that seems a good tradeoff for the simplicity. Is there a reason why this is not possible? Or undesirable?

@robacourt that's a good idea and does remove undesirable complexity from the Filter. it does add a coordination point where we have to add information about partitions to the partition "transformer" on shape creation. currently this is nicely encapsulated by the Shapes.Dispatcher subscribe/2 callback which updates the filter with partition information.

so though I like the encapsulation you suggest, I think it ends up as more complex as shape information has to be threaded into more places and fundamentally we end up with more moving parts.

I will look at the integration with the filter to see if I can encapsulate the partition handling a bit more.

update: thinking a bit more, I could maybe model the partition handling in the filter itself using this transformer approach.

[robacourt]

update: thinking a bit more, I could maybe model the partition handling in the filter itself using this transformer approach.

I can see the appeal of putting the logic in the filter as it needs a similar interface to the Filter. But it would be nice to keep the Filter ignorant of partitions. One possibility would be for the Dispatcher to call both, as in:

filter = Filter.add_shape(state.filter, from, shape)
partition_transformer = PartitionTransformer.register_table(state.transformer, shape.root_table)

then later

{partition_transformer, events} = PartitionTransformer.process_events([event])
...
Filter.affected_shapes(state.filter, event)

Ideally then the only thing that knows about partitions would be the PartitionTransformer.

[magnetised]

Ideally then the only thing that knows about partitions would be the PartitionTransformer.

@robacourt you're absolutely right! have pushed a version that works exactly as you propose, except I dropped the Transformer suffix.

this version is much simpler as we don't have manage relations as well as changes.

pls have a look. pr is not ready to merge as i have some serious issues with truncation to work out

[robacourt]

this version is much simpler as we don't have manage relations as well as changes.

pls have a look. pr is not ready to merge as i have some serious issues with truncation to work out

That's looks great! Nice one! There's various things I could point out, like reverting the Filter tests, but as you're not ready to merge I assume things like that are on your list so I won't point them out right now. In terms of the new structure it looks great 🚀