Context for deleted records on Firehose (com.atproto.sync.subscribeRepos)

[nicktelford]

I'm not certain this is the right repo to raise this request; I'm raising it here since indigo implements the Firehose, and this request should not require any changes the the API.

Note: all of my examples will be using the Jetstream JSON schema, but this applies to the CBOR com.atproto.sync.subscribeRepos too.

The Problem

When an atproto record is deleted, the firehose event for the deletion references only the rkey of the event being deleted. This requires that, in order to handle the deletion, the user must store every record (of that type) for the users being monitored, even if their use-case otherwise does not require it.

Example

A simple "follower count" widget, that shows the current follower count for a user.

When an app.bsky.graph.follow record is created, it includes the did of the user being followed, via commit.record.subject, enabling the widget to simply increment its count when the DID matches the user being monitored.

However, when a follow record is deleted, it only includes the rkey. In order to decrement your counter, you would need to store every matching follow record, and do look-ups against the rkey.

This is probably fine for a single user, but it doesn't scale as you want to monitor the follower counts of more and more users. You ultimately need to store the entire follower graph, just to keep an accurate by-user count.

Worse: if you do try to store the entire follower graph, that's not possible via just the Firehose: you'd need to "backfill" all previous follow records from the HTTP API, which due to rate-limits would likely take days already, and will only get worse.

This is the case irrespective of the type of record being deleted, which is why it's an atproto issue, not a bsky schema issue.

The Solution

atproto already allows commit objects to contain an optional Record field, which is populated for create and update operations.

My request is, when a record is being deleted, the existing record field should contain the full record being deleted.

This would solve the problem, without any schema change.

[yamarten]

This may not be good for users. This means that even after deleting records (especially posts containing confidential information), it will be visible during the window period of the relay or PDS firehose.

Bluesky has made changes to immediately reflect the deletion, and the policy is likely to conflict with this request.

bluesky-social/atproto#1410

Please note that even if an old record content is added to delete (and update) commit, it is not possible to count strictly without a graph. Account deletion, firehose failure, minor hacks (such as creating multiple follow records for the same DID), etc.