Fallback when checkpoints indicated by _last_checkpoint hint is missing

[Sevenannn]

Please describe why this is necessary.

When the checkpoints indicated by _last_checkpoint file is missing, the snapshot creation will simply fail. A fallback machenism to construct snapshot from a previous checkpoint versions, or a simpler fallback of completely construct snapshot from log files would be useful in this case, along with warning messages informing users that olders checkpoint file is used.

Describe the functionality you are proposing.

When the checkpoint indicated by _last_checkpoint file is missing, construct snapshot from the last valid checkpoint + subsequent logs file, or a more naive implementation of constructing snapshot purely from logs file.

Additional context

N/A

[zachschuermann]

Hi @Sevenannn thanks for raising! This has been a TODO for a while I will try to get to it soon :)

Looks like delta-spark just lists from 0 in case there isn't a last checkpoint hint? Seems reasonable to do in the near-term

[scovich]

There are two different issues here:

1. _last_checkpoint file is missing or stale (points to an older checkpoint). This can happen for various reasons and clients have to be ready to deal with it. We usually list from 0 because there's no way to guess where the listing should actually start.
2. _last_checkpoint file was wrong (points to a checkpoint that doesn't exist, and no newer checkpoint exists). This cannot happen under normal circumstances because we always write the checkpoint before updating the _last_checkpoint file, and metadata cleanup should never delete the newest checkpoint. Because this situation should not occur, it's less clear that we should try to handle it gracefully. The main reason I've seen for it to arise is when people physically delete files from the _delta_log directory. Either to "drop" and "recreate" the table (while workload is running), or to "recover" to an earlier state by deleting newer commits (again, while workload is running). Neither of those is a supported use case in Delta, tho delta-spark seems to go out of its way to tolerate them.

[Sevenannn]

Thanks for the replies! @scovich Yeah I can see the error "Had a _last_checkpoint hint but didn't find any checkpoints" under the abnormal circumstances. (e.g. checkpoint file get manually deleted). Do you suggest that this would be better to left as an error for users' to handle, instead of having a fall back within the delta-kernel rs?

[scovich]

Yeah, if some external entity is messing up the table's files while kernel and/or engine are trying to access them, a near-infinite set of things could go wrong. Delta spark is full of special-case code that attempts to compensate, but there's always one more case it missed, or ambiguous cases there's no good answer for. I'd much rather not get on a treadmill that encourages people to do bad/illegal things to their tables. Fail clean, and discourage users from breaking their tables.

[Sevenannn]

Thanks! That make sense. However I found it a bit hard to handle the checkpoint related error since they are all currently wrapped in Error::Generic. I created a PR to add a Error variation: InvalidCheckpoint, so that it will be easier to handle checkpoint specific error when using delta kernel crate. Would you help take a look? #593