bug: fetcher infinite loop

[samlaf]

Right now the fetcher naively retries fetching until it succeeds:

// Use a loop to keep retrying the prefetch as long as the key is not found
while preimage.is_none() && self.last_hint.is_some() {
    let hint = self.last_hint.as_ref().expect("Cannot be None");

    if let Err(e) = self.prefetch(hint).await {
        error!(target: "fetcher", "Failed to prefetch hint: {e}");
        warn!(target: "fetcher", "Retrying hint fetch: {hint}");
        continue;
    }

    let kv_lock = self.kv_store.read().await;
    preimage = kv_lock.get(key);
}

The error strategy (warning and continuing to try) doesn't separate temporary errors from unrecoverable errors, such as 404 type of errors where a hint is invalid:

if hint_data.len() != 32 {
    anyhow::bail!("Invalid hint data length: {}", hint_data.len());
}

So the fetcher will just fall into an infinite for loop.

[clabby]

Thanks. The infinite re-try is actually intentional (when it comes to RPC errors), but it would be a good idea for us to attempt to delineate between truly unrecoverable errors and RPC flakes.

Infinitely retrying is sort-of-better for production deployments (assuming we can't be certain if an error is unrecoverable), where RPC flakes are actually pretty common & runs of the proof have timeouts / alerts on long runs.