Fix error: content digest sha256: ***: not found

[davidferlay]

PR created from patch by @imeoer posted on #2631 (comment)

Prioritize checking that the layer content does not exist to make sure the cache is lazy.

All credits to: Yan Song [email protected]

[tonistiigi]

@imeoer Could you explain more how this patch works and in what cases it affects exporting.

BTW I also recently fixed e5f9959#diff-9dc5570df38eb8f9d15ac01a23974183e935a6825308f61007b98f6f5278102bL224 case that could be related to some these exporting errors. The case it can affect is when there are parallel builds that both do exporting of same manifest blobs.

[imeoer]

Thanks @davidferlay for posting the PR, sorry for the late reply.

@tonistiigi @sipsma We observed that this issue always occurs during the push layers phase. By adding some logs we found that the !cr.getBlobOnly() is always true in here, causing the push image process not to download the lazy layer into the content store first, then throw the error in here.

This issue seems difficult to reproduce (there is about a 1%+ chance of error in our production environment), this patch is a workaround, we still need to take a look deeper and add some test cases to reproduce.

[tonistiigi]

By adding some logs we found that the !cr.getBlobOnly() is always true in here,

What is the source of this layer? Is this a lazy layer reference from another image? In that case getBlobOnly should return true iiuc. Otherwise, if this is not a lazy layer then I'm not sure what other cases are possible.

[imeoer]

Is this a lazy layer reference from another image?

@tonistiigi It was not observed at the time whether it was a lazy layer, the issue did not always occur for the same Dockerfile, I can downgrade the version and look at it again at some time (sorry for being busy recently).

[imeoer]

It should have been fixed in #3566. cc @davidferlay