BUG - Builds images with sparse files which cannot be copied

[bgardner-noggin]

Problem

This runtime generates SOCI indexes that suffer from a bug whereby a copy of a file will not complete. The files generated are sparse.

Current Behavior

The image when loaded has many sparse files that a simple cp command will stall on.

Steps to Reproduce (if applicable)

With the following simple Docker file

`FROM oraclelinux:9

RUN dnf install -y libreoffice
&& echo "Hello world" > /tmp/test.txt
&& groupadd -g 679 convert
&& useradd -d /opt/noggin.io/soffice -u 679 convert -g 679

USER convert

RUN soffice --convert-to pdf --outdir /tmp /tmp/test.txt

CMD cp /opt/noggin.io/soffice/.config/libreoffice/4/user/basic/Standard/Module1.xba /tmp/Module1.xba.copied
`

Build an image, use this generator on it and create an ECS task.

This task will continue to run as the cp command does not complete.

Without the SOCI index, the task will quit immediately.

If the soci binary is used instead to build the SOCI index, the bug will not occur

Additional Context

Running the following in an ECS task for an image with the SOCI index

find / -type f -printf "%S\t%p\n" | gawk '$1 < 1.0 {print}

Any file that reports as less than 50% non sparse the cp command will stall on

[turan18]

Hey, thanks for opening an issue! I was able to reproduce the issue with index builder and the SOCI CLI. When you created the index with the CLI did you explicitly set a --min-layer-size? If not, I presume the layer with the sparse file (the last layer) was not lazy loaded since it was under the 10 MiB default min layer size threshold set by the CLI, and therefore the cp did not hang. The index builder sets the min-layer-size to 0 by default, which means that each layer in the image was lazy loaded and therefore susceptible to any SOCI FS bugs. See: #40

We are looking into this issue with sparse files on the SOCI side. We will keep you posted.

[turan18]

Hey, just wanted to provide a quick update. We have root caused the issue to be in one of our dependencies, go-fuse. We have a PR out against them upstream. In the meantime we are actively working on resolving #40, so you can build a working SOCI index for your image with the CFN solution. @bgardner-noggin

[turan18]

#40 (--min-layer-size issue) has now been resolved/merged. You should now be able to re-deploy your CFN stack and have a working SOCi index built for your image. Please let us know if you run into any issues and thanks for your patience. @bgardner-noggin

[turan18]

Hey, sorry for the wait, we had some issues with updating our release process but they are resolved now. Release v0.4.1 is now out, so if you are consuming SOCI via ECS on EC2 you can now pull this release in. If you are using Fargate this fix should already be available to you. Thank you for your patience!

[bgardner-noggin]

0.4.1 has been released, but this project has not been updated to use it.

I pulled this code, changed go.mod to use 0.4.1 and built the zip using the Dockerfile

However, when I execute the lambda against my ECR repo, it only generates a "Soci index" image in the ECR repo. Beforehand it would generate two items per image

"Soci index"
"Image Index"

I cannot see anything in the log to indicate why it isn't generating the "Image index" though

[turan18]

0.4.1 has been released, but this project has not been updated to use it.

The underlying issue was in the snapshotter, not in any of the build tools like the CLI or index builder, so you should be able to use the same index. If you are using ECS Fargate, you should just be able to deploy your tasks. If you are on ECS on EC2, you can pull the latest version of the snasphotter in. Let us know if you run into any other issues.

I pulled this code, changed go.mod to use 0.4.1 and built the zip using the Dockerfile
However, when I execute the lambda against my ECR repo, it only generates a "Soci index" image in the ECR repo. Beforehand it would generate two items per image

We have a PR out upgrading the index builder to pull the latest version of SOCI (v0.4.1). It seems like we were able to generate the image index, as-well. Once that is in you can re-deploy the solution and retry.