error syncing 'minio/...': Tar file extraction failed for file index: 2, with: EOF when updating minio #2229
Comments
|
Currently extracting the files from operator container, I calculated md5sums: $ md5sum /tmp/webhook/v1/update/*
59f91db02665c67c65f126437deab88c /tmp/webhook/v1/update/207e26731498cf8563a53387d874c6e55cda1834beddeb3fdf1be5c1834ea9bb.tar.gz
fbb6b01b6845bee9d8b8d8e24f557930 /tmp/webhook/v1/update/image.tar
201e622522ea221967b8e63165fe1794 /tmp/webhook/v1/update/minio |
|
Can you share the size of the files @pschichtel ? |
$ ls -l /tmp/webhook/v1/update/
total 192760
-rwxr-xr-x 1 1000 1000 37036866 Jul 20 14:41 207e26731498cf8563a53387d874c6e55cda1834beddeb3fdf1be5c1834ea9bb.tar.gz
-rwxr-xr-x 1 1000 1000 56932864 Jul 20 14:41 image.tar
-rwxr-xr-x 1 1000 1000 103407768 Jul 20 14:41 minio |
|
btw this was the upgrade from what Should I attach the actual files? |
|
You can thanks. |
|
It was too large for github, so: https://cubyte.cloud/s/CMyXTMKFBirSbLY (the share will expire in 2 days) |
|
Mirror here (will be available for 6 weeks): https://ramon.transferxl.com/details/08jXskbVkwJw02/ |
|
we'll try to reproduce and share our findings |
|
I've reproduced this again on my multi-node production cluster. Uploading its file as well doesn't seem to provide much value. As this might be related to k0s: Would it be helpful to provide a minimal k0sctl configuration to quickly setup a single-node cluster for testing? |
yes can you provide that? |
|
@harshavardhana https://gist.github.com/pschichtel/c1f45409a797d71eeb588e434947721b that's a bash script that creates a minimal setup with the operator and 1 tenant. the patch command printed by the script at the end will trigger the problem. the script embeds all yaml files for customizations, so you can override images however you need. |
|
the setup doesn't use k0sctl, it executes k0s directly with the same options k0sctl would render into the systemd service. It's derived from a setup of mine. |
|
I encountered the same issue when upgrading from
Some more detailsThe The manifest contains (formatted): [
{
"Config": "sha256:07520491faf15698a4853bcd6dd10fd05935f64ee5ed10a9fcfdfeefc6ae2d3a",
"RepoTags": [
"minio/minio:RELEASE.2024-08-03T04-33-23Z"
],
"Layers": [
"5f328c14e09dcf260227fb3a7eb4d0ef531ad567bd2868d0c5cef9f0190d1f5f.tar.gz",
"1f53073e01a069a457b592eaa38eddd12a5ff8df351a27085d5875b356cbe3c5.tar.gz",
"44cdc501a5a3f3c2ee4b03b01f4ad06e472f766d5e10ca1b95ce16a2058df041.tar.gz",
"a5140749f35fe0b43df9a368c47f422a879e4bc2afddf0f3a362ade19e3fb5ad.tar.gz",
"5977bc78f8400626b02d7c8ee659e9f22777abfc20ba480612bb76046e2eadf1.tar.gz",
"a7c0c330019377d8ba92526bec31fa1e23221e24b25570ad812193dcce8a90af.tar.gz",
"4086fb25cb3aecb8c62916a2063233d68d8f66df8372dde579338d24347c2cf1.tar.gz",
"9ebf38c2ce8c2b2b3698f0e8d7ba7356835275ded5e253d8ca9c7080be05e5a5.tar.gz"
]
}
]It did fetch the
This is the contents of this tar-file: This tar-ball only contains |
|
Yeah looks like we removed them when we moved to micro UBI8 base image. We can fix this though by copying the relevant files. |
|
This PR will improve the error message: #2278 if it can't find all files. |
|
@harshavardhana Okay, I'll submit a PR to fix the four Dockerfiles, see minio/minio#20270. |
|
@harshavardhana I just tested this on my single-node home-lab by upgrading to |
|
@pschichtel I checked by running the following: It seems the extra files are not in there: I'll investigate further. I expected the CI/CD build to use the Dockerfile in this patch. It looks like the build procedure uses |
That is not true @ramondeklein it does use the Dockerfile.release We do not do any |
The container has everything that is needed. |
|
The file is available in the second "layer" of the image. Operator needs to be updated to handle this or if you want to merge them into same layer you have to change the Dockerfile.release to diff --git a/Dockerfile.release b/Dockerfile.release
index ca0c3e688..fed3fa926 100644
--- a/Dockerfile.release
+++ b/Dockerfile.release
@@ -54,9 +54,7 @@ ENV MINIO_ACCESS_KEY_FILE=access_key \
MC_CONFIG_DIR=/tmp/.mc
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
-COPY --from=build /go/bin/minio /usr/bin/minio
-COPY --from=build /go/bin/minio.minisig /usr/bin/minio.minisig
-COPY --from=build /go/bin/minio.sha256sum /usr/bin/minio.sha256sum
+COPY --from=build /go/bin/minio* /usr/bin/
COPY --from=build /go/bin/mc /usr/bin/mc
COPY --from=build /go/bin/mc.minisig /usr/bin/mc.minisig
COPY --from=build /go/bin/mc.sha256sum /usr/bin/mc.sha256sum |
This is a follow-up to minio/minio#19510.
While the issue of rolling upgrade partially breaking minio has been solved in minio by properly waiting for all nodes to have a consistent version, actual root cause still seems to exist even with the new operator 6.0.0.
Expected Behavior
Current Behavior
What seems to be happening according to the discussion with @harshavardhana:
operator replaces the binary in all minio podsskippedoperator simultaneously restarts all minio processesskippedPossible Solution
no idea
Steps to Reproduce (for bugs)
Context
I have two affected setups:
This issue has been around for a while (part of it can be seen in the issue reference at the top and the issues that are referenced by that) across a bunch of k8s/k0s, minio and operator 5.x versions.
My setup is a single node k0s
Your Environment
minio-operator): 6.0.0uname -a):Linux server 6.9.10-arch1-1 #1 SMP PREEMPT_DYNAMIC Thu, 18 Jul 2024 18:06:13 +0000 x86_64 GNU/LinuxThe text was updated successfully, but these errors were encountered: