Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unknown blob when pushing to Artifactory , with s3 as the storage #1921

Open
RahmanBadru opened this issue Sep 12, 2024 · 6 comments
Open

Unknown blob when pushing to Artifactory , with s3 as the storage #1921

RahmanBadru opened this issue Sep 12, 2024 · 6 comments

Comments

@RahmanBadru
Copy link

RahmanBadru commented Sep 12, 2024 •
edited
Loading

Is this a request for help?:
Yes

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Version of Helm and Kubernetes: 3.12.3 and V1.29

Which chart:
Artifactory-107.90.8

Which product license (Enterprise/Pro/oss): Pro

JFrog support reference (if already raised with support team): N/A

What happened: Pushing images to our artifactory instance, throws the error unknown blob after pushing images

the error in the logs show

Deploy to 'docker-local:docker-local/mongo/_uploads/6d562ef3-cab2-4b8d-b60e-f4d5b6a05c2d' Content-Length: 0 (estimation) artificial: false

Failed while saving temp file /opt/jfrog/artifactory/var/data/artifactory/filestore/_pre/dbRecord870694586599854700-35d5a484b17b6f0a-docker-local.bin
org.jfrog.storage.binstore.ifc.ClientInputStreamException: Failed to read stream: Failed to read stream: java.io.EOFException
Caused by: org.jfrog.storage.binstore.ifc.ClientInputStreamException: Failed to read stream: java.io.EOFException
Caused by: org.apache.catalina.connector.ClientAbortException: java.io.EOFException
Caused by: java.io.EOFException: null
2024-09-12T08:20:21.331Z [jfrt ] [ERROR] [35d5a484b17b6f0a] [c.e.m.GlobalExceptionMapper:51] [ttp-nio-8081-exec-97] - org.jfrog.storage.StorageException: Failure during saveToTempFileAndUpload
org.artifactory.storage.StorageException: org.jfrog.storage.StorageException: Failure during saveToTempFileAndUpload
        at org.artifactory.sh.service.ArtifactoryBinaryServiceImpl.addBinary(ArtifactoryBinaryServiceImpl.java:140)
        at org.artifactory.repo.service.RepositoryServiceImpl.saveResource(RepositoryServiceImpl.java:2649)
        at jdk.internal.reflect.GeneratedMethodAccessor359.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:234)
        at jdk.proxy2/jdk.proxy2.$Proxy74.saveResource(Unknown Source)
        at org.artifactory.engine.UploadServiceImpl.uploadItemWithContent(UploadServiceImpl.java:795)
        at org.artifactory.engine.UploadServiceImpl.uploadItemWithProvidedContent(UploadServiceImpl.java:766)
        at org.artifactory.engine.UploadServiceImpl.uploadItem(UploadServiceImpl.java:629)
        at org.artifactory.engine.UploadServiceImpl.uploadFile(UploadServiceImpl.java:618)
        at org.artifactory.engine.UploadServiceImpl.uploadArtifact(UploadServiceImpl.java:597)
        at org.artifactory.engine.UploadServiceImpl.adjustResponseAndUpload(UploadServiceImpl.java:335)
        at org.artifactory.engine.UploadServiceImpl.validateRequestAndUpload(UploadServiceImpl.java:290)
        at org.artifactory.engine.UploadServiceImpl.processUpload(UploadServiceImpl.java:228)
        at org.artifactory.api.request.GracefulShutdownAware.track(GracefulShutdownAware.java:36)
        at org.artifactory.engine.UploadServiceImpl.upload(UploadServiceImpl.java:210)
        at jdk.internal.reflect.GeneratedMethodAccessor351.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)

What you expected to happen: the image pushes successfully

How to reproduce it (as minimally and precisely as possible): docker push artifactory-url/image-name

Anything else we need to know: No support, so we pretty much have to fix this on our own
@arikogan
Copy link

Hi @RahmanBadru. Can you please enable debug logging for org.jfrog.type.s3? This way we may be able to have better visibility of what may have caused this. Thanks.

<logger name="org.jfrog.type.s3" level="debug"/>

@RahmanBadru
Copy link
Author

Here is what i can see

Failed while saving temp file /opt/jfrog/artifactory/var/data/artifactory/filestore/_pre/dbRecord1770488452772219306-40f476e121806379-docker-local.bin
2024-09-12T10:54:35.729Z [jfrt ] [DEBUG] [40f476e121806379] [.j.t.s.S3AwsBinaryProvider:283] [http-nio-8081-exec-3] -
org.jfrog.storage.binstore.ifc.ClientInputStreamException: Failed to read stream: Failed to read stream: java.io.EOFException
        at org.jfrog.storage.binstore.ifc.ClientStream.read(ClientStream.java:36)
        at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1487)
        at org.apache.commons.io.IOUtils.copy(IOUtils.java:1107)
        at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1460)
        at org.apache.commons.io.IOUtils.copy(IOUtils.java:1085)
        at org.apache.commons.io.FileUtils.copyToFile(FileUtils.java:1026)
        at org.apache.commons.io.FileUtils.copyInputStreamToFile(FileUtils.java:934)
        at org.jfrog.storage.binstore.providers.tools.FilePersistenceHelper.saveStreamToTempFile(FilePersistenceHelper.java:52)
        at org.jfrog.type.s3.S3AwsBinaryProvider.saveToTempFileAndUpload(S3AwsBinaryProvider.java:269)
        at org.jfrog.type.s3.S3AwsBinaryProvider.addStream(S3AwsBinaryProvider.java:236)
        at org.jfrog.storage.binstore.providers.cachefs.FileCacheBinaryProviderImpl.addStream(FileCacheBinaryProviderImpl.java:225)
        at org.jfrog.storage.binstore.providers.federated.FederatedBinaryProvider.addStream(FederatedBinaryProvider.java:360)
        at org.jfrog.storage.binstore.BinaryServiceImpl.addStreamToBinaryProviderIfNeeded(BinaryServiceImpl.java:1342)
        at org.jfrog.storage.binstore.BinaryServiceImpl.addBinary(BinaryServiceImpl.java:435)
        at org.jfrog.storage.binstore.BinaryServiceImpl.addBinary(BinaryServiceImpl.java:422)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
--
        at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:312)
        at org.apache.catalina.connector.InputBuffer.checkByteBufferEof(InputBuffer.java:615)
        at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:358)
        at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:132)
        at org.glassfish.jersey.message.internal.EntityInputStream.read(EntityInputStream.java:79)
        at org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:205)
        at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:244)
        at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
        at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:343)
        at org.jfrog.storage.binstore.common.Sha1Sha2Md5ChecksumInputStream.read(Sha1Sha2Md5ChecksumInputStream.java:134)
        at org.jfrog.storage.binstore.common.Sha1Sha2Md5ChecksumInputStream.read(Sha1Sha2Md5ChecksumInputStream.java:118)
        at org.jfrog.storage.binstore.ifc.ClientStream.read(ClientStream.java:34)
        ... 145 common frames omitted
Caused by: java.io.EOFException: null
        at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.fillReadBuffer(NioEndpoint.java:1320)
        at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.read(NioEndpoint.java:1231)
        at org.apache.coyote.http11.Http11InputBuffer.fill(Http11InputBuffer.java:789)
        at org.apache.coyote.http11.Http11InputBuffer.access$400(Http11InputBuffer.java:41)
        at org.apache.coyote.http11.Http11InputBuffer$SocketInputBuffer.doRead(Http11InputBuffer.java:1195)
        at org.apache.coyote.http11.filters.ChunkedInputFilter.readBytes(ChunkedInputFilter.java:254)
        at org.apache.coyote.http11.filters.ChunkedInputFilter.fill(ChunkedInputFilter.java:295)
        at org.apache.coyote.http11.filters.ChunkedInputFilter.parseChunkHeader(ChunkedInputFilter.java:328)
        at org.apache.coyote.http11.filters.ChunkedInputFilter.doRead(ChunkedInputFilter.java:136)
        at org.apache.coyote.http11.Http11InputBuffer.doRead(Http11InputBuffer.java:244)
        at org.apache.coyote.Request.doRead(Request.java:626)
        at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:303)
        ... 156 common frames omitted

@arikogan
Copy link

Thank you for this. It's not related to the Helm Charts but seems to be something with your client interrupting the connection. Please contact our Support.

@RahmanBadru
Copy link
Author

Hi, currently dont have access to support, so would appreciate help

@arikogan
Copy link

@RahmanBadru, I replied on your Stack Overflow question, let's correspond there.

@RahmanBadru
Copy link
Author

RahmanBadru commented Sep 22, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@arikogan @RahmanBadru