new splitServicesToContainers default configuration breaks azure storage blob

[EvertonSA]

After upgrading our production instance to new jfrog helm chart, we notice that all services are deployed correctly, but jfrog stop working when uploading to azure storage account.

setting splitServicesToContainers to false make it work again.

I want to have splitServicesToContainers to true, because it's the default. Can you help me to fix? The open ticket to support is https://support.jfrog.com/s/tickets/5006900006gvRe5/migrating-from-75911-to-7637-breaks-azure-storage-account

The application logs indicate an error on the following:

2023-07-25T07:19:56.938Z �[1;32m[jfrt ]�[0;39m �[34m[INFO ]�[0;39m [517dd42dedac829f] [o.a.e.UploadServiceImpl:504   ] [http-nio-8081-exec-1] - Deploy to 'team-docker-local:bitnami/redis/_uploads/470a204b-0a5c-409d-916e-8374b4b83173' Content-Length: 1 (estimation) artificial: false
2023-07-25T07:19:56.938Z �[1;32m[jfrt ]�[0;39m �[39m[DEBUG]�[0;39m [517dd42dedac829f] [a.a.j.JFConnectServiceImpl:181] [http-nio-8081-exec-1] - Verifying if the entitlement "cold_storage" is available
2023-07-25T07:19:56.938Z �[1;32m[jfrt ]�[0;39m �[39m[DEBUG]�[0;39m [517dd42dedac829f] [o.a.a.AddonsManagerImpl:416   ] [http-nio-8081-exec-1] - The entitlement server return "false" for the entitlement "COLD_STORAGE"
2023-07-25T07:19:56.938Z �[1;32m[jfrt ]�[0;39m �[39m[DEBUG]�[0;39m [517dd42dedac829f] [lobStorageV2BinaryProvider:276] [http-nio-8081-exec-1] - Uploading to Azure by writing to temp file and upload it
2023-07-25T07:19:56.938Z �[1;32m[jfrt ]�[0;39m �[39m[DEBUG]�[0;39m [517dd42dedac829f] [b.c.SavedToFileInputStream:136] [http-nio-8081-exec-1] - Deleting temp file file /var/opt/jfrog/artifactory/data/cache/_pre/dbRecord8923238832558008007-517dd42dedac829f-team-docker-local.bin
2023-07-25T07:19:56.939Z �[1;32m[jfrt ]�[0;39m �[39m[DEBUG]�[0;39m [517dd42dedac829f] [s.d.s.DbTransactionManager:853] [http-nio-8081-exec-1] - Should roll back transaction but cannot - no transaction available
2023-07-25T07:19:56.939Z �[1;32m[jfrt ]�[0;39m �[39m[DEBUG]�[0;39m [517dd42dedac829f] [o.a.e.UploadServiceImpl:205   ] [http-nio-8081-exec-1] - Process 517dd42dedac829f tracking stopped
2023-07-25T07:19:56.939Z �[1;32m[jfrt ]�[0;39m �[39m[DEBUG]�[0;39m [517dd42dedac829f] [AggregatingEventServiceImpl:93] [http-nio-8081-exec-1] - An exception occurred within the invocation remove remote address 1REDACT6 and the events related to it.
2023-07-25T07:19:56.939Z �[1;32m[jfrt ]�[0;39m �[1;31m[ERROR]�[0;39m [517dd42dedac829f] [c.e.m.GlobalExceptionMapper:52] [http-nio-8081-exec-1] - org.jfrog.storage.StorageException: Failed to upload blob to Azure blob storage
2023-07-25T07:19:56.939Z �[1;32m[jfrt ]�[0;39m �[39m[DEBUG]�[0;39m [517dd42dedac829f] [c.e.m.GlobalExceptionMapper:53] [http-nio-8081-exec-1] - org.jfrog.storage.StorageException: Failed to upload blob to Azure blob storage
org.artifactory.storage.StorageException: org.jfrog.storage.StorageException: Failed to upload blob to Azure blob storage
	at org.artifactory.storage.db.binstore.service.ArtifactoryBinaryServiceImpl.addBinary(ArtifactoryBinaryServiceImpl.java:143)
	at org.artifactory.repo.service.RepositoryServiceImpl.saveResource(RepositoryServiceImpl.java:2578)
	at jdk.internal.reflect.GeneratedMethodAccessor489.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
	at jdk.proxy4/jdk.proxy4.$Proxy333.saveResource(Unknown Source)
	at org.artifactory.engine.UploadServiceImpl.uploadItemWithContent(UploadServiceImpl.java:702)
	at org.artifactory.engine.UploadServiceImpl.uploadItemWithProvidedContent(UploadServiceImpl.java:674)
	at org.artifactory.engine.UploadServiceImpl.uploadItem(UploadServiceImpl.java:537)
	at org.artifactory.engine.UploadServiceImpl.uploadFile(UploadServiceImpl.java:527)
	at org.artifactory.engine.UploadServiceImpl.uploadArtifact(UploadServiceImpl.java:506)
	at org.artifactory.engine.UploadServiceImpl.adjustResponseAndUpload(UploadServiceImpl.java:318)
	at org.artifactory.engine.UploadServiceImpl.validateRequestAndUpload(UploadServiceImpl.java:278)
	at org.artifactory.engine.UploadServiceImpl.processUpload(UploadServiceImpl.java:220)
	at org.artifactory.api.request.GracefulShutdownAware.track(GracefulShutdownAware.java:36)
	at org.artifactory.engine.UploadServiceImpl.upload(UploadServiceImpl.java:205)
	at jdk.internal.reflect.GeneratedMethodAccessor483.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.artifactory.request.aop.RequestAdvice.invoke(RequestAdvice.java:67)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:220)
	at jdk.proxy4/jdk.proxy4.$Proxy399.upload(Unknown Source)
	at org.artifactory.addon.common.repomd.ArtifactoryService.upload(ArtifactoryService.java:548)
	at org.jfrog.repomd.docker.v2.rest.handler.DockerV2LocalRepoHandler.uploadBlobToUuidPath(DockerV2LocalRepoHandler.java:155)
	at org.jfrog.repomd.docker.v2.rest.handler.DockerV2LocalRepoHandler.patchBlobUpload(DockerV2LocalRepoHandler.java:141)
	at org.jfrog.repomd.docker.v2.rest.DockerV2Resource.patchUpload(DockerV2Resource.java:123)
	at jdk.internal.reflect.GeneratedMethodAccessor482.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:475)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:397)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:209)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
	at org.artifactory.webapp.servlet.RepoFilter.execute(RepoFilter.java:204)
	at org.artifactory.webapp.servlet.RepoFilter.doFilter(RepoFilter.java:106)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
	at org.artifactory.webapp.servlet.AccessFilter.useAuthentication(AccessFilter.java:627)
	at org.artifactory.webapp.servlet.AccessFilter.authenticateAndExecute(AccessFilter.java:457)
	at org.artifactory.webapp.servlet.AccessFilter.doFilterInternal(AccessFilter.java:308)
	at org.artifactory.webapp.servlet.AccessFilter.doFilter(AccessFilter.java:226)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
	at org.artifactory.webapp.servlet.RequestFilter.doFilter(RequestFilter.java:88)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
	at org.artifactory.webapp.servlet.ArtifactoryCsrfFilter.doFilter(ArtifactoryCsrfFilter.java:96)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
	at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:164)
	at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:80)
	at org.artifactory.webapp.servlet.SessionFilter.doFilter(SessionFilter.java:67)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
	at org.artifactory.webapp.servlet.ArtifactoryTracingFilter.doFilter(ArtifactoryTracingFilter.java:38)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
	at org.artifactory.webapp.servlet.ArtifactoryFilter.doFilter(ArtifactoryFilter.java:126)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
	at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:661)
	at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:425)
	at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:357)
	at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:294)
	at org.artifactory.util.DockerInternalRewrite.redirect(DockerInternalRewrite.java:66)
	at org.artifactory.webapp.servlet.ArtifactoryFilter.redirectIfNeeded(ArtifactoryFilter.java:156)
	at org.artifactory.webapp.servlet.ArtifactoryFilter.doFilter(ArtifactoryFilter.java:116)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:481)
	at org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:289)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
	at org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:542)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:390)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:960)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1791)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.jfrog.storage.StorageException: Failed to upload blob to Azure blob storage
	at org.jfrog.type.azureblobstorage.v2.AzureBlobStorageV2BinaryProvider.saveToTempFileAndUpload(AzureBlobStorageV2BinaryProvider.java:560)
	at org.jfrog.type.azureblobstorage.v2.AzureBlobStorageV2BinaryProvider.addStream(AzureBlobStorageV2BinaryProvider.java:277)
	at org.jfrog.storage.binstore.providers.cachefs.FileCacheBinaryProviderImpl.addStream(FileCacheBinaryProviderImpl.java:220)
	at org.jfrog.storage.binstore.providers.federated.FederatedBinaryProvider.addStream(FederatedBinaryProvider.java:364)
	at org.jfrog.storage.binstore.BinaryServiceImpl.addBinary(BinaryServiceImpl.java:390)
	at jdk.internal.reflect.GeneratedMethodAccessor490.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:220)
	at jdk.proxy4/jdk.proxy4.$Proxy376.addBinary(Unknown Source)
	at org.artifactory.storage.db.binstore.service.ArtifactoryBinaryServiceImpl.addBinary(ArtifactoryBinaryServiceImpl.java:141)
	... 115 common frames omitted
Caused by: java.io.IOException: Directory 'data/filestore/temp' could not be created
	at org.apache.commons.io.FileUtils.openOutputStream(FileUtils.java:2436)
	at org.apache.commons.io.FileUtils.openOutputStream(FileUtils.java:2397)
	at org.apache.commons.io.FileUtils.copyToFile(FileUtils.java:1085)
	at org.apache.commons.io.FileUtils.copyInputStreamToFile(FileUtils.java:992)
	at org.jfrog.storage.binstore.providers.tools.FilePersistenceHelper.saveStreamToTempFile(FilePersistenceHelper.java:87)
	at org.jfrog.type.azureblobstorage.v2.AzureBlobStorageV2BinaryProvider.saveToTempFileAndUpload(AzureBlobStorageV2BinaryProvider.java:547)
	... 132 common frames omitted

[Logeshwarsn]

@EvertonSA Thanks for reporting this issue. We will look into this issue internally and get back to you.

[gitta-jfrog]

Hi @EvertonSA

To fix this issue, please remove the following parameters from your custom binarystore.xml:

<binariesDir>data/filestore</binariesDir>
<tempDir>temp</tempDir>

These parameters (which are part of the templates published in JFrog docs) are redundant and create a conflict with the filesystem structure of Artifactory with splitServicesToContainers enabled.

Following the removal of these parameters from your binarystore.xml, Artifactory will use the code default location (/opt/jfrog/artifactory/var/data/artifactory/filestore), and upload of binaries should work as expected.

I'll appreciate your response.
Yarden
JFrog Support.

[EvertonSA]

@gitta-jfrog right,

if I remove this, does it mean that we will not be able to use Direct (Eventual-less) uploads anymore?

[gitta-jfrog]

Hi @EvertonSA
Removing these values should not impact the chain template you are using.

When using a Direct Chain Template, Artifactory needs the binariesDir/temp location only during the upload operation. Following the removal of binariesDir and tempDir from the binarystore.xml (and restart Artifactory nodes), Artifactory will use the default location for temp files during uploads ($JFROG_HOME/artifactory/var/data/artifactory/filestore)

Hope this clarifies.

[EvertonSA]

I hereby confirm the solution works.