Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OAK-8413 Use the new Azure SDK in the Azure Segment Store #1748

Open
wants to merge 24 commits into
base: trunk
Choose a base branch
from
Open

OAK-8413 Use the new Azure SDK in the Azure Segment Store #1748

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
7e3dfc0
OAK-8413 Use the new Azure SDK in the Azure Segment Store
ierandra Sep 12, 2024
35ddcfa
Merge branch 'trunk' of https://github.com/ierandra/jackrabbit-oak in…
ierandra Sep 27, 2024
75edd07
OAK-8413 Use the new Azure SDK in the Azure Segment Store - configure…
ierandra Sep 30, 2024
2768330
OAK-8413 Use the new Azure SDK in the Azure Segment Store - PR review…
ierandra Oct 1, 2024
f98b20b
OAK-8413 Use the new Azure SDK in the Azure Segment Store - fix tests
ierandra Oct 2, 2024
9718bbc
OAK-8413 Use the new Azure SDK in the Azure Segment Store - fix tests
ierandra Oct 2, 2024
a125e1e
OAK-8413 Use the new Azure SDK in the Azure Segment Store - remove de…
ierandra Oct 29, 2024
18db288
Merge branch 'trunk' of https://github.com/ierandra/jackrabbit-oak in…
ierandra Oct 29, 2024
43df254
OAK-8413 Use the new Azure SDK in the Azure Segment Store - PR review…
ierandra Oct 29, 2024
045b4dd
OAK-8413 Use the new Azure SDK in the Azure Segment Store - fix rootP…
ierandra Nov 1, 2024
3ccdb10
OAK-8413 Use the new Azure SDK in the Azure Segment Store - fix rootP…
ierandra Nov 1, 2024
427ca79
OAK-8413 Use the new Azure SDK in the Azure Segment Store - PR review
ierandra Nov 4, 2024
a33dd37
OAK-8413 Use the new Azure SDK in the Azure Segment Store - fix rootP…
ierandra Nov 12, 2024
931cc3b
OAK-8413 Use the new Azure SDK in the Azure Segment Store - PR review
ierandra Nov 12, 2024
6374860
OAK-8413 Use the new Azure SDK in the Azure Segment Store - PR review
ierandra Nov 14, 2024
12c5eae
OAK-8413 Use the new Azure SDK in the Azure Segment Store - PR review
ierandra Nov 20, 2024
ffe0f40
Merge branch 'trunk' of https://github.com/ierandra/jackrabbit-oak in…
ierandra Nov 20, 2024
84180d8
OAK-8413 Use the new Azure SDK in the Azure Segment Store - PR merge
ierandra Nov 20, 2024
871294b
OAK-8413 Use the new Azure SDK in the Azure Segment Store - PR merge
ierandra Nov 21, 2024
d236a3c
OAK-8413 Use the new Azure SDK in the Azure Segment Store - PR review
ierandra Nov 26, 2024
6d5da69
OAK-8413 Use the new Azure SDK in the Azure Segment Store - fix tests
ierandra Dec 19, 2024
d038384
Merge branch 'trunk' of https://github.com/ierandra/jackrabbit-oak in…
ierandra Dec 19, 2024
bf3b7cc
OAK-8413 Use the new Azure SDK in the Azure Segment Store - fix tests
ierandra Dec 19, 2024
92e5d80
Merge branch 'trunk' of https://github.com/ierandra/jackrabbit-oak in…
ierandra Dec 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion oak-commons/src/test/java/org/apache/jackrabbit/oak/commons/FixturesHelper.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ private FixturesHelper() { }
* default fixtures when no {@code nsfixtures} system property is provided
*/
public enum Fixture {
DOCUMENT_NS, @Deprecated SEGMENT_MK, DOCUMENT_RDB, MEMORY_NS, DOCUMENT_MEM, SEGMENT_TAR, SEGMENT_AWS, SEGMENT_AZURE, COMPOSITE_SEGMENT, COMPOSITE_MEM, COW_DOCUMENT
DOCUMENT_NS, @Deprecated SEGMENT_MK, DOCUMENT_RDB, MEMORY_NS, DOCUMENT_MEM, SEGMENT_TAR, SEGMENT_AWS, SEGMENT_AZURE_V8, SEGMENT_AZURE, COMPOSITE_SEGMENT, COMPOSITE_MEM, COW_DOCUMENT
}

private static final Set<Fixture> FIXTURES;
Expand Down
6 changes: 6 additions & 0 deletions oak-it/src/test/java/org/apache/jackrabbit/oak/NodeStoreFixtures.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@
import org.apache.jackrabbit.oak.composite.CompositeSegmentStoreFixture;
import org.apache.jackrabbit.oak.segment.aws.fixture.SegmentAwsFixture;
import org.apache.jackrabbit.oak.segment.azure.fixture.SegmentAzureFixture;
import org.apache.jackrabbit.oak.segment.azure.fixture.SegmentAzureFixtureV8;
import org.apache.jackrabbit.oak.segment.fixture.SegmentTarFixture;

public class NodeStoreFixtures {
Expand All @@ -43,6 +44,8 @@ public class NodeStoreFixtures {

public static final NodeStoreFixture SEGMENT_AWS = new SegmentAwsFixture();

public static final NodeStoreFixture SEGMENT_AZURE_V8 = new SegmentAzureFixtureV8();

public static final NodeStoreFixture SEGMENT_AZURE = new SegmentAzureFixture();

public static final NodeStoreFixture DOCUMENT_NS = new DocumentMongoFixture();
Expand Down Expand Up @@ -80,6 +83,9 @@ public static Collection<Object[]> asJunitParameters(Set<FixturesHelper.Fixture>
if (fixtures.contains(FixturesHelper.Fixture.SEGMENT_AZURE)) {
configuredFixtures.add(SEGMENT_AZURE);
}
if (fixtures.contains(FixturesHelper.Fixture.SEGMENT_AZURE_V8)) {
configuredFixtures.add(SEGMENT_AZURE_V8);
}
if (fixtures.contains(FixturesHelper.Fixture.COMPOSITE_SEGMENT)) {
configuredFixtures.add(COMPOSITE_SEGMENT);
}
Expand Down
2 changes: 1 addition & 1 deletion oak-it/src/test/java/org/apache/jackrabbit/oak/spi/state/NodeStoreTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -459,7 +459,7 @@ public void moveToDescendant() {
if (fixture == NodeStoreFixtures.SEGMENT_TAR || fixture == NodeStoreFixtures.MEMORY_NS
|| fixture == NodeStoreFixtures.COMPOSITE_MEM || fixture == NodeStoreFixtures.COMPOSITE_SEGMENT
|| fixture == NodeStoreFixtures.COW_DOCUMENT || fixture == NodeStoreFixtures.SEGMENT_AWS
|| fixture == NodeStoreFixtures.SEGMENT_AZURE) {
|| fixture == NodeStoreFixtures.SEGMENT_AZURE_V8 || fixture == NodeStoreFixtures.SEGMENT_AZURE) {
assertTrue(x.moveTo(x, "xx"));
assertFalse(x.exists());
assertFalse(test.hasChildNode("x"));
Expand Down
5 changes: 5 additions & 0 deletions oak-parent/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -693,6 +693,11 @@
<artifactId>jackson-dataformat-smile</artifactId>
<version>${jackson.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.dataformat</groupId>
<artifactId> jackson-dataformat-xml</artifactId>
ierandra marked this conversation as resolved.
Show resolved Hide resolved
<version>${jackson.version}</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
Expand Down
6 changes: 3 additions & 3 deletions oak-run-commons/src/main/java/org/apache/jackrabbit/oak/fixture/SegmentTarFixture.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@
import org.apache.jackrabbit.oak.segment.aws.AwsContext;
import org.apache.jackrabbit.oak.segment.aws.AwsPersistence;
import org.apache.jackrabbit.oak.segment.aws.Configuration;
import org.apache.jackrabbit.oak.segment.azure.AzurePersistence;
import org.apache.jackrabbit.oak.segment.azure.v8.AzurePersistenceV8;
import org.apache.jackrabbit.oak.segment.compaction.SegmentGCOptions;
import org.apache.jackrabbit.oak.segment.file.FileStore;
import org.apache.jackrabbit.oak.segment.file.FileStoreBuilder;
Expand Down Expand Up @@ -289,7 +289,7 @@ public Oak getOak(int clusterId) throws Exception {
CloudBlobContainer container = cloud.createCloudBlobClient().getContainerReference(azureContainerName);
container.createIfNotExists();
CloudBlobDirectory directory = container.getDirectoryReference(azureRootPath);
fileStoreBuilder.withCustomPersistence(new AzurePersistence(directory));
fileStoreBuilder.withCustomPersistence(new AzurePersistenceV8(directory));
}

BlobStore blobStore = null;
Expand Down Expand Up @@ -341,7 +341,7 @@ public Oak[] setUpCluster(int n, StatisticsProvider statsProvider) throws Except
container.createIfNotExists();
containers[i] = container;
CloudBlobDirectory directory = container.getDirectoryReference(azureRootPath + "/primary-" + i);
builder.withCustomPersistence(new AzurePersistence(directory));
builder.withCustomPersistence(new AzurePersistenceV8(directory));
}

if (blobStore != null) {
Expand Down
8 changes: 4 additions & 4 deletions ...un-commons/src/main/java/org/apache/jackrabbit/oak/run/cli/SegmentTarFixtureProvider.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
import org.apache.jackrabbit.guava.common.io.Closer;
import org.apache.commons.io.FileUtils;
import org.apache.jackrabbit.oak.segment.SegmentNodeStoreBuilders;
import org.apache.jackrabbit.oak.segment.azure.AzureStorageCredentialManager;
import org.apache.jackrabbit.oak.segment.azure.v8.AzureStorageCredentialManagerV8;
import org.apache.jackrabbit.oak.segment.azure.tool.ToolUtils;
import org.apache.jackrabbit.oak.segment.file.FileStore;
import org.apache.jackrabbit.oak.segment.file.FileStoreBuilder;
Expand All @@ -53,12 +53,12 @@ static NodeStore configureSegment(Options options, BlobStore blobStore, Whiteboa

FileStoreBuilder builder;
if (segmentStoreType == ToolUtils.SegmentStoreType.AZURE) {
final AzureStorageCredentialManager azureStorageCredentialManager = new AzureStorageCredentialManager();
final AzureStorageCredentialManagerV8 azureStorageCredentialManagerV8 = new AzureStorageCredentialManagerV8();
SegmentNodeStorePersistence segmentNodeStorePersistence =
ToolUtils.newSegmentNodeStorePersistence(segmentStoreType, pathOrUri, azureStorageCredentialManager);
ToolUtils.newSegmentNodeStorePersistence(segmentStoreType, pathOrUri, azureStorageCredentialManagerV8);
File tempDir = Files.createTempDirectory("azure-segment-store").toFile();
closer.register(() -> FileUtils.deleteQuietly(tempDir));
closer.register(azureStorageCredentialManager);
closer.register(azureStorageCredentialManagerV8);
builder = fileStoreBuilder(tempDir).withCustomPersistence(segmentNodeStorePersistence);
} else {
builder = fileStoreBuilder(new File(pathOrUri)).withMaxFileSize(256);
Expand Down
2 changes: 1 addition & 1 deletion oak-run-elastic/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@
105 MB: Azure updates
107 MB: RDB/Tomcat (OAK-10752)
-->
<max.jar.size>112197632</max.jar.size>
<max.jar.size>113039632</max.jar.size>

</properties>

Expand Down
10 changes: 5 additions & 5 deletions ...un/src/main/java/org/apache/jackrabbit/oak/explorer/AzureSegmentStoreExplorerBackend.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
package org.apache.jackrabbit.oak.explorer;

import org.apache.jackrabbit.guava.common.io.Files;
import org.apache.jackrabbit.oak.segment.azure.AzureStorageCredentialManager;
import org.apache.jackrabbit.oak.segment.azure.v8.AzureStorageCredentialManagerV8;
import org.apache.jackrabbit.oak.segment.azure.tool.ToolUtils;
import org.apache.jackrabbit.oak.segment.file.InvalidFileStoreVersionException;
import org.apache.jackrabbit.oak.segment.spi.persistence.JournalFile;
Expand All @@ -39,16 +39,16 @@
public class AzureSegmentStoreExplorerBackend extends AbstractSegmentTarExplorerBackend {
private final String path;
private SegmentNodeStorePersistence persistence;
private final AzureStorageCredentialManager azureStorageCredentialManager;
private final AzureStorageCredentialManagerV8 azureStorageCredentialManagerV8;

public AzureSegmentStoreExplorerBackend(String path) {
this.path = path;
this.azureStorageCredentialManager = new AzureStorageCredentialManager();
this.azureStorageCredentialManagerV8 = new AzureStorageCredentialManagerV8();
}

@Override
public void open() throws IOException {
this.persistence = newSegmentNodeStorePersistence(ToolUtils.SegmentStoreType.AZURE, path, azureStorageCredentialManager);
this.persistence = newSegmentNodeStorePersistence(ToolUtils.SegmentStoreType.AZURE, path, azureStorageCredentialManagerV8);

try {
this.store = fileStoreBuilder(Files.createTempDir())
Expand All @@ -63,7 +63,7 @@ public void open() throws IOException {
@Override
public void close() {
super.close();
azureStorageCredentialManager.close();
azureStorageCredentialManagerV8.close();
}

@Override
Expand Down
6 changes: 3 additions & 3 deletions oak-run/src/main/java/org/apache/jackrabbit/oak/run/FileStoreDiffCommand.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@
import joptsimple.OptionSet;
import joptsimple.OptionSpec;
import org.apache.jackrabbit.oak.run.commons.Command;
import org.apache.jackrabbit.oak.segment.azure.AzureStorageCredentialManager;
import org.apache.jackrabbit.oak.segment.azure.v8.AzureStorageCredentialManagerV8;
import org.apache.jackrabbit.oak.segment.azure.tool.ToolUtils;
import org.apache.jackrabbit.oak.segment.file.ReadOnlyFileStore;
import org.apache.jackrabbit.oak.segment.spi.persistence.SegmentNodeStorePersistence;
Expand Down Expand Up @@ -87,8 +87,8 @@ public void execute(String... args) throws Exception {
}
} else {
if (pathOrURI.startsWith("az:")) {
try (AzureStorageCredentialManager azureStorageCredentialManager = new AzureStorageCredentialManager()) {
SegmentNodeStorePersistence azurePersistence = ToolUtils.newSegmentNodeStorePersistence(ToolUtils.SegmentStoreType.AZURE, pathOrURI, azureStorageCredentialManager);
try (AzureStorageCredentialManagerV8 azureStorageCredentialManagerV8 = new AzureStorageCredentialManagerV8()) {
SegmentNodeStorePersistence azurePersistence = ToolUtils.newSegmentNodeStorePersistence(ToolUtils.SegmentStoreType.AZURE, pathOrURI, azureStorageCredentialManagerV8);
ReadOnlyFileStore store = fileStoreBuilder(Files.createTempDir()).withCustomPersistence(azurePersistence).withBlobStore(newBasicReadOnlyBlobStore()).buildReadOnly();
statusCode = Diff.builder()
.withPath(pathOrURI)
Expand Down
12 changes: 6 additions & 6 deletions oak-run/src/test/java/org/apache/jackrabbit/oak/run/DataStoreCommandTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,8 +75,8 @@
import org.apache.jackrabbit.oak.run.cli.BlobStoreOptions.Type;
import org.apache.jackrabbit.oak.segment.SegmentNodeStore;
import org.apache.jackrabbit.oak.segment.SegmentNodeStoreBuilders;
import org.apache.jackrabbit.oak.segment.azure.AzureStorageCredentialManager;
import org.apache.jackrabbit.oak.segment.azure.AzureUtilities;
import org.apache.jackrabbit.oak.segment.azure.v8.AzureStorageCredentialManagerV8;
import org.apache.jackrabbit.oak.segment.azure.v8.AzureUtilitiesV8;
import org.apache.jackrabbit.oak.segment.azure.tool.ToolUtils;
import org.apache.jackrabbit.oak.segment.compaction.SegmentGCOptions;
import org.apache.jackrabbit.oak.segment.file.FileStore;
Expand Down Expand Up @@ -1141,7 +1141,7 @@ class SegmentStoreFixture implements StoreFixture {
class AzureSegmentStoreFixture extends SegmentStoreFixture {
private static final String AZURE_DIR = "repository";
private String container;
private final AzureStorageCredentialManager azureStorageCredentialManager = new AzureStorageCredentialManager();
private final AzureStorageCredentialManagerV8 azureStorageCredentialManagerV8 = new AzureStorageCredentialManagerV8();

@Override public NodeStore init(DataStoreBlobStore blobStore, File storeFile) throws Exception {
Properties props = AzureDataStoreUtils.getAzureConfig();
Expand All @@ -1151,14 +1151,14 @@ class AzureSegmentStoreFixture extends SegmentStoreFixture {
container = container + System.currentTimeMillis();
// Create the azure segment container
String connectionString = getAzureConnectionString(accessKey, secretKey, container, AZURE_DIR);
AzureUtilities.cloudBlobDirectoryFrom(connectionString, container, AZURE_DIR);
AzureUtilitiesV8.cloudBlobDirectoryFrom(connectionString, container, AZURE_DIR);

// get the azure uri expected by the command
storePath = getAzureUri(accessKey, container, AZURE_DIR);

// initialize azure segment for test setup
SegmentNodeStorePersistence segmentNodeStorePersistence =
ToolUtils.newSegmentNodeStorePersistence(ToolUtils.SegmentStoreType.AZURE, storePath, azureStorageCredentialManager);
ToolUtils.newSegmentNodeStorePersistence(ToolUtils.SegmentStoreType.AZURE, storePath, azureStorageCredentialManagerV8);
fileStore = fileStoreBuilder(storeFile).withBlobStore(blobStore)
.withCustomPersistence(segmentNodeStorePersistence).build();

Expand Down Expand Up @@ -1190,7 +1190,7 @@ protected String getAzureConnectionString(String accountName, String secret, Str
public void after() {
try {
AzureDataStoreUtils.deleteContainer(container);
azureStorageCredentialManager.close();
azureStorageCredentialManagerV8.close();
} catch(Exception e) {
log.error("Error in cleaning the container {}", container, e);
}
Expand Down
46 changes: 43 additions & 3 deletions oak-segment-azure/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,10 +48,12 @@
org.apache.jackrabbit.oak.segment.remote*,
com.fasterxml.jackson.annotation;resolution:=optional,
com.fasterxml.jackson.databind*;resolution:=optional,
com.fasterxml.jackson.dataformat.xml;resolution:=optional,
com.fasterxml.jackson.dataformat.*;resolution:=optional,
com.fasterxml.jackson.datatype*;resolution:=optional,
com.azure.identity.broker.implementation;resolution:=optional,
com.azure.xml;resolution:=optional,
com.azure.storage.blob*;resolution:=optional,
com.azure.storage.common*;resolution:=optional,
com.azure.storage.internal*;resolution:=optional,
com.microsoft.aad.msal4jextensions*;resolution:=optional,
com.nimbusds.langtag;resolution:=optional,
com.sun.jna*;resolution:=optional,
Expand All @@ -71,18 +73,28 @@
org.apache.jackrabbit.oak.segment.azure,
org.apache.jackrabbit.oak.segment.azure.queue,
org.apache.jackrabbit.oak.segment.azure.util,
com.fasterxml.jackson.dataformat.xml,
com.fasterxml.jackson.dataformat.xml.deser,
com.microsoft.azure.storage,
com.microsoft.azure.storage.core,
com.microsoft.azure.storage.blob,
com.azure.core.credential,
com.azure.identity
com.azure.identity,
com.azure.blob,
com.azure.storage.common,
com.azure.storage.internal.avro.implementation
</Export-Package>
<Embed-Dependency>
azure-storage,
azure-keyvault-core,
azure-core,
azure-identity,
azure-json,
azure-xml,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Arer we embedding the new SDK? Why?

(We embedded the old one due to the Guava dependency issue, but this should not be needed here)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried without and I got the following error at build:
[ERROR] Bundle oak-segment-azure:1.71-ierandra-T20241104112150-3ccdb109e0 is importing package(s) com.azure.xml in start level 15 but no bundle is exporting these for that start level.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the oak-it-osgi tests? In which case we may have to add the dependency there.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, when building CQ: https://jenkins-evergreen.ci.corp.adobe.com/job/Sprouts/job/aemcs-developer-releases/5335/artifact/1-BuildAEM.log

azure-storage-blob,
azure-storage-common,
azure-storage-internal-avro,
jackson-dataformat-xml,
guava,
jsr305,
reactive-streams,
Expand Down Expand Up @@ -188,6 +200,28 @@
<artifactId>azure-keyvault-core</artifactId>
</dependency>

<!-- Azure Blob Storage v12 dependencies -->
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-storage-blob</artifactId>
<version>12.25.3</version>
</dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-storage-common</artifactId>
<version>12.24.3</version>
</dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-storage-internal-avro</artifactId>
<version>12.10.3</version>
</dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-xml</artifactId>
<version>1.0.0</version>
</dependency>

<!-- Azure Identity Client for Microsoft Entra ID dependency -->
<dependency>
<groupId>com.azure</groupId>
Expand Down Expand Up @@ -365,6 +399,12 @@
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.projectreactor</groupId>
<artifactId>reactor-test</artifactId>
<version>3.6.10</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>testcontainers</artifactId>
Expand Down
Loading