Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: delta transitions in shared-memory mode #615

Open
wants to merge 33 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
1edaa13
Start adding delta support
Sunjeet May 20, 2023
9d98237
Continue adding delta support
Sunjeet May 20, 2023
0a73dd9
GapEncodedVariableLengthIntegerReader.java supports shared memory mode
Sunjeet May 20, 2023
798ad32
BlobByteBuffer supports putLong
Sunjeet May 20, 2023
67ee931
EncodedLongBuffer supports setElementValue
Sunjeet May 20, 2023
49429d4
EncodedLongBuffer supports testCopyBitRange
Sunjeet May 21, 2023
5dea324
EncodedLongBuffer supports increment/incrementMany
Sunjeet May 21, 2023
b3fe5e2
EncodedLongBuffer- add unit test for copy small bit range
Sunjeet May 21, 2023
359002d
EncodedLongBuffer implements clearElementValue
Sunjeet May 21, 2023
bf460ae
Run it up for simple data model
Sunjeet May 21, 2023
6fa9a07
Implement destroy for BlobByteBuffer, but dont invoke it yet
Sunjeet May 21, 2023
7d669a6
Refactor fixed length data provisioning
Sunjeet May 21, 2023
3446435
Cleanup before touching non object types
Sunjeet May 21, 2023
ae505e6
Support remaining fixed length types- list, set, map
Sunjeet May 21, 2023
55b655c
Shared memory mode delta transitions for variable length types
Sunjeet May 22, 2023
1bce619
Cleanup
Sunjeet May 22, 2023
770fac8
Bugfix for empty transition file being mmapped, other refactor/cleanup
Sunjeet Jun 10, 2023
46dfd6b
Gap encoded combined removed ordinals applicable conditionally- alway…
Sunjeet Jun 12, 2023
730cec9
Delta application performance- bulk copy fixed length data- but only …
Sunjeet Jun 13, 2023
a75f7a1
Delta target file- add schema name and shard num for diagnostic, disa…
Sunjeet Jun 17, 2023
f66b074
Take a stab at unmap/close
Sunjeet Jun 18, 2023
4a91f9b
Some minor fixes in file cleanup
Sunjeet Jun 18, 2023
37e9084
lifecycle staging files and cleanup
Sunjeet Jun 19, 2023
8770271
Delta transitions files under java.io.tmpdir
Sunjeet Jun 21, 2023
ab45e0d
Change unexpected read from exception to warn
Sunjeet Jun 21, 2023
ea9ef51
Fix for encoded long buffer setELementVAlue getElementValue for bytes…
Sunjeet Jun 22, 2023
5fabbd3
Trying removing explicit call to gc
Sunjeet Jun 22, 2023
013ca0f
Employ sun misc Cleaner for unmap - java8 only
Sunjeet Jun 22, 2023
5f91441
Fewer threads when computing history
Sunjeet Jun 23, 2023
6001a79
Patch allocated target file length
Sunjeet Jun 23, 2023
202cc43
Cleanup logs
Sunjeet Jun 23, 2023
ca84256
Temporarily disable cleaner
Sunjeet Jun 23, 2023
97a1030
Revert "Temporarily disable cleaner"
Sunjeet Jun 23, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Cleanup before touching non object types
Sunjeet committed May 21, 2023

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
commit 3446435a6429cc056ad78d52e0f8015b7ec25cb8
Original file line number Diff line number Diff line change
@@ -20,6 +20,22 @@ public static VariableLengthData get(MemoryMode memoryMode, ArraySegmentRecycler
}
}

public static VariableLengthData allocate(MemoryMode memoryMode, ArraySegmentRecycler memoryRecycler) {
if (memoryMode.equals(MemoryMode.ON_HEAP)) {
return new SegmentedByteArray(memoryRecycler);
} else {
// File targetFile = provisionTargetFile(numBytes, "/tmp/delta-target-" + target.schema.getName() + "_"
// + target.schema.getFieldType(i) + "_"
// + target.schema.getFieldName(i) + "_"
// + LocalDateTime.now().format(DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm"))+ "_" + UUID.randomUUID());
EncodedByteBuffer targetByteBuffer = new EncodedByteBuffer();
// TODO: resize file as needed
// target.varLengthData[i] = targetByteBuffer;
throw new UnsupportedOperationException("Shared memory mode doesnt support delta transitions for var length types (String and byte[])");
// SNAP: TODO: support writing to EncodedByteBuffers to support var length types like strings and byte arrays
}
}

public static void destroy(VariableLengthData vld) {
if (vld instanceof SegmentedByteArray) {
((SegmentedByteArray) vld).destroy();
Original file line number Diff line number Diff line change
@@ -16,10 +16,9 @@
*/
package com.netflix.hollow.core.read.engine.object;

import com.netflix.hollow.core.memory.EncodedByteBuffer;
import com.netflix.hollow.core.memory.FixedLengthDataFactory;
import com.netflix.hollow.core.memory.MemoryMode;
import com.netflix.hollow.core.memory.SegmentedByteArray;
import com.netflix.hollow.core.memory.VariableLengthDataFactory;
import com.netflix.hollow.core.memory.encoding.GapEncodedVariableLengthIntegerReader;
import com.netflix.hollow.core.schema.HollowObjectSchema.FieldType;
import java.io.IOException;
@@ -86,26 +85,12 @@ void applyDelta(MemoryMode memoryMode) throws IOException {
}

long numBits = (long) target.bitsPerRecord * (target.maxOrdinal + 1);
long numLongs = ((numBits - 1) >>> 6) + 1;
long numBytes = numLongs << 3;
target.fixedLengthData = FixedLengthDataFactory.allocate(numBits, memoryMode, target.memoryRecycler,
"/tmp/delta-target-" + target.schema.getName() + "_" + LocalDateTime.now().format(DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm"))+ "_" + UUID.randomUUID());

for(int i=0;i<target.schema.numFields();i++) {
if(target.schema.getFieldType(i) == FieldType.STRING || target.schema.getFieldType(i) == FieldType.BYTES) {
if (memoryMode.equals(MemoryMode.ON_HEAP)) {
target.varLengthData[i] = new SegmentedByteArray(target.memoryRecycler);
} else {
// File targetFile = provisionTargetFile(numBytes, "/tmp/delta-target-" + target.schema.getName() + "_"
// + target.schema.getFieldType(i) + "_"
// + target.schema.getFieldName(i) + "_"
// + LocalDateTime.now().format(DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm"))+ "_" + UUID.randomUUID());
EncodedByteBuffer targetByteBuffer = new EncodedByteBuffer();
// TODO: resize file as needed
target.varLengthData[i] = targetByteBuffer;
throw new UnsupportedOperationException("Shared memory mode doesnt support delta transitions for var length types (String and byte[])");
// SNAP: TODO: support writing to EncodedByteBuffers to support var length types like strings and byte arrays
}
target.varLengthData[i] = VariableLengthDataFactory.allocate(memoryMode, target.memoryRecycler);
}
}