Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PLT-2821 | Cassandra optimisation with Marker logic #3888

Closed
wants to merge 56 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
364c4aa
PLT-2568 optimise vertex fetch from janusgraph
sriram-atlan Nov 13, 2024
8c7ca54
remove method
aarshi0301 Nov 19, 2024
3df42f5
Fix property name
aarshi0301 Nov 18, 2024
0bb62fc
add null cehck
aarshi0301 Nov 18, 2024
9b966a6
remove submit call
aarshi0301 Nov 18, 2024
b0530b3
handle cocurrent modification exception
aarshi0301 Nov 18, 2024
52e645d
update libraries
aarshi0301 Nov 18, 2024
c323ad7
remove scrub results
aarshi0301 Nov 18, 2024
9741f59
fix results
aarshi0301 Nov 19, 2024
1545164
add dummy commit
aarshi0301 Nov 19, 2024
8259f67
Merge pull request #3767 from atlanhq/dummy
aarshi0301 Nov 19, 2024
953bce5
add branch
aarshi0301 Nov 19, 2024
c6c284b
add janus branch
aarshi0301 Nov 19, 2024
d27ea4d
scrub results
aarshi0301 Nov 19, 2024
0015ef7
add flag for janusgraph optimisation
aarshi0301 Nov 19, 2024
54103c8
add flag while retrieving from cassandra
aarshi0301 Nov 19, 2024
b903c7b
log ES query error
aarshi0301 Nov 19, 2024
dc77657
fix map vertex method
aarshi0301 Nov 20, 2024
ec4a32d
add configuration in flag
aarshi0301 Nov 20, 2024
f51f8f6
fix return type
aarshi0301 Nov 20, 2024
391e510
add flag for collapsed result
aarshi0301 Nov 20, 2024
8751387
fix updated by and createdby field
aarshi0301 Nov 20, 2024
6f1549d
fix if condition
aarshi0301 Nov 21, 2024
7feef65
remove branch deployment
aarshi0301 Nov 21, 2024
bd1ddb0
use multi query to improve response time of post elastic query lookups
sriram-atlan Nov 21, 2024
0bc01d6
Merge remote-tracking branch 'origin/janusoptimisation' into janusopt…
sriram-atlan Nov 21, 2024
650cfd1
import function
aarshi0301 Nov 21, 2024
f20e859
remove the mapVertex method that uses valueMap
sriram-atlan Nov 21, 2024
0022dae
build this branch as docker image
sriram-atlan Nov 21, 2024
1be3658
close thread bound janusgraph transaction
sriram-atlan Nov 21, 2024
d73d858
revert entity audit listener
sriram-atlan Nov 21, 2024
0af18e1
multi query with janusgraph 1.0.0
sriram-atlan Nov 22, 2024
4e39d43
add attributes part of multi query
sriram-atlan Nov 23, 2024
9c5a643
include janusoptimisation-multiQuery-1.0.0 in ecr build push
sriram-atlan Nov 23, 2024
01e5b3b
include janusoptimisation-1.0.0 in ecr build push
sriram-atlan Nov 23, 2024
ee86e6e
include janusupgrade in ecr build push
sriram-atlan Nov 23, 2024
e042b4f
bulk fetch with auto batching
sriram-atlan Nov 25, 2024
9ee5645
use properties method with janusgraph than gremlin
sriram-atlan Dec 7, 2024
c8578e4
use edge fetch always
sriram-atlan Dec 7, 2024
a47bdee
log edge and edge properties
sriram-atlan Dec 7, 2024
4dd2743
lookup only when marker exists or when attribute relationship is inwards
sriram-atlan Dec 7, 2024
76fe1c8
include all policy attributes in prefetch exclude
sriram-atlan Dec 7, 2024
358a7bc
revert using prefetch in mapSystemAttributes and toAtlasToObjectId
sriram-atlan Dec 7, 2024
5c6ad1b
remove logs. update comments
sriram-atlan Dec 7, 2024
b95c0e6
fix business attribute
aarshi0301 Dec 9, 2024
4cb985a
add check for business attributes
aarshi0301 Dec 10, 2024
8eef377
code cleanup
aarshi0301 Dec 11, 2024
3b009cd
add upgrade changes
aarshi0301 Dec 11, 2024
d97603d
fix build
aarshi0301 Dec 11, 2024
a6a7b92
fix build
aarshi0301 Dec 11, 2024
2692ae2
fetch only active edges
aarshi0301 Dec 12, 2024
b4caf2b
Minor refactoring
aarshi0301 Dec 13, 2024
9de9e60
replace in and out with both
aarshi0301 Dec 13, 2024
aa040f6
remove guid
aarshi0301 Dec 13, 2024
ab8ad60
Merge branch 'janusupgrade' into dev/stage/janusupgrade
aarshi0301 Dec 13, 2024
dc98a29
restor
aarshi0301 Dec 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
handle cocurrent modification exception
  • Loading branch information
aarshi0301 committed Nov 19, 2024
commit b0530b36b73774e3ff566112faa3e4a262aa6237
Original file line number Diff line number Diff line change
@@ -1096,6 +1096,7 @@ private void prepareSearchResult(AtlasSearchResult ret, DirectIndexQueryResult i

// Use ConcurrentHashMap for thread-safe access
ConcurrentHashMap<String, AtlasEntityHeader> headers = new ConcurrentHashMap<>();
ConcurrentHashMap<String, AtlasEntityHeader> entitiesSet = new ConcurrentHashMap<>();

// Run vertex processing in limited parallel threads
CompletableFuture.runAsync(() -> vertices.parallelStream().forEach(vertex -> {
@@ -1145,9 +1146,14 @@ private void prepareSearchResult(AtlasSearchResult ret, DirectIndexQueryResult i
}

if (header != null) {
ret.addEntity(header);
entitiesSet.put(header.getGuid(), header);
}
});

ret.setEntities(new ArrayList<>(entitiesSet.values()));
if (!searchParams.getEnableFullRestriction()) {
scrubSearchResults(ret, searchParams.getSuppressLogs());
}
}

// Non-recursive collapse processing