Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Semantic diff query optimized #577

Merged
merged 46 commits into from
Sep 12, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
773fb84
Optimized implementation of EquivalenceQuery.
tanyaveksler Jun 4, 2023
c5bf934
Added VacuityQuery and RedundancyQuery optimized implementation.
tanyaveksler Jun 20, 2023
aab6396
Added VacuityQuery and RedundancyQuery optimized implementation.
tanyaveksler Jun 20, 2023
54d3343
Ignoring 'complex function' lint error.
tanyaveksler Jun 11, 2023
adba6dd
Added VacuityQuery and RedundancyQuery optimized implementation.
tanyaveksler Jun 20, 2023
ab3a82e
Removed redundant method.
tanyaveksler Jun 13, 2023
c9393ee
Added VacuityQuery and RedundancyQuery optimized implementation.
tanyaveksler Jun 20, 2023
da98a3f
Fixed domain updating mechanism per rule (to avoid activating multipl…
tanyaveksler Jun 20, 2023
9cd810e
Fixed lint errors
tanyaveksler Jun 20, 2023
ac4f679
Enabled strongEquivalence optimized implementation.
tanyaveksler Jun 20, 2023
cbe8d1f
Implemented optimized ContainmentQuery.
tanyaveksler Jun 25, 2023
ac38097
Enabled optimized TwoContainmentQuery and PermitsQuery.
tanyaveksler Jun 25, 2023
6e632b5
Fixed small inaccuracy in handling host endpoints in optimized solution.
tanyaveksler Jun 27, 2023
af4c84f
Merge branch 'equivalence-based-queries-optmized' into containment-ba…
tanyaveksler Jun 27, 2023
a6ef67c
Implemented optimized InterferesQuery
tanyaveksler Jul 2, 2023
959d741
Merge with master
tanyaveksler Jul 9, 2023
3cd1578
Small improvement in print differences for two config queries
tanyaveksler Jul 9, 2023
951b40c
Merge branch 'master' into interferes-based-queries-optimized
tanyaveksler Jul 11, 2023
5b21d04
Optimized implementation of intersects and forbids queries.
tanyaveksler Jul 11, 2023
2b6b13b
Fixed bug in creation of optimized istio policy properties.
tanyaveksler Jul 11, 2023
3d840a4
Opened for optimized run those queries that do not call allowed_conne…
tanyaveksler Jul 11, 2023
a68de7e
Merge with master.
tanyaveksler Jul 16, 2023
22b8f2c
Implemented sanity query optimized.
tanyaveksler Jul 16, 2023
35e8a74
Implemented optimized semantic diff query.
tanyaveksler Jul 18, 2023
08b6b10
Change in semantic diff query: in case of optimized_run == 'debug', a…
tanyaveksler Jul 25, 2023
fc643da
Small fix
tanyaveksler Jul 25, 2023
054aaf6
Merge with master
tanyaveksler Jul 25, 2023
bd22ab9
Improving output of comparison between original and optimized runs in…
tanyaveksler Jul 30, 2023
fc6eaba
Merge branch 'master' into semantic-diff-query-optimized
tanyaveksler Jul 30, 2023
ed791e1
Fixing handling ipv6 blocks in optimized run.
tanyaveksler Jul 30, 2023
4867e95
Fixing lint errors
tanyaveksler Jul 30, 2023
a4e14b0
Update nca/FWRules/ConnectivityGraph.py
tanyaveksler Aug 6, 2023
2fd82da
merge with master
tanyaveksler Aug 6, 2023
8967e2b
Avoid using redundant all_peers (using inactive dimension instead).
tanyaveksler Aug 6, 2023
abb2fbe
Simplified filtering of ipv6 blocks
tanyaveksler Aug 6, 2023
e69b417
Fixed lint warnings.
tanyaveksler Aug 8, 2023
1caf9ce
merge with master
tanyaveksler Aug 8, 2023
6219813
Merge branch 'master' into semantic-diff-query-optimized
tanyaveksler Aug 8, 2023
9c77ec5
Optimized filtering IpBlocks by mask.
tanyaveksler Aug 8, 2023
22cefcb
Merge branch 'master' into semantic-diff-query-optimized
tanyaveksler Aug 15, 2023
d321200
Update nca/CoreDS/Peer.py
tanyaveksler Aug 15, 2023
334ba10
Fixed filtering IpBlocks by mask in ConnectivityMapQuery
tanyaveksler Aug 15, 2023
55ddd55
Refined filtering of IPv6 blocks in original and optimized solutions …
tanyaveksler Aug 15, 2023
13cdc5a
Merge branch 'master' into semantic-diff-query-optimized
tanyaveksler Aug 20, 2023
b24d06d
Merge branch 'master' into semantic-diff-query-optimized
tanyaveksler Sep 5, 2023
e3cbd0f
Merge branch 'master' into semantic-diff-query-optimized
tanyaveksler Sep 12, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion nca/CoreDS/ConnectionSet.py
Original file line number Diff line number Diff line change
Expand Up @@ -585,8 +585,9 @@ def get_non_tcp_connections():
# get rid of ConnectionSet and move the code below to ConnectivityProperties.py

@staticmethod
def get_connection_set_and_peers_from_cube(conn_cube, peer_container,
def get_connection_set_and_peers_from_cube(the_cube, peer_container,
relevant_protocols=ProtocolSet(True)):
conn_cube = the_cube.copy()
src_peers = conn_cube["src_peers"] or peer_container.get_all_peers_group(True)
conn_cube.unset_dim("src_peers")
dst_peers = conn_cube["dst_peers"] or peer_container.get_all_peers_group(True)
Expand Down
11 changes: 11 additions & 0 deletions nca/CoreDS/ConnectivityProperties.py
Original file line number Diff line number Diff line change
Expand Up @@ -491,3 +491,14 @@ def are_auto_conns(self):
if cube[src_peers_index] != cube[dst_peers_index] or not cube[src_peers_index].is_single_value():
return False
return True

def props_without_auto_conns(self):
"""
Return the properties after removing all connections from peer to itself
"""
peers = self.project_on_one_dimension("src_peers") | self.project_on_one_dimension("dst_peers")
auto_conns = ConnectivityProperties()
for peer in peers:
auto_conns |= ConnectivityProperties.make_conn_props_from_dict({"src_peers": PeerSet({peer}),
"dst_peers": PeerSet({peer})})
return self - auto_conns
6 changes: 4 additions & 2 deletions nca/CoreDS/Peer.py
Original file line number Diff line number Diff line change
Expand Up @@ -662,15 +662,17 @@ def get_ip_block_canonical_form(self):
res |= elem
return res

def filter_ipv6_blocks(self, ip_blocks_mask):
def filter_ip_blocks_by_mask(self, ip_blocks_mask):
"""
Update ip blocks in the peer set by keeping only parts overlapping with the given mask.
:param ip_blocks_mask: the mask according to which ip blocks should be updated
:param IpBlock ip_blocks_mask: the mask according to which ip blocks should be updated
"""
peers_to_remove = []
peers_to_add = []
for peer in self:
if isinstance(peer, IpBlock):
if peer.contained_in(ip_blocks_mask):
continue # optimization - avoid removing and adding the same peer
peers_to_remove.append(peer)
if peer.overlaps(ip_blocks_mask):
new_peer = peer.copy()
Expand Down
10 changes: 10 additions & 0 deletions nca/FWRules/ConnectivityGraph.py
Original file line number Diff line number Diff line change
Expand Up @@ -57,13 +57,23 @@ def add_edges_from_cube_dict(self, conn_cube, peer_container):
Add edges to the graph according to the give cube
:param ConnectivityCube conn_cube: the given cube
whereas all other values should be filtered out in the output
:param PeerContainer peer_container: the peer container
"""
conns, src_peers, dst_peers = \
ConnectionSet.get_connection_set_and_peers_from_cube(conn_cube, peer_container)
for src_peer in src_peers:
for dst_peer in dst_peers:
self.connections_to_peers[conns].append((src_peer, dst_peer))

def add_props_to_graph(self, props, peer_container):
"""
Add edges to the graph according to the given connectivity properties
:param ConnectivityProperties props: the given connectivity properties
:param PeerContainer peer_container: the peer container
"""
for cube in props:
self.add_edges_from_cube_dict(props.get_connectivity_cube(cube), peer_container)

def _get_peer_details(self, peer, format_requirement=False):
"""
Get the name of a peer object for connectivity graph, the type and the namespace
Expand Down
9 changes: 4 additions & 5 deletions nca/NetworkConfig/NetworkConfig.py
Original file line number Diff line number Diff line change
Expand Up @@ -176,7 +176,7 @@ def get_affected_pods(self, is_ingress, layer_name):

return affected_pods

def _check_for_excluding_ipv6_addresses(self, exclude_ipv6):
def check_for_excluding_ipv6_addresses(self, exclude_ipv6):
"""
checks and returns if to exclude non-referenced IPv6 addresses from the config
Excluding the IPv6 addresses will be enabled if the exclude_ipv6 param is True and
Expand All @@ -202,7 +202,7 @@ def get_referenced_ip_blocks(self, exclude_non_ref_ipv6=False):
if self.referenced_ip_blocks is not None:
return self.referenced_ip_blocks

exclude_non_ref_ipv6_from_policies = self._check_for_excluding_ipv6_addresses(exclude_non_ref_ipv6)
exclude_non_ref_ipv6_from_policies = self.check_for_excluding_ipv6_addresses(exclude_non_ref_ipv6)
self.referenced_ip_blocks = Peer.PeerSet()
for policy in self.policies_container.policies.values():
self.referenced_ip_blocks |= policy.referenced_ip_blocks(exclude_non_ref_ipv6_from_policies)
Expand Down Expand Up @@ -329,7 +329,7 @@ def append_policy_to_config(self, policy):
"""
self.policies_container.append_policy(policy)

def filter_conns_by_peer_types(self, conns, all_peers):
def filter_conns_by_peer_types(self, conns):
"""
Filter the given connections by removing several connection kinds that are never allowed
(such as IpBlock to IpBlock connections, connections from DNSEntries, and more).
Expand All @@ -346,7 +346,6 @@ def filter_conns_by_peer_types(self, conns, all_peers):
"dst_peers": all_ips | all_dns_entries})
res -= ip_to_ip_or_dns_conns
# avoid DNSEntry->anything connections
dns_to_any_conns = ConnectivityProperties.make_conn_props_from_dict({"src_peers": all_dns_entries,
"dst_peers": all_peers})
dns_to_any_conns = ConnectivityProperties.make_conn_props_from_dict({"src_peers": all_dns_entries})
res -= dns_to_any_conns
return res
Loading