Change list of match ids to set for json_handler.py

CHANGELOG.md

@@ -1,5 +1,11 @@
 # Change Log
 
+## v0.66
+
+- [#395](https://github.com/awslabs/amazon-s3-find-and-forget/issues/395):
+  Increase the speed of the json_handler by migrating from a list to a set.
+  Move from O(n) to O(1)
+
 ## v0.65
 
 - [#393](https://github.com/awslabs/amazon-s3-find-and-forget/issues/393): Fix

backend/ecs_tasks/delete_files/json_handler.py

@@ -52,7 +52,7 @@ def delete_matches_from_json_file(input_file, to_delete, compressed=False):
             for column in to_delete:
                 if column["Type"] == "Simple":
                     record = get_value(column["Column"], parsed)
-                    if record and record in column["MatchIds"]:
+                    if record and record in set(column["MatchIds"]):
                         should_delete = True
                         break
                 else:
@@ -61,7 +61,7 @@ def delete_matches_from_json_file(input_file, to_delete, compressed=False):
                         record = get_value(col, parsed)
                         if record:
                             matched.append(record)
-                    if matched in column["MatchIds"]:
+                    if tuple(matched) in set(map(tuple, column["MatchIds"])):
                         should_delete = True
                         break
             if should_delete:

templates/template.yaml

@@ -1,6 +1,6 @@
 AWSTemplateFormatVersion: "2010-09-09"
 Transform: AWS::Serverless-2016-10-31
-Description: Amazon S3 Find and Forget (uksb-1q2j8beb0) (version:v0.65) (tag:main)
+Description: Amazon S3 Find and Forget (uksb-1q2j8beb0) (version:v0.66) (tag:main)
 
 Parameters:
   AccessControlAllowOriginOverride:
@@ -206,7 +206,7 @@ Conditions:
 Mappings:
   Solution:
     Constants:
-      Version: 'v0.65'
+      Version: 'v0.66'
 
 Resources:
   TempBucket: