Merge pull request #31 from CybercentreCanada/update/dl-everything

Add service config to try to download everything

README.md

@@ -25,6 +25,7 @@ If you are using an Intezer Analyze On-Premise solution, then you do not need to
 * **private_only**: This is a flag that will only return private submissions on the Intezer Analyze system, if selected.
 * **is_on_premise**: This is a flag used for indicating if the Intezer Analyze system is on-premise, rather than the cloud API.
 * **retry_forever**: This is a flag used for indicating if the service should poll the Intezer Analyze system until it gets a response. If set to `false`, the service will raise an exception immediately.
+* **try_to_download_every_file**: This is a flag used for indicating if we want to attempt to download every available file, despite receiving an error on a previous attempt.
 
 ### Submission Parameters
 * **analysis_id**: This is the analysis ID of an analysis that is already on the system. The cloud API counts retrieving the analysis by file hash as a "File Scan" which counts towards an account's monthly quota. We can circumvent this by submitting the analysis ID of an analysis. That being said, this will ignore the file that you submit to Assemblyline.

intezer_static.py

@@ -789,7 +789,7 @@ def _handle_subanalyses(self, request: ServiceRequest, sha256: str, analysis_id:
                     sub_kv_section, file_verdict_map.get(sub_sha256)
                 )
 
-                if can_we_download_files:
+                if can_we_download_files or self.config.get("try_to_download_every_file", False):
                     file_was_downloaded = self.client.download_file_by_sha256(
                         sub_sha256, self.working_directory
                     )

service_manifest.yml

@@ -33,6 +33,7 @@ config:
   private_only: false
   is_on_premise: false
   retry_forever: true
+  try_to_download_every_file: false
 
 submission_params:
   - default: ""