Fix - Broken tests

[jbaptperez]

Description

Fixes a broken test.

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

I ran all tests, including the fixed one.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works

[jbaptperez]

@elegantmoose, @clenk, completes the Fix - Broken tests #3013 PR.

[uruwhy]

just for my awareness, what issue is this fixing? The current atomic tests seem to work for me locally

[jbaptperez]

@uruwhy, without this change, the test plugins.atomic.tests.test_atomic_svc.TestAtomicSvc.test_use_default_inputs (plugins/atomic/tests/test_atomic_svc.py:181) can fail in certain circumstances.

Summary

It happens when Caldera has already run once with the plugin atomic enabled before running this test.
This happens frequently while developing.

plugins/atomic/tests/test_atomic_svc.py:180 (TestAtomicSvc.test_use_default_inputs)
'64c4ae_qakbot.bat -a' != 'PathToAtomicsFolder\\T1016\\src\\qakbot.bat -a'

Expected :'PathToAtomicsFolder\\T1016\\src\\qakbot.bat -a'
Actual   :'64c4ae_qakbot.bat -a'
<Click to see difference>

self = <test_atomic_svc.TestAtomicSvc object at 0x10c1d9520>
atomic_svc = <plugins.atomic.app.atomic_svc.AtomicService object at 0x10c1da6f0>
atomic_test = {'auto_generated_guid': '121de5c6-5818-4868-b8a7-8fd07c455c1b', 'description': 'A list of commands known to be perform...omicsFolder\\T1016\\src\\qakbot.bat', 'description': 'File that houses commands to be executed', 'type': 'Path'}}, ...}

    def test_use_default_inputs(self, atomic_svc, atomic_test):
        platform = 'windows'
        string_to_analyze = '#{recon_commands} -a'
        got = atomic_svc._use_default_inputs(test=atomic_test,
                                                platform=platform,
                                                string_to_analyse=string_to_analyze)
>       assert got[0] == 'PathToAtomicsFolder\\T1016\\src\\qakbot.bat -a'
E       AssertionError: assert '64c4ae_qakbot.bat -a' == 'PathToAtomicsFolder\\T1016\\src\\qakbot.bat -a'
E         
E         - PathToAtomicsFolder\T1016\src\qakbot.bat -a
E         + 64c4ae_qakbot.bat -a

plugins/atomic/tests/test_atomic_svc.py:187: AssertionError

Details

The test code first calls plugins.atomic.app.atomic_svc.AtomicService._use_default_inputs, then plugins.atomic.app.atomic_svc.AtomicService._catch_path_to_atomics_folder.
The latter can return 2 different values.
The test success depends on to the returned value.

In the capture below, I called in advance the method in the debugger to see its result before its real call (overriding variables with returned values):

Everything happens in this method.

In the case you start Caldera with the atomic plugin at least once, the plugin, when starting, clones the atomic-red-team repository, and the testing path to the payload plugins/atomic/data/atomic-red-team/atomics/T1016/src/qakbot.bat becomes valid.

The if clause is validated, making a call to plugins.atomic.app.atomic_svc.AtomicService._handle_attachment and changing the resulting payload name (includes a hash)
Therefore, the whole test fails.

Reproducing

To make the test pass, just deletes the generated atomic-red-team repository (plugins/atomic/data/atomic-red-team).
This is the cas after cloning the Caldera repository without the first run.

To make the test fail again, deletes the plugins/atomic/data/abilities repository and start Caldera with the atomic plugin enabled.
The plugin will regenerate the atomic-red-team repository and the abilities directory.

My fix proposal

What I did here is to change the testing default path of the atomic_test fixture (dictionary) to make sure it won't exist, whatever the atomic-red-team repository is cloned or not.

This is one solution I found, but there should be other ways to make this this test deterministic.
However, I'm not expert in Caldera and its plugins.
I am open to suggestions.

[uruwhy]

@jbaptperez
Makes sense, good catch!

[uruwhy]

@elegantmoose approved and ready for merge