extract dynamic capabilities #1644

yelhamer · 2023-07-17T10:55:48Z

Checklist

No CHANGELOG update needed

No new tests needed

No documentation update needed

google-cla · 2023-07-17T10:55:58Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

yelhamer · 2023-07-17T11:31:30Z

will work on fixing the protobuf mypy issues and vverbose, then will remove the draft status. feel free to review in the meanwhile

mr-tz

looking good so far!

capa/main.py

capa/features/extractors/viv/extractor.py

yelhamer · 2023-07-19T13:06:05Z

Couldn't fix the mypy issues for some reason...

I tried adding the exclusions using pyproject.toml, mypy.ini, and pre-commit config files. All worked locally, but for some reason only tests/_test_proto,py was being excluded in the Github actions tests.

I'll take a look at this more afterwards, in the meantime, the tests succeed locally so I think a review would be nice (so that I could address the comments and the mypy issues together in future commits). The result_document.py tests might fail if you try to run them locally, and that's because I haven't updated all of the test result document (rd) data nor opened a PR to capa-testfiles, will do so after the dynamic statement PR gets merged (I'll be opening that shortly).

Here are the test results:

williballenthin

see inline comments.

overall, looks good! i'd like to run it locally and see how things work.

do you think we should separate StaticResultDocument from DynamicResultDocument? or, let the code switch on doc.meta.analysis?

capa/features/extractors/base_extractor.py

capa/features/extractors/ida/extractor.py

capa/features/extractors/dnfile/extractor.py

williballenthin · 2023-07-19T13:59:13Z

capa/main.py

+    meta = {
+        "feature_counts": feature_counts,
+    }
+
+    return matches, meta


we should probably introduce a dedicated type for the meta variable

is it okay to introduce a type with just 2 attributes max ("feature_counts" and "library_functions") ?

yup, its fine to introduce trivial types. the major benefit is that mypy can help is identify bugs, as well as serving as a single place for documenting what the data in each field means. otherwise, we're left marking up the various dict construction statements.

…ut_file_path() Co-authored-by: Willi Ballenthin <[email protected]>

capa/features/extractors/base_extractor.py

capa/features/extractors/binja/extractor.py

capa/features/extractors/ida/extractor.py

williballenthin · 2023-07-20T12:06:33Z

with the dynamic branch being sync'd, there will now be conflicts here to be fixed. you should try to merge dynamic-feature-extraction into this branch and update the PR.

…into find-dynamic-capabilities

Co-authored-by: Willi Ballenthin <[email protected]>

Co-authored-by: Moritz <[email protected]>

tests/test_binja_features.py

mr-tz

good progress, please see my inline comments and can you add a few tests on the dynamic rendering and calling capa main (test_main) with CAPE input?

williballenthin

some not-very-hard things to fix, and then i think we should go ahead and merge this. i think the logic looks quite good!

capa/features/extractors/base_extractor.py

capa/features/extractors/binja/extractor.py

capa/features/extractors/cape/extractor.py

capa/features/extractors/ida/extractor.py

capa/main.py

tests/test_binja_features.py

Co-authored-by: Willi Ballenthin <[email protected]>

williballenthin

see a few inline changes, then im happy for this to be merged.

capa/features/extractors/base_extractor.py

capa/features/extractors/ida/extractor.py

Co-authored-by: Willi Ballenthin <[email protected]>

mr-tz

should be good to merge, not sure what's wrong with the failing 3.11 test

initial commit

e3f60ea

yelhamer changed the base branch from master to dynamic-feature-extraction July 17, 2023 10:55

bugfixes

4af84e5

yelhamer marked this pull request as draft July 17, 2023 11:30

mr-tz reviewed Jul 17, 2023

View reviewed changes

capa/main.py Outdated Show resolved Hide resolved

capa/main.py Show resolved Hide resolved

add vverbose rendering

bc46bf3

williballenthin reviewed Jul 18, 2023

View reviewed changes

capa/features/extractors/viv/extractor.py Outdated Show resolved Hide resolved

yelhamer added 2 commits July 18, 2023 20:38

add removed tests

e5d7903

mypy.ini: ignore proto issues

4e4b123

yelhamer force-pushed the find-dynamic-capabilities branch 4 times, most recently from a51bf13 to 4e4b123 Compare July 19, 2023 13:00

update extractors and tests

c5d08ec

yelhamer requested review from mr-tz and williballenthin July 19, 2023 13:07

williballenthin reviewed Jul 19, 2023

View reviewed changes

Update capa/features/extractors/ida/extractor.py: add call to get_inp…

7de223f

…ut_file_path() Co-authored-by: Willi Ballenthin <[email protected]>

mr-tz reviewed Jul 20, 2023

View reviewed changes

capa/features/extractors/base_extractor.py Outdated Show resolved Hide resolved

capa/features/extractors/binja/extractor.py Outdated Show resolved Hide resolved

capa/features/extractors/ida/extractor.py Outdated Show resolved Hide resolved

yelhamer and others added 7 commits July 20, 2023 20:02

Merge remote-tracking branch 'parentrepo/dynamic-feature-extraction' …

1029b36

…into find-dynamic-capabilities

fix bugs

8ac9caf

fix tests

0a4fe58

add copyright and remove old test

d99b16e

use pathlib.Path() in binja and ida extractors

482e0d3

Update capa/features/extractors/base_extractor.py

fd7b926

Co-authored-by: Willi Ballenthin <[email protected]>

Update capa/features/extractors/base_extractor.py

2b2b2b6

Co-authored-by: Willi Ballenthin <[email protected]>

fix comment typo

da4e887

Co-authored-by: Moritz <[email protected]>

mr-tz reviewed Jul 21, 2023

View reviewed changes

tests/test_binja_features.py Show resolved Hide resolved

mr-tz reviewed Jul 21, 2023

View reviewed changes

williballenthin requested changes Jul 21, 2023

View reviewed changes

yelhamer and others added 11 commits July 21, 2023 13:15

update compute_dynamic_layout with the appropriate type

6f3fb42

Co-authored-by: Willi Ballenthin <[email protected]>

update compute_static_layout with the appropriate types

bd83316

Co-authored-by: Willi Ballenthin <[email protected]>

address @mr-tz main.py review comments

736b2cd

use ida's hash-extraction functions

3ab3c61

remove the usage of SampleHashes's __iter__() method

8085cae

migrate the get_sample_hashes() function to each individual extractor

6741229

add the skipif mark back

ab585ef

fix linting issues

4ec39d4

replace : FeatureSet annotations with a comment type annotation

c4ba5af

fix bugs

830bad5

add get_sample_hashes() to NullFeatureExtractor

3d1a1fb

yelhamer requested a review from williballenthin July 21, 2023 14:10

williballenthin approved these changes Jul 21, 2023

View reviewed changes

mr-tz mentioned this pull request Jul 21, 2023

Test IDA plugin after merge of changes from find-dynamic-capabilities branch #1660

Closed

yelhamer and others added 5 commits July 21, 2023 15:39

Update capa/features/extractors/base_extractor.py

90298fe

Co-authored-by: Willi Ballenthin <[email protected]>

remove SampleHashes __iter__method

d13114e

Co-authored-by: Willi Ballenthin <[email protected]>

Update capa/features/extractors/ida/extractor.py

c32ac19

Co-authored-by: Willi Ballenthin <[email protected]>

Update capa/features/extractors/base_extractor.py

344b3e9

Co-authored-by: Willi Ballenthin <[email protected]>

add get_sample_hashes() to elf extractor

d8c28e8

mr-tz approved these changes Jul 21, 2023

View reviewed changes

williballenthin merged commit f4bdff0 into mandiant:dynamic-feature-extraction Jul 21, 2023
13 of 15 checks passed

yelhamer added breaking-change introduces a breaking change that should be released in a major version gsoc Work related to Google Summer of Code project. dynamic related to dynamic analysis flavor labels Aug 3, 2023

yelhamer linked an issue Aug 3, 2023 that may be closed by this pull request

update main logic to recognize CAPE analysis reports and invoke the right extractor #1541

Closed

yelhamer mentioned this pull request Aug 3, 2023

update main logic to recognize CAPE analysis reports and invoke the right extractor #1541

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

extract dynamic capabilities #1644

extract dynamic capabilities #1644

yelhamer commented Jul 17, 2023

google-cla bot commented Jul 17, 2023

yelhamer commented Jul 17, 2023

mr-tz left a comment

yelhamer commented Jul 19, 2023

williballenthin left a comment

williballenthin Jul 19, 2023

yelhamer Jul 21, 2023

williballenthin Jul 21, 2023

williballenthin commented Jul 20, 2023

mr-tz left a comment

williballenthin left a comment

williballenthin left a comment

mr-tz left a comment

extract dynamic capabilities #1644

extract dynamic capabilities #1644

Conversation

yelhamer commented Jul 17, 2023

Checklist

google-cla bot commented Jul 17, 2023

yelhamer commented Jul 17, 2023

mr-tz left a comment

Choose a reason for hiding this comment

yelhamer commented Jul 19, 2023

williballenthin left a comment

Choose a reason for hiding this comment

williballenthin Jul 19, 2023

Choose a reason for hiding this comment

yelhamer Jul 21, 2023

Choose a reason for hiding this comment

williballenthin Jul 21, 2023

Choose a reason for hiding this comment

williballenthin commented Jul 20, 2023

mr-tz left a comment

Choose a reason for hiding this comment

williballenthin left a comment

Choose a reason for hiding this comment

williballenthin left a comment

Choose a reason for hiding this comment

mr-tz left a comment

Choose a reason for hiding this comment