Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BinaryView::AddUserSection gets increasingly slower for sections with semantics that aren't DefaultSectionSemantics #6271

Open
WeiN76LQh opened this issue Dec 22, 2024 · 1 comment
Open

BinaryView::AddUserSection gets increasingly slower for sections with semantics that aren't DefaultSectionSemantics #6271

WeiN76LQh opened this issue Dec 22, 2024 · 1 comment
Assignees
@bpotchik
Labels
State: Awaiting Triage Issue is waiting for more in-depth triage from a developer

Comments

@WeiN76LQh
Copy link

Version and Platform (required):

  • Binary Ninja Version: 4.3.6572-dev (37fd69ad)
  • OS: macOS
  • OS Version: 15.1
  • CPU Architecture: M1

Bug Description:
There's a performance issue with BinaryView::AddUserSection where the more sections a BinaryView has the longer it takes to complete if the semantics for the section aren't DefaultSectionSemantics. In the case of DefaultSectionSemantics it appears this always takes less than a millisecond (faster than the resolution of time I was reporting). Outside of DefaultSectionSemantics it doesn't seem to matter what the size of the section is or what the semantics for it are, the time it takes for BinaryView::AddUserSection to complete will be roughly the same and gradually increase as more sections are added.

Steps To Reproduce:
I'm using a modified version of the DSC plugin, which is roughly the same as this PR and then I'm using std::chrono::high_resolution_clock::now() to time how long this call to BinaryView::AddUserSection is taking.

I'm then just running the following script I have in the snippet editor plugin:

import binaryninja.sharedcache
dsc = binaryninja.sharedcache.SharedCache(bv)
bv.set_analysis_hold(True)
for image in dsc.images:
    dsc.load_image_with_install_name(image.name, True) # skip ObjC processing

NOTE: the extra argument to load_image_with_install_name is to skip Objective-C processing which is added with this PR that has not currently been merged.

The script pauses analysis and then just loads all images in the DSC one by one.

Expected Behavior:
It seems to me that this is an unnecessary performance issue. As in its likely that whatever is causing the slow down could be avoided with some optimisation. It would not surprise me if whats going on here is there's a data copy occurring each time BinaryView::AddUserSection based on the number of sections in the binary view. Without seeing the code I'd suspect there's something like an array of non-DefaultSectionSemantics sections and each time a new section is added to that array its getting resized and requiring a memcpy to move the data from the old array to the new one.

Binary:
I was testing on an iOS DSC extracted from iOS 18.1.1, it shouldn't matter exactly which.
@WeiN76LQh
Copy link
Author

For the first library loaded, which was /System/Library/PrivateFrameworks/TokenGenerationInference.framework/TokenGenerationInference, the following timings were recorded:

[SharedCache] AddUserSection 0x25ab022dc -> 0x25abe0530 (0xde254) TokenGenerationInference::__text ReadOnlyCodeSectionSemantics PURE_CODE - 0.003 seconds
[SharedCache] AddUserSection 0x25abe0530 -> 0x25abe3260 (0x2d30) TokenGenerationInference::__auth_stubs ReadOnlyCodeSectionSemantics SYMBOL_STUBS - 0.001 seconds
[SharedCache] AddUserSection 0x25abe3260 -> 0x25abe383c (0x5dc) TokenGenerationInference::__objc_methlist DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x25abe3840 -> 0x25abea20c (0x69cc) TokenGenerationInference::__const ReadOnlyDataSectionSemantics REGULAR - 0.001 seconds
[SharedCache] AddUserSection 0x25abea20c -> 0x25abed808 (0x35fc) TokenGenerationInference::__gcc_except_tab DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x25abed810 -> 0x25abf229c (0x4a8c) TokenGenerationInference::__cstring ReadOnlyDataSectionSemantics CSTRING_LITERALS - 0.001 seconds
[SharedCache] AddUserSection 0x25abf22a0 -> 0x25abf55c3 (0x3323) TokenGenerationInference::__oslogstring ReadOnlyDataSectionSemantics CSTRING_LITERALS - 0.001 seconds
[SharedCache] AddUserSection 0x25abf55c4 -> 0x25abf7050 (0x1a8c) TokenGenerationInference::__swift5_typeref DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x25abf7050 -> 0x25abf8b10 (0x1ac0) TokenGenerationInference::__constg_swiftt DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x25abf8b10 -> 0x25abfaa34 (0x1f24) TokenGenerationInference::__swift5_reflstr DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x25abfaa34 -> 0x25abfc8c4 (0x1e90) TokenGenerationInference::__swift5_fieldmd DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x25abfc8c4 -> 0x25abfc928 (0x64) TokenGenerationInference::__swift5_builtin DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x25abfc928 -> 0x25abfcc70 (0x348) TokenGenerationInference::__swift5_proto DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x25abfcc70 -> 0x25abfce48 (0x1d8) TokenGenerationInference::__swift5_types DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x25abfce48 -> 0x25abfd138 (0x2f0) TokenGenerationInference::__swift5_capture DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x25abfd138 -> 0x25abfd180 (0x48) TokenGenerationInference::__swift5_assocty DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x25abfd180 -> 0x25abfd1b4 (0x34) TokenGenerationInference::__swift5_protos DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x25abfd1b4 -> 0x25abfd248 (0x94) TokenGenerationInference::__swift5_mpenum DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x25abfd248 -> 0x25ac011d8 (0x3f90) TokenGenerationInference::__unwind_info DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x25ac011d8 -> 0x25ac07a50 (0x6878) TokenGenerationInference::__eh_frame DefaultSectionSemantics COALESCED - 0.000 seconds
[SharedCache] AddUserSection 0x25ac07a50 -> 0x25ac07b7d (0x12d) TokenGenerationInference::__objc_classname ReadOnlyDataSectionSemantics CSTRING_LITERALS - 0.001 seconds
[SharedCache] AddUserSection 0x25ac07b7d -> 0x25ac093e4 (0x1867) TokenGenerationInference::__objc_methname ReadOnlyDataSectionSemantics CSTRING_LITERALS - 0.001 seconds
[SharedCache] AddUserSection 0x25ac093e4 -> 0x25ac0b714 (0x2330) TokenGenerationInference::__objc_methtype ReadOnlyDataSectionSemantics CSTRING_LITERALS - 0.001 seconds
[SharedCache] AddUserSection 0x25ac0b720 -> 0x25ac0c000 (0x8e0) TokenGenerationInference::__objc_stubs ReadOnlyCodeSectionSemantics PURE_CODE - 0.001 seconds
[SharedCache] AddUserSection 0x2602bf4d0 -> 0x2602bfba0 (0x6d0) TokenGenerationInference::__got ReadOnlyDataSectionSemantics REGULAR - 0.001 seconds
[SharedCache] AddUserSection 0x2602bfba0 -> 0x2602bfd58 (0x1b8) TokenGenerationInference::__const ReadOnlyDataSectionSemantics REGULAR - 0.001 seconds
[SharedCache] AddUserSection 0x2602bfd58 -> 0x2602bfec8 (0x170) TokenGenerationInference::__objc_classlist ReadOnlyDataSectionSemantics REGULAR - 0.001 seconds
[SharedCache] AddUserSection 0x2602bfec8 -> 0x2602bfee8 (0x20) TokenGenerationInference::__objc_protolist ReadOnlyDataSectionSemantics REGULAR - 0.001 seconds
[SharedCache] AddUserSection 0x2602bfee8 -> 0x2602bfef0 (0x8) TokenGenerationInference::__objc_imageinfo ReadOnlyDataSectionSemantics REGULAR - 0.001 seconds
[SharedCache] AddUserSection 0x2602bfef0 -> 0x2602c02a8 (0x3b8) TokenGenerationInference::__objc_selrefs ReadOnlyDataSectionSemantics REGULAR - 0.001 seconds
[SharedCache] AddUserSection 0x2602c02a8 -> 0x2602c02b0 (0x8) TokenGenerationInference::__objc_protorefs ReadOnlyDataSectionSemantics REGULAR - 0.001 seconds
[SharedCache] AddUserSection 0x2602c02b0 -> 0x2602c02f8 (0x48) TokenGenerationInference::__objc_superrefs ReadOnlyDataSectionSemantics REGULAR - 0.001 seconds
[SharedCache] AddUserSection 0x26bab39e0 -> 0x26bab5088 (0x16a8) TokenGenerationInference::__auth_got DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x26bab5088 -> 0x26bab5740 (0x6b8) TokenGenerationInference::__auth_ptr DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x26bab5740 -> 0x26bab93c8 (0x3c88) TokenGenerationInference::__const ReadOnlyDataSectionSemantics REGULAR - 0.001 seconds
[SharedCache] AddUserSection 0x26bab93c8 -> 0x26bab97c8 (0x400) TokenGenerationInference::__cfstring DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x26bab97c8 -> 0x26babd1c0 (0x39f8) TokenGenerationInference::__objc_const DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x26697e7a0 -> 0x26697ed50 (0x5b0) TokenGenerationInference::__objc_data DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x26697ed50 -> 0x266980f08 (0x21b8) TokenGenerationInference::__data ReadWriteDataSectionSemantics REGULAR - 0.001 seconds
[SharedCache] AddUserSection 0x2654a5170 -> 0x2654a51f4 (0x84) TokenGenerationInference::__objc_ivar DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x2654a51f8 -> 0x2654a6498 (0x12a0) TokenGenerationInference::__data ReadWriteDataSectionSemantics REGULAR - 0.001 seconds
[SharedCache] AddUserSection 0x2654a64a0 -> 0x2654ab450 (0x4fb0) TokenGenerationInference::__bss ReadWriteDataSectionSemantics ZEROFILL - 0.001 seconds
[SharedCache] AddUserSection 0x2654ab450 -> 0x2654ab4e8 (0x98) TokenGenerationInference::__common ReadWriteDataSectionSemantics ZEROFILL - 0.002 seconds

By the 67th, for the library /System/Library/PrivateFrameworks/WorkflowEditor.framework/WorkflowEditor, they were:

[SharedCache] AddUserSection 0x231863ef4 -> 0x231a87370 (0x22347c) WorkflowEditor::__text ReadOnlyCodeSectionSemantics PURE_CODE - 0.190 seconds
[SharedCache] AddUserSection 0x231a87370 -> 0x231a8cb00 (0x5790) WorkflowEditor::__auth_stubs ReadOnlyCodeSectionSemantics SYMBOL_STUBS - 0.233 seconds
[SharedCache] AddUserSection 0x231a8cb00 -> 0x231a9351c (0x6a1c) WorkflowEditor::__objc_methlist DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x231a93520 -> 0x231ab5116 (0x21bf6) WorkflowEditor::__swift5_typeref DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x231ab5118 -> 0x231ab8980 (0x3868) WorkflowEditor::__swift5_capture DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x231ab8980 -> 0x231ac8550 (0xfbd0) WorkflowEditor::__const ReadOnlyDataSectionSemantics REGULAR - 0.196 seconds
[SharedCache] AddUserSection 0x231ac8550 -> 0x231acfa99 (0x7549) WorkflowEditor::__cstring ReadOnlyDataSectionSemantics CSTRING_LITERALS - 0.220 seconds
[SharedCache] AddUserSection 0x231acfa9c -> 0x231ad7ef0 (0x8454) WorkflowEditor::__constg_swiftt DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x231ad7ef0 -> 0x231adc79c (0x48ac) WorkflowEditor::__swift5_reflstr DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x231adc79c -> 0x231ae149c (0x4d00) WorkflowEditor::__swift5_fieldmd DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x231ae149c -> 0x231ae1730 (0x294) WorkflowEditor::__swift5_builtin DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x231ae1730 -> 0x231ae3160 (0x1a30) WorkflowEditor::__swift5_assocty DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x231ae3160 -> 0x231ae38c4 (0x764) WorkflowEditor::__swift5_proto DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x231ae38c4 -> 0x231ae3e44 (0x580) WorkflowEditor::__swift5_types DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x231ae3e44 -> 0x231ae3e6c (0x28) WorkflowEditor::__swift5_protos DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x231ae3e6c -> 0x231ae3eb4 (0x48) WorkflowEditor::__swift5_mpenum DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x231ae3ec0 -> 0x231ae4e85 (0xfc5) WorkflowEditor::__oslogstring ReadOnlyDataSectionSemantics CSTRING_LITERALS - 0.219 seconds
[SharedCache] AddUserSection 0x231ae4e88 -> 0x231ae5714 (0x88c) WorkflowEditor::__gcc_except_tab DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x231ae5714 -> 0x231ae5816 (0x102) WorkflowEditor::__ustring DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x231ae5818 -> 0x231aedfe8 (0x87d0) WorkflowEditor::__unwind_info DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x231aedfe8 -> 0x231af2160 (0x4178) WorkflowEditor::__eh_frame DefaultSectionSemantics COALESCED - 0.000 seconds
[SharedCache] AddUserSection 0x231af2160 -> 0x231af3487 (0x1327) WorkflowEditor::__objc_classname ReadOnlyDataSectionSemantics CSTRING_LITERALS - 0.219 seconds
[SharedCache] AddUserSection 0x231af3487 -> 0x231b0d18e (0x19d07) WorkflowEditor::__objc_methname ReadOnlyDataSectionSemantics CSTRING_LITERALS - 0.210 seconds
[SharedCache] AddUserSection 0x231b0d18e -> 0x231b13a11 (0x6883) WorkflowEditor::__objc_methtype ReadOnlyDataSectionSemantics CSTRING_LITERALS - 0.207 seconds
[SharedCache] AddUserSection 0x231b13a20 -> 0x231b22000 (0xe5e0) WorkflowEditor::__objc_stubs ReadOnlyCodeSectionSemantics PURE_CODE - 0.226 seconds
[SharedCache] AddUserSection 0x25f9a2bf8 -> 0x25f9a49f8 (0x1e00) WorkflowEditor::__got ReadOnlyDataSectionSemantics REGULAR - 0.209 seconds
[SharedCache] AddUserSection 0x25f9a49f8 -> 0x25f9a5fb8 (0x15c0) WorkflowEditor::__const ReadOnlyDataSectionSemantics REGULAR - 0.207 seconds
[SharedCache] AddUserSection 0x25f9a5fb8 -> 0x25f9a6450 (0x498) WorkflowEditor::__objc_classlist ReadOnlyDataSectionSemantics REGULAR - 0.214 seconds
[SharedCache] AddUserSection 0x25f9a6450 -> 0x25f9a65b0 (0x160) WorkflowEditor::__objc_catlist ReadOnlyDataSectionSemantics REGULAR - 0.226 seconds
[SharedCache] AddUserSection 0x25f9a65b0 -> 0x25f9a6a60 (0x4b0) WorkflowEditor::__objc_protolist ReadOnlyDataSectionSemantics REGULAR - 0.224 seconds
[SharedCache] AddUserSection 0x25f9a6a60 -> 0x25f9a6a68 (0x8) WorkflowEditor::__objc_imageinfo ReadOnlyDataSectionSemantics REGULAR - 0.218 seconds
[SharedCache] AddUserSection 0x25f9a6a68 -> 0x25f9abf00 (0x5498) WorkflowEditor::__objc_selrefs ReadOnlyDataSectionSemantics REGULAR - 0.233 seconds
[SharedCache] AddUserSection 0x25f9abf00 -> 0x25f9ac0e0 (0x1e0) WorkflowEditor::__objc_protorefs ReadOnlyDataSectionSemantics REGULAR - 0.211 seconds
[SharedCache] AddUserSection 0x25f9ac0e0 -> 0x25f9ac280 (0x1a0) WorkflowEditor::__objc_superrefs ReadOnlyDataSectionSemantics REGULAR - 0.215 seconds
[SharedCache] AddUserSection 0x25f9ac280 -> 0x25f9ac308 (0x88) WorkflowEditor::__objc_arraydata ReadOnlyDataSectionSemantics REGULAR - 0.192 seconds
[SharedCache] AddUserSection 0x269782db0 -> 0x269785990 (0x2be0) WorkflowEditor::__auth_got DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x269785990 -> 0x269787868 (0x1ed8) WorkflowEditor::__auth_ptr DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x269787868 -> 0x2697958d8 (0xe070) WorkflowEditor::__const ReadOnlyDataSectionSemantics REGULAR - 0.213 seconds
[SharedCache] AddUserSection 0x2697958d8 -> 0x269796e98 (0x15c0) WorkflowEditor::__cfstring DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x269796e98 -> 0x2697abb88 (0x14cf0) WorkflowEditor::__objc_const DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x2697abb88 -> 0x2697abc48 (0xc0) WorkflowEditor::__objc_intobj DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x2697abc48 -> 0x2697abc90 (0x48) WorkflowEditor::__objc_arrayobj DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x2697abc90 -> 0x2697abce0 (0x50) WorkflowEditor::__objc_dictobj DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x2697abce0 -> 0x2697abd00 (0x20) WorkflowEditor::__objc_floatobj DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x266457290 -> 0x26645bde0 (0x4b50) WorkflowEditor::__objc_data DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x26645bde0 -> 0x266460a30 (0x4c50) WorkflowEditor::__data ReadWriteDataSectionSemantics REGULAR - 0.232 seconds
[SharedCache] AddUserSection 0x263594ff0 -> 0x2635955ec (0x5fc) WorkflowEditor::__objc_ivar DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x2635955f0 -> 0x2635a04f8 (0xaf08) WorkflowEditor::__data ReadWriteDataSectionSemantics REGULAR - 0.210 seconds
[SharedCache] AddUserSection 0x2635a04f8 -> 0x2635a0500 (0x8) WorkflowEditor::__objc_stublist DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x2635a0500 -> 0x2635aec58 (0xe758) WorkflowEditor::__bss ReadWriteDataSectionSemantics ZEROFILL - 0.201 seconds
[SharedCache] AddUserSection 0x2635aec60 -> 0x2635aeec8 (0x268) WorkflowEditor::__common ReadWriteDataSectionSemantics ZEROFILL - 0.237 seconds
[SharedCache] AddUserSection 0x265bfe4d0 -> 0x265bfe610 (0x140) WorkflowEditor::__objc_data DefaultSectionSemantics REGULAR - 0.000 seconds
[SharedCache] AddUserSection 0x265bfe610 -> 0x265bfe620 (0x10) WorkflowEditor::__bss ReadWriteDataSectionSemantics ZEROFILL - 0.238 seconds

@xusheng6 xusheng6 added the State: Awaiting Triage Issue is waiting for more in-depth triage from a developer label Dec 23, 2024
@WeiN76LQh WeiN76LQh mentioned this issue Dec 24, 2024
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bpotchik bpotchik

Labels
State: Awaiting Triage Issue is waiting for more in-depth triage from a developer
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@bpotchik @xusheng6 @WeiN76LQh