Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for running OFRAK on arm64 Linux #562

Open
wants to merge 16 commits into
base: master
Choose a base branch
from
Open

Support for running OFRAK on arm64 Linux #562

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
34bb457
upgrading angr as it uses the newer packages with prebuilt arm binaries
Jepson2k Dec 26, 2024
33663ae
newer capstone with more arm pre-built binaries
Jepson2k Dec 26, 2024
ede6428
gcc10 was phased out for gcc12 in bookworm
Jepson2k Dec 26, 2024
fca3fc0
these packages needed newer versions to work with the newer python
Jepson2k Dec 26, 2024
4db5d21
added multi arch versions as they allow x86 binary introspective test…
Jepson2k Jan 3, 2025
9463331
fixing a couple of small bugs and changing some small details to enab…
Jepson2k Jan 3, 2025
a66c9b1
I upgrade this and then they just released a new version
Jepson2k Jan 3, 2025
cd13de2
Almost 100%...if the bcc toolchain had an arm version or build on arm…
Jepson2k Jan 3, 2025
49f7926
Fix missing newline in Dockerstub
alchzh Jan 7, 2025
e74ee56
Update Makefiles with Jacob's suggestion
alchzh Jan 9, 2025
a78622b
Add >= version specifier to angr requirement for different python ver…
alchzh Jan 9, 2025
0b0a23f
remove incorrect "shell" in subshell
alchzh Jan 9, 2025
9951881
Add separate configuration files for amd64 and arm64 docker images
alchzh Jan 17, 2025
35c79b4
Look in /etc/ofrak for toolchain.conf
alchzh Jan 17, 2025
aa70d01
Copy platform specific Docker toolchain.conf to /etc/ofrak/toolchain.…
alchzh Jan 17, 2025
82c7a54
Revert toolchain.conf to master to avoid breaking existing containers
alchzh Jan 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion disassemblers/ofrak_angr/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
angr==9.2.93
angr==9.2.133
importlib-resources # A workaround for https://github.com/redballoonsecurity/ofrak/issues/398
2 changes: 1 addition & 1 deletion disassemblers/ofrak_capstone/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
capstone==5.0.0.post1
capstone==5.0.3
14 changes: 14 additions & 0 deletions ofrak_core/Dockerstub
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,3 +68,17 @@ RUN cd /tmp && \
make install && \
cd /tmp && \
rm -r UEFITool-A68

# Multiarch packages necessary for various tests to work on ARM
RUN if [ "$TARGETARCH" = "arm64" ]; then \
dpkg --add-architecture amd64 && apt-get update && \
apt-get -y install --no-install-recommends \
libc6:amd64 \
zlib1g:amd64 \
libselinux1:amd64 \
libacl1:amd64 \
libmpc3:amd64 \
libisl23:amd64 \
libstdc++6:amd64 \
crossbuild-essential-amd64; \
fi;
6 changes: 5 additions & 1 deletion ofrak_core/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,11 @@ test: inspect
$(PYTHON) -m pytest -n auto test_ofrak --cov=ofrak --cov-report=term-missing
(sleep 2; echo 1; sleep 2; echo i agree) \
| python3 -m coverage run --append --source ofrak -m ofrak license --force
fun-coverage --cov-fail-under=100
@if [ "$(shell uname -m)" = "x86_64" ]; then \
fun-coverage --cov-fail-under=100; \
else \
fun-coverage; \
fi

ofrak/gui/public:
if [ -d /ofrak_gui ] ; then \
Expand Down
4 changes: 2 additions & 2 deletions ofrak_core/pytest_ofrak/elf/assets/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@

CC=gcc
# Specifying the full name is necessary for some tests to work on ARM
CC=x86_64-linux-gnu-gcc

default: program

Expand Down
2 changes: 1 addition & 1 deletion ofrak_core/pytest_ofrak/elf/fixtures.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@


# MAKEFILE_CONTENTS = """
# CC=gcc
# CC=x86_64-linux-gnu-gcc

# default: program

Expand Down
4 changes: 2 additions & 2 deletions ofrak_core/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ importlib-metadata>=4.13
intervaltree==3.1.0
keystone-engine==0.9.2
jefferson==0.4.5;python_version>="3.8"
lief==0.15.1
orjson~=3.9.15
lief==0.16.1
orjson~=3.10.12
pefile==2023.2.7
pycdlib==1.12.0
python-magic;platform_system!="Windows"
Expand Down
6 changes: 5 additions & 1 deletion ofrak_io/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,8 @@ inspect:
.PHONY: test
test: inspect
$(PYTHON) -m pytest -n auto --cov=ofrak_io --cov-report=term-missing --cov-fail-under=100 ofrak_io_test
fun-coverage --cov-fail-under=100
@if [ "$(shell uname -m)" = "x86_64" ]; then \
fun-coverage --cov-fail-under=100; \
else \
fun-coverage; \
fi
24 changes: 18 additions & 6 deletions ofrak_patch_maker/Dockerstub
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,13 +22,11 @@ RUN cd /tmp && \

#X64-64 toolchain for arm64 Docker images
RUN if [ "$TARGETARCH" = "arm64" ]; then \
apt-get update && apt-get install -y gcc-10-x86-64-linux-gnu; \
apt-get update && apt-get install -y gcc-12-x86-64-linux-gnu; \
fi;

#M68k GNU 10 Linux
#Only exists for x86
RUN if [ "$TARGETARCH" = "amd64" ]; then \
cd /tmp && \
RUN cd /tmp && \
apt-get update && apt-get install -y gcc g++ gperf bison flex texinfo help2man make libncurses5-dev python3-dev autoconf automake libtool libtool-bin gawk wget bzip2 xz-utils unzip patch libstdc++6 rsync && \
git clone https://github.com/crosstool-ng/crosstool-ng.git && \
cd crosstool-ng/ && \
Expand Down Expand Up @@ -81,8 +79,7 @@ RUN if [ "$TARGETARCH" = "amd64" ]; then \
'CT_ZSTD_PATCH_GLOBAL=y' \
>> .config && \
./ct-ng build CT_JOBS=`nproc` && \
cd /tmp && rm -rf crosstool-ng; \
fi;
cd /tmp && rm -rf crosstool-ng;

#M68k VBCC
RUN cd /tmp && \
Expand All @@ -108,6 +105,11 @@ RUN if [ "$TARGETARCH" = "amd64" ]; then \
wget https://armkeil.blob.core.windows.net/developer/Files/downloads/gnu-a/10.3-2021.07/binrel/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu.tar.xz --show-progress --progress=bar:force:noscroll && \
tar xf gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu.tar.xz -C /opt/rbs/toolchain && \
rm -rf gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu.tar.xz; \
elif [ "$TARGETARCH" = "arm64" ]; then \
cd /tmp && \
wget https://developer.arm.com/-/media/Files/downloads/gnu-a/10.3-2021.07/binrel/gcc-arm-10.3-2021.07-aarch64-aarch64-none-elf.tar.xz --show-progress --progress=bar:force:noscroll && \
tar xf gcc-arm-10.3-2021.07-aarch64-aarch64-none-elf.tar.xz -C /opt/rbs/toolchain && \
rm -rf gcc-arm-10.3-2021.07-aarch64-aarch64-none-elf.tar.xz; \
fi;

#AVR GCC
Expand All @@ -116,6 +118,11 @@ RUN if [ "$TARGETARCH" = "amd64" ]; then \
wget https://ww1.microchip.com/downloads/aemDocuments/documents/OTH/ProductDocuments/SoftwareLibraries/Firmware/avr8-gnu-toolchain-3.6.2.1778-linux.any.x86_64.tar.gz --show-progress --progress=bar:force:noscroll && \
tar xzf avr8-gnu-toolchain-3.6.2.1778-linux.any.x86_64.tar.gz -C /opt/rbs/toolchain && \
rm -rf avr8-gnu-toolchain-3.6.2.1778-linux.any.x86_64.tar.gz; \
elif [ "$TARGETARCH" = "arm64" ]; then \
cd /tmp && \
wget http://downloads.arduino.cc/tools/avr-gcc-7.3.0-atmel3.6.1-arduino7-aarch64-pc-linux-gnu.tar.bz2 --show-progress --progress=bar:force:noscroll && \
tar xf avr-gcc-7.3.0-atmel3.6.1-arduino7-aarch64-pc-linux-gnu.tar.bz2 -C /opt/rbs/toolchain && \
rm -rf avr-gcc-7.3.0-atmel3.6.1-arduino7-aarch64-pc-linux-gnu.tar.bz2; \
fi;

#PPC GNU 10 Linux
Expand All @@ -124,6 +131,11 @@ RUN if [ "$TARGETARCH" = "amd64" ]; then \
wget https://download.01.org/0day-ci/cross-package/gcc-10.3.0-nolibc/x86_64-gcc-10.3.0-nolibc_powerpc-linux.tar.xz --show-progress --progress=bar:force:noscroll && \
tar xf x86_64-gcc-10.3.0-nolibc_powerpc-linux.tar.xz -C /opt/rbs/toolchain && \
rm -rf x86_64-gcc-10.3.0-nolibc_powerpc-linux.tar.xz; \
elif [ "$TARGETARCH" = "arm64" ]; then \
cd /tmp && \
wget https://www.kernel.org/pub/tools/crosstool/files/bin/arm64/10.3.0/arm64-gcc-10.3.0-nolibc-powerpc-linux.tar.xz --show-progress --progress=bar:force:noscroll && \
tar xf arm64-gcc-10.3.0-nolibc-powerpc-linux.tar.xz -C /opt/rbs/toolchain && \
rm -rf arm64-gcc-10.3.0-nolibc-powerpc-linux.tar.xz; \
fi;

#BCC (GCC) SPARC v8
Expand Down
6 changes: 5 additions & 1 deletion ofrak_patch_maker/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,4 +17,8 @@ inspect:
.PHONY: test
test: inspect
$(PYTHON) -m pytest -n auto --cov=ofrak_patch_maker --cov-report=term-missing ofrak_patch_maker_test
fun-coverage --cov-fail-under=100
@if [ "$(shell uname -m)" = "x86_64" ]; then \
fun-coverage --cov-fail-under=100; \
else \
fun-coverage; \
fi
24 changes: 12 additions & 12 deletions ofrak_patch_maker/ofrak_patch_maker/toolchain.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,18 +42,18 @@ BIN_PARSER = /opt/rbs/toolchain/gcc-10.3.0-nolibc/x86_64-linux/bin/x86_64-linux-
LIB = /opt/rbs/toolchain/gcc-10.3.0-nolibc/x86_64-linux/lib

[GNU_AARCH64_LINUX_10]
PREPROCESSOR = /opt/rbs/toolchain/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu-gcc
COMPILER = /opt/rbs/toolchain/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu-gcc
LINKER = /opt/rbs/toolchain/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu-ld
BIN_PARSER = /opt/rbs/toolchain/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu-objdump
LIB = /opt/rbs/toolchain/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/lib
PREPROCESSOR = /opt/rbs/toolchain/gcc-arm-10.3-2021.07-*-aarch64-none-*/bin/aarch64-none-*-gcc
COMPILER = /opt/rbs/toolchain/gcc-arm-10.3-2021.07-*-aarch64-none-*/bin/aarch64-none-*-gcc
LINKER = /opt/rbs/toolchain/gcc-arm-10.3-2021.07-*-aarch64-none-*/bin/aarch64-none-*-ld
BIN_PARSER = /opt/rbs/toolchain/gcc-arm-10.3-2021.07-*-aarch64-none-*/bin/aarch64-none-*-objdump
LIB = /opt/rbs/toolchain/gcc-arm-10.3-2021.07-*-aarch64-none-*/lib

[GNU_AVR_5]
PREPROCESSOR = /opt/rbs/toolchain/avr8-gnu-toolchain-linux_x86_64/bin/avr-gcc
COMPILER = /opt/rbs/toolchain/avr8-gnu-toolchain-linux_x86_64/bin/avr-gcc
LINKER = /opt/rbs/toolchain/avr8-gnu-toolchain-linux_x86_64/bin/avr-ld
BIN_PARSER = /opt/rbs/toolchain/avr8-gnu-toolchain-linux_x86_64/bin/avr-objdump
LIB = /opt/rbs/toolchain/avr8-gnu-toolchain-linux_x86_64/lib
PREPROCESSOR = /opt/rbs/toolchain/avr*/bin/avr-gcc
COMPILER = /opt/rbs/toolchain/avr*/bin/avr-gcc
LINKER = /opt/rbs/toolchain/avr*/bin/avr-ld
BIN_PARSER = /opt/rbs/toolchain/avr*/bin/avr-objdump
LIB = /opt/rbs/toolchain/avr*/lib

[GNU_PPC_LINUX_10]
PREPROCESSOR = /opt/rbs/toolchain/gcc-10.3.0-nolibc/powerpc-linux/bin/powerpc-linux-gcc
Expand All @@ -73,7 +73,7 @@ LIB = /opt/rbs/toolchain/bcc-2.0.7-gcc/lib
ARM_ASM_PATH = /opt/rbs/toolchain/gcc-arm-none-eabi-10-2020-q4-major/bin/arm-none-eabi-as
X86_64_ASM_PATH = /opt/rbs/toolchain/gcc-10.3.0-nolibc/x86_64-linux/bin/x86_64-linux-as
M68K_ASM_PATH = /opt/rbs/toolchain/m68k-unknown-linux-gnu/bin/m68k-unknown-linux-gnu-as
AARCH64_ASM_PATH = /opt/rbs/toolchain/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu-as
AVR_ASM_PATH = /opt/rbs/toolchain/avr8-gnu-toolchain-linux_x86_64/bin/avr-as
AARCH64_ASM_PATH = /opt/rbs/toolchain/gcc-arm-10.3-2021.07-*-aarch64-none-*/bin/aarch64-none-*-as
AVR_ASM_PATH = /opt/rbs/toolchain/avr*/bin/avr-as
PPC_ASM_PATH = /opt/rbs/toolchain/gcc-10.3.0-nolibc/powerpc-linux/bin/powerpc-linux-as
SPARC_ASM_PATH = /opt/rbs/toolchain/bcc-2.0.7-gcc/bin/sparc-gaisler-elf-as
17 changes: 14 additions & 3 deletions ofrak_patch_maker/ofrak_patch_maker/toolchain/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import configparser
import os
import glob
from multiprocessing import Pool, cpu_count
from typing import Optional, Dict, Mapping, Tuple

Expand Down Expand Up @@ -28,6 +29,8 @@ def get_repository_config(section: str, key: Optional[str] = None):
"""
Get config values from toolchain.conf.

If the resulting value contains '*', we do a glob expansion.

:param section: section name in config file
:param key: key in `config[section]`

Expand All @@ -48,10 +51,18 @@ def get_repository_config(section: str, key: Optional[str] = None):
try:
config.read(conf)
if key:
ret = config.get(section, key)
raw_value = config.get(section, key)

if "*" in raw_value:
matches = glob.glob(raw_value)
if matches:
return matches[0]
else:
raise NotFoundError(f"No file matches wildcard {raw_value}")
else:
return raw_value
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't feel very positively about this change to use globbing. I personally feel that it makes the config file harder to read and understand. It also seems kind of error-prone for anyone not running OFRAK inside of Docker, who may have their own toolchains installed on their system.

@Jepson2k @alchzh what alternatives were considered here instead of globbing?

else:
ret = config.items(section) # type: ignore
return ret
return config.items(section) # type: ignore
except (configparser.NoSectionError, configparser.NoOptionError) as e:
error_by_config_file[conf] = e
continue
Expand Down
4 changes: 4 additions & 0 deletions ofrak_patch_maker/ofrak_patch_maker_test/test_sparc_toolchain.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import pytest
import platform

from ofrak_patch_maker.toolchain.gnu_bcc_sparc import GNU_BCC_SPARC_Toolchain
from ofrak_patch_maker_test import ToolchainUnderTest
Expand Down Expand Up @@ -28,14 +29,17 @@ def toolchain_under_test(request) -> ToolchainUnderTest:
return request.param


@pytest.mark.skipif(platform.machine() != "x86_64", reason="Test only supported on x86_64")
def test_monkey_patch(toolchain_under_test: ToolchainUnderTest):
run_monkey_patch_test(toolchain_under_test)


# C Tests
@pytest.mark.skipif(platform.machine() != "x86_64", reason="Test only supported on x86_64")
def test_bounds_check(toolchain_under_test: ToolchainUnderTest):
run_bounds_check_test(toolchain_under_test)


@pytest.mark.skipif(platform.machine() != "x86_64", reason="Test only supported on x86_64")
def test_hello_world(toolchain_under_test: ToolchainUnderTest):
run_hello_world_test(toolchain_under_test)
6 changes: 5 additions & 1 deletion ofrak_tutorial/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,11 @@ test: inspect
$(PYTHON) -m pytest -n auto ofrak_tutorial_test
# Note: this requires the Ghidra server to be up and running
$(PYTHON) -m pytest --nbval --sanitize-with=nbval_sanitizer.cfg --cov=ofrak_tutorial --cov-report=term-missing --cov-fail-under=100 notebooks_with_outputs
fun-coverage --cov-fail-under=100
@if [ "$(shell uname -m)" = "x86_64" ]; then \
fun-coverage --cov-fail-under=100; \
else \
fun-coverage; \
fi

.PHONY: generate_stripped_notebooks
generate_stripped_notebooks:
Expand Down
6 changes: 5 additions & 1 deletion ofrak_type/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,8 @@ inspect:
.PHONY: test
test: inspect
$(PYTHON) -m pytest -n auto --cov=ofrak_type --cov-report=term-missing --cov-fail-under=100 ofrak_type_test
fun-coverage --cov-fail-under=100
@if [ "$(shell uname -m)" = "x86_64" ]; then \
fun-coverage --cov-fail-under=100; \
else \
fun-coverage; \
fi
Loading