Skip to content

Commit

Permalink
New build system
Browse files Browse the repository at this point in the history
  • Loading branch information
@SeekyCt
SeekyCt committed Aug 28, 2024
1 parent 67392b5 commit 330bb9f
Show file tree
Hide file tree
Showing 70 changed files with 77,059 additions and 11,883 deletions.
13 changes: 13 additions & 0 deletions .gitattributes
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Auto detect text files and perform LF normalization
* text=auto

# Explicitly declare text files
*.py text

# Enforce platform-specific encodings
*.bat text eol=crlf
*.sh text eol=lf
*.sha1 text eol=lf

# decomp-toolkit writes files with LF
config/**/*.txt text eol=lf
82 changes: 45 additions & 37 deletions .github/workflows/build.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,55 +6,63 @@ on:

jobs:
build:
container: ghcr.io/seekyct/spm-docker-dtk:main

runs-on: ubuntu-latest
container: ghcr.io/seekyct/spm-docker:v1
strategy:
fail-fast: false
matrix:
version: [eu0]
version: [EU0]

steps:
# Checkout the repository (shallow clone)
- name: Checkout
uses: actions/checkout@v3
uses: actions/checkout@v4
with:
fetch-depth: 0
submodules: recursive

# Set Git config
- name: Git config
run: git config --global --add safe.directory "$GITHUB_WORKSPACE"

# Copy the original files to the workspace
- name: Prepare
run: |
cp /spmfiles/${{matrix.version}}/main.dol orig
cp /spmfiles/${{matrix.version}}/relF.rel orig
cp -R /spmfiles/4199_60831 tools
run: cp -R /orig .

# Build the project
- name: Build
run: |
python3 configure.py --wine wibo
ninja
- name: Upload maps
uses: actions/upload-artifact@v3
with:
name: maps-${{matrix.version}}
path: |
out/main.map
out/relF.map
- name: Upload analysis
uses: actions/upload-artifact@v3
with:
name: analysis-${{matrix.version}}
path: |
build/labels.pickle
build/relocs.pickle
build/rel_labels.pickle
build/rel_relocs.pickle
- name: Upload asm lists
uses: actions/upload-artifact@v3
python configure.py --map --version ${{ matrix.version }} \
--binutils /binutils --compilers /compilers
ninja all_source build/${{ matrix.version }}/progress.json \
build/${{ matrix.version }}/report.json
# # Upload progress if we're on the main branch
# - name: Upload progress
# # If you're using a different branch, change this to match
# if: github.ref == 'refs/heads/main'
# continue-on-error: true
# env:
# # Replace with your project slug
# PROGRESS_SLUG: prime
# # Set the API key in your repository secrets
# PROGRESS_API_KEY: ${{ secrets.PROGRESS_API_KEY }}
# run: |
# python tools/upload_progress.py -b https://progress.decomp.club/ \
# -p $PROGRESS_SLUG -v ${{ matrix.version }} \
# build/${{ matrix.version }}/progress.json

# Upload map files
- name: Upload map
uses: actions/upload-artifact@v4
with:
name: asm-list-${{matrix.version}}
path: |
build/main.dol.asml
build/relF.rel.asml
- name: Calculate progress
run: python3 progress.py --full > build/progress.txt
- name: Upload progrses
uses: actions/upload-artifact@v3
name: ${{ matrix.version }}_maps
path: build/${{ matrix.version }}/**/*.MAP

# Upload progress report
- name: Upload report
uses: actions/upload-artifact@v4
with:
name: progress-${{matrix.version}}
path: build/progress.txt
name: ${{ matrix.version }}_report
path: build/${{ matrix.version }}/report.json
13 changes: 7 additions & 6 deletions .gitignore
View file
Original file line number Diff line number Diff line change
@@ -1,15 +1,16 @@
__pycache__
.idea
.vscode
assets
.ninja_*
.mypy_cache
build
expected
out
*.dol
*.rel
relF.bin
*.exe
*.dll
build.ninja
objdiff.json
orig/*/*
!orig/*/.gitkeep
/*.txt
ctx.c
*.m2c
*.bat
3 changes: 0 additions & 3 deletions .gitmodules
View file

This file was deleted.

53 changes: 53 additions & 0 deletions .vscode.example/c_cpp_properties.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
{
"configurations": [
{
"name": "Linux",
"includePath": [
"${workspaceFolder}/spm-headers/include",
"${workspaceFolder}/spm-headers/decomp",
"${workspaceFolder}/build/EU0/include"
],
"defines": [
"DECOMP",
"SKIP_PPCDIS",
"SPM_EU0"
],
"cStandard": "c99",
"cppStandard": "c++98",
"intelliSenseMode": "linux-clang-x86",
"compilerPath": "",
"configurationProvider": "ms-vscode.makefile-tools",
"browse": {
"path": [
"${workspaceFolder}/spm-headers/include"
],
"limitSymbolsToIncludedHeaders": true
}
},
{
"name": "Win32",
"includePath": [
"${workspaceFolder}/spm-headers/include",
"${workspaceFolder}/spm-headers/decomp",
"${workspaceFolder}/build/EU0/include"
],
"defines": [
"DECOMP",
"SKIP_PPCDIS",
"SPM_EU0"
],
"cStandard": "c99",
"cppStandard": "c++98",
"intelliSenseMode": "windows-clang-x64",
"compilerPath": "",
"configurationProvider": "ms-vscode.makefile-tools",
"browse": {
"path": [
"${workspaceFolder}/spm-headers/include"
],
"limitSymbolsToIncludedHeaders": true
}
}
],
"version": 4
}
24 changes: 24 additions & 0 deletions .vscode.example/settings.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
{
"[c]": {
"files.encoding": "utf8"
},
"[cpp]": {
"files.encoding": "utf8"
},
"files.insertFinalNewline": true,
"files.trimFinalNewlines": true,
"files.associations": {
"*.inc": "cpp",
"*.h": "cpp",
"*.c": "cpp"
},
"search.useIgnoreFiles": false,
"search.exclude": {
"build/*/config.json": true,
"build/**/*.MAP": true,
"build.ninja": true,
".ninja_*": true,
"objdiff.json": true,
".mypy_cache": true
}
}
80 changes: 21 additions & 59 deletions CONTRIBUTING.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,82 +2,44 @@

If you're interested in contributing, it's recommended you reach out to us in the #reverse-engineering channel of the [SPM Speedrunning Discord](https://discord.gg/dbd733H) first. Here you can ask for any help you might need and discuss what you plan to decompile to prevent any duplicated work.

# ppcdis
# dtk

This project is build around the [ppcdis](https://github.com/SeekyCt/ppcdis) tools. These handle generation of assembly code (**you should never edit the assembly output manually**) and helpers for allowing partially-decompiled C files to match.
This project is build around encounter's [decomp-toolkit](https://github.com/encounter/decomp-toolkit).

## YMLs
## Configuration

### Overrides
### config.yml

Overrides allow hints to be given to the analysis / disassembly to improve accuracy or work around issues. Generally you won't need to change anything about analysis (`config/analysis_overrides.yml`), though changing the disassembly is more common (`config/analysis_overrides.yml`).
This file contains base properties for the project. Generally the only time you should need to edit this file is to add assets to the `extract` section.

See also the [ppcdis documentation of this](https://github.com/SeekyCt/ppcdis/blob/main/TOOLS.md#analyser-overrides).
See also the [dtk-template documentation](https://github.com/encounter/dtk-template/blob/main/config/GAMEID/config.example.yml) of this.

### Slices
### splits.txt

`config/dol_slices.yml` / `config/rel_slices.yml` control how the disassembly is split around the decompiled C code for the dol / rel.
See also the [ppcdis documentation of this](https://github.com/SeekyCt/ppcdis/blob/main/TOOLS.md#slice-yml-format)
This file controls the splitting of the game's address space into source files. The address range of a file in each of its sections needs to be added here before it can be decompiled.

**You must re-run configure.py after changing this file**
Each file added here should also have an entry in configure.py's `config.libs`.

### Symbols
See also the [dtk-template documentation](https://github.com/encounter/dtk-template/blob/main/docs/splits.md) of this.

`config/symbols.yml` controls the naming of functions and data in disassembly.
### symbols.txt

See also the [ppcdis documentation of this](https://github.com/SeekyCt/ppcdis/blob/main/TOOLS.md#symbol-map)
This file controls the naming of functions and data in disassembly.

## Generated includes
See also the [dtk-template documentation](https://github.com/encounter/dtk-template/blob/main/docs/symbols.md) of this

These can be generated by adding the include statement to a C file and re-running configure.py.
## Viewing Original Assembly

### Orderstrings
dtk will output the assembly of functions to `build/EU0/asm` for manual inspection. For files defined in splits.txt, there will be a file containing all contents of the file. For non-split functions, there will be an individual file for each one.

`#include orderstrings/XXXXXXXX_YYYYYYYY.inc` will generate a workaround for the ordering of strings in a file, with strings starting at address XXXXXXXX (inclusive) and ending at address YYYYYYYY (exclusive).
# objdiff

See also the [ppcdis documentation of this](https://github.com/SeekyCt/ppcdis/blob/main/TOOLS.md#orderstrings)
A setup for objdiff is included as the main way of checking whether functions match, and the progress of files in general.

### Orderfloats
Download the latest release from [encounter/objdiff](https://github.com/encounter/objdiff). Under project settings, set `Project directory`. The configuration should be loaded automatically.

`#include orderfloats/XXXXXXXX_YYYYYYYY.inc` will generate a workaround for the ordering of floats in a file, with floats starting at address XXXXXXXX (inclusive) and ending at address YYYYYYYY (exclusive).
`#include orderfloatsm/XXXXXXXX_YYYYYYYY.inc` will generate a 'manual' version of that (see documentation below).
Select an object from the left sidebar to begin diffing. Changes to the project will rebuild automatically: changes to source files, headers, `configure.py`, `splits.txt` or `symbols.txt`.

`#include orderdoubles/XXXXXXXX_YYYYYYYY.inc` will generate a version of that for doubles. 'Manual' doubles are not needed for this project.
# Visual Studio Code

See also the [ppcdis documentation of this](https://github.com/SeekyCt/ppcdis/blob/main/TOOLS.md#orderfloats)

### Jumptable

`#include jumptable/XXXXXXXX.inc` will generate a workaround for an undecompiled jumptable at address XXXXXXXX.

See also the [ppcdis documentation of this](https://github.com/SeekyCt/ppcdis/blob/main/TOOLS.md#c-jumptable-generation)

### Asm

`#include asm/XXXXXXXX.s` will generate inline assembly for an undecompiled function.

See also the [ppcdis documentation of this](https://github.com/SeekyCt/ppcdis/blob/main/TOOLS.md#inline-function-generation)

# diff.py

A setup for [asm-differ](https://github.com/simonlindholm/asm-differ/blob/main/diff.py) is included. To use it:
- First get a matching build and run expected.py
- If you use the `-o` flag (see below), this will need to be re-ran whenever you add new source files
- You can now run `diff.py symbolName` to diff a function
- It's recommended to run it with the flags `-mwos`, see `-h` for details on these and the other flags available
- If you want to diff a function in the rel, you'll need to add the `-r` flag

# Project-Specific Tools

## decompme.py

Generates a scratch on [decomp.me](https://decomp.me/) for a function. This gives a web-based UI for editing & diffing a function, and the ability to share it with other people. Usage is `decompme.py XXXXXXXX` to upload the function at XXXXXXXX.
By default, an M2C decompiler compatible context is made, which strips definitions and macros that you have wanted included. You can replace this with a context from makectx.py (see below) after the decompiler runs if you'd like.

## getfunction.py

Gets the assembly of a single function. Usage is `getfunction.py XXXXXXXX` to get the function at XXXXXXXX.

## makectx(_m2c).py

Creates a context of all header files in the project concatenated and prints it to stdout. If using makectx_m2c, this will have the C preprocessor ran on it.
If desired, use the recommended Visual Studio Code settings by renaming the `.vscode.example` directory to `.vscode`.
Loading

0 comments on commit 330bb9f

Please sign in to comment.