Plugin development

Plug-In Development

Unpacking is done recursevly layer by layer. The unpacker(-plug-in) for each individual file is selected by it's mime-type. If no unpacker is available a generic file carver is used as fallback.

Write an Unpacking Plug-In

FACT detects plug-ins automatically as long as they are stored in src/plugins/unpacking.

A plug-in consists of folders and files following the following template.

.
├── __init__.py
├── install.sh [OPTIONAL]
├── code
│   ├── __init__.py
│   └── PLUGIN_NAME.py
├── internal [OPTIONAL]
│   └── ADDITIONAL_SOURCES_OR_CODE
└── test
    ├── __init__.py
    ├── test_PLUGIN_NAME.py
    └── data [OPTIONAL]
        └── SOME DATA FILES TO TEST

Each file is described below.

./install.sh

install.sh is an optional file that can provide additional bash code that should be run on install. install.sh is automatically triggered by FACT's installation script.

A basic install.sh shall look like this.

#!/usr/bin/env bash

# change cwd to current file's directory
cd "$( dirname "${BASH_SOURCE[0]}" )" 

echo "------------------------------------"
echo "       SOME MEANINGFUL TITLE        "
echo "------------------------------------"

[YOUR CODE HERE]
	
exit 0

❗ Do not forget the "exit 0" at the end! If you miss it and something goes wrong, the installer will end-up in an infinite loop!

./code/PLUGIN_NAME.py

This is the actual plug-in code. You should use the following self explanatory template to write your plug-in.

'''
This plugin unpacks CONTAINER_NAME files.
'''
 
NAME = 'PLUGIN_NAME'
MIME_PATTERNS = ['MIME_PATTERN_1', 'MIME_PATTERN_2', ...]
VERSION = 'x.x'


def unpack_function(file_path, tmp_dir):
    '''
    file_path specifies the input file.
    tmp_dir must be used to store the extracted files.
    Optional: Return a dict with meta information
    '''
    INSERT YOUR UNPACKING CODE HERE

    return META_DICT


# ----> Do not edit below this line <----
def setup(unpack_tool):
    for item in MIME_PATTERNS:
        unpack_tool.register_plugin(item, (unpack_function, NAME, VERSION))

👍 Hint: An unpacking plug-in can provide several unpackers. This means you can add several PLUGIN_NAME.py files in the ./code/ directory.
👍 Hint: If you need to call an external program, you can use common_helper_process.execute_shell_command(COMMAND). It will return STDOUT and STDERR as a combined string and it will not raise an exception if the return code of COMMAND is not 0.

./internal

The optional folder internal can provide additional code or other stuff needed for the plug-in.
You can use the following code to get the absolute path of the internal directory:

from pathlib import Path

INTERNAL_DIR = Path(__file__).parent.parent / 'internal'

If you want to load python libraries/files from the internal directory you can use the following code.

import sys
from pathlib import Path

INTERNAL_DIR = Path(__file__).parent.parent / 'internal'
sys.path.append(str(INTERNAL_DIR))

from YOUR_PYTHON_FILE_IN_INTERNAL_DIRECTORY import WHATEVER

./test/test_PLUGIN_NAME.py

This file includes your test code.

./test/data

This folder shall contain any additional files, that are needed for your tests. You can address this folder using the following code in your test code file.

from pathlib import Path

TEST_DATA_DIR = Path(__file__).parent / 'data'

Special Cases

Encoding Overhead

Some unpackers decrease the file size of the resulting files (e.g. base64 decoder). Therefore, you can specify an "encoding_overhead" in your META_DICT to fix a false classification by the unpacking classifier.

Example: Base 64 has an encoding overhead of 33%

META_DICT['encoding_overhead'] = 0.33

Writing Mime-Type Definitions

It might be necessary to write your own mime-type definition. These should be added to src/mime/ (This is not part of the plug-in at the moment ❗).
Files in src/mime/ are compiled during the bootstrap procedure. You can use the following command to compile a magic file manually for testing: file -C -m custommime Afterwards you can check your new definition with: file -m custommime.mgc --mime-type FILE_PATH

A simple mime/file definition can look like this:

# Some comment describing your definition
0    string   \x00\x01\x02\x03   some text chown when file is used without --mime-type
!:mime MIME/TYPE

This signature matches all files with hex string "\x00\x01\x02\x03" at offset "0". Have a look at man file to learn about creating more complex signatures.

Your newly created mime-types should hold the following naming conditions:

Category	MIME schema
Firmware Container	firmware/NAME
File System	filesystem/NAME
Compression Streams	compression/NAME

👍 Hint: In many cases a file signature is already present in the official file repository, but no mime Type is defined. You could copy the signature definition and add a mime definition in this case.