Add module S28_python_run

[B1TC0R3]

• What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)
  Feature.

• What is the current behavior? (You can also link to an open issue here)
  EMBA cannot run Python scripts natively as modules.

• What is the new behavior (if this is a feature change)? If possible add a screenshot.
  This PR adds a new module ("S28_python_run") with the capability of running user-supplied python scripts as
  modules. Related to issue Python runner module #1264.

• Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)
  No. The module S28 is disabled in the default scan profile (default-scan.emba).
  In addition, when no Python scripts are manually specified, it will start and terminate without doing anything.

• Other information:

1. The changes run in strict mode (-S).
2. check_project.sh does not detect issues with the new code.
3. shell_check does not find errors with the new code.
4. A shell_check exception has been added in S28_python_run.sh to conform with EMBA coding standards. The exception allows the use of quotation marks in arithmetic operations. (See line 39 in S28_python_run.sh)
5. All Python code has been verified using pyright.

[github-actions]

Thank you for contributing a pull request!

Welcome to the EMBA firmware analysis community!

We are glad you are here and appreciate your contribution. Please keep in mind our contributing guidelines here and here.
Also, please check existing open issues and consider to open a discussion in the dedicated discussion area.
Additionally, we have collected a lot of details around EMBA, the installation and the usage of EMBA in our Wiki.

If you like EMBA you have the chance to support us by becoming a Sponsor or buying some beer here.

To show your love for EMBA with nice shirts or other merch you can check our Spreadshop.

This is an automatic message. Allow for time for the EMBA community to be able to read the pull request and comment on it.

[m-1-k-3]

I am currently on vacation and will come back to this PR soon.

[B1TC0R3]

Understood.
Enjoy your time off! :)

[m-1-k-3]

If someone can start with some initial rewiew and testing it would be highly appreciate.

[BenediktMKuehne]

For testing: where/how do I install pip packages?( does the venv work with the scripts)

[m-1-k-3]

I did a quick code review of the bash code. Please check my notes.
If we are going to include python code into EMBA we will use the same actions as for EMBArk (pylint/bandit/pycodestyle). This means, we need to setup further github actions and include them also in the checker script.

I am not completely sure how we should handle the logging and we need to link to the results from the python modules.

Finally, we need a way to setup a virtual environment dynamically for the container to run the python scripts. Probably we can create a virtual environment in the external directory and mount it via docker-compose into the container.

As there are currently some questions, I would recomment to draft this PR for now to work on these topics.

Good work and I think this will results in cool stuff for EMBA. Keep on going :)

[m-1-k-3]

We need to check this in the dependency checker. I also think this find is not needed as the installer should have already installed python3 in path. So, at the end we can do something like this here

emba/helpers/helpers_emba_dependency_check.sh

Line 709 in d829e18

check_dep_tool "ssdeep"

[m-1-k-3]

I think I was missing somthing like the following:

local lSCRIPT=""

Why ist the PYTHON_SCRIPTS array not local?
Please rename it to PYTHON_SCRIPTS_ARR

[B1TC0R3]

Considering the PYTHON_SCRIPTS array works essentially the same as the SELECT_MODULES array in an EMBA profile, should I still rename it?

[m-1-k-3]

we can do at least something like the following (unsure on the lRESULTS counting):

                lCOUNT_FINDINGS=$((lCOUNT_FINDINGS + "${#lRESULTS[@]}"))

lRESULTS is an array, please rename it the following lRESULTS_ARR

[B1TC0R3]

I will begin work on the requested changes.
Since this goes out of scope of my Bachelors Thesis, it might take a bit longer tho as I will be working on it in my free time.
Initially, I did not plan on adding any Python modules to EMBA permanently, just to implement an easy method
to run custom ones when required, but I will gladly expand the module to support this properly.
All minor changes (colorization, name changes, ...) should not take very long.

I will also update the copyright notices accordingly, I did not change them due to a misunderstanding on my end.

Please feel free to contact me for any additional questions or requirements.

---

To clarify some things:

For testing: where/how do I install pip packages?( does the venv work with the scripts)

As it is currently, the module uses EMBAs venv. PIP packages should work when installed into this environment.
However, I have not tested this. Will confirm that it works.
I also do not know whether this venv even should be used, or whether I need to create a custom one
for this module. That decision is up to you.

Why ist the PYTHON_SCRIPTS array not local?

The PYTHON_SCRIPTS array is not local because it is set in the EMBA profile.
It's supposed to allow a user to specify what Python scripts they want to run.
I can add a local version of the array with default settings, which can then be overwritten by the profile setting.
Would be similar to the SELECT_MODULES arrays functionality.

I am not completely sure how we should handle the logging and we need to link to the results from the python modules.

Right now, the Bash module reads the results from STDOUT of the Python script (each line = 1 results).
An alternative solution would be to use grep (or a similar tool) to parse the results from the Python scripts own log file.
This would require some formatting changes in the embamodule.py output to make it grep-able, but
should not take too much work.

Finally, we need a way to setup a virtual environment dynamically for the container to run the python scripts. Probably we can create a virtual environment in the external directory and mount it via docker-compose into the container.

Instead of using EMBAs venv, this might be possible. I will look into it.

[m-1-k-3]

Since this goes out of scope of my Bachelors Thesis, it might take a bit longer tho as I will be working on it in my free time. Initially, I did not plan on adding any Python modules to EMBA permanently, just to implement an easy method to run custom ones when required, but I will gladly expand the module to support this properly.

No hurry with this. I think this could be a very interesting addon for EMBA. Nevertheless, we need to design it in a way it will also fulfill the expectations into a python integration for EMBA.

[B1TC0R3]

As the next big change I want to change the logging method around.

As mentioned before, I plan on making the output from the python module grep-able, so it is easier
to determine what is a finding, what is just debugging output and what is an error.
The goal is to add a better result counter that does not have to rely on the amount of lines output by a Python module.

I could either implement this by reading STDOUT from each Python module directly like I am doing at the moment, or
by parsing the module specific log file after it has finished.

Is there a preferred method to go about this?

[m-1-k-3]

As the next big change I want to change the logging method around.

As mentioned before, I plan on making the output from the python module grep-able, so it is easier to determine what is a finding, what is just debugging output and what is an error. The goal is to add a better result counter that does not have to rely on the amount of lines output by a Python module.

Probably it makes sense to log from the python modules as from the main EMBA modules. All relevant output into a text file that we can then link from the main EMBA s28 module. With this we should be able to easily generate the html report page. I think this could be around line 37 of your module. There we will need a write_link after your print_output. This links to the original log file from your python module.

The grep-able thing is additionally a useful mechanism.

I could either implement this by reading STDOUT from each Python module directly like I am doing at the moment, or by parsing the module specific log file after it has finished.

Is there a preferred method to go about this?

See my notes before. I would recommend to log all relevant details in a nice way to a dedicated log file and link to this log file via write_link from the main s28 module.

[B1TC0R3]

That makes sense, I did not know about the write_link function.

[m-1-k-3]

That makes sense, I did not know about the write_link function.

The web reporter automatically picks it up and is doing the html stuff for you ... hopefully ;) You should see a link with the [REF] marker in the text report. See also here

emba/helpers/helpers_emba_print.sh

Line 401 in cee27c5

write_link()

[B1TC0R3]

I majorly screwed up my git history just now, which caused this pull request to close. :,)
All the changes are still backed up locally, so that is not too much of an issue.
I will see that I reopen a new pull request/issue as soon as I can, tho I might have to create a new fork of the repository.