Migrate to Python 3 (#211)

* Attempt to fix up gitbook samples. can't run tests since I'm on windows * A bit of py3k work * py3k broke into my house and ate my children * Fix the apidoc warnings filter * oh jesus * Update usage of clemory * okay. here we go * Update autodoc options for the greater good * *snaps fingers angrily* * There's that migration document * VERY important clarification * tick * Be CONSISTENT * tock * Destroy simuvex * Mention the deprecation removals * Attempt to fix up gitbook samples. can't run tests since I'm on windows * A bit of py3k work * py3k broke into my house and ate my children * Fix the apidoc warnings filter * oh jesus * Update usage of clemory * okay. here we go * Update autodoc options for the greater good * *snaps fingers angrily* * There's that migration document * VERY important clarification * tick * Be CONSISTENT * tock * Destroy simuvex * Mention the deprecation removals * Assume sphinx output is utf-8 so we can print the escape codes * Strip custom_ prefix from CLE arguments which aren't really all that custom * Add changelog entry, without finalized version number * HEY DUMBASS: DO NOT DO THAT * Add symbols_by_addr refactor information * Give install instrucitons a facelift, nuke the windows-install stuff * Un-deprecate simgr * Fix markdown style of migration guide link (#214) * CFGAccurate -> CFGEmulated (Do not lie to the users) * Do NOT lie to the users!!! * Update code samples for whatever * Add gym and arm_spotter to api-doc * Note the removal of immutable simgr
angr · Oct 1, 2018 · 139150e · 139150e
1 parent 707b7ed
commit 139150e
Show file tree

Hide file tree

Showing 94 changed files with 1,235 additions and 1,109 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -3,6 +3,22 @@
 This lists the *major* changes in angr.
 Tracking minor changes are left as an exercise for the reader :-)
 
+## angr 8.X.X.X
+
+Welcome to angr 8!
+The biggest change for this major version bump is the transition to python 3.
+You can read about this, as well as a few other breaking changes, in the [migration guide](MIGRATION.md).
+
+- Switch to python 3
+- Refactor to Clemory to clean up the API and speed things up drastically
+- Remove `object.symbols_by_addr` (dict) and add `object.symbols` (sorted list); add `fuzzy` parameter to `loader.find_symbol`
+- CFGFast is much, much faster now. CFGAccurate has been renamed to CFGEmulated.
+- Support for avx2 unpack instructions, courtesy of D. J. Bernstein
+- Removed support for immutable simulation managers
+- angr will now show you a warning when using uninitialized memory or registers
+- angr will now NOT show you a warning if you have a capstone 3.x install unless you're actually interacting with the relevant missing parts
+- Many, many, many bug fixes
+
 ## angr 7.8.7.1
 
 - Remove `LoopLimiter` and `DFG`.
@@ -58,7 +74,7 @@ Tracking minor changes are left as an exercise for the reader :-)
 
 Welcome to angr 7!
 We worked long and hard all summer to make this release the best ever.
-It introduces several breaking changes, so for a quick guide on the most common ways you'll need to update your scripts, take a look at the [migration guide](MIGRATION.md).
+It introduces several breaking changes, so for a quick guide on the most common ways you'll need to update your scripts, take a look at the [migration guide](docs/migration-7.md).
 
 - SimuVEX has been removed and its components have been integrated into angr
 - Path has been removed and its components have been integrated into SimState, notably the new `history` state plugin

diff --git a/INSTALL.md b/INSTALL.md
@@ -1,26 +1,26 @@
 # Installing angr
 
-angr is a python library, so it must be installed into your python environment before it can be used. It is built for Python 2: Py3k support is feasible somewhere out in the future, but we are a little hesitant to make that commitment right now (pull requests welcome!).
+angr is a python library, so it must be installed into your python environment before it can be used. It is built for Python 3: Python 2 support is not feasable due to the looming EOL and the small size of our team.
 
 We highly recommend using a [python virtual environment](https://virtualenvwrapper.readthedocs.org/en/latest/) to install and use angr. Several of angr's dependencies (z3, pyvex) require libraries of native code that are forked from their originals, and if you already have libz3 or libVEX installed, you definitely don't want to overwrite the official shared objects with ours. In general, don't expect support for problems arising from installing angr outside of a virtualenv.
 
 ### Dependencies
 
 All of the python dependencies should be handled by pip and/or the setup.py scripts. You will, however, need to build some C to get from here to the end, so you'll need a good build environment as well as the python development headers. At some point in the dependency install process, you'll install the python library cffi, but (on linux, at least) it won't run unless you install your operating system's libffi package.
 
-On Ubuntu, you will want: `sudo apt-get install python-dev libffi-dev build-essential virtualenvwrapper`. If you are trying out angr-management, you will need `sudo apt-get install libqt4-dev graphviz-dev`.
+On Ubuntu, you will want: `sudo apt-get install python3-dev libffi-dev build-essential virtualenvwrapper`. If you are trying out angr-management, you will need the [PySide 2 requirements](https://wiki.qt.io/Qt_for_Python/GettingStarted) along with `graphviz-dev`.
 
 ### Most Operating systems, all \*nix systems
 
-`mkvirtualenv angr && pip install angr` should usually be sufficient to install angr in most cases, since angr is published on the Python Package Index.
+`mkvirtualenv --python=$(which python3) angr && pip install angr` should usually be sufficient to install angr in most cases, since angr is published on the Python Package Index.
 
 Fish (shell) users can either use [virtualfish](https://github.com/adambrenecki/virtualfish) or the [virtualenv](https://pypi.python.org/pypi/virtualenv) package: `vf new angr && vf activate angr && pip install angr`
 
 Failing that, you can install angr by installing the following repositories (and the dependencies listed in their requirements.txt files), in order, from https://github.com/angr:
 
-- [claripy](https://github.com/angr/claripy)
 - [archinfo](https://github.com/angr/archinfo)
 - [pyvex](https://github.com/angr/pyvex)
+- [claripy](https://github.com/angr/claripy)
 - [cle](https://github.com/angr/cle)
 - [angr](https://github.com/angr/angr)
 
@@ -39,37 +39,54 @@ After installing angr, you will need to fix some shared library paths for the an
 Activate your virtual env and execute the following lines. [A script](https://github.com/angr/angr-dev/blob/master/fix_macOS.sh) is provided in the angr-dev repo.
 
 ```bash
-PYVEX=`python2 -c 'import pyvex; print pyvex.__path__[0]'`
-UNICORN=`python2 -c 'import unicorn; print unicorn.__path__[0]'`
-ANGR=`python2 -c 'import logging; logging.basicConfig(level=logging.CRITICAL); import angr; print angr.__path__[0]'`
+PYVEX=`python3 -c 'import pyvex; print pyvex.__path__[0]'`
+UNICORN=`python3 -c 'import unicorn; print unicorn.__path__[0]'`
+ANGR=`python3 -c 'import angr; print angr.__path__[0]'`
 
 install_name_tool -change libunicorn.1.dylib "$UNICORN"/lib/libunicorn.dylib "$ANGR"/lib/angr_native.dylib
 install_name_tool -change libpyvex.dylib "$PYVEX"/lib/libpyvex.dylib "$ANGR"/lib/angr_native.dylib
 ```
 
 ### Windows
 
-angr can _probably_ be installed from pip on Windows, given that you're in a shell with the visual studio build tools loaded.
+As usual, a virtualenv is very strongly recommended. You can use either the [virtualenv-win](https://pypi.org/project/virtualenvwrapper-win/) or [virtualenv](https://pypi.python.org/pypi/virtualenv) packages for this.
 
-Capstone is difficult to install on windows. You might need to manually specify a wheel to install, but sometimes it installs under a name different from "capstone", so if that happens you want to just remove capstone from the requirements.txt files in angr and archinfo.
+angr can be installed from pip on Windows, same as above: `pip install angr`.
+You should not be required to build any C code with this setup, since wheels (binary distributions) should be automatically pulled down for angr and its dependencies.
 
 # Development install
 
-We created a repo with scripts to make life easier for angr developers.
+There is a special repository `angr-dev` with scripts to make life easier for angr developers.
 You can set up angr in development mode by running:
 
 ```bash
-git clone git@github.com:angr/angr-dev.git
+git clone https://github.com/angr/angr-dev
 cd angr-dev
-mkvirtualenv angr
-./setup.sh
+./setup.sh -i -e angr
 ```
 
-This clones all of the repositories and installs them in editable mode.
-`setup.sh` can even create a PyPy virtualenv for you, resulting in significantly faster performance and lower memory usage.
+This creates a virtualenv (`-e angr`), checks for any dependencies you might need (`-i`), clones all of the repositories and installs them in editable mode.
+`setup.sh` can even create a PyPy virtualenv for you (replace `-e` with `-p`), resulting in significantly faster performance and lower memory usage.
 
 You can branch/edit/recompile the various modules in-place, and it will automatically reflect in your virtual environment.
 
+## Development install on windows
+
+The angr-dev repository has a setup.bat script that creates the same setup as above, though it's not as magical as setup.sh.
+Since we'll be building C code, you must be in the visual studio developer command prompt.
+*Make sure that if you're using a 64-bit python interpreter, you're also using the 64-bit build tools* (`VsDevCmd.bat -arch=x64`)
+
+```bash
+pip install virtualenv
+git clone https://github.com/angr/angr-dev
+cd angr-dev
+virtualenv -p "C:\Path\To\python3\python.exe" env
+env\Scripts\activate
+setup.bat
+```
+
+You may also substitute the use of `virtualenv` above with the `virtualenvwrapper-win` package for a more streamlined experience.
+
 ## Docker install
 
 For convenience, we ship a Docker image that is 99% guaranteed to work.
@@ -113,12 +130,23 @@ pip install -I --no-binary z3-solver z3-solver
 
 ## No such file or directory: 'pyvex_c'
 
-Are you running Ubuntu 12.04? If so, please stop using a 5 year old operating system! Upgrading is free!
+Are you running Ubuntu 12.04? If so, please stop using a 6 year old operating system! Upgrading is free!
 
-You can also try upgrading pip (`pip install -U pip`), which might solve the issue.
+You can also try upgrading pip (`python -m pip install -U pip`), which might solve the issue.
 
 ## AttributeError: 'FFI' object has no attribute 'unpack'
 
 You have an outdated version of the `cffi` Python module.  angr now requires at least version 1.7 of cffi.
 Try `pip install --upgrade cffi`.  If the problem persists, make sure your operating system hasn't pre-installed an old version of cffi, which pip may refuse to uninstall.
 If you're using a Python virtual environment with the pypy interpreter, ensure you have a recent version of pypy, as it includes a version of cffi which pip will not upgrade.
+
+## angr has no attribute Project, or similar
+
+If you can import angr but it doesn't seem to be the actual angr module... did you accidentally name your script `angr.py`?
+You can't do that. Python does not work that way.
+
+# AttributeError: 'module' object has no attribute 'KS_ARCH_X86'
+
+You have the `keystone` package installed, which conflicts with the `keystone-engine` package (an optional dependency of angr).
+Please uninstall `keystone`.
+If you would like to install `keystone-engine`, please do it with `pip install --no-binary keystone-engine keystone-engine`, as the current pip distribution is broken.