TUI Support Infrastructure (#1620)

* Support for TUI (#1605) * Update worker thread for server creation * Add necessary files for TUI connectivity * Add necessary files for TUI connectivity * Update MonitorWorker * Update protocol * Blacken * Update setup.py dependencies * Remove state debugging messages * Update setup.py to build protobuf protocol upon install * Remove previously generated state_pb2.py * Change subprocess.Popen to subprocess.check_output * Remove extraneous output * First attempt at fixing protobuf installation It might work, it might not. We'll let the CI sort it out. * Can't forget the f-string * Error on missing protoc * Disable auto-generation of protobuf file * Ignore pb2_errors * Disable monitor start See if this makes the EVM tests pass Co-authored-by: Eric Hennenfent <[email protected]> * Add log monitoring * Log monitoring via TCP * Swittch to rendering state lists directly * Extraneous line * Switch log buffer to multiprocessing queue * Create state transition events Should make it possible to track movements between state lists * Plug new events into context This will break the state merging plugin (but I'll fix it eventually) * move most enums to their own module * Blacken * Add DaemonThread from TUI branch * Add interface for registering daemon threads * Timestamp StateDescriptor upon updates * Capture return value * Blacken * Add solver wrapper to StateBase * Add `solve` events to all instances of SelectedSolver.instance() * Remove executor constraints from WASM * Add solve events to memory.py * Add intermittent execution event * Be more generous with states whose initialization we missed * Add Native callback for updating state descriptor * Fix state killing * Blacken * codecov: Remove outdated 'yml' entry in CI From these commits codecov/codecov-action@ebea5ca codecov/codecov-action@49c86d6 * Add solve event to evm Make warning messages better Debug GH actions Revert "Debug GH actions" This reverts commit f575eea. Fix some pycharm-detected problems Make symbolic function error message more verbose Add solve to published events Loud errors in callbacks by default Trying to find out what's killing truffle Revert "Trying to find out what's killing truffle" This reverts commit 8bd0224. Revert "Make symbolic function error message more verbose" This reverts commit bd3e90c. Debugging Truffle Restore introspector Add try_except on every callback Unconditionally print error message Add traceback Update event.py Debug subscriptions Debug arguments to callbacks Different debug msg 1ast arg Print statement debugging... Pass in `None` as state Revert "Add try_except on every callback" This reverts commit 1c689dd. * Drop solve events outside of a state context Forgot did_solve Remove traceback * Fix must/cannot_be_null usage * Fix missing solve event * Partially restore old did_fork_state ABI * Called internally * Clone iterators instead of creating a list * Use isgenerator instead of checking if iterable * Fix snapshot restoration * Slightly improve Unicorn test API usage * Temporarily disable property verifier tests * improper skip arg * Add simple tests for introspection API * Add test for custom introspector, improve base introspection test * Add intermittent update timestamp * Only allow daemon registration and introspection registration at initialization * Add docs to manticore.py * Add docs for plugin, add update_state_descriptor to EVM * Fix renamed will_start_run --> will_run * Docstrings for DaemonThread and EventSolver * Docs for enums * Improve pretty printer, add some mypy fixes * Don't run daemon threads if run is called multiple times * If at first you don't succeed, destroy all the evidence you tried. * Test the pretty printer * Add StateDescriptor to RTD * Add newlines for RTD parsing * Update to work with new state introspection API * Add termination messages * Also capture killed state messages * Make info logs debug logs * Apply suggestions from code review Newlines for doc comments Co-authored-by: Eric Kilmer <[email protected]> * Add some type hints to manticore.py * Add some type hints to plugin.py * Fix type hint for get_state * Add termination message from TUI PR * Add example script * Add docstrings to the example script * Pass introspection plugin type as an argument * Unskip property verifier tests * Add mypy-requests type hints * Remove itertools.tee The problem with usign tee is that only the first callback to use the iterator can write to it. In `ready_states`, the `save_state` after the `yield` statement is ignored for all others. * Make generator cloning a little bit more robust Now Manticore will give up and return the original argument instead of blowing up if it can't clone the generator * Clean up invalidated unit tests We now fire `introspect` for the first time before we have any states * Debug missing Truffle & Examples coverage * Merge coverage from XML file * Switch coverage to JSON, ignore debug logging and NotImplemented code * Fix copy commands * Move .coverage files directly * Set examples to append coverage * FLAG_NAME doesn't work the way we'd like * Use plugin dict to store introspector * Appease mypy * Fix missing property on unique name * Grab EVM PC * Blacken * Run black on all files if the git diff command fails * Fix mypy errors * Make plugin logging even less verbose * Move log capture and state monitoring to daemon threads * Use the config module for host & port * Fix worker configuration and add test for TUI API * Fix log messages breaking native tests * Split up base Manticore tests and logging tests The verbosity changes seem to be taking hold when they shouldn't * Merge LogTCPHandler and MonitorTCPHandler * Confirm that logging tests return to base level * Fix mypy * Switch back to using a deque for log buffering in the default case * Fix deque API * Update state_pb2.py * Reformat programatically generated files * Drop max verbosity in logging tests Haven't been able to figure out why, but somehow other loggers get "stuck" at this high verbosity and the integration tests try to print out the values of every single register. * Fix duplicated code from bad merge * Remove is_main from state_monitor * Add comment about log buffer size * Remove vestigial is_main * Blacken Co-authored-by: Philip Wang <[email protected]> Co-authored-by: Eric Kilmer <[email protected]>
trailofbits · Jan 27, 2021 · 5a258f4 · 5a258f4
1 parent 334b3aa
commit 5a258f4
Show file tree

Hide file tree

Showing 12 changed files with 801 additions and 25 deletions.
diff --git a/manticore/core/manticore.py b/manticore/core/manticore.py
@@ -2,6 +2,7 @@
 import itertools
 import logging
 import sys
+import time
 import typing
 import random
 import weakref
@@ -21,11 +22,18 @@
 from ..utils.deprecated import deprecated
 from ..utils.enums import StateLists, MProcessingType
 from ..utils.event import Eventful
-from ..utils.helpers import PickleSerializer, pretty_print_state_descriptors
+from ..utils.helpers import PickleSerializer, pretty_print_state_descriptors, deque
 from ..utils.log import set_verbosity
 from ..utils.nointerrupt import WithKeyboardInterruptAs
 from .workspace import Workspace, Testcase
-from .worker import WorkerSingle, WorkerThread, WorkerProcess, DaemonThread
+from .worker import (
+    WorkerSingle,
+    WorkerThread,
+    WorkerProcess,
+    DaemonThread,
+    LogCaptureWorker,
+    state_monitor,
+)
 
 from multiprocessing.managers import SyncManager
 import threading
@@ -88,6 +96,7 @@ def wait_for(self, condition, *args, **kwargs):
         self._terminated_states = []
         self._busy_states = []
         self._killed_states = []
+        self._log_queue = deque(maxlen=5000)
         self._shared_context = {}
 
     def _manticore_threading(self):
@@ -99,6 +108,7 @@ def _manticore_threading(self):
         self._terminated_states = []
         self._busy_states = []
         self._killed_states = []
+        self._log_queue = deque(maxlen=5000)
         self._shared_context = {}
 
     def _manticore_multiprocessing(self):
@@ -120,6 +130,9 @@ def raise_signal():
         self._terminated_states = self._manager.list()
         self._busy_states = self._manager.list()
         self._killed_states = self._manager.list()
+        # The multiprocessing queue is much slower than the deque when it gets full, so we
+        # triple the size in order to prevent that from happening.
+        self._log_queue = self._manager.Queue(15000)
         self._shared_context = self._manager.dict()
         self._context_value_types = {list: self._manager.list, dict: self._manager.dict}
 
@@ -370,8 +383,10 @@ def __init__(
         # Workers will use manticore __dict__ So lets spawn them last
         self._workers = [self._worker_type(id=i, manticore=self) for i in range(consts.procs)]
 
-        # We won't create the daemons until .run() is called
-        self._daemon_threads: typing.List[DaemonThread] = []
+        # Create log capture worker. We won't create the rest of the daemons until .run() is called
+        self._daemon_threads: typing.Dict[int, DaemonThread] = {
+            -1: LogCaptureWorker(id=-1, manticore=self)
+        }
         self._daemon_callbacks: typing.List[typing.Callable] = []
 
         self._snapshot = None
@@ -1102,21 +1117,27 @@ def run(self):
             # User subscription to events is disabled from now on
             self.subscribe = None
 
+        self.register_daemon(state_monitor)
+        self._daemon_threads[-1].start()  # Start log capture worker
+
         # Passing generators to callbacks is a bit hairy because the first callback would drain it if we didn't
         # clone the iterator in event.py. We're preserving the old API here, but it's something to avoid in the future.
         self._publish("will_run", self.ready_states)
         self._running.value = True
+
         # start all the workers!
         for w in self._workers:
             w.start()
 
         # Create each daemon thread and pass it `self`
-        if not self._daemon_threads:  # Don't recreate the threads if we call run multiple times
-            for i, cb in enumerate(self._daemon_callbacks):
+        for i, cb in enumerate(self._daemon_callbacks):
+            if (
+                i not in self._daemon_threads
+            ):  # Don't recreate the threads if we call run multiple times
                 dt = DaemonThread(
                     id=i, manticore=self
                 )  # Potentially duplicated ids with workers. Don't mix!
-                self._daemon_threads.append(dt)
+                self._daemon_threads[dt.id] = dt
                 dt.start(cb)
 
         # Main process. Lets just wait and capture CTRL+C at main
@@ -1173,6 +1194,17 @@ def finalize(self):
             self.generate_testcase(state)
         self.remove_all()
 
+    def wait_for_log_purge(self):
+        """
+        If a client has accessed the log server, and there are still buffered logs,
+        waits up to 2 seconds for the client to retrieve the logs.
+        """
+        if self._daemon_threads[-1].activated:
+            for _ in range(8):
+                if self._log_queue.empty():
+                    break
+                time.sleep(0.25)
+
     ############################################################################
     ############################################################################
     ############################################################################
@@ -1188,6 +1220,7 @@ def save_run_data(self):
             config.save(f)
 
         logger.info("Results in %s", self._output.store.uri)
+        self.wait_for_log_purge()
 
     def introspect(self) -> typing.Dict[int, StateDescriptor]:
         """

diff --git a/manticore/core/plugin.py b/manticore/core/plugin.py
@@ -670,6 +670,24 @@ def get_state_descriptors(self) -> typing.Dict[int, StateDescriptor]:
             out = context.copy()  # TODO: is this necessary to break out of the lock?
         return out
 
+    def did_kill_state_callback(self, state, ex: Exception):
+        """
+        Capture other state-killing exceptions so we can get the corresponding message
+
+        :param state: State that was killed
+        :param ex: The exception w/ the termination message
+        """
+        state_id = state.id
+        with self.locked_context("manticore_state", dict) as context:
+            if state_id not in context:
+                logger.warning(
+                    "Caught killing of state %s, but failed to capture its initialization",
+                    state_id,
+                )
+            context.setdefault(state_id, StateDescriptor(state_id=state_id)).termination_msg = repr(
+                ex
+            )
+
     @property
     def unique_name(self) -> str:
         return IntrospectionAPIPlugin.NAME
diff --git a/manticore/core/state.proto b/manticore/core/state.proto
@@ -0,0 +1,31 @@
+syntax = "proto3";
+
+package mserialize;
+
+message LogMessage{
+	string content = 1;
+}
+
+message State{
+
+    enum StateType{
+        READY = 0;
+        BUSY = 1;
+        KILLED = 2;
+        TERMINATED = 3;
+    }
+
+    int32 id = 2; // state ID
+    StateType type = 3; // Type of state
+    string reason = 4; // Reason for execution stopping
+    int32 num_executing = 5; // number of executing instructions
+    int32 wait_time = 6; 
+}
+
+message StateList{
+    repeated State states = 7; 
+}
+
+message MessageList{
+    repeated LogMessage messages = 8; 
+}