Add instructions on how to run tests.

vtjeng · Dec 31, 2023 · 134dbf5 · 134dbf5
1 parent 29c6063
commit 134dbf5
Showing 1 changed file with 78 additions and 0 deletions.
diff --git a/README.md b/README.md
@@ -60,6 +60,84 @@ Below is a modified version of Table 1 from our paper, where we report the adver
 
 \^ Values in this column represent previous state-of-the-art (SOA), as described in our paper.<br> \* Neural network available for import via listed name using `get_example_network_params`.
 
+## Contributing
+
+### Running Tests
+
+#### Running all tests
+
+To run all tests for the package, run this command from the directory containing this package.
+
+```sh
+$ julia --project -e 'using Pkg; Pkg.test("MIPVerify")'
+  [...]
+     Testing MIPVerify
+  [...]
+     Testing Running tests...
+  [...]
+```
+
+This runs all tests in [`test/runtests.jl`](./test/runtests.jl). ([Source](https://pkgdocs.julialang.org/v1/creating-packages/#Adding-tests-to-the-package)).
+
+Sample output for a successful test:
+
+```sh
+ ────────────────────────────────────────────────────────────────────────────────────────────────
+                                                        Time                    Allocations
+                                               ───────────────────────   ────────────────────────
+    Tot / % measured:                            211s /  99.1%           16.8GiB /  98.9%
+
+ Section                               ncalls     time    %tot     avg     alloc    %tot      avg
+ ────────────────────────────────────────────────────────────────────────────────────────────────
+ batch_processing_helpers/                  1    87.9s   42.1%   87.9s   4.14GiB   24.9%  4.14GiB
+   integration.jl                           1    85.5s   40.9%   85.5s   3.45GiB   20.8%  3.45GiB
+   unit.jl                                  1    2.36s    1.1%   2.36s    706MiB    4.2%   706MiB
+ integration/                               1    75.8s   36.3%   75.8s   4.64GiB   27.9%  4.64GiB
+   sequential/                              1    75.8s   36.2%   75.8s   4.64GiB   27.9%  4.64GiB
+     generated_weights/                     1    75.8s   36.2%   75.8s   4.64GiB   27.9%  4.64GiB
+       conv+fc+softmax.jl                   1    50.4s   24.1%   50.4s   1.29GiB    7.8%  1.29GiB
+         BlurringPerturbationFamily         1    35.0s   16.7%   35.0s   0.95GiB    5.7%  0.95GiB
+         Unrestricted[...]                  1    14.8s    7.1%   14.8s    302MiB    1.8%   302MiB
+           Minimizing lInf norm             1    8.57s    4.1%   8.57s    122MiB    0.7%   122MiB
+           Minimizing l1 norm               1    4.27s    2.0%   4.27s    103MiB    0.6%   103MiB
+           With multiple target  [...]      1    2.00s    1.0%   2.00s   77.2MiB    0.5%  77.2MiB
+         LInfNormBounded[...]               1    488ms    0.2%   488ms   36.9MiB    0.2%  36.9MiB
+       mfc+mfc+softmax.jl                   1    25.4s   12.1%   25.4s   3.35GiB   20.2%  3.35GiB
+ utils/                                     1    32.2s   15.4%   32.2s   6.50GiB   39.2%  6.50GiB
+   import_example_nets.jl                   1    24.1s   11.5%   24.1s   3.04GiB   18.3%  3.04GiB
+     get_example_network_params             1    24.1s   11.5%   24.1s   3.04GiB   18.3%  3.04GiB
+       MNIST.WK17a_linf0.1_authors          1    17.9s    8.5%   17.9s   1.54GiB    9.3%  1.54GiB
+       MNIST.RSL18a_linf0.1_authors         1    4.29s    2.1%   4.29s    786MiB    4.6%   786MiB
+       MNIST.n1                             1    1.91s    0.9%   1.91s    757MiB    4.5%   757MiB
+   import_datasets.jl                       1    8.08s    3.9%   8.08s   3.46GiB   20.8%  3.46GiB
+ net_components/                            1    13.1s    6.3%   13.1s   1.32GiB    7.9%  1.32GiB
+   layers/                                  1    6.86s    3.3%   6.86s    746MiB    4.4%   746MiB
+   core_ops.jl                              1    4.05s    1.9%   4.05s    449MiB    2.6%   449MiB
+   nets/                                    1   48.7ms    0.0%  48.7ms   5.17MiB    0.0%  5.17MiB
+ models.jl                                  1   10.0ms    0.0%  10.0ms   25.2KiB    0.0%  25.2KiB
+ ────────────────────────────────────────────────────────────────────────────────────────────────
+
+
+Test Summary: | Pass  Total
+MIPVerify     |  336    336
+     Testing MIPVerify tests passed
+```
+
+#### Running a subset of tests
+
+Running all tests takes ~5 minutes. For a faster iteration cycle, you can run a smaller set of tests. To run tests in a specific file (e.g., [`test/vendor.jl`](./test/vendor.jl)), run this command from the directory containing this package.
+
+```sh
+$ julia -e 'using Pkg; Pkg.add("TestEnv"); using TestEnv; Pkg.activate("."); TestEnv.activate(); include("test/vendor.jl")'
+    Updating registry at `~/.julia/registries/General.toml`
+   Resolving package versions...
+  No Changes to `~/.julia/environments/v1.7/Project.toml`
+  No Changes to `~/.julia/environments/v1.7/Manifest.toml`
+  Activating project at `~/development/vtjeng/MIPVerify.jl`
+Test Summary: | Pass  Total
+vendor/       |   22     22
+```
+
 ## Citing this Library
 
 ```bibtex