Write Readme; Include yaml for environment

.gitignore

@@ -1,3 +1,6 @@
+data/Irradiance_1sec_Anatolia_24.mat
+data/Loads_1sec_anatolia.mat
+
 # Byte-compiled / optimized / DLL files
 __pycache__/
 *.py[cod]

README.md

@@ -1,51 +1,60 @@
-# Name
+# PROF: Projected Feasibility
 
 This is the official repository that implements the following paper:
 
-...
+> *Chen, Bingqing, Priya Donti, Kyri Baker, J. Zico Kolter, and Mario Berges. "Enforcing Policy Feasibility Constraints through Differentiable Projection for Energy Optimization." In Proceedings of the Twelfth ACM International Conference on Future Energy Systems (e-energy '21). pp. 199–210. 2021.*
 
-[[slides]](docs/slides.pdf)[[paper]](https://dl.acm.org/doi/10.1145/3408308.3427980)[[video]](https://www.youtube.com/watch?v=rH64WyPHCVE) 
+[[slides]](docs/slides.pdf)[[paper]](https://dl.acm.org/doi/10.1145/3447555.3464874)<[[video]](https://www.youtube.com/watch?v=rH64WyPHCVE)> 
 
 # Overview
 
+PROjected Feasibility (PROF) is a method to enforce convex operational constraints within neural policies, by incorporating a differentiable projection layer within a neural network-based policy to enforce that all learned actions are feasible. We then update the policy end-to-end by propagating gradients through this differentiable projection layer, making the policy cognizant of the operational constraints. The result is a powerful neural policy that can flexibly optimize performance on the true underlying dynamics, while still satisfying the specified constraints.
+
+We demonstrate our method on two applications: energy-efficient building operation and inverter control. 
+- In the building control case, PROF outperforms other RL agents, while maintaining temperature within the deadband except when the control is saturated.
+- In the inverter control setting, PROF satisfies the constraints 100% of the time and minimizes curtailment as well as possible within its conservative safe set after learning safely for half a day.
+
 **Framework.** 
+<img src="docs/figs/framework.pdf" data-canonical-src="docs/figs/framework.pdf" width="600" />
+
 
 # Code Usage
 ### Clone repository
 ```
 git clone https://github.com/INFERLab/PROF.git
-cd COHORT
+cd PROF
 ```
 
 ### Set up the environment 
 Set up the virtual environment with your preferred environment/package manager.
 
 The instruction here is based on **conda**. ([Install conda](https://docs.anaconda.com/anaconda/install/))
 ```
-conda create --name cohort-env python=3.7 -c conda-forge -f requirements.txt
-condo activate cohort-env
+conda env create --file environment.yml 
+condo activate nn-w-proj
 ```
 
 ### File Structure
 ```
 .
 ├── agents
-│   ├── base.py             # Implement a controller that creates the CVXPY problem given building parameters
+│   ├── base.py             # Implement a controller that instantiate the projection problem given building parameters
 │   └── nn_policy.py        # Inherit the controller from base.py; Forward pass: NN + Differentiable projection
+│   └── inverter_policy.py  # Policy for inverter: NN + Differentiable projection
 ├── algo                    
 │   └── ppo.py	 	    # A PPO trainer 
 ├── env
 │   └── inverter.py         # Implements the IEEE 37-bus case
 ├── utils
-│   ├── network.py          # Implements vanilla MLP and LSTM
+│   ├── network.py          # Implements neural network modules, e.g. MLP and LSTM
 │   └── ppo_utils.py        # Helper function for PPO trainer, e.g. Replay_Memory, Advantage_func
-├── network		    # Matlab code for linearization; Data for grid;
-└── mypypower		    # Include some small changes from PyPower source code
+├── network		    # Matlab code for linearizing grid model; Data to construct IEEE 37-bus case;
+└── mypypower		    # Include some small changes from PyPower source code to allow customization
 
 ```
 
 ### Running
-You can replicate our experiments for *Energy efficiency in a single building* with `main_IW.py`.
+You can replicate our experiments for *Experiment 1: Energy-efficient Building Operation* with `main_IW.py` and *Experiment 2: Inverter Control* with `main_inverter.py`
 
 
 ### Feedback
@@ -54,8 +63,23 @@ Feel free to send any questions/feedback to: [Bingqing Chen](mailto:bingqinc@and
 
 ### Citation
 
-If you use COHORT, please cite us as follows:
+If you use PROF, please cite us as follows:
 
 ```
-test
+@inproceedings{chen2021enforcing,
+author = {Chen, Bingqing and Donti, Priya L. and Baker, Kyri and Kolter, J. Zico and Berg\'{e}s, Mario},
+title = {Enforcing Policy Feasibility Constraints through Differentiable Projection for Energy Optimization},
+year = {2021},
+isbn = {9781450383332},
+publisher = {Association for Computing Machinery},
+address = {New York, NY, USA},
+url = {https://doi.org/10.1145/3447555.3464874},
+doi = {10.1145/3447555.3464874},
+booktitle = {Proceedings of the Twelfth ACM International Conference on Future Energy Systems},
+pages = {199–210},
+numpages = {12},
+keywords = {inverter control, safe reinforcement learning, implicit layers, differentiable optimization, smart building},
+location = {Virtual Event, Italy},
+series = {e-Energy '21}
+}
 ```

agents/base.py

@@ -195,33 +195,3 @@ def u_update(self, v_bar, w_bar):
             controller.u_diff.value = u_list[idx] - u_bar
         return u_bar, np.array(u_list)
 
-class Aggregator():
-    def __init__(self, T, g_func, param_shapes, n_agent, rho):
-        self.N = n_agent
-        self.rho = rho
-        self.v_bar = cp.Variable(T)
-
-        self.u_bar = cp.Parameter(T)
-        self.w_bar = cp.Parameter(T)
-
-        self.param_list = []
-        for item in param_shapes:
-            self.param_list.append(cp.Parameter(item))
-        #print(self.param_list)
-
-        # Objective:
-        cost = g_func(self.N*self.v_bar, self.param_list)#
-
-        cost += self.N*self.rho/2*cp.sum_squares(self.u_bar-self.v_bar+self.w_bar)
-        self.Problem = cp.Problem(cp.Minimize(cost))
-
-    def setValues(self, value_list):
-        assert len(value_list) == len(self.param_list)
-        for idx, item in enumerate(self.param_list):
-            item.value = value_list[idx]
-
-    def v_update(self, u_bar, w_bar):
-        self.u_bar.value = u_bar
-        self.w_bar.value = w_bar
-        self.Problem.solve()
-        return self.v_bar.value

agents/inverter_policy.py

@@ -15,7 +15,6 @@
 DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
 
 
-### Can move to utils.network if appropriate
 class Net(nn.Module):
     def __init__(self, n_bus, n_inverters, shared_hidden_layer_sizes, indiv_hidden_layer_sizes, n_input = 3):
         super(Net, self).__init__()
@@ -239,15 +238,3 @@ def is_feasible(self, P, Q, P_nc, Q_nc, P_av):
         else:
             return True
 
-
-
-
-
-# if __name__ == "__main__":
-#     # Toy test code
-#     n = 3
-#     nbatch = 2
-#     model = Net(n, [200,200,40], [30,5])
-#     controller = NeuralController(n, model, np.random.randn(n, 2*n), 0.95, 1.05)
-#     action = controller(torch.randn(nbatch, 4*n), torch.randn(nbatch, n), torch.randn(nbatch, n))
-

environment.yml

@@ -0,0 +1,105 @@
+name: nn-w-proj
+channels:
+  - conda-forge
+  - defaults
+dependencies:
+  - _libgcc_mutex=0.1=conda_forge
+  - _openmp_mutex=4.5=1_gnu
+  - ampl-mp=3.1.0=h616b090_1004
+  - backcall=0.2.0=pyhd3eb1b0_0
+  - ca-certificates=2021.4.13=h06a4308_1
+  - certifi=2020.12.5=py39h06a4308_0
+  - cvxpy=1.1.12=py39hf3d152e_0
+  - cvxpy-base=1.1.12=py39hde0f152_0
+  - cyipopt=1.0.3=py39h3c5bb4f_0
+  - decorator=5.0.7=pyhd3eb1b0_0
+  - ecos=2.0.8=py39hce5d2b2_0
+  - future=0.18.2=py39hf3d152e_3
+  - ipopt=3.13.4=h7ede334_0
+  - ipykernel=5.3.4=py39hb070fc8_0
+  - ipython=7.22.0=py39hb070fc8_0
+  - ipython_genutils=0.2.0=pyhd3eb1b0_1
+  - jedi=0.17.2=py39h06a4308_1
+  - jupyter_client=6.1.12=pyhd3eb1b0_0
+  - jupyter_core=4.7.1=py39h06a4308_0
+  - ld_impl_linux-64=2.35.1=hea4e1c9_2
+  - libblas=3.9.0=9_openblas
+  - libcblas=3.9.0=9_openblas
+  - libffi=3.3=h58526e2_2
+  - libgcc-ng=9.3.0=h2828fa1_19
+  - libgfortran-ng=9.3.0=hff62375_19
+  - libgfortran5=9.3.0=hff62375_19
+  - libgomp=9.3.0=h2828fa1_19
+  - liblapack=3.9.0=9_openblas
+  - libopenblas=0.3.15=pthreads_h8fe5266_0
+  - libsodium=1.0.18=h7b6447c_0
+  - libstdcxx-ng=9.3.0=h6de172a_19
+  - metis=5.1.0=h58526e2_1006
+  - mumps-include=5.2.1=ha770c72_10
+  - mumps-seq=5.2.1=h47a8eb5_10
+  - ncurses=6.2=h58526e2_4
+  - numpy=1.20.2=py39hdbf815f_0
+  - openssl=1.1.1k=h27cfd23_0
+  - osqp=0.6.2=py39hde0f152_1
+  - parso=0.7.0=py_0
+  - pexpect=4.8.0=pyhd3eb1b0_3
+  - pickleshare=0.7.5=pyhd3eb1b0_1003
+  - pip=21.0.1=py39h06a4308_0
+  - prompt-toolkit=3.0.17=pyh06a4308_0
+  - ptyprocess=0.7.0=pyhd3eb1b0_2
+  - pygments=2.8.1=pyhd3eb1b0_0
+  - python=3.9.4=hffdb5ce_0_cpython
+  - python-dateutil=2.8.1=pyhd3eb1b0_0
+  - python_abi=3.9=1_cp39
+  - pyzmq=20.0.0=py39h2531618_1
+  - qdldl-python=0.1.5=py39hde0f152_0
+  - readline=8.1=h46c0cb4_0
+  - scipy=1.6.3=py39hee8e79c_0
+  - scotch=6.0.9=h0eec0ba_1
+  - scs=2.1.3=py39h3c5bb4f_0
+  - setuptools=49.6.0=py39hf3d152e_3
+  - six=1.16.0=pyh6c4a22f_0
+  - sqlite=3.35.5=h74cdb3f_0
+  - tk=8.6.10=h21135ba_1
+  - tornado=6.1=py39h27cfd23_0
+  - traitlets=5.0.5=pyhd3eb1b0_0
+  - tzdata=2021a=he74cb21_0
+  - wcwidth=0.2.5=py_0
+  - wheel=0.36.2=pyhd3deb0d_0
+  - xz=5.2.5=h516909a_1
+  - zeromq=4.3.4=h2531618_0
+  - zlib=1.2.11=h516909a_1010
+  - pip:
+    - absl-py==0.13.0
+    - cachetools==4.2.2
+    - charset-normalizer==2.0.5
+    - cvxpylayers==0.1.5
+    - cycler==0.10.0
+    - diffcp==1.0.16
+    - google-auth==1.35.0
+    - google-auth-oauthlib==0.4.6
+    - grpcio==1.40.0
+    - idna==3.2
+    - kiwisolver==1.3.1
+    - markdown==3.3.4
+    - matplotlib==3.4.2
+    - oauthlib==3.1.1
+    - pillow==8.2.0
+    - protobuf==3.17.3
+    - pyasn1==0.4.8
+    - pyasn1-modules==0.2.8
+    - pybind11==2.6.2
+    - pyparsing==2.4.7
+    - pypower==5.1.15
+    - requests==2.26.0
+    - requests-oauthlib==1.3.0
+    - rsa==4.7.2
+    - tensorboard==2.6.0
+    - tensorboard-data-server==0.6.1
+    - tensorboard-plugin-wit==1.8.0
+    - threadpoolctl==2.1.0
+    - torch==1.8.1
+    - typing-extensions==3.10.0.0
+    - urllib3==1.26.6
+    - werkzeug==2.0.1
+prefix: /home/bingqinc/anaconda3/envs/nn-w-proj