Differentiable Optimization-Based Modeling for Machine Learning

• This repository is by Brandon Amos and contains the full source code and data to produce my thesis document.
• The slides are available in pdf and pptx format.

Unpublished work in this thesis

• Chapter 2 provides some preliminaries and background information on differentiable convex optimization layers, including derivations for the optimization (or variational) viewpoints of the ReLU, sigmoid, and softmax.
• Chapter 7 presents an early version of differentiable CVXPY layers, which is now available here. As a bibliographic note, the cone program differentiation derivation in section 7.3 here remains unpublished in this thesis and was done concurrent to and independent of Differentiating Through a Cone Program.

Publications behind this thesis

Some of the content here is behind these publications:

Differentiable Convex Optimization Layers A. Agrawal*, B. Amos*, S. Barratt*, S. Boyd*, S. Diamond*, and J. Kolter* NeurIPS 2019 [1] [pdf] [code]

Differentiable MPC for End-to-end Planning and Control B. Amos, I. Rodriguez, J. Sacks, B. Boots, and J. Kolter NeurIPS 2018 [2] [pdf] [code]

Depth-Limited Solving for Imperfect-Information Games N. Brown, T. Sandholm, and B. Amos NeurIPS 2018 [3] [pdf]

Learning Awareness Models B. Amos, L. Dinh, S. Cabi, T. Rothörl, S. Colmenarejo, A. Muldal, T. Erez, Y. Tassa, N. de Freitas, and M. Denil ICLR 2018 [4] [pdf]

Task-based End-to-end Model Learning P. Donti, B. Amos, and J. Kolter NeurIPS 2017 [5] [pdf] [code]

OptNet: Differentiable Optimization as a Layer in Neural Networks B. Amos and J. Kolter ICML 2017 [6] [pdf] [code]

Input Convex Neural Networks B. Amos, L. Xu, and J. Kolter ICML 2017 [7] [pdf] [code]

Collapsed Variational Inference for Sum-Product Networks H. Zhao, T. Adel, G. Gordon, and B. Amos ICML 2016 [8] [pdf]

OpenFace: A general-purpose face recognition library with mobile applications B. Amos, B. Ludwiczuk, and M. Satyanarayanan CMU 2016 [9] [pdf] [code]

---

The experimental source code and libraries produced for this thesis are freely available as open source software and are available in the following repositories.

• [cvxgrp/cvxpylayers] Differentiable convex optimization layers in CVXPY.
• [locuslab/mpc.pytorch] A stand-alone PyTorch library for the differentiable model predictive control approach.
• [locuslab/differentiable-mpc] PyTorch experiments for the differentiable MPC work.
• [locuslab/qpth]: A stand-alone PyTorch library for the OptNet QP layers.
• [locuslab/optnet] PyTorch experiments for OptNet.
• [locuslab/icnn] TensorFlow experiments for input-convex neural networks.
• [cmusatyalab/openface] Face recognition with deep neural networks.
• [bamos/block] An intelligent block matrix library for numpy, PyTorch, and beyond.
• [bamos/dcgan-completion.tensorflow] Image Completion with Deep Learning in TensorFlow.
• [bamos/densenet.pytorch] A PyTorch implementation of DenseNet.

---

• This repository started from Cyrus Omar's thesis code, which is based on a CMU thesis template by David Koes and others before.
• Of standalone interest, refs.sort.sh uses biber to alphabetize and standardize my bibliography in refs.bib so it doesn't get too messy. This uses the configuration in refs.conf.
• I use update-pdf.sh to keep the latest PDF only in HEAD, although Git LFS or a related project may be a better solution.

---

---

The BibTeX for this document is:

@phdthesis{amos2019differentiable,
  author       = {Brandon Amos},
  title        = {{Differentiable Optimization-Based Modeling for Machine Learning}},
  school       = {Carnegie Mellon University},
  year         = 2019,
  month        = May,
}