Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Line search Docs #5

Open
wants to merge 2 commits into
base: docs
Choose a base branch
from
Open
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Update solvers.md
antoinebou12 authored Dec 19, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
commit 572797eeabd53b3b926f2c271153bd77a6a3e831
86 changes: 69 additions & 17 deletions docs/details/solvers.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
## Linear Solver

PolyFEM offers several linear solver options based on compilation options. For more information, see [PolySolve](../polysolve.md) a stand-alone linear solver wrapper library used by PolyFEM.
PolyFEM offers several linear solver options based on compilation options. For more information, see [PolySolve](../polysolve.md), a stand-alone linear solver wrapper library used by PolyFEM.

Options: `AMGCL`, `Eigen::BiCGSTAB`, `Eigen::CholmodSupernodalLLT`, `Eigen::ConjugateGradient`, `Eigen::DGMRES`, `Eigen::GMRES`, `Eigen::LeastSquaresConjugateGradient`, `Eigen::MINRES`, `Eigen::PardisoLDLT`, `Eigen::PardisoLU`, `Eigen::SimplicialLDLT`, `Eigen::SparseLU`, `Eigen::UmfPackLU`, `Hypre`, `Pardiso`
**Options:** `AMGCL`, `Eigen::BiCGSTAB`, `Eigen::CholmodSupernodalLLT`, `Eigen::ConjugateGradient`, `Eigen::DGMRES`, `Eigen::GMRES`, `Eigen::LeastSquaresConjugateGradient`, `Eigen::MINRES`, `Eigen::PardisoLDLT`, `Eigen::PardisoLU`, `Eigen::SimplicialLDLT`, `Eigen::SparseLU`, `Eigen::UmfPackLU`, `Hypre`, `Pardiso`

---

## Nonlinear Solver

@@ -14,8 +15,7 @@ The settings for the solver are stored inside the field `"nonlinear"`. General s
* `"grad_norm"` (default: `1e-8`): convergence tolerance on the norm ($L^2$) of the gradient
* `"nl_iterations"` (default: `1000`): maximum number of iterations to spend solving
* `"use_grad_norm"` (default: `true`): whether to use the gradient norm or update direction norm for convergence checks
* When optimizing a function it is natural to check for a zero (up to tolerance) gradient as this signifies an extremum. However, we also implement the convergence criteria used by [Li et al. [2020]](https://ipc-sim.github.io/). Where instead of the gradient's norm the update direction's $L^\infty$-norm is used. This provides two benefits: (1) it expresses the convergence criteria in the units of the variable (e.g., distance for elasticity) which (2) avoids applying small updates that lead to a marginal change in the variables. Note: this criterion has been well tested for nonlinear elasticity, but your mileage may vary for other formulations.
<!-- * As a sanity check, there is always a check that $\|\nabla f\| < 10^{-2}$ to make sure the solver does not converge with a large gradient. -->
* When optimizing a function, it is natural to check for a zero (up to tolerance) gradient as this signifies an extremum. However, we also implement the convergence criteria used by [Li et al., 2020](https://ipc-sim.github.io/). Instead of the gradient's norm, the update direction's $L^\infty$-norm is used. This provides two benefits: (1) it expresses the convergence criteria in the units of the variable (e.g., distance for elasticity) and (2) avoids applying small updates that lead to marginal changes in the variables. **Note:** This criterion has been well tested for nonlinear elasticity, but your mileage may vary for other formulations.

### Newton's Method

@@ -25,30 +25,82 @@ A (projected) Newton's method with line search.

A quasi-Newton method, L-BFGS requires more iterations than the full Newton's method but avoids expensive linear solves.

Reference: https://en.wikipedia.org/wiki/Limited-memory_BFGS<br>
Acknowledgments: The L-BFGS solver is implemented using the [LBFGS++](https://github.com/yixuan/LBFGSpp) library.
**Reference:** [Limited-memory BFGS](https://en.wikipedia.org/wiki/Limited-memory_BFGS)
**Acknowledgments:** The L-BFGS solver is implemented using the [LBFGS++](https://github.com/yixuan/LBFGSpp) library.

---

## Line Search

!!! todo
Describe line-search and its purpose.
Line search is a numerical optimization technique used to determine the appropriate step size along a given search direction in iterative optimization algorithms. The primary goal is to ensure sufficient decrease in the objective function, improving convergence speed and stability. By adaptively selecting a step size, line search helps to navigate the optimization landscape efficiently while avoiding overshooting or undershooting.

---

### Backtracking

!!! todo
Describe this method and its parameters.
Backtracking line search is an iterative method that starts with a large initial step size and reduces it iteratively until a sufficient decrease condition, often based on the robust Armijo criterion, is satisfied.

### Armijo
**Parameters:**

!!! todo
Describe this method and its parameters.
- **Initial Step Size:** Starting value for the step size, typically set to `1`.
- **Reduction Factor:** A constant in the range `(0, 1)` that determines how much the step size is reduced in each iteration. Typical values are `0.5` or `0.8`.
- **Armijo Parameter:** A small positive constant, e.g., `1e-4`, which defines the sufficient decrease condition for the step size.

#### Armijo (alt)
**Algorithm:**

1. Set the initial step size.
2. Evaluate the objective function at the current step size.
3. Check if the robust Armijo condition is satisfied:
\[
f(x + \alpha p) \leq f(x) + c \alpha \nabla f(x)^T p
\]
where \( \alpha \) is the step size, \( p \) is the search direction, and \( c \) is the Armijo parameter.
4. If the condition is not met, reduce the step size by multiplying it with the reduction factor and repeat.
5. Stop when the condition is satisfied or the step size becomes too small.

---

### Robust Armijo

The robust Armijo method improves the traditional Armijo criterion by incorporating additional tolerance checks to ensure meaningful convergence. It is particularly useful in preventing premature convergence when the objective function changes negligibly.

**Parameters:**

- **Armijo Parameter (c):** A small positive constant, typically `1e-4`, which sets the strictness of the sufficient decrease condition.
- **Delta Relative Tolerance:** A small positive constant defining the threshold for acceptable relative changes in the objective function.

**Mathematical Formulation:**

The standard robust Armijo condition is:
\[
f(x + \alpha p) \leq f(x) + c \alpha \nabla f(x)^T p
\]

If this condition is not met, the relative tolerance criterion is checked:
\[
\frac{|f(x + \alpha p) - f(x)|}{|f(x)|} \leq \text{tolerance}
\]

If both conditions fail, the step size is reduced further.

**Purpose:**

The robust Armijo method ensures optimization steps are meaningful by combining sufficient decrease with relative tolerance checks. This avoids poor solutions caused by steps that barely alter the objective function.

**References:**

- Longva et al., 2023: Introduced numerical robustness improvements for the Armijo line search, making it suitable for more challenging optimization problems.
- [Wikipedia: Line Search](https://en.wikipedia.org/wiki/Line_search)

---

!!! todo
Describe this method and its parameters.

### More-Thuente

!!! todo
Describe this method and its parameters.
Describe this method and its parameters.


**References:**

- Jorge Nocedal and Stephen J. Wright, *Numerical Optimization*, Springer, 2006.