Implement different rules for step size in derivatives approximation #28
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Step size can heavily influence the solving process based on the derivatives. For that reason, derivative approximations now allow to choose the relative epsilon value and step size computation and experiment with it to achieve the best results for the specific use case. Short notes about the implemented rules: * Fixed -- Using the machine precision without any scaling. * ValueScaled -- Scaling the machine precision by the current value, which avoids small steps for large variables and "big" steps for very small variables. This is often recommended on the internet. * AbsValueScaled -- A variation of the previous, which always uses positive value of the step. This "rule" is used in GSL. It is the default, because it was one of the rules for which our current test for scaled rosenbrock passed, together with Fixed (which I consider naive) and other always-positive step size rules (from which this is the most standard, more on that below). The real reason why the scaled rosenbrock test fails is that, when negative step size is used, it results in singular Jacobian in the first iteration, forcing the trust region algorithm to use LM step, which doesn't behave well apparently. * ValueOrMagScaled -- Scaling the machine precision by the absolute value of the variable or typical magnitude, whatever is larger, to avoid problems when the variable is close to zero. This "rule" is used in Numerical Methods for Unconstrained Optimization and Nonlinear Equations by Dennis and Schnabel. * AbsValueOrMagScaled -- A variation of the previous, which always uses positive value of the step. * ValueOrMagTimesValueScaled -- A variation of the next rule, which uses the sign of the variable value for the sign of the step size. * AbsValueOrMagTimesAbsValueScaled -- This is actually the original rule used in gomez, implemented by accident by swapping the arguments in copysign function (see 4161c56). So it was a bug, but for some reason from all these rules it works the best for my use case. I assume it relates to using bigger step size (if x > 1) and smaller step size (if x < 1). Nevertheless, using max(|x|, mag) * x is seems important, because variants like x^2 * sign(x) or similar did not work well. Anyway, I consider it non-standard, because it isn't based on any literature, but I want to have it in gomez, because it works well for me.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Step size can heavily influence the solving process based on the derivatives. For that reason, derivative approximations now allow to choose the relative epsilon value and step size computation and experiment with it to achieve the best results for the specific use case.