Calculate standard error on PS results #203
Conversation
This commit uses the central value theorem to facilitate calculation of the error on the mean results of the PS controller, using the standard error from a normal distribution. Two new functions are added to the WecOptTool.math package called standardErrorMeasure and standardErrorReduce, the first of which measures the standard error after a given number of samples and the second of which keeps sampling until a desired level of error is found. An example script to show how the standard error varies with the stroke of the RM3 device using the PS controller is added to the RM3 example directory called PSError.m
|
Hi Ryan, I implemented your algorithm, but I'm not sure it's behaving the way you expected it to do. The diagram below shows a simulation with the tolerance set to
I don't think this is converging as you were expecting, but I'm not really surprised because of the random numbers. I'm also not sure what the tolerance is telling you. I set the tolerance to
How do you want to proceed? |
|
I've added a metric which calculates the sum of the standard errors over the sum of the mean (per frequency component). It produces results as such:
I'll do the reverse and set a limit on the error and show how many iterations are required. |
|
OK, so here is the same situation recording the number of iterations required:
So, I really think the question is whether the user wants to set a particular accuracy and wait for the realisations, or choose a number of realisations and suffer the inaccuracy. I suggest we offer both options. |
When a struct is provided a target field must be given to use for the error calculation
Allow options to either measure the error or reduce it to a certain value in the PS controller option to simulateDevice. By default it will reduce the error to 1% of the mean.
Only targets 1% in the mean value with no options to change it.
|
Just a note that this would need to be updated if #170 is merged first and visa-versa. |
|
As part of the testing process I noticed that when summing the errors the results were somewhat better than expected. Turns out, that's because summing the standard error is not the correct approach and taking the 2norm is the correct way to go. I've implemented that and the results are now more consistent. |
|
Add option when reducing error for max samples - return error. Done. |
Adds the option 'maxN' to standardError to control the maximum number of samples when in reduce mode. standardError now always returns both the error and the number of samples in its results. The option 'onError' is also added to control behavior when maxN is exceeded. In 'warn' mode a warning will be issued and in 'raise' mode an error is raised.
Description
This commit uses the central value theorem to facilitate calculation of the error on the mean results of the PS controller, using the standard error from a normal distribution.
Two new functions are added to the WecOptTool.math package called standardErrorMeasure and standardErrorReduce, the first of which measures the standard error after a given number of samples and the second of which keeps sampling until a desired level of error is found.
An example script to show how the standard error varies with the stroke of the RM3 device using the PS controller is added to the RM3 example directory called PSError.m. The results of this simulation are shown here:
Fixes #56
Checklist:
[ ] Ifexamples/RM3/optimization.mhas been modified, the content / linenumbers in
docs/user/optimization.rstare still valid or have been fixed[ ] Add something to the documentation?-> Do as part of Document SeaState class example #153