Mercurial > octave
annotate scripts/optimization/fminsearch.m @ 21099:52af4092f863
For optimization scripts, correctly choose tolerance (eps) based on class of fun and X0.
* fminbnd.m, fminsearch.m, fminunc.m, fsolve.m: Use 'isa (..., "single")' to
check both initial guess X0 and result of function evaluation to decide whether
eps should be single or double.
| author | Rik <rik@octave.org> |
|---|---|
| date | Mon, 18 Jan 2016 10:00:56 -0800 |
| parents | 516bb87ea72e |
| children | b571fc85953f |
| rev | line source |
|---|---|
| 14895 | 1 ## Copyright (C) 2003,2012 Andy Adler |
|
17744
d63878346099
maint: Update copyright notices for release.
John W. Eaton <jwe@octave.org>
parents:
17281
diff
changeset
|
2 ## Copyright (C) 2002, 2013 N.J.Higham |
| 14895 | 3 ## |
| 4 ## This file is part of Octave. | |
| 5 ## | |
| 6 ## Octave is free software; you can redistribute it and/or modify it | |
| 7 ## under the terms of the GNU General Public License as published by | |
| 8 ## the Free Software Foundation; either version 3 of the License, or (at | |
| 9 ## your option) any later version. | |
| 10 ## | |
| 11 ## Octave is distributed in the hope that it will be useful, but | |
| 12 ## WITHOUT ANY WARRANTY; without even the implied warranty of | |
| 13 ## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU | |
| 14 ## General Public License for more details. | |
| 15 ## | |
| 16 ## You should have received a copy of the GNU General Public License | |
| 17 ## along with Octave; see the file COPYING. If not, see | |
| 18 ## <http://www.gnu.org/licenses/>. | |
| 19 | |
| 20 ## -*- texinfo -*- | |
|
20852
516bb87ea72e
2015 Code Sprint: remove class of function from docstring for all m-files.
Rik <rik@octave.org>
parents:
20735
diff
changeset
|
21 ## @deftypefn {} {@var{x} =} fminsearch (@var{fun}, @var{x0}) |
|
516bb87ea72e
2015 Code Sprint: remove class of function from docstring for all m-files.
Rik <rik@octave.org>
parents:
20735
diff
changeset
|
22 ## @deftypefnx {} {@var{x} =} fminsearch (@var{fun}, @var{x0}, @var{options}) |
|
516bb87ea72e
2015 Code Sprint: remove class of function from docstring for all m-files.
Rik <rik@octave.org>
parents:
20735
diff
changeset
|
23 ## @deftypefnx {} {[@var{x}, @var{fval}] =} fminsearch (@dots{}) |
| 14895 | 24 ## |
| 25 ## Find a value of @var{x} which minimizes the function @var{fun}. | |
|
20165
f1d0f506ee78
doc: Update more docstrings to have one sentence summary as first line.
Rik <rik@octave.org>
parents:
19596
diff
changeset
|
26 ## |
| 14895 | 27 ## The search begins at the point @var{x0} and iterates using the |
|
19040
0850b5212619
doc: Add @nospell macro around proper names in documentation.
Rik <rik@octave.org>
parents:
17744
diff
changeset
|
28 ## @nospell{Nelder & Mead} Simplex algorithm (a derivative-free method). This |
|
0850b5212619
doc: Add @nospell macro around proper names in documentation.
Rik <rik@octave.org>
parents:
17744
diff
changeset
|
29 ## algorithm is better-suited to functions which have discontinuities or for |
|
0850b5212619
doc: Add @nospell macro around proper names in documentation.
Rik <rik@octave.org>
parents:
17744
diff
changeset
|
30 ## which a gradient-based search such as @code{fminunc} fails. |
| 14895 | 31 ## |
|
20165
f1d0f506ee78
doc: Update more docstrings to have one sentence summary as first line.
Rik <rik@octave.org>
parents:
19596
diff
changeset
|
32 ## Options for the search are provided in the parameter @var{options} using the |
|
f1d0f506ee78
doc: Update more docstrings to have one sentence summary as first line.
Rik <rik@octave.org>
parents:
19596
diff
changeset
|
33 ## function @code{optimset}. Currently, @code{fminsearch} accepts the options: |
|
f1d0f506ee78
doc: Update more docstrings to have one sentence summary as first line.
Rik <rik@octave.org>
parents:
19596
diff
changeset
|
34 ## @qcode{"TolX"}, @qcode{"MaxFunEvals"}, @qcode{"MaxIter"}, @qcode{"Display"}. |
|
f1d0f506ee78
doc: Update more docstrings to have one sentence summary as first line.
Rik <rik@octave.org>
parents:
19596
diff
changeset
|
35 ## For a description of these options, see @code{optimset}. |
| 14895 | 36 ## |
|
20165
f1d0f506ee78
doc: Update more docstrings to have one sentence summary as first line.
Rik <rik@octave.org>
parents:
19596
diff
changeset
|
37 ## On exit, the function returns @var{x}, the minimum point, and @var{fval}, |
|
f1d0f506ee78
doc: Update more docstrings to have one sentence summary as first line.
Rik <rik@octave.org>
parents:
19596
diff
changeset
|
38 ## the function value thereof. |
| 14895 | 39 ## |
| 40 ## Example usages: | |
| 41 ## | |
| 42 ## @example | |
| 43 ## @group | |
| 44 ## fminsearch (@@(x) (x(1)-5).^2+(x(2)-8).^4, [0;0]) | |
| 45 ## | |
| 46 ## fminsearch (inline ("(x(1)-5).^2+(x(2)-8).^4", "x"), [0;0]) | |
| 47 ## @end group | |
| 48 ## @end example | |
| 49 ## @seealso{fminbnd, fminunc, optimset} | |
| 50 ## @end deftypefn | |
| 51 | |
| 52 ## PKG_ADD: ## Discard result to avoid polluting workspace with ans at startup. | |
| 53 ## PKG_ADD: [~] = __all_opts__ ("fminsearch"); | |
| 54 | |
|
21099
52af4092f863
For optimization scripts, correctly choose tolerance (eps) based on class of fun and X0.
Rik <rik@octave.org>
parents:
20852
diff
changeset
|
55 ## FIXME: Add support for "exitflag" output variable. |
|
52af4092f863
For optimization scripts, correctly choose tolerance (eps) based on class of fun and X0.
Rik <rik@octave.org>
parents:
20852
diff
changeset
|
56 ## FIXME: Add support for "output" output variable. |
| 14895 | 57 ## FIXME: For Display option, add 'final' and 'notify' options. Not too hard. |
|
21099
52af4092f863
For optimization scripts, correctly choose tolerance (eps) based on class of fun and X0.
Rik <rik@octave.org>
parents:
20852
diff
changeset
|
58 ## FIXME: Add support for OutputFcn. See fminunc for a template. |
| 14895 | 59 ## FIXME: Add support for exiting based on TolFun. See fminunc for an idea. |
| 60 | |
| 61 function [x, fval] = fminsearch (fun, x0, options = struct ()) | |
| 62 | |
| 63 ## Get default options if requested. | |
| 64 if (nargin == 1 && ischar (fun) && strcmp (fun, "defaults")) | |
| 65 x = optimset ("Display", "notify", "FunValCheck", "off", | |
| 66 "MaxFunEvals", 400, "MaxIter", 400, | |
| 67 "OutputFcn", [], | |
| 68 "TolFun", 1e-7, "TolX", 1e-4); | |
| 69 return; | |
| 70 endif | |
| 71 | |
| 72 if (nargin < 2 || nargin > 3) | |
| 73 print_usage (); | |
| 74 endif | |
| 75 | |
| 76 x = nmsmax (fun, x0, options); | |
| 77 | |
| 78 if (isargout (2)) | |
| 79 fval = feval (fun, x); | |
| 80 endif | |
| 81 | |
| 82 endfunction | |
| 83 | |
| 84 ##NMSMAX Nelder-Mead simplex method for direct search optimization. | |
| 85 ## [x, fmax, nf] = NMSMAX(FUN, x0, STOPIT, SAVIT) attempts to | |
| 86 ## maximize the function FUN, using the starting vector x0. | |
| 87 ## The Nelder-Mead direct search method is used. | |
| 88 ## Output arguments: | |
| 89 ## x = vector yielding largest function value found, | |
| 90 ## fmax = function value at x, | |
| 91 ## nf = number of function evaluations. | |
| 92 ## The iteration is terminated when either | |
| 93 ## - the relative size of the simplex is <= STOPIT(1) | |
| 94 ## (default 1e-3), | |
| 95 ## - STOPIT(2) function evaluations have been performed | |
| 96 ## (default inf, i.e., no limit), or | |
| 97 ## - a function value equals or exceeds STOPIT(3) | |
| 98 ## (default inf, i.e., no test on function values). | |
| 99 ## The form of the initial simplex is determined by STOPIT(4): | |
| 100 ## STOPIT(4) = 0: regular simplex (sides of equal length, the default) | |
| 101 ## STOPIT(4) = 1: right-angled simplex. | |
| 102 ## Progress of the iteration is not shown if STOPIT(5) = 0 (default 1). | |
| 103 ## STOPIT(6) indicates the direction (ie. minimization or | |
| 104 ## maximization.) Default is 1, maximization. | |
| 105 ## set STOPIT(6)=-1 for minimization | |
| 106 ## If a non-empty fourth parameter string SAVIT is present, then | |
|
15468
6437fa7263dd
use ' instead of ` in error messages, warnings and most comments
John W. Eaton <jwe@octave.org>
parents:
14895
diff
changeset
|
107 ## 'SAVE SAVIT x fmax nf' is executed after each inner iteration. |
| 14895 | 108 ## NB: x0 can be a matrix. In the output argument, in SAVIT saves, |
| 109 ## and in function calls, x has the same shape as x0. | |
| 110 ## NMSMAX(fun, x0, STOPIT, SAVIT, P1, P2,...) allows additional | |
| 111 ## arguments to be passed to fun, via feval(fun,x,P1,P2,...). | |
| 112 ## References: | |
| 113 ## N. J. Higham, Optimization by direct search in matrix computations, | |
| 114 ## SIAM J. Matrix Anal. Appl, 14(2): 317-333, 1993. | |
| 115 ## C. T. Kelley, Iterative Methods for Optimization, Society for Industrial | |
| 116 ## and Applied Mathematics, Philadelphia, PA, 1999. | |
| 117 | |
| 118 ## From Matrix Toolbox | |
|
17744
d63878346099
maint: Update copyright notices for release.
John W. Eaton <jwe@octave.org>
parents:
17281
diff
changeset
|
119 ## Copyright (C) 2002, 2013 N.J.Higham |
| 14895 | 120 ## www.maths.man.ac.uk/~higham/mctoolbox |
| 121 ## | |
| 122 ## Modifications for Octave by A.Adler 2003 | |
| 123 | |
| 124 function [stopit, savit, dirn, trace, tol, maxiter] = parse_options (options, x ); | |
| 125 | |
| 126 ## Tolerance for cgce test based on relative size of simplex. | |
| 127 stopit(1) = tol = optimget (options, "TolX", 1e-4); | |
| 128 | |
| 129 ## Max no. of f-evaluations. | |
| 130 stopit(2) = optimget (options, "MaxFunEvals", length (x) * 200); | |
| 131 | |
| 132 ## Max no. of iterations | |
| 133 maxiter = optimget (options, "MaxIter", length (x) * 200); | |
| 134 | |
| 135 ## Default target for f-values. | |
| 136 stopit(3) = Inf; # FIXME: expose this parameter to the outside | |
| 137 | |
| 138 ## Default initial simplex. | |
| 139 stopit(4) = 0; # FIXME: expose this parameter to the outside | |
| 140 | |
| 141 ## Default: show progress. | |
| 142 display = optimget (options, "Display", "notify"); | |
| 143 if (strcmp (display, "iter")) | |
| 144 stopit(5) = 1; | |
| 145 else | |
| 146 stopit(5) = 0; | |
| 147 endif | |
| 148 trace = stopit(5); | |
| 149 | |
| 150 ## Use function to minimize, not maximize | |
| 151 stopit(6) = dirn = -1; | |
| 152 | |
|
20713
2469d78a1d8b
Consistently use 'filename' rather than 'file name' throughout code base.
Rik <rik@octave.org>
parents:
20231
diff
changeset
|
153 ## Filename for snapshots. |
| 14895 | 154 savit = []; # FIXME: expose this parameter to the outside |
| 155 | |
| 156 endfunction | |
| 157 | |
| 158 function [x, fmax, nf] = nmsmax (fun, x, options, savit, varargin) | |
| 159 | |
| 160 [stopit, savit, dirn, trace, tol, maxiter] = parse_options (options, x); | |
| 161 | |
| 162 if (strcmpi (optimget (options, "FunValCheck", "off"), "on")) | |
| 163 ## Replace fcn with a guarded version. | |
| 164 fun = @(x) guarded_eval (fun, x); | |
| 165 endif | |
| 166 | |
| 167 x0 = x(:); # Work with column vector internally. | |
| 168 n = length (x0); | |
| 169 | |
| 170 V = [zeros(n,1) eye(n)]; | |
| 171 f = zeros (n+1,1); | |
| 172 V(:,1) = x0; | |
| 173 f(1) = dirn * feval (fun,x,varargin{:}); | |
| 174 fmax_old = f(1); | |
| 175 | |
| 176 if (trace) | |
| 177 fprintf ("f(x0) = %9.4e\n", f(1)); | |
| 178 endif | |
| 179 | |
| 180 k = 0; m = 0; | |
| 181 | |
| 182 ## Set up initial simplex. | |
| 183 scale = max (norm (x0,Inf), 1); | |
| 184 if (stopit(4) == 0) | |
| 185 ## Regular simplex - all edges have same length. | |
| 186 ## Generated from construction given in reference [18, pp. 80-81] of [1]. | |
| 187 alpha = scale / (n*sqrt (2)) * [sqrt(n+1)-1+n, sqrt(n+1)-1]; | |
| 188 V(:,2:n+1) = (x0 + alpha(2)*ones (n,1)) * ones (1,n); | |
| 189 for j = 2:n+1 | |
| 190 V(j-1,j) = x0(j-1) + alpha(1); | |
| 191 x(:) = V(:,j); | |
| 192 f(j) = dirn * feval (fun,x,varargin{:}); | |
| 193 endfor | |
| 194 else | |
| 195 ## Right-angled simplex based on co-ordinate axes. | |
| 196 alpha = scale * ones(n+1,1); | |
| 197 for j=2:n+1 | |
| 198 V(:,j) = x0 + alpha(j)*V(:,j); | |
| 199 x(:) = V(:,j); | |
| 200 f(j) = dirn * feval (fun,x,varargin{:}); | |
| 201 endfor | |
| 202 endif | |
| 203 nf = n+1; | |
| 204 how = "initial "; | |
| 205 | |
| 206 [~,j] = sort (f); | |
| 207 j = j(n+1:-1:1); | |
| 208 f = f(j); | |
| 209 V = V(:,j); | |
| 210 | |
| 211 alpha = 1; beta = 1/2; gamma = 2; | |
| 212 | |
| 213 while (1) # Outer (and only) loop. | |
|
20735
418ae0cb752f
Replace ++,-- with in-place operators for performance.
Rik <rik@octave.org>
parents:
20713
diff
changeset
|
214 k += 1; |
| 14895 | 215 |
| 216 if (k > maxiter) | |
| 217 msg = "Exceeded maximum iterations...quitting\n"; | |
| 218 break; | |
| 219 endif | |
| 220 | |
| 221 fmax = f(1); | |
| 222 if (fmax > fmax_old) | |
| 223 if (! isempty (savit)) | |
| 224 x(:) = V(:,1); | |
| 225 eval (["save " savit " x fmax nf"]); | |
| 226 endif | |
| 227 endif | |
| 228 if (trace) | |
| 229 fprintf ("Iter. %2.0f,", k); | |
| 230 fprintf ([" how = " how " "]); | |
| 231 fprintf ("nf = %3.0f, f = %9.4e (%2.1f%%)\n", nf, fmax, ... | |
| 232 100*(fmax-fmax_old)/(abs(fmax_old)+eps)); | |
| 233 endif | |
| 234 fmax_old = fmax; | |
| 235 | |
| 236 ## Three stopping tests from MDSMAX.M | |
| 237 | |
| 238 ## Stopping Test 1 - f reached target value? | |
| 239 if (fmax >= stopit(3)) | |
| 240 msg = "Exceeded target...quitting\n"; | |
| 241 break; | |
| 242 endif | |
| 243 | |
| 244 ## Stopping Test 2 - too many f-evals? | |
| 245 if (nf >= stopit(2)) | |
| 246 msg = "Max no. of function evaluations exceeded...quitting\n"; | |
| 247 break; | |
| 248 endif | |
| 249 | |
| 250 ## Stopping Test 3 - converged? This is test (4.3) in [1]. | |
| 251 v1 = V(:,1); | |
| 252 size_simplex = norm (V(:,2:n+1)-v1(:,ones (1,n)),1) / max (1, norm (v1,1)); | |
| 253 if (size_simplex <= tol) | |
| 254 msg = sprintf ("Simplex size %9.4e <= %9.4e...quitting\n", ... | |
| 255 size_simplex, tol); | |
| 256 break; | |
| 257 endif | |
| 258 | |
| 259 ## One step of the Nelder-Mead simplex algorithm | |
| 260 ## NJH: Altered function calls and changed CNT to NF. | |
|
15468
6437fa7263dd
use ' instead of ` in error messages, warnings and most comments
John W. Eaton <jwe@octave.org>
parents:
14895
diff
changeset
|
261 ## Changed each 'fr < f(1)' type test to '>' for maximization |
| 14895 | 262 ## and re-ordered function values after sort. |
| 263 | |
| 264 vbar = (sum (V(:,1:n)')/n)'; # Mean value | |
| 265 vr = (1 + alpha)*vbar - alpha*V(:,n+1); | |
| 266 x(:) = vr; | |
| 267 fr = dirn * feval (fun,x,varargin{:}); | |
|
20231
83792dd9bcc1
Use in-place operators in m-files where possible.
Rik <rik@octave.org>
parents:
20165
diff
changeset
|
268 nf += 1; |
| 14895 | 269 vk = vr; fk = fr; how = "reflect, "; |
| 270 if (fr > f(n)) | |
| 271 if (fr > f(1)) | |
| 272 ve = gamma*vr + (1-gamma)*vbar; | |
| 273 x(:) = ve; | |
| 274 fe = dirn * feval (fun,x,varargin{:}); | |
|
20231
83792dd9bcc1
Use in-place operators in m-files where possible.
Rik <rik@octave.org>
parents:
20165
diff
changeset
|
275 nf += 1; |
| 14895 | 276 if (fe > f(1)) |
| 277 vk = ve; | |
| 278 fk = fe; | |
| 279 how = "expand, "; | |
| 280 endif | |
| 281 endif | |
| 282 else | |
| 283 vt = V(:,n+1); | |
| 284 ft = f(n+1); | |
| 285 if (fr > ft) | |
| 286 vt = vr; | |
| 287 ft = fr; | |
| 288 endif | |
| 289 vc = beta*vt + (1-beta)*vbar; | |
| 290 x(:) = vc; | |
| 291 fc = dirn * feval (fun,x,varargin{:}); | |
|
20231
83792dd9bcc1
Use in-place operators in m-files where possible.
Rik <rik@octave.org>
parents:
20165
diff
changeset
|
292 nf += 1; |
| 14895 | 293 if (fc > f(n)) |
| 294 vk = vc; fk = fc; | |
| 295 how = "contract,"; | |
| 296 else | |
| 297 for j = 2:n | |
| 298 V(:,j) = (V(:,1) + V(:,j))/2; | |
| 299 x(:) = V(:,j); | |
| 300 f(j) = dirn * feval (fun,x,varargin{:}); | |
| 301 endfor | |
|
20231
83792dd9bcc1
Use in-place operators in m-files where possible.
Rik <rik@octave.org>
parents:
20165
diff
changeset
|
302 nf += n-1; |
| 14895 | 303 vk = (V(:,1) + V(:,n+1))/2; |
| 304 x(:) = vk; | |
| 305 fk = dirn * feval (fun,x,varargin{:}); | |
|
20231
83792dd9bcc1
Use in-place operators in m-files where possible.
Rik <rik@octave.org>
parents:
20165
diff
changeset
|
306 nf += 1; |
| 14895 | 307 how = "shrink, "; |
| 308 endif | |
| 309 endif | |
| 310 V(:,n+1) = vk; | |
| 311 f(n+1) = fk; | |
| 312 [~,j] = sort(f); | |
| 313 j = j(n+1:-1:1); | |
| 314 f = f(j); | |
| 315 V = V(:,j); | |
| 316 | |
| 317 endwhile # End of outer (and only) loop. | |
| 318 | |
| 319 ## Finished. | |
| 320 if (trace) | |
| 321 fprintf (msg); | |
| 322 endif | |
| 323 x(:) = V(:,1); | |
| 324 | |
| 325 endfunction | |
| 326 | |
| 327 ## A helper function that evaluates a function and checks for bad results. | |
| 328 function y = guarded_eval (fun, x) | |
| 329 | |
| 330 y = fun (x); | |
| 331 | |
| 332 if (! (isreal (f))) | |
| 333 error ("fminsearch:notreal", "fminsearch: non-real value encountered"); | |
| 334 elseif (any (isnan (f(:)))) | |
| 335 error ("fminsearch:isnan", "fminsearch: NaN value encountered"); | |
| 336 elseif (any (isinf (f(:)))) | |
| 337 error ("fminsearch:isinf", "fminsearch: Inf value encountered"); | |
| 338 endif | |
| 339 | |
| 340 endfunction | |
| 341 | |
| 342 | |
| 343 %!demo | |
| 344 %! fcn = @(x) (x(1)-5).^2 + (x(2)-8).^4 | |
| 345 %! x0 = [0;0]; | |
| 346 %! [xmin, fval] = fminsearch (fcn, x0) | |
| 347 | |
| 348 %!assert (fminsearch (@sin, 3, optimset ("MaxIter", 3)), 4.8750, 1e-4) | |
| 349 %!assert (fminsearch (@sin, 3, optimset ("MaxIter", 30)), 4.7124, 1e-4) | |
| 350 %!shared c | |
| 351 %! c = 1.5; | |
| 352 %!assert (fminsearch (@(x) x(1).^2+c*x(2).^2,[1;1]), [0;0], 1e-4) | |
| 353 |