octave: scripts/optimization/fminunc.m annotate

annotate scripts/optimization/fminunc.m @ 27978:a4268efb7334

maint: Eliminate single "Author:" lines from code base. * color-picker.cc, color-picker.h, octave-qscintilla.cc, octave-qscintilla.h, octave-txt-lexer.cc, octave-txt-lexer.h, balance.cc, kron.cc, lookup.cc, ls-hdf5.cc, ls-mat5.cc, ls-oct-text.cc, sylvester.cc, tsearch.cc, jit-ir.cc, jit-ir.h, jit-typeinfo.cc, jit-typeinfo.h, jit-util.cc, jit-util.h, pt-jit.cc, pt-jit.h, Array-jit.cc, bsxfun.h, url-transfer.cc, url-transfer.h, acosd.m, acotd.m, acscd.m, asecd.m, asind.m, atand.m, cosd.m, cotd.m, cscd.m, secd.m, sind.m, tand.m, cart2pol.m, cart2sph.m, curl.m, del2.m, divergence.m, flip.m, pol2cart.m, randi.m, sortrows.m, sph2cart.m, griddata3.m, griddatan.m, rectint.m, voronoi.m, dialog.m, guidata.m, guihandles.m, __file_filter__.m, __fltk_file_filter__.m, __get_funcname__.m, __is_function__.m, __uigetdir_fltk__.m, __uigetfile_fltk__.m, __uiobject_split_args__.m, __uiputfile_fltk__.m, uibuttongroup.m, uicontextmenu.m, uicontrol.m, uigetdir.m, uigetfile.m, uimenu.m, uipanel.m, uipushtool.m, uiputfile.m, uiresume.m, uitoggletool.m, uitoolbar.m, uiwait.m, doc.m, autumn.m, bone.m, cmpermute.m, cmunique.m, cool.m, copper.m, cubehelix.m, flag.m, frame2im.m, hot.m, hsv.m, hsv2rgb.m, im2frame.m, imfinfo.m, imformats.m, iscolormap.m, jet.m, pink.m, prism.m, __imfinfo__.m, rainbow.m, rgb2hsv.m, spinmap.m, spring.m, viridis.m, white.m, winter.m, importdata.m, strmatch.m, bandwidth.m, isbanded.m, isdiag.m, istril.m, istriu.m, krylov.m, linsolve.m, logm.m, lscov.m, subspace.m, bunzip2.m, compare_versions.m, edit.m, fullfile.m, getfield.m, gunzip.m, inputParser.m, license.m, nargchk.m, narginchk.m, nthargout.m, python.m, setfield.m, tar.m, unpack.m, ver.m, zip.m, odeplot.m, fminbnd.m, fminunc.m, fsolve.m, fzero.m, glpk.m, humps.m, pathdef.m, savepath.m, diffuse.m, lighting.m, material.m, shading.m, specular.m, contour.m, ellipsoid.m, isocaps.m, isocolors.m, isonormals.m, isosurface.m, light.m, pcolor.m, __interp_cube__.m, __marching_cube__.m, __patch__.m, __stem__.m, reducepatch.m, reducevolume.m, ribbon.m, shrinkfaces.m, slice.m, stem.m, surf.m, surfl.m, tetramesh.m, waterfall.m, __actual_axis_position__.m, __next_line_color__.m, allchild.m, figure.m, findall.m, findobj.m, gcf.m, hggroup.m, isprop.m, ndgrid.m, __add_default_menu__.m, saveas.m, __splinefit__.m, prefdir.m, preferences.m, profexplore.m, profexport.m, profile.m, profshow.m, setdiff.m, freqz.m, freqz_plot.m, sinc.m, unwrap.m, pcr.m, qmr.m, sprandn.m, betaincinv.m, cosint.m, ellipke.m, factor.m, gammaincinv.m, legendre.m, nchoosek.m, primes.m, invhilb.m, toeplitz.m, erase.m, mat2str.m, strtrim.m, validatestring.m, fail.m, compare_plot_demos.m, dump_demos.m, html_compare_plot_demos.m, addtodate.m, datenum.m, jit.tst: Eliminate single "Author:" lines from code base.

author	Rik <rik@octave.org>
date	Tue, 21 Jan 2020 12:53:13 -0800
parents	bd51beb6205e
children	28de41192f3c 0a5b15007766

rev	line source
27923 bd51beb6205e update formatting of copyright notices John W. Eaton <jwe@octave.org> parents: 27919 diff changeset	1 ########################################################################
bd51beb6205e update formatting of copyright notices John W. Eaton <jwe@octave.org> parents: 27919 diff changeset	2 ##
27919 1891570abac8 update Octave Project Developers copyright for the new year John W. Eaton <jwe@octave.org> parents: 27918 diff changeset	3 ## Copyright (C) 2008-2020 The Octave Project Developers
27918 b442ec6dda5c use centralized file for copyright info for individual contributors John W. Eaton <jwe@octave.org> parents: 27586 diff changeset	4 ##
27923 bd51beb6205e update formatting of copyright notices John W. Eaton <jwe@octave.org> parents: 27919 diff changeset	5 ## See the file COPYRIGHT.md in the top-level directory of this
bd51beb6205e update formatting of copyright notices John W. Eaton <jwe@octave.org> parents: 27919 diff changeset	6 ## distribution or <https://octave.org/copyright/>.
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	7 ##
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	8 ## This file is part of Octave.
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	9 ##
24534 194eb4bd202b maint: Update punctuation for GPL v3 license text. Rik <rik@octave.org> parents: 23220 diff changeset	10 ## Octave is free software: you can redistribute it and/or modify it
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	11 ## under the terms of the GNU General Public License as published by
24534 194eb4bd202b maint: Update punctuation for GPL v3 license text. Rik <rik@octave.org> parents: 23220 diff changeset	12 ## the Free Software Foundation, either version 3 of the License, or
22755 3a2b891d0b33 maint: Standardize Copyright formatting. Rik <rik@octave.org> parents: 22323 diff changeset	13 ## (at your option) any later version.
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	14 ##
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	15 ## Octave is distributed in the hope that it will be useful, but
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	16 ## WITHOUT ANY WARRANTY; without even the implied warranty of
22755 3a2b891d0b33 maint: Standardize Copyright formatting. Rik <rik@octave.org> parents: 22323 diff changeset	17 ## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3a2b891d0b33 maint: Standardize Copyright formatting. Rik <rik@octave.org> parents: 22323 diff changeset	18 ## GNU General Public License for more details.
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	19 ##
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	20 ## You should have received a copy of the GNU General Public License
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	21 ## along with Octave; see the file COPYING. If not, see
24534 194eb4bd202b maint: Update punctuation for GPL v3 license text. Rik <rik@octave.org> parents: 23220 diff changeset	22 ## <https://www.gnu.org/licenses/>.
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	23 ##
27923 bd51beb6205e update formatting of copyright notices John W. Eaton <jwe@octave.org> parents: 27919 diff changeset	24 ########################################################################
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	25
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	26 ## -- texinfo --
20852 516bb87ea72e 2015 Code Sprint: remove class of function from docstring for all m-files. Rik <rik@octave.org> parents: 20735 diff changeset	27 ## @deftypefn {} {} fminunc (@var{fcn}, @var{x0})
516bb87ea72e 2015 Code Sprint: remove class of function from docstring for all m-files. Rik <rik@octave.org> parents: 20735 diff changeset	28 ## @deftypefnx {} {} fminunc (@var{fcn}, @var{x0}, @var{options})
516bb87ea72e 2015 Code Sprint: remove class of function from docstring for all m-files. Rik <rik@octave.org> parents: 20735 diff changeset	29 ## @deftypefnx {} {[@var{x}, @var{fval}, @var{info}, @var{output}, @var{grad}, @var{hess}] =} fminunc (@var{fcn}, @dots{})
10793 be55736a0783 Grammarcheck the documentation from m-files. Rik <octave@nomad.inbox5.com> parents: 10687 diff changeset	30 ## Solve an unconstrained optimization problem defined by the function
be55736a0783 Grammarcheck the documentation from m-files. Rik <octave@nomad.inbox5.com> parents: 10687 diff changeset	31 ## @var{fcn}.
14895 e0525ecf156e Add new function fminsearch.m Andy Adler <andy@analyti.ca> parents: 14868 diff changeset	32 ##
18609 923614060f1d fminunc.m: Improve documentation. Rik <rik@octave.org> parents: 18608 diff changeset	33 ## @code{fminunc} attempts to determine a vector @var{x} such that
20165 f1d0f506ee78 doc: Update more docstrings to have one sentence summary as first line. Rik <rik@octave.org> parents: 19833 diff changeset	34 ## @code{@var{fcn} (@var{x})} is a local minimum.
f1d0f506ee78 doc: Update more docstrings to have one sentence summary as first line. Rik <rik@octave.org> parents: 19833 diff changeset	35 ##
27390 baeed03c3766 doc: Use common verbiage to describe input FUN in optimization functions. Rik <rik@octave.org> parents: 27069 diff changeset	36 ## @var{fun} is a function handle, inline function, or string containing the
baeed03c3766 doc: Use common verbiage to describe input FUN in optimization functions. Rik <rik@octave.org> parents: 27069 diff changeset	37 ## name of the function to evaluate. @var{fcn} should accept a vector (array)
baeed03c3766 doc: Use common verbiage to describe input FUN in optimization functions. Rik <rik@octave.org> parents: 27069 diff changeset	38 ## defining the unknown variables, and return the objective function value,
baeed03c3766 doc: Use common verbiage to describe input FUN in optimization functions. Rik <rik@octave.org> parents: 27069 diff changeset	39 ## optionally with gradient.
baeed03c3766 doc: Use common verbiage to describe input FUN in optimization functions. Rik <rik@octave.org> parents: 27069 diff changeset	40 ##
20165 f1d0f506ee78 doc: Update more docstrings to have one sentence summary as first line. Rik <rik@octave.org> parents: 19833 diff changeset	41 ## @var{x0} determines a starting guess. The shape of @var{x0} is preserved in
f1d0f506ee78 doc: Update more docstrings to have one sentence summary as first line. Rik <rik@octave.org> parents: 19833 diff changeset	42 ## all calls to @var{fcn}, but otherwise is treated as a column vector.
f1d0f506ee78 doc: Update more docstrings to have one sentence summary as first line. Rik <rik@octave.org> parents: 19833 diff changeset	43 ##
26137 1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab. Rik <rik@octave.org> parents: 26118 diff changeset	44 ## @var{options} is a structure specifying additional parameters which
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab. Rik <rik@octave.org> parents: 26118 diff changeset	45 ## control the algorithm. Currently, @code{fminunc} recognizes these options:
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab. Rik <rik@octave.org> parents: 26118 diff changeset	46 ## @qcode{"AutoScaling"}, @qcode{"FinDiffType"}, @qcode{"FunValCheck"},
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab. Rik <rik@octave.org> parents: 26118 diff changeset	47 ## @qcode{"GradObj"}, @qcode{"MaxFunEvals"}, @qcode{"MaxIter"},
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab. Rik <rik@octave.org> parents: 26118 diff changeset	48 ## @qcode{"OutputFcn"}, @qcode{"TolFun"}, @qcode{"TolX"}, @qcode{"TypicalX"}.
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab. Rik <rik@octave.org> parents: 26118 diff changeset	49 ##
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab. Rik <rik@octave.org> parents: 26118 diff changeset	50 ## If @qcode{"AutoScaling"} is @qcode{"on"}, the variables will be
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab. Rik <rik@octave.org> parents: 26118 diff changeset	51 ## automatically scaled according to the column norms of the (estimated)
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab. Rik <rik@octave.org> parents: 26118 diff changeset	52 ## Jacobian. As a result, @qcode{"TolFun"} becomes scaling-independent. By
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab. Rik <rik@octave.org> parents: 26118 diff changeset	53 ## default, this option is @qcode{"off"} because it may sometimes deliver
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab. Rik <rik@octave.org> parents: 26118 diff changeset	54 ## unexpected (though mathematically correct) results.
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	55 ##
27585 757a7119e319 doc: Create en-dashes and em-dashes correctly in documentation. Rik <rik@octave.org> parents: 26376 diff changeset	56 ## If @qcode{"GradObj"} is @qcode{"on"}, it specifies that @var{fcn}---when
26137 1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab. Rik <rik@octave.org> parents: 26118 diff changeset	57 ## called with two output arguments---also returns the Jacobian matrix of
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab. Rik <rik@octave.org> parents: 26118 diff changeset	58 ## partial first derivatives at the requested point.
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab. Rik <rik@octave.org> parents: 26118 diff changeset	59 ##
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab. Rik <rik@octave.org> parents: 26118 diff changeset	60 ## @qcode{"MaxFunEvals"} proscribes the maximum number of function evaluations
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab. Rik <rik@octave.org> parents: 26118 diff changeset	61 ## before optimization is halted. The default value is
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab. Rik <rik@octave.org> parents: 26118 diff changeset	62 ## @code{100 * number_of_variables}, i.e., @code{100 * length (@var{x0})}.
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab. Rik <rik@octave.org> parents: 26118 diff changeset	63 ## The value must be a positive integer.
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab. Rik <rik@octave.org> parents: 26118 diff changeset	64 ##
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab. Rik <rik@octave.org> parents: 26118 diff changeset	65 ## @qcode{"MaxIter"} proscribes the maximum number of algorithm iterations
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab. Rik <rik@octave.org> parents: 26118 diff changeset	66 ## before optimization is halted. The default value is 400.
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab. Rik <rik@octave.org> parents: 26118 diff changeset	67 ## The value must be a positive integer.
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab. Rik <rik@octave.org> parents: 26118 diff changeset	68 ##
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab. Rik <rik@octave.org> parents: 26118 diff changeset	69 ## @qcode{"TolX"} specifies the termination tolerance for the unknown variables
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab. Rik <rik@octave.org> parents: 26118 diff changeset	70 ## @var{x}, while @qcode{"TolFun"} is a tolerance for the objective function
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab. Rik <rik@octave.org> parents: 26118 diff changeset	71 ## value @var{fval}. The default is @code{1e-6} for both options.
11587 c792872f8942 all script files: untabify and strip trailing whitespace John W. Eaton <jwe@octave.org> parents: 11523 diff changeset	72 ##
18609 923614060f1d fminunc.m: Improve documentation. Rik <rik@octave.org> parents: 18608 diff changeset	73 ## For a description of the other options, see @code{optimset}.
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	74 ##
18609 923614060f1d fminunc.m: Improve documentation. Rik <rik@octave.org> parents: 18608 diff changeset	75 ## On return, @var{x} is the location of the minimum and @var{fval} contains
20165 f1d0f506ee78 doc: Update more docstrings to have one sentence summary as first line. Rik <rik@octave.org> parents: 19833 diff changeset	76 ## the value of the objective function at @var{x}.
f1d0f506ee78 doc: Update more docstrings to have one sentence summary as first line. Rik <rik@octave.org> parents: 19833 diff changeset	77 ##
f1d0f506ee78 doc: Update more docstrings to have one sentence summary as first line. Rik <rik@octave.org> parents: 19833 diff changeset	78 ## @var{info} may be one of the following values:
11587 c792872f8942 all script files: untabify and strip trailing whitespace John W. Eaton <jwe@octave.org> parents: 11523 diff changeset	79 ##
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	80 ## @table @asis
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	81 ## @item 1
10793 be55736a0783 Grammarcheck the documentation from m-files. Rik <octave@nomad.inbox5.com> parents: 10687 diff changeset	82 ## Converged to a solution point. Relative gradient error is less than
18609 923614060f1d fminunc.m: Improve documentation. Rik <rik@octave.org> parents: 18608 diff changeset	83 ## specified by @code{TolFun}.
10821 693e22af08ae Grammarcheck documentation of m-files Rik <octave@nomad.inbox5.com> parents: 10793 diff changeset	84 ##
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	85 ## @item 2
18609 923614060f1d fminunc.m: Improve documentation. Rik <rik@octave.org> parents: 18608 diff changeset	86 ## Last relative step size was less than @code{TolX}.
10821 693e22af08ae Grammarcheck documentation of m-files Rik <octave@nomad.inbox5.com> parents: 10793 diff changeset	87 ##
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	88 ## @item 3
18609 923614060f1d fminunc.m: Improve documentation. Rik <rik@octave.org> parents: 18608 diff changeset	89 ## Last relative change in function value was less than @code{TolFun}.
10821 693e22af08ae Grammarcheck documentation of m-files Rik <octave@nomad.inbox5.com> parents: 10793 diff changeset	90 ##
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	91 ## @item 0
18816 e275d15c27b5 doc: Periodic spellcheck of documentation. Rik <rik@octave.org> parents: 18609 diff changeset	92 ## Iteration limit exceeded---either maximum number of algorithm iterations
18609 923614060f1d fminunc.m: Improve documentation. Rik <rik@octave.org> parents: 18608 diff changeset	93 ## @code{MaxIter} or maximum number of function evaluations @code{MaxFunEvals}.
923614060f1d fminunc.m: Improve documentation. Rik <rik@octave.org> parents: 18608 diff changeset	94 ##
923614060f1d fminunc.m: Improve documentation. Rik <rik@octave.org> parents: 18608 diff changeset	95 ## @item -1
18816 e275d15c27b5 doc: Periodic spellcheck of documentation. Rik <rik@octave.org> parents: 18609 diff changeset	96 ## Algorithm terminated by @code{OutputFcn}.
10821 693e22af08ae Grammarcheck documentation of m-files Rik <octave@nomad.inbox5.com> parents: 10793 diff changeset	97 ##
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	98 ## @item -3
11587 c792872f8942 all script files: untabify and strip trailing whitespace John W. Eaton <jwe@octave.org> parents: 11523 diff changeset	99 ## The trust region radius became excessively small.
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	100 ## @end table
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	101 ##
21546 f7f97d7e9294 doc: Wrap m-file docstrings to 79 characters + newline (80 total). Rik <rik@octave.org> parents: 21178 diff changeset	102 ## Optionally, @code{fminunc} can return a structure with convergence
f7f97d7e9294 doc: Wrap m-file docstrings to 79 characters + newline (80 total). Rik <rik@octave.org> parents: 21178 diff changeset	103 ## statistics (@var{output}), the output gradient (@var{grad}) at the
f7f97d7e9294 doc: Wrap m-file docstrings to 79 characters + newline (80 total). Rik <rik@octave.org> parents: 21178 diff changeset	104 ## solution @var{x}, and approximate Hessian (@var{hess}) at the solution
f7f97d7e9294 doc: Wrap m-file docstrings to 79 characters + newline (80 total). Rik <rik@octave.org> parents: 21178 diff changeset	105 ## @var{x}.
9627 5bcfa0b346e8 fix extra outputs in fminunc Jaroslav Hajek <highegg@gmail.com> parents: 9623 diff changeset	106 ##
20695 6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	107 ## Application Notes: If the objective function is a single nonlinear equation
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	108 ## of one variable then using @code{fminbnd} is usually a better choice.
20165 f1d0f506ee78 doc: Update more docstrings to have one sentence summary as first line. Rik <rik@octave.org> parents: 19833 diff changeset	109 ##
22092 d3cef63f79ac * fminunc.m: Fix incorrect reference to fminunc as fminsearch (bug #48479) Mike Miller <mtmiller@octave.org> parents: 21758 diff changeset	110 ## The algorithm used by @code{fminunc} is a gradient search which depends
20165 f1d0f506ee78 doc: Update more docstrings to have one sentence summary as first line. Rik <rik@octave.org> parents: 19833 diff changeset	111 ## on the objective function being differentiable. If the function has
f1d0f506ee78 doc: Update more docstrings to have one sentence summary as first line. Rik <rik@octave.org> parents: 19833 diff changeset	112 ## discontinuities it may be better to use a derivative-free algorithm such as
f1d0f506ee78 doc: Update more docstrings to have one sentence summary as first line. Rik <rik@octave.org> parents: 19833 diff changeset	113 ## @code{fminsearch}.
14895 e0525ecf156e Add new function fminsearch.m Andy Adler <andy@analyti.ca> parents: 14868 diff changeset	114 ## @seealso{fminbnd, fminsearch, optimset}
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	115 ## @end deftypefn
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	116
13027 b9a89ca0fb75 prevent optimization functions from setting ans in workspace at startup John W. Eaton <jwe@octave.org> parents: 12578 diff changeset	117 ## PKG_ADD: ## Discard result to avoid polluting workspace with ans at startup.
b9a89ca0fb75 prevent optimization functions from setting ans in workspace at startup John W. Eaton <jwe@octave.org> parents: 12578 diff changeset	118 ## PKG_ADD: [~] = __all_opts__ ("fminunc");
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	119
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	120 function [x, fval, info, output, grad, hess] = fminunc (fcn, x0, options = struct ())
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	121
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	122 ## Get default options if requested.
20695 6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	123 if (nargin == 1 && strcmp (fcn, "defaults"))
26138 804e18e3e320 Reenable query of optimization options (bugs #54952 and #55089). Kai T. Ohlhus <k.ohlhus@gmail.com> parents: 26137 diff changeset	124 x = struct ("AutoScaling", "off", "FunValCheck", "off",
804e18e3e320 Reenable query of optimization options (bugs #54952 and #55089). Kai T. Ohlhus <k.ohlhus@gmail.com> parents: 26137 diff changeset	125 "FinDiffType", "forward", "GradObj", "off",
804e18e3e320 Reenable query of optimization options (bugs #54952 and #55089). Kai T. Ohlhus <k.ohlhus@gmail.com> parents: 26137 diff changeset	126 "MaxFunEvals", [], "MaxIter", 400, "OutputFcn", [],
804e18e3e320 Reenable query of optimization options (bugs #54952 and #55089). Kai T. Ohlhus <k.ohlhus@gmail.com> parents: 26137 diff changeset	127 "TolFun", 1e-6, "TolX", 1e-6, "TypicalX", []);
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	128 return;
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	129 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	130
21171 2935d56203a4 Fix regressions caused by ismatrix definition change (partial fix bug #47036). Rik <rik@octave.org> parents: 20165 diff changeset	131 if (nargin < 2 \|\| nargin > 3 \|\| ! isnumeric (x0))
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	132 print_usage ();
11587 c792872f8942 all script files: untabify and strip trailing whitespace John W. Eaton <jwe@octave.org> parents: 11523 diff changeset	133 endif
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	134
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	135 if (ischar (fcn))
26118 7502fce4cd3a str2func: eliminate optional second "global" argument John W. Eaton <jwe@octave.org> parents: 25054 diff changeset	136 fcn = str2func (fcn);
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	137 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	138
20695 6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	139 xsz = size (x0);
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	140 n = numel (x0);
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	141
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	142 has_grad = strcmpi (optimget (options, "GradObj", "off"), "on");
26137 1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab. Rik <rik@octave.org> parents: 26118 diff changeset	143 cdif = strcmpi (optimget (options, "FinDiffType", "forward"), "central");
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	144 maxiter = optimget (options, "MaxIter", 400);
26137 1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab. Rik <rik@octave.org> parents: 26118 diff changeset	145 maxfev = optimget (options, "MaxFunEvals", 100*n);
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	146 outfcn = optimget (options, "OutputFcn");
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	147
20695 6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	148 ## Get scaling matrix using the TypicalX option. If set to "auto", the
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	149 ## scaling matrix is estimated using the Jacobian.
10201 5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default Jaroslav Hajek <highegg@gmail.com> parents: 9899 diff changeset	150 typicalx = optimget (options, "TypicalX");
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default Jaroslav Hajek <highegg@gmail.com> parents: 9899 diff changeset	151 if (isempty (typicalx))
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default Jaroslav Hajek <highegg@gmail.com> parents: 9899 diff changeset	152 typicalx = ones (n, 1);
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default Jaroslav Hajek <highegg@gmail.com> parents: 9899 diff changeset	153 endif
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default Jaroslav Hajek <highegg@gmail.com> parents: 9899 diff changeset	154 autoscale = strcmpi (optimget (options, "AutoScaling", "off"), "on");
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default Jaroslav Hajek <highegg@gmail.com> parents: 9899 diff changeset	155 if (! autoscale)
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default Jaroslav Hajek <highegg@gmail.com> parents: 9899 diff changeset	156 dg = 1 ./ typicalx;
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default Jaroslav Hajek <highegg@gmail.com> parents: 9899 diff changeset	157 endif
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default Jaroslav Hajek <highegg@gmail.com> parents: 9899 diff changeset	158
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	159 funvalchk = strcmpi (optimget (options, "FunValCheck", "off"), "on");
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	160
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	161 if (funvalchk)
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	162 ## Replace fcn with a guarded version.
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	163 fcn = @(x) guarded_eval (fcn, x);
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	164 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	165
20695 6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	166 ## These defaults are rather stringent. I think that normally, user
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	167 ## prefers accuracy to performance.
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	168
10201 5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default Jaroslav Hajek <highegg@gmail.com> parents: 9899 diff changeset	169 tolx = optimget (options, "TolX", 1e-7);
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default Jaroslav Hajek <highegg@gmail.com> parents: 9899 diff changeset	170 tolf = optimget (options, "TolFun", 1e-7);
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	171
9623 bc0739d02724 update initial TR step for fsolve and fminunc Jaroslav Hajek <highegg@gmail.com> parents: 9464 diff changeset	172 factor = 0.1;
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	173 ## FIXME: TypicalX corresponds to user scaling (???)
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	174 autodg = true;
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	175
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	176 niter = 1;
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	177 nfev = 0;
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	178
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	179 x = x0(:);
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	180 info = 0;
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	181
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	182 ## Initial evaluation.
20695 6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	183 fval = fcn (reshape (x, xsz));
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	184 n = length (x);
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	185
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	186 if (! isempty (outfcn))
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	187 optimvalues.iter = niter;
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	188 optimvalues.funccount = nfev;
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	189 optimvalues.fval = fval;
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	190 optimvalues.searchdirection = zeros (n, 1);
20695 6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	191 state = "init";
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	192 stop = outfcn (x, optimvalues, state);
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	193 if (stop)
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	194 info = -1;
22785 9c6661004167 error if break statement is in script file separate from loop (bug #39168) John W. Eaton <jwe@octave.org> parents: 22755 diff changeset	195 return;
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	196 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	197 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	198
21099 52af4092f863 For optimization scripts, correctly choose tolerance (eps) based on class of fun and X0. Rik <rik@octave.org> parents: 20852 diff changeset	199 if (isa (x0, "single") \|\| isa (fval, "single"))
52af4092f863 For optimization scripts, correctly choose tolerance (eps) based on class of fun and X0. Rik <rik@octave.org> parents: 20852 diff changeset	200 macheps = eps ("single");
52af4092f863 For optimization scripts, correctly choose tolerance (eps) based on class of fun and X0. Rik <rik@octave.org> parents: 20852 diff changeset	201 else
52af4092f863 For optimization scripts, correctly choose tolerance (eps) based on class of fun and X0. Rik <rik@octave.org> parents: 20852 diff changeset	202 macheps = eps ("double");
52af4092f863 For optimization scripts, correctly choose tolerance (eps) based on class of fun and X0. Rik <rik@octave.org> parents: 20852 diff changeset	203 endif
52af4092f863 For optimization scripts, correctly choose tolerance (eps) based on class of fun and X0. Rik <rik@octave.org> parents: 20852 diff changeset	204
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	205 nsuciter = 0;
9199 399884c9d4a1 import the step adaptation strategy from fsolve to fminunc Jaroslav Hajek <highegg@gmail.com> parents: 9084 diff changeset	206 lastratio = 0;
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	207
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	208 grad = [];
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	209
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	210 ## Outer loop.
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	211 while (niter < maxiter && nfev < maxfev && ! info)
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	212
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	213 grad0 = grad;
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	214
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	215 ## Calculate function value and gradient (possibly via FD).
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	216 if (has_grad)
20695 6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	217 [fval, grad] = fcn (reshape (x, xsz));
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	218 grad = grad(:);
20695 6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	219 nfev += 1;
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	220 else
20695 6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	221 grad = __fdjac__ (fcn, reshape (x, xsz), fval, typicalx, cdif)(:);
9212 6feb27c38da1 support central differences in fminunc and fsolve Jaroslav Hajek <highegg@gmail.com> parents: 9207 diff changeset	222 nfev += (1 + cdif) * length (x);
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	223 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	224
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	225 if (niter == 1)
11587 c792872f8942 all script files: untabify and strip trailing whitespace John W. Eaton <jwe@octave.org> parents: 11523 diff changeset	226 ## Initialize by identity matrix.
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	227 hesr = eye (n);
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	228 else
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	229 ## Use the damped BFGS formula.
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	230 y = grad - grad0;
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	231 sBs = sumsq (w);
20695 6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	232 Bs = hesr' * w;
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	233 sy = y' * s;
9633 ecc2c556f844 simplify damped BFGS formula Jaroslav Hajek <highegg@gmail.com> parents: 9631 diff changeset	234 theta = 0.8 / max (1 - sy / sBs, 0.8);
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	235 r = theta * y + (1-theta) * Bs;
20695 6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	236 hesr = cholupdate (hesr, r / sqrt (s' * r), "+");
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	237 [hesr, info] = cholupdate (hesr, Bs / sqrt (sBs), "-");
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	238 if (info)
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	239 hesr = eye (n);
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	240 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	241 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	242
10201 5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default Jaroslav Hajek <highegg@gmail.com> parents: 9899 diff changeset	243 if (autoscale)
20695 6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	244 ## Second derivatives approximate the Hessian.
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	245 d2f = norm (hesr, "columns").';
10201 5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default Jaroslav Hajek <highegg@gmail.com> parents: 9899 diff changeset	246 if (niter == 1)
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default Jaroslav Hajek <highegg@gmail.com> parents: 9899 diff changeset	247 dg = d2f;
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default Jaroslav Hajek <highegg@gmail.com> parents: 9899 diff changeset	248 else
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default Jaroslav Hajek <highegg@gmail.com> parents: 9899 diff changeset	249 ## FIXME: maybe fixed lower and upper bounds?
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default Jaroslav Hajek <highegg@gmail.com> parents: 9899 diff changeset	250 dg = max (0.1*dg, d2f);
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default Jaroslav Hajek <highegg@gmail.com> parents: 9899 diff changeset	251 endif
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default Jaroslav Hajek <highegg@gmail.com> parents: 9899 diff changeset	252 endif
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default Jaroslav Hajek <highegg@gmail.com> parents: 9899 diff changeset	253
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	254 if (niter == 1)
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	255 xn = norm (dg .* x);
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	256 ## FIXME: something better?
9628 73e6ad869f08 further correct initial TR step strategy Jaroslav Hajek <highegg@gmail.com> parents: 9627 diff changeset	257 delta = factor * max (xn, 1);
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	258 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	259
18609 923614060f1d fminunc.m: Improve documentation. Rik <rik@octave.org> parents: 18608 diff changeset	260 ## FIXME: why tolfnxn? If abs (e) ~ abs(x) * eps is a vector
21751 b571fc85953f maint: Use two spaces after period to indicate sentence break. Rik <rik@octave.org> parents: 21578 diff changeset	261 ## of perturbations of x, then norm (hesre) <= epsxn, i.e., by
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	262 ## tolf ~ eps we demand as much accuracy as we can expect.
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	263 if (norm (grad) <= tolfnxn)
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	264 info = 1;
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	265 break;
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	266 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	267
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	268 suc = false;
9207 25f50d2d76b3 improve TR updating strategy for fminunc and fsolve Jaroslav Hajek <highegg@gmail.com> parents: 9199 diff changeset	269 decfac = 0.5;
25f50d2d76b3 improve TR updating strategy for fminunc and fsolve Jaroslav Hajek <highegg@gmail.com> parents: 9199 diff changeset	270
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	271 ## Inner loop.
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	272 while (! suc && niter <= maxiter && nfev < maxfev && ! info)
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	273
9631 00958d0c4e3c split __dogleg__ > __doglegm__ Jaroslav Hajek <highegg@gmail.com> parents: 9628 diff changeset	274 s = - __doglegm__ (hesr, grad, dg, delta);
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	275
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	276 sn = norm (dg .* s);
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	277 if (niter == 1)
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	278 delta = min (delta, sn);
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	279 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	280
20695 6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	281 fval1 = fcn (reshape (x + s, xsz))(:);
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	282 nfev += 1;
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	283
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	284 if (fval1 < fval)
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	285 ## Scaled actual reduction.
19833 9fc020886ae9 maint: Clean up m-files to follow Octave coding conventions. Rik <rik@octave.org> parents: 19697 diff changeset	286 actred = (fval - fval1) / (abs (fval1) + abs (fval));
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	287 else
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	288 actred = -1;
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	289 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	290
20695 6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	291 w = hesr * s;
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	292 ## Scaled predicted reduction, and ratio.
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	293 t = 1/2 * sumsq (w) + grad'*s;
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	294 if (t < 0)
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	295 prered = -t/(abs (fval) + abs (fval + t));
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	296 ratio = actred / prered;
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	297 else
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	298 prered = 0;
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	299 ratio = 0;
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	300 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	301
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	302 ## Update delta.
14868 5d3a684236b0 maint: Use Octave coding conventions for cuddling parentheses in scripts directory Rik <octave@nomad.inbox5.com> parents: 14552 diff changeset	303 if (ratio < min (max (0.1, 0.8*lastratio), 0.9))
9207 25f50d2d76b3 improve TR updating strategy for fminunc and fsolve Jaroslav Hajek <highegg@gmail.com> parents: 9199 diff changeset	304 delta *= decfac;
25f50d2d76b3 improve TR updating strategy for fminunc and fsolve Jaroslav Hajek <highegg@gmail.com> parents: 9199 diff changeset	305 decfac ^= 1.4142;
20695 6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	306 if (delta <= 10machepsxn)
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	307 ## Trust region became uselessly small.
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	308 info = -3;
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	309 break;
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	310 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	311 else
9199 399884c9d4a1 import the step adaptation strategy from fsolve to fminunc Jaroslav Hajek <highegg@gmail.com> parents: 9084 diff changeset	312 lastratio = ratio;
9207 25f50d2d76b3 improve TR updating strategy for fminunc and fsolve Jaroslav Hajek <highegg@gmail.com> parents: 9199 diff changeset	313 decfac = 0.5;
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	314 if (abs (1-ratio) <= 0.1)
9207 25f50d2d76b3 improve TR updating strategy for fminunc and fsolve Jaroslav Hajek <highegg@gmail.com> parents: 9199 diff changeset	315 delta = 1.4142*sn;
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	316 elseif (ratio >= 0.5)
9207 25f50d2d76b3 improve TR updating strategy for fminunc and fsolve Jaroslav Hajek <highegg@gmail.com> parents: 9199 diff changeset	317 delta = max (delta, 1.4142*sn);
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	318 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	319 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	320
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	321 if (ratio >= 1e-4)
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	322 ## Successful iteration.
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	323 x += s;
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	324 xn = norm (dg .* x);
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	325 fval = fval1;
20735 418ae0cb752f Replace ++,-- with in-place operators for performance. Rik <rik@octave.org> parents: 20715 diff changeset	326 nsuciter += 1;
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	327 suc = true;
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	328 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	329
20695 6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	330 niter += 1;
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	331
18609 923614060f1d fminunc.m: Improve documentation. Rik <rik@octave.org> parents: 18608 diff changeset	332 ## FIXME: should outputfcn be called only after a successful iteration?
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	333 if (! isempty (outfcn))
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	334 optimvalues.iter = niter;
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	335 optimvalues.funccount = nfev;
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	336 optimvalues.fval = fval;
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	337 optimvalues.searchdirection = s;
20695 6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	338 state = "iter";
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	339 stop = outfcn (x, optimvalues, state);
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	340 if (stop)
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	341 info = -1;
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	342 break;
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	343 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	344 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	345
20695 6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	346 ## Tests for termination conditions. A mysterious place, anything
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	347 ## can happen if you change something here...
11587 c792872f8942 all script files: untabify and strip trailing whitespace John W. Eaton <jwe@octave.org> parents: 11523 diff changeset	348
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	349 ## The rule of thumb (which I'm not sure M*b is quite following)
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	350 ## is that for a tolerance that depends on scaling, only 0 makes
20695 6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	351 ## sense as a default value. But 0 usually means uselessly long
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	352 ## iterations, so we need scaling-independent tolerances wherever
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	353 ## possible.
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	354
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	355 ## The following tests done only after successful step.
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	356 if (ratio >= 1e-4)
20695 6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	357 ## This one is classic. Note that we use scaled variables again,
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	358 ## but compare to scaled step, so nothing bad.
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	359 if (sn <= tolx*xn)
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	360 info = 2;
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	361 ## Again a classic one.
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	362 elseif (actred < tolf)
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	363 info = 3;
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	364 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	365 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	366
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	367 endwhile
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	368 endwhile
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	369
20695 6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	370 ## When info != 1, recalculate the gradient and Hessian using the final x.
20694 a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101) José Vallet <jose.vallet@aalto.fi> parents: 20165 diff changeset	371 if (nargout > 4 && (info == -1 \|\| info == 2 \|\| info == 3))
20715 5b7643257978 Remove trailing whitespace at end of lines. Rik <rik@octave.org> parents: 20695 diff changeset	372 grad0 = grad;
20694 a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101) José Vallet <jose.vallet@aalto.fi> parents: 20165 diff changeset	373 if (has_grad)
20695 6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	374 [fval, grad] = fcn (reshape (x, xsz));
20694 a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101) José Vallet <jose.vallet@aalto.fi> parents: 20165 diff changeset	375 grad = grad(:);
a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101) José Vallet <jose.vallet@aalto.fi> parents: 20165 diff changeset	376 else
20695 6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	377 grad = __fdjac__ (fcn, reshape (x, xsz), fval, typicalx, cdif)(:);
20694 a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101) José Vallet <jose.vallet@aalto.fi> parents: 20165 diff changeset	378 endif
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	379
20694 a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101) José Vallet <jose.vallet@aalto.fi> parents: 20165 diff changeset	380 if (nargout > 5)
a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101) José Vallet <jose.vallet@aalto.fi> parents: 20165 diff changeset	381 ## Use the damped BFGS formula.
a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101) José Vallet <jose.vallet@aalto.fi> parents: 20165 diff changeset	382 y = grad - grad0;
a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101) José Vallet <jose.vallet@aalto.fi> parents: 20165 diff changeset	383 sBs = sumsq (w);
a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101) José Vallet <jose.vallet@aalto.fi> parents: 20165 diff changeset	384 Bs = hesr' * w;
a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101) José Vallet <jose.vallet@aalto.fi> parents: 20165 diff changeset	385 sy = y' * s;
a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101) José Vallet <jose.vallet@aalto.fi> parents: 20165 diff changeset	386 theta = 0.8 / max (1 - sy / sBs, 0.8);
a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101) José Vallet <jose.vallet@aalto.fi> parents: 20165 diff changeset	387 r = theta * y + (1-theta) * Bs;
a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101) José Vallet <jose.vallet@aalto.fi> parents: 20165 diff changeset	388 hesr = cholupdate (hesr, r / sqrt (s' * r), "+");
a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101) José Vallet <jose.vallet@aalto.fi> parents: 20165 diff changeset	389 hesr = cholupdate (hesr, Bs / sqrt (sBs), "-");
a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101) José Vallet <jose.vallet@aalto.fi> parents: 20165 diff changeset	390 endif
a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101) José Vallet <jose.vallet@aalto.fi> parents: 20165 diff changeset	391 ## Return the gradient in the same shape as x
20695 6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	392 grad = reshape (grad, xsz);
20694 a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101) José Vallet <jose.vallet@aalto.fi> parents: 20165 diff changeset	393 endif
a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101) José Vallet <jose.vallet@aalto.fi> parents: 20165 diff changeset	394
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	395 ## Restore original shapes.
20695 6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	396 x = reshape (x, xsz);
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	397
20695 6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	398 if (nargout > 3)
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	399 output.iterations = niter;
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	400 output.successful = nsuciter;
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	401 output.funcCount = nfev;
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	402 endif
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	403
9627 5bcfa0b346e8 fix extra outputs in fminunc Jaroslav Hajek <highegg@gmail.com> parents: 9623 diff changeset	404 if (nargout > 5)
5bcfa0b346e8 fix extra outputs in fminunc Jaroslav Hajek <highegg@gmail.com> parents: 9623 diff changeset	405 hess = hesr'*hesr;
5bcfa0b346e8 fix extra outputs in fminunc Jaroslav Hajek <highegg@gmail.com> parents: 9623 diff changeset	406 endif
5bcfa0b346e8 fix extra outputs in fminunc Jaroslav Hajek <highegg@gmail.com> parents: 9623 diff changeset	407
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	408 endfunction
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	409
18609 923614060f1d fminunc.m: Improve documentation. Rik <rik@octave.org> parents: 18608 diff changeset	410 ## A helper function that evaluates a function and checks for bad results.
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	411 function [fx, gx] = guarded_eval (fun, x)
21758 ffad2baa90f7 maint: Use newlines to make code more readable. Rik <rik@octave.org> parents: 21751 diff changeset	412
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	413 if (nargout > 1)
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	414 [fx, gx] = fun (x);
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	415 else
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	416 fx = fun (x);
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	417 gx = [];
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	418 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	419
14383 07c55bceca23 Fix guarded_eval() subfunction in fminunc (bug #35534). Olaf Till <olaf.till@uni-jena.de> parents: 14138 diff changeset	420 if (! (isreal (fx) && isreal (gx)))
27069 0a62d9a6aa2d Place Octave's warning and error IDs in to the "Octave" namespace (bug #56213). Rik <rik@octave.org> parents: 26376 diff changeset	421 error ("Octave:fminunc:notreal", "fminunc: non-real value encountered");
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	422 elseif (any (isnan (fx(:))))
27069 0a62d9a6aa2d Place Octave's warning and error IDs in to the "Octave" namespace (bug #56213). Rik <rik@octave.org> parents: 26376 diff changeset	423 error ("Octave:fminunc:isnan", "fminunc: NaN value encountered");
14386 59aab666f2bf Extend "FunValCheck" option to optimization routines to detect Inf values. Rik <octave@nomad.inbox5.com> parents: 14383 diff changeset	424 elseif (any (isinf (fx(:))))
27069 0a62d9a6aa2d Place Octave's warning and error IDs in to the "Octave" namespace (bug #56213). Rik <rik@octave.org> parents: 26376 diff changeset	425 error ("Octave:fminunc:isinf", "fminunc: Inf value encountered");
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	426 endif
21758 ffad2baa90f7 maint: Use newlines to make code more readable. Rik <rik@octave.org> parents: 21751 diff changeset	427
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	428 endfunction
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	429
14386 59aab666f2bf Extend "FunValCheck" option to optimization routines to detect Inf values. Rik <octave@nomad.inbox5.com> parents: 14383 diff changeset	430
20695 6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	431 %!function f = __rosenb__ (x)
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	432 %! n = length (x);
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	433 %! f = sumsq (1 - x(1:n-1)) + 100 * sumsq (x(2:n) - x(1:n-1).^2);
13305 63463570d9fe Add %!endfunction block keyword to test.m Rik <octave@nomad.inbox5.com> parents: 13027 diff changeset	434 %!endfunction
19833 9fc020886ae9 maint: Clean up m-files to follow Octave coding conventions. Rik <rik@octave.org> parents: 19697 diff changeset	435 %!
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	436 %!test
20695 6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	437 %! [x, fval, info, out] = fminunc (@__rosenb__, [5, -5]);
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	438 %! tol = 2e-5;
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	439 %! assert (info > 0);
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	440 %! assert (x, ones (1, 2), tol);
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	441 %! assert (fval, 0, tol);
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	442 %!test
20695 6faaab833605 fminunc.m: Clean up function to meet Octave coding standards. Rik <rik@octave.org> parents: 20694 diff changeset	443 %! [x, fval, info, out] = fminunc (@__rosenb__, zeros (1, 4));
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	444 %! tol = 2e-5;
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	445 %! assert (info > 0);
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	446 %! assert (x, ones (1, 4), tol);
b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	447 %! assert (fval, 0, tol);
19833 9fc020886ae9 maint: Clean up m-files to follow Octave coding conventions. Rik <rik@octave.org> parents: 19697 diff changeset	448
9fc020886ae9 maint: Clean up m-files to follow Octave coding conventions. Rik <rik@octave.org> parents: 19697 diff changeset	449 ## Test FunValCheck works correctly
26137 1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab. Rik <rik@octave.org> parents: 26118 diff changeset	450 %!assert (fminunc (@(x) x^2, 1, optimset ("FunValCheck", "on")), 0, 1e-6)
14386 59aab666f2bf Extend "FunValCheck" option to optimization routines to detect Inf values. Rik <octave@nomad.inbox5.com> parents: 14383 diff changeset	451 %!error <non-real value> fminunc (@(x) x + i, 1, optimset ("FunValCheck", "on"))
59aab666f2bf Extend "FunValCheck" option to optimization routines to detect Inf values. Rik <octave@nomad.inbox5.com> parents: 14383 diff changeset	452 %!error <NaN value> fminunc (@(x) x + NaN, 1, optimset ("FunValCheck", "on"))
59aab666f2bf Extend "FunValCheck" option to optimization routines to detect Inf values. Rik <octave@nomad.inbox5.com> parents: 14383 diff changeset	453 %!error <Inf value> fminunc (@(x) x + Inf, 1, optimset ("FunValCheck", "on"))
59aab666f2bf Extend "FunValCheck" option to optimization routines to detect Inf values. Rik <octave@nomad.inbox5.com> parents: 14383 diff changeset	454
9084 b7210faa3ed0 implement fminunc using factored trust-region BFGS Jaroslav Hajek <highegg@gmail.com> parents: diff changeset	455
9899 9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	456 ## Solve the double dogleg trust-region minimization problem:
9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	457 ## Minimize 1/2norm(rx)^2 subject to the constraint norm(d.*x) <= delta,
9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	458 ## x being a convex combination of the gauss-newton and scaled gradient.
9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	459
21578 683a1beee538 maint: Use "FIXME:" for all code blocks needing further attention. Rik <rik@octave.org> parents: 21546 diff changeset	460 ## FIXME: error checks
683a1beee538 maint: Use "FIXME:" for all code blocks needing further attention. Rik <rik@octave.org> parents: 21546 diff changeset	461 ## FIXME: handle singularity, or leave it up to mldivide?
9899 9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	462
9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	463 function x = __doglegm__ (r, g, d, delta)
21758 ffad2baa90f7 maint: Use newlines to make code more readable. Rik <rik@octave.org> parents: 21751 diff changeset	464
9899 9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	465 ## Get Gauss-Newton direction.
9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	466 b = r' \ g;
9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	467 x = r \ b;
9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	468 xn = norm (d .* x);
9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	469 if (xn > delta)
9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	470 ## GN is too big, get scaled gradient.
9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	471 s = g ./ d;
9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	472 sn = norm (s);
9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	473 if (sn > 0)
9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	474 ## Normalize and rescale.
9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	475 s = (s / sn) ./ d;
9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	476 ## Get the line minimizer in s direction.
9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	477 tn = norm (r*s);
9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	478 snm = (sn / tn) / tn;
9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	479 if (snm < delta)
10549 95c3e38098bf Untabify .m scripts Rik <code@nomad.inbox5.com> parents: 10201 diff changeset	480 ## Get the dogleg path minimizer.
9899 9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	481 bn = norm (b);
9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	482 dxn = delta/xn; snmd = snm/delta;
9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	483 t = (bn/sn) * (bn/xn) * snmd;
9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	484 t -= dxn * snmd^2 - sqrt ((t-dxn)^2 + (1-dxn^2)*(1-snmd^2));
9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	485 alpha = dxn*(1-snmd^2) / t;
9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	486 else
9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	487 alpha = 0;
9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	488 endif
9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	489 else
9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	490 alpha = delta / xn;
9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	491 snm = 0;
9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	492 endif
9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	493 ## Form the appropriate convex combination.
9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	494 x = alpha * x + ((1-alpha) * min (snm, delta)) * s;
9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	495 endif
21758 ffad2baa90f7 maint: Use newlines to make code more readable. Rik <rik@octave.org> parents: 21751 diff changeset	496
9899 9f25290a35e8 more private function and subfunction changes John W. Eaton <jwe@octave.org> parents: 9758 diff changeset	497 endfunction