Mercurial > octave

annotate scripts/optimization/fminunc.m @ 33625:d213a148b3f1 default tip @

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
ensure exp. terminal widget has focus at startup * main-window.cc (main_window): call focus_command_window only if event loop is idle by using a single shot timer * main-window.h: make focus_command_window a public slot
author Torsten Lilge <ttl-octave@mailbox.org>
date Sun, 26 May 2024 02:29:44 +0200
parents 0d2a6a831574
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
27923
bd51beb6205e update formatting of copyright notices
John W. Eaton <jwe@octave.org>
parents: 27919
diff changeset
1 ########################################################################
bd51beb6205e update formatting of copyright notices
John W. Eaton <jwe@octave.org>
parents: 27919
diff changeset
2 ##
32632
2e484f9f1f18 maint: update Octave Project Developers copyright for the new year
John W. Eaton <jwe@octave.org>
parents: 31706
diff changeset
3 ## Copyright (C) 2008-2024 The Octave Project Developers
27918
b442ec6dda5c use centralized file for copyright info for individual contributors
John W. Eaton <jwe@octave.org>
parents: 27586
diff changeset
4 ##
27923
bd51beb6205e update formatting of copyright notices
John W. Eaton <jwe@octave.org>
parents: 27919
diff changeset
5 ## See the file COPYRIGHT.md in the top-level directory of this
bd51beb6205e update formatting of copyright notices
John W. Eaton <jwe@octave.org>
parents: 27919
diff changeset
6 ## distribution or <https://octave.org/copyright/>.
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
7 ##
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
8 ## This file is part of Octave.
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
9 ##
24534
194eb4bd202b maint: Update punctuation for GPL v3 license text.
Rik <rik@octave.org>
parents: 23220
diff changeset
10 ## Octave is free software: you can redistribute it and/or modify it
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
11 ## under the terms of the GNU General Public License as published by
24534
194eb4bd202b maint: Update punctuation for GPL v3 license text.
Rik <rik@octave.org>
parents: 23220
diff changeset
12 ## the Free Software Foundation, either version 3 of the License, or
22755
3a2b891d0b33 maint: Standardize Copyright formatting.
Rik <rik@octave.org>
parents: 22323
diff changeset
13 ## (at your option) any later version.
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
14 ##
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
15 ## Octave is distributed in the hope that it will be useful, but
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
16 ## WITHOUT ANY WARRANTY; without even the implied warranty of
22755
3a2b891d0b33 maint: Standardize Copyright formatting.
Rik <rik@octave.org>
parents: 22323
diff changeset
17 ## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3a2b891d0b33 maint: Standardize Copyright formatting.
Rik <rik@octave.org>
parents: 22323
diff changeset
18 ## GNU General Public License for more details.
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
19 ##
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
20 ## You should have received a copy of the GNU General Public License
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
21 ## along with Octave; see the file COPYING. If not, see
24534
194eb4bd202b maint: Update punctuation for GPL v3 license text.
Rik <rik@octave.org>
parents: 23220
diff changeset
22 ## <https://www.gnu.org/licenses/>.
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
23 ##
27923
bd51beb6205e update formatting of copyright notices
John W. Eaton <jwe@octave.org>
parents: 27919
diff changeset
24 ########################################################################
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
25
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
26 ## -*- texinfo -*-
30875
5d3faba0342e doc: Ensure documentation lists output argument when it exists for all m-files.
Rik <rik@octave.org>
parents: 30564
diff changeset
27 ## @deftypefn {} {@var{x} =} fminunc (@var{fcn}, @var{x0})
5d3faba0342e doc: Ensure documentation lists output argument when it exists for all m-files.
Rik <rik@octave.org>
parents: 30564
diff changeset
28 ## @deftypefnx {} {@var{x} =} fminunc (@var{fcn}, @var{x0}, @var{options})
5d3faba0342e doc: Ensure documentation lists output argument when it exists for all m-files.
Rik <rik@octave.org>
parents: 30564
diff changeset
29 ## @deftypefnx {} {[@var{x}, @var{fval}] =} fminunc (@var{fcn}, @dots{})
5d3faba0342e doc: Ensure documentation lists output argument when it exists for all m-files.
Rik <rik@octave.org>
parents: 30564
diff changeset
30 ## @deftypefnx {} {[@var{x}, @var{fval}, @var{info}] =} fminunc (@var{fcn}, @dots{})
5d3faba0342e doc: Ensure documentation lists output argument when it exists for all m-files.
Rik <rik@octave.org>
parents: 30564
diff changeset
31 ## @deftypefnx {} {[@var{x}, @var{fval}, @var{info}, @var{output}] =} fminunc (@var{fcn}, @dots{})
5d3faba0342e doc: Ensure documentation lists output argument when it exists for all m-files.
Rik <rik@octave.org>
parents: 30564
diff changeset
32 ## @deftypefnx {} {[@var{x}, @var{fval}, @var{info}, @var{output}, @var{grad}] =} fminunc (@var{fcn}, @dots{})
20852
516bb87ea72e 2015 Code Sprint: remove class of function from docstring for all m-files.
Rik <rik@octave.org>
parents: 20735
diff changeset
33 ## @deftypefnx {} {[@var{x}, @var{fval}, @var{info}, @var{output}, @var{grad}, @var{hess}] =} fminunc (@var{fcn}, @dots{})
10793
be55736a0783 Grammarcheck the documentation from m-files.
Rik <octave@nomad.inbox5.com>
parents: 10687
diff changeset
34 ## Solve an unconstrained optimization problem defined by the function
be55736a0783 Grammarcheck the documentation from m-files.
Rik <octave@nomad.inbox5.com>
parents: 10687
diff changeset
35 ## @var{fcn}.
14895
e0525ecf156e Add new function fminsearch.m
Andy Adler <andy@analyti.ca>
parents: 14868
diff changeset
36 ##
18609
923614060f1d fminunc.m: Improve documentation.
Rik <rik@octave.org>
parents: 18608
diff changeset
37 ## @code{fminunc} attempts to determine a vector @var{x} such that
20165
f1d0f506ee78 doc: Update more docstrings to have one sentence summary as first line.
Rik <rik@octave.org>
parents: 19833
diff changeset
38 ## @code{@var{fcn} (@var{x})} is a local minimum.
f1d0f506ee78 doc: Update more docstrings to have one sentence summary as first line.
Rik <rik@octave.org>
parents: 19833
diff changeset
39 ##
30893
e1788b1a315f maint: Use "fcn" as preferred abbreviation for "function" in m-files.
Rik <rik@octave.org>
parents: 30875
diff changeset
40 ## @var{fcn} is a function handle, inline function, or string containing the
27390
baeed03c3766 doc: Use common verbiage to describe input FUN in optimization functions.
Rik <rik@octave.org>
parents: 27069
diff changeset
41 ## name of the function to evaluate. @var{fcn} should accept a vector (array)
baeed03c3766 doc: Use common verbiage to describe input FUN in optimization functions.
Rik <rik@octave.org>
parents: 27069
diff changeset
42 ## defining the unknown variables, and return the objective function value,
baeed03c3766 doc: Use common verbiage to describe input FUN in optimization functions.
Rik <rik@octave.org>
parents: 27069
diff changeset
43 ## optionally with gradient.
baeed03c3766 doc: Use common verbiage to describe input FUN in optimization functions.
Rik <rik@octave.org>
parents: 27069
diff changeset
44 ##
20165
f1d0f506ee78 doc: Update more docstrings to have one sentence summary as first line.
Rik <rik@octave.org>
parents: 19833
diff changeset
45 ## @var{x0} determines a starting guess. The shape of @var{x0} is preserved in
f1d0f506ee78 doc: Update more docstrings to have one sentence summary as first line.
Rik <rik@octave.org>
parents: 19833
diff changeset
46 ## all calls to @var{fcn}, but otherwise is treated as a column vector.
f1d0f506ee78 doc: Update more docstrings to have one sentence summary as first line.
Rik <rik@octave.org>
parents: 19833
diff changeset
47 ##
26137
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab.
Rik <rik@octave.org>
parents: 26118
diff changeset
48 ## @var{options} is a structure specifying additional parameters which
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab.
Rik <rik@octave.org>
parents: 26118
diff changeset
49 ## control the algorithm. Currently, @code{fminunc} recognizes these options:
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab.
Rik <rik@octave.org>
parents: 26118
diff changeset
50 ## @qcode{"AutoScaling"}, @qcode{"FinDiffType"}, @qcode{"FunValCheck"},
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab.
Rik <rik@octave.org>
parents: 26118
diff changeset
51 ## @qcode{"GradObj"}, @qcode{"MaxFunEvals"}, @qcode{"MaxIter"},
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab.
Rik <rik@octave.org>
parents: 26118
diff changeset
52 ## @qcode{"OutputFcn"}, @qcode{"TolFun"}, @qcode{"TolX"}, @qcode{"TypicalX"}.
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab.
Rik <rik@octave.org>
parents: 26118
diff changeset
53 ##
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab.
Rik <rik@octave.org>
parents: 26118
diff changeset
54 ## If @qcode{"AutoScaling"} is @qcode{"on"}, the variables will be
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab.
Rik <rik@octave.org>
parents: 26118
diff changeset
55 ## automatically scaled according to the column norms of the (estimated)
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab.
Rik <rik@octave.org>
parents: 26118
diff changeset
56 ## Jacobian. As a result, @qcode{"TolFun"} becomes scaling-independent. By
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab.
Rik <rik@octave.org>
parents: 26118
diff changeset
57 ## default, this option is @qcode{"off"} because it may sometimes deliver
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab.
Rik <rik@octave.org>
parents: 26118
diff changeset
58 ## unexpected (though mathematically correct) results.
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
59 ##
27585
757a7119e319 doc: Create en-dashes and em-dashes correctly in documentation.
Rik <rik@octave.org>
parents: 26376
diff changeset
60 ## If @qcode{"GradObj"} is @qcode{"on"}, it specifies that @var{fcn}---when
26137
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab.
Rik <rik@octave.org>
parents: 26118
diff changeset
61 ## called with two output arguments---also returns the Jacobian matrix of
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab.
Rik <rik@octave.org>
parents: 26118
diff changeset
62 ## partial first derivatives at the requested point.
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab.
Rik <rik@octave.org>
parents: 26118
diff changeset
63 ##
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab.
Rik <rik@octave.org>
parents: 26118
diff changeset
64 ## @qcode{"MaxFunEvals"} proscribes the maximum number of function evaluations
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab.
Rik <rik@octave.org>
parents: 26118
diff changeset
65 ## before optimization is halted. The default value is
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab.
Rik <rik@octave.org>
parents: 26118
diff changeset
66 ## @code{100 * number_of_variables}, i.e., @code{100 * length (@var{x0})}.
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab.
Rik <rik@octave.org>
parents: 26118
diff changeset
67 ## The value must be a positive integer.
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab.
Rik <rik@octave.org>
parents: 26118
diff changeset
68 ##
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab.
Rik <rik@octave.org>
parents: 26118
diff changeset
69 ## @qcode{"MaxIter"} proscribes the maximum number of algorithm iterations
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab.
Rik <rik@octave.org>
parents: 26118
diff changeset
70 ## before optimization is halted. The default value is 400.
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab.
Rik <rik@octave.org>
parents: 26118
diff changeset
71 ## The value must be a positive integer.
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab.
Rik <rik@octave.org>
parents: 26118
diff changeset
72 ##
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab.
Rik <rik@octave.org>
parents: 26118
diff changeset
73 ## @qcode{"TolX"} specifies the termination tolerance for the unknown variables
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab.
Rik <rik@octave.org>
parents: 26118
diff changeset
74 ## @var{x}, while @qcode{"TolFun"} is a tolerance for the objective function
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab.
Rik <rik@octave.org>
parents: 26118
diff changeset
75 ## value @var{fval}. The default is @code{1e-6} for both options.
11587
c792872f8942 all script files: untabify and strip trailing whitespace
John W. Eaton <jwe@octave.org>
parents: 11523
diff changeset
76 ##
28958
6fd6ad758b10 doc: Use @xref, @pxref rather than "see @code{}" in TexInfo.
Rik <rik@octave.org>
parents: 28789
diff changeset
77 ## For a description of the other options,
6fd6ad758b10 doc: Use @xref, @pxref rather than "see @code{}" in TexInfo.
Rik <rik@octave.org>
parents: 28789
diff changeset
78 ## @pxref{XREFoptimset,,@code{optimset}}.
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
79 ##
18609
923614060f1d fminunc.m: Improve documentation.
Rik <rik@octave.org>
parents: 18608
diff changeset
80 ## On return, @var{x} is the location of the minimum and @var{fval} contains
20165
f1d0f506ee78 doc: Update more docstrings to have one sentence summary as first line.
Rik <rik@octave.org>
parents: 19833
diff changeset
81 ## the value of the objective function at @var{x}.
f1d0f506ee78 doc: Update more docstrings to have one sentence summary as first line.
Rik <rik@octave.org>
parents: 19833
diff changeset
82 ##
f1d0f506ee78 doc: Update more docstrings to have one sentence summary as first line.
Rik <rik@octave.org>
parents: 19833
diff changeset
83 ## @var{info} may be one of the following values:
11587
c792872f8942 all script files: untabify and strip trailing whitespace
John W. Eaton <jwe@octave.org>
parents: 11523
diff changeset
84 ##
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
85 ## @table @asis
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
86 ## @item 1
10793
be55736a0783 Grammarcheck the documentation from m-files.
Rik <octave@nomad.inbox5.com>
parents: 10687
diff changeset
87 ## Converged to a solution point. Relative gradient error is less than
18609
923614060f1d fminunc.m: Improve documentation.
Rik <rik@octave.org>
parents: 18608
diff changeset
88 ## specified by @code{TolFun}.
10821
693e22af08ae Grammarcheck documentation of m-files
Rik <octave@nomad.inbox5.com>
parents: 10793
diff changeset
89 ##
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
90 ## @item 2
18609
923614060f1d fminunc.m: Improve documentation.
Rik <rik@octave.org>
parents: 18608
diff changeset
91 ## Last relative step size was less than @code{TolX}.
10821
693e22af08ae Grammarcheck documentation of m-files
Rik <octave@nomad.inbox5.com>
parents: 10793
diff changeset
92 ##
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
93 ## @item 3
18609
923614060f1d fminunc.m: Improve documentation.
Rik <rik@octave.org>
parents: 18608
diff changeset
94 ## Last relative change in function value was less than @code{TolFun}.
10821
693e22af08ae Grammarcheck documentation of m-files
Rik <octave@nomad.inbox5.com>
parents: 10793
diff changeset
95 ##
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
96 ## @item 0
18816
e275d15c27b5 doc: Periodic spellcheck of documentation.
Rik <rik@octave.org>
parents: 18609
diff changeset
97 ## Iteration limit exceeded---either maximum number of algorithm iterations
18609
923614060f1d fminunc.m: Improve documentation.
Rik <rik@octave.org>
parents: 18608
diff changeset
98 ## @code{MaxIter} or maximum number of function evaluations @code{MaxFunEvals}.
923614060f1d fminunc.m: Improve documentation.
Rik <rik@octave.org>
parents: 18608
diff changeset
99 ##
923614060f1d fminunc.m: Improve documentation.
Rik <rik@octave.org>
parents: 18608
diff changeset
100 ## @item -1
18816
e275d15c27b5 doc: Periodic spellcheck of documentation.
Rik <rik@octave.org>
parents: 18609
diff changeset
101 ## Algorithm terminated by @code{OutputFcn}.
10821
693e22af08ae Grammarcheck documentation of m-files
Rik <octave@nomad.inbox5.com>
parents: 10793
diff changeset
102 ##
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
103 ## @item -3
11587
c792872f8942 all script files: untabify and strip trailing whitespace
John W. Eaton <jwe@octave.org>
parents: 11523
diff changeset
104 ## The trust region radius became excessively small.
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
105 ## @end table
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
106 ##
21546
f7f97d7e9294 doc: Wrap m-file docstrings to 79 characters + newline (80 total).
Rik <rik@octave.org>
parents: 21178
diff changeset
107 ## Optionally, @code{fminunc} can return a structure with convergence
f7f97d7e9294 doc: Wrap m-file docstrings to 79 characters + newline (80 total).
Rik <rik@octave.org>
parents: 21178
diff changeset
108 ## statistics (@var{output}), the output gradient (@var{grad}) at the
f7f97d7e9294 doc: Wrap m-file docstrings to 79 characters + newline (80 total).
Rik <rik@octave.org>
parents: 21178
diff changeset
109 ## solution @var{x}, and approximate Hessian (@var{hess}) at the solution
f7f97d7e9294 doc: Wrap m-file docstrings to 79 characters + newline (80 total).
Rik <rik@octave.org>
parents: 21178
diff changeset
110 ## @var{x}.
9627
5bcfa0b346e8 fix extra outputs in fminunc
Jaroslav Hajek <highegg@gmail.com>
parents: 9623
diff changeset
111 ##
33323
0d2a6a831574 doc: note fminX parameter passing removal, document alternatives (bug #55742)
Nicholas R. Jankowski <jankowski.nicholas@gmail.com>
parents: 32632
diff changeset
112 ## Application Notes:
0d2a6a831574 doc: note fminX parameter passing removal, document alternatives (bug #55742)
Nicholas R. Jankowski <jankowski.nicholas@gmail.com>
parents: 32632
diff changeset
113 ## @enumerate
0d2a6a831574 doc: note fminX parameter passing removal, document alternatives (bug #55742)
Nicholas R. Jankowski <jankowski.nicholas@gmail.com>
parents: 32632
diff changeset
114 ## @item
0d2a6a831574 doc: note fminX parameter passing removal, document alternatives (bug #55742)
Nicholas R. Jankowski <jankowski.nicholas@gmail.com>
parents: 32632
diff changeset
115 ## If the objective function is a single nonlinear equation
20695
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
116 ## of one variable then using @code{fminbnd} is usually a better choice.
33323
0d2a6a831574 doc: note fminX parameter passing removal, document alternatives (bug #55742)
Nicholas R. Jankowski <jankowski.nicholas@gmail.com>
parents: 32632
diff changeset
117 ## @item
22092
d3cef63f79ac * fminunc.m: Fix incorrect reference to fminunc as fminsearch (bug #48479)
Mike Miller <mtmiller@octave.org>
parents: 21758
diff changeset
118 ## The algorithm used by @code{fminunc} is a gradient search which depends
20165
f1d0f506ee78 doc: Update more docstrings to have one sentence summary as first line.
Rik <rik@octave.org>
parents: 19833
diff changeset
119 ## on the objective function being differentiable. If the function has
f1d0f506ee78 doc: Update more docstrings to have one sentence summary as first line.
Rik <rik@octave.org>
parents: 19833
diff changeset
120 ## discontinuities it may be better to use a derivative-free algorithm such as
f1d0f506ee78 doc: Update more docstrings to have one sentence summary as first line.
Rik <rik@octave.org>
parents: 19833
diff changeset
121 ## @code{fminsearch}.
33323
0d2a6a831574 doc: note fminX parameter passing removal, document alternatives (bug #55742)
Nicholas R. Jankowski <jankowski.nicholas@gmail.com>
parents: 32632
diff changeset
122 ## @item
0d2a6a831574 doc: note fminX parameter passing removal, document alternatives (bug #55742)
Nicholas R. Jankowski <jankowski.nicholas@gmail.com>
parents: 32632
diff changeset
123 ## Use @ref{Anonymous Functions} to pass additional parameters to @var{fcn}.
0d2a6a831574 doc: note fminX parameter passing removal, document alternatives (bug #55742)
Nicholas R. Jankowski <jankowski.nicholas@gmail.com>
parents: 32632
diff changeset
124 ## For specific examples of doing so for @code{fminunc} and other
0d2a6a831574 doc: note fminX parameter passing removal, document alternatives (bug #55742)
Nicholas R. Jankowski <jankowski.nicholas@gmail.com>
parents: 32632
diff changeset
125 ## minimization functions see the @ref{Minimizers} section of the GNU Octave
0d2a6a831574 doc: note fminX parameter passing removal, document alternatives (bug #55742)
Nicholas R. Jankowski <jankowski.nicholas@gmail.com>
parents: 32632
diff changeset
126 ## manual.
0d2a6a831574 doc: note fminX parameter passing removal, document alternatives (bug #55742)
Nicholas R. Jankowski <jankowski.nicholas@gmail.com>
parents: 32632
diff changeset
127 ## @end enumerate
14895
e0525ecf156e Add new function fminsearch.m
Andy Adler <andy@analyti.ca>
parents: 14868
diff changeset
128 ## @seealso{fminbnd, fminsearch, optimset}
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
129 ## @end deftypefn
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
130
13027
b9a89ca0fb75 prevent optimization functions from setting ans in workspace at startup
John W. Eaton <jwe@octave.org>
parents: 12578
diff changeset
131 ## PKG_ADD: ## Discard result to avoid polluting workspace with ans at startup.
b9a89ca0fb75 prevent optimization functions from setting ans in workspace at startup
John W. Eaton <jwe@octave.org>
parents: 12578
diff changeset
132 ## PKG_ADD: [~] = __all_opts__ ("fminunc");
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
133
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
134 function [x, fval, info, output, grad, hess] = fminunc (fcn, x0, options = struct ())
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
135
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
136 ## Get default options if requested.
20695
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
137 if (nargin == 1 && strcmp (fcn, "defaults"))
26138
804e18e3e320 Reenable query of optimization options (bugs #54952 and #55089).
Kai T. Ohlhus <k.ohlhus@gmail.com>
parents: 26137
diff changeset
138 x = struct ("AutoScaling", "off", "FunValCheck", "off",
804e18e3e320 Reenable query of optimization options (bugs #54952 and #55089).
Kai T. Ohlhus <k.ohlhus@gmail.com>
parents: 26137
diff changeset
139 "FinDiffType", "forward", "GradObj", "off",
804e18e3e320 Reenable query of optimization options (bugs #54952 and #55089).
Kai T. Ohlhus <k.ohlhus@gmail.com>
parents: 26137
diff changeset
140 "MaxFunEvals", [], "MaxIter", 400, "OutputFcn", [],
804e18e3e320 Reenable query of optimization options (bugs #54952 and #55089).
Kai T. Ohlhus <k.ohlhus@gmail.com>
parents: 26137
diff changeset
141 "TolFun", 1e-6, "TolX", 1e-6, "TypicalX", []);
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
142 return;
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
143 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
144
28789
28de41192f3c Eliminate unneeded verification of nargin, nargout in m-files.
Rik <rik@octave.org>
parents: 27978
diff changeset
145 if (nargin < 2 || ! isnumeric (x0))
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
146 print_usage ();
11587
c792872f8942 all script files: untabify and strip trailing whitespace
John W. Eaton <jwe@octave.org>
parents: 11523
diff changeset
147 endif
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
148
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
149 if (ischar (fcn))
26118
7502fce4cd3a str2func: eliminate optional second "global" argument
John W. Eaton <jwe@octave.org>
parents: 25054
diff changeset
150 fcn = str2func (fcn);
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
151 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
152
20695
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
153 xsz = size (x0);
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
154 n = numel (x0);
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
155
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
156 has_grad = strcmpi (optimget (options, "GradObj", "off"), "on");
26137
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab.
Rik <rik@octave.org>
parents: 26118
diff changeset
157 cdif = strcmpi (optimget (options, "FinDiffType", "forward"), "central");
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
158 maxiter = optimget (options, "MaxIter", 400);
26137
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab.
Rik <rik@octave.org>
parents: 26118
diff changeset
159 maxfev = optimget (options, "MaxFunEvals", 100*n);
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
160 outfcn = optimget (options, "OutputFcn");
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
161
20695
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
162 ## Get scaling matrix using the TypicalX option. If set to "auto", the
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
163 ## scaling matrix is estimated using the Jacobian.
10201
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default
Jaroslav Hajek <highegg@gmail.com>
parents: 9899
diff changeset
164 typicalx = optimget (options, "TypicalX");
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default
Jaroslav Hajek <highegg@gmail.com>
parents: 9899
diff changeset
165 if (isempty (typicalx))
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default
Jaroslav Hajek <highegg@gmail.com>
parents: 9899
diff changeset
166 typicalx = ones (n, 1);
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default
Jaroslav Hajek <highegg@gmail.com>
parents: 9899
diff changeset
167 endif
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default
Jaroslav Hajek <highegg@gmail.com>
parents: 9899
diff changeset
168 autoscale = strcmpi (optimget (options, "AutoScaling", "off"), "on");
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default
Jaroslav Hajek <highegg@gmail.com>
parents: 9899
diff changeset
169 if (! autoscale)
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default
Jaroslav Hajek <highegg@gmail.com>
parents: 9899
diff changeset
170 dg = 1 ./ typicalx;
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default
Jaroslav Hajek <highegg@gmail.com>
parents: 9899
diff changeset
171 endif
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default
Jaroslav Hajek <highegg@gmail.com>
parents: 9899
diff changeset
172
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
173 funvalchk = strcmpi (optimget (options, "FunValCheck", "off"), "on");
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
174
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
175 if (funvalchk)
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
176 ## Replace fcn with a guarded version.
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
177 fcn = @(x) guarded_eval (fcn, x);
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
178 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
179
20695
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
180 ## These defaults are rather stringent. I think that normally, user
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
181 ## prefers accuracy to performance.
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
182
10201
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default
Jaroslav Hajek <highegg@gmail.com>
parents: 9899
diff changeset
183 tolx = optimget (options, "TolX", 1e-7);
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default
Jaroslav Hajek <highegg@gmail.com>
parents: 9899
diff changeset
184 tolf = optimget (options, "TolFun", 1e-7);
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
185
9623
bc0739d02724 update initial TR step for fsolve and fminunc
Jaroslav Hajek <highegg@gmail.com>
parents: 9464
diff changeset
186 factor = 0.1;
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
187 ## FIXME: TypicalX corresponds to user scaling (???)
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
188 autodg = true;
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
189
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
190 niter = 1;
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
191 nfev = 0;
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
192
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
193 x = x0(:);
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
194 info = 0;
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
195
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
196 ## Initial evaluation.
20695
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
197 fval = fcn (reshape (x, xsz));
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
198 n = length (x);
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
199
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
200 if (! isempty (outfcn))
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
201 optimvalues.iter = niter;
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
202 optimvalues.funccount = nfev;
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
203 optimvalues.fval = fval;
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
204 optimvalues.searchdirection = zeros (n, 1);
20695
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
205 state = "init";
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
206 stop = outfcn (x, optimvalues, state);
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
207 if (stop)
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
208 info = -1;
22785
9c6661004167 error if break statement is in script file separate from loop (bug #39168)
John W. Eaton <jwe@octave.org>
parents: 22755
diff changeset
209 return;
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
210 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
211 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
212
21099
52af4092f863 For optimization scripts, correctly choose tolerance (eps) based on class of fun and X0.
Rik <rik@octave.org>
parents: 20852
diff changeset
213 if (isa (x0, "single") || isa (fval, "single"))
52af4092f863 For optimization scripts, correctly choose tolerance (eps) based on class of fun and X0.
Rik <rik@octave.org>
parents: 20852
diff changeset
214 macheps = eps ("single");
52af4092f863 For optimization scripts, correctly choose tolerance (eps) based on class of fun and X0.
Rik <rik@octave.org>
parents: 20852
diff changeset
215 else
52af4092f863 For optimization scripts, correctly choose tolerance (eps) based on class of fun and X0.
Rik <rik@octave.org>
parents: 20852
diff changeset
216 macheps = eps ("double");
52af4092f863 For optimization scripts, correctly choose tolerance (eps) based on class of fun and X0.
Rik <rik@octave.org>
parents: 20852
diff changeset
217 endif
52af4092f863 For optimization scripts, correctly choose tolerance (eps) based on class of fun and X0.
Rik <rik@octave.org>
parents: 20852
diff changeset
218
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
219 nsuciter = 0;
9199
399884c9d4a1 import the step adaptation strategy from fsolve to fminunc
Jaroslav Hajek <highegg@gmail.com>
parents: 9084
diff changeset
220 lastratio = 0;
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
221
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
222 grad = [];
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
223
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
224 ## Outer loop.
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
225 while (niter < maxiter && nfev < maxfev && ! info)
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
226
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
227 grad0 = grad;
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
228
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
229 ## Calculate function value and gradient (possibly via FD).
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
230 if (has_grad)
20695
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
231 [fval, grad] = fcn (reshape (x, xsz));
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
232 grad = grad(:);
20695
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
233 nfev += 1;
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
234 else
20695
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
235 grad = __fdjac__ (fcn, reshape (x, xsz), fval, typicalx, cdif)(:);
9212
6feb27c38da1 support central differences in fminunc and fsolve
Jaroslav Hajek <highegg@gmail.com>
parents: 9207
diff changeset
236 nfev += (1 + cdif) * length (x);
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
237 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
238
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
239 if (niter == 1)
11587
c792872f8942 all script files: untabify and strip trailing whitespace
John W. Eaton <jwe@octave.org>
parents: 11523
diff changeset
240 ## Initialize by identity matrix.
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
241 hesr = eye (n);
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
242 else
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
243 ## Use the damped BFGS formula.
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
244 y = grad - grad0;
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
245 sBs = sumsq (w);
20695
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
246 Bs = hesr' * w;
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
247 sy = y' * s;
9633
ecc2c556f844 simplify damped BFGS formula
Jaroslav Hajek <highegg@gmail.com>
parents: 9631
diff changeset
248 theta = 0.8 / max (1 - sy / sBs, 0.8);
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
249 r = theta * y + (1-theta) * Bs;
20695
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
250 hesr = cholupdate (hesr, r / sqrt (s' * r), "+");
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
251 [hesr, info] = cholupdate (hesr, Bs / sqrt (sBs), "-");
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
252 if (info)
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
253 hesr = eye (n);
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
254 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
255 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
256
10201
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default
Jaroslav Hajek <highegg@gmail.com>
parents: 9899
diff changeset
257 if (autoscale)
20695
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
258 ## Second derivatives approximate the Hessian.
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
259 d2f = norm (hesr, "columns").';
10201
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default
Jaroslav Hajek <highegg@gmail.com>
parents: 9899
diff changeset
260 if (niter == 1)
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default
Jaroslav Hajek <highegg@gmail.com>
parents: 9899
diff changeset
261 dg = d2f;
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default
Jaroslav Hajek <highegg@gmail.com>
parents: 9899
diff changeset
262 else
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default
Jaroslav Hajek <highegg@gmail.com>
parents: 9899
diff changeset
263 ## FIXME: maybe fixed lower and upper bounds?
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default
Jaroslav Hajek <highegg@gmail.com>
parents: 9899
diff changeset
264 dg = max (0.1*dg, d2f);
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default
Jaroslav Hajek <highegg@gmail.com>
parents: 9899
diff changeset
265 endif
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default
Jaroslav Hajek <highegg@gmail.com>
parents: 9899
diff changeset
266 endif
5c66978f3fdf support TypicalX and AutoScaling in fsolve/fminunc, don't autoscale by default
Jaroslav Hajek <highegg@gmail.com>
parents: 9899
diff changeset
267
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
268 if (niter == 1)
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
269 xn = norm (dg .* x);
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
270 ## FIXME: something better?
9628
73e6ad869f08 further correct initial TR step strategy
Jaroslav Hajek <highegg@gmail.com>
parents: 9627
diff changeset
271 delta = factor * max (xn, 1);
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
272 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
273
18609
923614060f1d fminunc.m: Improve documentation.
Rik <rik@octave.org>
parents: 18608
diff changeset
274 ## FIXME: why tolf*n*xn? If abs (e) ~ abs(x) * eps is a vector
21751
b571fc85953f maint: Use two spaces after period to indicate sentence break.
Rik <rik@octave.org>
parents: 21578
diff changeset
275 ## of perturbations of x, then norm (hesr*e) <= eps*xn, i.e., by
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
276 ## tolf ~ eps we demand as much accuracy as we can expect.
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
277 if (norm (grad) <= tolf*n*xn)
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
278 info = 1;
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
279 break;
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
280 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
281
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
282 suc = false;
9207
25f50d2d76b3 improve TR updating strategy for fminunc and fsolve
Jaroslav Hajek <highegg@gmail.com>
parents: 9199
diff changeset
283 decfac = 0.5;
25f50d2d76b3 improve TR updating strategy for fminunc and fsolve
Jaroslav Hajek <highegg@gmail.com>
parents: 9199
diff changeset
284
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
285 ## Inner loop.
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
286 while (! suc && niter <= maxiter && nfev < maxfev && ! info)
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
287
9631
00958d0c4e3c split __dogleg__ > __doglegm__
Jaroslav Hajek <highegg@gmail.com>
parents: 9628
diff changeset
288 s = - __doglegm__ (hesr, grad, dg, delta);
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
289
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
290 sn = norm (dg .* s);
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
291 if (niter == 1)
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
292 delta = min (delta, sn);
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
293 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
294
20695
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
295 fval1 = fcn (reshape (x + s, xsz))(:);
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
296 nfev += 1;
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
297
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
298 if (fval1 < fval)
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
299 ## Scaled actual reduction.
19833
9fc020886ae9 maint: Clean up m-files to follow Octave coding conventions.
Rik <rik@octave.org>
parents: 19697
diff changeset
300 actred = (fval - fval1) / (abs (fval1) + abs (fval));
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
301 else
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
302 actred = -1;
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
303 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
304
20695
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
305 w = hesr * s;
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
306 ## Scaled predicted reduction, and ratio.
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
307 t = 1/2 * sumsq (w) + grad'*s;
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
308 if (t < 0)
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
309 prered = -t/(abs (fval) + abs (fval + t));
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
310 ratio = actred / prered;
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
311 else
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
312 prered = 0;
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
313 ratio = 0;
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
314 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
315
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
316 ## Update delta.
14868
5d3a684236b0 maint: Use Octave coding conventions for cuddling parentheses in scripts directory
Rik <octave@nomad.inbox5.com>
parents: 14552
diff changeset
317 if (ratio < min (max (0.1, 0.8*lastratio), 0.9))
9207
25f50d2d76b3 improve TR updating strategy for fminunc and fsolve
Jaroslav Hajek <highegg@gmail.com>
parents: 9199
diff changeset
318 delta *= decfac;
25f50d2d76b3 improve TR updating strategy for fminunc and fsolve
Jaroslav Hajek <highegg@gmail.com>
parents: 9199
diff changeset
319 decfac ^= 1.4142;
20695
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
320 if (delta <= 10*macheps*xn)
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
321 ## Trust region became uselessly small.
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
322 info = -3;
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
323 break;
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
324 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
325 else
9199
399884c9d4a1 import the step adaptation strategy from fsolve to fminunc
Jaroslav Hajek <highegg@gmail.com>
parents: 9084
diff changeset
326 lastratio = ratio;
9207
25f50d2d76b3 improve TR updating strategy for fminunc and fsolve
Jaroslav Hajek <highegg@gmail.com>
parents: 9199
diff changeset
327 decfac = 0.5;
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
328 if (abs (1-ratio) <= 0.1)
9207
25f50d2d76b3 improve TR updating strategy for fminunc and fsolve
Jaroslav Hajek <highegg@gmail.com>
parents: 9199
diff changeset
329 delta = 1.4142*sn;
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
330 elseif (ratio >= 0.5)
9207
25f50d2d76b3 improve TR updating strategy for fminunc and fsolve
Jaroslav Hajek <highegg@gmail.com>
parents: 9199
diff changeset
331 delta = max (delta, 1.4142*sn);
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
332 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
333 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
334
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
335 if (ratio >= 1e-4)
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
336 ## Successful iteration.
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
337 x += s;
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
338 xn = norm (dg .* x);
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
339 fval = fval1;
20735
418ae0cb752f Replace ++,-- with in-place operators for performance.
Rik <rik@octave.org>
parents: 20715
diff changeset
340 nsuciter += 1;
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
341 suc = true;
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
342 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
343
20695
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
344 niter += 1;
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
345
18609
923614060f1d fminunc.m: Improve documentation.
Rik <rik@octave.org>
parents: 18608
diff changeset
346 ## FIXME: should outputfcn be called only after a successful iteration?
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
347 if (! isempty (outfcn))
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
348 optimvalues.iter = niter;
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
349 optimvalues.funccount = nfev;
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
350 optimvalues.fval = fval;
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
351 optimvalues.searchdirection = s;
20695
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
352 state = "iter";
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
353 stop = outfcn (x, optimvalues, state);
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
354 if (stop)
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
355 info = -1;
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
356 break;
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
357 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
358 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
359
20695
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
360 ## Tests for termination conditions. A mysterious place, anything
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
361 ## can happen if you change something here...
11587
c792872f8942 all script files: untabify and strip trailing whitespace
John W. Eaton <jwe@octave.org>
parents: 11523
diff changeset
362
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
363 ## The rule of thumb (which I'm not sure M*b is quite following)
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
364 ## is that for a tolerance that depends on scaling, only 0 makes
20695
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
365 ## sense as a default value. But 0 usually means uselessly long
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
366 ## iterations, so we need scaling-independent tolerances wherever
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
367 ## possible.
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
368
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
369 ## The following tests done only after successful step.
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
370 if (ratio >= 1e-4)
20695
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
371 ## This one is classic. Note that we use scaled variables again,
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
372 ## but compare to scaled step, so nothing bad.
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
373 if (sn <= tolx*xn)
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
374 info = 2;
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
375 ## Again a classic one.
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
376 elseif (actred < tolf)
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
377 info = 3;
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
378 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
379 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
380
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
381 endwhile
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
382 endwhile
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
383
20695
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
384 ## When info != 1, recalculate the gradient and Hessian using the final x.
20694
a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101)
José Vallet <jose.vallet@aalto.fi>
parents: 20165
diff changeset
385 if (nargout > 4 && (info == -1 || info == 2 || info == 3))
20715
5b7643257978 Remove trailing whitespace at end of lines.
Rik <rik@octave.org>
parents: 20695
diff changeset
386 grad0 = grad;
20694
a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101)
José Vallet <jose.vallet@aalto.fi>
parents: 20165
diff changeset
387 if (has_grad)
20695
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
388 [fval, grad] = fcn (reshape (x, xsz));
20694
a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101)
José Vallet <jose.vallet@aalto.fi>
parents: 20165
diff changeset
389 grad = grad(:);
a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101)
José Vallet <jose.vallet@aalto.fi>
parents: 20165
diff changeset
390 else
20695
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
391 grad = __fdjac__ (fcn, reshape (x, xsz), fval, typicalx, cdif)(:);
20694
a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101)
José Vallet <jose.vallet@aalto.fi>
parents: 20165
diff changeset
392 endif
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
393
20694
a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101)
José Vallet <jose.vallet@aalto.fi>
parents: 20165
diff changeset
394 if (nargout > 5)
a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101)
José Vallet <jose.vallet@aalto.fi>
parents: 20165
diff changeset
395 ## Use the damped BFGS formula.
a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101)
José Vallet <jose.vallet@aalto.fi>
parents: 20165
diff changeset
396 y = grad - grad0;
a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101)
José Vallet <jose.vallet@aalto.fi>
parents: 20165
diff changeset
397 sBs = sumsq (w);
a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101)
José Vallet <jose.vallet@aalto.fi>
parents: 20165
diff changeset
398 Bs = hesr' * w;
a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101)
José Vallet <jose.vallet@aalto.fi>
parents: 20165
diff changeset
399 sy = y' * s;
a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101)
José Vallet <jose.vallet@aalto.fi>
parents: 20165
diff changeset
400 theta = 0.8 / max (1 - sy / sBs, 0.8);
a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101)
José Vallet <jose.vallet@aalto.fi>
parents: 20165
diff changeset
401 r = theta * y + (1-theta) * Bs;
a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101)
José Vallet <jose.vallet@aalto.fi>
parents: 20165
diff changeset
402 hesr = cholupdate (hesr, r / sqrt (s' * r), "+");
a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101)
José Vallet <jose.vallet@aalto.fi>
parents: 20165
diff changeset
403 hesr = cholupdate (hesr, Bs / sqrt (sBs), "-");
a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101)
José Vallet <jose.vallet@aalto.fi>
parents: 20165
diff changeset
404 endif
a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101)
José Vallet <jose.vallet@aalto.fi>
parents: 20165
diff changeset
405 ## Return the gradient in the same shape as x
20695
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
406 grad = reshape (grad, xsz);
20694
a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101)
José Vallet <jose.vallet@aalto.fi>
parents: 20165
diff changeset
407 endif
a7dbc4fc3732 fminunc.m: Correctly calculate gradient and hessian even when fminunc fails (bug #40101)
José Vallet <jose.vallet@aalto.fi>
parents: 20165
diff changeset
408
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
409 ## Restore original shapes.
20695
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
410 x = reshape (x, xsz);
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
411
20695
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
412 if (nargout > 3)
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
413 output.iterations = niter;
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
414 output.successful = nsuciter;
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
415 output.funcCount = nfev;
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
416 endif
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
417
9627
5bcfa0b346e8 fix extra outputs in fminunc
Jaroslav Hajek <highegg@gmail.com>
parents: 9623
diff changeset
418 if (nargout > 5)
5bcfa0b346e8 fix extra outputs in fminunc
Jaroslav Hajek <highegg@gmail.com>
parents: 9623
diff changeset
419 hess = hesr'*hesr;
5bcfa0b346e8 fix extra outputs in fminunc
Jaroslav Hajek <highegg@gmail.com>
parents: 9623
diff changeset
420 endif
5bcfa0b346e8 fix extra outputs in fminunc
Jaroslav Hajek <highegg@gmail.com>
parents: 9623
diff changeset
421
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
422 endfunction
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
423
18609
923614060f1d fminunc.m: Improve documentation.
Rik <rik@octave.org>
parents: 18608
diff changeset
424 ## A helper function that evaluates a function and checks for bad results.
30893
e1788b1a315f maint: Use "fcn" as preferred abbreviation for "function" in m-files.
Rik <rik@octave.org>
parents: 30875
diff changeset
425 function [fx, gx] = guarded_eval (fcn, x)
21758
ffad2baa90f7 maint: Use newlines to make code more readable.
Rik <rik@octave.org>
parents: 21751
diff changeset
426
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
427 if (nargout > 1)
30893
e1788b1a315f maint: Use "fcn" as preferred abbreviation for "function" in m-files.
Rik <rik@octave.org>
parents: 30875
diff changeset
428 [fx, gx] = fcn (x);
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
429 else
30893
e1788b1a315f maint: Use "fcn" as preferred abbreviation for "function" in m-files.
Rik <rik@octave.org>
parents: 30875
diff changeset
430 fx = fcn (x);
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
431 gx = [];
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
432 endif
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
433
14383
07c55bceca23 Fix guarded_eval() subfunction in fminunc (bug #35534).
Olaf Till <olaf.till@uni-jena.de>
parents: 14138
diff changeset
434 if (! (isreal (fx) && isreal (gx)))
27069
0a62d9a6aa2d Place Octave's warning and error IDs in to the "Octave" namespace (bug #56213).
Rik <rik@octave.org>
parents: 26376
diff changeset
435 error ("Octave:fminunc:notreal", "fminunc: non-real value encountered");
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
436 elseif (any (isnan (fx(:))))
27069
0a62d9a6aa2d Place Octave's warning and error IDs in to the "Octave" namespace (bug #56213).
Rik <rik@octave.org>
parents: 26376
diff changeset
437 error ("Octave:fminunc:isnan", "fminunc: NaN value encountered");
14386
59aab666f2bf Extend "FunValCheck" option to optimization routines to detect Inf values.
Rik <octave@nomad.inbox5.com>
parents: 14383
diff changeset
438 elseif (any (isinf (fx(:))))
27069
0a62d9a6aa2d Place Octave's warning and error IDs in to the "Octave" namespace (bug #56213).
Rik <rik@octave.org>
parents: 26376
diff changeset
439 error ("Octave:fminunc:isinf", "fminunc: Inf value encountered");
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
440 endif
21758
ffad2baa90f7 maint: Use newlines to make code more readable.
Rik <rik@octave.org>
parents: 21751
diff changeset
441
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
442 endfunction
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
443
14386
59aab666f2bf Extend "FunValCheck" option to optimization routines to detect Inf values.
Rik <octave@nomad.inbox5.com>
parents: 14383
diff changeset
444
20695
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
445 %!function f = __rosenb__ (x)
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
446 %! n = length (x);
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
447 %! f = sumsq (1 - x(1:n-1)) + 100 * sumsq (x(2:n) - x(1:n-1).^2);
13305
63463570d9fe Add %!endfunction block keyword to test.m
Rik <octave@nomad.inbox5.com>
parents: 13027
diff changeset
448 %!endfunction
19833
9fc020886ae9 maint: Clean up m-files to follow Octave coding conventions.
Rik <rik@octave.org>
parents: 19697
diff changeset
449 %!
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
450 %!test
20695
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
451 %! [x, fval, info, out] = fminunc (@__rosenb__, [5, -5]);
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
452 %! tol = 2e-5;
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
453 %! assert (info > 0);
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
454 %! assert (x, ones (1, 2), tol);
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
455 %! assert (fval, 0, tol);
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
456 %!test
20695
6faaab833605 fminunc.m: Clean up function to meet Octave coding standards.
Rik <rik@octave.org>
parents: 20694
diff changeset
457 %! [x, fval, info, out] = fminunc (@__rosenb__, zeros (1, 4));
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
458 %! tol = 2e-5;
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
459 %! assert (info > 0);
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
460 %! assert (x, ones (1, 4), tol);
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
461 %! assert (fval, 0, tol);
19833
9fc020886ae9 maint: Clean up m-files to follow Octave coding conventions.
Rik <rik@octave.org>
parents: 19697
diff changeset
462
9fc020886ae9 maint: Clean up m-files to follow Octave coding conventions.
Rik <rik@octave.org>
parents: 19697
diff changeset
463 ## Test FunValCheck works correctly
26137
1ae11ca7dceb fminunc.m: Change algorithm defaults to match Matlab.
Rik <rik@octave.org>
parents: 26118
diff changeset
464 %!assert (fminunc (@(x) x^2, 1, optimset ("FunValCheck", "on")), 0, 1e-6)
14386
59aab666f2bf Extend "FunValCheck" option to optimization routines to detect Inf values.
Rik <octave@nomad.inbox5.com>
parents: 14383
diff changeset
465 %!error <non-real value> fminunc (@(x) x + i, 1, optimset ("FunValCheck", "on"))
59aab666f2bf Extend "FunValCheck" option to optimization routines to detect Inf values.
Rik <octave@nomad.inbox5.com>
parents: 14383
diff changeset
466 %!error <NaN value> fminunc (@(x) x + NaN, 1, optimset ("FunValCheck", "on"))
59aab666f2bf Extend "FunValCheck" option to optimization routines to detect Inf values.
Rik <octave@nomad.inbox5.com>
parents: 14383
diff changeset
467 %!error <Inf value> fminunc (@(x) x + Inf, 1, optimset ("FunValCheck", "on"))
59aab666f2bf Extend "FunValCheck" option to optimization routines to detect Inf values.
Rik <octave@nomad.inbox5.com>
parents: 14383
diff changeset
468
9084
b7210faa3ed0 implement fminunc using factored trust-region BFGS
Jaroslav Hajek <highegg@gmail.com>
parents:
diff changeset
469
9899
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
470 ## Solve the double dogleg trust-region minimization problem:
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
471 ## Minimize 1/2*norm(r*x)^2 subject to the constraint norm(d.*x) <= delta,
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
472 ## x being a convex combination of the gauss-newton and scaled gradient.
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
473
21578
683a1beee538 maint: Use "FIXME:" for all code blocks needing further attention.
Rik <rik@octave.org>
parents: 21546
diff changeset
474 ## FIXME: error checks
683a1beee538 maint: Use "FIXME:" for all code blocks needing further attention.
Rik <rik@octave.org>
parents: 21546
diff changeset
475 ## FIXME: handle singularity, or leave it up to mldivide?
9899
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
476
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
477 function x = __doglegm__ (r, g, d, delta)
21758
ffad2baa90f7 maint: Use newlines to make code more readable.
Rik <rik@octave.org>
parents: 21751
diff changeset
478
9899
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
479 ## Get Gauss-Newton direction.
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
480 b = r' \ g;
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
481 x = r \ b;
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
482 xn = norm (d .* x);
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
483 if (xn > delta)
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
484 ## GN is too big, get scaled gradient.
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
485 s = g ./ d;
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
486 sn = norm (s);
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
487 if (sn > 0)
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
488 ## Normalize and rescale.
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
489 s = (s / sn) ./ d;
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
490 ## Get the line minimizer in s direction.
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
491 tn = norm (r*s);
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
492 snm = (sn / tn) / tn;
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
493 if (snm < delta)
10549
95c3e38098bf Untabify .m scripts
Rik <code@nomad.inbox5.com>
parents: 10201
diff changeset
494 ## Get the dogleg path minimizer.
9899
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
495 bn = norm (b);
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
496 dxn = delta/xn; snmd = snm/delta;
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
497 t = (bn/sn) * (bn/xn) * snmd;
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
498 t -= dxn * snmd^2 - sqrt ((t-dxn)^2 + (1-dxn^2)*(1-snmd^2));
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
499 alpha = dxn*(1-snmd^2) / t;
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
500 else
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
501 alpha = 0;
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
502 endif
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
503 else
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
504 alpha = delta / xn;
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
505 snm = 0;
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
506 endif
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
507 ## Form the appropriate convex combination.
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
508 x = alpha * x + ((1-alpha) * min (snm, delta)) * s;
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
509 endif
21758
ffad2baa90f7 maint: Use newlines to make code more readable.
Rik <rik@octave.org>
parents: 21751
diff changeset
510
9899
9f25290a35e8 more private function and subfunction changes
John W. Eaton <jwe@octave.org>
parents: 9758
diff changeset
511 endfunction