Mercurial > octave
view doc/interpreter/testfun.txi @ 33584:3fe954c2fd25 default tip @
maint: merge stable to default
author | Rik <rik@octave.org> |
---|---|
date | Mon, 13 May 2024 11:41:11 -0700 |
parents | 0f9fe0289e9b |
children |
line wrap: on
line source
@c Copyright (C) 2002-2024 The Octave Project Developers @c @c This file is part of Octave. @c @c Octave is free software: you can redistribute it and/or modify it @c under the terms of the GNU General Public License as published by @c the Free Software Foundation, either version 3 of the License, or @c (at your option) any later version. @c @c Octave is distributed in the hope that it will be useful, but @c WITHOUT ANY WARRANTY; without even the implied warranty of @c MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the @c GNU General Public License for more details. @c @c You should have received a copy of the GNU General Public License @c along with Octave; see the file COPYING. If not, see @c <https://www.gnu.org/licenses/>. @node Test and Demo Functions @appendix Test and Demo Functions @cindex test functions Octave includes a number of functions to allow the integration of testing and demonstration code in the source code of the functions themselves. @menu * Test Functions:: * Demonstration Functions:: @end menu @node Test Functions @section Test Functions @DOCSTRING(test) @code{test} scans the named script file looking for lines which start with the identifier @samp{%!}. The prefix is stripped off and the rest of the line is processed through the Octave interpreter. If the code generates an error, then the test is said to fail. Since @code{eval()} will stop at the first error it encounters, you must divide your tests up into blocks, with anything in a separate block evaluated separately. Blocks are introduced by valid keywords like @code{test}, @code{function}, or @code{assert} immediately following @samp{%!}. A block is defined by indentation as in Python. Lines beginning with @samp{%!<whitespace>} are part of the preceding block. For example: @example @group %!test error ("this test fails!") %!test "test doesn't fail. it doesn't generate an error" @end group @end example When a test fails, you will see something like: @example @group ***** test error ("this test fails!") !!!!! test failed this test fails! @end group @end example Generally, to test if something works, you want to assert that it produces a correct value. A real test might look something like @example @group %!test %! @var{a} = [1, 2, 3; 4, 5, 6]; B = [1; 2]; %! expect = [ @var{a} ; 2*@var{a} ]; %! get = kron (@var{b}, @var{a}); %! if (any (size (expect) != size (get))) %! error ("wrong size: expected %d,%d but got %d,%d", %! size (expect), size (get)); %! elseif (any (any (expect != get))) %! error ("didn't get what was expected."); %! endif @end group @end example To make the process easier, use the @code{assert} function. For example, with @code{assert} the previous test is reduced to: @example @group %!test %! @var{a} = [1, 2, 3; 4, 5, 6]; @var{b} = [1; 2]; %! assert (kron (@var{b}, @var{a}), [ @var{a}; 2*@var{a} ]); @end group @end example @code{assert} can accept a tolerance so that you can compare results absolutely or relatively. For example, the following all succeed: @example @group %!test assert (1+eps, 1, 2*eps) # absolute error %!test assert (100+100*eps, 100, -2*eps) # relative error @end group @end example You can also do the comparison yourself, but still have assert generate the error: @example @group %!test assert (isempty ([])) %!test assert ([1, 2; 3, 4] > 0) @end group @end example Because @code{assert} is so frequently used alone in a test block, there is a shorthand form: @example %!assert (@dots{}) @end example @noindent which is equivalent to: @example %!test assert (@dots{}) @end example Occasionally a block of tests will depend on having optional functionality in Octave. Before testing such blocks the availability of the required functionality must be checked. A @code{%!testif HAVE_XXX} block will only be run if Octave was compiled with functionality @samp{HAVE_XXX}. For example, the sparse single value decomposition, @code{svds()}, depends on having the @sc{arpack} library. All of the tests for @code{svds} begin with @example %!testif HAVE_ARPACK @end example @noindent Review @file{config.h} or @code{__octave_config_info__ ("build_features")} to see some of the possible values to check. Sometimes during development there is a test that should work but is known to fail. You still want to leave the test in because when the final code is ready the test should pass, but you may not be able to fix it immediately. To avoid unnecessary bug reports for these known failures, mark the block with @code{xtest} rather than @code{test}: @example @group %!xtest assert (1==0) %!xtest fail ("success=1", "error") @end group @end example @noindent In this case, the test will run and any failure will be reported. However, testing is not aborted and subsequent test blocks will be processed normally. Another use of @code{xtest} is for statistical tests which should pass most of the time but are known to fail occasionally. Each block is evaluated in its own function environment, which means that variables defined in one block are not automatically shared with other blocks. If you do want to share variables, then you must declare them as @code{shared} before you use them. For example, the following declares the variable @var{a}, gives it an initial value (default is empty), and then uses it in several subsequent tests. @example @group %!shared @var{a} %! @var{a} = [1, 2, 3; 4, 5, 6]; %!assert (kron ([1; 2], @var{a}), [ @var{a}; 2*@var{a} ]) %!assert (kron ([1, 2], @var{a}), [ @var{a}, 2*@var{a} ]) %!assert (kron ([1,2; 3,4], @var{a}), [ @var{a},2*@var{a}; 3*@var{a},4*@var{a} ]) @end group @end example You can share several variables at the same time: @example %!shared @var{a}, @var{b} @end example Modifications to shared variables persist from one test to the next @strong{only} if the test succeeds. Thus, if one test modifies a shared variable, later tests cannot know which value of the shared variable to expect because the pass/fail status of earlier tests is unknown. For this reason, it is not recommended to modify shared variables in tests. You can also share test functions: @example @group %!function @var{a} = fn (@var{b}) %! @var{a} = 2*@var{b}; %!endfunction %!assert (fn(2), 4) @end group @end example Note that all previous variables and values are lost when a new shared block is declared. Remember that @code{%!function} begins a new block and that @code{%!endfunction} ends this block. Be aware that until a new block is started, lines starting with @samp{%!<space>} will be discarded as comments. The following is nearly identical to the example above, but does nothing. @example @group %!function @var{a} = fn (@var{b}) %! @var{a} = 2*@var{b}; %!endfunction %! assert (fn(2), 4) @end group @end example @noindent Because there is a space after @samp{%!} the @code{assert} statement does not begin a new block and this line is treated as a comment. Error and warning blocks are like test blocks, but they only succeed if the code generates an error. You can check the text of the error is correct using an optional regular expression @code{<pattern>}. For example: @example %!error <passes!> error ("this test passes!") @end example If the code doesn't generate an error, the test fails. For example: @example %!error "this is an error because it succeeds." @end example @noindent produces @example @group ***** error "this is an error because it succeeds." !!!!! test failed: no error @end group @end example It is important to automate the tests as much as possible, however some tests require user interaction. These can be isolated into demo blocks, which if you are in batch mode, are only run when called with @code{demo} or the @code{verbose} option to @code{test}. The code is displayed before it is executed. For example, @example @group %!demo %! @var{t} = [0:0.01:2*pi]; @var{x} = sin (@var{t}); %! plot (@var{t}, @var{x}); %! # you should now see a sine wave in your figure window @end group @end example @noindent produces @example @group funcname example 1: @var{t} = [0:0.01:2*pi]; @var{x} = sin (@var{t}); plot (@var{t}, @var{x}); # you should now see a sine wave in your figure window Press <enter> to continue: @end group @end example Note that demo blocks cannot use any shared variables. This is so that they can be executed by themselves, ignoring all other tests. If you want to temporarily disable a test block, put @code{#} in place of the block type. This creates a comment block which is echoed in the log file but not executed. For example: @example @group %!#demo %! @var{t} = [0:0.01:2*pi]; @var{x} = sin (@var{t}); %! plot (@var{t}, @var{x}); %! # you should now see a sine wave in your figure window @end group @end example @noindent The following trivial code snippet provides examples for the use of fail, assert, error, and xtest: @example @group function @var{output} = must_be_zero (@var{input}) if (@var{input} != 0) error ("Nonzero input!") endif @var{output} = @var{input}; endfunction %!fail ("must_be_zero (1)") %!assert (must_be_zero (0), 0) %!error <Nonzero> must_be_zero (1) %!xtest error ("This code generates an error") @end group @end example @noindent When putting this in a file @file{must_be_zero.m}, and running the test, we see @example @group test must_be_zero verbose @result{} >>>>> /path/to/must_be_zero.m ***** fail ("must_be_zero (1)") ***** assert (must_be_zero (0), 0) ***** error <Nonzero> must_be_zero (1) ***** xtest error ("This code generates an error") !!!!! known failure This code generates an error PASSES 3 out of 4 tests (1 expected failure) @end group @end example @subsubheading Block type summary: @table @code @item %!test @itemx %!test <MESSAGE> Check that entire block is correct. If @code{<MESSAGE>} is present, the test block is interpreted as for @code{xtest}. @item %!testif HAVE_XXX @itemx %!testif HAVE_XXX, HAVE_YYY, @dots{} @itemx %!testif HAVE_XXX, HAVE_YYY @dots{}; RUNTIME_COND @itemx %!testif @dots{} <MESSAGE> Check block only if Octave was compiled with feature @w{@code{HAVE_XXX}}. @w{@code{RUNTIME_COND}}@ is an optional expression to evaluate to check whether some condition is met when the test is executed. If @w{@code{RUNTIME_COND}}@ is false, the test is skipped. If @code{<MESSAGE>} is present, the test block is interpreted as for @code{xtest}. @item %!xtest @itemx %!xtest <MESSAGE> Check block, report a test failure but do not abort testing. If @code{<MESSAGE>} is present, then the text of the message is displayed if the test fails, like this: @example !!!!! known bug: MESSAGE @end example @noindent If the message is an integer, it is interpreted as a bug ID for the Octave bug tracker and reported as @example !!!!! known bug: https://octave.org/testfailure/?BUG-ID @end example @noindent in which BUG-ID is the integer bug number. The intent is to allow clearer documentation of known problems. @noindent If @code{MESSAGE} is an integer preceeded by an asterisk (e.g., @code{*12345}), it is interpreted as the id for a bug report that has been closed. That usually means that the issue probed in this test has been resolved. If such tests are failing, they are reported as regressions by the @code{test} function: @example !!!!! regression: https://octave.org/testfailure/?BUG-ID @end example @item %!error @itemx %!error <MESSAGE> @itemx %!warning @itemx %!warning <MESSAGE> Check for correct error or warning message. If @code{<MESSAGE>} is supplied it is interpreted as a regular expression pattern that is expected to match the error or warning message. @item %!demo Demo only executes in interactive mode. @item %!# Comment. Ignore everything within the block @item %!shared x,y,z Declare variables for use in multiple tests. @item %!function Define a function for use in multiple tests. @item %!endfunction Close a function definition. @item %!assert (x, y, tol) @item %!assert <MESSAGE> (x, y, tol) @item %!fail (CODE, PATTERN) @item %!fail <MESSAGE> (CODE, PATTERN) Shorthand for @code{%!test assert (x, y, tol)} or @code{%!test fail (CODE, PATTERN)}. If @code{<MESSAGE>} is present, the test block is interpreted as for @code{xtest}. @end table @anchor{test-message-anchor} When coding tests the Octave convention is that lines that begin with a block type do not have a semicolon at the end. Any code that is within a block, however, is normal Octave code and usually will have a trailing semicolon. For example, @example @group ## bare block instantiation %!assert (sin (0), 0) @end group @end example @noindent but @example @group ## test block with normal Octave code %!test %! assert (sin (0), 0); @end group @end example You can also create test scripts for built-in functions and your own C++ functions. To do so, put a file with the bare function name (no .m extension) in a directory in the load path and it will be discovered by the @code{test} function. Alternatively, you can embed tests directly in your C++ code: @example @group /* %!test disp ("this is a test") */ @end group @end example @noindent or @example @group #if 0 %!test disp ("this is a test") #endif @end group @end example @noindent However, in this case the raw source code will need to be on the load path and the user will have to remember to type @code{test ("funcname.cc")}. @DOCSTRING(assert) @DOCSTRING(fail) @node Demonstration Functions @section Demonstration Functions @DOCSTRING(demo) @DOCSTRING(example) @DOCSTRING(oruntests) @DOCSTRING(rundemos) @DOCSTRING(speed)