Mercurial > octave

annotate scripts/testfun/__run_test_suite__.m @ 33100:71bbf3d89efe stable

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
display total time to execute test suite * __run_test_suite__.m (__run_test_suite__): Also track and display total CPU and wall-clock time. (print_pass_fail): Use ID to pair tic and toc calls.
author John W. Eaton <jwe@octave.org>
date Tue, 27 Feb 2024 23:48:37 -0500
parents 4d24afc1d32b
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
27923
bd51beb6205e update formatting of copyright notices
John W. Eaton <jwe@octave.org>
parents: 27919
diff changeset
1 ########################################################################
bd51beb6205e update formatting of copyright notices
John W. Eaton <jwe@octave.org>
parents: 27919
diff changeset
2 ##
32632
2e484f9f1f18 maint: update Octave Project Developers copyright for the new year
John W. Eaton <jwe@octave.org>
parents: 32075
diff changeset
3 ## Copyright (C) 2005-2024 The Octave Project Developers
27918
b442ec6dda5c use centralized file for copyright info for individual contributors
John W. Eaton <jwe@octave.org>
parents: 27348
diff changeset
4 ##
27923
bd51beb6205e update formatting of copyright notices
John W. Eaton <jwe@octave.org>
parents: 27919
diff changeset
5 ## See the file COPYRIGHT.md in the top-level directory of this
bd51beb6205e update formatting of copyright notices
John W. Eaton <jwe@octave.org>
parents: 27919
diff changeset
6 ## distribution or <https://octave.org/copyright/>.
16019
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
7 ##
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
8 ## This file is part of Octave.
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
9 ##
24534
194eb4bd202b maint: Update punctuation for GPL v3 license text.
Rik <rik@octave.org>
parents: 24483
diff changeset
10 ## Octave is free software: you can redistribute it and/or modify it
16019
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
11 ## under the terms of the GNU General Public License as published by
24534
194eb4bd202b maint: Update punctuation for GPL v3 license text.
Rik <rik@octave.org>
parents: 24483
diff changeset
12 ## the Free Software Foundation, either version 3 of the License, or
22755
3a2b891d0b33 maint: Standardize Copyright formatting.
Rik <rik@octave.org>
parents: 22467
diff changeset
13 ## (at your option) any later version.
16019
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
14 ##
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
15 ## Octave is distributed in the hope that it will be useful, but
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
16 ## WITHOUT ANY WARRANTY; without even the implied warranty of
22755
3a2b891d0b33 maint: Standardize Copyright formatting.
Rik <rik@octave.org>
parents: 22467
diff changeset
17 ## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3a2b891d0b33 maint: Standardize Copyright formatting.
Rik <rik@octave.org>
parents: 22467
diff changeset
18 ## GNU General Public License for more details.
16019
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
19 ##
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
20 ## You should have received a copy of the GNU General Public License
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
21 ## along with Octave; see the file COPYING. If not, see
24534
194eb4bd202b maint: Update punctuation for GPL v3 license text.
Rik <rik@octave.org>
parents: 24483
diff changeset
22 ## <https://www.gnu.org/licenses/>.
27923
bd51beb6205e update formatting of copyright notices
John W. Eaton <jwe@octave.org>
parents: 27919
diff changeset
23 ##
bd51beb6205e update formatting of copyright notices
John W. Eaton <jwe@octave.org>
parents: 27919
diff changeset
24 ########################################################################
16019
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
25
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
26 ## -*- texinfo -*-
31664
505ed551e366 doc: Ensure DOCSTRING has an output when function returns a value (bug #61681)
Rik <rik@octave.org>
parents: 30875
diff changeset
27 ## @deftypefn {} {[@var{pass}, @var{fail}, @var{xfail}, @var{xbug}, @var{skip}, @var{rtskip}, @var{regress}] =} __run_test_suite__ (@var{fcndirs}, @var{fixedtestdirs})
505ed551e366 doc: Ensure DOCSTRING has an output when function returns a value (bug #61681)
Rik <rik@octave.org>
parents: 30875
diff changeset
28 ## @deftypefnx {} {[@var{pass}, @var{fail}, @var{xfail}, @var{xbug}, @var{skip}, @var{rtskip}, @var{regress}] =} __run_test_suite__ (@var{fcndirs}, @var{fixedtestdirs}, @var{topsrcdir}, @var{topbuilddir})
16019
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
29 ## Undocumented internal function.
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
30 ## @end deftypefn
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
31
24185
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
32 function [pass, fail, xfail, xbug, skip, rtskip, regress] = __run_test_suite__ (fcndirs, fixedtestdirs, topsrcdir = [], topbuilddir = [])
20426
fe488ffe5f4b * __run_test_suite__.m: Return number of passed, failed, and skipped tests.
John W. Eaton <jwe@octave.org>
parents: 19697
diff changeset
33
21434
1da428cbf401 deprecate octave_config_info
John W. Eaton <jwe@octave.org>
parents: 20852
diff changeset
34 testsdir = __octave_config_info__ ("octtestsdir");
16030
1af8d21608b7 rename all test files in the test directory from test_X.m to X.tst
John W. Eaton <jwe@octave.org>
parents: 16019
diff changeset
35 libinterptestdir = fullfile (testsdir, "libinterp");
1af8d21608b7 rename all test files in the test directory from test_X.m to X.tst
John W. Eaton <jwe@octave.org>
parents: 16019
diff changeset
36 liboctavetestdir = fullfile (testsdir, "liboctave");
16019
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
37 fixedtestdir = fullfile (testsdir, "fixed");
21434
1da428cbf401 deprecate octave_config_info
John W. Eaton <jwe@octave.org>
parents: 20852
diff changeset
38 fcnfiledir = __octave_config_info__ ("fcnfiledir");
16019
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
39 if (nargin == 0)
18028
9f59f4f74986 test: Run test suite tests in compilation order (liboctave, libinterp, scripts).
Rik <rik@octave.org>
parents: 17744
diff changeset
40 fcndirs = { liboctavetestdir, libinterptestdir, fcnfiledir };
16019
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
41 fixedtestdirs = { fixedtestdir };
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
42 endif
19369
29228fa3edba improve reporting for __run_test_suite__ (bug #43732)
John W. Eaton <jwe@octave.org>
parents: 18116
diff changeset
43 files_with_no_tests = {};
29228fa3edba improve reporting for __run_test_suite__ (bug #43732)
John W. Eaton <jwe@octave.org>
parents: 18116
diff changeset
44 files_with_tests = {};
30107
4fa7200f230a * __run_test_suite__.m: Show summary of failed tests before final summary.
John W. Eaton <jwe@octave.org>
parents: 29358
diff changeset
45 summary_failure_info = struct ([]);
17182
fa724bdd52d0 __run_test_suite__.m: Recode regexps for performance. Don't search private '.' dirs.
Rik <rik@octave.org>
parents: 16994
diff changeset
46 ## FIXME: These names don't really make sense if we are running
fa724bdd52d0 __run_test_suite__.m: Recode regexps for performance. Don't search private '.' dirs.
Rik <rik@octave.org>
parents: 16994
diff changeset
47 ## tests for an installed copy of Octave.
24185
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
48 if (isempty (topsrcdir))
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
49 topsrcdir = fcnfiledir;
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
50 endif
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
51 if (isempty (topbuilddir))
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
52 topbuilddir = testsdir;
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
53 endif
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
54
16019
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
55 pso = page_screen_output ();
28056
5f57a71d67db tests: restore warning "quiet" state after running test suite (bug #57786)
Mike Miller <mtmiller@octave.org>
parents: 27928
diff changeset
56 orig_wquiet = warning ("query", "quiet");
19444
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
57 orig_wstate = warning ();
32075
9583d971e603 mfile_encoding: Add to officially documented functions.
Markus Mützel <markus.muetzel@gmx.de>
parents: 31706
diff changeset
58 orig_mfile_encoding = mfile_encoding ("utf-8");
19369
29228fa3edba improve reporting for __run_test_suite__ (bug #43732)
John W. Eaton <jwe@octave.org>
parents: 18116
diff changeset
59 logfile = make_absolute_filename ("fntests.log");
19444
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
60 unwind_protect
17182
fa724bdd52d0 __run_test_suite__.m: Recode regexps for performance. Don't search private '.' dirs.
Rik <rik@octave.org>
parents: 16994
diff changeset
61 page_screen_output (false);
19444
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
62 warning ("on", "quiet");
16019
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
63 warning ("off", "Octave:deprecated-function");
25728
64fabfc191d7 __run_test_suite__.m: Ignore legacy/ directory files during testing.
Rik <rik@octave.org>
parents: 25054
diff changeset
64 warning ("off", "Octave:legacy-function");
23830
595d6716e3e2 ensure summary variables are defined in __run_test_suite__ (bug #51660)
John W. Eaton <jwe@octave.org>
parents: 23754
diff changeset
65 nfail = dp = dn = dxf = dxb = dsk = drtsk = drgrs = 0;
19444
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
66 try
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
67 fid = fopen (logfile, "wt");
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
68 if (fid < 0)
20726
25d676f9619c Preface error() messages with name of function when possible.
Rik <rik@octave.org>
parents: 20426
diff changeset
69 error ("__run_test_suite__: could not open %s for writing", logfile);
19444
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
70 endif
33100
71bbf3d89efe display total time to execute test suite
John W. Eaton <jwe@octave.org>
parents: 32899
diff changeset
71 tot_cpu_tm = cputime ();
71bbf3d89efe display total time to execute test suite
John W. Eaton <jwe@octave.org>
parents: 32899
diff changeset
72 tot_tic_tag = tic ();
19444
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
73 test ("", "explain", fid);
21634
96518f623c91 Backed out changeset dcf8922b724b
Mike Miller <mtmiller@octave.org>
parents: 21633
diff changeset
74 puts ("\nIntegrated test scripts:\n\n");
32899
4d24afc1d32b include timing info in test suite result summary output
John W. Eaton <jwe@octave.org>
parents: 32632
diff changeset
75 printf ("%101s\n", "[ CPU / CLOCK ]");
19444
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
76 for i = 1:length (fcndirs)
26278
01fdb337fa20 test: do not look for BISTs in .m files in fixed test dirs (bug #55250)
Mike Miller <mtmiller@octave.org>
parents: 25827
diff changeset
77 [p, n, xf, xb, sk, rtsk, rgrs] = run_test_dir (fid, fcndirs{i}, false);
19444
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
78 dp += p;
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
79 dn += n;
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
80 dxf += xf;
23363
0cd0db82123d distinguish between reported bugs and known failures in tests
John W. Eaton <jwe@octave.org>
parents: 23280
diff changeset
81 dxb += xb;
19444
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
82 dsk += sk;
23279
67d741321e21 allow skipping of tests based on run-time conditions
John W. Eaton <jwe@octave.org>
parents: 23220
diff changeset
83 drtsk += rtsk;
23571
b6144e6dda9e allow bugs to be tagged as fixed in test suite
John W. Eaton <jwe@octave.org>
parents: 23363
diff changeset
84 drgrs += rgrs;
19444
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
85 endfor
21634
96518f623c91 Backed out changeset dcf8922b724b
Mike Miller <mtmiller@octave.org>
parents: 21633
diff changeset
86 puts ("\nFixed test scripts:\n\n");
19444
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
87 for i = 1:length (fixedtestdirs)
26278
01fdb337fa20 test: do not look for BISTs in .m files in fixed test dirs (bug #55250)
Mike Miller <mtmiller@octave.org>
parents: 25827
diff changeset
88 [p, n, xf, xb, sk, rtsk, rgrs] = run_test_dir (fid, fixedtestdirs{i}, true);
19444
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
89 dp += p;
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
90 dn += n;
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
91 dxf += xf;
23363
0cd0db82123d distinguish between reported bugs and known failures in tests
John W. Eaton <jwe@octave.org>
parents: 23280
diff changeset
92 dxb += xb;
19444
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
93 dsk += sk;
23279
67d741321e21 allow skipping of tests based on run-time conditions
John W. Eaton <jwe@octave.org>
parents: 23220
diff changeset
94 drtsk += rtsk;
23571
b6144e6dda9e allow bugs to be tagged as fixed in test suite
John W. Eaton <jwe@octave.org>
parents: 23363
diff changeset
95 drgrs += rgrs;
19444
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
96 endfor
33100
71bbf3d89efe display total time to execute test suite
John W. Eaton <jwe@octave.org>
parents: 32899
diff changeset
97 tot_clock_tm = toc (tot_tic_tag);
71bbf3d89efe display total time to execute test suite
John W. Eaton <jwe@octave.org>
parents: 32899
diff changeset
98 tot_cpu_tm = cputime () - tot_cpu_tm;
71bbf3d89efe display total time to execute test suite
John W. Eaton <jwe@octave.org>
parents: 32899
diff changeset
99 printf ("%80s [%6.1fs / %6.1fs]", "total time (CPU / CLOCK)", tot_cpu_tm, tot_clock_tm);
30107
4fa7200f230a * __run_test_suite__.m: Show summary of failed tests before final summary.
John W. Eaton <jwe@octave.org>
parents: 29358
diff changeset
100 if (! isempty (summary_failure_info))
4fa7200f230a * __run_test_suite__.m: Show summary of failed tests before final summary.
John W. Eaton <jwe@octave.org>
parents: 29358
diff changeset
101 puts ("\nFailure Summary:\n\n");
4fa7200f230a * __run_test_suite__.m: Show summary of failed tests before final summary.
John W. Eaton <jwe@octave.org>
parents: 29358
diff changeset
102 for i = 1:numel (summary_failure_info)
4fa7200f230a * __run_test_suite__.m: Show summary of failed tests before final summary.
John W. Eaton <jwe@octave.org>
parents: 29358
diff changeset
103 info = summary_failure_info(i);
4fa7200f230a * __run_test_suite__.m: Show summary of failed tests before final summary.
John W. Eaton <jwe@octave.org>
parents: 29358
diff changeset
104 print_test_file_name (info.name);
4fa7200f230a * __run_test_suite__.m: Show summary of failed tests before final summary.
John W. Eaton <jwe@octave.org>
parents: 29358
diff changeset
105 print_pass_fail (info);
4fa7200f230a * __run_test_suite__.m: Show summary of failed tests before final summary.
John W. Eaton <jwe@octave.org>
parents: 29358
diff changeset
106 endfor
4fa7200f230a * __run_test_suite__.m: Show summary of failed tests before final summary.
John W. Eaton <jwe@octave.org>
parents: 29358
diff changeset
107 endif
21634
96518f623c91 Backed out changeset dcf8922b724b
Mike Miller <mtmiller@octave.org>
parents: 21633
diff changeset
108 puts ("\nSummary:\n\n");
23571
b6144e6dda9e allow bugs to be tagged as fixed in test suite
John W. Eaton <jwe@octave.org>
parents: 23363
diff changeset
109 nfail = dn - dp - dxf - dxb - drgrs;
23363
0cd0db82123d distinguish between reported bugs and known failures in tests
John W. Eaton <jwe@octave.org>
parents: 23280
diff changeset
110 printf (" %-30s %6d\n", "PASS", dp);
0cd0db82123d distinguish between reported bugs and known failures in tests
John W. Eaton <jwe@octave.org>
parents: 23280
diff changeset
111 printf (" %-30s %6d\n", "FAIL", nfail);
23754
afe45e9ff6b0 show regressions more prominently in test results and summary
John W. Eaton <jwe@octave.org>
parents: 23571
diff changeset
112 if (drgrs > 0)
afe45e9ff6b0 show regressions more prominently in test results and summary
John W. Eaton <jwe@octave.org>
parents: 23571
diff changeset
113 printf (" %-30s %6d\n", "REGRESSION", drgrs);
afe45e9ff6b0 show regressions more prominently in test results and summary
John W. Eaton <jwe@octave.org>
parents: 23571
diff changeset
114 endif
23363
0cd0db82123d distinguish between reported bugs and known failures in tests
John W. Eaton <jwe@octave.org>
parents: 23280
diff changeset
115 if (dxb > 0)
0cd0db82123d distinguish between reported bugs and known failures in tests
John W. Eaton <jwe@octave.org>
parents: 23280
diff changeset
116 printf (" %-30s %6d\n", "XFAIL (reported bug)", dxb);
19444
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
117 endif
24483
dac2ad033c43 use same wording and order for individual test results and summary
John W. Eaton <jwe@octave.org>
parents: 24185
diff changeset
118 if (dxf > 0)
dac2ad033c43 use same wording and order for individual test results and summary
John W. Eaton <jwe@octave.org>
parents: 24185
diff changeset
119 printf (" %-30s %6d\n", "XFAIL (expected failure)", dxf);
dac2ad033c43 use same wording and order for individual test results and summary
John W. Eaton <jwe@octave.org>
parents: 24185
diff changeset
120 endif
19444
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
121 if (dsk > 0)
24483
dac2ad033c43 use same wording and order for individual test results and summary
John W. Eaton <jwe@octave.org>
parents: 24185
diff changeset
122 printf (" %-30s %6d\n", "SKIP (missing feature)", dsk);
23279
67d741321e21 allow skipping of tests based on run-time conditions
John W. Eaton <jwe@octave.org>
parents: 23220
diff changeset
123 endif
67d741321e21 allow skipping of tests based on run-time conditions
John W. Eaton <jwe@octave.org>
parents: 23220
diff changeset
124 if (drtsk > 0)
24483
dac2ad033c43 use same wording and order for individual test results and summary
John W. Eaton <jwe@octave.org>
parents: 24185
diff changeset
125 printf (" %-30s %6d\n", "SKIP (run-time condition)", drtsk);
19444
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
126 endif
21634
96518f623c91 Backed out changeset dcf8922b724b
Mike Miller <mtmiller@octave.org>
parents: 21633
diff changeset
127 puts ("\n");
96518f623c91 Backed out changeset dcf8922b724b
Mike Miller <mtmiller@octave.org>
parents: 21633
diff changeset
128 printf ("See the file %s for additional details.\n", logfile);
23363
0cd0db82123d distinguish between reported bugs and known failures in tests
John W. Eaton <jwe@octave.org>
parents: 23280
diff changeset
129 if (dxf > 0 || dxb > 0)
21634
96518f623c91 Backed out changeset dcf8922b724b
Mike Miller <mtmiller@octave.org>
parents: 21633
diff changeset
130 puts ("\n");
24483
dac2ad033c43 use same wording and order for individual test results and summary
John W. Eaton <jwe@octave.org>
parents: 24185
diff changeset
131 puts ("XFAIL items are known bugs or expected failures.\n");
23883
b3b136bc7ce2 shorten messages displayed at end of make check
John W. Eaton <jwe@octave.org>
parents: 23830
diff changeset
132 puts ("Bug report numbers may be found in the log file:\n");
22389
663e30242c4b avoid using "expected failure" to describe known bugs
John W. Eaton <jwe@octave.org>
parents: 22323
diff changeset
133 puts (logfile);
22467
c61467fc0d3a Add missing newline in output of 'make check' (bug #49052)
Mike Miller <mtmiller@octave.org>
parents: 22389
diff changeset
134 puts ("\nPlease help improve Octave by contributing fixes for them.\n");
19444
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
135 endif
23279
67d741321e21 allow skipping of tests based on run-time conditions
John W. Eaton <jwe@octave.org>
parents: 23220
diff changeset
136 if (dsk > 0 || drtsk > 0)
21634
96518f623c91 Backed out changeset dcf8922b724b
Mike Miller <mtmiller@octave.org>
parents: 21633
diff changeset
137 puts ("\n");
23883
b3b136bc7ce2 shorten messages displayed at end of make check
John W. Eaton <jwe@octave.org>
parents: 23830
diff changeset
138 puts ("Tests are often skipped because required features were\n");
b3b136bc7ce2 shorten messages displayed at end of make check
John W. Eaton <jwe@octave.org>
parents: 23830
diff changeset
139 puts ("disabled or were not present when Octave was built.\n");
b3b136bc7ce2 shorten messages displayed at end of make check
John W. Eaton <jwe@octave.org>
parents: 23830
diff changeset
140 puts ("The configure script should have printed a summary\n");
b3b136bc7ce2 shorten messages displayed at end of make check
John W. Eaton <jwe@octave.org>
parents: 23830
diff changeset
141 puts ("indicating which dependencies were not found.\n");
19444
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
142 endif
16019
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
143
25728
64fabfc191d7 __run_test_suite__.m: Ignore legacy/ directory files during testing.
Rik <rik@octave.org>
parents: 25054
diff changeset
144 ## Weed out deprecated, legacy, and private functions
64fabfc191d7 __run_test_suite__.m: Ignore legacy/ directory files during testing.
Rik <rik@octave.org>
parents: 25054
diff changeset
145 weed_idx = cellfun (@isempty, regexp (files_with_tests, '\<deprecated\>|\<legacy\>|\<private\>', 'once'));
19444
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
146 files_with_tests = files_with_tests(weed_idx);
25728
64fabfc191d7 __run_test_suite__.m: Ignore legacy/ directory files during testing.
Rik <rik@octave.org>
parents: 25054
diff changeset
147 weed_idx = cellfun (@isempty, regexp (files_with_no_tests, '\<deprecated\>|\<legacy\>|\<private\>', 'once'));
19444
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
148 files_with_no_tests = files_with_no_tests(weed_idx);
16019
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
149
19444
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
150 report_files_with_no_tests (files_with_tests, files_with_no_tests, ".m");
16019
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
151
21634
96518f623c91 Backed out changeset dcf8922b724b
Mike Miller <mtmiller@octave.org>
parents: 21633
diff changeset
152 puts ("\nPlease help improve Octave by contributing tests for these files\n");
96518f623c91 Backed out changeset dcf8922b724b
Mike Miller <mtmiller@octave.org>
parents: 21633
diff changeset
153 printf ("(see the list in the file %s).\n\n", logfile);
16019
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
154
19444
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
155 fprintf (fid, "\nFiles with no tests:\n\n%s",
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
156 list_in_columns (files_with_no_tests, 80));
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
157 fclose (fid);
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
158 catch
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
159 disp (lasterr ());
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
160 end_try_catch
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
161 unwind_protect_cleanup
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
162 warning ("off", "all");
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
163 warning (orig_wstate);
28056
5f57a71d67db tests: restore warning "quiet" state after running test suite (bug #57786)
Mike Miller <mtmiller@octave.org>
parents: 27928
diff changeset
164 warning (orig_wquiet.state, "quiet");
32075
9583d971e603 mfile_encoding: Add to officially documented functions.
Markus Mützel <markus.muetzel@gmx.de>
parents: 31706
diff changeset
165 mfile_encoding (orig_mfile_encoding);
16019
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
166 page_screen_output (pso);
19444
3f29b433bd5d do a better job of preserving warning state when running tests
John W. Eaton <jwe@octave.org>
parents: 19377
diff changeset
167 end_unwind_protect
20426
fe488ffe5f4b * __run_test_suite__.m: Return number of passed, failed, and skipped tests.
John W. Eaton <jwe@octave.org>
parents: 19697
diff changeset
168
fe488ffe5f4b * __run_test_suite__.m: Return number of passed, failed, and skipped tests.
John W. Eaton <jwe@octave.org>
parents: 19697
diff changeset
169 if (nargout > 0)
23168
d7ea2662a7a8 * __run_test_suite__.m: fix typo in return values.
Colin Macdonald <cbm@m.fsf.org>
parents: 23083
diff changeset
170 pass = dp;
20426
fe488ffe5f4b * __run_test_suite__.m: Return number of passed, failed, and skipped tests.
John W. Eaton <jwe@octave.org>
parents: 19697
diff changeset
171 fail = nfail;
fe488ffe5f4b * __run_test_suite__.m: Return number of passed, failed, and skipped tests.
John W. Eaton <jwe@octave.org>
parents: 19697
diff changeset
172 xfail = dxf;
23363
0cd0db82123d distinguish between reported bugs and known failures in tests
John W. Eaton <jwe@octave.org>
parents: 23280
diff changeset
173 xbug = dxb;
23168
d7ea2662a7a8 * __run_test_suite__.m: fix typo in return values.
Colin Macdonald <cbm@m.fsf.org>
parents: 23083
diff changeset
174 skip = dsk;
23279
67d741321e21 allow skipping of tests based on run-time conditions
John W. Eaton <jwe@octave.org>
parents: 23220
diff changeset
175 rtskip = drtsk;
23571
b6144e6dda9e allow bugs to be tagged as fixed in test suite
John W. Eaton <jwe@octave.org>
parents: 23363
diff changeset
176 regress = drgrs;
20426
fe488ffe5f4b * __run_test_suite__.m: Return number of passed, failed, and skipped tests.
John W. Eaton <jwe@octave.org>
parents: 19697
diff changeset
177 endif
fe488ffe5f4b * __run_test_suite__.m: Return number of passed, failed, and skipped tests.
John W. Eaton <jwe@octave.org>
parents: 19697
diff changeset
178
26278
01fdb337fa20 test: do not look for BISTs in .m files in fixed test dirs (bug #55250)
Mike Miller <mtmiller@octave.org>
parents: 25827
diff changeset
179 function [dp, dn, dxf, dxb, dsk, drtsk, drgrs] = run_test_dir (fid, d, is_fixed = false)
24185
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
180
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
181 lst = dir (d);
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
182 dp = dn = dxf = dxb = dsk = drtsk = drgrs = 0;
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
183 for i = 1:length (lst)
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
184 nm = lst(i).name;
25827
d9d192928b45 process tests in @CLASS directories again (bug #54561)
John W. Eaton <jwe@octave.org>
parents: 25818
diff changeset
185 if (lst(i).isdir && nm(1) != "." && ! strcmp (nm, "private"))
26278
01fdb337fa20 test: do not look for BISTs in .m files in fixed test dirs (bug #55250)
Mike Miller <mtmiller@octave.org>
parents: 25827
diff changeset
186 [p, n, xf, xb, sk, rtsk, rgrs] = run_test_dir (fid, [d, filesep, nm], is_fixed);
24185
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
187 dp += p;
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
188 dn += n;
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
189 dxf += xf;
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
190 dxb += xb;
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
191 dsk += sk;
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
192 drtsk += rtsk;
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
193 drgrs += rgrs;
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
194 endif
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
195 endfor
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
196
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
197 saved_dir = pwd ();
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
198 unwind_protect
25827
d9d192928b45 process tests in @CLASS directories again (bug #54561)
John W. Eaton <jwe@octave.org>
parents: 25818
diff changeset
199 [dnm, fnm] = fileparts (d);
d9d192928b45 process tests in @CLASS directories again (bug #54561)
John W. Eaton <jwe@octave.org>
parents: 25818
diff changeset
200 if (fnm(1) != "@")
d9d192928b45 process tests in @CLASS directories again (bug #54561)
John W. Eaton <jwe@octave.org>
parents: 25818
diff changeset
201 cd (d);
d9d192928b45 process tests in @CLASS directories again (bug #54561)
John W. Eaton <jwe@octave.org>
parents: 25818
diff changeset
202 endif
24185
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
203 for i = 1:length (lst)
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
204 nm = lst(i).name;
25818
2be7cf62b9ee eliminate some nearly duplicate code in __run_test_suite__
John W. Eaton <jwe@octave.org>
parents: 25728
diff changeset
205 ## Ignore hidden files
2be7cf62b9ee eliminate some nearly duplicate code in __run_test_suite__
John W. Eaton <jwe@octave.org>
parents: 25728
diff changeset
206 if (nm(1) == '.')
27928
6804f2feea46 __run_test_suite__.m: Validate .m/.cc names are files before proceeding.
Rik <rik@octave.org>
parents: 27923
diff changeset
207 continue;
25818
2be7cf62b9ee eliminate some nearly duplicate code in __run_test_suite__
John W. Eaton <jwe@octave.org>
parents: 25728
diff changeset
208 endif
26278
01fdb337fa20 test: do not look for BISTs in .m files in fixed test dirs (bug #55250)
Mike Miller <mtmiller@octave.org>
parents: 25827
diff changeset
209 if ((! is_fixed && length (nm) > 2 && strcmpi (nm((end-1):end), ".m"))
01fdb337fa20 test: do not look for BISTs in .m files in fixed test dirs (bug #55250)
Mike Miller <mtmiller@octave.org>
parents: 25827
diff changeset
210 || (! is_fixed && length (nm) > 4 && strcmpi (nm((end-3):end), "-tst"))
01fdb337fa20 test: do not look for BISTs in .m files in fixed test dirs (bug #55250)
Mike Miller <mtmiller@octave.org>
parents: 25827
diff changeset
211 || (is_fixed && length (nm) > 4 && strcmpi (nm((end-3):end), ".tst")))
25827
d9d192928b45 process tests in @CLASS directories again (bug #54561)
John W. Eaton <jwe@octave.org>
parents: 25818
diff changeset
212 p = n = xf = xb = sk = rtsk = rgrs = 0;
24185
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
213 ffnm = fullfile (d, nm);
27928
6804f2feea46 __run_test_suite__.m: Validate .m/.cc names are files before proceeding.
Rik <rik@octave.org>
parents: 27923
diff changeset
214 if (! isfile (ffnm))
6804f2feea46 __run_test_suite__.m: Validate .m/.cc names are files before proceeding.
Rik <rik@octave.org>
parents: 27923
diff changeset
215 continue;
6804f2feea46 __run_test_suite__.m: Validate .m/.cc names are files before proceeding.
Rik <rik@octave.org>
parents: 27923
diff changeset
216 endif
6804f2feea46 __run_test_suite__.m: Validate .m/.cc names are files before proceeding.
Rik <rik@octave.org>
parents: 27923
diff changeset
217
25818
2be7cf62b9ee eliminate some nearly duplicate code in __run_test_suite__
John W. Eaton <jwe@octave.org>
parents: 25728
diff changeset
218 ## Only run if contains %!test, %!assert, %!error, %!fail, or %!warning
24185
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
219 if (has_tests (ffnm))
25818
2be7cf62b9ee eliminate some nearly duplicate code in __run_test_suite__
John W. Eaton <jwe@octave.org>
parents: 25728
diff changeset
220 tmp = reduce_test_file_name (ffnm, topbuilddir, topsrcdir);
2be7cf62b9ee eliminate some nearly duplicate code in __run_test_suite__
John W. Eaton <jwe@octave.org>
parents: 25728
diff changeset
221 print_test_file_name (tmp);
32899
4d24afc1d32b include timing info in test suite result summary output
John W. Eaton <jwe@octave.org>
parents: 32632
diff changeset
222 cpu_tm = cputime ();
33100
71bbf3d89efe display total time to execute test suite
John W. Eaton <jwe@octave.org>
parents: 32899
diff changeset
223 tic_tag = tic ();
25818
2be7cf62b9ee eliminate some nearly duplicate code in __run_test_suite__
John W. Eaton <jwe@octave.org>
parents: 25728
diff changeset
224 [p, n, xf, xb, sk, rtsk, rgrs] = test (ffnm, "quiet", fid);
33100
71bbf3d89efe display total time to execute test suite
John W. Eaton <jwe@octave.org>
parents: 32899
diff changeset
225 clock_tm = toc (tic_tag);
32899
4d24afc1d32b include timing info in test suite result summary output
John W. Eaton <jwe@octave.org>
parents: 32632
diff changeset
226 cpu_tm = cputime () - cpu_tm;
4d24afc1d32b include timing info in test suite result summary output
John W. Eaton <jwe@octave.org>
parents: 32632
diff changeset
227 print_pass_fail (p, n, xf, xb, sk, rtsk, rgrs, cpu_tm, clock_tm);
25818
2be7cf62b9ee eliminate some nearly duplicate code in __run_test_suite__
John W. Eaton <jwe@octave.org>
parents: 25728
diff changeset
228 dp += p;
2be7cf62b9ee eliminate some nearly duplicate code in __run_test_suite__
John W. Eaton <jwe@octave.org>
parents: 25728
diff changeset
229 dn += n;
2be7cf62b9ee eliminate some nearly duplicate code in __run_test_suite__
John W. Eaton <jwe@octave.org>
parents: 25728
diff changeset
230 dxf += xf;
2be7cf62b9ee eliminate some nearly duplicate code in __run_test_suite__
John W. Eaton <jwe@octave.org>
parents: 25728
diff changeset
231 dxb += xb;
2be7cf62b9ee eliminate some nearly duplicate code in __run_test_suite__
John W. Eaton <jwe@octave.org>
parents: 25728
diff changeset
232 dsk += sk;
2be7cf62b9ee eliminate some nearly duplicate code in __run_test_suite__
John W. Eaton <jwe@octave.org>
parents: 25728
diff changeset
233 drtsk += rtsk;
2be7cf62b9ee eliminate some nearly duplicate code in __run_test_suite__
John W. Eaton <jwe@octave.org>
parents: 25728
diff changeset
234 drgrs += rgrs;
24185
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
235 files_with_tests(end+1) = ffnm;
30107
4fa7200f230a * __run_test_suite__.m: Show summary of failed tests before final summary.
John W. Eaton <jwe@octave.org>
parents: 29358
diff changeset
236 if (n - p > 0)
4fa7200f230a * __run_test_suite__.m: Show summary of failed tests before final summary.
John W. Eaton <jwe@octave.org>
parents: 29358
diff changeset
237 ## Save info for summary if there were any failed
4fa7200f230a * __run_test_suite__.m: Show summary of failed tests before final summary.
John W. Eaton <jwe@octave.org>
parents: 29358
diff changeset
238 ## tests for this file.
4fa7200f230a * __run_test_suite__.m: Show summary of failed tests before final summary.
John W. Eaton <jwe@octave.org>
parents: 29358
diff changeset
239 failure_info = struct ("name", tmp, "pass", p, "ntst", n,
4fa7200f230a * __run_test_suite__.m: Show summary of failed tests before final summary.
John W. Eaton <jwe@octave.org>
parents: 29358
diff changeset
240 "xf", xf, "xb", xb, "rgrs", rgrs);
4fa7200f230a * __run_test_suite__.m: Show summary of failed tests before final summary.
John W. Eaton <jwe@octave.org>
parents: 29358
diff changeset
241 summary_failure_info(end+1) = failure_info;
4fa7200f230a * __run_test_suite__.m: Show summary of failed tests before final summary.
John W. Eaton <jwe@octave.org>
parents: 29358
diff changeset
242 endif
24185
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
243 else
25818
2be7cf62b9ee eliminate some nearly duplicate code in __run_test_suite__
John W. Eaton <jwe@octave.org>
parents: 25728
diff changeset
244 ## To reduce the list length, only mark .cc files that contain
2be7cf62b9ee eliminate some nearly duplicate code in __run_test_suite__
John W. Eaton <jwe@octave.org>
parents: 25728
diff changeset
245 ## DEFUN definitions.
24185
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
246 files_with_no_tests(end+1) = ffnm;
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
247 endif
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
248 endif
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
249 endfor
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
250 unwind_protect_cleanup
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
251 cd (saved_dir);
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
252 end_unwind_protect
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
253
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
254 endfunction
068556c5167e Add BIST test for clear() function (bug #35881).
Rik <rik@octave.org>
parents: 23883
diff changeset
255
16019
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
256 endfunction
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
257
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
258 function print_test_file_name (nm)
30379
363fb10055df maint: Style check m-files ahead of 7.1 release.
Rik <rik@octave.org>
parents: 30107
diff changeset
259
25818
2be7cf62b9ee eliminate some nearly duplicate code in __run_test_suite__
John W. Eaton <jwe@octave.org>
parents: 25728
diff changeset
260 nmlen = numel (nm);
2be7cf62b9ee eliminate some nearly duplicate code in __run_test_suite__
John W. Eaton <jwe@octave.org>
parents: 25728
diff changeset
261 filler = repmat (".", 1, 63-nmlen);
2be7cf62b9ee eliminate some nearly duplicate code in __run_test_suite__
John W. Eaton <jwe@octave.org>
parents: 25728
diff changeset
262 if (nmlen > 63)
2be7cf62b9ee eliminate some nearly duplicate code in __run_test_suite__
John W. Eaton <jwe@octave.org>
parents: 25728
diff changeset
263 nm = ["..", nm(nmlen-60:end)];
2be7cf62b9ee eliminate some nearly duplicate code in __run_test_suite__
John W. Eaton <jwe@octave.org>
parents: 25728
diff changeset
264 endif
21634
96518f623c91 Backed out changeset dcf8922b724b
Mike Miller <mtmiller@octave.org>
parents: 21633
diff changeset
265 printf (" %s %s", nm, filler);
30379
363fb10055df maint: Style check m-files ahead of 7.1 release.
Rik <rik@octave.org>
parents: 30107
diff changeset
266
16019
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
267 endfunction
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
268
32899
4d24afc1d32b include timing info in test suite result summary output
John W. Eaton <jwe@octave.org>
parents: 32632
diff changeset
269 function fail_info = print_pass_fail (p, n, xf, xb, sk, rtsk, rgrs, cpu_tm, clock_tm)
30107
4fa7200f230a * __run_test_suite__.m: Show summary of failed tests before final summary.
John W. Eaton <jwe@octave.org>
parents: 29358
diff changeset
270
4fa7200f230a * __run_test_suite__.m: Show summary of failed tests before final summary.
John W. Eaton <jwe@octave.org>
parents: 29358
diff changeset
271 if (nargin == 1)
4fa7200f230a * __run_test_suite__.m: Show summary of failed tests before final summary.
John W. Eaton <jwe@octave.org>
parents: 29358
diff changeset
272 ## The summary info struct just contains info about failures, not
32899
4d24afc1d32b include timing info in test suite result summary output
John W. Eaton <jwe@octave.org>
parents: 32632
diff changeset
273 ## skipped tests or timings.
30107
4fa7200f230a * __run_test_suite__.m: Show summary of failed tests before final summary.
John W. Eaton <jwe@octave.org>
parents: 29358
diff changeset
274 info = p;
4fa7200f230a * __run_test_suite__.m: Show summary of failed tests before final summary.
John W. Eaton <jwe@octave.org>
parents: 29358
diff changeset
275 p = info.pass;
4fa7200f230a * __run_test_suite__.m: Show summary of failed tests before final summary.
John W. Eaton <jwe@octave.org>
parents: 29358
diff changeset
276 n = info.ntst;
4fa7200f230a * __run_test_suite__.m: Show summary of failed tests before final summary.
John W. Eaton <jwe@octave.org>
parents: 29358
diff changeset
277 xf = info.xf;
4fa7200f230a * __run_test_suite__.m: Show summary of failed tests before final summary.
John W. Eaton <jwe@octave.org>
parents: 29358
diff changeset
278 xb = info.xb;
4fa7200f230a * __run_test_suite__.m: Show summary of failed tests before final summary.
John W. Eaton <jwe@octave.org>
parents: 29358
diff changeset
279 sk = 0;
4fa7200f230a * __run_test_suite__.m: Show summary of failed tests before final summary.
John W. Eaton <jwe@octave.org>
parents: 29358
diff changeset
280 rtsk = 0;
4fa7200f230a * __run_test_suite__.m: Show summary of failed tests before final summary.
John W. Eaton <jwe@octave.org>
parents: 29358
diff changeset
281 rgrs = info.rgrs;
32899
4d24afc1d32b include timing info in test suite result summary output
John W. Eaton <jwe@octave.org>
parents: 32632
diff changeset
282 cpu_tm = 0;
4d24afc1d32b include timing info in test suite result summary output
John W. Eaton <jwe@octave.org>
parents: 32632
diff changeset
283 clock_tm = 0;
30107
4fa7200f230a * __run_test_suite__.m: Show summary of failed tests before final summary.
John W. Eaton <jwe@octave.org>
parents: 29358
diff changeset
284 endif
21758
ffad2baa90f7 maint: Use newlines to make code more readable.
Rik <rik@octave.org>
parents: 21634
diff changeset
285
23571
b6144e6dda9e allow bugs to be tagged as fixed in test suite
John W. Eaton <jwe@octave.org>
parents: 23363
diff changeset
286 if ((n + sk + rtsk + rgrs) > 0)
27348
237b0bd20658 attempt to make test failures more visible
John W. Eaton <jwe@octave.org>
parents: 26376
diff changeset
287 printf (" pass %4d/%-4d", p, n);
32899
4d24afc1d32b include timing info in test suite result summary output
John W. Eaton <jwe@octave.org>
parents: 32632
diff changeset
288 if (cpu_tm != 0 || clock_tm != 0)
4d24afc1d32b include timing info in test suite result summary output
John W. Eaton <jwe@octave.org>
parents: 32632
diff changeset
289 printf (" [%6.3fs / %6.3fs]", cpu_tm, clock_tm);
4d24afc1d32b include timing info in test suite result summary output
John W. Eaton <jwe@octave.org>
parents: 32632
diff changeset
290 endif
23571
b6144e6dda9e allow bugs to be tagged as fixed in test suite
John W. Eaton <jwe@octave.org>
parents: 23363
diff changeset
291 nfail = n - p - xf - xb - rgrs;
16019
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
292 if (nfail > 0)
25818
2be7cf62b9ee eliminate some nearly duplicate code in __run_test_suite__
John W. Eaton <jwe@octave.org>
parents: 25728
diff changeset
293 printf ("\n%72s %3d", "FAIL ", nfail);
19597
db92e7e28e1f strip trailing whitespace from most source files
John W. Eaton <jwe@octave.org>
parents: 19596
diff changeset
294 endif
23754
afe45e9ff6b0 show regressions more prominently in test results and summary
John W. Eaton <jwe@octave.org>
parents: 23571
diff changeset
295 if (rgrs > 0)
25818
2be7cf62b9ee eliminate some nearly duplicate code in __run_test_suite__
John W. Eaton <jwe@octave.org>
parents: 25728
diff changeset
296 printf ("\n%72s %3d", "REGRESSION", rgrs);
23754
afe45e9ff6b0 show regressions more prominently in test results and summary
John W. Eaton <jwe@octave.org>
parents: 23571
diff changeset
297 endif
18167
c9b4d3177de3 Don't count expected failures as passed tests.
Carlo de Falco <cdf@users.sourceforge.net>
parents: 18116
diff changeset
298 if (sk > 0)
25818
2be7cf62b9ee eliminate some nearly duplicate code in __run_test_suite__
John W. Eaton <jwe@octave.org>
parents: 25728
diff changeset
299 printf ("\n%72s %3d", "(missing feature) SKIP ", sk);
23279
67d741321e21 allow skipping of tests based on run-time conditions
John W. Eaton <jwe@octave.org>
parents: 23220
diff changeset
300 endif
67d741321e21 allow skipping of tests based on run-time conditions
John W. Eaton <jwe@octave.org>
parents: 23220
diff changeset
301 if (rtsk > 0)
25818
2be7cf62b9ee eliminate some nearly duplicate code in __run_test_suite__
John W. Eaton <jwe@octave.org>
parents: 25728
diff changeset
302 printf ("\n%72s %3d", "(run-time condition) SKIP ", rtsk);
18167
c9b4d3177de3 Don't count expected failures as passed tests.
Carlo de Falco <cdf@users.sourceforge.net>
parents: 18116
diff changeset
303 endif
23363
0cd0db82123d distinguish between reported bugs and known failures in tests
John W. Eaton <jwe@octave.org>
parents: 23280
diff changeset
304 if (xb > 0)
25818
2be7cf62b9ee eliminate some nearly duplicate code in __run_test_suite__
John W. Eaton <jwe@octave.org>
parents: 25728
diff changeset
305 printf ("\n%72s %3d", "(reported bug) XFAIL", xb);
16019
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
306 endif
23571
b6144e6dda9e allow bugs to be tagged as fixed in test suite
John W. Eaton <jwe@octave.org>
parents: 23363
diff changeset
307 if (xf > 0)
25818
2be7cf62b9ee eliminate some nearly duplicate code in __run_test_suite__
John W. Eaton <jwe@octave.org>
parents: 25728
diff changeset
308 printf ("\n%72s %3d", "(expected failure) XFAIL", xf);
23571
b6144e6dda9e allow bugs to be tagged as fixed in test suite
John W. Eaton <jwe@octave.org>
parents: 23363
diff changeset
309 endif
16019
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
310 endif
21634
96518f623c91 Backed out changeset dcf8922b724b
Mike Miller <mtmiller@octave.org>
parents: 21633
diff changeset
311 puts ("\n");
21758
ffad2baa90f7 maint: Use newlines to make code more readable.
Rik <rik@octave.org>
parents: 21634
diff changeset
312
16019
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
313 endfunction
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
314
25007
2f695cf8af6f test: reduce relative file names in test suite output
Mike Miller <mtmiller@octave.org>
parents: 24534
diff changeset
315 function retval = reduce_test_file_name (nm, builddir, srcdir)
2f695cf8af6f test: reduce relative file names in test suite output
Mike Miller <mtmiller@octave.org>
parents: 24534
diff changeset
316
2f695cf8af6f test: reduce relative file names in test suite output
Mike Miller <mtmiller@octave.org>
parents: 24534
diff changeset
317 ## Reduce the given absolute file name to a relative path by removing one
2f695cf8af6f test: reduce relative file names in test suite output
Mike Miller <mtmiller@octave.org>
parents: 24534
diff changeset
318 ## of the likely root directory prefixes.
2f695cf8af6f test: reduce relative file names in test suite output
Mike Miller <mtmiller@octave.org>
parents: 24534
diff changeset
319
2f695cf8af6f test: reduce relative file names in test suite output
Mike Miller <mtmiller@octave.org>
parents: 24534
diff changeset
320 prefix = { builddir, fullfile(builddir, "scripts"), ...
25818
2be7cf62b9ee eliminate some nearly duplicate code in __run_test_suite__
John W. Eaton <jwe@octave.org>
parents: 25728
diff changeset
321 srcdir, fullfile(srcdir, "scripts"), ...
2be7cf62b9ee eliminate some nearly duplicate code in __run_test_suite__
John W. Eaton <jwe@octave.org>
parents: 25728
diff changeset
322 srcdir, fullfile(srcdir, "test") };
25007
2f695cf8af6f test: reduce relative file names in test suite output
Mike Miller <mtmiller@octave.org>
parents: 24534
diff changeset
323
2f695cf8af6f test: reduce relative file names in test suite output
Mike Miller <mtmiller@octave.org>
parents: 24534
diff changeset
324 retval = nm;
2f695cf8af6f test: reduce relative file names in test suite output
Mike Miller <mtmiller@octave.org>
parents: 24534
diff changeset
325
25818
2be7cf62b9ee eliminate some nearly duplicate code in __run_test_suite__
John W. Eaton <jwe@octave.org>
parents: 25728
diff changeset
326 for i = 1:numel (prefix)
25007
2f695cf8af6f test: reduce relative file names in test suite output
Mike Miller <mtmiller@octave.org>
parents: 24534
diff changeset
327 tmp = strrep (nm, [prefix{i}, filesep], "");
2f695cf8af6f test: reduce relative file names in test suite output
Mike Miller <mtmiller@octave.org>
parents: 24534
diff changeset
328 if (length (tmp) < length (retval))
2f695cf8af6f test: reduce relative file names in test suite output
Mike Miller <mtmiller@octave.org>
parents: 24534
diff changeset
329 retval = tmp;
2f695cf8af6f test: reduce relative file names in test suite output
Mike Miller <mtmiller@octave.org>
parents: 24534
diff changeset
330 endif
2f695cf8af6f test: reduce relative file names in test suite output
Mike Miller <mtmiller@octave.org>
parents: 24534
diff changeset
331 endfor
2f695cf8af6f test: reduce relative file names in test suite output
Mike Miller <mtmiller@octave.org>
parents: 24534
diff changeset
332
2f695cf8af6f test: reduce relative file names in test suite output
Mike Miller <mtmiller@octave.org>
parents: 24534
diff changeset
333 endfunction
2f695cf8af6f test: reduce relative file names in test suite output
Mike Miller <mtmiller@octave.org>
parents: 24534
diff changeset
334
16019
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
335 function retval = has_functions (f)
21758
ffad2baa90f7 maint: Use newlines to make code more readable.
Rik <rik@octave.org>
parents: 21634
diff changeset
336
16019
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
337 n = length (f);
17182
fa724bdd52d0 __run_test_suite__.m: Recode regexps for performance. Don't search private '.' dirs.
Rik <rik@octave.org>
parents: 16994
diff changeset
338 if (n > 3 && strcmpi (f((end-2):end), ".cc"))
16019
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
339 fid = fopen (f);
21758
ffad2baa90f7 maint: Use newlines to make code more readable.
Rik <rik@octave.org>
parents: 21634
diff changeset
340 if (fid < 0)
20726
25d676f9619c Preface error() messages with name of function when possible.
Rik <rik@octave.org>
parents: 20426
diff changeset
341 error ("__run_test_suite__: fopen failed: %s", f);
16019
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
342 endif
21758
ffad2baa90f7 maint: Use newlines to make code more readable.
Rik <rik@octave.org>
parents: 21634
diff changeset
343 str = fread (fid, "*char")';
ffad2baa90f7 maint: Use newlines to make code more readable.
Rik <rik@octave.org>
parents: 21634
diff changeset
344 fclose (fid);
ffad2baa90f7 maint: Use newlines to make code more readable.
Rik <rik@octave.org>
parents: 21634
diff changeset
345 retval = ! isempty (regexp (str,'^(DEFUN|DEFUN_DLD)\>',
ffad2baa90f7 maint: Use newlines to make code more readable.
Rik <rik@octave.org>
parents: 21634
diff changeset
346 'lineanchors', 'once'));
17182
fa724bdd52d0 __run_test_suite__.m: Recode regexps for performance. Don't search private '.' dirs.
Rik <rik@octave.org>
parents: 16994
diff changeset
347 elseif (n > 2 && strcmpi (f((end-1):end), ".m"))
16019
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
348 retval = true;
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
349 else
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
350 retval = false;
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
351 endif
21758
ffad2baa90f7 maint: Use newlines to make code more readable.
Rik <rik@octave.org>
parents: 21634
diff changeset
352
16019
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
353 endfunction
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
354
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
355 function retval = has_tests (f)
21758
ffad2baa90f7 maint: Use newlines to make code more readable.
Rik <rik@octave.org>
parents: 21634
diff changeset
356
16019
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
357 fid = fopen (f);
21758
ffad2baa90f7 maint: Use newlines to make code more readable.
Rik <rik@octave.org>
parents: 21634
diff changeset
358 if (fid < 0)
20726
25d676f9619c Preface error() messages with name of function when possible.
Rik <rik@octave.org>
parents: 20426
diff changeset
359 error ("__run_test_suite__: fopen failed: %s", f);
16019
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
360 endif
21758
ffad2baa90f7 maint: Use newlines to make code more readable.
Rik <rik@octave.org>
parents: 21634
diff changeset
361
ffad2baa90f7 maint: Use newlines to make code more readable.
Rik <rik@octave.org>
parents: 21634
diff changeset
362 str = fread (fid, "*char")';
ffad2baa90f7 maint: Use newlines to make code more readable.
Rik <rik@octave.org>
parents: 21634
diff changeset
363 fclose (fid);
ffad2baa90f7 maint: Use newlines to make code more readable.
Rik <rik@octave.org>
parents: 21634
diff changeset
364 retval = ! isempty (regexp (str,
ffad2baa90f7 maint: Use newlines to make code more readable.
Rik <rik@octave.org>
parents: 21634
diff changeset
365 '^%!(assert|error|fail|test|xtest|warning)',
ffad2baa90f7 maint: Use newlines to make code more readable.
Rik <rik@octave.org>
parents: 21634
diff changeset
366 'lineanchors', 'once'));
ffad2baa90f7 maint: Use newlines to make code more readable.
Rik <rik@octave.org>
parents: 21634
diff changeset
367
16019
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
368 endfunction
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
369
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
370 function n = num_elts_matching_pattern (lst, pat)
17386
6dbc866379e2 Replace cellfun() occurrences with faster code where possible.
Rik <rik@octave.org>
parents: 17338
diff changeset
371 n = sum (! cellfun ("isempty", regexp (lst, pat, 'once')));
16019
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
372 endfunction
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
373
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
374 function report_files_with_no_tests (with, without, typ)
30379
363fb10055df maint: Style check m-files ahead of 7.1 release.
Rik <rik@octave.org>
parents: 30107
diff changeset
375
16994
333243133364 Use matrix concatenation for strings, rather than cstrcat(), for clarity and performance.
Rik <rik@octave.org>
parents: 16816
diff changeset
376 pat = ['\' typ "$"];
16019
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
377 n_with = num_elts_matching_pattern (with, pat);
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
378 n_without = num_elts_matching_pattern (without, pat);
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
379 n_tot = n_with + n_without;
21634
96518f623c91 Backed out changeset dcf8922b724b
Mike Miller <mtmiller@octave.org>
parents: 21633
diff changeset
380 printf ("\n%d (of %d) %s files have no tests.\n", n_without, n_tot, typ);
30379
363fb10055df maint: Style check m-files ahead of 7.1 release.
Rik <rik@octave.org>
parents: 30107
diff changeset
381
16019
5b2126a8c84f install tests and functions to run them
John W. Eaton <jwe@octave.org>
parents:
diff changeset
382 endfunction
17338
1c89599167a6 maint: End m-files with 1 blank line.
Rik <rik@octave.org>
parents: 17182
diff changeset
383
19171
702aa79dc482 Add BIST tests for various quasi-internal m-files.
Rik <rik@octave.org>
parents: 18168
diff changeset
384
702aa79dc482 Add BIST tests for various quasi-internal m-files.
Rik <rik@octave.org>
parents: 18168
diff changeset
385 ## No test coverage for internal function. It is tested through calling fcn.
702aa79dc482 Add BIST tests for various quasi-internal m-files.
Rik <rik@octave.org>
parents: 18168
diff changeset
386 %!assert (1)