Rewrite gsvd function and return *correct* 3rd output (bug #60273). Correctly use the API interface to LAPACK which Octave got wrong for certain matrix combinations. Perform post-processing on LAPACK results to return Matlab-compatible outputs. * libinterp/corefcn/gsvd.cc (gsvd_type): Add second input (nargin) to be able to distinguish the type of gsvd to generate. * libinterp/corefcn/gsvd.cc (do_gsvd): Add another input (nargin) to be able to calculate the correct gsvd type. Update code for API change in liboctave which now returns a full Matrix for singular values rather than a DiagMatrix. Sort singular values in ascending order for compatibility with Matlab. * libinterp/corefcn/gsvd.cc (Fgsvd): Update documation to warn about rank-deficient input case. Add new input validation to check that number of columns of A and B match. Move special check for empty matrices from libinterp to liboctave. Update all BIST tests for new behavior of gsvd code in liboctave. * liboctave/numeric/gsvd.h (gsvd): Remove member variable R from gsvd class. * liboctave/numeric/gsvd.h (gsvd::singular_values_A, gsvd_singular_values_B): Change return type to T::real_matrix_type from T::real_diag_matrix_type. * liboctave/numeric/gsvd.h (gsvd::ggsvd): Change default gsvd::Type to "std" from "economy". * liboctave/numeric/gsvd.cc: Add #includes for <algorithm>, <undordered_map>, and "oct-locbuf.h". Change gsvd_fcn from std::map to std::unordered_map. * liboctave/numeric/gsvd.cc (initialize_gsvd): Throw an error if Octave is unable to connect to LAPACK library. * liboctave/numeric/gsvd.cc (ggsvd): Change function prototype to not use references to Octave Matrix datatypes, but rather point directly to underlying plain old datatype. For example, use "double *" rather than "Matrix&". Replace scratch memory instances created with Matrix() with OCTAVE_LOCAL_BUFFER which relies on C++ STL. * liboctave/numeric/gsvd.cc (left_singular_matrix_A, left_singular_matrix_B, right_singular_matrix): Use the fact that current_liboctave_error_handler never returns to eliminate else branch of if/else code. * liboctave/numeric/gsvd.cc (R_matrix): Delete unused function. * liboctave/numeric/gsvd.cc (gsvd): Add input validation for empty matrices. Simplify 'job' values processing which seems to have been fixed in LAPACK 3.0. Replace scratch memory created with std::vector with OCTAVE_LOCAL_BUFFER. Replace scratch memory created with Octave Matrix classes with OCTAVE_LOCAL_BUFFER. Use initializer list to std::max to simplify code. Re-code interface to LAPACK to extract 'R' matrix. Post-process to calculate third output of gsvd as X = Q*R'. Correct algorithm to extract singular values from LAPACK..

--- a/liboctave/numeric/gsvd.h	Thu May 27 22:49:26 2021 +0200
+++ b/liboctave/numeric/gsvd.h	Wed Sep 08 16:08:03 2021 -0700
@@ -50,13 +50,13 @@
       { }
 
       gsvd (const T& a, const T& b,
-            gsvd::Type gsvd_type = gsvd<T>::Type::economy);
+            gsvd::Type gsvd_type = gsvd<T>::Type::std);
 
       gsvd (const gsvd& a)
         : type (a.type),
           sigmaA (a.sigmaA), sigmaB (a.sigmaB),
-          left_smA (a.left_smA), left_smB (a.left_smB), right_sm (a.right_sm),
-          R(a.R) { }
+          left_smA (a.left_smA), left_smB (a.left_smB), right_sm (a.right_sm)
+      { }
 
       gsvd& operator = (const gsvd& a)
       {
@@ -68,7 +68,6 @@
             left_smA = a.left_smA;
             left_smB = a.left_smB;
             right_sm = a.right_sm;
-            R = a.R;
           }
 
         return *this;
@@ -76,26 +75,25 @@
 
       ~gsvd (void) = default;
 
-      typename T::real_diag_matrix_type
+      typename T::real_matrix_type
       singular_values_A (void) const { return sigmaA; }
 
-      typename T::real_diag_matrix_type
+      typename T::real_matrix_type
       singular_values_B (void) const { return sigmaB; }
 
       T left_singular_matrix_A (void) const;
       T left_singular_matrix_B (void) const;
 
       T right_singular_matrix (void) const;
-      T R_matrix (void) const;
 
     private:
       typedef typename T::value_type P;
       typedef typename T::real_matrix_type real_matrix;
 
       gsvd::Type type;
-      typename T::real_diag_matrix_type sigmaA, sigmaB;
+      real_matrix sigmaA, sigmaB;
       T left_smA, left_smB;
-      T right_sm, R;
+      T right_sm;
 
       void ggsvd (char& jobu, char& jobv, char& jobq, octave_f77_int_type m,
                   octave_f77_int_type n, octave_f77_int_type p,
@@ -106,7 +104,7 @@
                   P *u, octave_f77_int_type nrow_u,
                   P *v, octave_f77_int_type nrow_v,
                   P *q, octave_f77_int_type nrow_q,
-                  T& work, octave_f77_int_type lwork,
+                  P *work, octave_f77_int_type lwork,
                   octave_f77_int_type *iwork,
                   octave_f77_int_type& info);
     };