Mercurial > octave

changeset 24087:88a593fa1388

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Stop out-of-bounds memory access during convolution (bug #52067). * oct-convn.cc (convolve): Check size of return matrix and short-circuit return from function if it is empty (as can happen when shape == "valid").
author Rik <rik@octave.org>
date Fri, 22 Sep 2017 13:36:28 -0700
parents 2a9eec536cfd
children 0160a3199b2d
files liboctave/numeric/oct-convn.cc
diffstat 1 files changed, 5 insertions(+), 0 deletions(-) [+]
line wrap: on
line diff
--- a/liboctave/numeric/oct-convn.cc	Fri Sep 22 12:46:53 2017 -0700
+++ b/liboctave/numeric/oct-convn.cc	Fri Sep 22 13:36:28 2017 -0700
@@ -152,6 +152,11 @@
 
   MArray<T> c (cdims, T ());
 
+  // "valid" shape can sometimes result in empty matrices which must avoid
+  // calling Fortran code which does not expect this (bug #52067)
+  if (c.isempty ())
+    return c;
+
   convolve_nd<T, R> (a.fortran_vec (), adims, adims.cumulative (),
                      b.fortran_vec (), bdims, bdims.cumulative (),
                      c.fortran_vec (), cdims.cumulative (),