changeset 18952:c66029adf853

Fix out-of-bound indexing in update_normals (bug #42823). * graphics.cc (surface::properties::update_normals): Check that dimensions of xdata, ydata, and zdata are appropriately matched before attempting to calculate vertexnormals.
author Rik <rik@octave.org>
date Tue, 22 Jul 2014 22:46:48 -0700
parents 5bfedd39cc77
children 914c3ce73665
files libinterp/corefcn/graphics.cc
diffstat 1 files changed, 15 insertions(+), 6 deletions(-) [+]
line wrap: on
line diff
--- a/libinterp/corefcn/graphics.cc	Tue Jul 22 12:05:28 2014 +0200
+++ b/libinterp/corefcn/graphics.cc	Tue Jul 22 22:46:48 2014 -0700
@@ -8061,18 +8061,27 @@
       Matrix y = get_ydata ().matrix_value ();
       Matrix z = get_zdata ().matrix_value ();
 
-
       int p = z.columns ();
       int q = z.rows ();
+
+      // FIXME: There might be a cleaner way to do this.  When data is changed
+      // the update_xdata, update_ydata, update_zdata routines are called in a
+      // serial fashion.  Until the final call to update_zdata the matrices
+      // will be of mismatched dimensions which can cause an out-of-bound
+      // indexing in the code below.  This one-liner prevents calculating
+      // normals until dimensions match.
+      if (x.columns () != p || y.rows () != q)
+        return;
+
+      NDArray n (dim_vector (q, p, 3), 0.0);
+
+      bool x_mat = (x.rows () == q);
+      bool y_mat = (y.columns () == p);
+
       int i1, i2, i3, j1, j2, j3;
       i1 = i2 = i3 = 0;
       j1 = j2 = j3 = 0;
 
-      bool x_mat = (x.rows () == q);
-      bool y_mat = (y.columns () == p);
-
-      NDArray n (dim_vector (q, p, 3), 0.0);
-
       for (int i = 0; i < p; i++)
         {
           if (y_mat)