annotate libinterp/parse-tree/lex.h @ 16275:dd7de0345124

10/10 commits reworking the lexer This and the previous nine changesets completely revise the way significant whitespace is handled inside matrix and cell lists. Now instead of indefinite look ahead, we keep track of all tokens read and look back to previous tokens and ahead at most one character. If the surrounding context changes the parse, we unput the text for the current token and alter the start state, unput an additional character (typically a comma or semicolon) or both. Comments are now handled by a separate set of exlusive start states and there is no explicit handling of comments within any other patterns.
author John W. Eaton <jwe@octave.org>
date Mon, 11 Mar 2013 14:40:29 -0400
parents 15f55df088e7
children 26fd237a3427 8cb65fd72164
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1
78fd87e624cb [project @ 1993-08-08 01:13:40 by jwe]
jwe
parents:
diff changeset
1 /*
78fd87e624cb [project @ 1993-08-08 01:13:40 by jwe]
jwe
parents:
diff changeset
2
14138
72c96de7a403 maint: update copyright notices for 2012
John W. Eaton <jwe@octave.org>
parents: 13973
diff changeset
3 Copyright (C) 1993-2012 John W. Eaton
1
78fd87e624cb [project @ 1993-08-08 01:13:40 by jwe]
jwe
parents:
diff changeset
4
78fd87e624cb [project @ 1993-08-08 01:13:40 by jwe]
jwe
parents:
diff changeset
5 This file is part of Octave.
78fd87e624cb [project @ 1993-08-08 01:13:40 by jwe]
jwe
parents:
diff changeset
6
78fd87e624cb [project @ 1993-08-08 01:13:40 by jwe]
jwe
parents:
diff changeset
7 Octave is free software; you can redistribute it and/or modify it
78fd87e624cb [project @ 1993-08-08 01:13:40 by jwe]
jwe
parents:
diff changeset
8 under the terms of the GNU General Public License as published by the
7016
93c65f2a5668 [project @ 2007-10-12 06:40:56 by jwe]
jwe
parents: 5775
diff changeset
9 Free Software Foundation; either version 3 of the License, or (at your
93c65f2a5668 [project @ 2007-10-12 06:40:56 by jwe]
jwe
parents: 5775
diff changeset
10 option) any later version.
1
78fd87e624cb [project @ 1993-08-08 01:13:40 by jwe]
jwe
parents:
diff changeset
11
78fd87e624cb [project @ 1993-08-08 01:13:40 by jwe]
jwe
parents:
diff changeset
12 Octave is distributed in the hope that it will be useful, but WITHOUT
78fd87e624cb [project @ 1993-08-08 01:13:40 by jwe]
jwe
parents:
diff changeset
13 ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
78fd87e624cb [project @ 1993-08-08 01:13:40 by jwe]
jwe
parents:
diff changeset
14 FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
78fd87e624cb [project @ 1993-08-08 01:13:40 by jwe]
jwe
parents:
diff changeset
15 for more details.
78fd87e624cb [project @ 1993-08-08 01:13:40 by jwe]
jwe
parents:
diff changeset
16
78fd87e624cb [project @ 1993-08-08 01:13:40 by jwe]
jwe
parents:
diff changeset
17 You should have received a copy of the GNU General Public License
7016
93c65f2a5668 [project @ 2007-10-12 06:40:56 by jwe]
jwe
parents: 5775
diff changeset
18 along with Octave; see the file COPYING. If not, see
93c65f2a5668 [project @ 2007-10-12 06:40:56 by jwe]
jwe
parents: 5775
diff changeset
19 <http://www.gnu.org/licenses/>.
1
78fd87e624cb [project @ 1993-08-08 01:13:40 by jwe]
jwe
parents:
diff changeset
20
78fd87e624cb [project @ 1993-08-08 01:13:40 by jwe]
jwe
parents:
diff changeset
21 */
78fd87e624cb [project @ 1993-08-08 01:13:40 by jwe]
jwe
parents:
diff changeset
22
383
0b52c68ec81f [project @ 1994-03-09 21:19:44 by jwe]
jwe
parents: 247
diff changeset
23 #if !defined (octave_lex_h)
0b52c68ec81f [project @ 1994-03-09 21:19:44 by jwe]
jwe
parents: 247
diff changeset
24 #define octave_lex_h 1
1
78fd87e624cb [project @ 1993-08-08 01:13:40 by jwe]
jwe
parents:
diff changeset
25
16230
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
26 #include <deque>
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
27 #include <limits>
8745
6dc61981d18b better handling of object indexing in lexer
John W. Eaton <jwe@octave.org>
parents: 8701
diff changeset
28 #include <list>
16101
8d19626b38ae provide copy contructor and operator = for lexical_feedback class.
John W. Eaton <jwe@octave.org>
parents: 16100
diff changeset
29 #include <set>
11464
21b5284fa78d avoid error when parsing nested functions
John W. Eaton <jwe@octave.org>
parents: 11367
diff changeset
30 #include <stack>
8745
6dc61981d18b better handling of object indexing in lexer
John W. Eaton <jwe@octave.org>
parents: 8701
diff changeset
31
16228
e19b1632d7c1 revamp most comment handling
John W. Eaton <jwe@octave.org>
parents: 16224
diff changeset
32 #include "comment-list.h"
16195
b52d2f9294b6 use class for reading lexer input
John W. Eaton <jwe@octave.org>
parents: 16194
diff changeset
33 #include "input.h"
16230
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
34 #include "token.h"
16195
b52d2f9294b6 use class for reading lexer input
John W. Eaton <jwe@octave.org>
parents: 16194
diff changeset
35
13973
2c664266e9d0 clean up parser memory on exit
John W. Eaton <jwe@octave.org>
parents: 13237
diff changeset
36 extern OCTINTERP_API void cleanup_parser (void);
2c664266e9d0 clean up parser memory on exit
John W. Eaton <jwe@octave.org>
parents: 13237
diff changeset
37
4867
c7f9ea142fda [project @ 2004-04-20 19:41:11 by jwe]
jwe
parents: 4753
diff changeset
38 // Is the given string a keyword?
c7f9ea142fda [project @ 2004-04-20 19:41:11 by jwe]
jwe
parents: 4753
diff changeset
39 extern bool is_keyword (const std::string& s);
c7f9ea142fda [project @ 2004-04-20 19:41:11 by jwe]
jwe
parents: 4753
diff changeset
40
16113
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
41 class
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
42 stream_reader
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
43 {
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
44 public:
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
45 virtual int getc (void) = 0;
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
46 virtual int ungetc (int c) = 0;
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
47
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
48 protected:
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
49 stream_reader (void) { }
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
50 ~stream_reader (void) { }
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
51
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
52 private:
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
53
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
54 // No copying!
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
55 stream_reader (const stream_reader&);
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
56 stream_reader& operator = (const stream_reader&);
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
57 };
16111
3ec4f6488569 move token stack to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16110
diff changeset
58
1826
b14829582cc4 [project @ 1996-02-02 03:05:07 by jwe]
jwe
parents: 1315
diff changeset
59 // For communication between the lexer and parser.
b14829582cc4 [project @ 1996-02-02 03:05:07 by jwe]
jwe
parents: 1315
diff changeset
60
3585
d9803711e047 [project @ 2000-02-08 04:35:39 by jwe]
jwe
parents: 3351
diff changeset
61 class
16193
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
62 lexical_feedback
1826
b14829582cc4 [project @ 1996-02-02 03:05:07 by jwe]
jwe
parents: 1315
diff changeset
63 {
b14829582cc4 [project @ 1996-02-02 03:05:07 by jwe]
jwe
parents: 1315
diff changeset
64 public:
b14829582cc4 [project @ 1996-02-02 03:05:07 by jwe]
jwe
parents: 1315
diff changeset
65
16104
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
66 // Track nesting of square brackets, curly braces, and parentheses.
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
67
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
68 class bbp_nesting_level
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
69 {
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
70 private:
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
71
16107
3b791008b88e give name to anonymous enum
John W. Eaton <jwe@octave.org>
parents: 16106
diff changeset
72 enum bracket_type
16104
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
73 {
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
74 BRACKET = 1,
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
75 BRACE = 2,
16265
71ee3afedb69 5/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16263
diff changeset
76 PAREN = 3,
71ee3afedb69 5/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16263
diff changeset
77 ANON_FCN_BODY = 4
16104
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
78 };
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
79
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
80 public:
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
81
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
82 bbp_nesting_level (void) : context () { }
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
83
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
84 bbp_nesting_level (const bbp_nesting_level& nl) : context (nl.context) { }
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
85
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
86 bbp_nesting_level& operator = (const bbp_nesting_level& nl)
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
87 {
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
88 if (&nl != this)
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
89 context = nl.context;
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
90
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
91 return *this;
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
92 }
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
93
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
94 ~bbp_nesting_level (void) { }
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
95
16194
b7ca669af528 reset lexical_feedback state before starting a new parse
John W. Eaton <jwe@octave.org>
parents: 16193
diff changeset
96 void reset (void)
b7ca669af528 reset lexical_feedback state before starting a new parse
John W. Eaton <jwe@octave.org>
parents: 16193
diff changeset
97 {
b7ca669af528 reset lexical_feedback state before starting a new parse
John W. Eaton <jwe@octave.org>
parents: 16193
diff changeset
98 while (! context.empty ())
b7ca669af528 reset lexical_feedback state before starting a new parse
John W. Eaton <jwe@octave.org>
parents: 16193
diff changeset
99 context.pop ();
b7ca669af528 reset lexical_feedback state before starting a new parse
John W. Eaton <jwe@octave.org>
parents: 16193
diff changeset
100 }
b7ca669af528 reset lexical_feedback state before starting a new parse
John W. Eaton <jwe@octave.org>
parents: 16193
diff changeset
101
16104
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
102 void bracket (void) { context.push (BRACKET); }
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
103
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
104 bool is_bracket (void)
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
105 {
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
106 return ! context.empty () && context.top () == BRACKET;
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
107 }
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
108
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
109 void brace (void) { context.push (BRACE); }
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
110
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
111 bool is_brace (void)
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
112 {
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
113 return ! context.empty () && context.top () == BRACE;
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
114 }
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
115
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
116 void paren (void) { context.push (PAREN); }
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
117
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
118 bool is_paren (void)
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
119 {
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
120 return ! context.empty () && context.top () == PAREN;
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
121 }
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
122
16265
71ee3afedb69 5/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16263
diff changeset
123 void anon_fcn_body (void) { context.push (ANON_FCN_BODY); }
71ee3afedb69 5/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16263
diff changeset
124
71ee3afedb69 5/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16263
diff changeset
125 bool is_anon_fcn_body (void)
71ee3afedb69 5/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16263
diff changeset
126 {
71ee3afedb69 5/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16263
diff changeset
127 return ! context.empty () && context.top () == ANON_FCN_BODY;
71ee3afedb69 5/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16263
diff changeset
128 }
71ee3afedb69 5/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16263
diff changeset
129
16104
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
130 bool is_bracket_or_brace (void)
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
131 {
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
132 return (! context.empty ()
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
133 && (context.top () == BRACKET || context.top () == BRACE));
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
134 }
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
135
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
136 bool none (void) { return context.empty (); }
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
137
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
138 void remove (void)
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
139 {
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
140 if (! context.empty ())
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
141 context.pop ();
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
142 }
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
143
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
144 void clear (void)
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
145 {
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
146 while (! context.empty ())
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
147 context.pop ();
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
148 }
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
149
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
150 private:
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
151
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
152 std::stack<int> context;
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
153 };
16230
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
154
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
155 class token_cache
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
156 {
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
157 public:
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
158
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
159 // Store an "unlimited" number of tokens.
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
160 token_cache (size_t sz_arg = std::numeric_limits<size_t>::max ())
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
161 : buffer (), sz (sz_arg)
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
162 { }
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
163
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
164 void push (token *tok)
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
165 {
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
166 if (buffer.size () == sz)
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
167 pop ();
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
168
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
169 buffer.push_front (tok);
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
170 }
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
171
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
172 void pop (void)
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
173 {
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
174 if (! empty ())
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
175 {
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
176 delete buffer.back ();
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
177 buffer.pop_back ();
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
178 }
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
179 }
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
180
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
181 // Direct access.
16255
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
182 token *at (size_t n)
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
183 {
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
184 return empty () ? 0 : buffer.at (n);
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
185 }
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
186
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
187 const token *at (size_t n) const
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
188 {
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
189 return empty () ? 0 : buffer.at (n);
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
190 }
16230
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
191
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
192 // Most recently pushed.
16255
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
193 token *front (void)
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
194 {
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
195 return empty () ? 0 : buffer.front ();
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
196 }
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
197
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
198 const token *front (void) const
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
199 {
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
200 return empty () ? 0 : buffer.front ();
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
201 }
16230
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
202
16255
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
203 token *back (void)
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
204 {
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
205 return empty () ? 0 : buffer.back ();
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
206 }
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
207
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
208 const token *back (void) const
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
209 {
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
210 return empty () ? 0 : buffer.back ();
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
211 }
16230
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
212
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
213 // Number of elements currently in the buffer, max of sz.
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
214 size_t size (void) const { return buffer.size (); }
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
215
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
216 bool empty (void) const { return buffer.empty (); }
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
217
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
218 void clear (void)
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
219 {
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
220 while (! empty ())
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
221 pop ();
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
222 }
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
223
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
224 private:
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
225
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
226 std::deque<token *> buffer;
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
227
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
228 size_t sz;
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
229
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
230 // No copying!
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
231
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
232 token_cache (const token_cache&);
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
233
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
234 token_cache& operator = (const token_cache&);
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
235 };
16193
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
236
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
237 lexical_feedback (void)
16259
0b5ab09dfce4 2/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16257
diff changeset
238 : end_of_input (false), at_beginning_of_statement (true),
16193
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
239 looking_at_anon_fcn_args (false), looking_at_return_list (false),
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
240 looking_at_parameter_list (false), looking_at_decl_list (false),
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
241 looking_at_initializer_expression (false),
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
242 looking_at_matrix_or_assign_lhs (false),
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
243 looking_for_object_index (false),
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
244 looking_at_indirect_ref (false), parsing_class_method (false),
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
245 maybe_classdef_get_set_method (false), parsing_classdef (false),
16203
127cccb037bf move more global parser and lexer variables to classes
John W. Eaton <jwe@octave.org>
parents: 16199
diff changeset
246 quote_is_transpose (false), force_script (false),
127cccb037bf move more global parser and lexer variables to classes
John W. Eaton <jwe@octave.org>
parents: 16199
diff changeset
247 reading_fcn_file (false), reading_script_file (false),
127cccb037bf move more global parser and lexer variables to classes
John W. Eaton <jwe@octave.org>
parents: 16199
diff changeset
248 reading_classdef_file (false),
16193
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
249 input_line_number (1), current_input_column (1),
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
250 bracketflag (0), braceflag (0),
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
251 looping (0), defining_func (0), looking_at_function_handle (0),
16203
127cccb037bf move more global parser and lexer variables to classes
John W. Eaton <jwe@octave.org>
parents: 16199
diff changeset
252 block_comment_nesting_level (0), token_count (0),
16228
e19b1632d7c1 revamp most comment handling
John W. Eaton <jwe@octave.org>
parents: 16224
diff changeset
253 current_input_line (), comment_text (), help_text (),
e19b1632d7c1 revamp most comment handling
John W. Eaton <jwe@octave.org>
parents: 16224
diff changeset
254 fcn_file_name (), fcn_file_full_name (), looking_at_object_index (),
16207
0467d68ca891 move current_input_line to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16203
diff changeset
255 parsed_function_name (), pending_local_variables (),
16230
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
256 nesting_level (), tokens ()
16193
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
257 {
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
258 init ();
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
259 }
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
260
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
261 ~lexical_feedback (void);
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
262
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
263 void init (void);
16194
b7ca669af528 reset lexical_feedback state before starting a new parse
John W. Eaton <jwe@octave.org>
parents: 16193
diff changeset
264
b7ca669af528 reset lexical_feedback state before starting a new parse
John W. Eaton <jwe@octave.org>
parents: 16193
diff changeset
265 void reset (void);
b7ca669af528 reset lexical_feedback state before starting a new parse
John W. Eaton <jwe@octave.org>
parents: 16193
diff changeset
266
16255
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
267 int previous_token_value (void) const;
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
268
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
269 bool previous_token_value_is (int tok_val) const;
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
270
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
271 void mark_previous_token_trailing_space (void);
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
272
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
273 bool space_follows_previous_token (void) const;
12bf6a3f8c45 store more info in token value class
John W. Eaton <jwe@octave.org>
parents: 16253
diff changeset
274
16257
db7f07b22b9b 1/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16255
diff changeset
275 bool previous_token_is_binop (void) const;
db7f07b22b9b 1/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16255
diff changeset
276
16267
15f55df088e7 6/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16265
diff changeset
277 bool previous_token_is_keyword (void) const;
15f55df088e7 6/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16265
diff changeset
278
16257
db7f07b22b9b 1/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16255
diff changeset
279 bool previous_token_may_be_command (void) const;
db7f07b22b9b 1/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16255
diff changeset
280
16193
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
281 // true means that we have encountered eof on the input stream.
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
282 bool end_of_input;
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
283
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
284 // true means we are at the beginning of a statement, where a
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
285 // command name is possible.
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
286 bool at_beginning_of_statement;
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
287
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
288 // true means we are parsing an anonymous function argument list.
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
289 bool looking_at_anon_fcn_args;
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
290
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
291 // true means we're parsing the return list for a function.
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
292 bool looking_at_return_list;
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
293
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
294 // true means we're parsing the parameter list for a function.
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
295 bool looking_at_parameter_list;
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
296
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
297 // true means we're parsing a declaration list (global or
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
298 // persistent).
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
299 bool looking_at_decl_list;
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
300
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
301 // true means we are looking at the initializer expression for a
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
302 // parameter list element.
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
303 bool looking_at_initializer_expression;
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
304
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
305 // true means we're parsing a matrix or the left hand side of
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
306 // multi-value assignment statement.
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
307 bool looking_at_matrix_or_assign_lhs;
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
308
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
309 // object index not possible until we've seen something.
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
310 bool looking_for_object_index;
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
311
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
312 // true means we're looking at an indirect reference to a
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
313 // structure element.
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
314 bool looking_at_indirect_ref;
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
315
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
316 // true means we are parsing a class method in function or classdef file.
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
317 bool parsing_class_method;
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
318
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
319 // true means we are parsing a class method declaration line in a
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
320 // classdef file and can accept a property get or set method name.
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
321 // for example, "get.propertyname" is recognized as a function name.
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
322 bool maybe_classdef_get_set_method;
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
323
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
324 // true means we are parsing a classdef file
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
325 bool parsing_classdef;
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
326
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
327 // return transpose or start a string?
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
328 bool quote_is_transpose;
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
329
16203
127cccb037bf move more global parser and lexer variables to classes
John W. Eaton <jwe@octave.org>
parents: 16199
diff changeset
330 // TRUE means treat the current file as a script even if the first
127cccb037bf move more global parser and lexer variables to classes
John W. Eaton <jwe@octave.org>
parents: 16199
diff changeset
331 // token is "function" or "classdef".
127cccb037bf move more global parser and lexer variables to classes
John W. Eaton <jwe@octave.org>
parents: 16199
diff changeset
332 bool force_script;
127cccb037bf move more global parser and lexer variables to classes
John W. Eaton <jwe@octave.org>
parents: 16199
diff changeset
333
16199
810a71122c25 move more global variables to octave_lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16195
diff changeset
334 // TRUE means we're parsing a function file.
810a71122c25 move more global variables to octave_lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16195
diff changeset
335 bool reading_fcn_file;
810a71122c25 move more global variables to octave_lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16195
diff changeset
336
810a71122c25 move more global variables to octave_lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16195
diff changeset
337 // TRUE means we're parsing a script file.
810a71122c25 move more global variables to octave_lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16195
diff changeset
338 bool reading_script_file;
810a71122c25 move more global variables to octave_lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16195
diff changeset
339
810a71122c25 move more global variables to octave_lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16195
diff changeset
340 // TRUE means we're parsing a classdef file.
810a71122c25 move more global variables to octave_lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16195
diff changeset
341 bool reading_classdef_file;
810a71122c25 move more global variables to octave_lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16195
diff changeset
342
16193
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
343 // the current input line number.
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
344 int input_line_number;
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
345
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
346 // the column of the current token.
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
347 int current_input_column;
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
348
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
349 // square bracket level count.
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
350 int bracketflag;
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
351
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
352 // curly brace level count.
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
353 int braceflag;
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
354
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
355 // true means we're in the middle of defining a loop.
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
356 int looping;
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
357
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
358 // nonzero means we're in the middle of defining a function.
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
359 int defining_func;
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
360
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
361 // nonzero means we are parsing a function handle.
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
362 int looking_at_function_handle;
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
363
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
364 // nestng level for blcok comments.
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
365 int block_comment_nesting_level;
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
366
16203
127cccb037bf move more global parser and lexer variables to classes
John W. Eaton <jwe@octave.org>
parents: 16199
diff changeset
367 // Count of tokens recognized by this lexer since initialized or
127cccb037bf move more global parser and lexer variables to classes
John W. Eaton <jwe@octave.org>
parents: 16199
diff changeset
368 // since the last reset.
127cccb037bf move more global parser and lexer variables to classes
John W. Eaton <jwe@octave.org>
parents: 16199
diff changeset
369 size_t token_count;
127cccb037bf move more global parser and lexer variables to classes
John W. Eaton <jwe@octave.org>
parents: 16199
diff changeset
370
16207
0467d68ca891 move current_input_line to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16203
diff changeset
371 // The current line of input.
0467d68ca891 move current_input_line to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16203
diff changeset
372 std::string current_input_line;
0467d68ca891 move current_input_line to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16203
diff changeset
373
16228
e19b1632d7c1 revamp most comment handling
John W. Eaton <jwe@octave.org>
parents: 16224
diff changeset
374 // The current comment text.
e19b1632d7c1 revamp most comment handling
John W. Eaton <jwe@octave.org>
parents: 16224
diff changeset
375 std::string comment_text;
e19b1632d7c1 revamp most comment handling
John W. Eaton <jwe@octave.org>
parents: 16224
diff changeset
376
16203
127cccb037bf move more global parser and lexer variables to classes
John W. Eaton <jwe@octave.org>
parents: 16199
diff changeset
377 // The current help text.
127cccb037bf move more global parser and lexer variables to classes
John W. Eaton <jwe@octave.org>
parents: 16199
diff changeset
378 std::string help_text;
127cccb037bf move more global parser and lexer variables to classes
John W. Eaton <jwe@octave.org>
parents: 16199
diff changeset
379
127cccb037bf move more global parser and lexer variables to classes
John W. Eaton <jwe@octave.org>
parents: 16199
diff changeset
380 // Simple name of function file we are reading.
127cccb037bf move more global parser and lexer variables to classes
John W. Eaton <jwe@octave.org>
parents: 16199
diff changeset
381 std::string fcn_file_name;
127cccb037bf move more global parser and lexer variables to classes
John W. Eaton <jwe@octave.org>
parents: 16199
diff changeset
382
127cccb037bf move more global parser and lexer variables to classes
John W. Eaton <jwe@octave.org>
parents: 16199
diff changeset
383 // Full name of file we are reading.
127cccb037bf move more global parser and lexer variables to classes
John W. Eaton <jwe@octave.org>
parents: 16199
diff changeset
384 std::string fcn_file_full_name;
127cccb037bf move more global parser and lexer variables to classes
John W. Eaton <jwe@octave.org>
parents: 16199
diff changeset
385
16193
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
386 // if the front of the list is true, the closest paren, brace, or
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
387 // bracket nesting is an index for an object.
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
388 std::list<bool> looking_at_object_index;
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
389
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
390 // if the top of the stack is true, then we've already seen the name
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
391 // of the current function. should only matter if
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
392 // current_function_level > 0
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
393 std::stack<bool> parsed_function_name;
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
394
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
395 // set of identifiers that might be local variable names.
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
396 std::set<std::string> pending_local_variables;
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
397
16265
71ee3afedb69 5/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16263
diff changeset
398 // is the closest nesting level a square bracket, squiggly brace,
71ee3afedb69 5/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16263
diff changeset
399 // a paren, or an anonymous function body?
16193
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
400 bbp_nesting_level nesting_level;
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
401
16230
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
402 // Tokens generated by the lexer.
4bf907906134 use a queue to hold tokens in the lexer
John W. Eaton <jwe@octave.org>
parents: 16229
diff changeset
403 token_cache tokens;
16193
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
404
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
405 private:
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
406
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
407 // No copying!
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
408
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
409 lexical_feedback (const lexical_feedback&);
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
410
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
411 lexical_feedback& operator = (const lexical_feedback&);
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
412 };
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
413
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
414 // octave_lexer inherits from lexical_feedback because we will
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
415 // eventually have several different constructors and it is easier to
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
416 // intialize if everything is grouped in a parent class rather than
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
417 // listing all the members in the octave_lexer class.
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
418
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
419 class
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
420 octave_lexer : public lexical_feedback
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
421 {
d7392bf42fd1 use inheritance to simplify initialization in octave_lexer constructor
John W. Eaton <jwe@octave.org>
parents: 16185
diff changeset
422 public:
16104
c8974e28da59 move nesting_level to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16103
diff changeset
423
16183
359d56094efa handle lexer input buffering with class
John W. Eaton <jwe@octave.org>
parents: 16164
diff changeset
424 // Handle buffering of input for lexer.
359d56094efa handle lexer input buffering with class
John W. Eaton <jwe@octave.org>
parents: 16164
diff changeset
425
359d56094efa handle lexer input buffering with class
John W. Eaton <jwe@octave.org>
parents: 16164
diff changeset
426 class input_buffer
359d56094efa handle lexer input buffering with class
John W. Eaton <jwe@octave.org>
parents: 16164
diff changeset
427 {
359d56094efa handle lexer input buffering with class
John W. Eaton <jwe@octave.org>
parents: 16164
diff changeset
428 public:
359d56094efa handle lexer input buffering with class
John W. Eaton <jwe@octave.org>
parents: 16164
diff changeset
429
359d56094efa handle lexer input buffering with class
John W. Eaton <jwe@octave.org>
parents: 16164
diff changeset
430 input_buffer (void)
359d56094efa handle lexer input buffering with class
John W. Eaton <jwe@octave.org>
parents: 16164
diff changeset
431 : buffer (), pos (0), chars_left (0), eof (false)
359d56094efa handle lexer input buffering with class
John W. Eaton <jwe@octave.org>
parents: 16164
diff changeset
432 { }
359d56094efa handle lexer input buffering with class
John W. Eaton <jwe@octave.org>
parents: 16164
diff changeset
433
16195
b52d2f9294b6 use class for reading lexer input
John W. Eaton <jwe@octave.org>
parents: 16194
diff changeset
434 void fill (const std::string& input, bool eof_arg);
16183
359d56094efa handle lexer input buffering with class
John W. Eaton <jwe@octave.org>
parents: 16164
diff changeset
435
359d56094efa handle lexer input buffering with class
John W. Eaton <jwe@octave.org>
parents: 16164
diff changeset
436 // Copy at most max_size characters to buf.
359d56094efa handle lexer input buffering with class
John W. Eaton <jwe@octave.org>
parents: 16164
diff changeset
437 int copy_chunk (char *buf, size_t max_size);
359d56094efa handle lexer input buffering with class
John W. Eaton <jwe@octave.org>
parents: 16164
diff changeset
438
359d56094efa handle lexer input buffering with class
John W. Eaton <jwe@octave.org>
parents: 16164
diff changeset
439 bool empty (void) const { return chars_left == 0; }
359d56094efa handle lexer input buffering with class
John W. Eaton <jwe@octave.org>
parents: 16164
diff changeset
440
359d56094efa handle lexer input buffering with class
John W. Eaton <jwe@octave.org>
parents: 16164
diff changeset
441 bool at_eof (void) const { return eof; }
359d56094efa handle lexer input buffering with class
John W. Eaton <jwe@octave.org>
parents: 16164
diff changeset
442
359d56094efa handle lexer input buffering with class
John W. Eaton <jwe@octave.org>
parents: 16164
diff changeset
443 private:
359d56094efa handle lexer input buffering with class
John W. Eaton <jwe@octave.org>
parents: 16164
diff changeset
444
359d56094efa handle lexer input buffering with class
John W. Eaton <jwe@octave.org>
parents: 16164
diff changeset
445 std::string buffer;
359d56094efa handle lexer input buffering with class
John W. Eaton <jwe@octave.org>
parents: 16164
diff changeset
446 const char *pos;
359d56094efa handle lexer input buffering with class
John W. Eaton <jwe@octave.org>
parents: 16164
diff changeset
447 size_t chars_left;
359d56094efa handle lexer input buffering with class
John W. Eaton <jwe@octave.org>
parents: 16164
diff changeset
448 bool eof;
359d56094efa handle lexer input buffering with class
John W. Eaton <jwe@octave.org>
parents: 16164
diff changeset
449 };
359d56094efa handle lexer input buffering with class
John W. Eaton <jwe@octave.org>
parents: 16164
diff changeset
450
16158
7eb614760ddb rename lexical_feedback to octave_lexer
John W. Eaton <jwe@octave.org>
parents: 16149
diff changeset
451 octave_lexer (void)
16195
b52d2f9294b6 use class for reading lexer input
John W. Eaton <jwe@octave.org>
parents: 16194
diff changeset
452 : lexical_feedback (), scanner (0), input_buf (), input_reader ()
b52d2f9294b6 use class for reading lexer input
John W. Eaton <jwe@octave.org>
parents: 16194
diff changeset
453 {
b52d2f9294b6 use class for reading lexer input
John W. Eaton <jwe@octave.org>
parents: 16194
diff changeset
454 init ();
b52d2f9294b6 use class for reading lexer input
John W. Eaton <jwe@octave.org>
parents: 16194
diff changeset
455 }
b52d2f9294b6 use class for reading lexer input
John W. Eaton <jwe@octave.org>
parents: 16194
diff changeset
456
b52d2f9294b6 use class for reading lexer input
John W. Eaton <jwe@octave.org>
parents: 16194
diff changeset
457 octave_lexer (FILE *file)
b52d2f9294b6 use class for reading lexer input
John W. Eaton <jwe@octave.org>
parents: 16194
diff changeset
458 : lexical_feedback (), scanner (0), input_buf (),
b52d2f9294b6 use class for reading lexer input
John W. Eaton <jwe@octave.org>
parents: 16194
diff changeset
459 input_reader (file)
b52d2f9294b6 use class for reading lexer input
John W. Eaton <jwe@octave.org>
parents: 16194
diff changeset
460 {
b52d2f9294b6 use class for reading lexer input
John W. Eaton <jwe@octave.org>
parents: 16194
diff changeset
461 init ();
b52d2f9294b6 use class for reading lexer input
John W. Eaton <jwe@octave.org>
parents: 16194
diff changeset
462 }
b52d2f9294b6 use class for reading lexer input
John W. Eaton <jwe@octave.org>
parents: 16194
diff changeset
463
b52d2f9294b6 use class for reading lexer input
John W. Eaton <jwe@octave.org>
parents: 16194
diff changeset
464 octave_lexer (const std::string& eval_string)
b52d2f9294b6 use class for reading lexer input
John W. Eaton <jwe@octave.org>
parents: 16194
diff changeset
465 : lexical_feedback (), scanner (0), input_buf (),
b52d2f9294b6 use class for reading lexer input
John W. Eaton <jwe@octave.org>
parents: 16194
diff changeset
466 input_reader (eval_string)
16101
8d19626b38ae provide copy contructor and operator = for lexical_feedback class.
John W. Eaton <jwe@octave.org>
parents: 16100
diff changeset
467 {
8d19626b38ae provide copy contructor and operator = for lexical_feedback class.
John W. Eaton <jwe@octave.org>
parents: 16100
diff changeset
468 init ();
8d19626b38ae provide copy contructor and operator = for lexical_feedback class.
John W. Eaton <jwe@octave.org>
parents: 16100
diff changeset
469 }
8d19626b38ae provide copy contructor and operator = for lexical_feedback class.
John W. Eaton <jwe@octave.org>
parents: 16100
diff changeset
470
16158
7eb614760ddb rename lexical_feedback to octave_lexer
John W. Eaton <jwe@octave.org>
parents: 16149
diff changeset
471 ~octave_lexer (void);
1826
b14829582cc4 [project @ 1996-02-02 03:05:07 by jwe]
jwe
parents: 1315
diff changeset
472
16149
49dfba4fd3c5 use pure parser and reentrant lexer interfaces
John W. Eaton <jwe@octave.org>
parents: 16139
diff changeset
473 void init (void);
1826
b14829582cc4 [project @ 1996-02-02 03:05:07 by jwe]
jwe
parents: 1315
diff changeset
474
16139
2fd39ab12209 move a function and data member from lexical_feedback to octave_parser
John W. Eaton <jwe@octave.org>
parents: 16130
diff changeset
475 void reset (void);
16122
6884401b2fbb move reset_parser lexer helper function to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16120
diff changeset
476
16203
127cccb037bf move more global parser and lexer variables to classes
John W. Eaton <jwe@octave.org>
parents: 16199
diff changeset
477 void prep_for_file (void);
16124
3be725cd195b move more lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16122
diff changeset
478
16195
b52d2f9294b6 use class for reading lexer input
John W. Eaton <jwe@octave.org>
parents: 16194
diff changeset
479 int read (char *buf, unsigned int max_size);
16120
4b68eb9b98b0 move octave_read lexer helper function to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16118
diff changeset
480
16164
c5bfdc4c0963 move end_of_input flag from octve_parser class to octave_lexer class
John W. Eaton <jwe@octave.org>
parents: 16158
diff changeset
481 int handle_end_of_input (void);
c5bfdc4c0963 move end_of_input flag from octve_parser class to octave_lexer class
John W. Eaton <jwe@octave.org>
parents: 16158
diff changeset
482
16128
210039e91ad6 localize use of yytext with lexical_feedback member function
John W. Eaton <jwe@octave.org>
parents: 16126
diff changeset
483 char *flex_yytext (void);
210039e91ad6 localize use of yytext with lexical_feedback member function
John W. Eaton <jwe@octave.org>
parents: 16126
diff changeset
484
16130
421dea028bbf localize use of yyleng with lexical_feedback member function
John W. Eaton <jwe@octave.org>
parents: 16129
diff changeset
485 int flex_yyleng (void);
421dea028bbf localize use of yyleng with lexical_feedback member function
John W. Eaton <jwe@octave.org>
parents: 16129
diff changeset
486
16113
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
487 int text_yyinput (void);
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
488
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
489 void xunput (char c, char *buf);
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
490
16125
96a58f197f93 allow xunput to be called without buffer argument
John W. Eaton <jwe@octave.org>
parents: 16124
diff changeset
491 void xunput (char c);
96a58f197f93 allow xunput to be called without buffer argument
John W. Eaton <jwe@octave.org>
parents: 16124
diff changeset
492
16275
dd7de0345124 10/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16267
diff changeset
493 bool looking_at_space (void);
dd7de0345124 10/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16267
diff changeset
494
16113
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
495 bool inside_any_object_index (void);
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
496
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
497 int is_keyword_token (const std::string& s);
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
498
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
499 bool is_variable (const std::string& name);
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
500
16257
db7f07b22b9b 1/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16255
diff changeset
501 bool whitespace_is_significant (void);
db7f07b22b9b 1/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16255
diff changeset
502
16113
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
503 void handle_number (void);
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
504
16253
a89cf57ba3a5 partial cleanup of continuation handling in lexer
John W. Eaton <jwe@octave.org>
parents: 16231
diff changeset
505 void handle_continuation (void);
a89cf57ba3a5 partial cleanup of continuation handling in lexer
John W. Eaton <jwe@octave.org>
parents: 16231
diff changeset
506
a89cf57ba3a5 partial cleanup of continuation handling in lexer
John W. Eaton <jwe@octave.org>
parents: 16231
diff changeset
507 void finish_comment (octave_comment_elt::comment_type typ,
a89cf57ba3a5 partial cleanup of continuation handling in lexer
John W. Eaton <jwe@octave.org>
parents: 16231
diff changeset
508 bool looking_at_continuation = false);
a89cf57ba3a5 partial cleanup of continuation handling in lexer
John W. Eaton <jwe@octave.org>
parents: 16231
diff changeset
509
16113
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
510 bool have_continuation (bool trailing_comments_ok = true);
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
511
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
512 bool have_ellipsis_continuation (bool trailing_comments_ok = true);
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
513
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
514 int handle_string (char delim);
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
515
16259
0b5ab09dfce4 2/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16257
diff changeset
516 int handle_close_bracket (int bracket_type);
16113
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
517
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
518 bool next_token_can_follow_bin_op (void);
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
519
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
520 bool looks_like_command_arg (void);
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
521
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
522 int handle_superclass_identifier (void);
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
523
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
524 int handle_meta_identifier (void);
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
525
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
526 int handle_identifier (void);
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
527
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
528 void maybe_warn_separator_insert (char sep);
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
529
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
530 void gripe_single_quote_string (void);
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
531
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
532 void gripe_matlab_incompatible (const std::string& msg);
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
533
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
534 void maybe_gripe_matlab_incompatible_comment (char c);
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
535
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
536 void gripe_matlab_incompatible_continuation (void);
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
537
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
538 void gripe_matlab_incompatible_operator (const std::string& op);
7c5e5e97a3bc move static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16111
diff changeset
539
16126
5c885c13bfa3 localize use of yylval.tok_val in lexical_feedback member functions
John W. Eaton <jwe@octave.org>
parents: 16125
diff changeset
540 void push_token (token *);
5c885c13bfa3 localize use of yylval.tok_val in lexical_feedback member functions
John W. Eaton <jwe@octave.org>
parents: 16125
diff changeset
541
5c885c13bfa3 localize use of yylval.tok_val in lexical_feedback member functions
John W. Eaton <jwe@octave.org>
parents: 16125
diff changeset
542 token *current_token (void);
5c885c13bfa3 localize use of yylval.tok_val in lexical_feedback member functions
John W. Eaton <jwe@octave.org>
parents: 16125
diff changeset
543
16118
f8e463523229 move more static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16115
diff changeset
544 void display_token (int tok);
f8e463523229 move more static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16115
diff changeset
545
16129
053b0364b507 localize use of yy_fatal_error with lexical_feedback member function
John W. Eaton <jwe@octave.org>
parents: 16128
diff changeset
546 void fatal_error (const char *msg);
053b0364b507 localize use of yy_fatal_error with lexical_feedback member function
John W. Eaton <jwe@octave.org>
parents: 16128
diff changeset
547
16229
7b7b1e4968e8 use functions instead of token return macros in lexer
John W. Eaton <jwe@octave.org>
parents: 16228
diff changeset
548 void lexer_debug (const char *pattern);
16118
f8e463523229 move more static lexer helper functions to lexical_feedback class
John W. Eaton <jwe@octave.org>
parents: 16115
diff changeset
549
16149
49dfba4fd3c5 use pure parser and reentrant lexer interfaces
John W. Eaton <jwe@octave.org>
parents: 16139
diff changeset
550 // Internal state of the flex-generated lexer.
49dfba4fd3c5 use pure parser and reentrant lexer interfaces
John W. Eaton <jwe@octave.org>
parents: 16139
diff changeset
551 void *scanner;
49dfba4fd3c5 use pure parser and reentrant lexer interfaces
John W. Eaton <jwe@octave.org>
parents: 16139
diff changeset
552
16183
359d56094efa handle lexer input buffering with class
John W. Eaton <jwe@octave.org>
parents: 16164
diff changeset
553 // Object that reads and buffers input.
359d56094efa handle lexer input buffering with class
John W. Eaton <jwe@octave.org>
parents: 16164
diff changeset
554 input_buffer input_buf;
359d56094efa handle lexer input buffering with class
John W. Eaton <jwe@octave.org>
parents: 16164
diff changeset
555
16195
b52d2f9294b6 use class for reading lexer input
John W. Eaton <jwe@octave.org>
parents: 16194
diff changeset
556 octave_input_reader input_reader;
b52d2f9294b6 use class for reading lexer input
John W. Eaton <jwe@octave.org>
parents: 16194
diff changeset
557
16209
e7ff32e7cf82 move global promptflag variable to octave_reader class
John W. Eaton <jwe@octave.org>
parents: 16207
diff changeset
558 void increment_promptflag (void) { input_reader.increment_promptflag (); }
e7ff32e7cf82 move global promptflag variable to octave_reader class
John W. Eaton <jwe@octave.org>
parents: 16207
diff changeset
559
e7ff32e7cf82 move global promptflag variable to octave_reader class
John W. Eaton <jwe@octave.org>
parents: 16207
diff changeset
560 void decrement_promptflag (void) { input_reader.decrement_promptflag (); }
e7ff32e7cf82 move global promptflag variable to octave_reader class
John W. Eaton <jwe@octave.org>
parents: 16207
diff changeset
561
e7ff32e7cf82 move global promptflag variable to octave_reader class
John W. Eaton <jwe@octave.org>
parents: 16207
diff changeset
562 int promptflag (void) const { return input_reader.promptflag (); }
e7ff32e7cf82 move global promptflag variable to octave_reader class
John W. Eaton <jwe@octave.org>
parents: 16207
diff changeset
563
e7ff32e7cf82 move global promptflag variable to octave_reader class
John W. Eaton <jwe@octave.org>
parents: 16207
diff changeset
564 int promptflag (int n) { return input_reader.promptflag (n); }
e7ff32e7cf82 move global promptflag variable to octave_reader class
John W. Eaton <jwe@octave.org>
parents: 16207
diff changeset
565
16195
b52d2f9294b6 use class for reading lexer input
John W. Eaton <jwe@octave.org>
parents: 16194
diff changeset
566 std::string input_source (void) const
b52d2f9294b6 use class for reading lexer input
John W. Eaton <jwe@octave.org>
parents: 16194
diff changeset
567 {
b52d2f9294b6 use class for reading lexer input
John W. Eaton <jwe@octave.org>
parents: 16194
diff changeset
568 return input_reader.input_source ();
b52d2f9294b6 use class for reading lexer input
John W. Eaton <jwe@octave.org>
parents: 16194
diff changeset
569 }
b52d2f9294b6 use class for reading lexer input
John W. Eaton <jwe@octave.org>
parents: 16194
diff changeset
570
16203
127cccb037bf move more global parser and lexer variables to classes
John W. Eaton <jwe@octave.org>
parents: 16199
diff changeset
571 bool input_from_terminal (void) const
127cccb037bf move more global parser and lexer variables to classes
John W. Eaton <jwe@octave.org>
parents: 16199
diff changeset
572 {
127cccb037bf move more global parser and lexer variables to classes
John W. Eaton <jwe@octave.org>
parents: 16199
diff changeset
573 return input_source () == "terminal";
127cccb037bf move more global parser and lexer variables to classes
John W. Eaton <jwe@octave.org>
parents: 16199
diff changeset
574 }
127cccb037bf move more global parser and lexer variables to classes
John W. Eaton <jwe@octave.org>
parents: 16199
diff changeset
575
127cccb037bf move more global parser and lexer variables to classes
John W. Eaton <jwe@octave.org>
parents: 16199
diff changeset
576 bool input_from_file (void) const
127cccb037bf move more global parser and lexer variables to classes
John W. Eaton <jwe@octave.org>
parents: 16199
diff changeset
577 {
127cccb037bf move more global parser and lexer variables to classes
John W. Eaton <jwe@octave.org>
parents: 16199
diff changeset
578 return input_source () == "file";
127cccb037bf move more global parser and lexer variables to classes
John W. Eaton <jwe@octave.org>
parents: 16199
diff changeset
579 }
127cccb037bf move more global parser and lexer variables to classes
John W. Eaton <jwe@octave.org>
parents: 16199
diff changeset
580
16195
b52d2f9294b6 use class for reading lexer input
John W. Eaton <jwe@octave.org>
parents: 16194
diff changeset
581 bool input_from_eval_string (void) const
b52d2f9294b6 use class for reading lexer input
John W. Eaton <jwe@octave.org>
parents: 16194
diff changeset
582 {
b52d2f9294b6 use class for reading lexer input
John W. Eaton <jwe@octave.org>
parents: 16194
diff changeset
583 return input_source () == "eval_string";
b52d2f9294b6 use class for reading lexer input
John W. Eaton <jwe@octave.org>
parents: 16194
diff changeset
584 }
b52d2f9294b6 use class for reading lexer input
John W. Eaton <jwe@octave.org>
parents: 16194
diff changeset
585
16224
4a848eb52de2 use stack for tracking lexer start states
John W. Eaton <jwe@octave.org>
parents: 16209
diff changeset
586 void push_start_state (int state);
4a848eb52de2 use stack for tracking lexer start states
John W. Eaton <jwe@octave.org>
parents: 16209
diff changeset
587
4a848eb52de2 use stack for tracking lexer start states
John W. Eaton <jwe@octave.org>
parents: 16209
diff changeset
588 void pop_start_state (void);
4a848eb52de2 use stack for tracking lexer start states
John W. Eaton <jwe@octave.org>
parents: 16209
diff changeset
589
4a848eb52de2 use stack for tracking lexer start states
John W. Eaton <jwe@octave.org>
parents: 16209
diff changeset
590 void clear_start_state (void);
4a848eb52de2 use stack for tracking lexer start states
John W. Eaton <jwe@octave.org>
parents: 16209
diff changeset
591
4a848eb52de2 use stack for tracking lexer start states
John W. Eaton <jwe@octave.org>
parents: 16209
diff changeset
592 int start_state (void) const { return start_state_stack.top (); }
4a848eb52de2 use stack for tracking lexer start states
John W. Eaton <jwe@octave.org>
parents: 16209
diff changeset
593
4a848eb52de2 use stack for tracking lexer start states
John W. Eaton <jwe@octave.org>
parents: 16209
diff changeset
594 void display_start_state (void) const;
4a848eb52de2 use stack for tracking lexer start states
John W. Eaton <jwe@octave.org>
parents: 16209
diff changeset
595
16275
dd7de0345124 10/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16267
diff changeset
596 int handle_op (const char *pattern, int tok, bool bos = false);
16229
7b7b1e4968e8 use functions instead of token return macros in lexer
John W. Eaton <jwe@octave.org>
parents: 16228
diff changeset
597
16275
dd7de0345124 10/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16267
diff changeset
598 int handle_incompatible_op (const char *pattern, int tok, bool bos = false);
16229
7b7b1e4968e8 use functions instead of token return macros in lexer
John W. Eaton <jwe@octave.org>
parents: 16228
diff changeset
599
16263
9acb86e6ac90 4/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16259
diff changeset
600 bool maybe_unput_comma_before_unary_op (int tok);
9acb86e6ac90 4/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16259
diff changeset
601
16275
dd7de0345124 10/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16267
diff changeset
602 int handle_unary_op (int tok, bool bos = false);
16263
9acb86e6ac90 4/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16259
diff changeset
603
16275
dd7de0345124 10/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16267
diff changeset
604 int handle_incompatible_unary_op (int tok, bool bos = false);
16263
9acb86e6ac90 4/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16259
diff changeset
605
16257
db7f07b22b9b 1/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16255
diff changeset
606 int handle_assign_op (const char *pattern, int tok);
db7f07b22b9b 1/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16255
diff changeset
607
db7f07b22b9b 1/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16255
diff changeset
608 int handle_incompatible_assign_op (const char *pattern, int tok);
db7f07b22b9b 1/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16255
diff changeset
609
16275
dd7de0345124 10/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16267
diff changeset
610 int handle_op_internal (int tok, bool bos, bool compat);
16229
7b7b1e4968e8 use functions instead of token return macros in lexer
John W. Eaton <jwe@octave.org>
parents: 16228
diff changeset
611
16231
2b15ae55c721 put all tokens in the token cache
John W. Eaton <jwe@octave.org>
parents: 16230
diff changeset
612 int handle_token (const std::string& name, int tok);
16229
7b7b1e4968e8 use functions instead of token return macros in lexer
John W. Eaton <jwe@octave.org>
parents: 16228
diff changeset
613
16231
2b15ae55c721 put all tokens in the token cache
John W. Eaton <jwe@octave.org>
parents: 16230
diff changeset
614 int handle_token (int tok, token *tok_val = 0);
16229
7b7b1e4968e8 use functions instead of token return macros in lexer
John W. Eaton <jwe@octave.org>
parents: 16228
diff changeset
615
7b7b1e4968e8 use functions instead of token return macros in lexer
John W. Eaton <jwe@octave.org>
parents: 16228
diff changeset
616 int count_token (int tok);
7b7b1e4968e8 use functions instead of token return macros in lexer
John W. Eaton <jwe@octave.org>
parents: 16228
diff changeset
617
16257
db7f07b22b9b 1/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16255
diff changeset
618 int count_token_internal (int tok);
db7f07b22b9b 1/10 commits reworking the lexer
John W. Eaton <jwe@octave.org>
parents: 16255
diff changeset
619
16229
7b7b1e4968e8 use functions instead of token return macros in lexer
John W. Eaton <jwe@octave.org>
parents: 16228
diff changeset
620 int show_token (int tok);
7b7b1e4968e8 use functions instead of token return macros in lexer
John W. Eaton <jwe@octave.org>
parents: 16228
diff changeset
621
16126
5c885c13bfa3 localize use of yylval.tok_val in lexical_feedback member functions
John W. Eaton <jwe@octave.org>
parents: 16125
diff changeset
622 // For unwind protect.
16158
7eb614760ddb rename lexical_feedback to octave_lexer
John W. Eaton <jwe@octave.org>
parents: 16149
diff changeset
623 static void cleanup (octave_lexer *lexer) { delete lexer; }
16126
5c885c13bfa3 localize use of yylval.tok_val in lexical_feedback member functions
John W. Eaton <jwe@octave.org>
parents: 16125
diff changeset
624
5c885c13bfa3 localize use of yylval.tok_val in lexical_feedback member functions
John W. Eaton <jwe@octave.org>
parents: 16125
diff changeset
625 private:
5c885c13bfa3 localize use of yylval.tok_val in lexical_feedback member functions
John W. Eaton <jwe@octave.org>
parents: 16125
diff changeset
626
16224
4a848eb52de2 use stack for tracking lexer start states
John W. Eaton <jwe@octave.org>
parents: 16209
diff changeset
627 std::stack<int> start_state_stack;
4a848eb52de2 use stack for tracking lexer start states
John W. Eaton <jwe@octave.org>
parents: 16209
diff changeset
628
16110
7302f8a4df83 use pointer for global lexical_feedback structure
John W. Eaton <jwe@octave.org>
parents: 16107
diff changeset
629 // No copying!
7302f8a4df83 use pointer for global lexical_feedback structure
John W. Eaton <jwe@octave.org>
parents: 16107
diff changeset
630
16158
7eb614760ddb rename lexical_feedback to octave_lexer
John W. Eaton <jwe@octave.org>
parents: 16149
diff changeset
631 octave_lexer (const octave_lexer&);
16110
7302f8a4df83 use pointer for global lexical_feedback structure
John W. Eaton <jwe@octave.org>
parents: 16107
diff changeset
632
16158
7eb614760ddb rename lexical_feedback to octave_lexer
John W. Eaton <jwe@octave.org>
parents: 16149
diff changeset
633 octave_lexer& operator = (const octave_lexer&);
1826
b14829582cc4 [project @ 1996-02-02 03:05:07 by jwe]
jwe
parents: 1315
diff changeset
634 };
b14829582cc4 [project @ 1996-02-02 03:05:07 by jwe]
jwe
parents: 1315
diff changeset
635
1
78fd87e624cb [project @ 1993-08-08 01:13:40 by jwe]
jwe
parents:
diff changeset
636 #endif