Recognition of Tokens

• We would like to recognize the following input

  stmt → if expr then stmt
       |  if expr then stmt else stmt
       | ε
  expr → term relop term
       | term
  term → id
       | number
      

• relop =, <>, <, >, >=, <=
• The terminals are if, then, else, realop, id, number
• The patterns are:

  digit	→	[0-9]
  digits	→	digit+
  number	→	digits (. digits)? (E [+-]? digits)?
  letter	→	[A-Za-z]
  id	→	letter (letter|digit)*
  if	→	if
  then	→	then
  else	→	else
  relop	→	=|<>|<|>|>=|<=
  ws	→	(blank | tab | newline)+

• The pseudo-token ws will allow us to recognize whitespace, but will not be returned to the parser.
• Build a table to show the lexemes, token names and attribute values. (3.12)
• Transition Diagrams
  • This seems to be a pre-automata step.
  • We have a number of states, represented by circles
  • A transition function, represented by edges
  • A start state
  • A set of accepting or final states.
  • These will be used to represent regular expressions
    • It summarizes the information between lexemeBegin and forward
    • They will be done by hand, for now.
    • As we encounter a symbol, we will move along an edge.
    • If it fails, we return a fail, and probably reset forward
    • If we hit an accepting state, we return success, and perhaps perform an action
    • consider relop → =|<>|<|>|>=|<=
      • rewrite as relop → = | < [>|=]? | >=?
      • 
      • Trace each of the valid operators
  • Reserved Words and Identifiers
    • id→ letter (letter|digit)^*
    • 
    • then → then
    • 
    • At least two choices here:
      • Preload the symbol table with all of the reserved words
        • installID() installs an identifier into the symbol table as an ID, if it is not there
        • installID() also returns a symbol table pointer
        • getToken() returns the token type for a given string.
      • Build a transition diagram for each keyword
        • These must be run first, before the ID transition diagram.
    • Build a transition diagram for number → digits (. digits)? (E [+-]? digits)?
    • We will keep a different numeric constants.
  • Architecture of a Transition-Diagram Based lexical Analyzer
    • We could build a mondo-transition diagram based on all of the above
      • This seems bad to me.
    • We could run all transition diagrams in parallel,
      • We need to select the output that produces the longest lexeme.
    • We could :
      • Make a state variable.
      • Build a series of functions (getRealop() is described on page 135)
        • getId()
        • getNumber()
        • getWS()
      • They either return a TOKEN pair or a FAIL
      • In the case of a fail, use a function to determine what to do next
        • If state < 8, reset the forward pointer, set state to be 9
        • If 21 <state <28, we must be at EOF, quit scanning.