Lex/Flex

• This is by no means a complete treatment of flex.
• There are MANY references on line
  • http://flex.sourceforge.net/
• But it is sufficient as a boot-strap for what we want to do.
• Lex built in 75, was proprietary (Unix)
• flex (1987), open source, and clean (ie developed without looking at the lex code)
• F/Lex
  • Reads the inpfile, (.l)
    • This is a combination of C/C++ and the lex language.
  • It generates a number of translation diagrams,
  • Which it then translates into C/C++ code.
  • Which can then be compiled.
• There are versions in other languages (java, python, ...)
• And while it is primarliarly for *nix environments, it can work elsewhere too.
• Make knows about .l files
• The structure of a Lex Program

  • declarations
    %%
    translation rules
    %% 
    auxiliary functions
    	

  • Declarations
    • C/C++ code which are copied directly into the output file.
      • This must either be indented
      • or be between %{ and }%
      • But %{ must begin a line with no other values
      • And }% must be the same.
    • This section also contains a set of definitions (d_i in a RE)
      • Begin with a name,
        • letter|underscore (letter|digit|_|-)^*
      • A space⁺
      • Then a regexp, with {d_j}, 1≤j<i
  • Transition Rules
    • pattern {action}
    • Patterns follow regexp rules.
    • actions are C/C++ code
      • if the action is empty, then nothing is done and the state machine continues.
        • A white space rule
        • Or perhaps a comment eating segment.
      • The program will then attempt to match a pattern
        • Prefering the longest
        • And prefering the first defined (then vs id in the example)
        • The matching lexeme is stored in yytext
        • The corresponding action is executed.
  • User Code Section
    • This is copied verbatim to the output file.
    • It is for routines needed by the the action rules
• RE's in flex
  • . matches any single character except \n
    • Frequently in the last line of set of rules to eat left over stuff.
  • [] to define a character class
    • delim [ \t\m]
    • a [aA] so you don't have to worry about case
    • digit [0-9]
    • letter [a-zA-Z]
  • ^ beginning of a line
  • $ end of a line
  • *,+,? | as expected
• Heading section
  • Look at scan.l from the book's code.
  • Note, it also has a main.
  • Look at that Makefile to see how it is compiled.
• Look at wombat scan.l for rules section
  • use of auxilary routines within lex
  • in the book code there is use of the symbolTable
  • Note return value here is passed back to calling routine.
  • See extern int yyparse(); in was.C
  • Traditional yylval is a union, or other for the second part of the token
  • See ch3* for returning the book style tokens
  • And for use of a symbol table.