TokenType

Types (categories) of tokens such as “number”, “symbol” and “word”.

Description

The TokenType class defines different types (categories) of tokens, such as “number”, “symbol” and “word”.

Enumeration members

The codes are as follows:

  • Unknown = iota
  • Eof
  • Eol
  • Float
  • Integer
  • HexDecimal
  • Number
  • Symbol
  • Quoted
  • Word
  • Keyword
  • Whitespace
  • Comment
  • Special