ExpressionWordState

Implements a word state object.

Implements: GenericWordState

Description

The ExpressionTokenizer class allows you to implement a word state object.

Constructors

Constructs an instance of this class.

ExpressionWordState()

Fields

Supported expression keywords.

keywords: List[str] = [ “AND”, “OR”, “NOT”, “XOR”, “LIKE”, “IS”, “IN”, “NULL”, “TRUE”, “FALSE” ]

next_token

Gets the next token from the stream started from the character linked to this state.

next_token(scanner: IScanner, tokenizer: ITokenizer): Token

  • scanner: IScanner - textual string to be tokenized.
  • tokenizer: ITokenizer - tokenizer class that controls the process.
  • returns: Token - next token from the top of the stream.