Tokenizers

This sub-package contains classes used to create expression specific state objects and tokenizers for lexical analysis.

Classes

ExpressionNumberState

Implements an expression-specific number state object.

ExpressionQuoteState

Implements an expression-specific quote string state object

ExpressionSymbolState

Implements a symbol state object.

ExpressionTokenizer

Implements a tokenizer to perform lexical analysis for expressions.

ExpressionWordState

Implements a symbol state object.