Tokenizers

This package provides different sub-packages containing classes used to define and manage tokenizers.

Packages

Generic

The sub-package provides classes that define different tokenizer states and symbol nodes.

Utilities

This sub-package contains utility classes used to create character intervals, keep references associated with specific characters and char validators.

Constants

TokenType

Types (categories) of tokens such as “number”, “symbol” and “word”.

Interfaces

ICommentState

Defines an interface for tokenizer state that processes comments.

INumberState

Defines an interface for tokenizer state that processes numbers - Integers, Floats, HexDec.

IQuoteState

Defines an interface for tokenizer state that processes quoted strings.

ISymbolState

Defines an interface for tokenizer state that processes delimiters.

ITokenizer

Interface for tokenizers.

ITokenizerState

Interface for TokenizerStates.

IWhitespaceState

Defines an interface for tokenizer state that processes whitespaces (' ‘, ‘\t’).

IWordState

Defines an interface for tokenizer state that processes words, identificators or keywords.