Generic

The sub-package provides classes that define different tokenizer states and symbol nodes.

Classes

CCommentState

This state will either delegate to a comment-handling state, or return a token with just a slash in it.

CppCommentState

This state will either delegate to a comment-handling state, or return a token with just a slash in it.

GenericCommentState

A CommentState object returns a comment from a scanner.

GenericNumberState

A NumberState object returns a number from a scanner.

GenericQuoteState

A quoteState returns a quoted string token from a scanner.

GenericSymbolState

The GenericSymbolState class allows you to add multi-character symbols and obtain a symbol token from a scanner.

GenericTokenizer

Implements a default tokenizer class.

GenericWhitespaceState

Creates a whitespace state.

GenericWordState

A WordState returns a word from a scanner.

SymbolNode

Constructs a SymbolNode.

SymbolRootNode

This class is a special case of a SymbolNode. A SymbolRootNode object has no symbol of its own, but has children that represent all possible symbols.