Generic

The sub-package provides classes that define different tokenizer states and symbol nodes.

This state will either delegate to a comment-handling state, or return a token with just a slash in it.

This state will either delegate to a comment-handling state, or return a token with just a slash in it.

A CommentState object returns a comment from a scanner.

A NumberState object returns a number from a scanner.

A quoteState returns a quoted string token from a scanner.

The GenericSymbolState class allows you to add multi-character symbols and obtain a symbol token from a scanner.

Implements a default tokenizer class.

Creates a whitespace state.

A WordState returns a word from a scanner.

Constructs a SymbolNode.

This class is a special case of a SymbolNode. A SymbolRootNode object has no symbol of its own, but has children that represent all possible symbols.