Implements a word state object.
Extends: GenericWordState
Description
The ExpressionTokenizer class allows you to implement a word state object.
Constructors
Constructs an instance of this class.
public
constructor()
Fields
nextToken
Gets the next token from the stream started from the character linked to this state.
public
nextToken(scanner: IScanner, tokenizer: ITokenizer): Token
- scanner: IScanner - textual string to be tokenized.
- tokenizer: ITokenizer - tokenizer class that controls the process.
- returns: Token - next token from the top of the stream.