Implements a word state object.
The ExpressionTokenizer class allows you to implement a word state object.
Constructs an instance of this class.
Gets the next token from the stream started from the character linked to this state.
- scanner: IScanner - textual string to be tokenized.
- tokenizer: ITokenizer - tokenizer class that controls the process.
- returns: Token - next token from the top of the stream.