Implements: IWhitespaceState

Description

The GenericWhitespaceState allows you to create a whitespace state.

Important points

• A whitespace state ignores whitespace (such as blanks and tabs), and returns the tokenizer’s next token. By default, all characters from 0 to 32 are whitespace.

Constructors

Constructs a whitespace state with a default idea of what characters are, in fact, whitespace.

GenericWhitespaceState()

Instance methods

clearWhitespaceChars

Clears definitions of whitespace characters.

@override

void clearWhitespaceChars()

nextToken

Ignores whitespace (such as blanks and tabs), and returns the tokenizer’s next token.

@override

Token? nextToken(IScanner scanner, ITokenizer? tokenizer)

• scanner: IScanner - text string to be tokenized.
• tokenizer: ITokenizer? - tokenizer class that controls the process.
• returns: Token? - next token from the top of the stream.

setWhitespaceChars

Establishes the given characters as whitespace to ignore.

@override

void setWhitespaceChars(int fromSymbol, int toSymbol, bool enable)

• fromSymbol: int - first character index of the interval.
• toSymbol: int - last character index of the interval.
• enable: bool - true if this state should ignore characters in the given range.