GenericWhitespaceState

Implements: IWhitespaceState

Description

The GenericWhitespaceState allows you to create a whitespace state.

Important points

• A whitespace state ignores whitespace (such as blanks and tabs), and returns the tokenizer’s next token. By default, all characters from 0 to 32 are whitespace.

Constructors

Constructs a whitespace state with a default idea of what characters are, in fact, whitespace.

public GenericWhitespaceState() throws Exception

Instance methods

clearWhitespaceChars

Clears definitions of whitespace characters.

public void clearWhitespaceChars()

nextToken

Ignores whitespace (such as blanks and tabs), and returns the tokenizer’s next token.

public Token nextToken(scanner: IScanner, tokenizer: ITokenizer)

• scanner: IScanner - text string to be tokenized.
• tokenizer: ITokenizer - tokenizer class that controls the process.
• returns: Token - next token from the top of the stream.

setWhitespaceChars

Establishes the given characters as whitespace to ignore.

public void setWhitespaceChars(int fromSymbol, int toSymbol, boolean enable) throws Exception

• fromSymbol: int - first character index of the interval.
• toSymbol: int - last character index of the interval.
• enable: boolean - true if this state should ignore characters in the given range.