Implements: ITokenizer

Description

The AbstractTokenizer class defines a general tokenizer.

Fields

lastTokenType

Last token type

lastTokenType: TokenType = TokenType.Unknown

nextTokenValue

Next token

nextTokenValue: Token?

_scanner

Scanner

_scanner: IScanner?

commentState

Comment state

@override

commentState: ICommentState?

decodeStrings

bool that defines the option to decode strings or not.

@override

decodeStrings: bool

mergeWhitespaces

bool that defines the option to unify white spaces.

@override

mergeWhitespaces: bool

numberState

Number state

@override

numberState: INumberState?

quoteState

Quote state

@override

quoteState: IQuoteState?

skipComments

bool that defines the option to skip comments. @override

skipComments: bool

skipEof

bool that defines the option to skip EOF. @override

skipEof: bool

skipUnknown

bool that defines the option to skip unknowns. @override

skipUnknown: bool

skipWhitespaces

bool that defines the option to skip white spaces. @override

skipWhitespaces: bool

symbolState

Symbol state @override

symbolState: ISymbolState?

unifyNumbers

bool that defines the option to unify numbers. @override

unifyNumbers: bool

whitespaceState

White space state. @override

whitespaceState: IWhitespaceState?

wordState

Word state. @override

wordState: IWordState?

Properties

scanner

Scanner

@override

IScanner? get scanner

• returns: IScanner? - scanner

@override

set scanner(IScanner? value)

• value: IScanner? - scanner

Instance methods

clearCharacterStates

Clears all character states.

void clearCharacterStates()

getCharacterState

Gest the state for a given character.

ITokenizerState? getCharacterState(int symbol)

• symbol: int - symbol
• returns: ITokenizerState? - tokenizer state

hasNextToken

Finds out if the tokenizer has a next token.

@override

bool hasNextToken()

• returns: bool - true if it has a next token, false otherwise.

nextToken

Gets the next token.

@override

Token? nextToken()

• returns: Token? - next token

readNextToken

Reads the next token.

@override

Token? readNextToken()

• returns: Token? - next token

setCharacterState

Sets the characters' state.

void setCharacterState(int fromSymbol, int toSymbol, ITokenizerState state)

• fromSymbol: int - first symbol
• toSymbol: int - last symbol
• state: ITokenizerState - tokenizer state

tokenizeBuffer

Provides a token for a string buffer.

@override

List<Token> tokenizeBuffer(String buffer)

• buffer: String - buffer
• returns: List<Token> - token

tokenizeBufferToStrings

Creates a list of token values. @override

List<String?> tokenizeBufferToStrings(String buffer)

• buffer: String - buffer
• returns: List<String?> - list of token values

tokenizeStream

Creates a list of tokens @override

List<Token> tokenizeStream(IScanner scanner)

• scanner: IScanner - scanner
• returns: List<Token> - list of tokens

tokenizeStreamToStrings

Creates a list of token values.

List<String?> tokenizeStreamToStrings(IScanner scanner)

• scanner: IScanner - scanner
• returns: List<String?> - list of token values