Implements: ITokenizer

Description

The AbstractTokenizer class defines a general tokenizer.

Fields

_lastTokenType

Last token type

protected _lastTokenType: TokenType = TokenType.Unknown

_nextToken

Next token

protected _nextToken: Token

_scanner

Scanner

protected _scanner: IScanner

commentState

Comment state

public commentState: ICommentState

decodeStrings

Boolean that defines the option to decode strings or not.

public decodeStrings: boolean

mergeWhitespaces

Boolean that defines the option to unify white spaces.

public mergeWhitespaces: boolean

numberState

Number state

public numberState: INumberState

quoteState

Quote state

public quoteState: IQuoteState

skipComments

Boolean that defines the option to skip comments.

public skipComments: boolean

skipEof

Boolean that defines the option to skip EOF.

public skipEof: boolean

skipUnknown

Boolean that defines the option to skip unknowns.

public skipUnknown: boolean

skipWhitespaces

Boolean that defines the option to skip white spaces.

public skipWhitespaces: boolean

symbolState

Symbol state

public symbolState: ISymbolState

unifyNumbers

Boolean that defines the option to unify numbers.

public unifyNumbers: boolean

whitespaceState

White space state.

public whitespaceState: IWhitespaceState

wordState

Word state.

public wordState: IWordState

Properties

scanner

Scanner

public scanner(): IScanner

• returns: IScanner - scanner

public scanner(value: IScanner)

• value: IScanner - scanner

Instance methods

clearCharacterStates

Clears all character states.

public clearCharacterStates(): void

getCharacterState

Gest the state for a given character.

public getCharacterState(symbol: number): ITokenizerState

• symbol: number - symbol
• returns: ITokenizerState - tokenizer state

hasNextToken

Finds out if the tokenizer has a next token.

public hasNextToken(): boolean

• returns: boolean - true if it has a next token, false otherwise.

nextToken

Gets the next token.

public nextToken(): Token

• returns: Token - next token

readNextToken

Reads the next token.

protected readNextToken(): Token

• returns: Token - next token

setCharacterState

Sets the characters' state.

public setCharacterState(fromSymbol: number, toSymbol: number, state: ITokenizerState): void

• fromSymbol: number - first symbol
• toSymbol: number - last symbol
• state: ITokenizerState - tokenizer state

tokenizeBuffer

Provides a token for a string buffer.

public tokenizeBuffer(buffer: string): Token[]

• buffer: string - buffer
• returns: Token[] - token

tokenizeBufferToStrings

Creates a list of token values.

public tokenizeBufferToStrings(buffer: string): string[]

• buffer: string - buffer
• returns: string[] - list of token values

tokenizeStream

Creates a list of tokens

public tokenizeStream(scanner: IScanner): Token[]

• scanner: IScanner - scanner
• returns: Token[] - list of tokens

tokenizeStreamToStrings

Creates a list of token values.

public tokenizeStreamToStrings(scanner: IScanner): string[]

• scanner: IScanner - scanner
• returns: string[] - list of token values