Inherits: ITokenizer

Description

The AbstractTokenizer class defines a general tokenizer.

Fields

_lastTokenType

Last token type

protected _lastTokenType: TokenType = TokenType.Unknown

Properties

NextToken

Next token

public Token NextToken()

Scanner

Scanner

public IScanner Scanner { get; set; }

CommentState

Comment state

public ICommentState CommentState { get; set; }

DecodeStrings

Boolean that defines the option to decode strings or not.

public bool DecodeStrings { get; set; }

MergeWhitespaces

Boolean that defines the option to unify white spaces.

public bool MergeWhitespaces { get; set; }

NumberState

Number state

public INumberState NumberState { get; set; }

QuoteState

Quote state

public IQuoteState QuoteState { get; set; }

SkipComments

Boolean that defines the option to skip comments.

public bool SkipComments { get; set; }

SkipEof

Boolean that defines the option to skip EOF.

public bool SkipEof { get; set; }

SkipUnknown

Boolean that defines the option to skip unknowns.

public bool SkipUnknown { get; set; }

SkipWhitespaces

Boolean that defines the option to skip white spaces.

public bool SkipWhitespaces { get; set; }

SymbolState

Symbol state

public ISymbolState SymbolState { get; set; }

UnifyNumbers

Boolean that defines the option to unify numbers.

public bool UnifyNumbers { get; set; }

WhitespaceState

White space state.

public IWhitespaceState WhitespaceState { get; set; }

WordState

Word state.

public IWordState WordState { get; set; }

Instance methods

ClearCharacterStates

Clears all character states.

public void ClearCharacterStates()

GetCharacterState

Gest the state for a given character.

public ITokenizerState GetCharacterState(char symbol)

• symbol: char - symbol
• returns: ITokenizerState - tokenizer state

HasNextToken

Finds out if the tokenizer has a next token.

public bool HasNextToken()

• returns: bool - true if it has a next token, false otherwise.

NextToken

Gets the next token.

public Token NextToken()

• returns: Token - next token

ReadNextToken

Reads the next token.

protected virtual Token ReadNextToken()

• returns: Token - next token

SetCharacterState

Sets the characters' state.

public void SetCharacterState(char fromSymbol, char toSymbol, ITokenizerState state)

• fromSymbol: char - first symbol
• toSymbol: char - last symbol
• state: ITokenizerState - tokenizer state

TokenizeBuffer

Provides a token for a string buffer.

public IList<Token> TokenizeBuffer(string buffer)

• buffer: string - buffer
• returns: Token - token

TokenizeBufferToStrings

Creates a list of token values.

public IList<string> TokenizeBufferToStrings(string buffer)

• buffer: string - buffer
• returns: IList<string> - list of token values

TokenizeStream

Creates a list of tokens

public IList<Token> TokenizeStream(IScanner scanner)

• scanner: IScanner - scanner
• returns: Token - list of tokens

TokenizeStreamToStrings

Creates a list of token values.

public IList<string> TokenizeStreamToStrings(IScanner scanner)

• scanner: IScanner - scanner
• returns: IList<string> - list of token values