ITokenizer

Interface for tokenizers.

Description

The ITokenizer interface defines the main elements of a tokenizer.

Important points

  • A tokenizer divides a string into tokens. This class is highly customizable with regard to exactly how this division occurs, but it also has defaults that are suitable for many languages. This class assumes that the character values read from the string lie in the range 0-255. For example, the Unicode value of a capital A is 65, so String.fromCharCode(65) prints out a capital A.

  • The behavior of a tokenizer depends on its character state table. This table is an array of 256 TokenizerState states. The state table decides which state to enter upon reading a character from the input string.

  • For example, by default, upon reading an ‘A’, a tokenizer will enter a “word” state. This means the tokenizer will ask a WordState object to consume the ‘A’, along with the characters after the ‘A’ that form a word. The state’s responsibility is to consume characters and return a complete token.

  • The default table sets a SymbolState for every character from 0 to 255, and then overrides this with:

From    To        State
 0     ' '   whitespaceState 
'a'    'z'      wordState
'A'    'Z'      wordState
160     255     wordState
'0'    '9'     numberState
'-'    '-'     numberState
'.'    '.'     numberState
'"'    '"'      quoteState
'\''   '\''     quoteState
'/'    '/'      slashState
  • In addition to allowing modification of the state table, this class makes each of the states above available. Some of these states are customizable. For example, wordState allows customization of what characters can be part of a word, after the first character.

Properties

CommentState

Token state to process comments.

CommentState() ICommentState

DecodeStrings

Gets the decode quoted strings flag.

DecodeStrings() bool

  • returns: bool - decodes quoted strings flag.

SetDecodeStrings

Sets the decode quoted strings flag.

SetDecodeStrings(value bool)

  • value: bool - decodes quoted strings flag.

MergeWhitespaces

Gets the merge whitespaces flag.

MergeWhitespaces() bool

  • returns: bool - whitespaces flag.

SetMergeWhitespaces

Sets the merge whitespaces flag.

SetMergeWhitespaces(value bool)

  • value: bool - whitespaces flag.

NumberState

Gets a token state to process numbers.

NumberState() INumberState

QuoteState

Gets a token state to process quoted strings.

QuoteState() IQuoteState

  • returns: IQuoteState - token state to process quoted strings.

Reader

Gets the stream scanner to tokenize.

Reader() IScanner

SetReader

Sets the stream scanner to tokenize.

SetReader(scanner IScanner)

SkipComments

Gets the skip comments flag.

SkipComments() bool

  • returns: bool - skip comments flag.

SetSkipComments

Sets the skip comments flag.

SetSkipComments(value bool)

  • value: bool - skip comments flag.

SkipEof

Gets the skip End-Of-File token at the end of stream flag.

SkipEof() bool

  • returns: bool - stream flag.

SetSkipEof

Sets the skip End-Of-File token at the end of stream flag.

SetSkipEof(value bool)

  • value: bool - stream flag.

SkipUnknown

Gets the skip unknown characters flag.

SkipUnknown() bool

  • returns: bool - skip unknown characters flag.

SetSkipUnknown

Sets the skip unknown characters flag.

SetSkipUnknown(value bool)

  • value: bool - skip unknown characters flag.

SkipWhitespaces

Gets the skip whitespaces flag.

SkipWhitespaces() bool

  • returns: bool - skip whitespaces flag.

SetSkipWhitespaces

Sets the skip whitespaces flag.

SetSkipWhitespaces(value bool)

  • value: bool - skip whitespaces flag.

SymbolState

Gets a token state to process symbols (single like “=” or muti-character like “<>")

SymbolState() ISymbolState

UnifyNumbers

Gets unify numbers: “Integers” and “Floats” makes just “Numbers”.

UnifyNumbers() bool

  • returns: bool - unifies numbers.

SetUnifyNumbers

Sets unify numbers: “Integers” and “Floats” makes just “Numbers” flag

SetUnifyNumbers(value bool)

  • value: bool - unifies numbers.

WhitespaceState

Gets a token state to process white space delimiters.

WhitespaceState() IWhitespaceState

WordState

Gets a token state to process words or indentificators.

WordState() IWordState

  • returns: IWordState - a token state to process words.

Methods

HasNextToken

Checks if there is a next token.

HasNextToken() bool

  • returns: bool - true if scanner has a next token.

NextToken

Gets the next token from the scanner.

NextToken() *Token

  • returns: *Token - next token or nil if there are no more tokens left.

TokenizeBuffer

Tokenizes a string buffer into a list of tokens structures.

TokenizeBuffer(buffer string) []*Token

  • buffer: string - string buffer to be tokenized.
  • returns: []*Token - list of token structures.

TokenizeBufferToStrings

Tokenizes a string buffer into a list of strings.

TokenizeBufferToStrings(buffer string) []string

  • buffer: string - string buffer to be tokenized.
  • returns: []string - list of token strings.

TokenizeStream

Tokenizes a text stream into a list of token structures.

TokenizeStream(scanner IScanner) []*Token

  • scanner: IScanner - text stream to be tokenized.
  • returns: []*Token - list of token structures.

TokenizeStreamToStrings

Tokenizes a text stream into a list of strings.

TokenizeStreamToStrings(scanner IScanner) []string

  • scanner: IScanner - text stream to be tokenized.
  • returns: []string - list of token strings.