Implements a symbol state to tokenize delimiters in CSV streams.
Implements: GenericSymbolState
Description
The CsvSymbolState implements a symbol state to tokenize delimiters in CSV streams.
Constructors
Constructs this object with specified parameters.
CsvSymbolState()
Instance methods
next_token
Gets the next token from the stream started from the character linked to this state.
next_token(scanner: IScanner, tokenizer: ITokenizer): Token
- scanner: IScanner - text string to be tokenized.
- tokenizer: ITokenizer - tokenizer class that controls the process.
- returns: Token - next token from the top of the stream.