Defines an interface for a tokenizer state that processes delimiters.
Implements: ITokenizerState
Description
The ISymbolState interface is used for tokenizer states that process delimiters.
Instance methods
add
Add a multi-character symbol.
add(value: str, token_type: TokenType)
- value: str - symbol to add, such as "=:=".
- token_type: TokenType - token type.