CppCommentState

This state will either delegate to a comment-handling state, or return a token with just a slash in it.

Implements: GenericCommentState

Description

The CppCommentState class allows you to create a state that will either delegate to a comment-handling state, or return a token with just a slash in it.

Fields

STAR

Represents a star (*) char

STAR: int = ord('*')

SLASH

Represents a forward slash (/) char.

SLASH: int = ord('/')

Instance methods

get_multi_line_comment

Ignores everything up to a closing star and slash, and then returns the tokenizer’s next token.

get_multi_line_comment(scanner: IScanner): str

  • scanner: IScanner - scanner
  • returns: str - comment

get_single_line_comment

Ignores everything up to an end-of-line, and then returns the tokenizer’s next token.

get_single_line_comment(scanner: IScanner): str

  • scanner: IScanner - scanner
  • returns: str - comment

next_token

Either delegates to a comment-handling state, or returns a token with just a slash in it.

next_token(scanner: IScanner, tokenizer: ITokenizer): Token

  • scanner: IScanner - text string to be tokenized.
  • tokenizer: ITokenizer - tokenizer class that controls the process.
  • returns: Token - next token from the top of the stream.