Implements a default tokenizer class.
Implements: AbstractTokenizer
Description
The GenericTokenizer allows you to implement a default tokenizer class.
Constructors
NewGenericTokenizer
Creates an instance of GenericTokenizer
NewGenericTokenizer() *GenericTokenizer