sys::Obj compiler::CompilerSupport compiler::Tokenizer
Tokenizer inputs a Str and output a list of Tokens
Return a CompilerException for current location in source.
Parse an escapse sequence which starts with a \
Process hex int/long literal starting with 0x
Construct with characters of source file. The buffer passed must be normalized in that all newlines must be represented strictly as \n and not \r or \r\n (see File.readAllStr). If isDoc is false, we skip all star-star Fandoc comments.
Return the next token in the buffer.
Tokenize the entire input into a list of tokens.