IntConst, int marker. For example, if a program's source file contains string: "", the lexer will treat it as token Number, meaning it is a number with value of Until I do that, you have one of two options: Figure out on your own how to adapt these posts to a bit instruction set.
That's the value of a lexer: to simplify the parser by converting the stream of source code into token stream. It turns out that scanners, especially for non-ambiguously defined languages, are fairly easy to write.
Ident], matching the previously shown regular expression. A literal string constant [TokenType.
That will explain the existance of while: to skip unknown characters in the source code.