Lexer component of the preprocessor.
This is similar to the parser's lexer, but for the preprocessor.