Tokenizer Class
Assembly: Topten.Cminor.Compiler.dll
Namespace: Topten.Cminor.Lexical
Source code tokenzier
public class TokenizerInheritance
Tokenizer → object
Constructors
| Tokenizer(CodeFile) | Constructor |
Properties
| ParsingType | When set to true to the bitwise shift operators '<<' and '>>' tokens are instead returned multiple Token.LT and Token.GT tokens. Used when parsing generic type declarations |
| Source | Gets the source code being tokenized |
| Token | The current token |
| TokenIdentifier | For Token.Identifier, the identifier name |
| TokenLiteral | For Token.Literal, the literal value |
| TokenPosition | The position of the current token |
| TokenRaw | The raw text associated with the token |
Methods
| CheckToken(Token, string) | Checks for a specific token and throws a CodeException if not |
| DescribeToken(Token, string) | Provdes a description of the current token |
| IsIdentifier(string) | Checks if the current token is a particular identifier |
| Next() | Moves to the next token |
| NextInterpolatedStringPart() | Gets the next part of a interpolated string |
| Rewind(CodePosition) | Rewinds the tokenizer to a previously saved position |
| SkipIdentifier() | Returns the current identifier, throwing an CodeException if the current token is not an identifier |
| SkipIdentifier(string) | Skips a specified identifier, throwing a CodeException if the current token is not an identifier or doesn't match the specified identifier name |
| SkipInterpolatedStringFormat() | Should be called after parsing interpolated string expression,alignment to read the format part of the string |
| SkipToken(Token) | Checks for a specific token and if so skips it, otherwise throws a Codeexception |
| TrySkipIdentifier(string) | Checks if the current token is a particular identifier and if so skip it |
| TrySkipToken(Token) | Checks if the current token matches and if so skips it |
| Unexpected() | Creates a syntax error CodeException for the current token |
| Unexpected(string) | Creates a syntax error CodeException for the current token, along with a message describing what was expected |