Next: Super-Mutants, Previous: Style Features, Up: Components [Contents][Index]
The Clang C Tokenizer component offers the ability to construct a sequence of tokens that represent a software object.
The sel/cp/clang-tokens:clang-tokens method takes a sel:software object and an
optional list of roots indicating the elements that should be
tokenized. It returns a list of symbols representing all the tokens
under each of the roots.
The Clang C tokenizer uses the following conventions:
| Token Kind | Represented By |
|---|---|
| identifier | interned string “identifier”. |
| special character (operator, parenthesis, brace, etc) | the interned string of the special character, e.g., (intern "+"). |
| primitive type | the interned string for that type, e.g., (intern "int"). |
| keyword | the interned string for that keyword, e.g., (intern "if"). |
| macros | the interned string “macro” (macros are generally not expanded in the AST). |
| literals | character, float, imaginary, integer, and string literals are represented by the following interned strings, respectively: “char-literal”, “float-literal”, “i-literal”, “int-literal”, “string-literal”. |