2.1.1 Source Code with tree-sitter

The sel/sw/ts:tree-sitter software object class is the primary base class for representing software source code which has been parsed into structured abstract syntax trees (ASTs) or more accurately Concrete Syntax Trees (CSTs). This class uses GitHub’s tree-sitter (see https://tree-sitter.github.io/tree-sitter/) libraries to parse source code into ASTs.

tree-sitter allows for many different languages to be represented uniformly through libtree-sitter and the analysis of its corresponding language modules. This cuts down on maintenance time across different languages and enables immediate inclusion of any language that has a language module for libtree-sitter.

Tree sitter has support for many source code languages. See Available Parsers for full list. The tree-sitter classes are generated at compile time. This is done by analyzing the node-types.json and grammar.json files that are located in tree-sitter language modules, of which the format can be found here: https://tree-sitter.github.io/tree-sitter/using-parsers#static-node-types. The SEL_TREE_SITTER_LANGUAGE_DIR environment variable can be set to modify where the JSON files are expected. By default, /usr/share/tree-sitter/,/usr/local/share/tree-sitter/, and $HOME/.local/share/tree-sitter/ are searched for tree-sitter files. If a directory is found and has node-types.json and grammar.json in it, the directory’s name is added to the list of languages to create classes for. For example, a directory named /usr/share/tree-sitter/python/ that has the relevant JSON files will result in class for python being generated. On the lisp side, sel looks in the directories specified by sel/sw/ts:*tree-sitter-language-directories* to find the grammar.json and node-types.json files that define a tree-sitter language. By default these files are expected to be in /usr/share/tree-sitter/$language/. The sel/sw/ts:*tree-sitter-language-directories* variable may be customized to control where sel searches for these files. For even more control over the tree-sitter files used the sel/sw/ts:*tree-sitter-language-files* variable may be set directly. These files are then parsed to automatically define associated common lisp classes for every type of AST node defined by the parser. Currently only the sel/sw/ts:python and sel/sw/ts:javascript tree-sitter classes have the full complement of more sophisticated sel/sw/parseable:parseable methods (e.g., sel/sw/parseable:scopes) defined.

tree-sitter support currently depends on cl-tree-sitter and the fork of cffi it depends on. It also depends on libtree-sitter and tree-sitter language modules for every desired language.