tokenize_syntax.ts view source
(text: string, grammar: SyntaxGrammar): SyntaxTokenStream Accepts a string of text as input and the language definitions to use, and returns an array with the tokenized code.
When the language definition includes nested tokens, the function is called recursively on each of these tokens.
This method could be useful in other contexts as well, as a very crude parser.
text
a string with the code to be styled
type
stringgrammar
a SyntaxGrammar object containing the tokens to use.
Usually a language definition like syntax_styler.get_lang('markup').
type SyntaxGrammar
returns
SyntaxTokenStream a SyntaxTokenStream array of strings and tokens
examples
var code = `var foo = 0;`;
var tokens = tokenize_syntax(code, SyntaxStyler.langs.js);
for (var token of tokens) {
if (token instanceof SyntaxToken && token.type === 'number') {
console.log(`Found numeric literal: ${token.content}`);
}
}