Hand-written TypeScript/JavaScript lexer.
Disambiguation rests on three rules: previous-token tracking separates a
regex literal from division, template literals nest their ${…}
interpolations, and matching is position-major (a string wins over a comment
that opens inside it).
Nesting (template interpolations, generic argument lists, : type
annotations) runs on an explicit pooled frame stack, so arbitrarily deep
input tokenizes fully without touching the JS call stack.
Resilience: unterminated strings extend to end of line; unterminated
templates, block comments, and interpolations extend to end of window.
Interpolation bodies discover their own closing } during real tokenization
(nothing is prescanned), so constructs that contain a } — regex literals,
strings, comments — never end an interpolation early. The tradeoff is that
damage propagates rather than being contained: a malformed interior that
consumes the closing } (say an unterminated block comment) extends the
interpolation past it, editor-style.
SyntaxLang import {lexer_ts} from '@fuzdev/fuz_code/lexer_ts.js'; The TypeScript language registration for the lexer engine.
JavaScript reuses this lexer via the js/javascript aliases: TypeScript is
a syntactic superset, and the TS-only constructs the lexer recognizes (type
annotations, as, generics before a call) can't appear in valid JS, so
running the full TS lexer on JS is a no-op for those paths. There is no
separate JS lexer.