lexer.ts

Flat-event lexer substrate — the engine beneath the hand-written per-language lexers (lexer_json.ts, lexer_ts.ts, …).

Tokens are emitted as variable-length records into one Int32Array:

  • leaf: [type_id, start, end] where type_id > 0
  • open: [-type_id, start] — opens a container token
  • close: [0, end] — closes the innermost open container

Untyped text is implicit — it's the gap between events, recovered from offsets. Adjacent same-type leaves are coalesced at emit time so runs like }); become one span.

view source

Declarations
#

28 declarations

advance_probe
#

lexer.ts view source

(text: string, cached: number, from: number, ch: string): number import {advance_probe} from '@fuzdev/fuz_code/lexer.js';

Returns the cached next occurrence of ch at or after from, re-probing with indexOf only when the cached position has fallen behind. Infinity when the text has no further occurrence — a monotonic probe that keeps delimiter scans linear across a construct that is dense in ch.

text

type string

cached

type number

from

type number

ch

type string

returns

number

is_ascii_alnum
#

lexer.ts view source

(c: number): boolean import {is_ascii_alnum} from '@fuzdev/fuz_code/lexer.js';

c

type number

returns

boolean

is_ascii_word
#

lexer.ts view source

(c: number): boolean import {is_ascii_word} from '@fuzdev/fuz_code/lexer.js';

A \w word char — ASCII letters, digits, and _.

c

type number

returns

boolean

is_digit
#

lexer.ts view source

(c: number): boolean import {is_digit} from '@fuzdev/fuz_code/lexer.js';

c

type number

returns

boolean

is_hex_digit
#

lexer.ts view source

(c: number): boolean import {is_hex_digit} from '@fuzdev/fuz_code/lexer.js';

c

type number

returns

boolean

is_ident
#

lexer.ts view source

(c: number): boolean import {is_ident} from '@fuzdev/fuz_code/lexer.js';

c

type number

returns

boolean

is_ident_start
#

lexer.ts view source

(c: number): boolean import {is_ident_start} from '@fuzdev/fuz_code/lexer.js';

c

type number

returns

boolean

is_space
#

lexer.ts view source

(c: number): boolean import {is_space} from '@fuzdev/fuz_code/lexer.js';

c

type number

returns

boolean

lex_syntax
#

lexer.ts view source

(text: string, lang: SyntaxLang, langs?: Map<string, SyntaxLang> | undefined, types?: TokenTypeRegistry): LexedSyntax import {lex_syntax} from '@fuzdev/fuz_code/lexer.js';

Lexes text with lang, returning the flat token event stream.

text

type string

lang

type SyntaxLang

langs?

registry used to resolve embedded languages by id

type Map<string, SyntaxLang> | undefined
optional

types

token-type registry stamped on the result; must be the one lang (and any embedded language) interned its type ids into

type TokenTypeRegistry
default token_types_global

returns

LexedSyntax

LexedSyntax
#

lexer.ts view source

LexedSyntax import type {LexedSyntax} from '@fuzdev/fuz_code/lexer.js';

The result of lexing: the source text plus its flat token event stream.

text

type string

events

type Int32Array

events_len

type number

types

The registry that interned the type ids in events — consumers resolve ids through it, so a stream can never be rendered against the wrong id space.

type TokenTypeRegistry

Lexer
#

lexer.ts view source

import {Lexer} from '@fuzdev/fuz_code/lexer.js';

Shared lexing context passed to language lex functions. Holds the text window, the event buffer, and the language registry for embedding.

text

type string

pos

type number

end

type number

langs

Language registry for embed — set by lex_syntax.

type Map<string, SyntaxLang> | null

events

type Int32Array

events_len

type number

constructor

type new (capacity?: number): Lexer

capacity

type number
default 256

leaf

Emits a leaf token. Empty spans are dropped; a leaf adjacent to a preceding leaf of the same type extends it instead (span coalescing).

type (type_id: number, start: number, end: number): void

type_id

type number

start

type number

end

type number
returns void

open

Opens a container token at start. Must be balanced by a later close.

type (type_id: number, start: number): void

type_id

type number

start

type number
returns void

close

Closes the innermost open container at end.

type (end: number): void

end

type number
returns void

embed

Lexes [start, end) with the language registered as lang_id, restoring this lexer's window afterward. Returns false (leaving the region as plain text) when the language isn't registered or embedding is nested past MAX_EMBED_DEPTH.

type (lang_id: string, start: number, end: number): boolean

lang_id

type string

start

type number

end

type number
returns boolean

matches_ci
#

lexer.ts view source

(text: string, from: number, word: string): boolean import {matches_ci} from '@fuzdev/fuz_code/lexer.js';

Case-insensitive ASCII match of word (must be lowercase) at text[from..], via code | 0x20 folding — never allocates, unlike toLowerCase() comparisons.

text

type string

from

type number

word

type string

returns

boolean

render_syntax_html
#

lexer.ts view source

(lexed: LexedSyntax): string import {render_syntax_html} from '@fuzdev/fuz_code/lexer.js';

Renders a lexed token event stream to HTML in one forward pass. Gap text is copy-escaped; token spans use the precomputed open tags.

lexed

type LexedSyntax

returns

string

scan_balanced_braces
#

lexer.ts view source

(text: string, i: number, end: number): number import {scan_balanced_braces} from '@fuzdev/fuz_code/lexer.js';

Finds the matching } for the { at i, skipping js-style strings, templates, and comments. Returns -1 when unbalanced within the window.

text

type string

i

type number

end

type number

returns

number

scan_ident
#

lexer.ts view source

(text: string, from: number, end: number): number import {scan_ident} from '@fuzdev/fuz_code/lexer.js';

Scans an identifier starting at from (assumed to be an identifier start), returning the exclusive end index.

text

type string

from

type number

end

type number

returns

number

scan_to_line_end
#

lexer.ts view source

(text: string, i: number, end: number): number import {scan_to_line_end} from '@fuzdev/fuz_code/lexer.js';

Returns the index of the next \n at or after i (exclusive end of the line's content, excluding a preceding \r), or end when there is none. Uses native indexOf — the fast path for line-oriented scans.

text

type string

i

type number

end

type number

returns

number

skip_quoted
#

lexer.ts view source

(text: string, from: number, end: number, quote: number): number import {skip_quoted} from '@fuzdev/fuz_code/lexer.js';

Skips a js-style quoted span from the quote at from (used inside balanced scans), returning the index after the closing quote. Unterminated '/" strings stop at the newline; templates (`` `) span lines.

text

type string

from

type number

end

type number

quote

type number

returns

number

skip_space
#

lexer.ts view source

(text: string, from: number, end: number): number import {skip_space} from '@fuzdev/fuz_code/lexer.js';

Skips whitespace (including newlines) from from, returning the next non-space index.

text

type string

from

type number

end

type number

returns

number

syntax_events_to_tokens
#

lexer.ts view source

(lexed: LexedSyntax): SyntaxEventToken[] import {syntax_events_to_tokens} from '@fuzdev/fuz_code/lexer.js';

Flattens a lexed event stream to SyntaxEventTokens in document order (containers precede their children).

lexed

type LexedSyntax

returns

SyntaxEventToken[]

SyntaxEventToken
#

lexer.ts view source

SyntaxEventToken import type {SyntaxEventToken} from '@fuzdev/fuz_code/lexer.js';

A flattened token span, in document order with containers before their children. Used by fixtures, tests, and range building.

type

type string

start

type number

end

type number

SyntaxLang
#

lexer.ts view source

SyntaxLang import type {SyntaxLang} from '@fuzdev/fuz_code/lexer.js';

A lexer-based language registration.

id

Primary language id, e.g. 'ts'.

type string

aliases?

Alternate ids resolving to this language, e.g. ['typescript'].

type Array<string>

lex

Lexes the lexer's current [pos, end) window, emitting token events. Must never throw and must always terminate with lexer.pos === lexer.end.

type (lexer: Lexer) => void

token_type
#

lexer.ts view source

(name: string, alias?: string | string[] | undefined): number import {token_type} from '@fuzdev/fuz_code/lexer.js';

Interns a token type into token_types_global — the zero-config authoring path used by the built-in lexers' module-load type constants.

name

type string

alias?

type string | string[] | undefined
optional

returns

number

token_types_global
#

lexer.ts view source

TokenTypeRegistry import {token_types_global} from '@fuzdev/fuz_code/lexer.js';

The shared default registry — the single id space used by the built-in lexers' module-load token_type constants and by any SyntaxStyler not given its own registry. The type vocabulary is global by design, mirroring the global .token_* CSS namespace; per-registry isolation exists for fully-custom stylers and tests, whose lexers must intern into the same registry they're rendered against.

TokenTypeInfo
#

lexer.ts view source

TokenTypeInfo import type {TokenTypeInfo} from '@fuzdev/fuz_code/lexer.js';

Interned metadata for a token type.

id

type number

name

type string

aliases

type Array<string>

classes

Space-separated CSS classes, e.g. 'token_null token_keyword'.

type string

open_tag

Precomputed HTML open tag, e.g. '<span class="token_null token_keyword">'.

type string

TokenTypeRegistry
#

lexer.ts view source

import {TokenTypeRegistry} from '@fuzdev/fuz_code/lexer.js';

An id space of interned token types with precomputed CSS classes and HTML open tags. Ids are only meaningful against the registry that interned them — a lexed event stream resolves back through the registry stamped on its LexedSyntax.

infos

Interned infos indexed by id. Hot loops hoist and index this directly; grow it only via intern.

type Array<TokenTypeInfo>

readonly

intern

Interns a token type by name (+ optional aliases) and returns its id. Repeated calls with the same name and aliases return the same id. The CSS class list and HTML open tag are precomputed here so emitters never build class strings at runtime.

type (name: string, alias?: string | string[] | undefined): number

name

type string

alias?

type string | string[] | undefined
optional
returns number

info

Looks up the interned info for a token type id.

type (id: number): TokenTypeInfo

id

type number
returns TokenTypeInfo

trim_space_end
#

lexer.ts view source

(text: string, from: number, to: number): number import {trim_space_end} from '@fuzdev/fuz_code/lexer.js';

Trims trailing whitespace from a [from, to) span, returning the new exclusive end.

text

type string

from

type number

to

type number

returns

number

validate_syntax_events
#

lexer.ts view source

(lexed: LexedSyntax): string[] import {validate_syntax_events} from '@fuzdev/fuz_code/lexer.js';

Validates a lexed event stream's structural invariants, returning a list of human-readable issues (empty when valid): records well-formed, offsets monotonic and in-bounds, containers balanced.

lexed

type LexedSyntax

returns

string[]

words_map
#

lexer.ts view source

(...entries: [kind: number, words: string][]): Map<string, number> import {words_map} from '@fuzdev/fuz_code/lexer.js';

Builds a word→kind classification map from [kind, words] entries, where words is space-separated — the shared shape of the lexers' keyword tables.

entries

type [kind: number, words: string][]

returns

Map<string, number>

Imported by
#