import {MdzLexer} from '@fuzdev/mdz/mdz_lexer.js'; Tokenizes mdz input into a flat MdzToken stream for MdzTokenParser.
The lexer owns all structural decisions — block boundaries, list level
stacks, blockquote prefix stripping (sub-lexing quote content as a
mini-document with positions remapped to source offsets), and inline
delimiter pairing bounded by #max_search_index — so the token parser
only assembles the tree.
constructor
type new (text: string): MdzLexer
text
stringrebind
Rebind to a new text, so one lexer instance can be reused across inputs —
the streaming table path reuses one across a table's rows via
MdzTableCellParser instead of allocating a lexer (and its memo Map)
per row. The search memos survive when the text is identical (they're
valid over the immutable text — the common case for rows parsed from one
buffered chunk) and clear otherwise. Per-parse cursor state is reset by
lex_table_cell itself.
type (text: string): void
text
stringvoidtokenize
type (): MdzToken[]
MdzToken[]lex_table_cell
Tokenize a table cell's inline content [cell_start, cell_end) as a
bounded inline run, with no wrapping cell tokens (the bare form of
#emit_table_cell, sharing #tokenize_cell_inline). The streaming parser
reuses it across a row's cells via MdzTableCellParser: the token buffer
resets per call while the search memo — valid over the immutable bound
text — carries across cells (the same reuse the sync lexer does in place).
type (cell_start: number, cell_end: number): MdzToken[]
cell_start
numbercell_end
numberMdzToken[]