Fullmoon Syntactic Code Highlighter: Design Notes on Safe Code Display

A highlighter design that separates TextMate tokenization from safe DOM output and limits the rendering cost of long code samples.

1. Separate tokenization from display

These are design notes for a code highlighter. TextMate grammars are tokenization rules that assign scopes to code; they are not AST parsers with a complete understanding of a language's semantics. Oniguruma is a regular expression engine, so running it in WebAssembly does not eliminate regular expression processing. Distinguish the roles of vscode-textmate and vscode-oniguruma.

2. A safe fallback display

The code below is not a syntax highlighter. It is a basic output example that safely displays the original source when tokenization is unavailable. Assign the input through textContent instead of inserting it directly into an HTML string. See the textContent documentation.

/* typescript */
export function renderPlainCode(container: HTMLElement, source: string): void {
  const pre = document.createElement("pre");
  const code = document.createElement("code");
  code.textContent = source;
  pre.append(code);
  container.replaceChildren(pre);
}

When tokenization succeeds, construct spans using permitted theme classes and text nodes. Code must remain readable through a fallback like the one above if grammar or theme loading fails.

3. Long code samples and validation

• Load language grammars and WebAssembly on demand, and handle loading errors.
• Measure tokenization and DOM construction separately; consider splitting work or virtualization for long inputs.
• Test input resembling HTML tags, very long lines, invalid syntax, and documents containing multiple languages.
• Evaluate performance using results that record code size, language, and device conditions, rather than guaranteeing a fixed speedup or frame rate.