Parsing #placeholder# templates when a regex is not enough.

Generative Grammar Engine writes rules in Tracery’s notation: #name# marks a placeholder, the rest is literal text. To run a rule, the engine first splits a string like #greeting#, #name#! into placeholders and literals. The obvious tool is a regular expression, and it works until someone needs a literal # in their text.

Where the regex breaks

The common pattern is /#([^#]+)#/g: a hash, some non-hash characters, a closing hash. It knows nothing about escaping. Once you let \# mean a literal hash, the regex cannot tell these two apart:

#weight\#1#      one placeholder whose inner text is  weight#1
#a# #b#          two placeholders, a and b

To the regex both are just hashes and characters. A cleverer expression is possible, but an escapable delimiter is exactly the case regular expressions handle badly, so a small scanner takes its place instead. It comes out both clearer and correct.

Finding the real closing hash

Scanning forward from an opening hash, a backslash escapes the next character, so \\ and \# are skipped as pairs and never end the placeholder. The first unescaped hash closes it:

function findClosingHash(template, openIdx) {
  let j = openIdx + 1;
  while (j < template.length) {
    if (template[j] === '\\' && j + 1 < template.length) { j += 2; continue; }  // skip escaped pair
    if (template[j] === '#') return j;
    j += 1;
  }
  return -1;   // unterminated
}

Splitting into segments

The main scan walks the string, building a literal run until it hits a placeholder. A backslash collapses its pair into the literal. An unescaped hash flushes the literal and emits a placeholder using the closing index above. An unterminated hash falls back to literal, which is what the naive regex would have left alone anyway:

while (i < template.length) {
  const c = template[i];
  if (c === '\\' && i + 1 < template.length) {            // escape: keep next char literally
    lit += template[i + 1] === '#' ? '#' : template[i + 1];
    i += 2;
    continue;
  }
  if (c === '#') {
    const close = findClosingHash(template, i);
    if (close < 0) { lit += template.slice(i); break; }   // unterminated -> literal
    flushLit();
    out.push({ kind: 'placeholder', innerRaw: template.slice(i + 1, close) });
    i = close + 1;
    continue;
  }
  lit += c;
  i += 1;
}

One job per pass

The split keeps a placeholder’s inner text raw. Decoding the escapes is a separate step that runs on the inner string before it is read as a rule name and modifiers:

// "weight\#1" -> "weight#1",  "a\\b" -> "a\b"
function decodePlaceholderInner(raw) { /* same backslash walk, returns decoded text */ }

Tokenization and decoding stay apart, and that is what keeps each one small. The scanner only decides where a placeholder starts and ends; the decoder only turns \# into #. Neither reasons about the other, and the same two functions back the engine, the parameter extractor, and the graph view, so every part of the app reads a grammar the same way.

When to drop the regex

A regex is fine for flat patterns. The moment a delimiter can be escaped, nested, or carries its own mini-syntax, a short character scanner is easier to read and to trust than the expression that tries to do everything at once, mostly because it keeps finding tokens and decoding them as two separate jobs.

Generative Grammar Engine is free at grammar.tinygods.dev, and the code is open source at github.com/tinygodsdev/metatracery.