Gema

Gema - the name stands for general purpose macro processor - is a small pattern matching language for transforming text. It reads an input stream, matches it against a set of user-defined rules, and writes the transformed result. That description also fits sed, awk, and m4, and gema’s own documentation invites the comparison. What separates it is the shape of its patterns: gema rules can span line boundaries, recurse into themselves to handle nested constructs, and switch between named rule sets depending on context. The result is that a gema pattern file for a nested notation tends to look strikingly like a BNF grammar for it.

The language is the work of one person, David N. Gray, who wrote it in 1994 and 1995 and released it without charge, asking only that the acknowledgment of the original source be kept. It has never had a large user base. It has, however, had an unusually persistent small one - and after nineteen years without a release, it received a major new version in March 2024.

A note on the date: this language is sometimes catalogued with a first-appearance year of 1991. That does not appear to be supportable. Every source file in the distribution carries the header “written by David N. Gray in 1994 and 1995”, the user manual is dated 17 March 1995, and the public announcement went out on 26 March 1995. This page uses 1995.

History and Origins

Gema was announced to the comp.compilers newsgroup on 26 March 1995. The announcement is brief and slightly harried - it ends by noting that the FTP host offering the source was being switched off five days later - but it does the one thing a language announcement most needs to do, which is name its ancestry:

This approach was inspired by W. M. Waite’s STAGE2 processor, but it also has some similarities to “awk.”

STAGE2 is the interesting half of that sentence. Waite’s macro processor, from around 1970, was the bootstrapping layer of the STAGE2/FLUB portable-software system: a template-driven text expander used to carry compilers and utilities between machines in an era when there was no portable implementation language to write them in. STAGE2 matched whole templates against input lines and substituted parameters. Gema takes that template-and-substitution model, removes the line orientation, and adds recursion - which is precisely the step that turns a macro expander into something that can parse.

The awk comparison is a comparison, not a lineage claim, and the manual makes the same kind of remark about cpp, grep, sed, and strings: gema can do the sorts of things those tools do. The claim of novelty is narrower and better founded - that unlike sed or awk it handles multi-line patterns and nested constructs, and that unlike cpp or m4 it imposes no fixed syntax on what a macro invocation looks like.

The author’s other work leaves fingerprints on the distribution. The examples directory contains a C-to-Dylan syntax converter and a Lisp-oriented pattern file, and the manual’s very first tutorial example for recursive arguments is converting Lisp s-expressions to infix function-call notation. A David N. Gray of Texas Instruments is well known in the Common Lisp world as the author of the “Gray streams” proposal to the ANSI X3J13 committee; gema’s documentation never makes the identification, but the Lisp and Dylan orientation of the examples is at least consistent with it.

Design Philosophy

Gema’s central bet is that context beats regular expressions. A regular expression describes a flat set of strings. Real text formats - markup, source code, s-expressions, nested delimiters - are not flat, and the standard workaround is to escape from the pattern language into a general-purpose one as soon as nesting appears. Gema instead makes recursion and context switching primitives of the pattern language itself.

Three ideas carry the design:

A rule is a template and an action. Written template=action, separated by =. Text matching the template is replaced by the result of evaluating the action. Actions are not merely replacement strings - they can call functions, set variables, redirect output, and change domains - but the common case is exactly as simple as sed’s s///.

Rules are grouped into domains. A domain is a named rule set. Translation begins in the default domain, whose name is the empty string, and both templates and actions can hand a stretch of input to a different domain. Domains may inherit: a::b means that if nothing in a matches, the rules of b are tried. The same mechanism serves three purposes at once - a domain used in a template is a custom argument type, a domain used in an action is a user-defined function, and a domain entered on a delimiter is a lexical context.

Matching and function definition are unified. This is the point David A. Mundie singles out in his essay Why I Love Gema, included in the distribution: in gema, defining a new kind of thing to recognize and defining a new operation to apply are the same act. His comparison table gives sed, awk, and Perl regular expressions, gives awk and Perl non-line-oriented matching, but reserves user-defined recognizers, unification of matching and function definition, and multiple rule sets for gema alone.

There is a fourth, quieter commitment: gema is deliberately unopinionated about syntax. Nearly every special character can be reassigned with @set-syntax or disabled with -literal, because a tool for translating arbitrary notations cannot afford to reserve characters that those notations need.

Key Features

Arguments

A template is literal text plus arguments - the parts that match variable input. Gema offers five kinds, and choosing among them is most of the skill of writing gema:

The distinction between * and # is the language’s crux. Consider the manual’s own example - converting Lisp s-expressions to function-call notation:

Rule:   (* * *)=*(*,*)
Input:  (fn (g a b) z)
Output: fn((g,a b)

The wildcards have no idea that parentheses nest. Change three characters:

Rule:   (# # #)=#(#,#)
Input:  (fn (g a b) z)
Output: fn(g(a,b),z)

The recursive argument re-enters the same rule set on the inner text, so the inner parentheses are consumed by an inner match, and the result is translated too. One character bought both bracket matching and structural recursion.

Predefined recognizers

Angle-bracket arguments come with a built-in set covering the character classes a translator usually wants: <D> digits, <L> letters, <I> identifiers, <N> numbers with optional sign and decimal point, <A> alphanumerics, <F> file pathnames, <G> graphic characters, <C> control characters. The notation is compact and systematic - an uppercase letter requires at least one character, lowercase makes the argument optional, a leading - inverts the test, and a trailing digit fixes or caps the count. So <D3> is exactly three digits, <d3> is up to three, <-D> is one or more non-digits, and <U10> is any ten characters, which is how you split a fixed-width record.

Operators

Backslash followed by an uppercase letter is a matching operator rather than an escape. The most-used are \N (a line boundary, matched without consuming the newline - which is how you write a whole-line rule that also works on the first line of the file), \I (identifier boundary, so \Ix\I replaces the variable x without touching xyz), \W (skip optional whitespace), \L (forbid the following arguments from crossing lines), \P (match as pure lookahead, leaving the input position untouched), and \G (mark the goal point that terminates the preceding argument). A bare space in a template already means “one or more whitespace characters”; matching exactly one space requires \s.

User-defined recognizers

Because a domain can serve as an argument type, extending the pattern vocabulary takes one line. To match either yes or no:

yesno:yes=yes@end;no=no@end;=@fail

and then use it anywhere:

done\? <yesno>=Finished \= $1

The empty template in the last rule is the default case, reached only when nothing else in the domain matches; @fail aborts the enclosing match. The manual adds a practical warning - give domains at least two letters, or they will collide with the single-letter predefined recognizers.

Contexts

The canonical use of domains is protecting regions that should not be transformed. Renaming identifiers in C source without touching string literals starts as "<sbody>"="$1", with a companion rule sbody:\\"=\\" so that an escaped quote inside a string does not terminate the argument. This is the same job that in a regex-based tool requires either a heroic lookbehind or a hand-written tokenizer.

Built-in functions

Actions can call functions with @name{arg;arg}. The set is small but practical: arithmetic, string manipulation with padding, filling and wrapping, case conversion and comparison, variables, pathname manipulation, alternate input and output files, file context queries such as @file and @line, control flow (@end, @fail, @terminate, @abort), and shell invocation. A default error rule is a one-liner:

\N.*\N=@err{@file line @line\: Unrecognized\: $1}

Evolution

The language settled quickly and then barely moved. Version 1.2 is what the 1995 manual documents; the subsequent releases were consolidation rather than redesign. The RCS history in the source shows most 1995 activity going into portability - MS-DOS wildcard expansion, Windows NT, a workaround splitting a large string constant into five pieces to placate Apple’s MPW compiler - rather than into the pattern language.

Two changes are worth singling out. The -ml option, added in December 2001, lets patterns use [...] instead of <...> for domain arguments - a small ergonomic fix with an obvious motive, since angle brackets are the one thing you are guaranteed to be matching literally when the input is HTML or XML. And GeL, contributed by Remo Dentato for version 1.4, binds the matching engine to Lua, giving actions a real programming language instead of the built-in function list. Dentato became co-maintainer, and the project’s later life is largely his.

The 2024 revival is the surprise. After nineteen dormant years, versions 1.5 and 2.0 appeared within weeks of each other in March 2024, splitting the line: 1.5 preserves 8-bit behaviour, and 2.0 adds full Unicode while, per its release notes, maintaining compatibility with previous usage. That ordering is a considerate piece of release engineering for a tool whose users are likely to have decade-old pattern files in production.

Portability

Gema is ANSI C with no dependencies beyond the standard library - the Lua-bound gel and gua binaries are optional, built only if the gbuild script locates Lua headers. The distribution README states support for several varieties of Unix, MS-DOS, Microsoft Windows, and Macintosh including both OS X and classic MacOS under MPW, with #ifdef unix, #ifdef MSDOS, and #ifdef MACOS marking the places needing attention on a new system. The SourceForge project additionally lists BSD, Linux, and Solaris.

The clearest evidence is the 2.0 release itself, which ships prebuilt binaries for Windows 32-bit and 64-bit, Linux x64, macOS x64, ARM64 and universal, and 32-bit SPARC. A source tarball and a Git repository cover everything else. No official or community Docker image is published.

Current Relevance

Gema occupies a niche that has not gone away: one-off translation between text formats that are structured enough to defeat sed and not important enough to justify writing a parser. That work is now mostly done in Python or Perl, and gema’s practical disadvantage against them is not expressiveness but familiarity - a # argument is genuinely more elegant than a hand-rolled bracket counter, but only if you already know what a # argument is.

Activity is real but slight. The SourceForge project reports download counts in the low dozens per week and carries a Community Choice badge, which SourceForge awards for cumulative download milestones. There is a discussion mailing list, a Git repository, and a bug tracker, and the 2024 releases prove the maintainers are still reachable. It is reportedly packaged in the text-processing sections of some Unix package collections. What there is not is a community in any meaningful sense - no active forum, no ecosystem, no third-party libraries.

The idea, however, has descendants in spirit. Parsing expression grammars, the parser-combinator libraries of the functional languages, and tree-sitter’s query language all rest on the same observation gema made in 1995: that once patterns can recurse and carry context, the boundary between “search and replace” and “parse and translate” stops being a boundary at all.

Why It Matters

It shows what regular expressions are missing, precisely. The gap is not power in the abstract - it is nesting and context, two things that recur in every real text format and that regular languages cannot express. Gema isolates exactly those two additions and shows how cheap they can be: one sigil for recursion, one colon for context. That is a clean, small result about language design, and the (# # #) example demonstrates it in three lines.

It is a rare survival from the STAGE2 tradition. Template-driven macro processors were serious infrastructure in the 1970s, the mechanism by which software crossed between incompatible machines. Almost nothing from that lineage is still downloadable and still building on current hardware. Gema is, and it carries the acknowledgment forward in its announcement.

It documents a genuine minority taste. Mundie’s Why I Love Gema, written by someone who had already been through Snobol, Pascal, Ada, ML, Prolog, awk, Tcl and Perl, is an unusually clear articulation of why a rule-based, non-procedural view of text appeals to some programmers and not others. Gema never became popular. The essay explains, better than adoption statistics could, why the people who used it kept using it for thirty years.

Sources

• gema project homepage - overview, documentation, license, and downloads
• gema user manual - the full specification, dated 17 March 1995 and revised 30 November 2003
• gema manual page - tutorial introduction and reference summary
• Announcing “gema” - the general purpose macro processor - the comp.compilers announcement of 26 March 1995, with the STAGE2 attribution
• gema update - follow-up posting on comp.compilers
• gema on SourceForge - registration date, license, maintainers, and release history
• gema 2.0 release files - March 2024 binaries and Unicode release notes
• gema source mirror on GitHub - source headers, RCS logs, and bundled example pattern files
• General-purpose macro processor - context on the tool category and gema’s place in it
• Gray Streams in Allegro CL - background on the X3J13 proposal by a David N. Gray of Texas Instruments