CRM114

CRM114 - the “Controllable Regex Mutilator”, named after the CRM 114 Discriminator, the radio receiver in Stanley Kubrick’s Dr. Strangelove that ignores any message lacking the right code prefix - is a programming language that most people encountered without ever knowing it was a programming language. They installed it as a spam filter. Underneath the mail filter scripts sits a Turing-complete language for chopping up, matching, mutating, and statistically classifying text, written by Bill Yerazunis and, according to the project’s own documentation, first released publicly under the GPL in early 2001. It is one of the very few general-purpose languages in which “learn this document as spam” and “which of these two corpora does this document resemble” are single statements rather than library calls.

History and Origins

Yerazunis was a senior research scientist at Mitsubishi Electric Research Laboratories in Cambridge, Massachusetts. Like everyone else with a public email address at the turn of the century, he was drowning in spam, and unlike most people he found the available filters - rule sets and heuristics that were, in his description, 90% or 95% accurate - unsatisfying. His answer was to build a language whose native operations were exactly the ones a text classifier needs, and then write the filter in it.

The project became widely known on October 16, 2002, when Paul Graham posted Yerazunis’s results to his site under the heading “CRM114 gets 99.87%”. The measurement was a live-mail test: 2,374 fresh incoming messages, 1,518 of them spam and 856 not, classified with the sparse binary polynomial hash (SBPH) classifier and no blacklists or whitelists, producing three errors - one false rejection and two false acceptances. Yerazunis noted in the same report that when he hand-classified about 1,900 of the same messages twice he made three errors of his own, a personal accuracy of 99.84%, which framed the result honestly: the filter had reached the neighbourhood of human accuracy on that corpus, not some absolute ceiling. That number, on that corpus, in that year, is the origin of nearly every “CRM114 is 99.9% accurate” claim you will find repeated since; results vary substantially with the corpus, and the TREC 2005 and 2006 evaluations put well-configured CRM114 setups above 99% with meaningful spread between test sets.

John Goerzen packaged CRM114 for Debian on January 21, 2003, and it spread through the Linux distributions from there. By March 2007 Yerazunis was being profiled in Network World as “the antispam man”, with CRM114 reportedly in use by individuals, corporations, and some ISPs.

Design Philosophy: Declension, Not Position

The single most unusual thing about CRM114 is its syntax. In almost every programming language, the meaning of an argument comes from where it sits in the call. In CRM114 it comes from what punctuation surrounds it - the language is declensional rather than positional, borrowing the idea from inflected natural languages where a noun’s ending, not its place in the sentence, marks its grammatical role.

The declensions are:

Only the action keyword must come first. match <nocase> (:word:) [:buffer:] /foo/ and match (:word:) /foo/ [:buffer:] <nocase> mean the same thing. The payoff is that statements with a dozen possible modifiers - and classify accepts a great many - stay readable without a fixed argument order nobody could remember.

The second unusual choice is that everything is text. Variables are named with surrounding colons (:name:, :_dw:, :_env_PATH:), and by default a variable is not storage at all: it is a start/length pair pointing into the default data window, the buffer holding the text being processed. Altering a non-isolated variable edits the underlying document in place. isolate is the statement that gives a variable its own storage and detaches it from the window - which makes isolate the closest thing CRM114 has to a declaration, and forgetting it the classic CRM114 bug.

A First Program

By default a CRM114 program slurps all of standard input into the data window before executing. A leading window statement overrides that, which is why even “hello world” needs one:

#! /usr/bin/crm
window
output /Hello, world!\n/

Something closer to the language’s purpose - read a message, decide which of two learned corpora it belongs to:

#! /usr/bin/crm
isolate (:stats:)
{
    classify <osb unique microgroom> (spam.css | good.css) (:stats:) /[[:graph:]]+/
    output /SPAM\n:*:stats:/
}
alius
{
    output /GOOD\n:*:stats:/
}

:*:stats: is variable expansion - the value of :stats: interpolated into the surrounding text. Training is the mirror image, and equally short:

learn <osb unique microgroom> (spam.css) /[[:graph:]]+/

Control flow is where the language shows its age and its strangeness. There is no if and no for. Statements either succeed or FAIL; a FAIL skips to the end of the enclosing { } block and exits it with failure status. alius (“otherwise”, from Latin) runs only if the preceding block failed, giving you if/else and switch/case. liaf - fail spelled backwards - jumps back to the start of the enclosing block, which is how you write loops. Real CRM114 code is full of match-then-liaf idioms that walk a buffer one match at a time:

match [:option_txt:] //
{
    match <fromend nomultiline> (:line: :name: :value:) \
          [:option_txt:] /^([[:graph:]]+).*\/(.*)\/.*$/
    isolate (:*:name:) /:*:value:/
    liaf
}

The language is Turing complete, but it is unapologetically specialised: as Wikipedia’s article dryly observes, even a recursive definition of factorial takes almost ten lines. Arithmetic lives inside eval, which repeatedly evaluates a string until it stops changing, and supports both algebraic and RPN math modes.

Key Features

• Approximate regex matching. CRM114 uses the TRE regex engine, so patterns can match with a bounded number of errors. A filter can catch V1agra without an exhaustive list of obfuscations.
• A menu of classifiers. learn and classify take the same arguments regardless of which algorithm is selected by flags: the Markovian/SBPH default, orthogonal sparse bigrams (OSB), Littlestone’s Winnow, a KNN variant called Hyperspace, a bit-entropy classifier that works one bit at a time via entropy coding, the fast substring compression matcher (FSCM, essentially LZ77 similarity), full character correlation, an SVM, a string-kernel SVM, and an experimental three-layer neural net with back-propagation.
• Microgrooming. Statistics files (.css) are fixed-size hash tables; the microgroom flag lets the engine evict low-value features when a chain gets too long, so a filter can run indefinitely without unbounded growth.
• Null-safe, UTF-8-capable text handling. Buffers are length-delimited rather than NUL-terminated, so binary and multilingual data pass through intact - a property that helped in work such as the Japanese-language document classification presented at Black Hat in 2010.
• syscall and window. Programs can shell out and stream text through the data window, which is what lets a mail filter integrate with procmail, MTAs, and external tools.
• A full statement set. match, alter, learn, classify, input, output, isolate, window, hash, translate, union, intersect, eval, syscall, trap/fault, goto, call/return, accept, exit, plus fail, liaf, alius, and noop.

Evolution

The classifier story is the language’s real evolution. The original SBPH scheme hashed sparse phrase patterns up to five words long into a Markov random field - far more expressive than single-word Bayes, but expensive in both memory and time. The 2004 ECML/PKDD paper by Siefkes, Assis, Chhabra, and Yerazunis introduced orthogonal sparse bigrams, which keep most of the contextual power at a fraction of the cost, and paired them with Winnow instead of naive Bayes. OSB became the default recommendation, and the flag <osb unique microgroom> the canonical incantation.

Training methodology evolved alongside the algorithms. mailtrainer.crm takes directories of example messages and trains repeatedly until the filter classifies its own training corpus correctly, an approach in the same family as train-on-error and thick-threshold training that the spam-filtering community converged on in the mid-2000s.

The last language release, crm114-20100106-BlameMichelson, shipped on January 6, 2010 - CRM114 versions are dates plus a whimsical codename rather than semantic version numbers. Later that year Yerazunis released libcrm114, an LGPL-licensed C library exposing most of the classifiers directly, in memory, with no filesystem and no CRM114 language at all. It is a fair reading of that release as the author’s own verdict: the classifiers were the durable contribution, and the language was the harness that produced them.

Current Relevance

CRM114 is dormant. Upstream development stopped after January 2010, the announce mailing list went quiet in March 2007, and FreeBSD deleted its libcrm114 port in April 2021 after deprecating it with the note “no known users, dead upstream”. What survives is packaging inertia and a residue of working installations: Debian still carries crm114 20100106 into its current releases, and third-party bindings such as Text::AI::CRM114 and AI::CRM114 on CPAN and crm114/pycrm114 on PyPI are still published, if largely unmaintained.

The reason it stopped mattering is not that it stopped working. It is that the problem moved. Spam filtering migrated to hosted mail providers running reputation systems, network-level signals, and large-scale machine learning over billions of messages - none of which a per-user statistics file can compete with. The content-classification battle CRM114 was designed to win is now fought somewhere the individual user cannot see.

Why It Matters

CRM114 is worth studying for two reasons that have nothing to do with spam.

The first is the declensional syntax. It is one of the few serious attempts to answer “how should a statement with twenty optional modifiers be written?” with something other than keyword arguments or a builder pattern, and the answer - let punctuation carry grammatical role, free the order entirely - is genuinely novel and genuinely usable in the domain it was built for. Reading CRM114 code is a good cure for the assumption that argument position is a law of nature.

The second is the demonstration that statistical classification can be a language primitive. In 2001, putting learn and classify on the same footing as match and output was an odd thing to do; two decades later, with model inference showing up as a first-class operation in new languages and frameworks, it looks less like an eccentricity and more like an early instance of a recurring idea. That researchers repurposed a spam filter into a defect predictor, a bot detector, and a confidential-document scanner - largely by pointing it at different training corpora and choosing among its existing classifiers - is the strongest evidence that the abstraction was the right one.