The Language Inside the Language: A Tour of Metaprogramming

Every programming language eventually runs into the same wall: the programmer knows something at write time that the language cannot express directly. How to serialize a thousand different types without writing a thousand different functions. How to build a generic container that works for any type. How to implement a new control structure the language forgot.

The solutions define some of the most distinctive — and divisive — features in language design. They are collectively called metaprogramming: code that generates or transforms other code. The approaches span a gulf so wide that languages from the same decade look almost unrecognizable to each other.

The story starts in 1958. It is still unfolding today.

The First Metaprogrammer: Lisp (1958)

John McCarthy did not set out to invent metaprogramming. He set out to build a list processing language for AI research. What he produced was an accident that the rest of the programming world is still absorbing.

Lisp is built on a single insight: code and data have the same shape. A Lisp program is a list. A list is data. Therefore, a Lisp program is data. This property — code representable as a data structure the language can manipulate — is called homoiconicity, and it is the foundation of everything that follows.

A Lisp macro receives unevaluated code as a data structure, transforms it, and returns a new expression to be evaluated. The macro does not see values — it sees the abstract syntax tree itself.

1
2
3
4
5
6
7
8
9

;; A simple macro that creates an "unless" (opposite of "when")
(defmacro unless (condition &rest body)
  `(if (not ,condition)
       (progn ,@body)))

;; This usage expands at macro-evaluation time:
;; (unless (= x 0) (print "not zero"))
;; becomes:
;; (if (not (= x 0)) (progn (print "not zero")))

The backtick and comma syntax is Lisp’s template mechanism: backtick quotes the whole form, comma splices in an evaluated part. The result is that you construct arbitrary code as data, return it, and it runs.

Crucially, Lisp macros operate on the language’s actual syntax tree. They understand structure and scope. The famous LOOP macro in Common Lisp implements an entirely different sub-language for iteration — with clauses like for x in list, collect, and finally — built using the same macro mechanism any programmer can use. No preprocessor has ever matched this.

The C Preprocessor (1972): Text Manipulation Disguised as Metaprogramming

C arrived fourteen years after Lisp and took a radically different approach to the same problem. The C preprocessor — the #define and #include machinery that runs before the compiler sees your code — is not part of the C language at all. It is a separate text-substitution program that happens to run first.

1
2
3
4
5
6
7

/* This looks like a function. It is not. */
#define MAX(a, b) ((a) > (b) ? (a) : (b))

/* These calls look safe. They are not. */
int x = 5, y = 3;
int result = MAX(x, y);         /* Fine: expands to ((x) > (y) ? (x) : (y)) */
int danger = MAX(x++, y++);     /* Bug: x and y each increment twice */

That MAX(x++, y++) bug is not a contrived example. It is a textbook demonstration of why the preprocessor is considered one of C’s most dangerous features. The preprocessor does not know what x++ means. It does not know that x++ is an expression with a side effect. It substitutes the text x++ wherever a appears in the macro template, producing:

1

((x++) > (y++) ? (x++) : (y++))

Both x and y are incremented twice — once in the comparison and once when the larger value is used. The behavior is undefined by the C standard (incrementing the same variable twice before the next sequence point), meaning the compiler is free to produce any result, including one that appears to work, which is arguably worse because it makes the bug intermittent.

The preprocessor has no concept of types, scope, or any semantic property of the language it processes. It is, technically speaking, not metaprogramming — it is text processing with macros. The CVE database contains numerous vulnerabilities from unsafe macro expansion: security-critical code that read correctly but behaved differently because an expression with side effects appeared in a macro argument. Lisp macros operate on the syntax tree. C preprocessor macros operate on raw text. The distance between these is the distance between metaprogramming and find-and-replace.

C++ Templates (1985+): A Turing-Complete Accident

C++ added templates in the early 1990s, initially as a mechanism for writing generic data structures — containers that could hold any type rather than a specific one. The intent was practical and modest.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

// A simple generic max function — the intent
template<typename T>
T max(T a, T b) {
    return a > b ? a : b;
}

// No preprocessor tricks. No text substitution.
// The compiler generates a specialized version for each type used.
int a = max(3, 5);         // max<int>
double b = max(3.1, 5.7);  // max<double>

Unlike the C preprocessor, C++ templates are part of the language. The compiler understands types, resolves overloads, and checks semantics. max with two arguments of incompatible types is a type error, not a silently misbehaving text substitution.

Then someone noticed something unexpected: template specialization — choosing different template implementations based on types — combined with template parameters that could themselves be integers, created a mechanism for computation at compile time. Erwin Unruh famously demonstrated this in 1994 by writing a C++ program that, as a side effect of generating compiler error messages, printed prime numbers. The compiler was doing arithmetic during compilation, not at runtime.

The classic demonstration is compile-time factorial:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

// Template metaprogramming: factorial computed entirely at compile time
template<int N>
struct Factorial {
    static const int value = N * Factorial<N - 1>::value;
};

template<>
struct Factorial<0> {
    static const int value = 1;
};

// At runtime, this is just loading a constant.
// The multiplication happened during compilation.
int result = Factorial<5>::value;  // 120, computed at compile time

Template metaprogramming (TMP) became a serious discipline. Entire libraries — most famously Boost — were built on it, computing type lists, detecting type properties, and specializing algorithms at zero runtime cost.

The cost was brutal elsewhere. A type mismatch deep in a chain of template instantiations produced error output running to thousands of lines, because the compiler reported the failure at every layer of the instantiation stack simultaneously. A beginner misusing the STL could receive output longer than their source file. Senior engineers learned to parse these messages. It was not an easy skill.

The C++ committee recognized this as a design failure. C++11 introduced constexpr — a direct mechanism for running ordinary C++ code at compile time:

1
2
3
4
5
6
7

// constexpr factorial: looks like a normal function
constexpr int factorial(int n) {
    return n <= 1 ? 1 : n * factorial(n - 1);
}

// Evaluated at compile time when the argument is a compile-time constant
constexpr int result = factorial(5);  // 120, in the binary as a constant

C++20 dramatically expanded constexpr to allow most ordinary C++ code — loops, local variables, most standard library functions — to run at compile time. The explicit goal was to reduce the industry’s dependence on template metaprogramming. You can still write TMP. You no longer need to in most cases where you previously had to.

Rust’s Two Macro Systems

Rust, releasing its 1.0 in 2015, arrived having watched C++ templates and the C preprocessor from a distance and made deliberate choices about what to take and what to avoid.

Rust has two distinct macro systems. The first, declarative macros (macro_rules!), is pattern-based. You describe patterns in the input token stream and what to produce when they match:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

// A simple macro that creates a vector from its arguments
macro_rules! my_vec {
    ( $( $x:expr ),* ) => {
        {
            let mut v = Vec::new();
            $(
                v.push($x);
            )*
            v
        }
    };
}

let numbers = my_vec![1, 2, 3, 4, 5];

The $( $x:expr ),* pattern matches zero or more comma-separated expressions. The $( v.push($x); )* in the output repeats the statement for each matched expression. This is more structured than the C preprocessor — the pattern system understands Rust’s token categories (expressions, types, identifiers, blocks) rather than treating everything as raw text. A macro that says it expects an expr will not match a bare type name.

The second system, procedural macros, is a different mechanism entirely. A procedural macro is a full Rust program that runs at compile time: it receives a token stream as input, manipulates it using Rust code, and returns a new token stream that the compiler then compiles as though the programmer had written it directly.

The most consequential demonstration of procedural macros is serde, Rust’s serialization framework. To serialize and deserialize a struct in virtually any format — JSON, YAML, MessagePack, TOML, and dozens more — you annotate it with #[derive(Serialize, Deserialize)]:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

use serde::{Serialize, Deserialize};

#[derive(Serialize, Deserialize)]
struct User {
    name: String,
    email: String,
    age: u32,
}

// At compile time, serde generates complete serialization/deserialization
// code specialized for this exact struct — no runtime reflection required.
let json = serde_json::to_string(&user)?;
let user: User = serde_json::from_str(&json)?;

The #[derive(Serialize, Deserialize)] annotation triggers procedural macros that inspect the struct’s fields at compile time and generate type-aware serialization code for every field. No runtime reflection, no dynamic dispatch, no hash map lookup. The generated code is as efficient as hand-written code for that specific struct — which, for a library supporting dozens of formats across thousands of user-defined types, no developer could ever write by hand.

This is categorically different from the C preprocessor. Serde’s macros understand Rust’s type system. A bug in the generated code is a compile error, not a runtime mystery.

D’s CTFE and Zig’s comptime

Two languages took the template/macro approach and asked a sharper question: why does compile-time code need to look different from runtime code at all?

D, from Walter Bright in the early 2000s, introduced CTFE — compile-time function execution. Almost any D function can run at compile time if called in a compile-time context, simply by placing it in a context that demands a compile-time value. D also has string mixins — constructing a string at compile time and compiling it as D source — which are powerful but bring the predictable risks of string-based code generation. They operate on text rather than on a syntax tree.

Zig, arriving in 2016, made the unification its central design philosophy. No macro system. No template system. No preprocessor. Instead: comptime, a keyword marking a value as needing to be known at compile time, with the guarantee that any function whose arguments are all comptime-known can be evaluated at compile time.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

// A perfectly ordinary Zig function
fn factorial(n: u64) u64 {
    if (n <= 1) return 1;
    return n * factorial(n - 1);
}

// Runtime call:
var a: u64 = factorial(5);

// Compile-time call — comptime forces evaluation at compile time:
const b: u64 = comptime factorial(5);  // 120 in the binary

Generics in Zig are implemented as comptime functions that return types — because types are valid comptime values:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

// A generic stack: a function that takes a type and returns a type
fn Stack(comptime T: type) type {
    return struct {
        items: []T,
        len: usize,

        pub fn push(self: *@This(), item: T) void {
            // ...
        }
    };
}

// Usage looks like a function call, because it is one
const IntStack = Stack(i32);
const StringStack = Stack([]u8);

No template syntax. No macro syntax. No separate compile-time language. The Stack(i32) call is literally a function call that returns a type. Its body is ordinary Zig. Its loops and conditionals work identically to their runtime equivalents.

This is one of Zig’s most explicitly stated design motivations: replace the C preprocessor with something honest. Where C’s preprocessor is a separate text-substitution language bolted on before the compiler sees the code, Zig’s comptime is simply Zig, evaluated earlier.

Runtime Metaprogramming: Ruby and Julia

The languages above primarily concern compile-time metaprogramming: code that generates or transforms code before the program runs. But metaprogramming has a runtime dimension too, and no language demonstrates it more vividly than Ruby.

Ruby’s object model is mutable at runtime. Classes can be opened and modified after they are defined. Methods can be added, removed, or redefined. method_missing intercepts calls to methods that don’t exist, allowing objects to respond dynamically to any message:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

class FlexibleProxy
  def method_missing(name, *args)
    if name.to_s.start_with?("find_by_")
      field = name.to_s.sub("find_by_", "")
      puts "Searching by #{field} with value #{args.first}"
    else
      super
    end
  end

  def respond_to_missing?(name, include_private = false)
    name.to_s.start_with?("find_by_") || super
  end
end

proxy = FlexibleProxy.new
proxy.find_by_email("[email protected]")  # Works, no method defined
proxy.find_by_name("Alice")              # Works too

This was the mechanism behind early Rails’ ActiveRecord magic: User.find_by_email("[email protected]") worked without any method named find_by_email ever being defined because method_missing caught the call and implemented the query dynamically. (Modern Rails replaced these underscore-based dynamic finders — deprecated in Rails 4 — with the explicit User.find_by(email: "[email protected]") syntax.) define_method creates methods from strings at runtime. class_eval opens a class and executes code in its context. instance_variable_set sets instance variables by name.

Ruby’s metaprogramming happens at runtime, in a live object model willing to be modified while the program runs. This enables patterns impossible in statically compiled languages, at the cost of performance and the ability to catch errors before execution.

Julia, the scientific computing language released in 2012, occupies a middle ground. Julia macros work like Lisp macros — they receive unevaluated code as an AST and return a new AST:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

macro assert_positive(expr)
    return quote
        value = $(esc(expr))
        if value <= 0
            error("Expected positive, got: $value")
        end
        value
    end
end

result = @assert_positive(sqrt(4.0))  # Passes
result = @assert_positive(-1.0)       # Error at runtime with context

Julia also has generated functions (@generated), which specialize their implementation at compile time based on argument types — similar to C++ template specialization but using ordinary Julia code rather than template syntax. And Julia is homoiconic in the Lisp sense: code is representable as data structures the language can manipulate.

The Spectrum

Stand back from the individual features and a clear axis emerges.

At one end: the C preprocessor. A separate program that knows nothing about C, operating on text before the compiler sees it, unable to understand types or scope or semantics. Powerful enough to enable widespread C library development. Dangerous enough to have caused decades of security vulnerabilities. The “meta-language” here is not really a language — it is find-and-replace with recursion.

One step in: C++ templates. Part of the language, type-aware, Turing-complete, but requiring a separate syntax and a separate mental model. The compile-time computation that happens in TMP looks nothing like the runtime C++ that surrounds it. Error messages from the boundary between these worlds are famously unreadable. constexpr represents the committee’s attempt to paper over this gap.

Further in: Rust’s procedural macros. Full Rust programs that run at compile time, operating on typed token streams rather than raw text. Type-aware. Checked by the compiler. Powerful enough to generate an entire serialization framework. Still a separate system — you write a procedural macro differently from ordinary Rust code, and it runs in a different context.

Lisp: the oldest design, still perhaps the cleanest. Code is data. Macros manipulate the actual syntax tree using the same language they produce code in. The separation between object-level code and meta-level code is syntactic, not conceptual.

At the other end: Zig’s comptime. No separate macro language. No template syntax. No preprocessor. The same language, evaluated at a different time. The programmer learns one thing and uses it everywhere. The compiler decides which invocations happen during compilation and which happen at runtime based on whether the inputs are known at compile time.

The direction of progress is not subtle. The oldest serious metaprogramming system (Lisp, 1958) operated on the language’s own syntax tree. The most recently designed system with significant adoption (Zig, 2016) eliminates the distinction between meta-language and language entirely. The decades between are the story of systems approaching that ideal from various angles — templates, procedural macros, CTFE — each iteration more integrated than the last.

What all the mature systems share, and what C’s preprocessor lacks, is type awareness. Every system after the preprocessor — templates, procedural macros, comptime — understands the type system it operates on. This is not coincidental. Metaprogramming that ignores types cannot offer correctness guarantees. It produces bugs that are invisible until runtime, for precisely the same reason C’s unchecked error codes do: nothing in the system has the information to notice.

The logical endpoint is Zig’s formulation: comptime and runtime are the same language. You write Zig. Some of it runs at compile time. Some runs in production. The distinction is when, not what. There is no meta-language to learn. There is just Zig, evaluated at different times.

Whether that is strictly better than Lisp’s homoiconic approach is a real question. Lisp macros can restructure the code they receive; Zig’s comptime executes functions but does not rewrite them. For most use cases, the distinction is academic. For the cases where LOOP and similar macro systems have historically shone, it matters. The debate between “powerful but dangerous” and “principled but constrained” has not been resolved — it has moved up a level, from the runtime object model to the compile-time meta-language.

Explore the Languages

Each language discussed in this post has full examples on CodeArchaeology:

• C — The preprocessor in context, alongside modern C
• Common Lisp — Macros, the REPL, and homoiconicity
• C++ — Templates, constexpr, and the evolution of compile-time code
• Rust — macro_rules!, procedural macros, and serde
• Zig — comptime and generics as functions
• Ruby — Runtime metaprogramming and the open object model
• Julia — Macros, generated functions, and scientific computing

Or browse our encyclopedia of 1,200+ programming languages to see how the full history of language design has wrestled with the question of code that generates code.

What’s the most useful — or most dangerous — metaprogramming pattern you’ve shipped in production? Open a discussion on GitHub.