coNCePTuaL

coNCePTuaL — the capitalization spells out Network Correctness and Performance Testing Language — is a domain-specific language for writing network benchmarks. Instead of a general-purpose language with a messaging library bolted on, it offers a keyword-heavy, almost English-language syntax in which a complete ping-pong latency test, command-line parsing and statistics included, fits in under twenty lines. Those lines are then compiled into a real, efficient C+MPI program. It was created by Scott Pakin at Los Alamos National Laboratory in 2003, and it remains one of the clearest attempts to treat benchmark description as a language design problem rather than a documentation problem.

History and Origins

Pakin’s motivating observation, laid out in the language’s own user’s guide, is that network researchers reinvent the same suite of performance tests over and over, and that the resulting numbers are not comparable. Consider something as apparently simple as a bandwidth test. Does it run for a fixed number of iterations or a fixed length of time? Is bandwidth measured as ping-pong bandwidth or as unidirectional throughput with a single acknowledgement at the end? Does the acknowledgement’s length count toward the total? How many warmup messages precede the timing loop, and is there a pause afterward to let the network reclaim scarce resources? Are receives blocking or nonblocking?

Every one of those choices moves the reported number, and almost none of them appear in the paper that reports it. Pakin named the underlying problem benchmark opacity: the practical impossibility of presenting performance measurements in a way that lets someone else reproduce them or evaluate them independently. coNCePTuaL’s answer is that if the benchmark’s full specification is short enough to print in the paper itself, the reader can simply read what was measured.

The language emerged in 2003 — its source files carry 2003 copyright notices, it is registered internally at LANL as LA-CC-03-099, and the first user’s guide was issued as technical report LA-UR 03-7356. Pakin presented it at IPDPS 2004 in Santa Fe, followed the same year with a Euro-Par paper on reproducibility, and gave the definitive treatment of the compiler in IEEE Transactions on Parallel and Distributed Systems in 2007. The project was published to GitHub under the lanl organization in 2011, where it remains available under a BSD-style license.

Design Philosophy

Three commitments shape the language.

Read like a description, not like code. coNCePTuaL statements are sentences. task 0 sends a msgsize byte message to task 1 is a statement; all tasks await completion is a statement. Statements are chained with then to impose ordering and with and to run concurrently. A reader who has never seen the language can generally follow a program on first reading, which is exactly the property a printed benchmark specification needs.

Cover what benchmark writers forget. Because the language is special-purpose, its runtime can do work that hand-written tests routinely skip: recording the environment the run took place in, computing descriptive statistics, aborting a job that overruns a time limit, and emitting data in formats that plotting programs can consume directly.

Do not misattribute overhead. The hard part of compiling such a high-level language, as the TPDS paper frames it, is that the generated code has to be extremely efficient — any overhead the compiler introduces gets charged to the messaging library under test — while the language itself must not be dumbed down to make the compiler’s job easier.

A Complete Program

This is latency.ncptl, shipped with the distribution — a full ping-pong latency test:

# A ping-pong latency test written in coNCePTuaL

Require language version "1.5".

# Parse the command line.
reps is "Number of repetitions of each message size" and comes from
 "--reps" or "-r" with default 1000.
maxbytes is "Maximum number of bytes to transmit" and comes from
 "--maxbytes" or "-m" with default 1M.

# Ensure that we have a peer with whom to communicate.
Assert that "the latency test requires at least two tasks" with num_tasks>=2.

# Perform the benchmark.
For each msgsize in {0}, {1, 2, 4, ..., maxbytes} {
  for reps repetitions {
    task 0 resets its counters then
    task 0 sends a msgsize byte message to task 1 then
    task 1 sends a msgsize byte message to task 0 then
    task 0 logs the msgsize as "Bytes" and
                the median of elapsed_usecs/2 as "1/2 RTT (usecs)"
  } then
  task 0 computes aggregates
}

Note how much is handled declaratively. The comes from clauses generate command-line option parsing and the --help text. The set notation {1, 2, 4, ..., maxbytes} expresses a geometric sweep. the median of elapsed_usecs/2 names a reduction over the repetition loop, so the aggregation is part of the specification rather than a downstream script. Assert that produces a clear failure message when the job is launched with too few tasks.

Verification programs are equally compact. verifyall.ncptl fills messages with a known pattern using the with verification modifier and logs bit_errors, turning the same language into a fabric-integrity test.

Key Features

• Task-set expressions. Statements are addressed to task sets — task 0, all tasks, all tasks src, task (src+ofs) mod num_tasks — so all-to-all and shifted-ring patterns are one line each.
• Ordering operators. then sequences; and runs concurrently. Data dependencies do not have to be inferred by the reader.
• Message attributes. Messages can be declared synchronous or asynchronous, page-aligned or misaligned, and verified — the alignment attributes are what made the 2005 buffer-alignment study practical.
• Built-in measurement vocabulary. Counters, timers, elapsed_usecs, bit_errors, computes aggregates, and named log columns are language-level constructs, not library calls.
• Simulated computation and I/O. Alongside communication, programs can express compute delays and file I/O, so a benchmark can approximate a real application’s mix rather than only its message traffic.

Compiler Backends

The compiler is written in Python and is deliberately modular; version 1.5.1b ships twelve backends:

The interpret backend is worth dwelling on: because it can simulate an arbitrary task count on one machine and detect deadlocks and mismatched operations, a benchmark can be debugged before it ever touches a supercomputer’s job queue. The visualization backends serve the same reproducibility goal from the other direction — a communication pattern that can be drawn is a pattern that can be checked by eye.

The distribution also includes a Java GUI for assembling programs from dialog-driven building blocks, and a C runtime library (libncptl) that provides the timing, logging, memory-touching, and statistics machinery the generated code calls into.

Self-Documenting Log Files

The log format is where the reproducibility argument is cashed out. Log files are plain text with comma-separated columns, and the header comments record an unusually complete picture of the run: coNCePTuaL version and backend, executable name, working directory, command line, task count, host name, operating system version, CPU vendor/architecture/count/frequency, cycle-counter frequency, page size, physical memory, the exact configure line coNCePTuaL was built with, the compiler and its version and flags, the dynamic libraries actually loaded, the timer type in use, and calibration measurements of the timer’s overhead, granularity, and error — with explicit warnings when the platform’s sleep or process timers behave poorly. Environment variables are dumped too.

Companion tools round this out: ncptl-logextract converts log data into other formats, while ncptl-logmerge and ncptl-logunmerge combine and separate logs for comparison across runs.

Limitations

The user’s guide is candid about the boundary. coNCePTuaL expresses race-free communication patterns, and it cannot express data-dependent communication — a master/worker pattern where the master responds to whichever worker happens to report first is outside the language, because the behavior depends on message arrival order. For the same reason it cannot use runtime measurements to steer itself, so a benchmark that repeats until the standard error of a metric falls below a threshold cannot be written. The guide notes these “may be lifted in a future release”; they were not.

Evolution and Current Status

Development proceeded steadily through the 2000s — the sample log file in the manual is from a 0.6.4a-era run — reaching the 1.x series by the end of the decade. The public tag history runs 1.2 (2009), 1.3 (2011), 1.4 (2012), 1.5 (2014), 1.5.1 (2015), and 1.5.1b, tagged 4 March 2019. The bundled user’s guide carries the same 1.5.1b stamp and a 4 March 2019 date, and no further changes have been pushed to the repository since.

The project is therefore dormant rather than abandoned: the source is public on GitHub under the lanl organization, the copyright now sits with Triad National Security, LLC (which took over LANL’s management in 2018), and the BSD-style license permits redistribution and modification. Building it follows the ordinary Unix ./configure && make && make install path, with make check available to verify the build.

Why It Matters

coNCePTuaL’s importance is out of proportion to its user count. It made an argument — that a benchmark is a specification, and specifications belong in languages — and then demonstrated it end to end: a syntax terse enough to print in a paper, a compiler careful enough not to pollute the measurement, a log format that captures the experimental conditions nobody remembers to write down, and a typesetting package so the program can go straight into the publication.

That argument has aged well. The reproducibility problems Pakin catalogued in 2004 are the same ones that motivated later work on rigorous HPC benchmarking practice, and the design lives on indirectly through work like Union, which translates coNCePTuaL programs into workload skeletons for the CODES network simulator. For anyone studying domain-specific language design, it is a compact case study in what a DSL buys you: not just less typing, but a runtime that can be responsible for the parts of the problem the language’s users kept getting wrong.

Learn More

• lanl/coNCePTuaL on GitHub — source, examples, and the bundled user’s guide (doc/conceptual.pdf)
• Pakin, S. “coNCePTuaL: A Network Correctness and Performance Testing Language.” IPDPS 2004. DOI: 10.1109/IPDPS.2004.1303014
• Pakin, S. “The Design and Implementation of a Domain-Specific Language for Network Performance Testing.” IEEE TPDS 18(10), 2007, pp. 1436-1449. DOI: 10.1109/TPDS.2007.1065
• Pakin, S. “Reproducible Network Benchmarks with coNCePTuaL.” Euro-Par 2004, pp. 64-71
• Arber, L. and Pakin, S. “The Impact of Message-buffer Alignment on Communication Performance.” Parallel Processing Letters 15(1-2), 2005, pp. 49-66
• Wang, X., Mubarak, M., Kang, Y., Ross, R. B., and Lan, Z. “Union: An Automatic Workload Manager for Accelerating Network Simulation.” IPDPS 2020 — translates coNCePTuaL programs into CODES simulation workloads