SymbEL

SymbEL is an interpreted, C-like language whose entire reason for existing is stated in the first sentence of its own manual: it “was created to address the need for simplified access to data residing in the SunOS kernel.” It was written by Richard Pettit in 1993, it never ran anywhere except Solaris, and it was the language of the SE Toolkit - the freeware performance-monitoring kit that, for roughly the decade between Solaris 2.3 and the arrival of DTrace, was what Sun systems administrators reached for when the shipped tools stopped being enough.

Two things about the catalogue row that brings most readers here need correcting at the outset. SymbEL is not a symbolic language and has nothing to do with mathematics: there is no symbolic algebra in it, no term rewriting, no s-expressions, and its type system tops out at 64-bit integers, doubles and strings. And “1990s” can be tightened to 1993, which is both the first year in the copyright notice on every source file and the year of SunOS 5.2, the release the manual says the original interpreter was developed under.

The name is pronounced “symbol” - the manual says so, and then declines to expand the acronym.

The problem it was built for

The manual’s opening chapter is unusual in that it spends several pages arguing for the language’s existence before describing any of it, and the argument is worth following because it explains every design decision that comes after.

Solaris exposed kernel performance data through five distinct interfaces: kvm (raw kernel virtual memory, requiring root and /dev/kmem), kstat (the world-readable kernel statistics framework), mib (the MIB-II structures reached through STREAMS ioctls), ndd (network driver tunables) and procfs. Each had a C library. That meant that anyone who wanted a number the shipped vmstat did not print had to write, compile and ship a C program.

The alternative - shell scripts wrapping vmstat, iostat and sar - had a different problem, and the manual is precise about it. Such a script “impacts the performance of the machine under analysis enough to render the script useless as a ‘round the clock’ diagnostician.” Forking three utilities every few seconds to monitor a machine perturbs the thing being measured.

So the goal was a language that needed no compiler, produced scripts small enough to paste into an email, went straight to the kernel interfaces, and imposed as little load as possible while doing it. That last constraint shaped the language more than anything else: as the manual notes when explaining why local variables are initialised only once, “the overhead of maintaining automatic variables in an interpretive environment would be too high for the language to perform reasonably. One of the goals of ‘se’ is put as little load on the system as possible and still provide usable runtime performance.”

Active variables

The idea that makes SymbEL more than a small C interpreter is the active variable.

A variable declared with a language-class prefix in its name is not an ordinary variable. Every time one of its members is read, the runtime goes out to the corresponding kernel interface and refreshes it. There is no fetch call, no open, no handle:

1
2
3
4
5
6
7
8

#include <kstat.se>

main()
{
  ks_system_misc kstat$misc;

  printf("This machine has %u CPU(s) in it.\n", kstat$misc.ncpus);
}

The access to kstat$misc.ncpus inside the printf argument list is what triggers the read. Four language classes exist - kvm$, kstat$, mib$ and ndd$ - and the same trick handles iteration over device instances. Every multi-instance kstat structure carries two special members, number$ and name$, and the end of the list is signalled by number$ becoming -1, which makes device enumeration an ordinary for loop:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

#include <kstat.se>

main()
{
  ks_disks kstat$disk;

  printf("Disks currently seen by the system:\n");
  for(kstat$disk.number$=0; kstat$disk.number$ != -1; kstat$disk.number$++) {
    printf("\t%s\n", kstat$disk.name$);
  }
}

Setting number$ and then comparing it against -1 is what causes the runtime to check whether an nth disk exists; reading name$ is what causes it to be fetched. The loop reads like C and behaves like a query.

User-defined classes

The natural next step - and the one the manual spends its longest chapter on - is to let scripts define their own classes. A SymbEL class is a struct with a block of code attached, and that block runs whenever any member of an instance is accessed. The manual demonstrates it by taking a program that computes system uptime from kstat$misc.clk_intr inline, and folding the arithmetic into a class so that up.days, up.hours and up.minutes are simply there, recomputed on demand.

This is what let the toolkit ship things like vmstat_class.se, p_iostat_class.se, netif.se and mnt_class.se - 55 include files in the final release - as reusable derived measurements layered on top of the raw kernel data. A script author writing a disk monitor never touched kstat directly; they declared a variable of the appropriate class and read percentages and rates off it. It is a pull-based dataflow system built out of struct member access, and it is genuinely the language’s own idea.

There is a refresh$ builtin as an escape hatch for forcing an update, and the manual warns about it, because active variables that update themselves at unpredictable moments are exactly as confusing as they sound.

The language proper

Everything else is C with the sharp edges filed off, and the manual lists the removals frankly in a chapter titled “Pitfalls”:

What it keeps is the parts that make a C programmer immediately productive: if/else, switch, while, for, do, break, continue, the ternary operator, structs, arrays of structs, aggregate initialisation with C syntax, and compound assignment operators. It runs its own source through the C preprocessor, so #include and #define are the real thing. Its builtins are libc by another name - printf, fopen, fgets, popen, sprintf, strtok, qsort, bsearch, getopt, syslog. A main() is required.

There are a few additions that betray the domain. The comparison operators work on strings as well as numbers, so ("hello" == "world") is a valid expression that evaluates to false. There is a regular-expression match operator, =~. And there are dynamic constants - MAX_CPU, MAX_DISK, MAX_IF - filled in by the interpreter from the hardware it finds itself running on.

The most important extension is attach, added in release 2.1, which lets a script bind directly to functions in any shared object:

1
2
3
4

attach "libc.so" {
  int fclose(int fp);
  string getenv(string name);
};

That single mechanism is why release 2.1’s ChangeLog entry reads “Many builtins deleted in lieu of the new attachment mechanism”. Rather than growing the interpreter to cover every library anyone might want, Pettit made the interpreter able to call anything. The distribution’s 55 .se include files are, in large part, attach blocks mirroring the corresponding headers in /usr/include. It also means SymbEL can crash: the manual supplies a four-line script whose only statement is puts(nil) and notes that calling attached functions with wrong values “can result in a core dump and is not avoidable by the interpreter.”

History

Sun, but not Sun’s

The toolkit’s relationship with its birthplace was always slightly awkward, and the ChangeLog preserves the moment it was settled. Release 2.0 made se an installable Solaris package named SMCCse, after Sun Microsystems Computer Corporation. Release 2.4, on 12 June 1995, renamed it: “SMCCse now RICHPse so noone will think that SMCC supports it. They don’t.” The name RICHPse, and the install path /opt/RICHPse, survived to the last release in 2013.

The installation notes shipped with that release, signed by both authors, set the terms plainly:

This package is not a Sun supported product. Think of it as a vehicle for documenting the data sources, processing and behavioural rules of Solaris 2. All the real work is done in scripts, so you can fix, maintain and modify the package. If you want to incorporate the scripts in a product you should recode them in C first, so you do not depend on the unsupported SymbEL interpreter.

That framing - the toolkit as executable documentation for how Solaris performance actually works - is the fairest description of what SymbEL was for. Its distribution was as much a knowledge transfer exercise as a software release, which is why the rules were shipped as readable, editable, heavily commented source rather than compiled into a product.

The books and the column

SymbEL had an unusually strong publication channel for a language nobody outside one operating system used. Cockcroft’s Sun Performance and Tuning: SPARC and Solaris (SunSoft Press/Prentice Hall, 1995) supplied the ruleset that the toolkit’s ANCrules package implemented; the second edition, Sun Performance and Tuning: Java and the Internet (1998), added Pettit as co-author and carried the toolkit’s user manual as a chapter. Between and around them, Cockcroft’s monthly Performance Q&A column for SunWorld Online, which he wrote from 1995 onwards, repeatedly published SE scripts and explained the kernel data they read. For a period in the late 1990s, learning Solaris performance analysis and learning to read SymbEL were close to the same activity.

Second edition of the manual

The manual’s “Preface to the second edition” is one of the more honest pieces of writing in any language reference, and it is Pettit’s own assessment of where SymbEL sat:

It has been pointed out to me by several people that SymbEL represents “yet another language”. The problem that SymbEL solves is a specific one as shown by the applications written thus far. By no means does this language represent an attempt to provide a solution to any programming problem. Operator rich languages like Perl and Tcl provide ways to solve any problem with a minimum of typing. A medium level language such as Java can solve any problem with a little more effort. All of these languages are either source interpreted or i-code interpreted and have associated overhead and complexity depending on the problem set. Given the narrow focus of the problem set for SymbEL it’s overhead and complexity should also be narrow. I have endeavored to keep it that way and will continue to do so.

He also notes, ruefully, that the change between editions “came about from the use of the interpreter by so few people other than myself.”

DTrace

In January 2005, Sun shipped Solaris 10 with DTrace. Pettit’s last signed release, SE Toolkit 3.4, had gone out the month before, on 9 December 2004.

DTrace was better than SymbEL at SymbEL’s job in every respect that mattered: it instrumented the kernel dynamically rather than reading its variables from outside, it was designed to be safe on production systems, it was supported by the vendor, and it did not need setgid binaries to read a MIB counter. The SE Toolkit’s own README, written by its later maintainers, concedes the point and asks for the historical allowance:

The functionality is a subset of DTrace where kernel state can be inspected but not dynamically configured. Please keep in mind that the SE Toolkit started 10 years before DTrace.

Opening the source

The source had been closed for thirteen years - the final README states it plainly: “Starting with SE Toolkit 3.4.1 the sourcecode has been published under GPL. Earlier versions are provided in binary form only.” That was released under the GPL in 2006, per the AUTHORS file, and a SourceForge project was created on 15 February 2007 to host it. Dagobert Michelsen took over as maintainer, uploaded the historical binaries in February 2007, and published 3.4.1 in April 2007 as the first release with source.

What followed was a competent community maintenance phase rather than a revival. SE Toolkit 3.5.0, uploaded on 5 August 2008, made the interpreter work on x64 - 32-bit x86 had returned in 3.4 and amd64 support had first appeared in 3.4.1 - moved to autoconf/automake/libtool, made the grammar bison-compatible, added extended regular expressions and a test suite, converted the documentation to reStructuredText, and taught the kstat definitions about a decade of network hardware the language had missed. 3.5.1 followed, dated 12 February 2010, adding NFSv4 support and a few more drivers. The last commit to the repository is “Update to 3.5.2” on 29 March 2013, and 3.5.2 was never packaged.

The rewrites that were not SymbEL

Two successors exist, and neither is a SymbEL implementation. The project’s archive area holds two unreleased trees uploaded on 1 November 2008: se4.0, a portability rewrite whose version.c is stamped 6 October 2004 and carries build conditionals for Linux, HP-UX, AIX, Darwin, BSD and Cygwin across SPARC, x86, AMD64, PowerPC and PA-RISC, and “Caribou”, a C++ tree whose version.cpp announces itself as “Caribou - Version 5.0 (alpha)” and is dated 29 April 2008. Caribou carries the same operating-system conditionals and ships binary directories for i386 on Solaris, Linux, BSD and Cygwin, SPARC on Solaris, PowerPC on Darwin, and x86_64. Neither was ever released, and their presence in an “OldFiles” directory alongside the shipping versions is the clearest surviving evidence of how far Pettit intended to take the language off Solaris before stopping.

What he shipped instead was the XE Toolkit, a portable rewrite in Java sold through Captive Metrics, whose version 1.2 Cockcroft announced on his blog on 28 April 2008; that release added AIX (5.3), Linux on Power and Linux on s390 to the platforms XE already covered. XE reimplemented the toolkit’s purpose in a language that already ran everywhere. SymbEL did not come with it.

Why it matters

SymbEL is a good example of a language built for exactly one job and abandoned the moment the platform absorbed that job into itself. It is not a general-purpose language and never pretended to be; Pettit said as much in his own preface. Judged on its own terms, it did something specific and well: it took five awkward C APIs and made them look like ordinary variable reads, and by doing so it moved Solaris performance analysis out of compiled tools and into scripts that could be read, argued with, customised and mailed around.

The active variable is the idea worth remembering. Wiring a data source to a variable so that reading the variable performs the query is a pattern that turns up repeatedly - in reactive frameworks, in observability query languages, in ORM lazy loading - and SymbEL implemented it in 1993, in C syntax, against a Unix kernel, purely so that a monitoring script would not have to say kstat_read().

Its second legacy is documentary. The toolkit was distributed as source specifically so that its rules would be legible, and the result is that virtual_adrian.se and the kstat header files preserve, in executable form, a detailed record of what Solaris performance engineers in the 1990s actually measured and what thresholds they considered alarming. Very little software of that era left behind a comparable artefact.

Today SymbEL is a dead language on a fading platform: no released version builds anywhere but Solaris, its last release is from 2010, its last commit from 2013, and DTrace replaced it more than twenty years ago. But orcallator.se kept feeding Orca graphs on Solaris fleets long after anyone was writing new SymbEL, and the language’s central trick - the variable that fetches itself - was a good one, arrived at early, by someone who needed it to work.