NEWLIB

NEWLIB was the program-management system and command language that DESY, the German particle-physics laboratory in Hamburg, built for its IBM mainframes. It sat on top of IBM’s TSO (Time Sharing Option). Through it, physicists and programmers could keep their source code and program systems in libraries, edit them on a terminal, link them and submit jobs. Judging by the one program that survives, its procedures were written in a small procedural language with variables, loops and conditionals. NEWLIB was in use by 1975, and DESY’s own history credits it with a full-screen terminal interface in 1977, at a time when TSO itself still worked one line at a time. It was still being documented in 1988 and still being programmed in 1996. Then it disappeared with the mainframe it ran on. Its manuals survive only as catalogue entries.

What the record says

The encyclopedia catalog that lists NEWLIB records the following:

History & Origins

Terminals arrive at DESY

In the early 1970s, DESY’s computer centre ran like most scientific computing centres. Programs arrived as decks of punched cards and output came back on paper. DESY’s account of its own IT history marks TSO as the turning point. With TSO, program writing, job preparation and viewing the results could all be done at a terminal. The first four display terminals went into the computer centre in 1975, and by 1976 there were nineteen. The history notes that these screens could at first only be used for alphanumeric work. It also notes that it took several more years before punched cards were dropped altogether.

TSO gave a user a command line, not a working environment. Keeping track of hundreds of source members, the libraries they were compiled into and the JCL needed to build and run them was still the user’s job. NEWLIB was DESY’s answer to that problem.

The NEWLIB reports

The INSPIRE-HEP database catalogues four DESY internal reports about NEWLIB:

The first three are recorded as German-language reports, so the English titles are probably the catalogue’s translations. Two details matter. First, the 1975 report is already called a new version, so some form of NEWLIB existed before November 1975. No record of the original has been found. Second, the reference manual grew from 35 pages in 1975 to 508 pages by 1988. That says something about how much the system grew over its life, though without the manuals the details are unknown.

None of the four reports has been scanned. DESY’s library publishes scans of many internal reports, but the R-series reports that hold the NEWLIB documentation are not among them. Their contents could therefore not be checked for this article.

Full screen, 1977

DESY’s IT history gives NEWLIB a short but emphatic paragraph. It calls NEWLIB a “pioneering development” (bahnbrechende Entwicklung) of 1977, built jointly by the R1 and R2 groups as software for the terminal user interface. Its key advance was that the whole screen, not just a single line, could be used as an input form. The history claims that DESY was a world leader in this technology for a long time and gave important impulses to its further development. It adds that DESY’s director general demonstrated NEWLIB to the then head of IBM during a visit to DESY in 1977/78.

Those claims come from DESY itself and cannot be independently checked here. The 1977 date also does not conflict with the 1975 report. The most likely reading is that NEWLIB began as a library-management program and gained its full-screen interface around 1977, but the sources do not say this outright. In the same year, DESY’s R1 group also issued a twelve-page report by D. Mönkemeyer titled FSP: Full Screen Programming (R1-77-02). Whether it describes part of NEWLIB is not recorded.

Design Philosophy

The catalogue titles describe NEWLIB as a program for the administration of programs and programming systems. That puts the emphasis on managing software, not on computation. A physicist’s analysis program at a laboratory like DESY was rarely a single file. It was a collection of FORTRAN routines, shared libraries from the experiment and the computer centre, and the job-control statements needed to compile, link and run them. NEWLIB appears to have bundled those chores into one environment that worked on a terminal.

The 99 Bottles of Beer entry describes it as “a programming aid that operates as a command processor in the Time Sharing Option (TSO) environment”. In IBM terms, a command processor is a program that TSO starts when a user types a command. NEWLIB was therefore not a separate operating environment like Wylbur. It ran inside a TSO session and took over the terminal. Its command language appears to have let users automate their own sequences of NEWLIB and system actions, much as IBM’s own CLIST language did for TSO commands, though without the manuals this is an inference.

Key Features

The surviving program

The only NEWLIB source code known to be public is a 99 Bottles of Beer program by R. P. Hofmann of DESY. The comment header gives his DESY IBM mail address, and the library-member header line is dated 21/02/96. It was posted to 99-bottles-of-beer.net as entry 424, dated 04/20/05, under the site’s “real language” category. The submitter is listed as anonymous.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
$   21/02/96 602211216  MEMBER NAME  <<EDTCL  (SPECTRUM)    CLIST
$ NEWLIB VERSION OF 99 BOTTLES OF BEER
$ R.P.HOFMANN [email protected]
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
STORE
ESC := '%'
DO N0 = 1 TO 99
  N1 := 100-%N0
  IF N1 > 1 THEN
    IF N0 > 1  SHOW N1||' BOTTLES OF BEER ON THE WALL'
    SHOW N1||' BOTTLES OF BEER ON THE WALL'
    SHOW N1||' BOTTLES OF BEER'
  ELSE
    SHOW '1 BOTTLE OF BEER ON THE WALL'
    SHOW '1 BOTTLE OF BEER ON THE WALL'
    SHOW '1 BOTTLE OF BEER'
  ENDIF
  SHOW 'TAKE 1 DOWN AND PASS IT ALL AROUND'
END
SHOW '0 BOTTLES OF BEER ON THE WALL'
RESTORE

NEWLIB “99 Bottles of Beer” by R. P. Hofmann (1996), from 99-bottles-of-beer.net entry 424 as preserved by the Wayback Machine.

What the program shows

No implementation survives, so this program could not be run. The readings below are inferences from the code, not documented behaviour.

The loop is arranged so that each pass first prints the last line of the previous verse (skipped on the first pass by IF N0 > 1) and then the opening of the current one. The text differs from the usual song in small ways: “take 1 down and pass it all around”, and a final “0 bottles” rather than “no more”. It reads as a quick demonstration of the control structures rather than a polished program.

The PL/I connection

The 99 Bottles page says NEWLIB is “mostly based on PL/I, therefore the similarity”. The program bears that out on the surface: DO ... TO, END, || and IF ... THEN ... ELSE all match PL/I, which was IBM’s own general-purpose language and was widely used on IBM mainframes. This is the claim of one anonymous submitter, not a statement from the NEWLIB documentation, so this page does not list PL/I as a documented influence. The comment block also labels the member as a CLIST, the name of IBM’s TSO command-procedure language. That may only describe the member’s type in the library, since the syntax is not CLIST syntax.

Commands from inside the environment

A 1988 DESY diploma thesis on the data link between the HERA beam-orbit processor and DESY’s IBM gives one more glimpse of NEWLIB in use. It describes an IBM-side ONLINE command that lists every front-end module connected to the IBM, with the online job and user module currently running for each. The thesis says the command can be executed on the IBM “for example under Newlib” by typing a character string that the scanned text renders as ((ONLINE)). The scan is of poor quality, so the exact syntax is uncertain. It does suggest that NEWLIB had a special notation for passing commands through to other programs.

Evolution

What can be said about NEWLIB’s evolution comes from its documentation dates:

• Before November 1975: an original version, never catalogued.
• 1975-1976: two revisions, each a “new version”, documented by H. Butenschön in about 35-40 pages.
• Around 1977: the full-screen interface that DESY’s IT history highlights.
• 1978: a tutorial introduction for users, by K. Dahlmann.
• 1988: a 508-page reference manual by Butenschön and Sommer, more than ten times the size of the 1975 document.
• 1996: still in use at DESY, to judge by the date in the 99 Bottles program’s member header.

When NEWLIB was retired is not recorded. It was tied to DESY’s IBM TSO service and would have ended with it at the latest, but no date for either has been confirmed for this article.

Current Relevance

NEWLIB has no known surviving implementation, users or online documentation. Its manuals exist as DESY internal reports catalogued in INSPIRE-HEP, but they are not scanned. DESY’s library also had no scan under the obvious file names. The 99 Bottles of Beer site that held the only public program no longer resolves, and entry 424 survives in the Wayback Machine.

The KEK connection given on the 99 Bottles page could not be confirmed. KEK, in Tsukuba, Japan, is a high-energy physics laboratory with long-standing ties to DESY, but no KEK report or user guide mentioning NEWLIB was found, so the claim rests on the 99 Bottles page alone.

Not to be confused with

Why It Matters

NEWLIB is a good example of a kind of software that rarely reaches language histories: the in-house environment a large scientific computing centre wrote for itself. When interactive terminals replaced punched cards in the mid-1970s, the tools that IBM supplied were basic. Laboratories such as DESY wrote their own to fill the gap. DESY’s own history calls NEWLIB a pioneering development and says the lab was long a world leader in full-screen terminal work, and its documentation grew to more than 500 pages.

Yet almost nothing about it is publicly readable today. The manuals sit in a laboratory archive. Outside that archive, the language survives in one short program written for fun in 1996, two passing references in physics reports, and a paragraph on a DESY web page. If the R-88-01 reference manual is ever scanned, it will be the main source on NEWLIB. Until then, this page records what the surviving traces show and where each of them comes from.

Sources

• DESY IT, “Wir über uns” history page (German, English): TSO and the first terminals (1975-1976), Newlib (1977), the IBM visit (1977/78)
• INSPIRE-HEP catalogue records for DESY reports R2-75/2, R2-76-01, R1-78-01 and R-88-01
• DESY internal report F1-82/01, The Input/Output Software for the 370/E (August 1982), scan at DESY library
• T. Woeniger, Entwicklung einer VME-PADAC Schnittstelle zum Transport von Strahl-Simulationsdaten des HERA Speicherringes, DESY F35D-88-02 (February 1988), INSPIRE record
• 99-bottles-of-beer.net, “Language NEWLIB”, entry 424, via the Wayback Machine, June 2006 capture