The Evolution of C Programming Practices:
A Study of the Unix Operating System 1973-2015

Diomidis Spinellis
Panos Louridas
Maria Kechagia

Department of Management Science and Technology
Athens University of Economics and Business
Patision 76, GR-104 34 Athens, Greece

Abstract

Tracking long-term progress in engineering and applied science allows us to take stock of things we have achieved, appreciate the factors that led to them, and set realistic goals for where we want to go. We formulate seven hypotheses associated with the long term evolution of C programming in the Unix operating system, and examine them by extracting, aggregating, and synthesising metrics from 66 snapshots obtained from a synthetic software configuration management repository covering a period of four decades. We found that over the years developers of the Unix operating system appear to have evolved their coding style in tandem with advancements in hardware technology, promoted modularity to tame rising complexity, adopted valuable new language features, allowed compilers to allocate registers on their behalf, and reached broad agreement regarding code formatting. The progress we have observed appears to be slowing or even reversing prompting the need for new sources of innovation to be discovered and followed.

CCS Concepts

• Software and its engineering → Software evolution; Imperative languages; Software creation and management; Open source model; • General and reference → Empirical studies; Measurement; • Social and professional topics → Software maintenance; History of software;

Keywords

C; coding style; coding practices; Unix; BSD; FreeBSD

1  Introduction

Tracking long-term progress in engineering and applied science allows us to take stock of things we have achieved, appreciate the factors that led to them, and set realistic goals for where we want to go. Progress can be tracked along two orthogonal axes. We can look at the processes (inputs) or at the resulting artefacts (outputs). Furthermore, we can examine both using either qualitative or quantitative means. The objective of this work is to study the long term evolution of C programming in the context of the Unix operating system development. The practice of programming is affected by tools, languages, ergonomics, guidelines, processing power, conventions, as well as business and societal trends and developments. Specific factors that can drive long term progress in programming practices include the affordances and constraints of computer architecture, programming languages, development frameworks, compiler technology, the ergonomics of interfacing devices, programming guidelines, processing memory and speed, and social conventions. These might allow, among other things, the more liberal use of memory, the improved use of types, the avoidance of micro-optimisations, the writing of more descriptive code, the choice of appropriate encapsulation mechanisms, and the convergence toward a common coding style. Here are a few examples. The gradual replacement of clunky teletypewriters with addressable-cursor visual display terminals in the 1970s may have promoted the use of longer, more descriptive identifiers and comments. Compilers using sophisticated graph colouring algorithms for register allocation and spilling [12] may have made it unnecessary to allocate registers in the source code by hand. The realisation that the overuse of the goto statement can lead to spaghetti code [13] might have discouraged its use. Similarly, one might hope that the recognition of the complexity and problems associated with the (mis)use of the C preprocessor [48,34,15,49] may have led to a reduced and more disciplined application of its facilities. Also, one would expect that the introduction and standardisation of new language features [45,2,23] would lead to their adoption by practitioners. Finally, the formation of strong developer communities, the maturing of the field, and improved communication facilities may lead to a convergence on code style. In more formal terms, based on a simple-regression exploratory study [54], we established the following hypotheses, which we then proceeded to test with our data. H1: Programming practices reflect technology affordances If screen resolutions rise we expect developers to become more liberal with their use of screen space, as they are no longer constrained to use shorter identifiers and shorter lines. Higher communication bandwidth (think of the progress from a 110 bps ASR-33 teletypewriter, to a 9600 bps VT-100 character addressable terminal, to a 10MB Ethernet-connected bitmap-screen workstation) makes typing more responsive and the refresh of large code bodies faster. Similarly, if barriers imposed to compilation unit sizes by the lack of memory or processing capacity are removed, we expect developers to abandon artificially-imposed file size limits, and move towards longer files packed with more functionality. H2: Modularity increases with code size As the Unix source code evolves and multiplies in size and complexity we expect developers to manage this growth through mechanisms that promote modularity. These include the use of the static keyword to limit the visibility of globally visible identifiers, and the pairing of header files with C implementation into discrete modules. H3: New language features are increasingly used to saturation point The C language has been changing over time with new features being added, albeit sparingly. We expect them to be increasingly used after being introduced, up to a point - certainly, a language feature can only be used a certain number of times in a program. For example, when the unsigned keyword was introduced, it could in theory, be used at most on all integer definitions and declarations of a program, and in practice only on those where the underlying value was indeed non-negative. We call this limit the feature's saturation point. H4: Programmers trust the compiler for register allocation In the days of yore, programmers had to deal themselves with the nitty-gritty details of register allocation by declaring variables with the register keyword. As compiler technology has improved, compilers have become more adept at the task. We expect that developers have been noticing this and have therefore been more and more trusting of the compilers with handling register allocation optimizations. H5: Code formatting practices converge to a common standard We expect developers working on a single project, such as Unix, to adopt a common coding standard, making the code base more homogeneous over time. This can be aided through the increased availability and use of collaboration mechanisms such as version control systems and online communities. H6: Software complexity evolution follows self correction feedback mechanisms We expect that as software evolves it becomes more complex, to a degree. A successful project cannot grow in complexity beyond the confines of human comprehension. Therefore we also expect that beyond a certain point self correction feedback mechanisms should kick in, bringing code complexity down. H7: Code readability increases As software and its development process and community evolve, growing in size and in complexity (to a degree), we expect its readability to increase, in order to make it manageable by its evolving community of developers. In addition, a long lived project will outlive its original developer cast and accommodations must be made for new cohorts. The preceding hypotheses can be tested by examining instances of the code over time. By looking at the long term evolution of associated metrics we can determine whether they indeed change over time, as well as the direction and rate of change. The results of the study on long term evolution of programming practices can be used to allocate the investment of effort in areas where progress has been efficiently achieved, and to look for new ways to tackle problems in areas showing a lack of significant progress. Also, given the hypothesis that the structure and internal quality attributes of a working, non-trivial software artefact will represent first and foremost the engineering requirements of its construction [51], the results can also indicate areas where developers rationally allocated improvement effort and areas where developers did not see a reason to invest. This paper builds on the earlier short exploratory study [54] by explicitly stating and testing hypotheses, by using a more sophisticated statistical method for the analysis, by a detailed explanation of the employed methods, and by presenting possible causality links between external changes, the metrics, and the hypotheses. To formulate the hypotheses we took into account the general technological trends, such as improved screen technologies and software techniques, combined with metrics, which we could measure from our data and could bear on the trends. In the following sections we describe the methods of our study (Section 2), we present and discuss the results we obtained (Section 3), and outline related work (Section 4). Section 5 summarises our conclusions and provides directions for further work.

2  Methods

Our study is based on a synthetic software configuration management repository tracking the long term evolution of the Unix operating system. At successive time points of significant releases we process the C source code files with a custom-developed tool to extract a variety of metrics for each file. We then synthesise these metrics into weighted values that are related to our hypotheses, and analyse the results over time using established statistical techniques.

2.1  Data Collection and Primary Metrics

The analysis of the Unix source code over the long term was made possible by the fact that important Unix material of historical importance has survived and is nowadays openly available. Although Unix was initially distributed with relatively restrictive licenses, significant parts of its early development have been released by one of its rights-holders (Caldera International) under a liberal license. Additional parts were developed or released as open source software by (among others) the University of California, Berkeley and the FreeBSD Project. The primary sources of the material used in this study include source code snapshots of early released versions, which were obtained from the Unix Heritage Society archive, the CD-ROM images containing the full source archives of Berkeley's Computer Science Research Group (CSRG), the OldLinux site, and the FreeBSD archive. These snapshots were merged with past and current repositories, namely the CSRG SCCS [46] repository, the FreeBSD 1 CVS repository, and the Git mirror of modern FreeBSD development. This material plus results of primary research regarding authorship and genealogy formed the basis for constructing a synthetic Git repository, which allows the efficient retrieval and processing of the Unix source code covering a period of 44 years [52]. Due to the fact that early releases are only available as snapshots, it was decided to study the code at points of significant software releases, rather than at fixed time intervals. This was done because obtaining code snapshots at fixed time intervals was only possible after 1990, when all software began to be tracked through various revision control systems. The 66 releases examined, the mean date of the files comprising them, and the size of C-proper source code files are listed in Table 1. We did not include header files in our study, to avoid skewing metrics associated with executable code (such as the number of functions per file) with results from files that typically do not include any such code. Interestingly, over the examined period the code body grew by more than three orders of magnitude, from six thousand to ten million lines. A few earlier revisions that existed in the repository were not examined, because they did not contain any C source code files. We show and use each release's mean file date - based on averaging each file's last modification time - rather than the release date, because this reflects better the age of the corresponding code base. This also avoids the distortion that would be introduced by treating what were sometimes parallel lines of development as a linear sequence. Such parallel development took place during the Berkeley Unix evolution: over the early years with Research Unix and over the late years with 386BSD and FreeBSD. Obtaining metrics from large code bodies is difficult for technical and operational reasons [39,20]. On the technical side, code dependencies make it difficult to establish the full context needed in order to parse and semantically analyse the code. This is especially true for C code, where the compilation depends on system header files, compiler-defined macros, search paths, and compile-time flags passed through the build process [49,33,18]. The operational reasons are associated with the required throughput, though due to the relatively small number of releases we examined, this was not a major issue in this study. We addressed the difficulty of parsing C source code without access to the original compilation environment by extending and using our cmcalc¹ open source tool, which efficiently calculates through static analysis a variety of C code quality metrics, without requiring full access to the compilation environment's parameters. The tool's operation is based on state machine logic (described in reference [53]), and will therefore produce reasonably accurate results without requiring access to header files and the like. Extensive unit tests (48% of the source code is devoted to them) were used to verify the tool's operation in diverse corner cases. The cmcalc tool calculates size, language feature, code style, and commenting metrics; see references [25], [50,pp. 326-333] for more details. The metrics we used are the following. Size metrics: number of characters, lines, statements, functions, identifiers. Language features: declarations with internal linkage; number of C preprocessor directives; number of C preprocessor include directives; number of C preprocessor conditional directives (#if, #ifdef, #elif); number of const, enum, goto, inline, noalias, register, restrict, signed, struct, union, unsigned, void, and volatile keywords. Code style: mean identifier length; spaces used for indentation and their standard deviation; mean nesting level; code style infractions. Commenting: number of comments and comment characters. In addition, we derived a sloppiness metric (DKLUDGE) by using fgrep to count in the code the number of "kludge" words that may indicate problems in the code (FIXME, XXX, TODO, BUGBUG, and swearwords that cannot be reproduced in this paper). The cmcalc tool calculates code style infractions from commonly agreed formatting guidelines. As there are a number of different approaches for formatting C code, cmcalc also measures the consistency of their application. Specifically, for each way to format a particular construct (for example putting a space after the while keyword) cmcalc measures the times, a, the rule is applied in the one way (e.g., putting a space) and the times, b, the rule is applied in the other way (omitting the space). Then, the file's style inconsistency for n style rules (19 in our case) as a percentage of possible cases is defined as follows.

SI = n ∑ i=1  min (ai, bi) n ∑ i=1  ai + bi

(1)

We based the checked style rules on the Google,² FreeBSD,³ GNU,⁴ and the updated Indian Hill⁵ guidelines. We collected the metrics by iterating through the 66 releases listed in Table 1, checking out the code for each one of them, and running cmcalc on all C files of that release. Through this process we collected 490 thousand records containing in total 56 million values. The processing took about five hours on an eight-core server. A considerable speedup was achieved by parallelising the analysis of each file through the use of GNU parallel [57].

2.2  Derived Metrics

We aggregated the raw metrics at the level of each release, and calculated weighted derivative values needed to track the long term evolution of programming practices. The derived aggregate metrics are listed in Table 2. Column-wise, the metrics are roughly clustered into those that are affected () by programming language evolution, ergonomics (workstation technology and screen resolution), programming guidelines, processing capacity (CPU performance and RAM size), conventions, and tools. Further columns on the right indicate the hypothesis that each derived metric affects (). To control for size and thus make metrics comparable across releases, absolute numbers in the primary collected metrics were replaced in the derived metrics by corresponding density figures. The denominator for calculating each density is based on the numerator unit, and can be the corresponding number of files, lines, characters, statements, or identifiers. As an example, the comment character density is calculated as the fraction of comment characters over the number of characters across all source code files. We expect that metrics that are not density figures (FILINE, FISTMT, LICHAR, FULINE, IDCHAR, STNEST, and CMCHAR) are implicitly controlled for size through coding practice and convention. For example, when a function or a file increases in too much in size, developers will split it into smaller modules. The following paragraphs outline how metrics are associated with the factors that may influence them. The analysis follows roughly the order in which the metrics are listed in Table 2. The length of files and the corresponding functionality they provide can be decided to promote proper encapsulation and modularity. However, on resource constrained computers - think of a 128kB PDP-11 - large files could take overly long to compile, forcing developers to split them in order to minimise the impact of small changes on build performance. As an example, the 7th Research Edition implementation of the refer program seems to be arbitrarily split into nine files named refer0.c ... refer8.c. Programming guidelines also typically dictate line length (lines should not exceed the number of characters that can fit in a display row), function length (functions should fit on a screen), identifier length (most names should be descriptive but not overly long [8]), and statement nesting (deep nesting should be avoided). On the other hand, all these are also affected by ergonomics. The lengths to which past developers went to avoid long identifiers survives to this day through the Unix creat (rather than create) system call. Also note that 1970s linkers only recognized the first six characters of identifiers, thus restricting the length of globally-visible identifiers. Higher resolution screens can display more characters per row, more rows on a screen, and thus allow for deeper nesting, while fast workstations make it easy to type and display long identifiers. However, long functions and corresponding deep nesting may be required on slower CPUs in order to avoid the overhead of function calls and the danger of hitting the limits of a shallow stack size. Also, IDEs with code completion can help entering long identifiers, while optimising compilers can avoid the function call cost through inlining. The declaration of identifiers that are only internally visible within the compilation unit (static), depends on the existence of the corresponding language feature. The use of such declarations to limit the identifiers' global visibility is prescribed in guidelines, and can be assisted by tools that identify such problems. The use of several keywords (const, enum, unsigned, inline, noalias, restrict, signed, register, void, volatile) is made possible through their introduction as language features and corresponding compiler support. In addition, the use of some (const, enum, signed, unsigned) can clarify the programmer's intent and avoid some errors. Furthermore, when these four keywords are used, static analysis methods can identify common error cases. On the other hand, the register and inline keywords are there to address deficiencies in the way compilers allocate registers [12] and inline functions, so their use should become less common as technology advances. The noalias keyword was a mistaken attempt by the ANSI C standardization committee to handle aliasing through pointers. We expect it to make an appearance for a brief (if any) period of time. The volatile and restrict keywords control the caching of values and express programmer intent regarding aliasing for cases that the compiler could not detect. We expect them to be used sparingly, with volatile used for only a small fixed number of values. The goto statement has been with us from the dawn of C and has been "considered harmful" for almost half a century [13]. The formatting style and the number of spaces used for indentation are a matter of convention. This can be established when code is collaboratively edited through version control systems and shared through online communities and as open source software. Consistency in both areas can also be aided through tools, such as code formatters, editors, and IDEs. Spacing and the density of comments and statements, are also a matter of guidelines and ergonomics. The level of nesting and available screen real estate can affect the number of spaces used for indentation. Comments should be long and plentiful, while white space among statements should be used to separate code into logical blocks. Fast high-resolution workstations make it easy to type in and display such code. Research findings regarding spacing and comments present a mixed picture. According to Miara et. al, the amount of spacing used for indentation significantly affects program comprehension: an indentation of six spaces or more can confuse developers [37]. On the other hand, Posnett et al. found that indentation measures did not have a significant impact on their code readability models [42]. Arafat and Riehle, in an empirical study on 5,229 active open source projects, found that commenting the source code is a practice that is consistently followed by successfull projects [3]. However, Buse and Weimer found that even though one would expect that the presence of comments improves readability, the correlation of comments with readability is modest [7]. In addition, Fluri et al. found that code and comments seldom co-evolve and that new code is often sparsely commented [16]. The problems associated with the use of the C preprocessor are well known [48,34,15,49], and most guidelines advocate the avoidance of macro definitions and conditional compilation. Conversely, guidelines also advocate the use of header files (and the #include directive) in order to promote code modularity and portability. According to Steve Johnson [43,Ch. 17], early versions of Unix did not share structure definitions through header files.

"Another major change in V7 was that Unix data structure declarations were now documented on header files, and included. Previous Unixes had actually printed the data structures (e.g., for directories) in the manual, from which people would copy it into their code. Needless to say, this was a major portability problem."

C preprocessor macros can often be replaced by exploiting newer language features, such as enumerations and inlined functions. However, these features and header file inclusion require additional processing power and more sophisticated compiler support. We end the description of the factors associated with the metrics we tracked by noting that the annotation of code through colourful symbols and swearwords (DKLUDGE ), is a matter of convention, which may be fading, being replaced by comments in issue tracking systems, online forums, and configuration management tools.

2.3  Statistical Analysis

We subjected the derived aggregate metrics to statistical analysis in order to discern longitudinal trends. As all the metrics we collected were ordered by date, we performed regression analysis with the days elapsed since the first release as the independent variable and each metric as the dependent variable. We used a General Additive Model (GAM) method [62,63] with cubic splines as our regression model. We gauged the goodness of fit of each trend by calculating the adjusted R² value of each regression.

2.4  Threats to Validity

The internal validity of this study's findings would be threatened by inferring causal relationships without actually demonstrating the mechanism through which the cause drives the effect. In our study we have identified some factors that could affect long term programming practices. However, we have been careful not to draw any conclusions regarding causality, using the factors merely as a starting point for determining and arranging the metrics to examine. The external validity of any findings is limited by the fact that only a single large system (Unix) has been studied. The developers of Unix are not likely to match the general population of programmers, for they include two winners of the National Medal of Technology Award and the Turing Award (Ken Thompson and Dennis Ritchie), two founders of Fortune 500 companies (Bill Joy and Eric Schmidt), and many highly accomplished scientists, technology authors, and practitioners. However, given the system's continued prevalence, utility, and importance, the lack of generalisability, is not the fatal problem it could otherwise be.

3  Results and Discussion

Figure 2 contains the plots of all metrics over time, ordered alphabetically. The x axis of all plots is the release date. In every plot we show the regression line with 95% confidence intervals around it and we include the adjusted R² value as an indicator of the goodness of fit. In the following paragraphs we detail our findings regarding each hypothesis and discuss their meaning. Figure 1 shows the timeline of analyzed revisions and milestones associated with factors that could influence the evolution of programming practices. Although we have not explicitly performed regression analysis against these factors it may be helpful to refer to them in the discussion of our findings. H1: Programming practices reflect technology affordances The metrics FILINE, mean file length (R² = 0.91); FISTMT, mean file functionality (R² = 0.90); LICHAR, mean line length (R² = 0.93); and IDCHAR, mean identifier length (R² = 0.94), increase steadily over time. This lends strong support to the H1; as bigger, higher resolution, and increased bandwidth terminals or workstations became available, Unix developers have been eager to embrace new capabilities and evolve their programming practices to take advantage of them. The FULINE metric, mean function length, (R² = 0.91), is also related to this hypothesis, as bigger screens and higher resolutions allow longer functions to fit on screen. This could explain why mean function size increases steadily to about 60 lines. The drop that follows may be explained as a reaction to inreased function complexity, which we discuss in H6 (Software complexity evolution follows self correction feedback mechanisms). Our findings here lend support to technology and infrastructure investments that aid developers in their work. Nowadays these can include multiple high-resolution screens, fast computing clusters for compiling and testing the software, and a workplace that allows developers to concentrate. H2: Modularity increases with code size The metric DSTATIC, internally visible declaration density (R² = 0.97), increased steadily until reaching a peak, from which it has receded somewhat. Therefore for most of Unix history there is strong evidence supporting the hypothesis. The recent small decline in the use of the static keyword may be counterbalanced through the use of extra-lingual structuring mechanisms, namely the use of kernel modules and shared libraries. A different measure of software modularity, the importing of functionality through header files as witnessed by DCPINC, C preprocessor include statement density (R² = 0.37) has, after an initial bumpy period until the early 1980s, remained remarkably stable over time. A significant rise in this metric (more imports) would indicate a rising number of efferent (outward) couplings. This increased number of dependencies across diverse modules might indicate a lower quality of modular design. In general, the findings regarding this hypothesis support the conjecture that engineering requirements drive an artefact's quality features [51]. In terms of actions, they also support investment in modularity mechanisms. In modern systems the two levels of identifier visibility provided by the C programming language are showing their limits. (This might be another explanation for the recent decline in the static keyword density.) Mechanisms that can handle multiple levels of module federations from diverse sources, such as the C++ namespaces and the Java packages, show the way forward. However, these mechanisms cover only identifiers. Other important modularity issues remain open, and will become increasingly important as code size and reuse increase. These include versioning, deployment, (possibly live) updates, and access to shared resources, such as computing threads, power, the cache, and middleware.