Tackling Large Projects
Diomidis Spinellis
Department of Management Science and Technology
Athens University of Economics and Business
Athens, Greece
dds@aueb.gr
Design and Implementation Techniques
- Visible software process
- Disciplined practices
- Programming style
- Organization
- Documentation
- Processes
- Nontrivial architecture
- Merciless decomposition
- Support for multiple platforms
- Libraries, components, processes
- Custom, domain-specific languages and tools
- Implementation style
- Agressive use of the C preprocessor
- Object-oriented techniques
- Operator overloading
FreeBSD 5.2 Release Policy
Subject: 5.2 is coming!!
Date: Wed, 7 Jan 2004 23:57:22 -0700 (MST)
From: Scott Long <scottl@FreeBSD.org>
To: developers@FreeBSD.org
All,
We've delayed quite bit while install problems and various show-stopper
bugs got fixed. I think that it's safe to say that we are getting
pretty close to closing the door on 5.2 now. There are still a few MFC
requests trickling in, so I wanted to publicize the policy that needs
to be followed for the remainder of the 5.2 cycle.
First, there can be no 'instant MFCs'. All 5.2 MFC candidates must spend
at least 36 hours in HEAD before going into RELENG_5_2. It's fine to
contact re@ after committing to HEAD to state your desire to also commit
to RELENG_5_2, but don't expect instant approval. I encourage the 36
hours to be used for as much review, testing, and verification as
possible.
Unless another show-stopper arises, I intend to tag the tree late
Friday night or early Saturday morning (MST -0700). In accordance
with the previous paragraph, this means that there are approximately
12 more hours for bug fixes to go in and be 5.2 candidates. This is
_not_ intended to be a signal for people to start rushing stuff in,
but rather a reminder that we cannot hold the release up for forever.
MFC approval is still at the discretion of the RE team. After this
cut-off, only show-stoppers will be considered for approval.
While 5.2 will certainly have bugs, I'm happy to hold the release up for
true show-stoppers. A show-stopper generally is a bug that causes a panic
[crash] during normal and reasonable operation of the OS or prevents the
OS from being installed in a way that is documented, and has no reasonable
work-around. These bugs must be confirmed by a third party to ensure
that it is a problem that is likely to be reproduced in the user base. It
also includes security vulnerablilities that are published by mainstream
security outlets and have not already been addressed in some fashion.
Thanks a lot!
Scott
Directory Structure
- src
- TLD (e.g. .org)
- main / program-name
- lib
- common
- include
- doc
- man
- arch / os
- .git / .svn / CVS
- build / compile / classes / obj / Release / Debug
- tools
- test
- conf
- etc
- eg
- bin
- contrib
Project Organization
Two approaches:
- Use a single structure for all parts (doc/named)
- Replicate directory structure for each part (named/doc)
Keep in mind: large structures are often very easy to navigate.
- 3000 directories.
- Source is related to the installation path.
Source Code is Not Only Code
- Specifications
- Design documentation (rarely)
- End user documentation
- Developer documentation
- Test scripts
- Multimedia resources
- Build tools
- Examples
- Localization files
- Revision history
- Installation procedures
- Licensing information
Key Processes
- Configuration management
- Building automation
- Issue tracking
- Tooling
Project-Specific Tools
- Used to automate the build process (X: imake, makedepend)
- Generate specialized code
- Handle regression tests
- Automatically generate documentation (javadoc, pod)
Building the IBM 3270 terminal emulator
Debug Messages
- Debug messages can help us understand the rest of the code
- Console debug messages:
#ifdef DEBUG
if (trace == NULL)
trace = fopen("bgtrace", "w");
fprintf(trace, "\nRoll: %d %d%s\n", D0, D1, race ? " (race)" : "");
fflush(trace);
#endif
- System log messages:
syslog(LOG_DEBUG, "Successful lookup: %d , %d : %s\n",
lport, fport, pwp->pw_name);
- Different debug levels:
if (debug > 4)
printf("systime: offset %s\n", lfptoa(now, 6));
Logging
Logging offers a number of advantages over using a debugger:
- Placing and output is program-specific
- Can perform data-structure traversals
printf(PDQ_OS_PREFIX "FDDI Port%s = %c (PMD = %s)",
PDQ_OS_PREFIX_ARGS,
rsp->status_chars_get.station_type == PDQ_STATION_TYPE_DAS ? "[A]" : "",
pdq_phy_types[rsp->status_chars_get.phy_type[0]],
pdq_pmd_types[rsp->status_chars_get.pmd_type[0] / 100]
[rsp->status_chars_get.pmd_type[0] % 100]);
- Stored with the program source code
- Careful formatting
printf("\n");
printf("%s %s trap:\n", isfatal? "fatal" : "handled",
user ? "user" : "kernel");
printf("\n");
printf(" trap entry = 0x%lx (%s)\n", entry, entryname);
printf(" a0 = 0x%lx\n", a0);
printf(" a1 = 0x%lx\n", a1);
printf(" a2 = 0x%lx\n", a2);
printf(" pc = 0x%lx\n", framep->tf_regs[FRAME_PC]);
printf(" ra = 0x%lx\n", framep->tf_regs[FRAME_RA]);
printf(" curproc = %p\n", curproc);
if (curproc != NULL)
printf(" pid = %d, comm = %s\n", curproc->p_pid,
curproc->p_comm);
printf("\n");
- Adjustable and filterable (syslog, log4j, Apple logging)
import org.apache.log4j.*;
Argo.log.info ("Loading modules from " + file);
Argo.log.warn("Could not instantiate module " + classname);
Argo.log.warn ("Class " + pluginType.getName() + " is not a core Argo pluggable type.");
- Useful when a debugger can't be used
Assertions
- Can be used to identify how the program should behave
- Are used to verify:
- Steps of an operation
- Algorithm input and output values
- Function parameters
- Flow of control
- Hardware properties
- Test results
Example (from regular expression engine)
if (dp != NULL)
break;
/* uh-oh... we couldn't find a subexpression-level match */
assert(g->backrefs); /* must be back references doing it */
assert(g->nplus == 0 || m->lastpos != NULL);
Assertions in Java
Example
/*
* Run With java -ea FindMax
*/
class FindMax {
/** Return the maximum number in non-empty array v */
public static int findMax(int v[]) {
int max = Integer.MIN_VALUE;
// Precondition: v[] is not empty
assert v.length > 0 : "v[] is empty";
// Precondition: max <= v[i] for every i
for (int i = 0; i < v.length; i++)
assert max <= v[i] : "Found value < MIN_VALUE";
// Locate the real maximum value
for (int i = 0; i < v.length; i++)
if (v[i] > max)
max = v[i];
// Postcondition: max >= v[i] for every i
for (int i = 0; i < v.length; i++)
assert max >= v[i] : "Found value > MIN_VALUE";
return max;
}
// Test harness
public static void main(String argv[]) {
int t[] = new int[5];
t[0] = 4;
t[1] = -4;
t[2] = 145;
t[3] = 0;
t[4] = Integer.MIN_VALUE;
System.out.println("Max value is " + findMax(t));
}
}
Further Reading
- Moshe Bar and Karl Franz
Fogel.
Open
Source Development with CVS.
The Coriolis Group, Scottsdale, AZ, 2001.
- Kent Beck and Erich
Gamma.
Test infected: Programmers love writing tests.
Java Report, 3(7):37–50, July 1998.
- Stephen P. Berczuk
and Brad Appleton.
Software Configuration Management Patterns: Effective Teamwork, Practical
Integration.
Addison-Wesley, Boston, MA, 2002.
- Don Bolinger, Tan
Bronson, and Mike Loukides.
Applying RCS and SCCS: From Source Control to Project Control.
O'Reilly and Associates, Sebastopol, CA, 1995.
- Per Cederqvist
et al.
Version Management with
CVS, 2001.
Available online http://www.cvshome.org/docs/manual/ (January 2002).
- Roger F. Crew.
ASTLOG: A
language for examining abstract syntax trees.
In Ramming [Ramming, 1997], pages 229–242.
- Rohan T. Douglas.
Error message management.
Dr. Dobb's Journal, 15(1):48–51, January 1990.
- Paul Dubois and
Gigi Estabrook.
Software Portability with Imake.
O'Reilly and Associates, Sebastopol, CA, second edition, 1996.
- Rickard E. Faith, Lars S.
Nyland, and Jan F. Prins.
KHEPERA: A
system for rapid implementation of domain specific languages.
In Ramming [Ramming, 1997], pages 243–255.
- Stuart I. Feldman.
Make—a program for maintaining computer programs.
Software: Practice & Experience, 9(4):255–265, 1979.
- Cem Kaner, Jack Falk, and
Hung Quoc Nguyen.
Testing Computer Software.
Wiley, New York, second edition, 1999.
- Nils Klarlund
and Michael I. Schwarzbach.
A
domain-specific language for regular sets of strings and trees.
In Ramming [Ramming, 1997], pages 145–156.
- Andrew Oram and Steve
Talbott.
Managing Projects with make.
O'Reilly and Associates, Sebastopol, CA, second edition, 1991.
- J. Christopher Ramming, editor.
USENIX Conference on Domain-Specific Languages, Berkeley, CA, October
1997. Usenix Association.
- M. J. Rochkind.
The source code control system.
IEEE Transactions on Software Engineering, SE-1(4):255–265,
1975.
- Diomidis
Spinellis and V. Guruprasad.
Lightweight languages as software engineering tools (http://www.dmst.aueb.gr/dds/pubs/conf/1997-DSL-Lightweight/html/paper.html).
In Ramming [Ramming, 1997], pages 67–76.
- Diomidis Spinellis.
Implementing Haskell: Language implementation as a tool building
exercise.
Structured Programming (Software Concepts and Tools), 14:37–48,
1993.
- Diomidis Spinellis.
Reliable software implementation using domain specific languages (http://www.dmst.aueb.gr/dds/pubs/conf/1999-ESREL-SoftRel/html/dsl.html).
In G. I. Schuëller and P. Kafka, editors, Proceedings ESREL '99 —
The Tenth European Conference on Safety and Reliability, pages
627–631, Rotterdam, September 1999. ESRA, VDI, TUM, A. A. Balkema.
- Diomidis Spinellis.
Notable design patterns for domain specific languages (http://www.dmst.aueb.gr/dds/pubs/jrnl/2000-JSS-DSLPatterns/html/dslpat.html).
Journal of Systems and Software, 56(1):91–99, February 2001.
- Diomidis Spinellis.
Code Reading: The Open
Source Perspective, pages 179–224.
Effective Software Development Series. Addison-Wesley, Boston, MA, 2003.
- James M. Stichnoth
and Thomas Gross.
Code
composition as an implementation language for compilers.
In Ramming [Ramming, 1997], pages 119–132.
- Walter F. Tichy.
Design, implementation, and evaluation of a revision control system,.
In Proceedings of the 6th International Conference on Software
Engineering. IEEE, September 1982.
- Gary V. Vaughan, Ben
Elliston, Tom Tromey, and Ian Lance Taylor.
GNU
Autoconf, Automake, and Libtool.
New Riders Publishing, Indianapolis, IN, 2000.
Exercises and Discussion Topics
-
Propose ways to quickly determine whether a given project follows
one of the design or implementation techniques we described.
Test your proposal against one of the major projects available in
the course's reference source code .
-
Locate recommended design and implementation practices in a software
engineering book.
Explain how these are reflected in a project's source code.
-
How can a standardized directory structure be utilized for automating
aspects of the software development process?
-
Examine and describe the directory structure of an installed version
of Microsoft Windows.
-
Configure a program from the course's reference source code to compile and run in a
supported environment.
Examine the configuration process output files and
manually verify four different configuration options.
-
Read the source code of a configuration script,
explaining how two different configuration options
are determined.
-
How do you track revision histories in your environment?
If you are not using a revision control system, consider
adopting one.
-
How is the build process managed in your favorite
integrated development environment?
Discuss the advantages and shortcomings of the employed approach.
-
Locate two different test cases in the course's reference source code and explain how their
execution is, or could be, automated.
-
The Perl test cases suggest that
``A good test
case has most of these attributes: fewest possible
number of lines; few dependencies on external
commands, modules, or libraries; runs on most
platforms unimpeded; and is self-documenting.''
Locate test cases in the course's reference source code and judge them
against the above standard.