Purpose:

Over the years numerous writers, researchers, and software vendors have proposed myriad methodologies and technologies for use in the software development process. A simple example of competing methodologies is Structured Analysis with its Structured Analysis Diagrams on the one hand and Object-Oriented Analysis with its UML Diagrams --thirteen different varieties at last count-- on the other. An equally simple example of computing technologies is Sun's Java/Corba and Microsoft's COM, both intended for use in the development of distributed software. In both cases, that of methodologies and that of technologies, important adoption decisions are often made on the basis of anecdotal evidence. It is not surprising, however, that is the case; while it would be beneficial to experimentally test newly-introduced methodologies and technologies, both individually and comparatively --on such metrics as steepness of learning curve, productivity once mastered frequency of software defects-- it is virtually impossible to do so in an industrial/commercial setting; the cost of developing software is so great that repeating the experiment of developing a software product, using different methodologies/technologies, is all but impossible.

Approach:

In the context of a Software Engineering course at a university, it is, however, possible to repeat software development experiments, at least on small to medium-sized project, sufficiently many times to produce statistically significant answers to interesting questions. We will set up an experimental infrastructure to perform software engineering evaluations as a research objective. The merits and advantages of specific software engineering approaches for the design of distributed computing systems will be identified and shared with industry.

Status:

Experimental Research on Software Engineering Process, Technology, and Pedagogy

We are revising the curriculum for CS460-461, the required senior-level course in Software Engineering, with an eye toward instrumenting the courses for the purpose of acquiring experimental data relating to software engineering method, process, and pedagogy as a byproduct of the year-long design and implementation projects in which CS460-461 students engage.

Issues being studied include the relative efficacies of competing software engineering methods and processes, and the efficacies of different pedagogical approaches toward teaching the course, issues which are all-too-frequently discussed impressionistically, but rarely with strong experimental support.

The novelty of our approach is that in the setting of this course sequence, with 5-10 project groups each time the sequence is offered, and with each sequence being offered multiple times, it will be possible to run experiments with sufficient numbers of repetitions to obtain statistically significant results. We plan to recruit other universities into a Consortium for Experimental Software Engineering Research, so that experiments with even larger repetition factors can be performed. To date negotiations are underway for collaboration with Software Engineering faculty at Rutgers University, the University of Maine, and the University of Southern California.

The following is the description of a pedagogy experiment currently underway:

The typical Computer Science student who has had no industrial experience, even if -- or, perhaps, especially if -- he or she has had great success in personal software development projects, is entirely unaware of the process-related know-how required to successfully utilize technical skills in group software projects; in fact, many students will even deny the existence of anything but technology expertise as a requirement for success in group software development.

One begins to understand the reason by noting that the typical Computer Science student is oriented towards the solution of problems which provide immediate feedback; for example, the correctness of a small code module written to a given simple specification is easily tested by compiling and executing it with sample data; the validity of the solution to a simple quantitative problem is often directly verifiable through substitution of the result into the specification of the problem. The typical computer science student is probably considerably above the population average at solving such problems. 
The typical Computer Science student is probably not, however, particularly oriented toward, or endowed with more than average talent at, the solution of problems in psychology or group dynamics. In fact, it may very well be the case that the typical Computer Science student is endowed with less-than-average talent in these areas. As a result, the student often defensively views a course designed to teach the principles of software process as irrelevant or trivial, and of no use to those with highly specialized technological expertise. In the worst case students resent being required to take such a soft course.

In order to quickly convince students that if a group software project is to be successful, there actually is something to be learned over and above the relevant technology, we plan to devote the first month or so of the Stevens Senior Project/Software Engineering course (CS460-461) to a Naive Software Development Project, i.e., a project which will shock them into the necessity of software process. At the first meeting of the course students will be divided into groups of approximately ten and will be given (rough) specifications of a software project which they will be required to complete in a month -- without benefit of any training in software development process.
In order to test the pedagogical effectiveness of this approach, students are being surveyed through the use of questionnaires. Students are filling out questionnaires:

*on the first day of the course, to determine students' backgrounds/knowledge of, both software technology and software process, and their expectations of (and prejudices about) the course.
*at the end of the Naive Software Development Project to determine the extent of students' increased awareness/appreciation of the need for software process

Related Publications: