The JOT Blog

The trouble with configuration management in some large technical organizations is that it is not just configuration management. An organization often assigns to a single role responsibility for software builds, configuration management (which often includes change management), and releases of products to customers. While these responsibilities are mutually dependent they are distinct roles, have different goals, and require very different skills. When we evaluate a product line organization this role is often identified as a source of problems. One or more of the dimensions is either neglected or incorrectly executed by the personnel assigned to the position.

There are several issues related to this approach. People in this integrated role are often not familiar with how software development will be carried out on a specific project and as a result they will often design a repository structure that does not permit, or at least hinders, parallel development. This has to do with the grain size of each versioned blob and the dependencies among blobs. A poor structure results in the possibility of two people needing to working on the same file at the same time. Developers who are unaware of the structure of the repository or who do not understand how that structure relates to the structure of the products will not be able to write build scripts that are sufficiently modular to be reused at every level up the aggregation hierarchy and may codify dependencies incorrectly resulting in an improperly linked module.

The CM staff will also define one process that is applied to both development activities and delivery activities. While this makes the release part of the CM job simpler, it slows down the development process which needs quick, frequent access to repositories to commit, update, and retrieve program elements. Some companies address this by having separate development and release CM processes without sufficiently differentiating between the processes.

First I will talk about the goals of each role, then the required skills, and finally dependencies among the roles. I will describe a company doing it right and will briefly describe how they perform these three functions.

Goals

Each of these roles has a specific goal that distinguishes it from the other roles.

The goal of the build role is to provide an executable that reflects the current development state of the product. Often legacy modules are not recompiled during a build. This is sometimes because of the time it would take to build the module but more often it is because there is a risk that the module will not build in the current environment. The result of partial recompilation can be that other modules that have changed and should be rebuilt aren’t.  Building is a sufficiently complex task that many organizations require an independent auditor to witness the actual building to affirm that the correct modules were used to produce an executable with no errors.

The goal of the (real) configuration management role is to allow controlled changes to the source code by multiple people at the same time while protecting the integrity of previous work. This sounds simple, but in an environment such as a software product line it is not so simple. The structure of the repository of versions of the code facilitates specific development patterns more than others. The CM role must anticipate the eventual complexity of a development effort that will involve multiple players for multiple products. In a software product line the emphasis is on multiple references from multiple products to single copies of assets.

The goal of release management is to deliver to the customer a complete and consistent set of modules and resources that represent a correct product at a specific point in time. The product release and related resources such as the installation mechanism must be tested and the release manager determines whether particular features meet quality standards. Each release represents a build that has a permanent life in the CM system.

Skills

The build role needs an understanding of the product architecture, compile and link time variation points, and, obviously, the build tools. The build manager assures that the final build and installation package includes all the required resources including appropriate licenses.  In a software product line, the build script for a product specifies a configuration of the product with compile and link time variations resolved.  The script is placed under management because it captures the variant choices. The build role provides build automation to the development and test staff. In many projects every commit is a compile and test before the commit is confirmed. If every developer commits every day the organization has “always running” code. I have developed for many years using this style and it is a very efficient approach, but it is only feasible if the builds and tests are automatic.

The configuration management role requires knowledge of design, variation mechanisms, and the development process. The structure of the repository must support the approach taken to development. Some parallelism in work is required. The CM process should support concurrent work in a baseline. One way this happens is by having assigned ownership of assets so that only one person does work in a specific asset. Or at least having a separate trunk for each asset with only one person allowed in the asset at a time. Newer build tools allow logical references to files removing the need to physically copy code, but many organizations still do copy and are then faced with a diverging code base.

Release management requires an individual who understands the development process, the needs of the customer, the install time variations, and the paperwork required to support a release.  In many companies the release manager is responsible for negotiating with the customer and development team for what will be ready for delivery in the next release and by what date. The manager determines that every module in the release has been through the full life cycle and is ready for release. The release manager is also responsible for assembling the installation package. Automation is critical to be able to exactly repeat a build and to quickly get ready for the next build.

Every project should establish and maintain a regular schedule of releases.  This steady rhythm will establish a positive expectation on the part of clients. Many large open source projects release nightly builds, which are less well tested than the periodic stable builds. Software product line organizations often make both core asset and product releases on a regular schedule. Eclipse creates a stable, deliverable build for a large percentage of its project twice a year with nightly builds in-between. While the rhythms may vary from one organization to another, this is still a useful approach.

Dependencies

The three roles all manipulate the source code for the products. Essentially CM manages a set of files, some subset of which are selected to be built together, and the result of that build is the key element in a release. While the build process could take place with almost any physical organization of the code, the CM role can facilitate the build process through the structures that are chosen. An organization of the code that reflects some logical structure can reduce the chances of build errors. Clear labeling of versions allows for the automatic generation of new build scripts. The Build role needs to understand the content of a release and know any specific constraints for resolving any compile/link time variations.

Figure 1 Interdependencies

CM needs to know logical dependencies among the source files. The CM role captures a complete snapshot of its database for each release. This does not have to be a separate copy. It can be a versioned script that specifies a specific version for every element in the release. This goes beyond the scope of a “build” which is limited to the elements needed to compile and link the code to produce an executable. The copy includes all tests, test results, the development environment, and more.

The release role needs to know the exact contents of a release, which means the contents of one of many builds and the structure of the repository. If build scripts are managed in the repository by the CM function than the release role can retrieve a specific one at any time.

A Modest Proposal

Much of this confusion can be clarified by automating the build process and treating the build script, which may be more than a single text file, as a first class object. In fact, the build script should be thought of as the product. The script is placed under management and is versioned as changes are made to the contents of the product. The CM role creates a structure for the product that will include the build script and any product unique code. The build role creates the script and submits it to the configuration management system. The release of a product simply requires that the build script be baselined, references into the core asset base are changed from relative to absolute addresses, a separate branch is created to assemble the elements of the product.

Even if the three roles are assigned to a single person, this is a sufficiently simple process that can be handled by a single person. Many tools such as Eclipse and Visual Studio support this approach with integrated build tools. Eclipse has several projects such as Buckminster and b3 that manage the build process (and much more) and consist of artifacts that can be managed in the configuration management system.

Case in Point

My example company will have to remain anonymous but it is a major software development house specializing in a single domain but producing a software product line of consumer and professional products in that domain. This company separates the build, CM, and release roles and is large enough to have several people in each role.

Build – Building a product begins with builds of individual executables. These builds are initiated by development team members.  The owner of a module develops a build script for that module. Successive layers of integration also produce build scripts that utilize the lower level scripts to build from the ground up. Finally each product has a script that automates the instantiation of the lower level pieces. Since a fundamental responsibility of the build role is to ensure that the correct code is compiled and linked, this bottom up aggregation of scripts provides a natural traceability mechanism.

CM – The configuration management is largely automated at the start of a project. The structure of the core asset base has been fixed for some time and each new product sets up to utilize that base in their scheme. The developer sets up a directory structure for their code and includes references to each core asset used rather than copying the asset into their project. The CM role establishes and evolves the basic structure and is responsible for “baselining” deliveries, which only requires creating a branch for the build scripts and making variable references in the script to the latest versions of assets into fixed references to specific versions of those assets instead.

Release management – The role of release manager carries both responsibility and authority.  Once a delivery date is set and the scope of the release is negotiated with all development streams, the release manager has the authority to remove a feature from a product if including that feature puts the delivery date at risk.   The manager also has the responsibility to ensure that any included features have been appropriately tested and have satisfied quality requirements. The release itself is based on the baselined build scripts that have been used from initial development through testing and now to release.

Summary

These infrastructure roles are important to the success of any software development effort but they are particularly critical in a software product line organization. The role definitions and training for each role need to be sharply focused and we need to push for trained personnel in each role. In the event that two or more of these roles are assigned to the same person, controls should be put in place to ensure even-handed and competent treatment in each of the three areas.

Running through this entire discussion is the notion of automation. There are many tools that can be used make building, change management, and product release much more repeatable and robust. Most of these tools support a variety of process structure and styles but most also make the lines between the three roles much more discernable.