So is Agile just hype or is there something to it?

There are strengths to the agile way of thinking, and many of them bring useful perspectives to software and systems development that are new and even revolutionary. Here are some of the things that work — and, potentially, that radically change our old-world practices.

Whereas most legacy methods stem from industrial process — that is, assembling a product using a set of defined, predictable steps — the agile method is empirical. It recognizes that development is more like invention and research, more akin to scientific study, than assembly. This empirical nature is at the heart of the agile mantra: Deliver, measure, adjust and repeat. The strength of this approach often bears itself out in fantastically hyperproductive teams that deliver working product far more quickly than legacy methods, such as waterfall, could ever achieve.

Agile does this by cutting through complexity. Every agile-based methodology focuses on simplification of otherwise complicated problems. For example, XP and Scrum both emphasize development of near-term, complete deliverables. This means carving out tangible and reasonably independent pieces of work, focusing on that work, and then — at least as much as possible — moving on to other work. This approach requires that large, complex problems are broken down into manageable pieces and thought of on a micro-deliverable level. Likewise, this approach minimizes ceremony and eschews as much procedure as possible. Some agile methods go to extremes in this regard, focusing entirely on delivering work product and not at all on procedure. This translates into minimizing complexity on a large scale.

Closely related to eliminating complexity is agile’s focus on progress measurement. Most agile methods measure progress chiefly, if not exclusively, in terms of delivered work product. Most methods also are quite stringent in defining progress only when finished work is delivered, which means you can’t work for nine months on a single big feature. Instead, micro-deliverables target key features, deliver those features into the customer’s hands, then moving on to new features. This can be a huge strength because the customer gets working product in-hand to review early, and often. It involves the customer early in product evolution, leading to a host of benefits including better product targeting, prioritized development and improved quality.

These characteristics of agile methods combine to fundamentally change the way software and systems development is practiced. Agile also empowers individuals to become stellar performers. In fact, all forms of agile rely on this to some degree — with more lightweight agile methods being completely dependent on individual empowerment. The idea is that an empowered team will leap over constraints to get the job done, no matter how “out of the box” the thinking needs to be. It’s a refreshing concept and one that can indeed be supremely successful, but it does require the team to embrace the idea wholeheartedly.

Another benefit, at least in some situations, is the creation of self-organizing teams. Partly because of the light ceremony, the fast pace, and the penchant for empowerment and accountability, teams become self-organizing. This works when the team has the right make-up, as individuals step up to take on tasks best suited to individual skills. Self-organized, empowered teams become very powerful and very productive, provided that the team members are up to the job.

There is absolutely no doubt that agile methods make it possible to get things done quickly. That’s what it’s all about, after all. The real question to me is how much of this tradeoff is really desirable? How often do we want to eschew process and maturity in favor of getting things done quickly?

More importantly: Can we effectively merge the best attributes of Agility with the most valuable benefits of established processes and standards?

Why Agile doesn’t work

When an agile project fails, it generally does so spectacularly and predictably. The common failings of agile-based projects are just that… common. We see the same problems over and over again, and this has become the basis for many critiques of agile methodology. After all, if we keep seeing the same problems crop up again and again, isn’t this proof enough that the process is flawed? This becomes clear in hindsight, so why do we continue to see 90 percent of projects missing the mark?

The fact is, agile by itself is just one tool in the toolbox that should be applied with other implements of the trade. In my experience, the problem comes in most often because small- and mid-sized organizations experience brilliant success with agile and then assume it can work everywhere. They throw out the toolbox (or perhaps never buy one in the first place). Yes, agile can succeed. Yes, it can deliver fantastic productivity and stellar results. But not always — in fact, I will go so far as to say not often.

This isn’t because of agile’s limitations. Instead, it’s because of overconfidence by those putting it to use, and the mistakes an immature organization makes as it grows and applies it inappropriately.

Immature companies and teams are cutting their teeth, again and again, on the limitations of agile.

All agile methods make it easy to oversimplify complexity. In fact, agile’s strength of eliminating complexity might be better stated as “ignoring complexity.” There are appropriate situations for this but, more often than not, ignoring complexity leads to problems. Most business cases don’t call for undefined delivery dates or loosely changing requirements and partial deliveries. These are risks that most business models are incompatible with. If the risks aren’t something that your business can sustain, adopting a purely agile process is taking a huge gamble.

Likewise, focusing on the near-term is an agile attribute that introduces a lot of unknowns into the business-end of an equation. Few people will contend that agile is appropriate for mission critical efforts such as, say, launch vehicle development, as sometimes requirements need to be set in stone before anyone starts development. But what about situations where some degree of fuzziness is acceptable or even beneficial? Agile advocates compatibility with change, sidestepping change control procedures that would otherwise place tight controls over requirements. Requirements change carries with it a heavy burden, particularly when it comes to the cross-organizational impact to marketing, budget, quality management and the customer. However, cutting change control, requirements management, and configuration management from the process can lead to long-term disaster that the short-term perspective of most agile methods will overlook.

This theme of reducing structure and control has cut out many waterfall-origin processes. The danger often manifests as small-scale agile projects are successful, leading to wider-scale adoption of agile. But, as the projects grow in complexity and criticality, major missing components in the process become evident. For example, no agile methods today integrate comprehensive quality assurance procedures (in fact, thanks to some early mistakes, such as MIL-STD-498[#], most people think quality assurance is software testing — it’s not). Structured software testing often becomes an afterthought, and risk management programs tend to be regarded as “fuzzy disciplines.” Yet, these are the processes that successfully put man on the moon, that develop health care and financial services systems, and ensure that nuclear plant regulatory systems don’t fail after delivery. Of course, there is a cost to each of these processes, and every business needs to weigh the cost-benefit of adopting more process against cutting those processes. This needs to be an on-going evaluation, made as projects, organizations, and teams evolve — it’s not a decision that stands alone.

From a purely hands-on, management level, agile methods pose “people problems” as well. The strong emphasis on self-organization and empowerment can easily backfire. The former relies heavily on people that are capable of self-management and self-direction. Not everyone can live up to that expectation. The latter, delivering empowerment to the team and individual, can lead to a hero mentality and silo’d teams that refuse to play well with others. As projects grow in size, complexity, and dependency on other teams and resources, these characteristics become the drawback of an immature organization.

Almost all agile methods oversimplify valuable processes. In some situations, the project survives the oversimplification. Sometimes the business is tolerant of the fallout. In every case, agile methods expose the project to risks that stakeholders should be — and often are not — aware of.

What to do about it

We need to be cognizant that one solution does not fit all problems. While an agile method such as XP or Scrum may have led to success in one project, this doesn’t make it a foregone conclusion that it will do so again. Each project is different, and organizations evolve over time. Adopting one process to solve all problems is a sure recipe for failure. On the other hand, having a well-versed team that can draw on several methodologies, as appropriate for the job, is a recipe for success.

If your organization is looking for the one-size hammer to hit every nail, make sure it’s as configurable a hammer as possible. Don’t choose something that is either too lightweight, such as XP, because many projects will overreach the capabilities of such a lightweight process. Likewise, don’t try to implement a full-on waterfall style methodology either because, while definitely thorough and capable of getting the job done, it’s just overkill for many smaller projects. If you must choose a single process, pick one that’s efficient, borrows from both agile and waterfall, and is highly configurable, such as Rational Scrum or the Rational Unified Process. Both of these have the maturity to deliver large-scale projects, but also support starting small and adopting minimum ceremony.

A better awareness of what specific agile practices can and cannot accomplish is key. For example, Scrum is not a development methodology, and it cannot effectively deliver software or hardware projects unless it wraps itself around one. Yet today many organizations are employing Scrum as if it were a development methodology. I’ve even seen an organization of several hundred developers “force fed” Extreme Programming from the top down. The outcome of that particular operation: Mid-level management hid the fact that they didn’t use XP from top-level management after everyone realized what a mistake it was. Perhaps we’ll have to wait for mature standards in education and certification to evolve, but personally I’m not sitting idly by.

One of my personal pet peeves in the technology industry is a relative lack of standards and qualifications. Would you go to a doctor that didn’t have a medical degree? Would you hire an architect that didn’t have an appropriate engineering degree? Yet we hire software professionals (much less often hardware professionals) without adequate education, current qualifications, or meaningful certifications. For that matter, the proliferation of meaningless qualifications (such as Scrum Master certification) continues to weaken the industry. In the long run, we need better standards regarding education, accreditation and certification.

Understand agile methods for what they are. Keep in mind that lightweight process carries risk. Use the right tools in the right situation.

Coming full circle

If we add all of these things to agile methods, won’t we just end up using waterfall process all over again?

I don’t think so. Waterfall-based process, the original behemoth processes born out of industrial process, are widely recognized as inefficient. There are tremendous advantages to pressing forward with a merger between waterfall practices and agile practices. I hope the end result is a new generation of software and hardware development methodology — a generation that we’re just starting to see as processes such as Rational Scrum come to the fore. It’s time for development methodologies to evolve, and there’s no holding that back.

Related posts:

1. Making Scrum work: Common failings in adopting Scrum
2. Scrum is not an agile methodology
3. Common oversights in choosing methodology

Probably the most difficult situation is the latter — a boss that’s too hands-on, or perhaps an external resource that is too busy, yet absolutely must approve every decision. The story usually goes something like this: He’s a great guy, pretty easy to get along with when it comes right down to it — but he’s also insufferably “hands-on.” He’s just got to review every critical decision, contributing, fine-tuning and tweaking until it’s just right. This micro-managing mentality is causing all kinds of bottlenecks: Decisions are held until the last moment, changes are made late in the game, and your entire department suffers — staying late to “catch up” after your boss delivers his final word on any given issue. But here’s the real kicker: He’s good at what he does, and even though it creates internal chaos the finished product is that much better for his input.

Consider the image to the right: A critical “quality spike” representing the single project resource that is overloaded. This resource does raise project quality, but the spike also represents a huge bottleneck to the project.

Your department has tried everything: Getting him involved earlier in the game (it doesn’t work, his schedule is booked and it always presses to the last moment); Involving other review sources so your boss’ input is minimized (that helped a bit, but the other sources don’t like it when your boss overrides their contribution); You tried publishing the “departmental cost” figures that show all this inefficiency (but the powers-that-be seem to feel this is “just the cost of doing business”).

This is not a simple problem. It gets right to the root of the dynamics created in working teams. How can the situation be improved, realizing positive gains in a habitually entrenched process that some recognize as painful, but overall is regarded as good enough?

There are really two issues that need to be dealt with here. One is obvious, one perhaps a little bit less so. Most strategies up until now have focused chiefly on the primary goal:

“How can we minimize the negative impact of a critical resource (our ‘hands-on boss,’ for example) that doesn’t work efficiently with your team?”

The problem with this approach is that it attacks an entrenched problem head-on, and often does so using a single head-on approach to solve the problem. When an attempt fails, that approach is discarded and another tried. The principle flaw here is that each attempt to solve the problem is, in and of itself, not effective enough to get the job done. A secondary flaw is that each separate attempt tends to get lost in the chaff of day-to-day operations. We all know there’s a problem, but nobody is really tasked with studying it, compiling the entire scope of the problem and its impact, and somehow bringing about a solution.

For example, when your department produced some figures on costs to the organization, those figures clearly showed that, through inefficiency, rework, and last-minute changes, your team suffered. You even went so far as to tie it to actuals, a dollar figure that represented all that overtime and rework — but, it was accepted as “the cost of doing business.” Since it didn’t work, it was discarded — but the fact is, it’s only part of the overall puzzle.

So the head-on solution won’t work. Let’s consider a more subtle approach to the problem:

“How can we make the critical resource less critical?”

This would achieve the same result, but it’s really an entirely different problem — one that can be attacked somewhat obliquely. Rather than focusing on how to change the way your boss works (aka “the critical resource”), try focusing on changing what your boss needs to work on.

The first step toward a solution is realizing that solving this problem is a project, just like any other. It needs a project lead and sponsor. It needs to be handled as an iterative project, and the team needs to recognize that incremental improvement toward an ideal is acceptable. You aren’t going to discover a single silver bullet that solves the whole problem — that’s just not the nature of human team dynamics.

Make it a project initiative

Once your “quality improvement project” is up and running it will snowball. Here are a few steps that will get the ball rolling:

1. Initiate the project. Don’t think of this as a “workplace problem” that needs to be fixed in the short term. Instead, create a project initiative around it and get some mindshare going. Someone is going to need to lead the project, even if it’s unofficial — that person will keep the forward momentum.
2. Focus on making many small improvements across the board. For example, in the case I described above, consider how involving other review sources did in fact help, but not always. That’s an incremental improvement, and it raised the quality of the project a little bit.

Now, consider the impact of this approach over time. If numerous improvements can be implemented, and each one raises project quality just a little bit, it has an inevitable outcome: Your boss is going to have less to worry about, and less to be involved in. This is shown visually in the second image — as the overall quality matrix delivers improvement across the project, the “critical spike” that represents your boss’ workload begins to shrink. This is a quality matrix at work.

The goal is simply this: Dilute the need for your boss to be involved in every decision, by raising the bar in as many places as possible. You will gain a “double whammy” by not only reducing the amount of work he needs to contribute to, but also by increasing his own efficiency since he’ll have less to worry about. The bottleneck will begin to dissipate.

Assemble your team

Get involvement from everyone that’s affected by the problem — your team, your peers, other departments that might be affected. Of course, part of the finesse in this kind of project is to make it clear exactly what problem you are solving without making the project sound subversive or offending the wrong people. Your boss may be a terrible bottleneck, but also remember how valuable he or she is — this isn’t about cutting him out of the picture, going around him, or changing policies you don’t like. It’s much better to focus on things like qualitative improvement, streamlining projects to avoid inefficiency, or developing lean principles. Make it positive, in other words.

Identify key risks

Your probably already know what the main pain points are. Your team experiences them all the time: Loss of productivity, overtime, last minute changes, introducing new errors, reworking something that could already be finished. These are the lesser symptoms of an endemic problem: What are the potential risks if the problem goes unaddressed? Certainly continued loss of efficiency, higher expenses to the company and lower employee satisfaction come to mind. Perhaps there’s even a trend of some team members transferring out of the department to find a better working environment. More dramatic effects can include missed product deadlines, problems with released products or lowered perception of the department’s attention to quality. All of these can have a real financial impact on the company.

Identify those risks that are most significant. Those should become the focal point for initial improvement. This is a “risk driven” model, where high risk is identified — in other words, the most painful or potentially painful result of the problem gets the most attention as early as possible. Generally it’s best to tackle a few things at a time. While the list of risks could be very long, if you try to solve every problem at once it’s going to be overwhelming.

Mitigate

Once you’ve identified the top few risks, pull together your project team and identify candidate solutions. You’re designing something new here, so it’s going to seem like brainstorming — which is exactly what it is:

1. Can more attention from outside experts improve the overall product, lowering your boss’ need to be involved?
2. Would the quality assurance organization be a helpful partner in achieving this?
3. Could changes in the project timeline better accommodate your boss’ hectic schedule? Perhaps driving for earlier involvement (or longer project schedules) would help.
4. Can the team multitask, effectively putting several critical paths into play so that if one gets stalled waiting for your boss, the others move forward?
5. Would better tracking systems and information management help solve the problem? Perhaps your boss would benefit from a system that brings more organization and immediate answers directly to him.
6. Perhaps greater visibility into the project timeline and reasonable deadlines would both improve responsiveness and provide some firm schedule guidance.
7. Information radiators (such as task boards, timelines and assignment lists) might help keep people up to speed on overall project status (and what has moved from “medium” to “high” priority).
8. A widely accessible system that assigns tasks and prioritizes those tasks would increase visibility of current goals as well as get everyone united about what needs attention first.

Every organization is going to discover different solutions that help its specific situation. It will likely take time to hit on the first few steps that move in a positive direction — but those are the ones you want to hang on to.

Include a quality improvement element

Organizations that have the benefit of a formal quality assurance and improvement group should try to leverage the group’s knowledge. Quality assurance is about auditing and measuring progress toward improvement. Most quality assurance groups tend to be very good at measuring improvement over time, and putting that knowledge to use will help in gathering metrics and identifying what’s working, and what isn’t.

Track your progress

Measure, act, measure again, and adjust. This is the heart of most agile development methodologies — and most scientific methods.

The most important thing to do is realize when improvement has taken place. That means tracking information — metrics — about your organization’s efficiency. This can seem pretty amorphous when dealing with a problem such as this. Remember that it comes down to improved efficiency of individuals. Some teams will have simple metrics that are easy enough to track, such as number of projects completed on a monthly or quarterly basis, or number of cases filed. When all else fails, the number of hours committed to rework is a great metric, because it tracks directly to risk identification and mitigation up front. That is, if you properly identify a risk and mitigate it early in the project, the amount of time spent in rework related to that risk goes down.

This is one reason it’s important to have everyone affected by the problem involved: It’s going to require everyone to look for these signs of improvement. It might even require tracking hours spent in rework. Fortunately, with the goal in sight (less time spent reworking and fewer overtime hours) it shouldn’t be too hard to commit people to some moderate tracking.

Gather metrics on what works and what doesn’t work. Emphasize incremental improvement as a desired outcome: Track the results of each effort, and keep the best practices. The important thing here is to demonstrate improvement over time, so that everyone sees each action as contributing to the solution, not as a failure in delivering the end result.

One of the most important goals in creating a project mentality is the positive group effect. Seeing each practice as part of a solution keeps it alive — as opposed to trying something and perceiving failure. As the team, department or company works together to deliver a solution it becomes a self-reinforcing effect. Even your boss might notice, and actively start to take part.

Related posts:

1. Quality assurance as a way of life
2. Exposing the enterprise to risk: Who decides what not to test?
3. Why heroes are bad

1. Quality Assurance is responsible for auditing and ensuring all aspects of work meet agreed upon quality standards.
2. Therefore, if QA is also responsible for Structured Software Testing, who is going to audit the testing team for compliance and quality of deliverables?

Quality assurance is the “cop” that makes sure we all do our job right. It has the authority to say “hold it, something’s not right.” Testing is the organization that performs regression analysis on a product to see if it works right. The skills required for these two disciplines are dramatically different — as much as business management and programming.

The value in a QA organization is that it is independent. It focuses entirely on ensuring quality across the organization. Would you want development, project management, requirements management, configuration management, verification & validation, customer service or any other project discipline to report to QA? Why should testing be any different? The bottom line is simply that if testing reports to quality assurance its independence is compromised — the organization becomes vested in representing testing in the best possible light and, just possibly, taking shortcuts or letting a few things slide.

I realize that a large segment of the industry seems to use a different terminology, lumping testing under quality assurance. It’s unfortunate, because doing so handicaps both organizations. It’s important to realize the difference between the activities of ensuring quality in a project (largely focused on standards of process), and fault testing (activities that perform hands on fault detection and diagnosis).

The activities of quality assurance focus on things like quality assurance plans, project audits, requirement audits, checklists, enforcing standards and process, checking the results and deliveries of different teams, and discovering discrepancies between requirements, project artifacts, and functional goals. Quality assurance, as a general rule, spends more time looking at what the project teams are doing, performing audits on the work generated by each team, and figuring out what hasn’t been done according to agreed upon standards.

Testing, on the other hands, focuses on test planning and management, test case development, test execution, regression analysis, performance testing and defect diagnosis. These are “hands on the software” activities and much more akin to programming than auditing. Indeed, with today’s testing tools and complicated programming environments it can be a very “in the code” experience.

Keep your organization healthy and your teams focused on their competencies. Don’t disregard the value in centralizing authority for specific roles with specific teams — it lets them do their job well, without distractions and without muddying the water with conflicting interests.

Related posts:

1. Exposing the enterprise to risk: Who decides what not to test?
2. Fix your boss (or, reduce risk to quality using a matrix approach)
3. Scrum is not an agile methodology

One such tool is comindwork.com, a fabulously rich project management software as a service (SaaS) site. While not right for everyone or for every situation it’s definitely worth taking a look at.

Comindwork combines over 250 project management related capabilities under one roof, yet does it with a web interface that is, by and large, a breeze to use. Some of the strengths of the service include traditional project management tools, knowledge management, collaboration tools, information sharing and versioning, and both agile and traditional waterfall management tools (e.g.: think Gantt).

The entry point is easy, and that’s another strength for comindwork: A small team can get started for somewhere around $20 a month (for teams of 10 or fewer, it’s $1 per day that you actually use the service — if nobody logs in, it’s no charge). This offers up a wealth of really advanced tools at a fraction of the cost of most large scale management infrastructure. For companies that don’t have a system in place, it’s easy to give comindwork a spin.

Here are a few of the things I like about comindwork.

It all starts with the personal dashboard. I’m a huge believer in personal (meaning, customized and personally relevant) dashboards, especially if they follow the basic principle of “Getting Things Done” methodologies. Distraction is bad, focus is good:

With the project dashboard, you can:

1. Get a bird-view on all activities where you are involved, see who changed what and when
2. See your nearest milestones
3. Check team members’ status and mood
4. Easily access detailed project-specific dashboards

It offers both traditional (meaning, typically, “large scale”) project management and agile management philosophies living under one roof. At first glance I was taken aback by any system that can claim to offer this mix of tools, but comindwork manages to pull it off. On the traditional side, there are Gantt charts and round-trip import and export of Microsoft Project files, not to mention a whole host of reporting capabilities. On the agile end of the spectrum, to-do lists, tasks and very easy time tracking support simple progress monitoring. Unfortunately, burndown charts have yet to make an appearance, although there’s enough information available that they may not be entirely necessary.

Knowledge management and collaboration are central to the product. Blogs, to-do lists, milestones and business wiki support which codify and share tacit knowledge are tightly integrated into the project. In fact, one of comindwork’s strengths is that so many services are so tightly integrated. For example, linked business wiki entries, tasks and time commitments can be shared and kept up-to-date, with progress being reflected in round-trip Gantt tracking in Project. Notifications of all activity take place automatically, sending out instant email messages or daily digests that summarize project activity.

One of the problems I’ve run into with customers that have no pre-existing system is simply keeping track of all the project artifacts and versions of each. With email flying everywhere, documents being authored, and half the team not knowing how to use source management repositories, how can you hope to keep track of every artifact the team produces? I like to implement a policy of “email doesn’t exist,” but this means you need a tool that’s going to support the policy.

In other words, if someone wants to get something done, it should be in a system, not flying around in email. The idea of half a dozen versions of a document living on everyone’s desktop is just unacceptable to me. Comindwork provides file versioning and drag-and-drop upload. This makes it possible to implement the “email doesn’t exist” policy, and comindwork does a great job of delivering an easy to use tool:

Having a convenient and ubiquitous place to store project artifacts makes it easy for the team to share, manage, and collaborate. Comindwork put enough effort into the interface that it’s not painful:

1. Create a convenient tree-structure of your documents
2. Make common actions on a set of files (mass-delete, mass-move)
3. Provide comments to any file version
4. Versions are stored automatically whenever you upload a file with the same name. Check all revisions and easily revert if required
5. Use drag & drop area for native multiple files upload

You can even email files directly to a folder in your comindwork project.

For more demanding projects, you can design custom workflows to support the security, policies and customer demands of the project. This is an essential tool in my mind. Any project management solution needs to be able to grow with the project. Custom workflow makes it possible:

You can create graphical representation of your business process, modify the process with appropriate business rules and make sure your project is enforcing necessary policy:

1. Define, control and track states and transitions in your business process
2. Encourage process automation and standardization
3. Break your business process into easy to follow step-by-step workflow diagram
4. Reconfigure your business process as needed
5. Clearly define your business process and avoid miscommunication and inconsistency

If you’ve been casting about looking for how to get project tracking off the ground, take the quick tour and see what you think. It’s not a panacea that will fit all project’s needs, but it is a very solid tool that’s been seeing a lot of success lately.

If you decide it’s not for you, take a look at Atlassian’s JIRA too. JIRA is by far my favorite project management tool, especially if you’re agile-oriented. It requires a bit more of an up-front investment to get off the ground (both in terms of deployment and financial cost), but in my opinion, it’s one of very few first-rate tools available today. I’ve been trying to finish a detailed article on JIRA for some time now, so check back in a bit.

Related posts:

1. Fix your boss (or, reduce risk to quality using a matrix approach)
2. Why Agile isn’t enough (and why it doesn’t work)
3. Whole teams

Testing is becoming as necessary a profession as design and coding. Skills and experience matter. Process matters. Tools matter. Let the tests begin.

Our systems are becoming more complex — in fact, logarithmically so. Not long ago, entire projects were conceived, designed and coded entirely in-house. Projects were measured in thousands of lines of code, perhaps hundreds of thousands for large scale projects. Everything was created “here,” so to speak, and it was conceivable to understand and test the entire system. Not necessarily so in today’s world.

Today, most of the code in our projects is written by someone else. We integrate third-party libraries, both commercial and open source. We use code repositories that were written by co-workers no longer with the company. As Parkinson points out, testing projects in this environment becomes a far more complex task:

Modern code has millions (perhaps billions) of possible execution paths and program states, and we cannot test every unique combination. So we have evolved as an industry, consciously or unconsciously, toward a testing strategy based on a combination of materiality and risk. … But to do it right, you have to do it consciously. And I suspect that in some shops this isn’t always the case.

Despite this, testing continues to fall under scrutiny and remains one of the first programs to suffer budget cuts or face time constraints. This is evidenced by strategies that balance “materiality and risk.” Stated another way, testing programs are routinely under tight constraints to reduce cost and “get out the door faster.” Consequently, testing program managers are pressured to focus efforts in high-risk areas, choosing which areas to test knowing that comprehensive testing is not possible.

Is an increased need to focus testing on high-risk areas a sign of the times, or a sign of system complexity, or a sign of poor understanding of the software process? Granted, vastly more complex systems are more difficult to test — but does this justify a reduction in the quality of our testing efforts? Isn’t it more logical to conclude that more complex systems require more rigorous, more thorough testing programs?

After decades of experience we’ve learned that it is far more costly to correct defects after they enter an operational phase than before. Finding and fixing a software defect after delivery is often as much as 100 times more costly than finding and fixing the problem during requirements or design (according to Boehm, Software Engineering, IEEE Computer Society, 2007).

By trading-off testing activity in return for lowered up-front costs, we run the constant risk of introducing more defects and facing dramatically higher defect-correction costs down the road. Weighing materiality and risk becomes a slippery slope — one that a strong quality assurance organization will keep an eye on. Of course there are reasonable levels of risk, but the analysis of what is acceptable risk cannot lie with the testing team or quality assurance team. Risk must be clearly and concisely presented to the business. The decision to trade thorough testing for market-based and consumer-based risk should only be made by the business unit.

Placing the responsibility for this decision with the business unit ensures adequate communication of the issues at hand, and also assures that the potential risk will correctly be weighed against long-term cost. With luck, the business unit will make the right risk versus reward decisions — or, at the very least, learn from its mistakes in relatively short order.

Related posts:

1. What do you mean, SQA isn’t testing?
2. Quality assurance as a way of life
3. Fix your boss (or, reduce risk to quality using a matrix approach)

It turns out that when you search for defects determines how much it costs to identify and fix those defects. The formal inspections process emphasizes discovering defects early in the software life cycle. This is reflected in the average industry costs associated with defect resolution:

• Without formal inspections:
• 50 defects found at testing stage
• 10 defects found after the project ships, at operations phase
• Average industry cost: $700,000
• Using logic and code inspections only:
• 35 defects found in logic and code stage
• 25 found at testing stage
• 5 found during operations
• Average industry cost: $525,000
• Using requirements, design, logic and code formal inspections:
• 16 defects found in requirements and design stage
• 30 found in logic and code stage
• 13 found in test stage
• 1 found in operations stage
• Average industry cost: $300,000

Attaining such drastic reduction in defect prevention costs did not become possible overnight. Looking back over the history of software development and quality assurance methodology there are clear evolutionary breaks that have fueled strides toward better, more efficient defect prevention. Early efforts in the software industry focused on process definition as a primary means to eliminate defects. By the 1970’s, efforts in process measurement and improvement, combined with formal defect detection techniques, made significant strides in software quality possible. It wasn’t until the 1980’s that defect prevention techniques—such as formal inspections and unit testing frameworks—began to evolve.

Formal Inspection refers to a structured process of trying to find defects in development documents, programming code, specifications, designs and others artifacts during various phases of the software development program. Sometimes called a “Fagan inspection,” after Michael Fagan who is credited with being the inventor of formal software inspections, the process has a dramatic effect in reducing defects early in the project life cycle. This of course translates into dramatic project cost reduction and more reliable schedule projections.

In fact, according to a JPL Executive Briefing given to NASA in the late 1980’s, NASA’s projected budget for the International Space Station software program without the use of formal inspections would top $6 billion—compared to a revised budget of $4 billion after applying the formal inspection process.

So, what are formal inspections and how can they improve your projects?

Formal inspection is a defect detection, removal and correction verification process carried out by a small group during the pre-test phases of the development life cycle. It involves the interaction of the following five components of the software life cycle:

• Well-defined inspection steps
• Well-defined roles for the participants of the inspection (moderator, reader, recorder, author, inspector)
• The software product subject to inspection
• An infrastructure that supports the formal inspections

The primary objective of formal inspections is to remove defects as early as possible in the development process. Formal inspections achieve this objective by:

• Identifying potential defects during individual preparation
• Verifying that identified defects are actual defects
• Recording the existence of defects
• Providing the code author with a list of defects to fix
• Ensuring that fixes are performed and are correct

Formal inspections were designed to help organizations involved in software projects develop better products. While its primary focus is on software quality there are a number of peripheral benefits—dramatic budget benefits being the most obvious. The overall software life cycle cost is lower since defects are found early and are easier and less expensive to fix. By introducing formal software review and testing steps early in the software development life cycle a more technically correct architecture is developed and tested earlier. This fuels improved component reuse and drives development efficiency up. The effectiveness of the testing process and closely-related software quality assurance processes are improved because testing begins early, as well. This generally means that less time needs to be devoted to testing processes on a recurring basis. Another benefit of formal inspections is the immediate feedback on the software from a group of peers and target users. This creates a support loop that integrates well with modern ideas of iterative development and creates an environment where agility and end user feedback become a target of the development cycle.

There is a significant emphasis on the “formal” part of the inspection process. Formal inspections, while designed to be effective without impacting team schedules heavily, are more rigorous and well-defined than other peer review techniques such as pair programming and walkthroughs. Because of this they are significantly more effective, but they do not take the place of milestone reviews, progress reports, status review, testing or walkthroughs. Also, the process clearly defines phases of the software life cycle at which the inspections should take place.

Inspections must be carried out by peers representing the areas of the life cycle affected by the material being inspected. The team should be limited to six or seven people at most, and everyone participating should have a vested interest in the work product. This is a particularly important requirement of the process. Also, inspections must not be used to judge the quality of the software work product or the authors that developed the product. For this reason managers should not be present in the formal inspection itself; findings should be presented to management statistically or as a group of work product findings. This will demonstrate the value of the inspection process without singling out any individual author. Using the inspection to judge the capability of authors may result in less than honest and thorough results: If coworkers feel their efforts could result in a poor performance review for the team they may be reluctant to identify defects.

The Formal Inspection Team

Every formal inspection is carried out by a team of inspectors. The inspection team consists of four to seven individuals, and no more. In fact, the team should be kept as small as possible and yet be able to complete the job at hand. The goal of keeping the team small is to keep decision making and process to a minimum. It is also important to minimize impact to the overall development life cycle—superfluous involvement ultimately means time taken away from productive work on the project.

The inspection team consists of a moderator, a reader, a recorder, the software author and other inspectors. Each team member has a specific, defined role to fulfill. In addition to each individual role, it is everyone’s job to find and report defects; all team members are considered inspectors in this regard. Where necessary, more than one author will be involved in the inspection, although it is best to keep the inspection focused to an area that involves a minimum number of authors.

The moderator’s primary role is to achieve a good inspection. This includes planning the inspection, assembling the team and managing the overall process from beginning to end. The reader’s responsibility is to guide the inspection team through the work product being inspected (keep in mind that “work product” can include specifications, designs, actual software code or functioning software). The recorders job is essentially to record every discovered defect. The author’s job is to answer questions throughout the process, present the work product as needed by the inspection team and, ultimately, to correct the defects. In addition, larger projects may require support from the project librarian in keeping track of status, statistics and overall progress of the formal inspections throughout the development life cycle.

The most important guidelines to keep in mind when creating the formal inspection team are:

• Use a fair and unbiased moderator
• Keep inspection team size reasonable (between 4 and 7 individuals)
• Select inspection team members who have a vested interest in the work product

This last point is a particularly important one. Only those individuals that care about the project will be invested in the inspection process. Team members should be close to the project; peers working in the same life cycle phase, authors of the relevant specifications or code, direct users of the work product, product quality assurance and testing personnel are all excellent candidates.

Supporting Roles

Formal inspections are an organization-wide activity. As such there are several supporting roles that are essential to assure success from the process.

Most visible is the need for support at the Project and Program Management level. This means that the appropriate manager needs to establish a schedule that allows adequate time for all stages of the inspection process. This includes monitoring the inspection process and making sure that there is sufficient inspection preparation time, that individuals are not over-burdened with tasks and that all team members understand the critical nature of the inspection process. Ensuring that all team members are properly trained in the formal inspection process will help in maintain the schedule and making sure that everyone is prepared for an inspection in the alloted time. After the formal inspections have taken place, managers must meet with the moderator and author to review open items and rework estimates. The project manager will likely review inspection summary reports for defect trends and perform defect trend analysis.

Organizations may also find a need for a Chief Moderator. The Chief Moderator will chair monthly meetings of all inspection moderators and guide the evolution of the inspection process, forms and checklists, and information gathering. The Chief Moderator may also analyze the effectiveness of inspections across participating projects, provide support in the form of guidance and answers to questions and keep current on recent information and developments in the formal inspection arena.

Stages of a Formal Inspection

The formal inspection process is broken into seven stages, although two of these are optional and can be excised from the inspection process on a case-by-case basis, at the moderator’s choice. These stages are:

• Planning—The period of time used to determine whether the product to be inspected meets the entry criteria, set the inspection schedule, plan the inspection itself, select a team of inspectors and assign respective roles, and prepare and distribute the inspection materials. This is when the moderator decides whether an overview will be necessary, as well.
• Overview—An optional stage in the inspection process. The overview provides the inspection team with background information for the inspection. This stage may not be necessary if the team is already intimately familiar with the work product being inspected.
• Preparation—A critical stage during which each member of the inspection team individually prepares for the inspection. It is crucial that individual inspectors be given adequate time to prepare, otherwise the inspection process will not be efficient and may well fail to identify defects that could otherwise be discovered. Each team member prepares for the inspection by reviewing and finding potential defects in the product being inspected before the inspection meeting. Potential defects are then discussed during the inspection meeting as a group.
• Inspection Meeting—Meeting in which team members, as a group, review the product to discover, categorize and record defects. Defects are not resolved during this meeting.
• Third Hour—Literally, a third hour to the inspection meeting (the formal inspection meeting is limited to two hours). This is an optional additional time that can be used to discuss, possibly solve or further investigate defects that have already been discovered during the Inspection Meeting.
• Rework—The period of time that the author uses to correct identified defects.
• Follow-up—A short meeting between the author and the inspection moderator used to determine if the defects found during the inspection have been corrected and to ensure that no additional defects have been introduced.

The following few sections discuss each of these stages in more detail.

The Planning Stage

Chiefly, the moderator uses the planning stage to prepare for the inspection process. This entails making sure the product is ready for inspection, selecting the inspection team and assigning team members appropriate roles, scheduling the inspection meeting and making sure meeting facilities will be available, and distributing inspection materials such as forms and background information. The inspection materials should also include detailed information on the work product being inspected, the scope of the inspection and any specific components or functionalities that will be excluded from the inspection process, and individual checklists for each inspection team member to follow.

Another key role the moderator will play is making sure that the work product being inspected is of appropriate size for the inspection. If not, the product is divided into manageable pieces and separate inspections are scheduled for each piece.

The moderator also must decide whether the inspection team members are adequately familiar with the material to be inspected or whether an overview must be held. The team should know the background material from which the product was derived and should know how the material fits into the overall system being developed.

The Overview

This is an optional stage that chiefly flows out of the moderator’s work in making sure that the team is adequately familiar with the work product being inspected. If any team members do not have sufficient depth of knowledge about the product, an overview is warranted. At the overview, the author will present the rationale behind the product, its relationship to other components in the system, its current state of development (e.g.: how complete it is and how well integrated into the overall system), its function and intended use and how it was developed.

An overview should be scheduled if any of the following criteria exist:

• The inspection team is not familiar with the product
• The product is new or is being inspected for the first time
• Inspections are new to the project
• Novel techniques have been used in the product

Preparation

This is the most critical stage of the formal inspection process. During the preparation process, each inspection team member individually prepares for the role in the formal inspection meeting. Each inspector will review the work product thoroughly; user interface inspections will be conducted field-by-field while code level inspections will be conducted by reviewing the work product line by line. Higher level specifications should be reviewed by reading the specifications and comparing them to system requirements and expected system behaviors. Comparison against relevant work product and best practices should be made during the preparation process, providing a basis for quality standards and acceptable behaviors. Checklists should be used in the work product review for guidance on typical types of defects to be found. All potential defects are carefully logged and will be discussed during the formal inspection meeting. Complete preparation logs are submitted to the moderator prior to the meeting.

Prior to the inspection meeting the moderator will review preparation logs and draft defect reports. This review will both ensure that the inspection team is ready for the formal inspection meeting and also highlight any areas that may require additional attention. If the logs indicate that the team is not ready, the inspection meeting should be rescheduled. Areas of common concern between inspectors can often provide valuable guidance on areas of the work product that require particular scrutiny.

The Inspection Meeting

The inspection meeting is the period during which the inspection team reviews the work product as a group. The meeting is held to a limit of two hours and is carefully moderated. The meeting is run by the moderator and usually starts with a very brief restatement of the goals of the formal inspection. If an overview has been scheduled it is held prior to the formal inspection meeting itself.

During the meeting, the reader interprets and provides a reading of the work product, the author clarifies information as needed, and the team identifies defects that are then classified and recorded by the team recorder.

The reader’s description should note the function of components being review (whether paragraphs of text, code blocks or a functioning user interface) and their relationship to the product and higher level documents. The moderator is tasked with keeping the meeting on track, but the reader can be interrupted at any point a potential defect enters into the discussion. A short discussion of the defect is permissible, but the moderator should limit time accordingly. Any issue that cannot be resolved within the time limit will be marked as an open issue and can be revisited during the third hour.

The team should reach consensus on whether a potential defect is, in fact, a real defect. Quite often what appears to be a defect could be a mistake on the part of the inspector. At the same time it could be a flaw or deficiency in the requirements or other specifications (in fact it is very common to discover edge cases that were never considered during the specification and design process). These are considered defects as well and must be logged as such, even though resolution steps outside the bounds of the author’s work. During this process the recorder will note down every defect, its reproduction criteria or location as appropriate, and classify the defect. Ideally, if defect tracking software is readily available and defect entry is efficient enough that it will not slow down the inspection meeting, direct entry into the system is favorable. This provides the team with an opportunity to review the defect as it is recorded, avoiding possible interpretation or notation mistakes.

If the work product cannot be fully inspected within the two hour limit, to avoid fatigue and missing defects, the moderator should conclude the meeting and schedule a follow-on inspection at a later time.

The moderator will conclude the meeting with a brief recap of the number of defect found and the severity of the defects. All information should be recorded to track the effectiveness of the formal inspection process and to look to defect trends at a later date. If a third hour is needed, the moderator will assign action items to individual inspectors at this time.

Third Hour

The third hour is used for discussion or to close open issues—it should not be an extension of the formal inspection meeting. The third hour does not need to take place immediately after the formal inspection meeting either, nor does it need to be limited to one hour: It should be scheduled at a time when the author is prepared to discuss corrections to discovered defects or if major issues, such as defects or omissions in the higher level specifications, have been discovered. Attendees to the third hour are free to obtain support from outside sources or invite third parties, including managers or technical experts, to the third hour purely for technical reasons.

Rework

The rework period is used by the author to correct discovered defects, or to coordinate the correction of defects that fall outside the author’s domain (for example, if the defects occur in a higher level specification document). The author is responsible for ensuring that all defects have been corrected before the follow-up period with the inspection moderator.

Reinspection

This is an optional stage; the moderator and the team will jointly decide whether a reinspection is warranted based the inspection meeting findings. Reinspection may be required when there are a large number of defects in the product or when one or more defects require extensive or complicated corrections. Reinspection allows the changes to the product to be reviewed by the entire team instead of just the moderator.

Follow-up

The goal of the follow-up meeting is to confirm closure on major defects that were discovered during the formal inspection and that no secondary defects have arisen during the process. The meeting is held between the moderator and the author. The author reviews the steps taken to correct the defects and the moderator ensures that all open issues have been addressed. Minor defects do not necessarily need to be addressed in the follow-up meeting. Additional reviewers can be present during the follow-up meeting if either the moderator or author feel it will benefit the process.

Once all open defects have been corrected and any other open issues have been resolved, and once the work product itself passes the exit criteria (this could be reaching a compile state or passing quality assurance testing, depending on the nature of the work product) the moderator marks the work product as having “passed” the formal inspection. The moderator will note the product as having passed the inspection on the formal inspection summary report.

Scheduling Guidelines

The scheduling of formal inspections throughout the software development life cycle targets specific phases to optimally eliminate defects as early as possible.

Traditional defect discovery and removal techniques target defects significantly later and less frequently than the formal inspection process. Expectations with traditional methods are correspondingly limited: Experience is that 60 defects escape pre-testing phases for every 1,000 lines of code written. Traditional testing steps are each approximately 50% efficient in the identification and removal of defects. Provided a typical, traditional quality assurance method that emphasizes testing of a coded and working product, followed by repeated late-stage testing, an average of 3.75 defects per thousand lines of code reaches the operational phase—and ends up in customer’s hands. The following figure demonstrates the effect of traditional testing steps on the development process:

While traditional methods tend to focus on reviewing and testing completed software, formal inspections begins the process much earlier at the requirements, specification and coding phases. In comparison to traditional defect discovery and removal this is much more effective at identifying defects early and correcting them before they become a problem. The formal inspection process achieves this by:

• Inserting inspections into the pre-test phase of the software life cycle
• The strategy is to find and fix defects when and where they are injected
• Inspections mean more defect detection steps: At least 9 as compared to 4 traditional steps

The frequent and repeated inspection process also keeps defects counts manageable, as shown in the following figure:

The formal inspection process introduces inspections as early as possible into the software development life cycle, and continues to repeat the inspection process at key stages of the life cycle. This means introducing formal inspections after the program functional design, requirements, architectural design, detailed design and software coding stages. Iterative projects can take advantage of additional inspection steps after operational code delivery as well. Note that source code level inspections should take place before unit testing, as the corrections made during unit testing can hide larger scale defects that should be exposed.

There are two striking differences realized between the formal inspection process and traditional detection process: First, the average defect rate in operational systems drops from 3.75 defects per thousand lines of code to 1.05 per thousand delivered source instructions, about one fourth the defect level. Second, because defects are identifying and corrected early in the process total defect counts remain much lower throughout the software development life cycle. This has a positive, cascading effect throughout development: Fewer defects means fewer negative behaviors become “coded in” to interfaces—and that means codependent defects are also kept to a minimum.

This implies that formal inspections are about 7.4 times more productive than formal testing practices alone (based on 3.25 errors per unit effort with inspections compared to 0.44 errors per unit effort with testing, given that inspections yield a lower return to effort against testing).

Planning the Inspection

Inspections should not be cumbersome. In fact, the formal inspection process is designed to be quick, efficient and require surprisingly little preparation time. Even so, it is important the the inspection team be well trained in the inspection process and its benefits—and fully realize the relative effort that can be saved through formal inspections.

The inspection moderator will invest more time into the process than other team members, chiefly because of the advanced planning and preparation that goes into each inspection. Likewise, the author is likely to invest more time because of the rework phase—however, if we discount the actual defect correction activities, the author’s investment in the inspection is largely in line with the rest of the team: Only a matter of a few hours. Following are the approximate time estimates recommended in a single formal inspection:

• Planning (moderator): 2 to 4 hours
• Overview: Between 30 minutes and 1.5 hours
• Preparation: 3 hours per inspector
• Inspection: 2 hours maximum
• Rework (author): Up to 5 hours
• Follow-up: 2 to 3 hours

Keeping the inspection process tightly focused on the work product and on defect discovery will help keep the inspection on-track and minimize time taken away from other tasks.

It is crucial that the inspection team be fully trained in the formal inspection process. The moderator and project manager must also make sure that there is adequate time available for the inspection, and that the team has properly prepared. If there is less than five hours available for preparation time, the inspection should be rescheduled.

Summary

Formal inspections have proven themselves through the test of time. We have an extensive body of work that demonstrates their value in effort, time and cost savings across many projects. Reports produced by organizations such as the IBM Federal Systems Division (1986) show that the formal inspection process can find between 75% and 90% of all defects at early phases of the software development life cycle. And, we can look forward to a commensurate reduction in maintenance costs: As much as 90% of corrective maintenance is eliminated (Ackerman, 1989).

In fact, formal inspections are the single most important quality improvement technique a development can make, according to experts at IBM, Bell Labs, the Software Engineering Institute and other sources.

As well as these very tangible benefits, a number of less tangible benefits come from the inspection process. Frequent inspections aid in project tracking accuracy and reduce the effort involved in traditional project management. The frequent deep-dives into the system also works to foster better team communication and collaboration on the work product. This, combined with the iterative cycle of inspections works against repeated mistakes, fueling better software quality and team development.

All of these benefits conspire to produce a better product, at a lower cost and within a shorter time than would otherwise be the case.

Related posts:

1. Quality assurance as a way of life
2. Don’t ship broken software
3. Exposing the enterprise to risk: Who decides what not to test?

Managing software quality is not simply creating a test program during a late-phase testing period. In fact, addressing quality assurance in this way is too little, too late. This far into the software life cycle, defects have become an intrinsic part of the architecture.

Quality begins with examining project goals early on, assessing the quality of requirements gathering and business goals, and developing an execution plan that addresses system architecture as well as specific feature-level testing.

This is a process that begins even before engineering becomes involved in product development.

Quality assurance, as a department, needs to be an independent organization from the development group. It must become involved in the early planning phases of every project. The influence of the quality assurance team covers the entire spectrum of the software life cycle:

• Customer goals and business goals are managed by the quality control process, making sure that the goals of the project are aligned with the same.
• The development case is reviewed early by quality assurance, making sure that development goals fit in with overall architecture and planning goals.
• Planning for acceptance testing begins early and delivers clear-cut, specific acceptance criteria that guide the project from the beginning.
• Acceptance tests become the basis for the development case and, therefore, the roadmap to which development executes.
• Test planning begins at the same time as development execution — this means that the first operational unit of code is tested early, and often.
• As the product life cycle is taken into account, changes to architecture are discussed and accommodated throughout the team. This leads to platform testing, advanced configuration management and delivery planning — again, early in the life cycle.
• Configuration management, platform testing and platform certification become early concerns that are addressed before release.
• By controlling the entire life cycle, quality assurance lowers overall defects and makes sure that customers experience a minimum of problems.
• Quality assurance remains involved after release, for example, providing a key role in diagnosing customer-facing defects, preparing reproductions, and managing the defect resolution process. This assures that bugs are fixed once and enter into regression testing thereafter.

This life cycle oriented approach addresses quality early and continuously, leading to higher quality software and lower defect counts. Contributing to the overall quality of software becomes an essential role. Likewise, with quality assurance being addressed externally, development can focus on its core competency: Implementation.

Related posts:

1. Fix your boss (or, reduce risk to quality using a matrix approach)
2. Don’t ship broken software
3. Formal inspections: An introduction