So is Agile just hype or is there something to it?

There are strengths to the agile way of thinking, and many of them bring useful perspectives to software and systems development that are new and even revolutionary. Here are some of the things that work — and, potentially, that radically change our old-world practices.

Whereas most legacy methods stem from industrial process — that is, assembling a product using a set of defined, predictable steps — the agile method is empirical. It recognizes that development is more like invention and research, more akin to scientific study, than assembly. This empirical nature is at the heart of the agile mantra: Deliver, measure, adjust and repeat. The strength of this approach often bears itself out in fantastically hyperproductive teams that deliver working product far more quickly than legacy methods, such as waterfall, could ever achieve.

Agile does this by cutting through complexity. Every agile-based methodology focuses on simplification of otherwise complicated problems. For example, XP and Scrum both emphasize development of near-term, complete deliverables. This means carving out tangible and reasonably independent pieces of work, focusing on that work, and then — at least as much as possible — moving on to other work. This approach requires that large, complex problems are broken down into manageable pieces and thought of on a micro-deliverable level. Likewise, this approach minimizes ceremony and eschews as much procedure as possible. Some agile methods go to extremes in this regard, focusing entirely on delivering work product and not at all on procedure. This translates into minimizing complexity on a large scale.

Closely related to eliminating complexity is agile’s focus on progress measurement. Most agile methods measure progress chiefly, if not exclusively, in terms of delivered work product. Most methods also are quite stringent in defining progress only when finished work is delivered, which means you can’t work for nine months on a single big feature. Instead, micro-deliverables target key features, deliver those features into the customer’s hands, then moving on to new features. This can be a huge strength because the customer gets working product in-hand to review early, and often. It involves the customer early in product evolution, leading to a host of benefits including better product targeting, prioritized development and improved quality.

These characteristics of agile methods combine to fundamentally change the way software and systems development is practiced. Agile also empowers individuals to become stellar performers. In fact, all forms of agile rely on this to some degree — with more lightweight agile methods being completely dependent on individual empowerment. The idea is that an empowered team will leap over constraints to get the job done, no matter how “out of the box” the thinking needs to be. It’s a refreshing concept and one that can indeed be supremely successful, but it does require the team to embrace the idea wholeheartedly.

Another benefit, at least in some situations, is the creation of self-organizing teams. Partly because of the light ceremony, the fast pace, and the penchant for empowerment and accountability, teams become self-organizing. This works when the team has the right make-up, as individuals step up to take on tasks best suited to individual skills. Self-organized, empowered teams become very powerful and very productive, provided that the team members are up to the job.

There is absolutely no doubt that agile methods make it possible to get things done quickly. That’s what it’s all about, after all. The real question to me is how much of this tradeoff is really desirable? How often do we want to eschew process and maturity in favor of getting things done quickly?

More importantly: Can we effectively merge the best attributes of Agility with the most valuable benefits of established processes and standards?

Why Agile doesn’t work

When an agile project fails, it generally does so spectacularly and predictably. The common failings of agile-based projects are just that… common. We see the same problems over and over again, and this has become the basis for many critiques of agile methodology. After all, if we keep seeing the same problems crop up again and again, isn’t this proof enough that the process is flawed? This becomes clear in hindsight, so why do we continue to see 90 percent of projects missing the mark?

The fact is, agile by itself is just one tool in the toolbox that should be applied with other implements of the trade. In my experience, the problem comes in most often because small- and mid-sized organizations experience brilliant success with agile and then assume it can work everywhere. They throw out the toolbox (or perhaps never buy one in the first place). Yes, agile can succeed. Yes, it can deliver fantastic productivity and stellar results. But not always — in fact, I will go so far as to say not often.

This isn’t because of agile’s limitations. Instead, it’s because of overconfidence by those putting it to use, and the mistakes an immature organization makes as it grows and applies it inappropriately.

Immature companies and teams are cutting their teeth, again and again, on the limitations of agile.

All agile methods make it easy to oversimplify complexity. In fact, agile’s strength of eliminating complexity might be better stated as “ignoring complexity.” There are appropriate situations for this but, more often than not, ignoring complexity leads to problems. Most business cases don’t call for undefined delivery dates or loosely changing requirements and partial deliveries. These are risks that most business models are incompatible with. If the risks aren’t something that your business can sustain, adopting a purely agile process is taking a huge gamble.

Likewise, focusing on the near-term is an agile attribute that introduces a lot of unknowns into the business-end of an equation. Few people will contend that agile is appropriate for mission critical efforts such as, say, launch vehicle development, as sometimes requirements need to be set in stone before anyone starts development. But what about situations where some degree of fuzziness is acceptable or even beneficial? Agile advocates compatibility with change, sidestepping change control procedures that would otherwise place tight controls over requirements. Requirements change carries with it a heavy burden, particularly when it comes to the cross-organizational impact to marketing, budget, quality management and the customer. However, cutting change control, requirements management, and configuration management from the process can lead to long-term disaster that the short-term perspective of most agile methods will overlook.

This theme of reducing structure and control has cut out many waterfall-origin processes. The danger often manifests as small-scale agile projects are successful, leading to wider-scale adoption of agile. But, as the projects grow in complexity and criticality, major missing components in the process become evident. For example, no agile methods today integrate comprehensive quality assurance procedures (in fact, thanks to some early mistakes, such as MIL-STD-498[#], most people think quality assurance is software testing — it’s not). Structured software testing often becomes an afterthought, and risk management programs tend to be regarded as “fuzzy disciplines.” Yet, these are the processes that successfully put man on the moon, that develop health care and financial services systems, and ensure that nuclear plant regulatory systems don’t fail after delivery. Of course, there is a cost to each of these processes, and every business needs to weigh the cost-benefit of adopting more process against cutting those processes. This needs to be an on-going evaluation, made as projects, organizations, and teams evolve — it’s not a decision that stands alone.

From a purely hands-on, management level, agile methods pose “people problems” as well. The strong emphasis on self-organization and empowerment can easily backfire. The former relies heavily on people that are capable of self-management and self-direction. Not everyone can live up to that expectation. The latter, delivering empowerment to the team and individual, can lead to a hero mentality and silo’d teams that refuse to play well with others. As projects grow in size, complexity, and dependency on other teams and resources, these characteristics become the drawback of an immature organization.

Almost all agile methods oversimplify valuable processes. In some situations, the project survives the oversimplification. Sometimes the business is tolerant of the fallout. In every case, agile methods expose the project to risks that stakeholders should be — and often are not — aware of.

What to do about it

We need to be cognizant that one solution does not fit all problems. While an agile method such as XP or Scrum may have led to success in one project, this doesn’t make it a foregone conclusion that it will do so again. Each project is different, and organizations evolve over time. Adopting one process to solve all problems is a sure recipe for failure. On the other hand, having a well-versed team that can draw on several methodologies, as appropriate for the job, is a recipe for success.

If your organization is looking for the one-size hammer to hit every nail, make sure it’s as configurable a hammer as possible. Don’t choose something that is either too lightweight, such as XP, because many projects will overreach the capabilities of such a lightweight process. Likewise, don’t try to implement a full-on waterfall style methodology either because, while definitely thorough and capable of getting the job done, it’s just overkill for many smaller projects. If you must choose a single process, pick one that’s efficient, borrows from both agile and waterfall, and is highly configurable, such as Rational Scrum or the Rational Unified Process. Both of these have the maturity to deliver large-scale projects, but also support starting small and adopting minimum ceremony.

A better awareness of what specific agile practices can and cannot accomplish is key. For example, Scrum is not a development methodology, and it cannot effectively deliver software or hardware projects unless it wraps itself around one. Yet today many organizations are employing Scrum as if it were a development methodology. I’ve even seen an organization of several hundred developers “force fed” Extreme Programming from the top down. The outcome of that particular operation: Mid-level management hid the fact that they didn’t use XP from top-level management after everyone realized what a mistake it was. Perhaps we’ll have to wait for mature standards in education and certification to evolve, but personally I’m not sitting idly by.

One of my personal pet peeves in the technology industry is a relative lack of standards and qualifications. Would you go to a doctor that didn’t have a medical degree? Would you hire an architect that didn’t have an appropriate engineering degree? Yet we hire software professionals (much less often hardware professionals) without adequate education, current qualifications, or meaningful certifications. For that matter, the proliferation of meaningless qualifications (such as Scrum Master certification) continues to weaken the industry. In the long run, we need better standards regarding education, accreditation and certification.

Understand agile methods for what they are. Keep in mind that lightweight process carries risk. Use the right tools in the right situation.

Coming full circle

If we add all of these things to agile methods, won’t we just end up using waterfall process all over again?

I don’t think so. Waterfall-based process, the original behemoth processes born out of industrial process, are widely recognized as inefficient. There are tremendous advantages to pressing forward with a merger between waterfall practices and agile practices. I hope the end result is a new generation of software and hardware development methodology — a generation that we’re just starting to see as processes such as Rational Scrum come to the fore. It’s time for development methodologies to evolve, and there’s no holding that back.

Related posts:

1. Making Scrum work: Common failings in adopting Scrum
2. Scrum is not an agile methodology
3. Common oversights in choosing methodology

I don’t remember where I heard the story about a truck driver named John Barstow. Nearly every day of his life he would drive down Main Street making his delivery to a local store. His goal was actually on 4th Street, which was a one way street and he needed to go about half a block in the wrong direction, so he would pass 4th and turn right on 5th Street, go around the block, and pull up in front of his destination.

One of these many deliveries days, something unexpected happened on the way to his drop off. As he approached 3rd Street, his delivery truck blew a tire. He had to pull over and, upon climbing out of his truck and inspecting the damaged tire, realized he needed to call a tow truck. Being close to his destination he decided to proceed the remaining block and call for help from the store where he made his deliveries. He continued on, passing the one-way 4th Street and turning right on 5th, as usual. He was halfway down the 5th Street, getting ready to turn right and circle back on to 4th, before he realized he could have taken 4th Street this time.

He was on foot.

The quantitative decision — the habit of going around the block to avoid a one way street — won out over a qualitative choice, that of taking a shorter route. We get used to set patterns and “business as usual.”

The technology industry is among a small set of disciplines that takes the consequence of making quantitative decisions to an extreme. Software and hardware are both progressing at a remarkable pace that is, if anything, accelerating. We see this every day as new technologies develop and old technologies evolve. The open source community is driving this effect to an even more frantic extreme, as hundreds of contributors pour their effort into a single product. It is, and always will be, impossible to totally keep up with the pace of change and the challenges of an evolving world.

It’s not enough to define our process or methodology and let it settle in. Yes, we absolutely need to have a clearly defined and adopted set of processes and procedures to ensure a good product. And, those processes and procedures need to become a part of our daily lives so that we don’t take shortcuts and miss important steps. But at the same time, it’s important to never let it become too rote. Watch out for doing something just because that’s the way it’s supposed to be done. Challenge yourself to find out how things can be improved on a daily basis.

Related posts:

1. Making Scrum work: Common failings in adopting Scrum
2. Fix your boss (or, reduce risk to quality using a matrix approach)
3. Why Agile isn’t enough (and why it doesn’t work)

Let’s start at the beginning and reiterate this original statement from Schwaber and Beedle, the creators of Scrum:

Scrum is a management and control process that cuts through complexity to focus on building software that meets business needs. Scrum is superimposed on top of and wraps existing engineering practices, development methodologies, or standards.

Scrum is a process. One process, that must be taken and combined with other practices, methodologies and standards. A process does not, in and of itself, create a methodology and for this reason I say that “Scrum is not an agile methodology.” I might even go so far as to say “Scrum is not Agile,” but that’s misleading — because as a process, Scrum is compatible with and enhances agility, either taken as an “Agile methodology” or a general practice of being agile.

As Scrum has become more of a buzzword it runs into this dichotomy more often. I experience this frequently with my clients, where the dialogue goes something this this:

Me: “So, what methodologies do you follow?”

Client: “We use Scrum.”

At this point, I feel as if, say, I’d asked for a bowl of fruit and being given a bit of jam.

Me: “Yes, but that doesn’t answer my question since Scrum is a management process, not a methodology — do you use any development methodologies?”

This last is often poorly received. It’s at this point that I typically remind my client the reason I’m here is because they’re having trouble, and the reason is likely because of poor internal processes.

Methodology is a body of practices, procedures, and rules used by those who work in a discipline or engage in an inquiry. In the context of software and systems engineering, it specifically addresses how we manage the conceptualization, specification and delivery of the software. Scrum does not cover any of this. Scrum is just a management process that can be applied to pretty much any business situation.

A good methodology is going to provide a framework in which requirements capture and management is specified. Project parameters and management criteria, such as reporting structure, authority and responsibility, status management and progress management will be specified. Program assessment goals and objectives will be specified. Assessing organizational capability and planning growth or acquisition may be needed. How requirements are stated, what depth they must go to and what standards must be met, and how to manage change in requirements will be specified. Likewise, quality assurance processes will be set forth (for example, audits and controls will be put in place to ensure requirements meet agreed upon standards). Testing of the product will be standardized as well: What techniques will be used, what are the goals of the test program, and what will the output of the test program be? User acceptance standards should be specified and usability testing programs implemented. Transitional phase operations, often overlooked until “after we’re done,” need to be planned and prepared for — and then executed. Scheduling, resource planning and budget management are of course a key component of any well-run program. And through it all, meaningful measurement criteria must be established and communicated to stakeholders.

Scrum touches only the tip of the iceberg in regard to much of this. This is intentional: Since it’s designed to work compatibly with just about any methodology, Scrum explicitly avoids putting too many constraints in place. More than anything it’s intended to refine an existing methodology and process, improving its productivity through streamlining.

Of course, sometimes Scrum is all you need. A sufficiently senior team, comfortable with the problem domain, combined with a project of reasonable size and a business that’s willing to launch without a clear end-point can work. But more often than not, one of these four essential elements is missing. Perhaps the business isn’t comfortable with an undefined, vague end-point (they may want to know what the budget is, or have a hard launch date). Perhaps the team is new, hence the necessity for more structure and measurement. It’s a rare situation that we can dive into a project with no methodology, only the Scrum process, and come out the other side in a favorable position. Sometimes it will work, but it’s a gamble — and often a gamble that the business isn’t willing to bet on.

I’d like to stop hearing “we use Scrum” in response to the question “what’s your methodology?” It’s great to say, “Rational Unified Process with Scrum to streamline it” or perhaps “XP with a Scrum wrapper so we have better visibility,” but please don’t try to deliver software with “just Scrum.”

Related posts:

1. Making Scrum work: Common failings in adopting Scrum
2. Why Agile isn’t enough (and why it doesn’t work)
3. Common oversights in choosing methodology

Scrum can be remarkably beneficial in many kinds of software projects. But, as with any process, methodology or management technique, when used inappropriately it can cause more problems that it solves.

In this article I’ll discuss some of the common misconceptions and “lessons learned” as related to Scrum.

You don’t need training to use Scrum

This is the most important one in my mind: Training is just as necessary with Scrum as it is with any other process. I’ve run into many teams that eschew formal training in favor of simply “reading the book” or sending one lead engineer to a quick Scrum certification program.

Let’s look at it another way: For upwards of 30 years, the software development industry has followed a largely industrial process (sometimes called “waterfall” or “spiral,” depending on your particular experience). Across that time, many professionals have struggled with the complexity of software development, striving to create a process that is simple, reliable and repeatable. More often than not, to varying degrees, the industry has met with failure: Nobody has turned software development into a wholly reliable, repeatable procedure that always hits its goals dead-on.

Scrum is a process that has evolved out of this 30 year history. While the process itself focuses on simplicity and emphasizes a set of easy to follow practices, this does not make it inherently intuitive or trivial. In some regards Scrum is so contrary to established process that many team members just “won’t get it” unless they experience it in a learning-enabled setting. Training is critical, and I highly advocate sending your entire team to a Scrum workshop — or, better yet, bring the workshop to your team. Good programs exist that will turn your training session into a guided working session, which means your productivity will not be hurt by sending people off-site for training. Just the contrary — your productivity will leap ahead in just a week or two as training, learning absorption, experience and actual hands-on work all combine to create a Ken Schwaber’s original goal for Scrum: A hyperproductive team.

Scrum is all we need

Often sung to a Beatles inspired tune, I’m afraid it’s not true: Scrum is not all you need, and any idyllic fantasies otherwise will be met with dashed and broken hopes on the rocks of failure. Scrum is not a complete methodology, it’s a management control process — and that implies that you still need methodology. Don’t take my word for it, take Ken Schwaber’s, the creator of Scrum: “Scrum is superimposed on top of and wraps existing engineering practices, development methodologies and standards.” (Schwaber & Beedle, 2002; Agile Development with Scrum, p. 2).

Scrum is an excellent control process that helps us cut through the red tape of more traditional, bulkier control processes. It helps us to focus in on what’s important. It enables accountability to an extreme degree and, through accountability, achieves greater productivity. Its empirically driven measurement technique is well suited to the often experimental development environment of new products, such as inventing new software.

Nowhere in the Scrum process itself do we discuss development methodology. This means we must bring our own methodology to the table and wrap Scrum around it, enhancing processes that have not been hyperproductive and increasing their productivity. In other words: Software quality assurance, formal testing, configuration management, risk analysis, budget planning and project scope planning, verification, requirements management and specific methodologies such as the Rational Unified Process, Extreme Programming or Spiral programming must also be part of the equation.

The backlog replaces the Project Design Document

Here I’m essentially belaboring the point that Scrum is not all you need. Different development methodologies employ different kinds of design strategies and documentation strategies, but one thing is common across almost every single methodology: There is always a Project Design Document (PDD) in one form or another. The project backlog is not a complete design document. A well-implemented project backlog is going to provide enormous technical detail and will probably fulfill all the requirements for a technical specification — but this is only part of the PDD. Team collaboration is not a substitute for someone putting the product design specification in writing.

The Project Design Document addresses a host of project objectives and requirements that simply won’t show up in the project backlog, at least not in an easily interpreted, cohesive fashion. Some of these include: Overall project scope; market objectives; user interface models; business models; user acceptance criteria; original customer requirement statements or vision statements; a project baseline; key risks and mitigation strategies; user interface style guides; software architecture and software integration procedures; and the list goes on. Many of these artifacts will, in time, turn into very specific project backlog items, but many will not. It’s important to document not only the specific, day-to-day deliverables of the project, but also the overall goals, architecture, and design guidelines.

One of my favorite product suites for managing both the product backlog and project documentation is Atlassian’s JIRA and Confluence. Combined, the tools provide fantastic support for managing tasks and deliverables (the project backlog) and a collaborative documentation environment. The ability to seamlessly create links between the JIRA tickets and more in-depth, collaborative Confluence documentation is an excellent tool. It allows the team to expand on technical detail and cross reference extended documentation, all within the context of the product backlog.

The 15 minute daily scrum is unnecessary

I’ve run into a lot of resistance to the daily scrum, especially very early in the process of introducing Scrum to a new team. Nevertheless, within a matter of a month or two, the value of the daily scrum has been realized and wholly adopted by all of the teams I’ve led.

One of the worst situations you can get into is a “fuzzy scrum,” one that doesn’t start on time or that exceeds the boundaries of the meeting on a regular basis. If the Scrum Master shows up five minutes late, then politely waits ten minutes for everyone to gather, a focused 15 minute briefing turns into a 30 or 40 minute distraction that everyone resents. Stay focused on the daily scrum rules of conduct — specifically, start on-time every day, in the same meeting place. The Scrum Master must arrive early and be ready to kick off precisely on time. If someone is late, don’t hold up the meeting. Use a timer to casually ensure that everyone gets a turn, and nobody goes significantly over their time limit.

A well-structured 15 minute daily scrum will benefit the team enormously. It keeps the entire team apprised of progress and current goals. It helps to focus the team, avoiding situations where one person gets derailed, instead giving them support from the team. Remote workers and partner-vendors will stay in-touch with current goals.

I can’t count the number of times something has popped up in a daily scrum that resulted in days, potentially even weeks of gained productivity. I’ve seen team leaders that sit next to their developers react with surprise when they hear what an individual is actually working on. In situations like this, enormous effort has been saved when the team leader corrects the objective right then and there.

I’m not allowed to work outside of the sprint goals

Another very frequent misconception is that Scrum introduces artificial roadblocks; for example, forcing the team to work on current sprint goals and nothing else. This is only partially true: It is correct that Scrum focuses the team on the current sprint. The objective is to complete the sprint and then move ahead to the next set of objectives. This means that an individual team member must finish their sprint goals and, ideally, try to help out other team members with other goals, if possible.

However, once the team member is done with their sprint goals they are free to “look ahead.” There is nothing that says a team member can’t start working on the next sprint ahead of schedule or — if the next sprint doesn’t have enough definition yet — even draw directly from the product backlog.

Scrum is about productivity. It will never require the team to stop work because of an arbitrary set of procedures or policies. Quite the opposite: Well implemented, Scrum enables the team to become far more productive than previously possible.

We don’t have to adopt all of Scrum

This issue has come up with every single client and, I fully expect, will continue to do so. Technically, it’s true — but not initially.

As with any process or methodology, it’s critical to understand how the process works properly before customizing it. Scrum should be followed entirely and rigorously for at least the first month or two of a project. Ken Schwaber’s recommendation is to do so until “several sprints” have been successfully delivered and the goals of achieving a hyperproductive team are met. As Clara Ko writes in Jeff Sutherland Talks Shock Therapy For Scrum, “Perhaps there are good reasons for excluding some parts of Scrum, but without truly understanding the reason behind a Scrum practice or the implications of skipping it, agile teams struggle to become hyperproductive, or mistakenly, struggle with Scrum itself.”

In other words, before changing Scrum, become an expert in it. Once the Scrum process has become second-nature it becomes much easier to begin customizing it or cutting pieces of it out. It will also become clear why something stopped working smoothly. Past experience will give the team the awareness to know the process broke because it was changed, and the motivation to move back to a working process.

With proper training and guidance, teams uniformly experience incredible productivity gains with the Scrum process. They key is adopting Scrum correctly, and that means doing it with the right level of training and commitment, especially early on. Process change is never easy, and while Scrum may appear to be a simple process it is not an intuitive one.

Related posts:

1. Why Agile isn’t enough (and why it doesn’t work)
2. Common oversights in choosing methodology
3. Scrum is not an agile methodology

A ScrumButt is a sort of like Scrum implementation… but some changes that were too painful have been left out… Companies in this category tend to only experience moderate success with Scrum, i.e revenues up 0-35%. This is very different from the design goals Jeff Sutherland had for Scrum, i.e. to create hyper productive teams and hyper-profitable companies.

In 2005 Bas Vodde, agile pioneer and coach within Nokia Networks, created the original test. In 2008, Scrum co-creator Jeff Sutherland extended the test by introducing new questions and a scoring model. Today, Jeff works with openview venture partners to help investors make money by only investing in companies with state of the art software development capabilities.

The test itself is incredibly informative — and the results usually much more so. Merely answering the questions posed in the Nokia Test will reveal some obvious shortfalls in most Scrum implementations. But take care to answer honestly and objectively. It’s not a bad idea to make the exercise a team activity, reaching serious consensus on the answer to each question.

Related posts:

1. Making Scrum work: Common failings in adopting Scrum
2. Scrum is not an agile methodology
3. Rational Scrum

Just as people have specializations, so do processes. Applying one process to all situations is just as wrong as calling your dentist when you need brain surgery.

Rational itself is an excellent methodology and scales very well. Starting with only a few Rational artifacts, it can almost feel like Extreme Programming. Yet, the Rational body of work provides a framework that supports a much more comprehensive methodology… something less than full-on Spiral development, but still robust enough to handle large-scale, distributed team development.

However, for all that I like Rational, it does have some holes… and these are plugged most wonderfully by Scrum. In fact, Rational and Scrum benefit each other so well I’ve started referring to the combination as “Rational Scrum.” As with most methodologies, it does not define every possible response to every possible situation, but the combination of these two techniques is very robust and complete.

The processes complete each other by addressing mutual weaknesses. Scrum steps in to fill a gap in organization and team management. Conversely, Rational brings a structured, risk-driven approach to Scrum that is lacking.

Most recently I was able to put the principles of Rational and Scrum into effect with a client. The initial engagement was, let’s say, “unfortunately typical.” The client had about 45 technical staff working in a disorganized fashion on unclear requirements. Most of the work was outsourced to a third party, further distancing the team from key business knowledge. Product releases had been very slow or nonexistent for the past year—what releases there were had been seriously compromised in quality. Development felt isolated from the business and quality assurance was a poorly funded afterthought. The production support group of about five people were out of the loop on software deliveries and constantly scrambling to stay on top of customer driven emergencies.

My first task was to get my mind around what was going on. I took a trip to the outsource vendor (amazingly, this was the first trip anyone from the company had taken to the vendor) to get to know the team. I spent several weeks understanding the issues and bringing the team up to speed on what the business was really all about. Because of time differences (about thirteen hours) this sort of cross-pollination had never happened. Ideally, I would have set up a program where the entire team would have rotated through on-site “shifts” of several weeks duration, but budget was limited. This would have created a far more integrated team and given each person on-the-ground experience with the business, its goals, its priorities. I also delivered some training on Rational methodology in a compressed one-week course to the entire development team.

One of the most glaring problems was a complete lack of information share between the business and the development teams. We immediately purchased and implemented task and defect tracking (with Atlassian Jira) and a document repository (with Atlassian Confluence). Effective immediately, all information had to be migrated to the new system. If it wasn’t in there, it didn’t exist. I went so far as to return, unread, any technical email or document I received unless it was in Jira or Confluence. Absolutely no defect, feature request or deliverable could be discussed until it was put into Jira, given a once over to make sure the issue was stated clearly and, in most cases, given a quick first-pass effort estimate from development.

We also started having daily Scrums. This proved to be the most difficult challenge—the time difference pretty well necessitated a tight meeting window. Once back in the U.S., I ended up scheduling nightly telephone calls with the development teams from 8:00 PM until about 11:00 PM every day. This went on for months, but was necessary to drive improvement. We typically kicked off an evening with the “management Scrum,” limiting it to no more than half an hour (I doubled the time limit because it was virtual, organization was difficult, and at least initially there were a lot of issues to work through). Once the management Scrum wrapped up, splinter Scrums started up with each of the individual development, quality assurance, testing and professional services teams. Even following Scrum guidelines to keep the meetings on-track, we initially ended up with plenty of overflow. This was followed with a management Scrum on U.S. timetables, the following morning. Initially, it was a grueling meeting schedule for me, but we kept individual team member involvement to a minimum—preferably one daily Scrum.

Another major change was implementing the Rational Unified Process. This took upwards of six months to effect as we introduced light processes initially, adding to the Rational methodology over time. A few of the most important aspects of Rational that we adopted early on were risk-driven development and truly iterative releases. While development had been using iterative terminology, the fact of the matter is that monolithic development practice was still being followed (for example, creating massive specifications and trying to build to the specifications rather than focusing on short-term release goals and continuous improvement). By focusing the team on short-term releases, iterative release cycles and a product backlog (all done in Jira and Confluence) we quickly raised the level of understanding, across the board, of what was coming and what it would take to deliver. This information started to flow throughout the organization and we started to see its effects in how product features were prioritized, how releases where scheduled, and in customer service and the product life cycle itself. Soon, the company went from half a dozen silos of locked away information to a single, relatively smoothly operating organization with across-the-board visibility.

During this evolutionary phase of these changes we discovered all kinds of interesting things were going on. We found separate development teams working on the same foundation technology. Optimizations that could be made with small shifts in the business goals. Streamlining and efficiency gains by changing our release cycle. Massive effort savings by disregarding a few relatively unimportant features. More streamlining that came from the information sharing. The product backlog started to become self-prioritizing as business owners and other stakeholders participated in setting the upcoming release goals. Distractions, fire-fighting, context-switching and misdirection disappeared and the development team started producing true forward progress for the first time in a couple of years.

In fact, the team went from essentially zero forward progress to a steady, major new software release every six months.

We also trimmed operational costs by about $1.2 million dollars per year. Of the 45 outsourced development staff originally on the project, we reduced the count to fifteen in less than a year. And they still kept producing a major new software release every six months. Despite a team size of about one third, we were delivering more and getting more done in less time.

We implemented improved quality assurance measures and actually increased the size of the software testing team from one person to three (all within the 15 person head-count). Software quality started to improve at a dramatic rate.

Through Rational Scrum we had realized a productivity increase of somewhere in the vicinity of 300-400% if we account for the team size reduction. We had achieved positive forward progress where it had been lacking for well over a year, if not two, and had done so after cutting the development team down to one third of its original size.

Toward the tail end of my involvement, after about 18 months on the project, the team size had been scaled back a bit more. The daily Scrum meetings had become smooth and unintrusive, going from two or three hours down to about 15-20 minutes per team. New technology teams focusing on completely new product lines were being hired and integrated into the process very effectively. Even the professional services group had adopted the process and, likewise, had reduced its staffing requirements from five to two and was operating more smoothly. Customer satisfaction was up and the fire-fighting had pretty well come to a stop.

The two processes—Rational Unified Process and Scrum—gave the company the focus and clarity it need to get back on track. Working together, these processes had delivered improvement to every aspect of the organization:

1. All stakeholders in the project become a part of the development process. Everyone involved is genuinely interested in the project output and actively contributing torward progress.
2. Visibility is raised across the board through information sharing. This improves efficiency, mindshare and contributes to progress as well. The group contribution is far more valuable than individual “information silos.”
3. Truly iterative development delivers working, progressively improving software rapidly. Incremental change is easily accommodated by avoiding the “big bang” and monolithic development approach.
4. Rational’s focus on risk-driven development keeps the team focused on the most difficult, potentially risky aspects of the project first. This contributes on a number of fronts: Major problems stemming from unknowns become identified and are dealt with early in the project. Major change stemming from these unknowns has less impact because it happens early. The riskiest, most complex portions of the project are developed first, and thus mature first and receive the most testing.
5. Risk analysis and mitigation “bubbles to the top” through the daily Scrum meetings.
6. Priorities come in-line with business and technical reality as the whole team works on sensible, informed schedules.
7. Ownership and accountability become the mainstay as everyone develops a long-term interest in the project and pride of work product in their contribution. Commitments are taken seriously as a team.
8. Visibility into the entire process means there are no surprises and no last minute bad news. The business can plan according to reality and influence the overall program in an informed manner.
9. Progress is realized for effort and planning, not for putting “hours in the cubicle.” People are rewarded for good work and not for punching a clock.
10. Quality becomes integral and continuous. The Rational process integrates quality assurance and structured testing from the very beginning of the project. It is never an afterthought.

Rational Scrum delivers a solution for small-scale as well as large-scale project development. It supports the essential aspects of project and program management that make quality software development a possibility. It adds the dimensions of reliability and repeatability to projects and focuses the team on continuous improvement and productive work.

Related posts:

1. Making Scrum work: Common failings in adopting Scrum
2. Why Agile isn’t enough (and why it doesn’t work)
3. Scrum is not an agile methodology

In some industries involvement of the “whole team” is almost assumed. Would you buy a car that had never been tested in a safety lab? How about try out a new instrument configuration on an airplane, knowing it had been redesigned by engineers and not pilots? Does it seem reasonable to build a house without hiring an architect? Each of these examples will more often lead to failure than success.

Yet the software industry, particularly the commercial industry (as compared to Military, for example) has been ploughing along without whole teams for decades—a trend that seems to be getting more and more negative attention. Recently evolving standards such as Extreme Programming, Agile methods, Scrum and the Rational Unified Process have all incorporated whole team concepts into their methodologies.

The Whole Team Approach

The term “Whole Team” defined: What is it and why is it so important?

The concept behind the “whole team” is that building a product without representation from every stakeholder is fundamentally flawed. It takes constant, comprehensive involvement through every aspect of the product life cycle in order to guarantee timely delivery of a well-designed product. For example, without a quality assurance organization’s involvement, it is almost certain that customers will discover incomplete requirements after delivery. If the user perspective is not considered early on, it’s very likely the product won’t be accepted or even usable until a second generation is built. Failing to involve software testing and configuration management means failure to deliver working code. And not involving program management, marketing and even financial oversight can lead to a host of problems in execution. All of these situations arise from lack of whole team involvement and can easily lead to a product stumbling out the gate or outright failing.

Creating a cohesive whole team means building a project team with representation from all stakeholders in the project. Doing so requires both defining the team and creating involvement. It also means creating a structure for involvement that does not slow the project down. Creating this environment of constant, informed involvement does not mean throwing everyone into long (and ultimately unproductive) meetings every day.

In some ways defining the team is the easy part: It’s answering the question “who cares about the outcome of this project?” Project analysts, business stakeholders (such as the project sponsor and someone that represents the financial investment), designers, usability analysts, quality assurance, development, software testing, configuration management are a few. Quite often some of these groups can be represented by a single person, as is often the case when it comes to marketing, finance and business interest.

Once the stakeholders are known creating an effective, productive environment can be quite a challenge. This is particularly true with large efforts—perhaps 50 people or more. The goal is to achieve higher bandwidth of communication between team members. In fact, achieving this goal has been documented (see Agile Software Development With Scrum, Schwaber, Beedle) to increase efficiency by an order of magnitude: Over 100% not just 10 or 20%. To achieve this, you must:

1. Organize your projects around cross-functional teams—the core of the “whole team” concept. The team must represent the business and must contain analysts, developers, quality assurance and testers (as well as the related disciplines).
2. Provide the team with the tools needed to create an effective collaboration across all functions.
3. Foster an environment that creates an attitude of “I’ll do what it takes to exceed the goals of the project, on time, and with attention to quality,” not “I’ll do my small part in the overall life cycle.”

Small teams are almost self-organizing, but larger groups require more forethought. The project team must consist of analysts, developers, testers, quality assurance, a project manager, architects and so on. Once the project scope grows beyond what can be feasibly (and efficiently) organized into a single team we can organize into “teams of teams.” This means having an architecture team in charge of system architecture and deciding on the subsystems and how they interrelate. Then, a cross-functional team (including analysts, developers, testers, etc.) are in charge of each subsystem. The entire project is tied together with the management team, with representation from each of the subsystem teams, and through communication channels that must be free and open between all teams. (Several examples and case studies are cited in Agile Software Development With Scrum, among other sources).

An essential element of this process is open communication and information access. In order for each team to work closely within itself, and for it to work closely with other teams, they need access to information. Creating an environment that ensures access to requirements, design details, customer requests, business goals, defects, test plans and status, and so on is critical. Furthermore, communication must be unified. This means establishing an infrastructure for communication and setting it as the sole authoritative source for information. Invest in a document management system and an issue tracking system and develop the attitude that “if it isn’t in the system, it doesn’t exist.” Make sure there is a sole-source for information, and that email in particular does not become a “transient information store.”

Traditionally, at least in some cultures, functional organizations have evolved: All the analysts are in one group, all the designers are in another, all the quality assurance and testing in yet another. This silos information and creates a barrier to ownership that is almost impossible to overcome. Even more so in organizations that use matrix models—where, for example, an analyst works on several different projects at the same time. This leads to treating people like interchangeable components. It also generates a huge waste of effort and efficiency in context switching between projects.

In contrast, the whole team approach creates a vested interest and ownership in the project goals. Each team member develops a long-term investment in their contribution to the project. The ability to concentrate on a single goal creates focus. This is not only more efficient, but working as a team reinforces the sense of ownership between team members. Accountability is high, as is pride in work product. The team orientation avoids finger pointing and assertions such as “that wasn’t my job,” and “it works for me.” Everyone must share project responsibility and, as part of the small team dynamic, will work together to solve issues as they arise.

I think this paragraph, from The Rational Unified Process Made Easy (Booch, Jacobson, Rumbaugh), summarizes the goals quite well: “An iterative approach increases the need for working closely as a team. Avoid functional organizations, and instead use cross-functional teams of generalists, analysts, developers, and testers. Ensure that the tool infrastructure provides each team member with the right information and promotes synchornization and round-trip engineering of artifacts across disciplines. Finally, make sure that team members take joint ownership of the project results.”

Or, perhaps a bit more passionately (the “gang of three” books do tend a bit on the dry side), as Mike Beedle and Ken Schwaber wrote: “When we say Scrum provides higher productivity, we often mean several orders of magnitude higher i.e. several 100 percents higher. When we say higher adaptability we mean coping with radical change… we show through case studies that Scrum reduces risk and uncertainty by making everything visible early and often to all the people involved and by allowing adjustments to be made as early as possible.”

However, it is important to distinguish that Scrum in and of itself is not a complete process. As pointed out in Mike and Ken’s excellent book, Scrum wraps other processes and, in fact, we find that most processes tend toward artifacts that share a great deal in common with Scrum.

Processes such as Rational, spiral and traditional waterfall methods have a lot to offer when combined with agile methods, including Scrum.

Many of the traditional methods of monolithic software development also have a great deal to offer. Formal Software Configuration Management, Quality Assurance and Structured Software Testing are three such examples. Many agile methods focus so closely on rapid progress and lightweight process (or lack of process) that they verge dangerously close to unstructured chaos in and of themselves.

Finding the right balance between agile methods and heavy process is an ongoing decision—one that takes place on a per-project basis and sometimes changes over the life cycle of a project. The best we can hope to do is develop a solid foundation in good implementation principles, keep the best, and put the rest on the shelf for future reference. I’m sure whatever we’re doing in a couple of decades, it’s not going to look much like today’s practices. But I do know this: A whole team approach is one best practice that’s here to stay.

Related posts:

1. Fix your boss (or, reduce risk to quality using a matrix approach)
2. Why Agile isn’t enough (and why it doesn’t work)
3. Making Scrum work: Common failings in adopting Scrum