So is Agile just hype or is there something to it?

There are strengths to the agile way of thinking, and many of them bring useful perspectives to software and systems development that are new and even revolutionary. Here are some of the things that work — and, potentially, that radically change our old-world practices.

Whereas most legacy methods stem from industrial process — that is, assembling a product using a set of defined, predictable steps — the agile method is empirical. It recognizes that development is more like invention and research, more akin to scientific study, than assembly. This empirical nature is at the heart of the agile mantra: Deliver, measure, adjust and repeat. The strength of this approach often bears itself out in fantastically hyperproductive teams that deliver working product far more quickly than legacy methods, such as waterfall, could ever achieve.

Agile does this by cutting through complexity. Every agile-based methodology focuses on simplification of otherwise complicated problems. For example, XP and Scrum both emphasize development of near-term, complete deliverables. This means carving out tangible and reasonably independent pieces of work, focusing on that work, and then — at least as much as possible — moving on to other work. This approach requires that large, complex problems are broken down into manageable pieces and thought of on a micro-deliverable level. Likewise, this approach minimizes ceremony and eschews as much procedure as possible. Some agile methods go to extremes in this regard, focusing entirely on delivering work product and not at all on procedure. This translates into minimizing complexity on a large scale.

Closely related to eliminating complexity is agile’s focus on progress measurement. Most agile methods measure progress chiefly, if not exclusively, in terms of delivered work product. Most methods also are quite stringent in defining progress only when finished work is delivered, which means you can’t work for nine months on a single big feature. Instead, micro-deliverables target key features, deliver those features into the customer’s hands, then moving on to new features. This can be a huge strength because the customer gets working product in-hand to review early, and often. It involves the customer early in product evolution, leading to a host of benefits including better product targeting, prioritized development and improved quality.

These characteristics of agile methods combine to fundamentally change the way software and systems development is practiced. Agile also empowers individuals to become stellar performers. In fact, all forms of agile rely on this to some degree — with more lightweight agile methods being completely dependent on individual empowerment. The idea is that an empowered team will leap over constraints to get the job done, no matter how “out of the box” the thinking needs to be. It’s a refreshing concept and one that can indeed be supremely successful, but it does require the team to embrace the idea wholeheartedly.

Another benefit, at least in some situations, is the creation of self-organizing teams. Partly because of the light ceremony, the fast pace, and the penchant for empowerment and accountability, teams become self-organizing. This works when the team has the right make-up, as individuals step up to take on tasks best suited to individual skills. Self-organized, empowered teams become very powerful and very productive, provided that the team members are up to the job.

There is absolutely no doubt that agile methods make it possible to get things done quickly. That’s what it’s all about, after all. The real question to me is how much of this tradeoff is really desirable? How often do we want to eschew process and maturity in favor of getting things done quickly?

More importantly: Can we effectively merge the best attributes of Agility with the most valuable benefits of established processes and standards?

Why Agile doesn’t work

When an agile project fails, it generally does so spectacularly and predictably. The common failings of agile-based projects are just that… common. We see the same problems over and over again, and this has become the basis for many critiques of agile methodology. After all, if we keep seeing the same problems crop up again and again, isn’t this proof enough that the process is flawed? This becomes clear in hindsight, so why do we continue to see 90 percent of projects missing the mark?

The fact is, agile by itself is just one tool in the toolbox that should be applied with other implements of the trade. In my experience, the problem comes in most often because small- and mid-sized organizations experience brilliant success with agile and then assume it can work everywhere. They throw out the toolbox (or perhaps never buy one in the first place). Yes, agile can succeed. Yes, it can deliver fantastic productivity and stellar results. But not always — in fact, I will go so far as to say not often.

This isn’t because of agile’s limitations. Instead, it’s because of overconfidence by those putting it to use, and the mistakes an immature organization makes as it grows and applies it inappropriately.

Immature companies and teams are cutting their teeth, again and again, on the limitations of agile.

All agile methods make it easy to oversimplify complexity. In fact, agile’s strength of eliminating complexity might be better stated as “ignoring complexity.” There are appropriate situations for this but, more often than not, ignoring complexity leads to problems. Most business cases don’t call for undefined delivery dates or loosely changing requirements and partial deliveries. These are risks that most business models are incompatible with. If the risks aren’t something that your business can sustain, adopting a purely agile process is taking a huge gamble.

Likewise, focusing on the near-term is an agile attribute that introduces a lot of unknowns into the business-end of an equation. Few people will contend that agile is appropriate for mission critical efforts such as, say, launch vehicle development, as sometimes requirements need to be set in stone before anyone starts development. But what about situations where some degree of fuzziness is acceptable or even beneficial? Agile advocates compatibility with change, sidestepping change control procedures that would otherwise place tight controls over requirements. Requirements change carries with it a heavy burden, particularly when it comes to the cross-organizational impact to marketing, budget, quality management and the customer. However, cutting change control, requirements management, and configuration management from the process can lead to long-term disaster that the short-term perspective of most agile methods will overlook.

This theme of reducing structure and control has cut out many waterfall-origin processes. The danger often manifests as small-scale agile projects are successful, leading to wider-scale adoption of agile. But, as the projects grow in complexity and criticality, major missing components in the process become evident. For example, no agile methods today integrate comprehensive quality assurance procedures (in fact, thanks to some early mistakes, such as MIL-STD-498[#], most people think quality assurance is software testing — it’s not). Structured software testing often becomes an afterthought, and risk management programs tend to be regarded as “fuzzy disciplines.” Yet, these are the processes that successfully put man on the moon, that develop health care and financial services systems, and ensure that nuclear plant regulatory systems don’t fail after delivery. Of course, there is a cost to each of these processes, and every business needs to weigh the cost-benefit of adopting more process against cutting those processes. This needs to be an on-going evaluation, made as projects, organizations, and teams evolve — it’s not a decision that stands alone.

From a purely hands-on, management level, agile methods pose “people problems” as well. The strong emphasis on self-organization and empowerment can easily backfire. The former relies heavily on people that are capable of self-management and self-direction. Not everyone can live up to that expectation. The latter, delivering empowerment to the team and individual, can lead to a hero mentality and silo’d teams that refuse to play well with others. As projects grow in size, complexity, and dependency on other teams and resources, these characteristics become the drawback of an immature organization.

Almost all agile methods oversimplify valuable processes. In some situations, the project survives the oversimplification. Sometimes the business is tolerant of the fallout. In every case, agile methods expose the project to risks that stakeholders should be — and often are not — aware of.

What to do about it

We need to be cognizant that one solution does not fit all problems. While an agile method such as XP or Scrum may have led to success in one project, this doesn’t make it a foregone conclusion that it will do so again. Each project is different, and organizations evolve over time. Adopting one process to solve all problems is a sure recipe for failure. On the other hand, having a well-versed team that can draw on several methodologies, as appropriate for the job, is a recipe for success.

If your organization is looking for the one-size hammer to hit every nail, make sure it’s as configurable a hammer as possible. Don’t choose something that is either too lightweight, such as XP, because many projects will overreach the capabilities of such a lightweight process. Likewise, don’t try to implement a full-on waterfall style methodology either because, while definitely thorough and capable of getting the job done, it’s just overkill for many smaller projects. If you must choose a single process, pick one that’s efficient, borrows from both agile and waterfall, and is highly configurable, such as Rational Scrum or the Rational Unified Process. Both of these have the maturity to deliver large-scale projects, but also support starting small and adopting minimum ceremony.

A better awareness of what specific agile practices can and cannot accomplish is key. For example, Scrum is not a development methodology, and it cannot effectively deliver software or hardware projects unless it wraps itself around one. Yet today many organizations are employing Scrum as if it were a development methodology. I’ve even seen an organization of several hundred developers “force fed” Extreme Programming from the top down. The outcome of that particular operation: Mid-level management hid the fact that they didn’t use XP from top-level management after everyone realized what a mistake it was. Perhaps we’ll have to wait for mature standards in education and certification to evolve, but personally I’m not sitting idly by.

One of my personal pet peeves in the technology industry is a relative lack of standards and qualifications. Would you go to a doctor that didn’t have a medical degree? Would you hire an architect that didn’t have an appropriate engineering degree? Yet we hire software professionals (much less often hardware professionals) without adequate education, current qualifications, or meaningful certifications. For that matter, the proliferation of meaningless qualifications (such as Scrum Master certification) continues to weaken the industry. In the long run, we need better standards regarding education, accreditation and certification.

Understand agile methods for what they are. Keep in mind that lightweight process carries risk. Use the right tools in the right situation.

Coming full circle

If we add all of these things to agile methods, won’t we just end up using waterfall process all over again?

I don’t think so. Waterfall-based process, the original behemoth processes born out of industrial process, are widely recognized as inefficient. There are tremendous advantages to pressing forward with a merger between waterfall practices and agile practices. I hope the end result is a new generation of software and hardware development methodology — a generation that we’re just starting to see as processes such as Rational Scrum come to the fore. It’s time for development methodologies to evolve, and there’s no holding that back.

Related posts:

1. Making Scrum work: Common failings in adopting Scrum
2. Scrum is not an agile methodology
3. Common oversights in choosing methodology

Indeed, a good question: What would we do without the hero?

Let’s flash forward a little ways. The hero has grown tired of the project and leaves. Fears abound that the project is doomed; there’s no way the team will survive without him. Concerns run deep, until the hero tells us he’ll stay on-board as a part-time consultant. Of course, he’s pretty busy these days — but, well, he’ll give us a “preferred” rate and show up at least a few days each week. He’ll keep the project afloat, no worries… in between getting his new company off the ground, of course. The one we’re funding with expensive consulting dollars. Either that, or the hero feels the project isn’t challenging enough, and leaves to greener pastures and more exciting projects, still promising to provide part-time support to keep the project afloat.

Either situation is dismal. Both demonstrate a project and a team that’s being held hostage to the whims of the “hero.”

The truth of the matter is simply this: Heroes are bad for the team, bad for the project and bad for the company.

Heroes are motivated by making themselves indispensable to the project by becoming irreplaceable. Most often this takes the form of information hoarding. The hero understands quite well that the established culture supports his role only while he and he alone can solve the project’s problems. One of the most effective ways to make sure this happens is to keep critical information away from the team. He’s the only expert in a few critical areas, and refuses to share his knowledge because it would be inefficient or too difficult to convey to someone else.

This leads to situations where the hero is subtly motivated to make sure there are instances in which only he can save the project. By ensuring the hero is the only one with the answers, the only one who can solve the problem at hand, he becomes indispensable — and, also, a tremendous project risk. Not only is everyone on the team constantly put in second-place to the hero, they also suffer as a whole when the hero is at his best. In times when the hero “shines,” the team is struggling to overcome roadblocks they haven’t been prepared to deal with. The project falls into jeopardy because the team cannot focus on a solution as a whole. Instead, the team churns in vain trying to contribute to the solution that only the indispensable hero can tackle.

Consequently, team morale is often a casualty of the hero culture. The continued failure of the team to exceed project goals, instead running into roadblock after roadblock, is tiring. Compound this with the fact that only one person, the hero, is enough of an over-achiever to solve the problems of a seemingly over-ambitious project and the team begins to become demoralized. Nobody likes to come face to face with failure repeatedly. Doing so in an environment that doesn’t provide the tools to better oneself is futile, and the hero team leader is certainly not motivated to fix the situation. In point of fact, most heroes tend to be pretty lousy team mentors, as being a hero implies putting oneself above everyone else. This means that while the hero has holed up inventing the next great solution, the team is left to its own devices. This vacuum of leadership provides a poor environment for anyone interested in advancing their skills and career, let alone seeking out a workplace that provides opportunity to learn.

Environments that support the hero become detrimental to the team. With the team leader focused on fulfilling the role of hero, collaboration suffers. Without a truly open, collaborative, information-sharing environment projects function at low efficiency (or, in some cases, fail to function completely as information is silo’d and isolated). In contrast, a healthy team elevates everyone, sharing information and skills, making the hero obsolete and making the entire team indispensable. A collaborative team rapidly turns into a formidable force when its collective attention turns to any problem — such teams turn out productive results faster and more efficiently. Each individual’s growth becomes a focal point, and the positive experience and knowledge gained from such a working environment becomes a lesson that each team member can share. In an open, collaborative, delegating environment the team lead will mentor the team; constantly challenging each person to take on more responsibility and grow into new opportunities. The best team leads are not information hoarders but information sharers, almost trying to engineer themselves out of the equation by teaching everything they know. In fact, this kind of leadership becomes truly indispensable because so few people are able to teach, motivate, mentor and unify a group of people toward a single purpose effectively. It’s not the information hoarded in the team lead’s head that makes him invaluable, it’s his ability to create a hyper-productive team and get things done.

As a project leader it’s important to confront the hero mentality head-on, making it clear that a hero is known for what it is: A detriment and risk to the project. Heroes are a liability. They are a bottleneck to progress, introduce the risk of “hostage projects,” generally make poor mentors and leaders, and always create specific situations that are harmful or dangerous to the project.

Unfortunately, putting an end to hero culture is often not an easy task. Many workplaces that embody hero culture don’t understand the problem — they think everything is “just fine,” and look to the hero as a critical resource, someone that has saved the project or the company time and again. Especially in young or inexperienced organizations, the difference between a supportive environment and a destructive one is unclear. The hero continues to operate above and apart from the team, often disregarding what little authority or direction he disagrees with. Inexperienced team members don’t know they should have a team lead that is mentor, guide and teacher. Organizations can be set up in such a way that the hero figure is empowered beyond reasonable boundaries, as often happens when clear structure and accountability is not in place. Combined, these problems can create the environment where a hero-mentality, embodied in someone who is ill-suited to create unity, lead, and mentor, ends up holding the project hostage. Most often, the evidence will point to the root problem: The team will constantly run into problems only one person can solve; team members will disagree with the hero and often not achieve consensus; much of the team will be “out of the loop,” particularly where the hero is involved; overall, the team will operate either apart from the hero or as individuals, not a cohesive group. Perhaps most indicative: The hero has sole authority over the project, yet little accountability. This is common in so-called “flat” organizations. I tend to avoid organizations that strive to be “flat,” as it’s really just a way of saying “we don’t want to deal with the fact that someone has to be in charge.” The simple truth is that leaders and managers need to have the authority to implement policy. Good leaders and managers will find ways to do it without using their authority unless necessary.

The easiest way to fight hero culture is from a position of authority, such as the project sponsor or project manager role. Given the advantage of authority, the hero can be given a choice: Either become a collaborative leader, or face what amounts to demotion as a new leader steps in. Some heroes won’t be able to make the right decision and will end up leaving the project — but others will embrace the idea of transforming themselves into a positive influence. I’ve seen this happen in a few cases and can honestly say the results were extraordinary. It could be that your hero’s spirit is willing but he doesn’t recognize there’s a problem. Likewise, it could be that your hero is struggling to move into a management role and needs guidance himself. Sometimes you’ll find out the hero was thrust into a leadership role without wanting it, and will happily step aside.

Whether tackling the problem from a position of authority, or from the inside, there are a few strategies that will help to ease the process. Developing a collaborative environment is a first step at ending the hero culture. This means putting in place tools and processes to share information, including an open information environment. Get information out of everyone’s head and into a tool that facilitates group review and commentary, such as a wiki or document repository. This can begin by emphasizing brainstorming sessions, collaborative documentation and group exercises to design and implement solutions. Often group development can be a productive tool, as well (some environments, such as Extreme Programming, push pair programming as one example — I don’t believe pair programming should always be put in practice, but this is a good example of when it makes sense). Project deliverables should also be managed in an open, collaborative environment. Use a project or task tracking system that lets everyone see what’s going on, what’s being delivered and how it’s being done, and most important: Focus on shared responsibility and avoid having unique specialties in favor of collaboration.

The team should also push for opportunities to advance skills and individual knowledge. Avoid “information silos,” or individuals who hold all the answers to a specific problem. It’s healthy for teams to have more than one expert in an area. Not only does it reduce risk, it also affords a more collaborative environment where two or more people can openly discuss a solution and work together on implementation. Any time you hear “only one of us knows how to do that,” immediately think in terms of how you can turn one into two or three people.

Demand a leader that is a good mentor. If the team hero isn’t up to the job, find someone who is — and be vocal about wanting an environment that supports growth. A healthy work environment includes ample opportunity to take on more responsibility. Any project manager that hears you want the responsibility will start casting about for a means to give it to you — and that usually means making sure the team leader can advance his team.

Ultimately, your work environment is in your hands. If you’re aware of the problem, identify it and and surface it — but do so in a constructive way. Have ideas and solutions ready to solve the problem, and emphasize the value it will bring to the team or organization. Focus on the facts as much as possible, like productivity gains the team will experience when it has greater bandwidth, and how getting to market more rapidly (and probably with a better, more robust product) will boost return on investment.

Related posts:

1. Hiring for the culture
2. Fix your boss (or, reduce risk to quality using a matrix approach)
3. Whole teams