Why Transformations Fail

Over the years, we’ve all heard about, seen, or participated in Agile Transformations (or when ‘agile’ passed its prime, Digital Transformations, or some other phrasing that means the same thing). Almost every one of them fails. Many of the companies that run them will claim that they succeeded, but by objective measures, almost every one of them fails to meet the stated outcomes — improve the business’s ability to react to changing customer demands in a manner that keeps those customers satisfied.

For all intents and purposes, engaging in an Agile Transformation is a public recognition that the old ways aren’t suitable any longer, and something new has to be done. Unfortunately, the ‘something new’ is almost always indistinguishable from the old ways.

Solving the wrong problem

The things that are tackled during an Agile Transformation usually aren’t what are stopping the company from adapting. It’s not ceremonies, process, or teams that are hindering progress. It’s bigger (and also smaller) than that.

A large number of Agile Transformations are undertaken by executives as a way of getting their teams to do better. It’s an example of the old classic: “Change for thee, not for me.” The executives who start these transformations see a problem in their ranks, and set out to address it. But the problem they see is a symptom of another, deeper problem.

Fundamentally, systems reflect the assumptions and beliefs of their creators and leaders. The reason companies function the way they do is the collective set of beliefs of their leadership. These beliefs lead to assumptions, assumptions lead to behaviour, which lead to policies, practices, and ceremonies that propagate (both formally and informally) through the organisation. What executives are seeing when they look at their problematic organisations, and what they’re trying to change, are their own beliefs. They just don’t realise it. And unfortunately, the things they do in order to change things aren’t aimed at their beliefs, they’re aimed at the symptoms of their beliefs. And just like treating the symptoms of a cold, treating the symptoms of their beliefs can lead to temporary improvement, but the ultimate fix requires internal change.

The Cause

Executives believe lots of things. Which belief(s), in particular, is the issue? The biggest, is trust.

In waterfall development practices, all the thinking is done up front. It’s done by leaders who develop the plans, architects who design the systems, and project managers who define what ‘right’ means. And it’s all enforced by testers who ensure that what’s done matches what the thinkers said needed to be done. That means that developers, the doers, don’t have to think. The thinking has been done for them, by people who are trusted to make the right decision — people who understand the context, the need, the customer, etc. Whether or not executives realise it, organising things this way indicates a significant lack of trust in the people doing the work. Every step along the way is there to ensure that nobody deviates, adds anything original, or otherwise diverges from the original plan. People are treated like unthinking automata, or cogs in a machine. Absent little ‘a’ agile, that’s the default position for most organisations. (Waterfall development also creates an illusion of certainty or predictability that exectives find hard to let go of, but that’s a conversation for another time.)

At this point, I can hear the objections from people who say they do trust their staff. They want them to think. They want them to do better. I’m not here to argue whether that’s true or not. Maybe it’s true. But what matters is what perceptions people have. Instead of focusing on whether or not they trust their staff, I want to look at what practices, behaviours, governance, restrictions, and rules have been put in place. Do they indicate trust?

One of my favourite set of questions that I like asking executives is, “If you trusted all of your staff to do the right thing, what would your organisation look like? How would it behave? What freedoms, power, autonomy, and choices would people have? And is that how your company behaves today?” Most executives, answering honestly, have to acknowledge that their systems, policies, and practices do not indicate trust. They indicate control, or lack of trust.

Why is agile different?

Underpinning the agile manifesto, beneath the surface, and implied by the agile principles is something antithetical to traditional enterprise development: Trust. Agile software development requires trust to work. (It also engenders trust, creating a virtuous circle, but that’s a separate conversation.) Development teams are free to use whatever tools, methods, approaches, and decisions work. They’re expected to work directly with customers to identify their needs. They’re then expected to prioritise those needs and satisfy them in the best way for the business. Teams are expected to know what the business needs, what the customer needs, and what the software requires, and to figure out how to satisfy, or at least balance, all three. Power, control, autonomy, influence, and customer relationships all sit in the hands of the team. The team is trusted to make the right decision. And if it makes a mistake (by mistake I mean ‘makes an objectively poor decision given the data available to the team at the time’), they’re expected to reflect on what happened in order to figure out if it was avoidable and how to reduce the likelihood of it happening again. Teams are expected to know what’s best. They’re closest to the customer, so there’s nobody better placed to understand customer needs.

Having this much power in the hands of development teams is not something that many executives are comfortable with. They’re used to a world of false certainty, where they hold all the cards. A piece of work is worth undertaking because they say it is. Because they know best. Effective agile doesn’t work this way. It can’t.

Conflict

This different in approach doesn’t inherently breed conflict. In fact, many top-down/command-and-control organisations are very successful, even today. Conflict comes when executives who hold these assumptions decide they want to adopt agile — when they see other companies changing faster, and winning more market share than they are — or rather, they want their people to adopt agile. They don’t want to do it. They think adopting agile is a software process change. They don’t recognise the role of their own beliefs in their organisation, and its behaviours. Organisations in this state will often hire external help who will come in and help them restructure to be more Agile. They’ll adopt Agile frameworks. They’ll create Value Streams. And 2 years later, things will be worse than ever.

This is because when your beliefs, architecture, and team structures are all aligned, things get done. They might get done slowly, but they get done. When executives look at how things are going in other companies, and they see them going much faster, they want the same thing. What they don’t realise is that in companies that have effectively adopted agile, their beliefs, architectures, and team structures are all aligned differently to their own.

Most Agile Transformations don’t tackle all 3 areas. They usually only tackle team structures. They talk about aligning around value streams, but they compromise in order to keep people as happy as possible. The end result is an organisation that has all its old architectural practices, all its old beliefs (absent a quick course on ‘agile leadership,’ ‘agile fundamentals,’ or something similar that doesn’t change anything), but new structures. And that mismatch creates conflict, as dependencies and delays show up that never hapepened before. On top of that, most organisations that go through this change are unaware that in addition to structures, they also have to change architectures and beliefs. And without awareness, there can be no planning or consideration of how to make the change. This leaves organisations in a state of conflict, and ultimately disillusionment, wherein they come to believe that Agile is flawed, or that their adoption failed.

The long-term impact of such Agile Transformations on the people who were supposed to benefit are almost always negative. People who liked the old way get isolated or leave, and people who like working in agile organisations can’t be kept, because they can tell something isn’t right. This leads to a high degree of churn, and a significant loss of domain knowledge. The people who remain are those that are doing their best, but have been (unintentionally) set up to fail. At the end of the day, the organisation is left with new language, and new ceremonies, but the same old problems, made worse because it’s now taboo to acknowledge them, and their structures don’t match their architecture.

Conclusion

As leaders, it’s our job to challenge ourselves and our beliefs in order to help our organisations do better, for our employees and our customers. We can do better for them and ourselves.

If, fundamentally, you are comfortable with a system that puts all the thinking and control in to the hands of a few, or you want to hold power, don’t adopt agile. It won’t work out well. Keep going the way you have been. Your organisation will be fine (at least for now).

If, on the other hand, you (want to) believe people can be trusted to do the right thing, given a chance, and you’re ready to try to make it work, agile might be a good choice for you. The first steps are all internal to the executive team — changing thinking, changing behaviours, changing incentives (implicit and explicit), changing decisions — long before you’ll be ready to even think about your teams, your structures, or your architecture. The amazing part of changing beliefs, is that a lot of additional change comes for free. Your questions will change, your expectations will change, the things you notice will change, and people will respond to it, adding their own changes to a system that will adapt and improve over time. As your beliefs shift, so will your organisation.

By focusing your time and attention on the right things, you can have an outsized impact on the entire organisation. After all, change comes from within.

NB: The shift to agile (and everything after) is an ongoing process. It’s not once and done. Agile is about continuous improvement, which means continuous change.

NB: If you want help with this change, let me know.