Mind The Agile Gap
Updated: Jul 14, 2019
Agile is everywhere now. Companies of all shapes and sizes are embracing the agile way.
Iterative development, emergent design, time-boxed delivery and self-organised teams are no longer just the buzzwords of software houses and the startup world. Large enterprises also want a slice of the agile pie. Who wouldn’t want faster delivery, higher software quality and increased customer satisfaction?
Agile is about creating software that can handle change because, as we know, requirements are never stable. That false assumption was partly why the waterfall model failed to deliver. Big up front designs result in rigid, over-engineered software that doesn’t respond well to change.
Even though it’s taken about 20 years, the agile movement has finally ‘crossed the chasm’ into the large enterprise mainstream. And developers have been at the forefront of this revolution.
After years of telling ‘the business’ how software should be built, traditional large enterprises are finally listening to the developers. But it’s not all roses. Within the development community there’s a collective memory of distrust and resentment at having been ‘commanded and controlled’ from the top for so long.
‘Big Up Front Design’ and ‘waterfall’ are swear words in many development teams. ‘Management’ is often the worst word of all. You can see this when you look at the job titles of leadership roles in development. Apparently, there’s no need for managers or even architects in Agile. Instead we have Scrum Masters, Product Owners, Lead Developers and Agile Coaches.
When it comes to developing software, power and technical decision-making has finally shifted into developers hands. But this poses a challenge for large enterprises that naturally aren’t very agile.
They’re often built around a complex web of red tape, managerial hierarchies and isolated departments. This creates an ‘us and them’ mindset where the necessary governance and business constraints coming from ‘the business’ are seen as a threat to the agile way of development teams.
You might think that Development and IT Infrastructure departments would get along nicely. But then you probably haven’t worked as a dev in an organisation where you need to complete a 4-page application form just to spin up a new staging environment. Oh and it’s going to take 2 weeks.
The divide between dev and IT infrastructure departments has only increased with the advent of PaaS and Serverless technologies. These empower developers by abstracting away infrastructure concerns, making development and deployment way more efficient and enjoyable. Understandably, this can leave traditional infrastructure teams, who are usually responsible for managing IT environments, wondering about their role.
Bridging the gap
Despite their expertise in software development, developers usually know (or care) very little about business or leadership. This is a problem. You might argue that Scrum Masters or Product Owners fulfil this role but if you’ve ever worked in an agile team you’ll know that this isn’t the reality.
While an essential part of a Scrum team, the Scrum Master is there to act as ‘servant leader’. In reality, this usually means running the rituals of Scrum, removing ‘blockers’ and doing whatever the developers tell them to do.
Product Owners, while useful members of the agile team, are rarely actually part of the team. Perhaps they might wander in to the odd meeting or settle a dispute. What Product Owners should do according to Scrum and what they actually end up doing are two very different things.
This isn’t, however, the fault of the ‘Product Owner’. The main blocker for a Product Owner is that he or she usually doesn’t have enough authority to make any strategic decisions. She or he may simply be a middleman conveying the vision of the product without any real involvement in its construction. Not even the Scrum Master can unblock that.
What about developers?
Developers care about software craftsmanship and reducing technical debt. And they’re experts in bottom-up, emergent design (which is the best way to build software). But emergent design needs to be complemented with a top-down approach that brings vision, strategy and governance to the development lifecycle.
If emergent design isn’t constrained, or rather gently moulded, an architecture may evolve that is led by the ‘latest and greatest tech’ rather than by any strategic business vision. When the tech and business goals are not aligned, businesses can’t move forward (or scale) in any productive way.
When this happens, you often find that ‘the business’ doesn’t understand what the development team is actually building, how they built it or more importantly why they built it. Conversely, developers see ‘the business’ as a threat to their autonomy and agility. There’s nothing agile about this stalemate.
The solution is to bring top-down design back into agile. The technical design process needs to be guided by the business vision because technical decisions affect business outcomes.
Agile top-down design means doing ‘just enough, just in time’. It involves a whiteboard (not reams of documentation). It involves suitably experienced individuals from all relevant departments who can bring clear business objectives and vision to the table as well as technical expertise.
Agile, after all, is about getting business people and developers to work closely together. Only then can we be sure that we’re building the right thing and we’re building it right.