Why agile IT projects still fail
"It's meant to be an agile project why are we still making changes?"
"Why didn't you capture those requirements earlier?"
"It meets MVP why can't you release?"
"Why are we only discovering these issues now? It needs to hit this deadline so no more changes!"
Does any of that sound familiar? It's something I've heard lots of times.
Unfortunately by the time people are asking those questions it's probably already far too late. I'd probably go further... Most IT projects have already failed before a single line of code is ever written!
So why? The problem certainly isn't because it's an 'agile' project! I've seen many of the same problems with waterfall deliveries!
The Waterfall Sandwich!
So... You take a development team and decide they'll be agile, Agile with a little 'a' of cause - We couldn't possibly be totally agile because the risk of failure is too great! We'll use Scrum, Perhaps use a manager or PM to be Scrum Master and then surround this with the most expensive bread we can! This bread is the existing procedures and mindsets that have always existed.
We don't deliver value until we can deliver everything, The plan is rear loaded with risk, Pressure around release is enormous! Because we don't release until the very end - It's all or nothing. Senior managers have not brought into agile thinking.... Agile is something thats only done in development! The pressure gets high, We try and remember agile principles but the support from PMO, SLT and senior stakeholders just isn't there... They've still in the waterfall mindset!
MVP
Minimum Viable Product - A term that I try and never use!!! It's been picked up and used by so many and understood by so few!!
Projects often have many risky areas and assumptions - The way we build software all too often still loads these risks towards the end of the project! We need to turn that thinking on it's head! Areas which are tricky or dangerous need to be tackled first!
Lets scrap the term MVP! Instead how are we going to test our riskiest assumptions? Lets develop the software to validate our thinking before building the pretty UX or main functionality.
Try and get the software in some format in front of a customer ASAP - Get some feedback - Test the need/demand/appetite for the solution! Design the software so it can run (live) in parallel with existing systems to reduce risk. Get something live fast... Test the target hardware early. Test scalability early on don't build a system that you know needs to run for 1000's of users and only test in the last few weeks as if it's a mere check box activity.
If you want your Product Owner to agree to releasing a cut down version make sure that you continue to deliver software after that point.... One retailer I was doing work for would constantly complain that their PO's didn't understand MVP as they would insist on everything being complete before agreeing the software to be released... But the same organisation would axe funding and break up scrum teams as soon as MVP was delivered! No wonder the PO's acted in the way they did! The system in which the teams and PO found themselves led to this behaviour!
We use the wrong metrics and reporting to give us confidence
There's probably a RAG status somewhere that gets reported back on... We all know how the RAG status works... It's green for 80% of the project, goes amber and then red just before the end when the development team suddenly discovers another six months of work to do! Everybody acts surprised and shocked even the IT veterans that have survived many campaigns in the past seem genuinely shocked that this could have happened - Despite as far as I know this is how the RAG status has always worked on IT Projects?
Why does the RAG status start life green? The start of a project is where we have all the risk! We have unknowns, unvalidated assumptions, assumptions we don't yet realise are assumptions! Why don't we start red and go amber and green as we move through a number of points that validates our thinking? Why don't we turn the RAG status on it's head and assume risk is upfront until we set out how to prove otherwise? Dedicate effort not to be seen starting new functionality but in prototyping and conducting meaningful spikes.
We use burn down charts (Something that the CSM course and software vendors have made the accepted standard) - I never use burn-down charts, they are more or less useless at best and possibly dangerous in the wrong hands! If you're going to use something like this go for the CFD (Cumulative flow diagram)
However I seem to remember reading somewhere that 'Working software is the primary measure of progress' so why don't we use that metric? Probably because we're in a waterfall sandwich and telling those we can't measure progress until the very end wouldn't go down well...... And it really shouldn't go down well.... Unfortunately rather than changing the system so we see working software early we come up with metrics to help us feel better and to give everyone false comfort.
An awful lot of companies would do well to report back on the number of releases to production per quarter for their projects! A simple metric which takes almost no effort to collect! But would be very telling and might change behaviour in a positive way!
Estimates
Estimates are always wrong - So get over it! We're not working in manufacturing or even batch job production, We're working in complex, highly variable knowledge based work. Much of it is more akin to R&D - genuinely no 2 jobs are the same. Teams are not building brick-walls over and over again.
I know that this is unpalatable - But lets just accept that estimates are always wrong - stop wasting effort trying to come up with better estimates (which are equally wrong) and perhaps start to think of other ways of working to reassure those who fund projects.
The rest of the Waterfall sandwich!
Sorry to all my Project Manager friends! It's not their fault! And they are often just part of a flawed broken system! But PM's are measured against certain criteria... Usually if a project is delivered on time or cost! Or delivery against a plan! We've been taught that veering away from a plan is a sign of failure! People are scared to admit that reality is different - We're told that the plan can't change again! The reality is that the plan WILL change and we should welcome that - Not live in a world where we fear admitting to deviating!
Unfortunately all of us are rarely measured on what really matters! What if instead successes for the PMO was based on the client loving the new product? Or the original intention of the solution being met!!! Not does it meet the requirements that they thought they probably needed many years ago!! But has the solution enabled that saving of X million as promised? Or enabled a strategic portfolio goal to be met.
Most people's behaviour is based around how they are measured.... The Firewall team that has to approve and make those changes required to go live are not measured by the success of your project... They are measured against a SLA.... If they do exactly what's been requested not what's needed well thats not there fault! They've done what's been asked of them! And so this depressing trend continues throughout the system!
What if the security team doing that penetration test decided to get involved early with the team... Spoke about what usually causes problems.... Looked at ways of working with the team to try and prevent... Rather than what happens so often - Which is that the security team are thrown a solution to look at a week before go-live... And again the security team are not measured by how many projects they enable to succeed - If so they might behave in a different manner... Their measure of success is that a security breach never occurs! I'm not saying that's a bad thing... But the safest way to ensure that you don't have any type of breach is to always say 'No'!
Group Think
This is something I want to dedicate an entire blog post too. I've been in many meetings/planning sessions with senior managers, Executives sometimes development teams - The plan is talked through.. There's a unspoken pressure on people to conform and accept the plan... Disagreement is frowned upon.. Perhaps it isn't but it feels like it is or will be... Those that perhaps do challenge are told that we must hit the deadline, Failure is not an option, There is no plan B! Teams go along with plans which are flawed from the start.... A month in and the project is already falling behind.. Do we put our hands up and say that the plan has been invalidated and we need to re-plan? Or do we convince ourselves that the time will be made up later? Its nonsense... We've designed a plan which probably loads most of the risk towards the end and we still think that lost time early on will be made up in the most riskiest phases of the project - But still everyone nods there heads and goes along with it.
We just need to be seen starting this
I've lost count of the number of times I've heard someone say the line 'We just need to be seen starting this, We need to give so and so confidence that it's in hand' Everyone knows the Kanban line 'Stop starting, Start finishing' But it couldn't be more true.
Projects loose focus, They start more and more things, WIP (work in progress) balloons, Queues start to form, Context switching begins... We have a hundred different journeys to test and go through but none of them work... We don't stop and finish what we have! Instead we start something else - We can finish that code later... We'll integrate at the end of project.... Just before that enormous big bang release we've planned!
Complex and Expensive release procedures
All too often the process of pushing software out is hugely complex - I don't want to go into DevOps or automated testing in this blog post.. It's already long enough! But highly complex release procedures add to the risk! Its a barrier to releasing frequently which results in a lack of learning early. It encourages a culture of trying to push as much into each release as possible which then ironically tends to further delay releases, increases risk............
Process which prevents software going into production which isn't fully tested and UAT'd is part of the problem.... Design systems in a way where it can be safely pushed into production even untested! Release software switched off, Release API's in parallel to existing API's - Design software which can test API's in production before they are fully switched on. Put effort into designing ways to release software that might not be fully working early but safely!
Distributed teams
If team members are on different floors communication begins to fail! I've recently read research that shows that once team members are more than 20meters apart communication drops significantly... And it’s true! Throw outsourced teams into the mix - particularly when they are working on non T&M contracts and things just get worse!
I'm not saying that distributed teams cannot work... But you have to work an awful lot harder to ensure that they do work! And that also costs money!
Training
So often training seems to be missing... At all levels! Many developers have never worked in a 'genuine agile' environment - Usually they may have done bits of Scrum only, Everyone on the team will have different understandings of what Scrum and Agile mean..... Which is fine but they often don't have agreements as to how they will work together.
Many people believe that being agile is something that applies to the development team only and so they don't need training... Senior leadership are rarely well versed in being agile or what it means to be agile. Don't underestimate the need for training and to have coaches on site... Coaches to the business not just IT
Projects!
The constant ramping up and down of project teams is wasteful - It disrupts flow, knowledge, team cohesion the list goes on! Projects are by default temporary structures created to implement highly complex change. But we know that change is continual albeit capacity/demand may be variable. Project funding leads to poor decision making, deadlines don't make sense and actions are taken which increase risk.