Are Story Points useful?
I wrote a post back in 2015 in defence of planning poker and story points, why I still used them and why I still found them relevant and in fact useful despite the increasing criticisms of the technique. Read it here
When I talk about story points… which I still do… often! I list the following main points :-
Estimates are always wrong - If you’re going to be wrong, don’t waste too much time/effort on it, don’t worry about it, do it and move on quickly.
Story points estimates are quick, If I ask for an estimate in days people will make an effort to understand and plan how to accomplish and give a number (which will be wrong) - If I ask for a story point estimate it’s quick (although still wrong)
People are poor at estimating the cycle time that a piece of work will take - But we’re better at comparing jobs to other jobs (Hence relative sizing)
Estimates have a shelf life - The more time that passes the less accurate estimates are - Story points tend to be ‘more accurate’ for longer, they decay slower especially if the team alters the way it works (increases/decreases velocity)
Over the years consulting… I have found story points are often miss-understood, abused and the cause of so many arguments. I often find teams engaging in long ‘arguments’ over if something is a 3SP or 5SP, pressure from project managers and stakeholders for smaller estimates (as if making a story 5SP rather than 8SP will mean it’s delivered quicker)
Over time I’ve come to find story points the source of perhaps as much trouble as time estimates once were… But are they useful? Is the benefit worth the hassle?
I’m going to say something which is controversial… story point estimates are NOT about value or complexity (Just because something is more complex doesn’t mean it will take longer) they are about effort and by effort I mean time. I’ve had arguments with few CST’s (Certified Scrum Trainers) on this but I still maintain that Story Points ARE LINKED TO DAYS…. Although not directly.
Let me explain, imagine we have a team with a stable velocity of 100SP per sprint - and a sprint is 2 weeks (10 days) And let’s say it’s a team of 5 so we have a theoretical 50days per sprint (10days * 5team members) - And yes I’m including sprint ceremonies/events because planning and talking about work… is work.
We now know that 2SP is equal to 1 Day - Yes I know, Shock horror! How dare I equate story points to time when agile coaches/trainers/consultants (Whatever you call us) have spent years telling us they are not - In fact I’ve even used Story points to back charge projects for work delivered (perhaps worthy of it’s own blog)
If story points were not linked to time what is their purpose? Velocity is directly linked to the amount of time (a sprint) to perform an amount of work - If that statement is not true then why track velocity or use it to help forecast what is effectively an amount of work over time?
Over the last 5 years since writing that first blog - I have become more and more disillusioned with story points, partially because a technique that should be quick and easy has perhaps become such a bone of contention and partially because I’ve had a sneaky suspicion that story point estimates are useless.
So I’ve been collecting data from various teams over years trying to collate actual cycle times against story points. Recording at a sprint level and over multiple sprints and attempting to normalise data where paired development took place, etc,
Example scatter graph showing cycle times against Story Points - Source That’s Intelligence
The above graph is a good example (although perhaps extreme) recorded over a number of Sprints :-
The team have 10 completed 5SP stories and the cycle times range from 1 day to 15 days
3SP stories ranges from < 1Day to greater than 30Days
Here’s another team I collected some data on
Scatter graph, Story points against cycle times - source That’s Intelligence
There’s certainly questions you’d want to ask about how these teams were using Scrum - However I’ve found similar wide spreads with many teams using Story points to estimate.
At best with the above team I could probably make the following commitments :-
Any 2SP story should be completed in less than 10 days
Any 3SP story has a 50% probability of being completed within 10 days
Any 8SP story will take longer than 10 days
These above teams took story point estimation seriously - They had an external consultancy firm helping them with their agile process and providing consultancy (In addition to mine). They were making forecasts based on these story point estimates and making business commitments and decisions on them.
I’ve been collecting data on teams and story points against cycle times for years and have found low statistical correlation.
The above teams had a correlation of <0.3 and I frequently find teams with a correlation of less than 0.5 (0.5 would be considered a moderate correlation)
So given that for many teams a 5SP story could range so much - what value are you getting from story points? What are they telling you? What do they allow you to achieve?
Lets have a practical example - Imagine you have a new 3SP story using the teams historical data you know that the cycle time should be between 1 day and 33days with a medium of 9 days or to put it another way, with a blended rate of £550 per person per day, the story as Product Owner could cost between £550 and £18,150 to complete - given the range should that story be started?
I wonder how many teams have looked at the correlation between their story points and cycle times?
Interestingly many of these teams had a fairly predictable velocity with a low standard deviation between sprints.
The below is another team I did some work with - In this case I asked the team not to look at estimates, not to use story points for capacity planning - but to pull in the number of stories they thought they could deliver in a sprint - I then recorded the number of stories completed each sprint verses committed.
Stories committed v Stories delivered - Source That’s Intelligence
I then used the number of stories completed per sprint to forecast delivery dates and I did the same using story points and velocity. The result, both forecasting methods delivered a similar result (within 2 weeks of each other). Now I am by no means the first person to notice this! I remember listening to a talk at London Lean Kanban day in 2015 by Vasco Duarte where he’d observed the same.
I still agree with many of the arguments I made in favour of story points back in 2015, in some scenarios they have usefulness - However given much of the evidence I’ve collated from various teams and the sheer confusion and arguments I see over story points estimates I would suggest they are now just another waste and many teams would be better off simply counting stories and the time saved used for better purposes.