Estimating programming tasks is a complex process. There is no one-size-fits-all solution. Furthermore, skimping on resources to get it right turns the entire procedure into reading tea leaves. However, we can improve the accuracy of our predictions by applying the following practices.
I have prepared a PDF version of this article. You can open it in your browser or download it to your computer and print it:
1. Expect trust when making decisions
Having the “productive partnerships” value from the Manifesto for Software Craftsmanship in mind, verify that the relationship between a Development Team, Domain Experts and a Product Owner is free from the mis-engagement of management. This advice is extremely helpful in reducing the time we waste on work that doesn’t add much value to the product. Moreover, we’ll be able to produce more accurate estimates.
The team decides “how” to achieve goals, the Product Owner decides “what” is most important. The Domain Experts provide crucial knowledge about their area of expertise. Don’t let this balance suffer from micromanagement or unnecessary intermediaries and remember that developers are not just code monkeys. Otherwise, estimating tasks lacks the insight that the team could have brought in personally discussing goals and requirements with the Product Owner and Domain Experts. Ultimately, the estimates are unreliable and the time spent on tasks is wasted because the value provided is less relevant to end users than it should be.
You’re a professional
Ignoring developers when estimating tasks will result in numbers that are appealing form a marketing point of view. However, it shows a serious lack of competence. Unfortunately, there are many projects that have a technological stack, due date and feature list sealed on paper before even a single programmer is consulted.
It makes no sense to have estimates that are not based on reality. To cope with the imminent problems, programmers are asked to abandon rules of software craftsmanship and cease writing unit tests. This is a quick and stupid way to completely destroy a project by making it unmaintainable, unusable, buggy and even dangerous. The fact that management may not understand the importance of the principles guiding professional programmers does not mean that they can view them as a waste of resources.
Trust facilitates task estimation
One of the most important strategies to ensure that a task actually benefits the project is to allow developers to review both its requirements and design with the strong and direct support from the Domain Experts and Product Owner. As many team members as possible should be involved in this review to get reliable and thorough conclusions that are priceless when estimating tasks.
By default, accept that the responsibility for making decisions about architecture, technology and organizing work rests with the team. Each developer involved in the project should make a number of decisions for themselves. If the results are not adequate, the team will reexamine them during pair programming and code review. The collective code ownership practice guarantees that the whole team ensures project quality. In addition, the extreme programming and software craftsmanship principles greatly reduce the impact of inaccurate projections – it’s easy to add new functions efficiently when the existing code is clean.
Micromanagement is a waste of time that has no place in a dignified and well-functioning organization. Moreover, a project that relies on a manager who used to have something to do with IT many years ago and now only they can make decisions is dead on arrival. Keep in mind that the more valuable and productive developers are, the sooner they leave an organization that doesn’t trust them and doesn’t treat them like professionals. Even the most sophisticated methodologies will not help in obtaining valuable estimates when no one wants to work in the organization long enough to learn about the specifics of the project.
2. Don’t avoid consulting problem finders
People competent enough to identify the nuances and risks of software development should feel welcome when the team defines and estimates tasks. They can tell with great accuracy how long it will take to achieve a goal and how problematic it is. However, often what they have to say does not match our wishful thinking. Therefore, they protect us from committing to overly optimistic estimates. They are invaluable in adjusting the scope of a project and making it more realistic.
Never sign all documents confirming the terms of a contract without consulting with specialists. Take them seriously when they point out unrealistic expectations or promises conjured out of thin air. Don’t underestimate their cries for more down-to-earth estimates – this protects the entire company from becoming addicted to miracles as the due date approaches. Any wishful thinking about a project deadline or scope is unacceptable.
A real professional will do all it takes to keep their employer from financial consequences of overcommitting. This includes providing the necessary expertise to completely deprive management of any illusions before they seal any promises on paper.
3. Allocate enough time for a proper research
Even an experienced team needs time to research all new, better, and more optimized ways to implement features. In other words, we cannot rely on implementing familiar and common features in the way that was used three years ago.
Lack of research results in unoptimized, outdated code that is vulnerable to security breaches and more difficult to maintain by younger developers, accustomed to modern, better-suited tools. If you don’t include the research phase as part of the assignment you evaluate, you will lose the opportunity to learn about new security issues that could endanger the project or external libraries that would perfectly suit your needs, saving a lot of time for implementation.
A research phase is flexible and it takes various amount of time for different developers and different tasks. However, ignoring it during estimating tasks will more likely rise maintenance costs as the code may introduce bugs or fail to solve the real problem.
How to research a programming assignment so the time is well invested
- Develop a comprehensive understanding of the problem. Book some time to consult Domain Experts, don’t be timid about the amount of questions – ask as many as you need.
- Rephrase the problem from the point of view of users. How much value solving the task will bring to the project?
- Create sketches, diagrams, make notes in plain English, enumerate algorithmic steps, write test cases. Ask other programmers for peer reviewing your answer to a problem, verify whether you’re thinking in the right direction or need to change the approach. Cross question the solution design while it’s still only on paper.
- Ask what are the possible blockers, difficulties, complications. Don’t attempt to solve them at this point! The sole purpose of this list is to prevent surprise from foreseeable problems. In addition, it will make the research phase more thorough and provide you with a list of exceptions to deal with.
- Research the place in the existing codebase where the solution is meant to fit. Which parts of the system will be interacting with each other?
- Be clear about features that the solution won’t provide. Mark the boundaries to protect yourself from feature creep.
- Search Google for ready to use solutions. Examine available open source and commercial libraries and asses their usefulness, availability and maintenance cost. Ponder on the pros and cons of adding an external dependency to the project.
- Use a spike to verify that your solution passes unit testing that reflects the most important requirements. Responding to business needs is a very valuable part of our job, but it may be cumbersome. Therefore, we must be able to check many ideas quickly and easily against these tests.
Attach the artifacts created during the research to the task description. Make your research explicit to other programmers. Not only will it save a lot of time when someone has to pick up your assignment, it will also provide test cases, a list of exceptions to handle, and details that can be copied into the project documentation.
4. Break down features into easily manageable tasks
Make sure that the tasks are actually achievable in a reasonably short time, e.g. they have been evaluated by the team at 2-5 points. As a result, any mistakes made when estimating tasks will appear early, will be minor, and won’t disrupt the project schedule.
Don’t hesitate to break a user story into smaller components. It’ll help you define the requirements more clearly and identify all corner cases. Moreover, the whole story will be implemented more accurately, without taking any shortcuts that could harm the project in the long run. This will keep everyone more focused on solving real problems and adding substantial value to the project.
Another key point is that more concise tasks make continuous integration much easier. A regularly updated codebase means we don’t have to make as many assumptions when estimating future workload. Moreover, we can follow the progress on a burn down chart more accurately and quickly notice even a slight delay. Thus, allowing us to apply our back-up plan in good time to meet the deadline (e.g. cut the sprint scope, do pair programming, use an open source library instead of custom code, etc.).
5. Keep priorities in check
The better defined and divided the tasks are, the easier it is to prioritize them. Estimates are only forecasts based on many assumptions, guesses, and, usually, a small amount of data. It’s irrational to expect them to be 100% correct. Therefore, we must be prepared when reality does not follow them. Fortunately, thanks to reasonably short sprints (e.g. two weeks) and well-defined, small tasks, we can quickly realise that the predictions may not be correct. When the priorities are clear, we can engineer a back-up plan for the situation when some of the most valuable features can’t be delivered on time.
Separating what’s crucial from what’s nice to have in a project might even save the deadline as we can easily decide how to narrow the scope of the upcoming release. This way, we can meet the deadline and still deliver a valuable product.
|How Dead Space’s Scariest Scene Almost Killed the Game – the interview with Glen Schofield, the co-founder of Sledgehammer Games and the creator/director of Dead Space, gives a great example of how adjusting a project scope, maintaining a laser focus, dividing problems into manageable chunks and sticking to priorities can save a project.|
6. Estimate in appropriate units
Be aware that an hour from an experienced developer gives a different amount of work than an hour from a person learning a particular technology. One hour is not equal to another. Estimating in abstract points and according to the Fibonacci sequence will reflect differences in tasks complexity.
Moreover, defective leadership can make programmers prioritize meeting a specific deadline over solving a problem, shifting their attention from understanding a task to its due date. Thinking about how much time we have before a task has to be marked as “Done” on the project board makes it difficult for us to focus on dealing with the actual problem. As a result, many tasks “completed” on time provide only superficial solutions to persistent problems. As we can read in the Atlassian estimation guide:
Story points reward team members for solving problems based on difficulty, not time spent. This keeps team members focused on shipping value, not spending time.https://www.atlassian.com/agile/project-management/estimation
When you can try estimating tasks in hours
A team can estimate in tangible units without bringing much harm, if:
- tasks are usually small, well-defined and most of them received the same (and small) number of points (there is little or no difference in complexity between them),
- you collected enough statistics for the particular team to calculate how many hours on average a task takes for 2, 3 and 5 points (the average for tasks of more than 5 points can be deceptive),
- you realise stats expire when a team gains or loses a member – coding, pair programming, and code review are conducted by a different team from then on,
- the team is strongly unified in experience, skills and competence within the project’s technological stack, which means that one developer working hour gives a similar result to another developer working hour,
- the team agrees on this,
- estimating tasks in hours is carried out in each team separately (do not use statistics collected for one team to another).
7. Remember that estimates are not transitive between teams
Estimates only belong to those who have committed to them. You can’t freely apply them to new team members or even other teams. Take into account all differences in experience and proficiency between programmers – they are not faceless, indistinguishable beings. Expect teams to take responsibility only for the estimates they have committed to. Don’t compare estimates between different groups to force one of them to scale-up.
Each team will estimate work on a slightly different scale, which means their velocity (measured in points) will naturally be different. This, in turn, makes it impossible to play politics using velocity as a weapon.https://www.atlassian.com/agile/project-management/estimation
Other people’s predictions or management’s ideas about the duration of the assignment are in vain without the team’s confirmation. Always see yourself and other developers as professionals who can evaluate their own work.
8. Ask for the actual scope and deadlines
Make sure everyone on your team knows the true deadlines and project requirements. Agreeing to work with managers who keep this knowledge to themselves is asking for trouble. There are already enough assumptions that the team must make to maintain an agile approach and deliver software tailored to the ever-changing needs of consumers.
If somebody is deliberately hiding the business knowledge to force more optimistic estimates, the project is doomed to failure. If you are consistently in a situation where you learned about a requirement crucial for estimating a task AFTER you’d started working on it, you have to take a stand. First, when someone wants to add an expensive requirement to an already estimated task, agree on what can be removed from the sprint scope if the deadline is compromised. Second, make sure that transparency is a value shared by everyone involved in the project. Otherwise, the estimates are useless.
The team should refuse to estimate when communication with stakeholders lacks transparency or someone is lying about the requirements to get better estimates. If developers don’t know the context, strategic goals, and priorities of a project, their performance suffers. They can’t foresee the consequences of their code and technological choices, leading to unsustainable and expensive code. In summary, agreeing to work in a dishonest environment takes away the opportunity to solve real problems. Instead, we provide only superficial solutions and a rapidly growing technical debt.
9. Use past data to better estimate future tasks
Being consistent about measuring velocity pays off when we need to predict future productivity. There is no point in planning tasks worth 60 points when, on average, our team manages to complete 30 in one sprint.
Ignoring statistics not only destroys confidence in the team’s past performance, but also makes prioritizing difficult. Wishful thinking brings more harm than good. When we disregard the actual team performance, the scope of the sprint may expand in an uncontrolled manner. Trying to force estimates that are inconsistent with data and facts is an easy way to destroy the project and team. Agree on the techniques you use in your sprint review to gain valuable insight, such as:
Try, for example, pulling up the last 5 user stories the team delivered with the story point value 8. Discuss whether each of those work items had a similar level of effort. If not, discuss why. Use that insight in future estimation discussions.https://www.atlassian.com/agile/project-management/estimation
The hope that we’ll automatically “get better” at estimating tasks is unreasonable. We can’t even copy estimates for the same tasks! Assuming you repeatedly evaluate and do one type of task, you will also be more efficient in completing it, which impacts each subsequent prediction. If the tasks are identical, you deal with the problem only once and then somebody can reuse the solution (unless you solved the issue poorly). Furthermore, as a developer, you have to work with different technologies, resolve various problems and constantly learn new tools. Therefore, even when working with similar applications, you are never asked to estimate two exactly the same tasks.
|We can’t effectively learn from the past if we haven’t measured progress or recorded any lessons from previous sprints.|
10. Review both failed and successful estimates as a team
If you have to explain overdue tasks, take your time to identify the actual sources of the delay. Reiterate over the steps presented in this article and answer the following questions:
- Who decided on the estimates? Was it the team in agreement with the Product Owner and Domain Experts or was it someone from the management?
- Has anyone tried to brainstorm possible problems and blockers that might hinder progress?
- Was the research phase a planned and organized undertaking?
- What is the size of the overdue tasks? How often was the new code merged with the codebase? In which stage of the sprint did you realise that something won’t be delivered?
- Did the priorities keep changing during the sprint? Who decided on what should be done? Did you have a list of features that are not essential and could be cut from the sprint’s scope? Did the team spend time on nice-to-have features before moving on to must-haves?
- In which units were the estimates made? Did you estimate in hours for a development team of varying skill levels?
- Did anyone decide to move the work to another team and keep the original projections?
- Were the goals clear to everyone? Or did the management keep the team in the dark to avoid tough discussions about scope and priorities?
- On what basis were the estimates made? Do you have any historical data for the team such as its velocity to use as a reference?
- What conclusions from the last review were implemented in this sprint? Did they work?
Group the problems
Trying to tackle all obstacles at once can be discouraging and ineffective. As a remedy, try to assign the problems you identified to the following categories:
- requirements that were kept hidden from the developers during estimating tasks;
- actual surprises – the problems that could not be predicted;
- what was poorly estimated – include both too optimistic and too pessimistic estimates.
Determine which group is the most numerous. As a result, you will know which area’s improvement will have the most positive impact on better task estimation.
Discard what doesn’t work, embrace what’s worth keeping
Consult every member of the team and use their feedback to identify the estimation mistakes. Furthermore, don’t forget about all the techniques and procedures that improved your accuracy. Make sure that the conclusions are clear and everyone involved in the process agrees on them. Transfer the truly beneficial practices to next sprints and other projects if applicable.
|However, if the estimates were imposed on the team, the developers don’t own them. This limits the space for improvements as they can’t do much until the management starts to trust them.|
More on estimating tasks
- The How do you estimate on an Agile project? document published by ThoughtWorks presents many valuable insights on estimating tasks.
- The price of faking agility
- Get the most out of the new hire