Purpose

Effort translates the size of functionality (measured in points) into a detailed estimate of effort, typically measured in units of days or hours. The Planned versus Actual Effort metric is used to monitor variance in the amount of actual hours worked compared to the amount planned. This helps the team identify the degree to which effort variance across the lifecycle might impact the cost of the project. It also helps the team monitor the accuracy of their estimates in order to improve project predictability.

Definition

Planned versus Actual Effort is captured at the work item level, and totals are rolled up for each iteration and the overall release as follows:

Work Item

• Work Item Planned Effort - Estimated number of hours it will take to complete a specific work item
• Work Item Actual Effort - Actual number of hours it took to complete a specific work item.

Iteration

• Iteration Planned Effort - Estimated number of hours it will take to complete all work items planned for a specific iteration.
• Iteration Actual Effort - Actual number of hours it took to complete all work items planned for a specific iteration.

Release

• Total Planned Effort - Estimated number of hours it will take to complete all work items for the project
• Total Actual Effort - Actual number of hours it took to complete all work items for the project
  

Analysis

A good way to monitor Planned versus Actual Effort throughout the lifecycle is to use two trend lines. Plot Planned Effort and Actual Effort on the Y axis and iterations on the X axis. This allows you to monitor Planned versus Actual Effort at the iteration and release levels. Monitor the hours variance between planned and actual effort in each iteration to identify trends across the lifecycle.

Using Planned versus Actual Effort to monitor project execution

Planned versus Actual Effort is used as a project execution metric to help a team monitor and steer their project performance. As actual and planned effort is tracked in each iteration, trends are identified that help the team understand if their estimates are improving.

Expected trend - Estimates should improve iteration to iteration as the amount of uncertainty in the project decreases. Trend lines will show less and less variance between planned and actual effort as the project progresses. The project is likely on track for cost, and the team can have high confidence in their predictability.

Effort variance remains high throughout the lifecycle - This trend can occur when the team is estimating work in chunks that are too large. The larger the task, the more room for error there is in estimation. Break large tasks down into a number of smaller work items. Ongoing poor estimation can also occur when the team continues to struggle with a new technology. Additional training or mentoring might be required to get the team on track and enable better estimates. Confirm that the team is not estimating too far out into the future. Provide just-in-time detailed estimates during iteration planning. Team members should collaborate to validate individual estimates. When it is too hard to provide an estimate for a particular work item, the team must take action to resolve any open issues. Address uncertainty throughout the lifecycle rather than waiting until the end to deal with outstanding issues. When requirements are not detailed enough, estimates are likely to be off. Perform sufficient analysis and receive clarification as needed prior to producing detailed estimates for iteration planning. Determine if process or tooling efficiencies should be introduced to eliminate overhead. When estimation variance is high throughout the lifecycle, the project is at risk of being over budget (meaning the team should probably cut scope) or under budget (meaning the team could be adding more value by delivering more features).

Sudden spike - A sudden increase in effort variance typically indicates a surprise of some sort. The team may not have fully understood a particular requirement, waited too long to address any associated risks, or an unexpected risk has emerged. Effort variance can also increase when a team member completes a work item but did not create the original estimate and is not familiar with that part of the system. The introduction of new team members can also result in a sudden change in effort variance. Confirm that the team is addressing uncertainty throughout the lifecycle in order to minimize surprises. When this trend occurs, identify the cause of the unusual variance and determine if it is likely to be repeated. If so, cost is likely to be impacted resulting in the need to reduce scope and adjust future iteration plans. If the cause it not likely to be a recurring problem, and has been addressed by the team, there is no need to dramatically change plans for the next iteration.

Up and down - Effort variance that fluctuates drastically throughout the lifecycle indicates a serious problem. The team is experiencing a great deal of surprises and disruptions. Some factors are affecting the project that cannot be controlled or managed. Perhaps requirements are continually changing, or a particular technology has a cycle of working as expected in a given iteration and then causing unexpected problems in the next. When this trend occurs it means that you cannot manage the project effectively throughout the lifecycle and that your cost and schedule are at risk. Collaborate with stakeholders to address requirements churn. Focus on addressing technology risks to uncover any additional unknowns. Bring in mentors to help where needed. Confirm that the team is gathering all information needed before providing estimates at the beginning of each iteration.

Monitoring Planned versus Actual Effort at the work item level

If estimation variance is high, you can generate a scatter chart of work item-level variance to identify the source of the problem in a given iteration. It could be that most of the estimates provided by the team in a given iteration were reasonably accurate, but a few estimates were off by a large margin. Use a scatter chart to monitor Planned versus Actual Effort for each work item in the iteration to determine the standard deviation in estimates. Monitor the standard deviation in planned versus actual effort in each iteration to identify trends across the lifecycle. You may discover that only certain work items were estimated poorly, or that particular team members often over or under-estimate. This knowledge helps the team take corrective action and improve their predictability.

Using Planned versus Actual Effort to monitor capability improvement

Planned Versus Actual Effort is also used as a capability improvement metric. It helps a team and middle management (project manager, product owner) monitor improvements made during the project lifecycle in adopting practices such as:

• Team Change Management
• Test-Driven Development
• Continuous Integration
• Concurrent Testing
• Iterative Development
• Whole Team
• Continuous Integration
• Shared Vision
• User Story-Driven Development
• Evolutionary Architecture

Teams successfully adopting these practices will demonstrate improvements across the lifecycle that will help to reduce effort variance. Operational Executives can also use this metric to monitor systematic improvement in reducing costs across the organization by adopting these practices.

Expected trend - Successful teams decrease their effort variance as the project progresses. They consistently stay on budget and have high project predictability.

Increasing or fluctuating effort variance - When teams consistently display these trends, they should analyze their projects to identify any common causes for the variance in order to take appropriate corrective action. For example:

• If developers are clear on requirements and are addressing uncertainty throughout the lifecycle, they may have a problem with efficiency. Reduce manual and inefficient build and test procedures by adopting Continuous Integration, Test Driven Development, and Concurrent Testing practices.
• If teams are performing their work efficiently, but changes are not managed well in projects, they may need to focus on better software configuration management. Implement tooling to track and control changes, and implement best practices of Team Change Management.
• When projects typically experience surprises later in the lifecycle and scope is not controlled well, teams may not be collaborating well across the lifecycle. Introduce practices that encourage teams to communicate well and take action on feedback, deliver working software frequently, integrate their work, and increase visibility of their progress throughout the lifecycle such as Iterative Development, Continuous Integration, Whole Team, and Test Driven Development.
• When projects often have high requirements churn and stakeholders are not satisfied that the right capabilities are delivered, teams should improve their ability to elicit requirements by adopting the Shared Vision and User Story-Driven Development practices.
  

Frequency and reporting

Detailed estimates are provided at the beginning of each iteration. Actuals are reported at the end of each iteration for work items completed in that iteration. The Planned versus Actual Effort chart is reviewed with the team during each iteration retrospective and discussed with stakeholders to help identify trends.

Collection and reporting tools

Teams can capture estimated and actual effort for work items in IBM® Rational® Team Concert®.

Assumptions and prerequisites

• User Stories/ Use Cases are elaborated
• Detailed estimates for work items assigned to an iteration are logged during iteration planning
• Detailed estimates are provided by the team member who will perform the work.
• All actual hours worked by the team to complete work items in a given iteration must be logged at the end of that iteration

Pitfalls, advice, and countermeasures

• Estimates will differ depending on the person who will perform the work. It can be useful for individual team members to monitor their own estimation variance for the sake of improvement, but overall team estimation variance is what is most useful for project monitoring.
• Estimates will change as more information is learned about a particular work item. However, in order to analyze trends, estimates should be frozen at the beginning of the iteration.
• Make sure that any tool used to capture effort data is simple to use and does not introduce too much overhead to the team.
• Do not exclude overtime hours in order to appear more productive, or to stay within original estimates. This will decrease the effectiveness of your trend analysis, and impair the team's ability to track real improvements or issues with estimating.
• Communicate key constraints and assumptions with estimates.
• Continue to validate estimates by comparing with past experience.
• To calculate a variance percentage for any level (release, iteration, or work item), use the following calculation: Effort Variance = ((Actual Effort- Planned Effort) / Planned Effort) *100
• Use quality metrics as a countermeasure to Planned versus Actual effort. If teams are highly incented to reduce estimation variance, they spend less time testing in order to deliver work items within their original estimates.
• Customer Satisfaction can also be used as a countermeasure to this metric. If too much scope is removed from the release to make up for underestimating, stakeholders may report low customer satisfaction with the project.