Hi Everyone,

Welcome to the third post in my conference and journal paper post series. This series will contain ten conference and journal papers from my time working in the Queensland Government. In my post, My Peer Reviewed Conference and Journal Papers, I explain the purpose of this series.

Road Program Evaluation: Assessing the Bigger Picture was my second conference paper and also my second attempt at presenting at the ARRB Conference. The ARRB and the ATRF Conferences ran in alternating years. The conferences were organised and ran by some of the same people and organisations. My experience from the previous year’s conference definitely helped me prepare for this conference.

In the year leading up to this paper, I had become more heavily involved in conducting cost benefit analysis for programs. These were not programs in any meaningful way. They were projects along a particular corridor or of a particular type that grouped together for funding purposes. These programs were used to obtain more funding. The whole approach is backwards. The program should come first, and projects should be included to help achieve the goals of the program.

This paper introduced three possible methods that could be used to rank projects within a program. The aim was to maximise the net benefits for the program. Some projects by themselves have very little value. However, when included with other projects, it might become critical to a program. For example, the benefit of upgrading a road that has very little traffic is unlikely to outweigh its cost. However, if other projects cause a significant change in traffic flow, this project could become critical.

The paper was well received by the ARRB Conference, and Transport and Main Roads were mostly indifferent to my presentation. I used two case studies in my paper to describe two of the methods. The Warrego Highway Upgrade was not considered political; therefore, it did not attract attention. The Bruce Highway Overtaking Lanes program was more political, but the results of the cost benefit analysis reflected positively on these projects. The findings of this paper were not taken seriously, and none of my recommendations were implemented in any form of future guidance.

Management considered funding attendance to conferences as a privilege. They did not care if an attendee presented or just watched. The content presented was little more than an exercise and not something that would influence how things were actually done. Changes were only implemented based on direct instruction from higher in the organisation or from a highly paid consultant. I did not understand that at the time.

Road Program Evaluation: Assessing the Bigger Picture

Abstract

Road projects are typically evaluated individually before submission for funding to either State or Federal Government Programs. However, there are occasions when projects are grouped together and evaluated as a program and the program is then submitted for funding. This paper focuses on the quantitative evaluation of programs and the methods of ranking and evaluating proposed projects within a program.

Three methods of program evaluation are discussed in this paper. The first method applies an approach dubbed the ‘simultaneous’ approach. This approach involves evaluating all projects simultaneously under the assumption that either all the proposed projects are constructed or none of the proposed projects is constructed. The second method applies an approach dubbed the ‘sequential’ approach. This approach involves evaluating projects in the sequence of a predetermined level of priority. The third method adopts both the simultaneous and sequential approaches.

The $670,000,000 Warrego Highway Upgrade (WHU) program and the $40,000,000 Nation Building Bruce Highway Overtaking Lane (NBBHOL) program have been included in this paper as examples of programs with different approaches to evaluation. The WHU program was evaluated using the sequential approach while the NBBHOL program was evaluated using the simultaneous approach.

1. Introduction

This paper focuses on the quantitative evaluation of road programs and the methods of ranking and evaluating proposed road project options and road projects within a program in the context of the Queensland Department of Transport and Main Roads (TMR). The approaches and methods applied to program evaluation, project ranking and project option selection are explained using a hypothetical example and demonstrated with two real world examples. Discussed in this paper are a brief comparison of the advantages and disadvantages of evaluating projects as part of a program as well as possible changes to funding policies to facilitate the submission of programs for funding.

“A program is a group of projects managed in a coordinated way to obtain benefits not available from managing them individually” (Duncan 1996, p. 8). A program approach allows for the accounting of benefits and costs that are difficult to recognise when projects are evaluated in isolation. Road projects are often located in close proximity to each other causing overlaps of benefits. This is particularly true for the major highways located in Queensland. Nine projects were proposed in The Warrego Highway Upgrade (WHU) Program¹ along the 84km stretch of road between Toowoomba and Dalby (TMR 2010c) and eight projects were proposed in the Nation Building Bruce Highway Overtaking Lane (NBBHOL) Program along a 50km stretch of road between Curra and Childers (TMR 2010a).

The size of the Queensland road network and the quantity of road projects required to maintain and improve this network are key impetuses to the development of methods of evaluating programs. As at May 2010, TMR had over 2500 state funded road projects and over 600 federally funded road projects on its 5-year rolling Roads Implementation Program Database (RIP Live) and over 150 of those projects had capital expenditure of over $10,000,000 (TMR 2010d).

2. Program Evaluation Models and Approaches

There is a limited amount of literature pertaining to the evaluation of programs. A focus of the available literature is the optimal selection and scheduling of interdependent projects. Nemhauser and Ullmann (1969) developed one of the earliest models to consider project interdependence. This model is stated in Equation 1 and Equation 2.

Equation 1: Nemhauser and Ullmann Model

Subject to:

Equation 2: Nemhauser and Ullmann Model budget constraint

Where:
b_k = expected value with implementation of kth project
d_km = deviation in expected value from linear addition of two interacting projects k and m
x = binary variable equal to one if project is selected and zero if project is not selected
A_k = construction expenditure of kth project
V_km = deviation in costs from linear addition of two interacting projects k and m
C = Maximum budget expenditure for projects

The Nemhauser and Ullmann model only considers the interdependence of two projects and therefore, the application of this model to a typical road program of more than two projects is limited. Gear and Cowie (1980) and Fox et al (1984) have suggested the expansion of Nemhauser and Ullmann model to include more projects but these deviations of the model are complex and difficult to apply to programs. Another issue with such models is the problem of circular referencing in regards to the binary variable when the selection of projects is dependent on the results of the model itself.

Tao and Schonfeld (2006) have developed island models, these models deviate from the Nemhauser and Ullmann model and other previous models by including more interactions between projects. The models discussed by Tao and Schonfeld (2006) relate mostly to urban transport models and travel time cost savings but these models do not appear to rank projects within the program but rather provide a solution of optimal project combinations dependent on a fixed budget.

Of the models reviewed, the principles behind the Tao and Schonfeld (2006) models are most relevant to the evaluation of programs in the TMR context. In this paper, a complete model to program evaluation is not provided but instead three methods of quantitatively evaluating a program are developed. Applied to these methods are two evaluation approaches, the simultaneous approach and the sequential approach. The two approaches consider different assumptions regarding the base cases and project cases² of options, therefore potentially producing different results.

The simultaneous approach is based on the assumption that all projects in the program will proceed in the project case and none of the projects will proceed in the base case. This assumption enables an unbiased calculation of benefits of each project in the program if no prior project selection has been undertaken. The formula for calculating benefits of a project evaluated using the simultaneous approach is given in Equation 3.

Equation 3: Calculation of project benefits using simultaneous approach

Where:
R_k = discounted road user costs (base case) of k^th project/project option
p_k = adjustment to discounted road user costs (base case) from implementation of k^th project/ project option
d_i = deviation in discounted road user costs from the interaction with other projects/project options
N = number of proposed projects in the program

The simultaneous approach is not designed as a theoretical optimization of resources but a practical approach to evaluating projects and determining close to optimal project selection and ranking of projects in a program. The simultaneous approach is more relevant at the earlier stages of the concept phase³ of a program when there is the greatest uncertainty about the viability of proposed projects.

The sequential approach is based on the assumptions that projects have been ranked prior to the evaluation. In the base case, all projects ranked above the proposed project under evaluation are constructed and no projects ranked below and including the project under evaluation are constructed. In the project case, the same assumptions hold true but the proposed project under evaluation is assumed to be constructed. The benefits/costs each project imposes on the base cases of projects ranked lower in the program and the benefits/costs each project imposes on the project cases of projects ranked higher in the program are apportioned to the project. These assumptions allow for the incremental impact of adding each project to the program. The formula for calculating benefits of a project evaluated using the sequential approach is given in Equation 4.

Equation 4: Calculation of project benefits using sequential approach

Where:
d_k = deviation in discounted road user costs (base case) of other projects from interaction with k^th project
p_i= adjustment to discounted road user costs (base case) from implementation of project

The sequential approach, like the simultaneous approach, is not designed as a theoretical optimization of resources but an approach to assess the change in net welfare of society from the inclusion of additional projects to a program of works. The results of the sequential approach can be used as an argument for additional funding if projects currently excluded from the program add sufficiently to the net welfare of society. The sequential approach is more relevant at the later stages of the concept phase when project options have been determined and projects have been ranked. The sequential approach should not be used in isolation, as the sequential approach does not have a project ranking function.

2.1. Hypothetical Program

A simple hypothetical program is used to demonstrate the application of the simultaneous approach and the sequential approach to program evaluation. Contained in Figure 1, are four projects of a hypothetical program proposed along a major highway. Project 1 is a road duplication project⁴ that also includes the upgrade of off-ramps and on-ramps to the highway, Projects 2 and 4 are road duplication projects, and Project 3 is an intersection upgrade. For Project 3, there are a number of options available, such as signalising the intersection, adding right turn lanes or left turn lanes or putting in a roundabout. All projects require some form of options analysis but for simplicity of demonstrating the approaches described in this paper, only Project 3 is assumed to be subject to the options analysis stage. This program is also assumed to have a cut-off benefit cost ratio (BCR) of 1⁵ .

Figure 1: Location of Projects

The four projects are located in close proximity of each other and therefore, some of the projects are interdependent. Table 1 provides an overview of the interdependent projects.

Table 1: Overview of Relationship between Projects

On completion, Project 1 is expected to increase the traffic flow to the project sites of projects 2 and 3. In the base case, the project site of Project 1 will become congested, thus restricting traffic flow to project sites 2 and 3. In the project case, traffic flow is not restricted to these project sites; thus, the projects sites of projects 2 and 3 will have increased traffic flows. On completion, Project 3 is expected to increase the traffic flows to all the project sites. The increased capacity of the intersection will enable increased traffic flows both east towards projects 1 and 2 as well as west towards Project 4. Projects 2 and 4 do not have an impact on the other projects in the program.

Identifying the interdependent projects is critical, as the extent of the interdependency between projects determines the assumptions made in both the base and project cases. The extent of interdependency also determines the most appropriate approaches and methods to apply to an evaluation of a program.

2.2. Application of Simultaneous Approach to Program

The base cases of all projects evaluated using the simultaneous approach will be subject to the assumption that none of the projects will be constructed while the project cases will be subject to the assumption that all projects will be constructed. The base cases of all projects will not be influenced by other project outcomes. The simultaneous approach can be applied as early as the options analysis stage of the concept phase of the project life cycle. Projects with options with a various degree of interdependency in relation to other projects need to be identified early and evaluated first as the selected option could influence the assumptions applied to the other projects in the program.

For Project 3, assume there are three hypothetical options, Option A: left turn lanes; Option B: signals and left turn lanes; and; Option C: roundabout. The project cases of the options will include traffic growth from the increased capacity of the intersection as well as additional traffic growth from the upgraded first section of road. Benefits, costs, net present value (NPV), BCR and Incremental BCR (IBCR) of these options are given Table 2.

Table 2: Options Analysis of Project 3 (Intersection Upgrade)

In this example, Option B has the highest NPV and an IBCR greater than 1, therefore Option B is mostly likely to be the recommended option to be included in the program based on a quantitative analysis. The project cases of projects 1, 2 and 4 should be subject to the assumption that Project 3 is Option B.

The project case of Project 1 will include traffic growth from the increased capacity of the upgraded road as well as traffic growth produced from the upgraded intersection in Project 3. The project case of Project 2 will include additional traffic growth from both projects 1 and 3. The project case of Project 4 will include additional traffic growth from the upgraded intersection in Project 3. Table 3 contains the benefits, costs, BCR and NPV of each project. Projects are ranked according to BCR.

Table 3: Results of Program using the Simultaneous Approach

The primary functions of the simultaneous approach is to rank project options according to incremental returns to incremental capital cost, rank projects based on returns to capital or cost and provide an economic value for the entire program, measured by NPV.

A possible problem with the simultaneous approach when projects are heavily interdependent on each other is that different options of Project X⁶ have different interrelatedness with the options of Project Y⁷ , thus causing uncertainty about which option is optimal. A solution to this problem would be to evaluate the various combinations of Project X and Project Y options in the same evaluation⁸ . Table 4 contains an example of a combined options analysis if Project X and Project Y have three options each.

Table 4: Combined Options Analysis

Option 3 and Option 1 (X3Y1) are the optimal options for Projects X and Y respectively if a cut-off BCR of 1 is assumed¹¹ . This solution can make the evaluation of project options complicated especially if there a large number of options that are interdependent, for example, three interdependent projects with four options each would produce an options analysis of sixty-four options¹² .

The assumption that all projects will proceed in the project case may not be a realistic assumption for many programs, hence may result in understating or overstating of project benefits. This problem is difficult to avoid using the simultaneous approach as projects are yet to be eliminated from the program.

2.3. Application of Sequential Approach to Program

The hypothetical program can be evaluated using the sequential approach. The sequential approach to evaluating a program requires the input of a predetermined ranking of projects. This ranking could be determined through the simultaneous approach, other quantitative methods or through qualitative methods. This ranking is critical to the order projects are evaluated using the sequential approach and this order can influence the apportionment of benefits to individual projects within the program. The ranking applied to the sequential approach in this paper is the ranking that was determined using the simultaneous approach described in Section 2.2.

Project 3 is evaluated first as it has the highest ranking. The base case will be subject to the assumption that none of the projects will be constructed and the project case will be subject to the assumption that only Project 3 will be constructed. The differences in the project case costs between the simultaneous approach and sequential approach will be apportioned to Project 1.

Project 1 is evaluated second based on ranking. The base case will be subject to the assumption that Project 3 will be constructed. The increased traffic growth produced by the upgrade of the intersection in Project 3 will be included in the base case. The project case will be subject to the assumption that only projects 1 and 3 are constructed thus, will include the increased traffic growth from the upgraded intersection in Project 3 and the increased traffic growth from the increased capacity produced by Project 1. The differences in base case costs between the simultaneous approach and sequential approach will be apportioned to Project 3.

Project 2 is evaluated third based on ranking, the base case will be subject to the assumption that projects 1 and 3 will be constructed. Increased traffic growth from the upgrades of both projects 1 and 3 will be included. The project case will be subject to the assumption that projects 1, 2 and 3 will be constructed. All increases in traffic growth produced by projects 1, 2 and 3 are included in the project case. As Project 2 is impacted by both projects 1 and 3, an additional base case subject to the assumption that only Project 3 is constructed is required for the apportionment of costs to projects 1 and 3.

Project 4 is evaluated fourth based on ranking, the base case will be subject to the assumption that projects 1, 2 and 3 will be constructed, increased traffic growth from the upgrade of the intersection in Project 3 will be included. The project case will be subject to the assumption that all projects will be constructed, thus including all the increased traffic growth stated in the base case. The differences in the base case costs between the simultaneous approach and sequential approach will be apportioned to Project 3. Further base cases need not be created for Project 4, as Project 4 is only impacted by Project 3. If the benefits and costs of the projects in the program are deemed to influence beyond the project sites of the proposed projects, base and project case scenarios may need to incorporate additional sections of road beyond projects 1 and 4.

Table 5 contains the results of the program using the sequential approach.

Table 5: Results of Program using the Sequential Approach

The value of the road user costs calculated using the sequential approach are different from the value of the road user costs calculated using the simultaneous approach. The different assumptions applied to the base and project cases cause this difference. Neither approach is incorrect but each approach reflects a different purpose. Benefits calculated using the simultaneous approach reflects the overall benefits directly attributed to each project when the program is evaluated as a whole. The sequential approach reflects the incremental benefits each project adds to the program on the inclusion of each project to the program, thus provides a more accurate apportionment of benefits to each project within the program. The NPV of the program under either approach should be the same if capital expenditure is assumed to occur in the same financial year under each approach. If ranking influences the financial year of construction of the projects, the NPV of the program will change to reflect project timing.

The primary functions of the sequential approach are to assess the incremental impact of including each additional project to the program and provide an economic value for the entire program, measured by NPV. The sequential approach can also incorporate the impact of project timing on the evaluation of the program that is not possible with the simultaneous approach. The sequential approach does not primarily have a ranking function but if the results from the sequential approach suggest that certain projects have strong incremental impacts, the ranking of projects could be reconsidered.

3. Methods

Two approaches to program evaluation have been discussed. These approaches can be applied together or independently to evaluate programs and rank the projects within these programs. In this paper, three methods are discussed. Method A applies the simultaneous approach to the options analysis of projects and is also used to rank projects within a program. Method B applies the sequential approach to the quantitative evaluation of a program where projects have been ranked according to a qualitative criterion. Method C applies the simultaneous approach to the options analysis and initial ranking of projects early in the concept phase of a program and the sequential approach after projects have been ranked in the program and when data is more readily available.

3.1. Method A

Method A involves a quantitative evaluation of project options for every project in the program. The quantitative evaluation proposed in most cases would be a rapid cost benefit analysis (CBA)¹³ . A rapid CBA is possible if traffic data, road characteristics¹⁴ and base or preliminary cost estimates are available for proposed upgrades. A detailed CBA is not normally possible given time, data and costing limitations. The simultaneous approach should be applied to the rapid CBA if projects do not have a pre-determined ranking. The incremental net benefit investment ratio (INBIR) or incremental benefit cost ratio (IBCR) should be used to rank project options assuming other project constraints have been considered. The criterion proposed in this paper is that the IBCR should be greater than the BCR of the marginal project (BCR_MP) to qualify to the program¹⁵ , i.e. IBCR>BCR_MP, since at this stage the BCR_MP is not known, a criteria of BCR>cut-off BCR (Miller 2005), can be applied and revised later if required.

Once the options have been evaluated and the most appropriate option has been selected for each project, the projects should then be ranked according to net benefit investment ratio (NBIR)¹⁶ or BCR until the program budget is reached. The remaining projects are not selected for the program. If the last project to qualify for the program based on the above criteria exceeds the budget, another option for that project or another project with a lower BCR and a lower capital cost can replace the last project to qualify for the program. The BCR of the last project to qualify for the program (BCR_MP) can be used as a new cut-off BCR. This cut-off BCR can be used to moderate the project options selected. The NBBHOL is an example of where method A was applied to evaluate a program.

Method A is a quick method of evaluating a program early in the concept phase to provide decision makers with a good idea of the optimal combination of projects to include in a program. Method A is ideal for programs where the interdependency between most projects is limited and when additional data is likely to have little impact on the benefits and distribution of benefits to projects across the program. Method A does not provide much rigour to programs with many complex interdependent projects. Method A also cannot be adequately applied to a large program with timelines that span over a number of financial periods as all projects are assumed to be evaluated in the same period.

3.2. Method B

Method B involves an initial qualitative analysis of all options for all projects in a program. A strategic merits test (SMT)¹⁷ is often applied to determine which project options most closely meet the requirements of the program. A strategic merits test requires a minimal amount of data to make a decision as to which projects should be of high priority. A strategic merits test allows for project prioritization very early in the concept phase of a program.

The sequential approach is then applied to the program. Projects are evaluated in the order they are prioritized according to the strategic merits test. Sufficient data should be available at this stage to conduct an accurate CBA. The BCR or NBIR produced by the CBA can be applied to change the ranking of projects if required. The CBA should also produce an NPV to determine the quantified net gain to society from the program. Sensitivity analysis of key parameters of each project should be conducted to add additional rigour to the CBA.

Method B provides a rigorous approach to evaluating a program; interdependencies between projects are recognised and apportioned appropriately. Qualitative aspects of the program are considered early in the concept phase and not neglected. Method B incorporates project timings and the impact of these timings on the net welfare of society. Method B does not apply a quantitative approach to rank projects but rather adjusts rankings when required. Method B does not apply any quantitative analysis to project options, optimal project options are likely to be overlooked. Method B could become a reactive approach to program evaluation where projects in the program are evaluated with rigour but project options and ranking are insufficiently considered producing a program with suboptimal projects, which are inefficiently ranked.

3.3. Method C

Method C incorporates both the simultaneous approach and sequential approach to program evaluation. As described in Method A, the simultaneous approach is applied to the options analysis of projects to determine the optimal projects to be included in the program and then further applied to rank projects in the program based on returns to investment using either the BCR or NBIR. The sequential approach as described in Method B is applied later in the concept phase of the program to determine the incremental benefits of each project to the program and provide more rigour to the analysis of each project. If necessary, projects can be re-ranked according to the results produced by the sequential approach. An NPV will be produced to determine the quantified net gain to society from the program and a sensitivity analysis of key parameters will add even further rigour to the analysis of the program.

Method C enables the projects within a program to be ranked and evaluated in early in the concept phase as well as providing more rigorous evaluation later in concept phase when projects have been prioritized and data is more readily available. Complex interdependencies between projects are evaluated twice using both approaches to ensure benefits are adequately evaluated and apportioned. Method C of the three methods is the most thorough quantitative approach to evaluating a program. Method C is time consuming and not appropriate for simple programs with projects that are not interdependent on each other. The qualitative aspects of the program may become neglected, if only quantitative approaches are applied to the evaluation.

3.4. Summary of Methods

All three of the above methods have merit and can be applied successfully under different circumstances. Method A is appropriate for smaller programs with limited number of dependencies between projects and intervals in construction periods of projects are short. For programs of this nature, not evaluating projects in the order of proposed construction will not have a large impact on the value of benefits and costs calculated for each project within the program. Method B is appropriate for programs that strongly support strategic objectives of governments where quantitative values are of less significance to ranking. Method C is appropriate for large programs and programs with a large number of complex and interdependent projects.

4. Case Studies

The methods described in Section 3 have been broadly applied to two recently evaluated programs in TMR. The Nation Building Bruce Highway Overtaking Lane (NBBHOL) Program between Curra and Childers has been evaluated using Method A and the Warrego Highway Upgrade (WHU) Program has been evaluated using Method B. These programs demonstrate the real life application of the methods described in Section 3.

4.1. Nation Building Bruce Highway Overtaking Lane Program

The NBBHOL Program is a proposed $40,000,000 upgrade of the Bruce Highway between Curra and Childers. The program consists of nine overtaking lanes over a stretch of 40 km. The objectives of the program are to improve travel time and safety of road users by allowing safe overtaking of heavy vehicles using an additional lane rather than overtaking into oncoming traffic. Method A in part is applied to this program. The options analysis was not conducted as described in Section 3.1, but was instead based on engineering design parameters of overtaking lanes (Department of Main Roads 2002). Method A was applied to the evaluation and ranking of projects in the program.

The close proximity of the projects and the overtaking lanes that currently exist on this section of the Bruce Highway strongly implies that the proposed overtaking lanes are interdependent. Figure 2 diagrammatically represents the location of the current and proposed overtaking lanes in respect to each other.

Figure 2: Bruce Highway Overtaking Lanes

Source: Figure 2, Benefit Cost Analysis, Nation Building Program – Overtaking Lanes, TMR 2010a

Existing overtaking lanes are depicted in black and the proposed overtaking lanes are depicted in red. Overtaking lanes are assumed to have a 5km downstream and a 3km upstream (Austroads 2001). The proximity of the overtaking lanes to each other causes the upstream and downstream areas to overlap; for example, Project 1 does not have an upstream area and only a downstream area of 3.1km because of the location of Projects 8 and 9. Projects 1 and 9 combine with existing single overtaking lanes to form side-by-side and head-to-head overtaking lanes respectively (TMR 2010a). Table 6 contains the ranking of projects and the results of the evaluation of the program.

Table 6: Results of NBBHOL Program (Discount Rate of 4.4%)

Source: Executive Summary, Benefit Cost Analysis, Nation Building Program – Overtaking Lanes, TMR 2010a

Project 6 has been ranked first according to BCR; the higher traffic volumes further south is the fundamental reason why Project 6 had a very high BCR (TMR 2010a). Project 6 does not have a ‘competitive’ relationship with any of the other proposed overtaking lanes. All the proposed projects were included in the program as all projects had BCRs greater than the program cut-off BCR of 1. The program overall had a BCR of 2.66 at a discount rate of 4.4% indicating that the program improves the net welfare of society.

4.2. Warrego Highway Upgrade Program

The WHU program is a proposed $670,000,000 upgrade of the Warrego Highway. This program contains a variety of different projects ranging from overtaking lanes to intersection upgrades to bridge strengthening projects. In 2010, the program was projected to contain twenty-seven projects but this figure has been revised several times as budget constraints have changed. In this paper, the 2010 approach to the evaluation of the program is discussed. Method B was applied to the evaluation of the program. The Darling Downs Regional office conducted the options analysis and qualitative evaluation of projects (TMR 2010b). The qualitative analysis was used to rank projects within the program. The sequential approach to program evaluation was applied based on the order the projects were ranked within the program.

Altogether twenty-four evaluations were conducted for the twenty-seven projects of the WHU program. There were fewer evaluations than projects mostly due to the rapid nature of the program evaluation and the interdependent nature of the projects due to their proximity and influence on projected traffic flows.

The evaluation of the road widening along the Warrego Highway between Macalister and Chinchilla contained two projects, the widening of the section between Macalister and Warra and the widening of the section between Brigalow and Chinchilla. As the two projects were adjacent to each other, they shared key characteristics and had benefits and costs that overlapped; therefore, the two projects were combined and treated as one evaluation. The rapid nature of the Program CBA did not sufficiently allow the two projects to be split and therefore, combined NPV and BCRs were derived.

The evaluation of duplication of the Warrrego Highway between Nugents Pinch Road and Oakey Turnoff contained three projects, the duplication of Nugents Pinch Road to Gowrie Mountain, Gowrie Mountain to Kingsthorpe Road and Kingsthorpe Road to Oakey Turnoff. The project sites of the duplication upgrades were close to capacity and high projected traffic flows would result in the project sites reaching capacity in the base case before the end of the evaluation period. In the project case, the extra lanes are expected to prevent the sections of road reaching capacity, therefore inducing higher traffic volumes. The induced traffic from the upgrade of the first section of the Warrego Highway will flow into the second and third sections. The benefits to the induced traffic is a result of the upgrade of the three projects rather than any individual project, therefore the benefits are attributed to the three projects as these benefits cannot be fully realised unless all three projects are constructed. Additional costs of the duplication projects on the sections of road following the Oakey Turnoff were also included and attributed to the three duplication projects.

Table 7 contains the results of the evaluation of the program and the predetermined ranking of the projects prior to the application of the sequential approach.

Table 7: Results of WHU Program (Discount Rate of 7%)

Sources: Appendix B, Rapid Benefit Cost Analysis Warrego Highway Investment Strategy, TMR (2010c) & Appendix National Network Investment Program – Regional Input, TMR 2010b

Jingi Jingi Bridge upgrade produced the highest BCR of 16.6. The Jingi Jingi Bridge upgrade involved upgrading a bridge with several structural problems. These structural problems will force heavy vehicles to take a diversion to avoid crossing this bridge. The cost of diverting for these heavy vehicles was estimated to be very high as the traffic growth rates of heavy vehicles over the next twenty years along this section of the Warrego Highway were predicted to be high (TMR 2010c). The overtaking lane projects produced high BCRs that dropped as the highway progressed further west where traffic volumes and projected traffic growth rates are lower. Unfortunately, even though the overtaking lane projects have high BCRs, they do not have a high priority in the WHU program.

The WHU program has a BCR of 1.89 at a discount rate of 7%, which indicates the program as a whole improves net welfare to society even though, only 10 of the 24 projects have BCRs above one and three of the top four ranked projects have BCRs below one. The primary justifications of most of the projects in the program are not determined by quantitative benefits but instead by qualitative benefits. This approach hinders the transparency of the ranking and selection of projects.

The ranking of projects in the program do not closely correspond with the BCR. The qualitative assessment of projects produced by the Darling Downs Region was focused on strategic objectives rather than individual project viability. The application of the sequential approach to the evaluation of the program produced results that highlighted the weakness of the qualitative approach applied to the ranking of projects. The WHU program was revisited in late 2011 and was under re-evaluation in early 2012, ranking and selection of projects have changed since the 2010 report.

5. Advantages and Disadvantages of Program Evaluation

Quantitative evaluation of programs has a number of advantages and disadvantages compared to evaluating projects individually. The advantage emphasized in this paper is the identification, evaluation and apportionment of benefits of interdependent projects in a program. For the program to receive the benefits or costs evaluated from interdependent projects suggests that projects selected for a program should be based on proximity rather than project type. The problem with selecting projects for a program based on proximity is that funding is allocated based on project type or project objectives such as safety. Projects will need to be extracted from the program they are evaluated in and submitted into the appropriate respective federal or state programs.

Program evaluation allows for improved network planning as relationships between different types of projects becomes more apparent (Davies 2011). For example, adding additional lanes to a road may generate traffic, while an adjacent intersection requiring safety improvements will require increased capacity to cater for the traffic generated from the road upgrade. The need for extra capacity could be overlooked if the projects are not evaluated as a program. Flood immunity projects are another example of projects that are likely to be interdependent; projects can be ‘competitively’ related if located on competing links to a location or ‘complementary’ related if along the same link to a location (Davies 2011).

Quantitatively evaluating project options and projects together as a program improves the rigour and transparency of option selection and project ranking. When projects are presented as part of a program the quantitative rational for project selection is transparent as projects are visibly compared with each other rather than submitted in isolation for funding. Program evaluation reduces the chances of omitting projects from evaluation if all projects are presented clearly in a program.

Evaluating projects as part of a program saves time and resources. Program evaluations can be completed faster as duplication of tasks are reduced. For example, all projects can be included in a single program report rather than several individual reports. In 2012, the TMR CBA Team is developing a CBA program model that can be applied to a wide range of projects. This model is designed to produce rapid evaluations of all projects and a range of results for the overall program (TMR 2012a).

Quantitative evaluation of programs still has its disadvantages. One such disadvantage is that urgent projects may not be addressed quickly if the funding request for the project depends on the evaluation of other projects in the program. Program evaluation is also likely to be more complex than project evaluation, thus requiring higher levels of expertise that may not always be available or affordable. If a program evaluation is conducted erroneously, a large amount of resources could easily be misallocated. As mentioned in Section 3, there are various methods that can be applied to evaluating a program; the most appropriate method is not always obvious, thus creating possible inconsistencies. Table 8 summarizes the advantages and disadvantages of the approach to program evaluation described in this paper.

Table 8: Summary of Advantages and Disadvantages of Evaluating Programs

The advantages described in this section present a reasonably strong case to support the application of program evaluation to road programs. Some of the disadvantages described can be mitigated with policy changes to align program evaluation with funding requirements.

6. Policy requirements and frameworks

The NBBHOL Program (Curra to Childers) and the WHU Program are two examples of entire programs submitted for state or federal funding. The overtaking lanes in the NBBHOL Program have been approved for funding under the Nation Building Program and commenced construction in March 2011 (Department of Infrastructure and Transport 2012). The WHU Program is to be joint funded by the Queensland Government and the Commonwealth Government of Australia (Queensland Government 2012). The Queensland Government has approved funding for some of the higher-ranking projects of this program but many of the projects remain unfunded as at February 2012. Road projects are typically evaluated individually before submission for funding to either state or federal government programs that are categorized according to broad objectives such as road safety; heavy vehicle safety and productivity (Department of Infrastructure and Transport 2012); and flood recovery (Department of Local Government and Planning 2012). Programs of work with a diversity of projects that address various aspects of a network may not directly receive funding, but instead projects are required to be submitted individually for funding to different state and national programs (Department of Transport and Regional Services 2006). The NBBHOL program received funding because all the projects in the program met the requirements of the Nation Building Program. The WHU program has a diversity of projects that do not meet the requirements of any individual government program.

Federal and state policy should allow entire programs to be submitted for funding even if these programs contain a variety of projects with different objectives. Splitting programs according to project type or objective could affect the viability of the entire program if some of the projects do not receive funding. Splitting programs will also require the preparation of additional reports, which will be time consuming and may create the perception that the program report is redundant if it is not required for funding. The Nation Building Program caters for program evaluations to a certain extent by allocating funds per major highway but place a cap on the amount available per project type (National Land Transport 2009). If policy changes are not made, programs can still be evaluated based on project type or objectives of projects but at the expense of the evaluation of some of the benefits and costs of interdependent projects.

7. Conclusion

In this paper, two approaches to quantitative evaluation of programs have been suggested and three methods of applying the two approaches have been discussed. The application of each method depends on the nature of the programs evaluated. The quantitative evaluation of programs offers many advantages in regards to improved rigour in options analysis, project ranking, and recognition of benefits and costs attributed to interdependent projects within a program.

The methods described in this paper have been applied by TMR to evaluate programs with mixed success. The NBBHOL Program has been funded and construction of the proposed projects began in March 2011. The simultaneous approach described in this paper was applied to the program evaluation and the projects in the program were aligned with the Nation Building Program. WHU Program has a wide variety of projects, not all of these projects can be funded under one state or federal program. As at February 2012, the Queensland Government has funded some of the higher-ranked projects while most projects still wait funding from the Australian Government. If program evaluation is to become a standard method of evaluation for transport and road departments, state and federal programs will need to be changed to facilitate the submission of programs with a diversity of projects that have a diversity of objectives. The Nation Building Program has taken a positive step in supporting the submission of programs for funding, which has prompted the program evaluation of programs such as the NBBOL program.

End Notes

Warrego Highway Upgrade Program is also referred to as the Warrego Highway Investment Strategy or the Warrego Highway Upgrade Strategy from various sources.
The base case is the state of the world in the absence of the initiative being implemented, The project case is the state of the world with the initiative being implemented (ATC 2006, 105).
The concept phase referred too in this paper is the first stage in the project life cycle according to OnQ Project Management Methodology (TMR 2012b).
A road duplication project is designed to double the number of existing lanes of a road (TMR 2011a, 3.138)
The cut-off BCR is minimum BCR required for a project to be accepted into a program. In this example the BCR is applied to project ranking, this is consistent with Infrastructure Australia frameworks (Infrastructure Australia 2010).
Project X denotes a project with options that have an impact on other options.
Project Y denotes a project with options that are impacted by other options.
Tao and Schonfeld (2006), adopt a similar approach of evaluating projects together to determine if the projects are ‘competitive’ or ‘complementary’. ‘Competitive’ projects are projects that have benefits that can be in part, addressed by just one or some of the proposed projects. ‘Complementary’ projects are projects that have benefits that can be only obtained if both or all projects are constructed.
COM is the option compared with for the calculation of the IBCR
Project X Option 1 combined with Project Y Option 1
The optimal project combination of X3Y1 is determined through the application of the IBCR. Option X1Y1 has the lowest capital expenditure with a BCR greater than 1 and therefore is the base for the first comparison. As Options X1Y2 and X2Y1 have the same capital value, they are compared with Option X1Y1. Both options have IBCRs above 1 but X1Y2 has the higher IBCR and therefore, is used as the base for comparison with higher capital cost options. As Options X1Y3, X2Y2 and X3Y1 have the same capital value, they are compared with Option X1Y2. Option X3Y1 is the only option with an IBCR greater than therefore is the base for comparison with higher capital options. As Options X2Y3 and X3Y2 have the same capital value, they are compared with Option X3Y1. Neither Option X2Y3 nor X3Y2 have IBCRs above 1, therefore X3Y1 remains the base for comparison with X3Y3, the highest capital option. Option X3Y3 has an IBCR of less than 1, therefore Option X3Y1 is the optimal option for Project X and Project Y.
Number of possible combinations for three projects with four options each is 4×4×4=64
CBA is the analysis of the benefits and costs to society of a proposed initiative. CBA aims to value benefits and cost in monetary terms and provide a summary indication of the net benefit (ATC 2006, 105).
A road management information system (ARMIS) data is regularly updated for the majority of the network, thus providing the analyst with sufficient road and traffic data to conduct a basic or rapid CBA (Austroads 2002).
The marginal project or incremental project is the project with the lowest economic value per dollar invested, which has been ranked according to monetised economic benefits. The BCR of the marginal project can act as a BCR cut-off value for which other projects can be compared too. Projects with lower economic value per dollar and are not prioritized based on monetised economic benefits and should not be considered (Davies 2010).
The NBIR is another economic indicator that can be applied to project ranking (Campbell and Brown 2007).
SMT is normally a qualitative series of questions that provides a first-order determination of the ‘strategic merit or fit’ of an identified initiative. SMT identifies proposals that should proceed to the next stage of appraisal, proposals that require further scoping, and proposals that should be abandoned because they lack strategic fit. SMT also includes checks to ensure that the initiative has been properly formulated and is feasible (ATC 2006, 110).
For combined projects, the NPV is the total NPV for the two projects and not the NPV for the individual projects.
‘General’ refers to the reduction in benefits of the existing overtaking lanes because of the reduction in their upstream and downstream areas based on the proximity of the proposed overtaking lanes.
Two projects combined into one evaluation.
Three projects combined into one evaluation.
More than 80% of projects on QTRIP Live do not have benefit cost ratios (TMR 2011b)
The CBA team has received verbal feedback from visits to Regional TMR offices that less prominent projects are considerably less likely to receive funding than projects that receive more attention.

References

Australian Transport Council (2006): National Guidelines for Transport Systems Management in Australia, 3 Appraisal of initiatives, Commonwealth of Australia, Canberra, 105 and 110

Austroads (2001), Economic Evaluation of Road Investment Proposals: Improved Predictions Model for Road Crash Savings, Austroads, AP-R184

Austroads (2002), Road Network Asset Management: International Benchmarking Study, Austroads, AP-R204, Sydney

Campbell, H. F. and Brown, R. P. C., (2007), Benefit-Cost Analysis Financial and Economic Appraisal Using Spreadsheets, Cambridge University Press, Cambridge

Davies, W. (2010), Proposed Modifications to the Cost Benefit Analysis Decision Criteria for Road Project Evaluation to Improve Decision Making, available at: www.wmdblueberry.com, accessed 19/03/12

Davies, W. (2011), Advanced Methods of Evaluating Benefits from Improved Flood Immunity in Queensland, Australasian Transport Research Forum 2011

Department of Infrastructure and Transport (2012), Nation Building Program, Australian Government available at: www.nationbuildingprogram.gov.au, accessed on 19/03/12

Department of Infrastructure and Transport (2012), Bruce Highway – Curra to Sarina – Overtaking Lanes, Australian Government available at: www.nationbuildingprogram.gov.au, accessed on 19/03/12

Department of Local Government and Planning (2012), Natural Disaster Relief and Recovery Arrangements, Queensland Government, available at: dlgp.qld.gov.au, accessed on 19/03/12

Department of Main Roads 2002, Road Planning and Design Manual 2002, Queensland Government

Department of Transport and Regional Services (2006), Auslink Investment Program: National Projects – Notes on Administration, Australian Government

Duncan, W. R. (1996), A Guide to the Project Management Body of Knowledge, Project Management Institute, USA

Fox, G. E., Baker, N. R. and Bryant, L. J. (1984), Economic Models for R&D Project Selection in the Presence of Project Interactions, Management Science, 30, 890-902

Gear, T. E. and Cowie, G. C. (1980), A Note on Modeling Project Interdependence in Research and Development, Decision Science, 11, 738-748

Infrastructure Australia (2010), Reform and Investment Framework Templates for use by Proponents, Australian Government

Miller, E. M. (2005), The Cutoff Benefit-Cost Ratio Should Exceed One.: An article from: Engineering Economist, Institute of Industrial Engineers

National Land Transport (2009), Notes on Administration for the Nation Building Program, Australian Government

Nemhauser, G. L. and Ullmann, Z. (1969), Discrete Dynamic Programming and Capital Allocation, Management Science, 15, (9), 494-505

Queensland Government (2012), Warrego Highway Upgrade Strategy 2012, Queensland Government

Tao, X. and Schonfeld, P. (2006), Selection and Scheduling of Interdependent Transportation Projects with Island Models, Transportation Research Record: Journal of the Transportation Research Board, No. 1981, Transportation Research Board of the National Academies, Washington, D.C., 133-141

Transport and Main Roads (2010a), Benefit Cost Analysis: Nation Building Program – Overtaking Lanes, Queensland Government (unpublished)

Transport and Main Roads (2010b), National Network Investment Program – Regional Input, Queensland Government (unpublished)

Transport and Main Roads (2010c), Rapid Benefit Cost Analysis: Warrego Highway Investment Strategy, Queensland Government (unpublished)

Transport and Main Roads (2010d), RIP Benefit Cost Analysis Status Report, Queensland Government (unpublished)

Transport and Main Roads (2011a), Cost-benefit Analysis Manual: Road Projects, Queensland Government, Brisbane

Transport and Main Roads (2011b), QTRIP Cost Benefit Analysis Status Report and Recommendations for Future Action July 2011 Update, Queensland Government (unpublished)

Transport and Main Roads (2012a), Program Delivery Improvement (PDI) Specialist Advice Team 2011-12 PDI Teams Program of Work (Summary), Queensland Government (unpublished)

Transport and Main Roads (2012b), Project Management Reference Guide Appendix A On-Q, Queensland Government