Peer review of proposals

This page relates to the 2021-24 National Land Transport Programme.

Introduction

This page outlines the requirements of peer reviews to ensure the robustness evaluations of an approved organisation or Waka Kotahi NZ Transport Agency (for its own activities), and to reduce risk to the delivery of desired outcomes from our (Waka Kotahi as investor) investment decisions.

The purpose of the peer review is to reduce the risks that projects either do not deliver on the outcomes forecast in the funding application, or they fail to deliver the outcomes at the level of efficiency and effectiveness stated in the application.

Requirements of an improvement activity peer review

We require a peer review of improvement activity business cases with estimated whole-of-life costs over $15 million or where a significant level of risk is involved.

We encourage approved organisations and Waka Kotahi (for its own activities) to have small projects (between $2 million and $15 million implementation cost) peer reviewed if the cost and/or benefit risks associated with these are considered high or the applicant lacks experience in the development and implementation of such projects. In any event, all small project evaluations should be internally peer reviewed.

Raising concerns

The reviewer must raise in writing with the applicant organisation any:

• discrepancies or departures from the requirements of this knowledge base or the Monetised benefits and costs manual (MBCM)
  Monetised benefits and costs manual
• material concerns about the activity as a result of the review.

The reviewer must request that the applicant organisation:

• responds to the reviewer within a reasonable period
• provides reasons for any discrepancies or departures and answers to resolve the reviewer’s concerns.

The reviewer must note any outstanding concerns in the review report.

Scope of an improvement activity peer review

The peer review of improvement activities must include a range of considerations.

Selection and independence of peer reviewer

We require an independent, external peer review for any improvement activity business case with estimated whole-of-life costs over $15 million or any that involves a significant level of risk. Where a peer review is required or warranted, the peer reviewer shall be selected and appointed by the applicant and must:

• be independent of the organisation and the project, unless otherwise formally agreed with us
• take an objective, professional stance at all times in undertaking the peer review, keeping in mind that the key aim of the review is to reduce the risk to the delivery of desired outcomes, including economic efficiency, from our investment decisions
• be competent with respect to the specific nature of the activity and not exceed its level of competence in undertaking the review.

For very large, complex programmes and activities, a peer review panel, covering a range of competencies, may be more appropriate.

We reserve the right to undertake our own peer review of any activity or to require the approved organisation or Waka Kotahi (for its own activities) to appoint a specific peer reviewer or to establish a peer review panel with appropriate competencies. 

Conformity

The reviewer must first determine whether the activity is eligible for funding in that it fits the description of one of the activity classes in the current Government Policy Statement on land transport (GPS).

Government Policy Statement on land transport 2021(external link)

The reviewer must ensure that the activity evaluation conforms to the requirements of this Knowledge Base, including that it has been assessed by the applicant in conformance with the Waka Kotahi Investment Prioritisation Method.

2021–24 NLTP Investment Prioritisation Method 

Credibility

To check credibility, the reviewer must:

• ensure the transport issue, priority or opportunity has been identified, is reasonable and is adequately described.
• critically assess the results of each stage of the activity’s economic efficiency evaluation, avoiding unnecessary detail where possible. The test as to the level of detail to consider is whether the conclusion reached in the report is a reasonable and credible result from the information and data used in the analysis.
• assess the costs estimated for the activity and consider how realistic these are, taking into account current market rates.
• identify the key benefits and determine whether they are realistic (eg are the travel time savings realistic or are excessive delays being forecast under congested conditions in the do-minimum?). Some quick ‘back-of-the-envelope’ calculations are necessary to check the level of forecast benefits.
• identify the factors or assumptions, particularly forecasted estimates that have a major influence on the evaluation. Describe each of these factors/assumptions and include a commentary on the sensitivity of the evaluation to each factor or assumption.
• highlight any significant areas of risk for costs and benefits.

Choice of do-minimum

The reviewer must assess the do-minimum as stated in the activity report and must determine whether it is realistic, and does not represent another option to be considered in the analysis. 

Identification and selection of alternatives and options

The reviewer must examine the evaluation and judge whether all feasible alternatives and options have been identified and considered adequately. These should consider the hierarchy of interventions, including low-cost options.

The reviewer needs to be satisfied that the process for selecting the preferred alternative and option(s) has been robust and includes incremental assessment where appropriate.

GPS alignment rating

The reviewer needs to be satisfied that the GPS alignment rating for the activity is correct. 

Cost estimate

The reviewer shall check compliance with parallel cost estimate process requirements, where applicable. 

See the Cost estimation manual (SM014) for details.

Cost estimation manual (SM014)

Cost–benefit appraisal rating

The reviewer must determine whether the cost–benefit appraisal has conformed to all the relevant requirements of the MBCM. The reviewer must determine whether there are any outstanding issues not addressed in the activity report.

Monetised benefits and costs manual

If there is a departure from the requirements, or any defect or omission, the reviewer must comment on its significance.

Where the reviewer considers there have been discrepancies and departures from procedure, or has concerns about cost and/or benefit estimation, the reviewer will determine the activity benefit–cost ratio (BCR) and compare this with the applicant’s calculations.

The reviewer must determine whether the options identified in the analysis are mutually exclusive options of the same activity. If the options identified:

• are mutually exclusive, then the reviewer must determine that an incremental assessment of the options has been carried out correctly, as set out in the incremental assessment section of this knowledge base, or
• are not mutually exclusive, then by definition they must be either:
  • independent activities, in which case the reviewer must determine that the analysis has been undertaken in terms of independent activities and has been undertaken correctly, and should comment whether they should be resubmitted as separate activities; or
  • inter-dependent activities, eg components of a package, in which case the reviewer should consider whether the analysis that has been undertaken is valid.

In special cases, other economic impacts may be considered (eg wider economic benefits). These are to be shown as sensitivity analyses, in addition to the MBCM procedure economic analysis.

Where supplementary (third party) funding is involved, a government BCR must be determined in addition to the national BCR. 

Risk assessment, analysis and mitigation

The reviewer must ensure that:

• risks have been assessed adequately in the applicant’s evaluation
• realistic mitigation measures have been considered
• a full risk analysis has been undertaken for large/complex and high-risk activities. 

Sensitivity analysis

The reviewer must consider whether the sensitivity of critical aspects of the activity evaluation has been covered off adequately, paying particular attention to:

• key assumptions that underlie the activity and its delivery of desired outcomes, in particular future growth and demand assumptions
• information and data values that are ‘out of the ordinary’ or unusual
• the sensitivity of the activity’s outcomes to the input parameters. 

Requirement for external peer reviews of research projects

We require all projects in the research programme to undergo external peer reviews. The participation of peer reviewers is integral to the success of the research programme and ensures that we receive a satisfactory report at the completion of a research project.

Peer review guidelines and the agreement form for research projects are available on the Waka Kotahi Research programme page.

Research programme

Timing of peer review reviews of research projects

Peer reviews are required on completion of a research project and before the presentation of any final report to Waka Kotahi for editing and publishing. Peer reviews may also be required after the completion of individual stages or tasks if specified as milestones in the approved research project plan.

The researcher must organise the peer reviewers and peer review process. This requirement is set out in the procurement documents for research projects.

The research proposal must include a clear description of the reviews that will be undertaken during the course of the project, at what point those reviews will be completed, and by whom. The reviewers must be expert peers or users who will have given their prior agreement to the project’s timeframe. This will ensure the timely provision of comments or advice.

Nomination for peer reviewer

Peer reviewers independently review research reports, whereas steering group members may be called on to act as mentors during the progress of a project.

Reviewers are often in a position to provide guidance and professional advice.