Peer review of proposals

Waka Kotahi requires a peer review of improvement activity business cases with a likely implementation cost of over $5 million or that involves a significant level of risk.

Approved organisations and Waka Kotahi (state highways) are encouraged to have small projects (between $1m and $5m implementation cost) peer reviewed if the cost and/or benefit risks associated with these are considered high or the applicant lacks experience in the development and implementation of such projects. In any event, all small project evaluations should be internally peer reviewed.

The reviewer must raise any concerns in writing with the applicant organisation for funding assistance (and its representative).

The reviewer must raise in writing with the applicant organisation for funding assistance (and its representative) any:

• discrepancies or departures from the requirements of this Knowledge Base or the Monetised benefits and costs manual (from August 2020), or the Economic evaluation manual (external link)(superseded August 2020)
  
• material concerns about the project as a result of the review.

The reviewer must request that the applicant organisation:

• responds to the reviewer within a reasonable period
• provides reasons for any discrepancies or departures and answers to resolve the reviewer’s concerns.

The review must note any outstanding concerns in the review report.

Waka Kotahi requires an independent, external peer review for any improvement project over $15m whole-of-life cost. Where a peer review is required or warranted, the peer reviewer shall be selected and appointed by the applicant, and must:

• be independent of the organisation and the project, unless otherwise  formally agreed with Waka Kotahi (planning and investment)
• take an objective, professional stance at all times in undertaking the peer review, keeping in mind that the key aim of the review is to reduce the risk to the delivery of desired outcomes, including economic efficiency, from Waka Kotahi investment decisions
• be competent with respect to the specific nature of the project and must not exceed its level of competence in undertaking the review.

For very large, complex programmes and projects, a peer review panel, covering a range of competencies, may be most appropriate.

Waka Kotahi reserves the right to undertake its own peer review of any project or to require the approved organisation or Waka Kotahi (state highways) to appoint a specific peer reviewer or to establish a peer review panel with appropriate competencies.

The reviewer must first determine whether the project is eligible for funding in that it fits the description of one of the activity classes in the current Government Policy Statement on Land Transport (GPS).

The reviewer must ensure that the project evaluation conforms to the requirements of this Knowledge Base, including that it has been assessed by the applicant in conformance with Waka Kotahi’s Investment Assessment Framework.

To check credibility, the reviewer must:

• Ensure the transport issue, priority or opportunity has been identified, is reasonable and is adequately described.
• Critically assess the results of each stage of the project’s economic efficiency evaluation, avoiding unnecessary detail where possible. The test as to the level of detail to consider is whether the conclusion reached in the report is a reasonable and a credible result from the information and data used in the analysis.
• Assess the costs estimated for the project and consider how realistic these are, taking into account current market rates.
• Identify the key benefits and determine whether they are realistic (eg are the travel time savings realistic or are excessive delays being forecast under congested conditions in the do-minimum?). Some quick ‘back-of-the-envelope’ calculations are necessary to check the level of forecast benefits.
• Identify the factors or assumptions, particularly forecasted estimates that have a major influence on the evaluation. Describe each of these factors/assumptions and include a commentary on the sensitivity of the evaluation to each factor or assumption.
• Highlight any significant areas of risk for costs and benefits.

The reviewer must assess the do-minimum as stated in the project report and must determine whether it is realistic, and does not represent another option to be considered in the analysis.

The reviewer must examine the evaluation and judge whether all feasible alternatives and options have been identified and considered adequately. These should include alternative transport modes, where applicable, and low cost options.

The reviewer needs to be satisfied that the process to select the preferred alternative and option(s) has been robust and includes incremental assessment where appropriate.

The reviewer needs to be satisfied that the results alignment rating for the activity is correct.

The reviewer shall check compliance with parallel cost estimate process requirements, where applicable.

The reviewer must determine whether the cost-benefit appraisal has conformed to all the relevant requirements of the Waka Kotahi Monetised benefits and costs manual (MBCM; from August 2020) and  Economic evaluation manual (EEM; superseded August 2020). The reviewer must determine whether there are any outstanding issues not addressed in the project report.

If there is a departure from the requirements, or any defect or omission, the reviewer must comment on its significance.

Where the reviewer considers that there have been discrepancies and departures from procedure, or has concerns on cost and/or benefit estimation, the reviewer will determine the project benefit–cost ratio (BCR) and compare this with the applicant’s calculations.

The reviewer must determine whether the options identified in the analysis are mutually exclusive options of the same project. If the options identified:

• are mutually exclusive, then the reviewer must determine that an incremental assessment of the options has been carried out correctly, as set out in the Incremental Assessment section of this Knowledge Base, or
• are not mutually exclusive, then by definition they must be either:
  • independent projects, in which case the reviewer must determine that the analysis has been undertaken in terms of independent projects and has been undertaken correctly, and should comment whether they should be resubmitted as separate projects; or
  • inter-dependent projects, e.g. components of a package, in which case the reviewer should consider whether the analysis that has been undertaken is valid.

In special cases, other economic impacts may be considered (eg wider economic benefits). These are to be shown as sensitivity analyses, in addition to the MBCM (from August 2020) and EEM (superseded August 2020) procedure economic analysis.

Where supplementary (third party) funding is involved, a government BCR must be determined in addition to the national BCR.

The reviewer must ensure that:

• risks have been assessed adequately in the applicant’s evaluation
• realistic mitigation measures have been considered
• a full risk analysis has been undertaken for large/complex and high-risk projects.

The reviewer must consider whether the sensitivity of critical aspects of the project evaluation has been covered off adequately, paying particular attention to:

• key assumptions that underlie the project and its delivery of desired outcomes, in particular future growth and demand assumptions
• information and data values that are ‘out of the ordinary’ or unusual
• the sensitivity of the project’s outcomes to the input parameters.

Peer reviews are required on completion of a research project and before the presentation of any final report to Waka Kotahi for editing and publishing. Peer reviews may also be required after the completion of individual stages or tasks if specified as milestones in the approved research project plan.

The researcher must organise the peer reviewers and peer review process. This requirement is set out in the procurement documents for research projects.

The research proposal must include a clear description of the reviews that will be undertaken during the course of the project, at what point those reviews will be completed, and by whom. The reviewers must be expert peers or users who will have given their prior agreement to the project’s timeframe. This will ensure the timely provision of comments or advice.

Peer reviewers independently review research reports, whereas steering group members may be called on to act as mentors during the progress of a project.

Reviewers are often in a position to provide guidance and professional advice.