Framework and metric enhancements

The framework and metrics are continually evolving and enhanced to provide more meaningful results.

The ongoing enhancements and changes to the framework and metrics affect the comparability with the published previous annual results.

Transport Insights reports the metric results based on the current framework. This is the best place for reviewing the outcome of any data improvement initiatives undertaken and improvement opportunities.

Below is a summary of changes made by reporting year.

Details regarding changes at a metric level are available in the data quality metric library within Transport Insights.

Data quality metric library(external link)

2023/24

The 2023/24 reports include a few enhancements to the metrics and reporting framework. These planned changes for 2023/24 were made within Transport Insights during April 2024, providing opportunity for RCAs to see their 2022/23 results as well as any early imports for 2023/24 with the changes implemented.

Key changes for 2023/24 include:

• LIGHTS4, RAIL3, and SIGNS5 have been paired to report as a single indicator.
• SIGNS5 minimum height check has been changed to >=50mm.
• RETAIN3 maximum height check has been changed to <10m.
• LIGHTS5 metric now checks unique streetlight lights, rather than streetlight poles.

The descriptions, details and additional notes for each metric have been reviewed and some updated in the metric library. These changes will not affect the calculations of metric results, and will not affect a RCA’s data quality results.

2022/23

Enhancements to the data quality framework and metrics were made for 2022/23.  A number of these were identified through a detailed of the framework on review of the 2020/21 national annual results with implementation deferred until the 2022/23 financial year.

The planned changes for 2022/23 were made within Transport Insights during March 2023 providing opportunity for RCAs to see their 2021/22 results against the enhanced framework.

The 2022/23 changes made are in the following two areas:

• At an overall framework level addressing the balance of the framework across the data categories and quality dimensions tested. Metrics of the same data subcategory, importance level and quality dimension have been combined to form single indicators. These are still reported as individual metrics in the results, and retain their existing metric reference, but are aggregated in the dashboard results and calculation of the Score. Details on which metrics are paired together is available in the data quality metric library with Transport Insights.
• At a metric level considering the individual metrics based on feedback from the sector, bring in industry developments (eg ONF, AMDS, CCDC, etc.) and analysis to gain a deeper understanding of the metric results that have consistently been below the current expected standard and showing little improvement.

2021/22

The 2021/22 reports include further enhancements to the metrics and reporting framework. Key changes for 2021/22 include:

• Changes to the Asset Inventory sub-categories.
• CWAY3, CWAY5 and ESTIM3 have been retired.
• CWAY8 and ESTIM5 have been added to the framework.
• MAINT2, RATING1 and ROUGH1 have been changed to the “Timeliness” quality dimension.
• CWAY6a, CWAY6b, TREAT2a and TREAT2b have been changed to Low importance.
• TREAT5, SURF1a, SURF2, SURF3, SURF4 and SURF5 will now report “NA” for networks with no surfacing data for the reported financial year.
• Other minor changes have been made to TREAT1, SURF1a, SURF1b, SURF2, SURF6, PAVE1, COUNT1.

We have deferred some planned changes until early 2023. Having early visibility of these will provide the opportunity of RCAs to understand the impact of the framework changes on their data quality results prior to them becoming live for 2022/23.

2020/21

The 2020/21 reports include several changes to the metrics and reporting framework, some resulting from sector feedback. As a result, there is a reduction to 60 metrics in the 2020/21 reports. Key changes include:

• SURF2 has been updated to include further surface material types where two recorded chip sizes are expected.
• SURF4 and SURF5 have been updated to report on surfaces with a surface date in the reported financial year only.
• RETAIN1 has been updated to allow for retaining walls that support the road to be recorded with a negative value.
• SWC1 and SWC 2 have been updated to exclude an additional earth channel type (ESWC).
• ROUGH1 has been updated to report the greater percentage when calculated using either the 100m or high-speed roughness data.
• ROUGH2 has been updated to report based on the data table providing the ROUGH1 result.
• MAINT4, MAINT5 and MAINT7 have been combined into a single MAINT4 metric.
• MAINT5 and MAINT7 have been retired.

A data quality framework and suite of metrics for the state highway network have been developed. Several metrics have either been adapted, or excluded, from the existing TLA report due to differences in data sources and asset portfolios. The state highway report consists of 54 metrics and is reported at a Waka Kotahi region level.

The TLA (inc DoC) and Waka Kotahi regional results are reported separately from each other.

2019/20

The previously separate ONRC and asset management reports have been combined into a single Asset Management data quality framework and report. This includes assigning an importance level to the previous ONCR metrics and expanding the ‘Score’ calculation to include all 62 metrics.

With the combining into a single report the metric reference codes have been updated to standardise the codes into a single style for all metrics.

Additional dashboard charts and annual reports show the results by importance.

Metric calculations have been updated as follows, for the relevant metrics:

• Excluding carriageways with an ONRC of Not Required.
• Checking the asset owner type rather than the asset owner.
• Checking the road type and owner type are for the RCA.
• Adjusting calculations to allow for specific situations.

Two metrics have changed their underlying calculations:

• MAINT1 (previously AM-MA1) now reports on units by fault type rather than units by activity.
• LIGHTS3 (previously AM-SL3) reports on light replacement rather than pole replacement.

Metric TREAT4 (TL4) Network coverage with STE reading has been retired.

2018/19

The 2018/19 reports include several changes to the metrics and reporting framework, some as a result of sector feedback. Key changes include:

• Removing the upper time limit for the Demand data category metrics.
• Reporting of two metrics as NA when they are not applicable to a RCA.
• ‘Assets known’ type metrics updated to include condition as an alternative to constructed/installed date.

2017/18

The final 2017/18 reports include several changes to the metrics and reporting framework, some as a result of sector feedback. Key changes include:

• Reporting the work origin for the first time.
• Updated the original cost metric to report for surfaces associated with work category 212 and 214 activities only.
• Updating of the traffic count metrics to include both the sealed and unsealed network.
• Retirement of four ONRC metrics.
• Slackening of the asset management grade ranges for non-high importance asset inventory metrics.
• Rationalising metric dimensions to be consistent across both reports and definitions – including removing the previously reported usability and consistency dimensions. from the asset management report.

2016/17

Minor changes were made to the ONRC data quality metrics and the queries used to determine the results, including reporting results as “NA” when a metric is not applicable to a network. For example, metric TE2a reports as NA for networks that have no carriageway sections with an ONRC category of arterial or higher. Any NA results are also excluded from the aggregated dashboard results.

Prototype release of the new asset management data quality framework consisting of 32 metrics and report. Three metrics in the asset management report include the use of data sourced from Transport Investment Online (TIO) in addition to each RCA’s RAMM data.

Data sourced changed to use the annual import into Transport Insights instead of the Waka Kotahi data warehouse.

2015/16

The second release of the ONRC data quality report. Minor changes have been made to the metrics and the queries used to determine the results. Data sourced from the Waka Kotahi data warehouse.

2014/15

Protype release of the ONRC data quality framework and report. Framework consisting of 33 data quality metrics interrogating each RCA’s RAMM data that underpins the ONRC Performance Measures. Data was sourced from the Waka Kotahi data warehouse.