Producers of statistics and data should explain clearly how they assure themselves that statistics and data are accurate, reliable, coherent and timely.
Q3.1 Statistics should be produced to a level of quality that meets users’ needs. The strengths and limitations of the statistics and data should be considered in relation to different uses, and clearly explained alongside the statistics.
Q3.2 Quality assurance arrangements should be proportionate to the nature of the quality issues and the importance of the statistics in serving the public good. Statistics producers should be transparent about the quality assurance approach taken throughout the preparation of the statistics. The risk and impact of quality issues on statistics and data should be minimised to an acceptable level for the intended uses.
Q3.3 The quality of the statistics and data, including their accuracy and reliability, coherence and comparability, and timeliness and punctuality, should be monitored and reported regularly. Statistics should be validated through comparison with other relevant statistics and data sources. The extent and nature of any uncertainty in the estimates should be clearly explained.
Q3.5 Systematic and periodic reviews on the strengths and limitations in the data and methods should be undertaken. Statistics producers should be open in addressing the issues identified and be transparent about their decisions on whether to act.
The Consumer Price Index including Owner Occupiers’ Housing Costs (CPIH) is published monthly by the Office for National Statistics (ONS) in its UK Consumer Price Inflation bulletin.
ONS publishes information about the quality of the Valuation Office Agency (VOA) private rents data, which are used to estimate owner occupiers’ housing costs, a key component of the inflation measure:
- Its Quality Assurance of Administrative Data (QAAD) document includes a detailed overview of the quality of these data, as well as a range of other administrative and non-administrate data sources
- ONS produced a flow diagram (PDF) that provides an overview of the quality assurance processes of the private rents data at each stage of data collection and processing
ONS communicates clearly with VOA to understand the quality assurance of these data. ONS is currently looking into gaining access to the private rents microdata, using the powers granted through the Digital Economy Act 2017. This is expected to help ONS further understand data quality issues.
In addition, ONS has developed several comparative analyses to provide assurance to itself and to users about the behaviour of CPIH:
- One analysis compared different methods of estimating owner occupiers’ housing (OOH) costs
- Another analysis compared the CPIH private rents data with other data source
By publishing clear and detailed information about data quality assurance and embedding quality assurance practices in its production process, ONS provides reassurances to itself and users about the quality of the data used to produce CPIH.
The NHS Digital Mental Health Act annual statistics bulletin contains official statistics about uses of the Mental Health Act in England during 2016/17.
In 2015 NHS Digital announced changes to the way it sources and produces these statistics. Previously these statistics were produced from the KP90 aggregate data collection. They are now produced from the Mental Health Services Data Set (MHSDS). This transition to a new data source was a cost saving factor as well as a programme of work to improve data quality. MHSDS provides a much richer data source for these statistics, allowing for new insights into uses of The Act.
For the October 2017 release, NHS Digital published the annual statistics with the new data source, and also produced a background data quality report that clearly communicates this assurance to their users. The document highlighted the improvements to data, methods and source and provided information on data relevance, reliability, coherence, timeliness, and clarity. NHS Digital included detailed information on, and published, missing data they were using to identify the most efficient way to increase coverage. Other positive mentions within the report include a section on the trade-offs between output quality components.
The report is a good example of what to include in a background quality document that accompanies the statistics bulletin and the data. The result of the overall quality improvements ensures the Mental Health Act annual statistics fit their intended use.
|Guidance on quality assuring administrative data used to create statistics. It includes explanatory notes, case examples, FAQs, the actual toolkit (audit questionnaire), and questions to prompt thinking when conducting the audit.||Quality Assurance of Administrative Data (QAAD)||UKSA|
|Guidance on quality assuring management information (MI) – aggregate information collated during the normal course of business to inform operational delivery, policy development or the management of performance. It includes examples of the practices to use.||Quality Assurance of Management Information (QAMI)||UKSA|
|Another way to access the QAAD and QAMI guidance.||Administrative data and official statistics (QAAD and QAMI)||UKSA|
|Guidelines that aid producers in the production of official statistics. They include information about measuring and reporting statistical quality, quality assurance, and reporting metadata.||GSS Quality Guidelines||GSS|
|National Statistician’s guidance on interpreting and implementing the principles and practices of Code which relate to the quality and methodological basis of their official statistics. This document is scheduled for revision in 2018 to reflect version 2.0 of the Code.||National Statistician’s Guidance: Quality, Methods, and Harmonisation (2009)||GSS|
|A webpage with links to a series of guidance documents on harmonisation, including what harmonisation is and its aims, the Harmonisation Handbook and the GSS Harmonised Principles.||Harmonisation within the GSS webpage||GSS|
|Guidance that provides a common approach to aid the clear communication of uncertainty and change. It can be applied to all sources of information, including surveys, censuses, administrative data and other sources, as well as estimates derived from a combination of these. It includes examples of good practice, as well as standard wording to be used when appropriate. This document is scheduled for revision in 2018 to reflect version 2.0 of the Code.||Communicating Uncertainty and Change: Guidance for Official Statistics Producers||GSS|
|Guidance for producing quality analysis for government. It is used by the UK Government Policy Profession. For those within the Policy Profession, the Code of Practice for Statistics does not supercede the Aqua Book, but complements it.||The Aqua Book||UK Government|
|A template developed by the United Nations (UN) to help countries develop and implement national quality frameworks of their own, or to build on existing ones. The website includes a list of tools and resources.||UN Generic National Quality Assurance Framework (NQAF)||UN|
|The United Nations Economic Commission for Europe’s (UNECE) handbook for using administrative and secondary sources for official statistics provides international methodological guidelines to help those in the early stages of using administrative data.||Using Administrative and Secondary Sources for Official Statistics: A Handbook of Principles and Practices (2011)||UNECE|
|The Quality Assurance Framework of the European Statistical System (ESS QAF) is a supporting document that assists producers with implementing the European Statistics Code of Practice. It contains recommendations on activities, methods and tools that facilitate the practical and effective implementation of quality.||Quality Assurance Framework of the European Statistical System (PDF)||Eurostat|
|The ESS Handbook for Quality Reports provides recommendations on how to prepare comprehensive quality reports for the full range of statistical processes and their output. It also provides detailed guidelines and examples of quality reporting practices.||European Statistical System (ESS) Handbook for Quality Reports (PDF)||Eurostat|