Q2: Sound methods


Producers of statistics and data should use the best available methods and recognised standards, and be open about their decisions.

 

Q2.1 Methods and processes should be based on national or international good practice, scientific principles, or established professional consensus.

Q2.2 Statistics, data and metadata should be compiled using recognised standards, classifications and definitions. They should be harmonised to be consistent and coherent with related statistics and data where possible. Users should be provided with reasons for deviations from these standards and explanations of any related implications for use.

Q2.3 Statistics producers should be transparent about methods used, giving the reasons for their selection. The level of detail of the explanation should be proportionate to the complexity of the methods chosen and reflect the needs of different types of users and uses.

Q2.4 Relevant limitations arising from the methods and their application, including bias and uncertainty, should be identified and explained to users. An indication of their likely scale and the steps taken to reduce their impact on the statistics should be included in the explanation.

Q2.5 Producers of statistics and data should provide users with advance notice about changes to methods, explaining why the changes are being made. A consistent time series should be produced, with back series provided where possible. Users should be made aware of the nature and extent of the change.

Q2.6 Statistics producers should collaborate with topic and methods experts and producers of related statistics and data wherever possible.


Case studies

The UK House Price Index (UK HPI) has been published since June 2016 and is produced by HM Land Registry in partnership with the Office for National Statistics (ONS), Registers of Scotland and Land and Property Services Northern Ireland (referred to as HM Land Registry and partners).

The method used to produce the UK HPI was originally published in Development of a single Official House Price Index which set out the rationale for the approach, the data sources used and how it complied with international standards. It also considered users’ questions raised during an earlier methods consultation and from a peer review conducted by the Government Statistical Service Methodology Advisory Committee.

Each month, the UK HPI presents a first estimate of average house prices in the UK based on the available sales transactions data for the latest reference period. The first estimate then updated in subsequent months as more sales transaction data become available for inclusion in the calculation.

In March 2017, there was a large increase in the magnitude of revisions between first and subsequent estimates of annual change to average house prices. This negatively affected some users’ confidence in UK HPI as they were unable to understand or explain house price trends using the first estimate with certainty. After investigating, ONS established that they were being driven by volatility in new build property prices, compounded by an operational backlog in HM Land Registry registering new build sales transactions.

HM Land Registry and partners took steps to improve the methods by changing the calculation for the first estimate to reduce its sensitivity to the impact of new build transactions. The approach was developed by GSS methodologists, and several options were tested before a final one was chosen.

HM Land Registry and partners communicated the method change to users prior to its implementation through the About the UK HPI section of the UK HPI release, a blog, and later produced an enhanced Quality and Methodology report which includes details of the impact of the changes and supporting analysis. Details about the HM Land Registry operational backlog have also been included in Section 4.4 of About the UK HPI, with a reference to HM Land Registry’s speed of service and its future plans, which present information about average completion times for new build registrations.

As a result, the scale of revisions to the first estimate of UK HPI annual change to average house prices has reduced, and is more stable over time. HM Land Registry and partners and UK HPI users are now more assured that delays in processing new build registrations are not adversely impacting on the robustness of the UK HPI first estimates.

HM Land Registry and partners also compare UK HPI with other non-official house prices indices to identify and explain any differences between the series, and publish their analyses in an annual article Comparing house price indices in the UK.

This example shows how HM Land Registry and partners have transparently developed UK HPI’s methods by collaborating with relevant experts during their development, informed users in advance about methods changes with clear reasons and explanations of their impact, and published supporting information that helpfully sets out the rationale behind their various decisions.

In 2018, in response to the manifesto published by the Jo Cox Commission on Loneliness, The Prime Minister called loneliness “one of the greatest public health challenges of our time”. As such, a consistent approach to measuring how loneliness affects people’s lives and who is more susceptible to it, was needed. The Prime Minister tasked the Office for National Statistics with developing the evidence base and to develop national indicators of loneliness, suitable for use on major studies, to inform future policy in England.

The harmonisation of the new loneliness indicators was important for enabling more surveys to measure loneliness in the same way, in order to build a better evidence base more quickly. This is needed to enable a better understanding about what factors are most associated with loneliness, what the effects of loneliness are for different people, and how it can be prevented or alleviated. As this is a devolved matter, ONS took this work forward for England, with scope for future work to harmonise across the Devolved Administrations.

In December 2018, following consultations with key stakeholders and experts, and extensive collaboration with the ONS Quality of Life team, the GSS Harmonisation Team published the Harmonised Principles for measuring loneliness. The principles can be used to measure loneliness using any survey or administrative data source, which ensures a consistent approach can be adopted across major studies to inform future policy in England.

After identifying the need for indicators across all ages, the GSS Harmonisation Team agreed upon two sets of indicator questions and one direct loneliness question. The first set of four indicator questions is recommended for use with adults while there is an alternatively worded set recommended for use with children. The questions were tested and then used on several established surveys using different survey modes, including paper self-completion (English Longitudinal Study of Aging), online self-completion (Community Life Survey, Good Childhood Index Survey), and telephone interview (Opinions Survey).

All four questions are also due to be adopted on the:

And the direct loneliness question is due to be included on the:

Given the important link between health and loneliness, there is also ongoing work with various agencies including Public Health England, NHS England and NHS Digital to include the loneliness measures in key surveys, such as the Health Survey for England. Work is also ongoing to continue harmonisation of the loneliness indicators across the GSS including consultation with the Devolved Administrations.

This example shows how the GSS Harmonisation Team have worked effectively with statistics producers across government and experts in loneliness measurement, to develop consistent methods for measuring loneliness in both adults and children. These measures can then be adopted in a comparable way across major studies to help inform effective government policy responses in this area of current public debate.

 


Guidance and resources

DescriptionLinkSource
A resource for official statistics producers to develop their knowledge and understanding of the broad range of methodological approaches used across the Government Statistical Service (GSS).GSS methodology webpageGSS
Guidelines that aid producers in the production of official statistics. They include information about measuring and reporting statistical quality, quality assurance, and reporting metadata.GSS Quality GuidelinesGSS
Guidance that sets out the UK Statistics Authority (UKSA) policy on experimental statistics. Experimental statistics are a subset of new developed or innovative official statistics undergoing evaluation. The guidance sets out when to use the experimental statistics label, when to introduce experimental statistics, and removal of the experimental statistics label.Guidance on Experimental StatisticsGSS
National Statistician’s guidance on interpreting and implementing the principles and practices of Code which relate to the quality and methodological basis of their official statistics. This document is scheduled for revision in 2018 to reflect version 2.0 of the Code.National Statistician’s Guidance: Quality, Methods, and Harmonisation (2009)GSS
A webpage with links to a series of guidance documents on harmonisation, including what harmonisation is and its aims, the Harmonisation Handbook and the GSS Harmonised Principles.Harmonisation within the GSS webpageGSS
A one-page guide for authors and reviewers on how to conduct an effective peer review of statistics. It contains information on why a peer review is helpful, what the process can look like, and questions to ask.Hints and Tips: Conducting a Peer ReviewGSS
Information on a range of statistical classifications and standards, including the UK Standard Industrial Classification of Economic Activities, the Standard occupational classification, Economic statistics classifications, and other national and international classifications.Statistical classificationsONS
Guidance on collecting and classifying data on ethnic group, national identity, religion, and sexual identity, and an overview of ONS’s work on gender identity.Guidance on measuring equalityONS
Guidance that provides a common approach to aid the clear communication of uncertainty and change. It can be applied to all sources of information, including surveys, censuses, administrative data and other sources, as well as estimates derived from a combination of these. It includes examples of good practice, as well as standard wording to be used when appropriate. This document is scheduled for revision in 2018 to reflect version 2.0 of the Code.Communicating Uncertainty and Change: Guidance for Official Statistics ProducersGSS
Guidance on how to use a standard and straightforward way of assessing comparability of their statistics with other parts of the UK. It also allows users to better understand what is comparable so that they can quickly get to the numbers they want.Guidance on comparing official statistics across the UKGSS
Guidance for government analysts on when and how to use quota sampling. The target audience is government analysts involved with quota surveys – whether commissioning research, designing and running surveys, or interpreting the results to inform policy colleagues.Government Social Research (GSR) quota sampling guidanceGSR
The European Statistics Code of Practice, adopted by the European Statistical System (ESS), aims to ensure that statistics produced within the ESS are not only relevant, timely and accurate but also comply with principles of professional independence, impartiality and objectivity. The Code of Practice for Statistics is aligned with the ESS Code of Practice.European Statistics Code of Practice (2011 edition) Eurostat
Information on all main international statistical methods and classifications used by the United Nations (UN).UN international classificationsUN
A UN list of agreed international statistical principles and good practice tips that will enhance the functioning of the international statistical system. The Code of Practice for Statistics is aligned with these principles.UN principles governing international statistical activitiesUN
The United Nations Economic Commission for Europe’s (UNECE) Fundamental Principles of Official Statistics sets out the standards of official statistics that have been adopted at all levels of the UN. It recognises that reliable and objective information is crucial for decision making.UN Fundamental Principles of Official StatisticsUNECE

Please help us improve our website by answering a few quick questions.