Error in GDP Numbers is Obscured by the ONS

It's really rather difficult to find information on the statistical errors in GDP figures released by the Office for National Statistics (ONS). Today's release for instance gives no direct information, but only refers the reader to another document: Quality and Methodology Information for Gross Domestic Product. It says:

Validation and quality assurance
Accuracy

The degree of closeness between an estimate and the true value.

There is no simple way of measuring the accuracy of GDP, that is, the extent to which the estimate measures the underlying ‘true’ value of GDP in the UK for a particular period. Blue Book 1 2008 (pp 27-30) provides more information on this.

One dimension of measuring accuracy is reliability, which is measured using evidence from analyses of revisions to assess the closeness of early estimates to subsequently estimated values. The results of revisions analysis are regularly presented in the background notes of GDP Statistical Bulletins and revisions spreadsheets containing the data behind this.

The QMI:GDP document doesn't discuss error generally but does give figures for the revisions. They say that the total revisions are not statistically different from zero for 2011. This might be difficult to accept for those of us used to non-zero revisions, such as the revision from -0.7% to -0.5% for Q2 2012; which is very far from being zero! Indeed in recent years the revisions between the first and the current estimates have been very much non-zero and increasingly volatile. The graph below shows the difference in percentage points per quarter from Q1 1997 to Q2 2012 (not including the most recent revision).

However this document does lead to a third document Accuracy Assessment of National Accounts Statistics (2002). The author of this paper doesn't really come out and say what the margin of error is, but does discuss sources of error and gives some indication of the magnitude of errors. This is partly due to the complexity of the calculation. But let's recall that when we add to uncertain figures the margin of errors are added together as well.

GDP is calculated in different ways and, as best as I can tell from these obscure documents this leads to an error of between 1.5% and 3.5% depending on the method. However this is far from being clear, and the way the figures are stated seems designed to hedge and fudge.

Contrarily we have the document Understanding the quality of early estimates of Gross Domestic Product. This document estimates the change in the estimates of GDP to average around 0.05 percentage points. But of course the data for this claim are in a separate document! The graphs are presented so as to obscure any differences between first estimates and subsequent estimates - which seems to be what this one is about, rather than error margins generally. The conclusion here is that the

"since the mid-1990s, revisions have been smaller than in previous periods. Over maturities up to T+24 [months], when most of the non-methodological changes will have been taken on board, the average revision is only +0.05 percentage points."

However in the graph I show above the average may be 0.05 percentage points but the standard deviation must be large because changes of 1 percentage per quarter (which could be 4 points over a year) are not uncommon.

As far as I can work out--though I have hardly exhausted all of the many OND documents available--the ONS do not supply error margins with their GDP figures.They do supply information on the differences between first estimates and later estimates.


Why Is It So Hard Get a Straight Answer About the Margin of Error?

Flying in the face of all good practice when dealing with statistics, the figures produced are treated as absolute. Not only the media (who probably can't be expected to know better), but the ONS themselves skate over the issue of statistical errors. It is reprehensible of ONS not to indicate the level of confidence they have in these figures at every point. No figure should be quoted without an indication at least of the calculated margin of error.

What this pattern of interlocking documents reminds me of is ISO9000 Quality Control documentation of a process. This in no way defines the quality of the product, but only provides for the process to happen the same each time. That is it guarantees that reports will be produced with figures in them, and the figures will be produced by the same method, but in fact says nothing at all about the quality of those figures.

The GDP guestimate we get from the ONS have a built in margin of error. It's unlikely to be small since it involves compounding errors from a series of other statistical measures. A lot rides on small changes in GDP, but the irony is that the smaller the change the less confidence we can have that it isn't just a statistical blip.