Error in GDP Numbers is Obscured by the ONS

It's really rather difficult to find information on the statistical errors in GDP figures released by the Office for National Statistics (ONS). Today's release for instance gives no direct information, but only refers the reader to another document: Quality and Methodology Information for Gross Domestic Product. It says:

Validation and quality assurance
Accuracy

The degree of closeness between an estimate and the true value.

There is no simple way of measuring the accuracy of GDP, that is, the extent to which the estimate measures the underlying ‘true’ value of GDP in the UK for a particular period. Blue Book 1 2008 (pp 27-30) provides more information on this.

One dimension of measuring accuracy is reliability, which is measured using evidence from analyses of revisions to assess the closeness of early estimates to subsequently estimated values. The results of revisions analysis are regularly presented in the background notes of GDP Statistical Bulletins and revisions spreadsheets containing the data behind this.

The QMI:GDP document doesn't discuss error generally but does give figures for the revisions. They say that the total revisions are not statistically different from zero for 2011. This might be difficult to accept for those of us used to non-zero revisions, such as the revision from -0.7% to -0.5% for Q2 2012; which is very far from being zero! Indeed in recent years the revisions between the first and the current estimates have been very much non-zero and increasingly volatile. The graph below shows the difference in percentage points per quarter from Q1 1997 to Q2 2012 (not including the most recent revision).

However this document does lead to a third document Accuracy Assessment of National Accounts Statistics (2002). The author of this paper doesn't really come out and say what the margin of error is, but does discuss sources of error and gives some indication of the magnitude of errors. This is partly due to the complexity of the calculation. But let's recall that when we add to uncertain figures the margin of errors are added together as well.

GDP is calculated in different ways and, as best as I can tell from these obscure documents this leads to an error of between 1.5% and 3.5% depending on the method. However this is far from being clear, and the way the figures are stated seems designed to hedge and fudge.

Contrarily we have the document Understanding the quality of early estimates of Gross Domestic Product. This document estimates the change in the estimates of GDP to average around 0.05 percentage points. But of course the data for this claim are in a separate document! The graphs are presented so as to obscure any differences between first estimates and subsequent estimates - which seems to be what this one is about, rather than error margins generally. The conclusion here is that the

"since the mid-1990s, revisions have been smaller than in previous periods. Over maturities up to T+24 [months], when most of the non-methodological changes will have been taken on board, the average revision is only +0.05 percentage points."

However in the graph I show above the average may be 0.05 percentage points but the standard deviation must be large because changes of 1 percentage per quarter (which could be 4 points over a year) are not uncommon.

As far as I can work out--though I have hardly exhausted all of the many OND documents available--the ONS do not supply error margins with their GDP figures.They do supply information on the differences between first estimates and later estimates.


Why Is It So Hard Get a Straight Answer About the Margin of Error?

Flying in the face of all good practice when dealing with statistics, the figures produced are treated as absolute. Not only the media (who probably can't be expected to know better), but the ONS themselves skate over the issue of statistical errors. It is reprehensible of ONS not to indicate the level of confidence they have in these figures at every point. No figure should be quoted without an indication at least of the calculated margin of error.

What this pattern of interlocking documents reminds me of is ISO9000 Quality Control documentation of a process. This in no way defines the quality of the product, but only provides for the process to happen the same each time. That is it guarantees that reports will be produced with figures in them, and the figures will be produced by the same method, but in fact says nothing at all about the quality of those figures.

The GDP guestimate we get from the ONS have a built in margin of error. It's unlikely to be small since it involves compounding errors from a series of other statistical measures. A lot rides on small changes in GDP, but the irony is that the smaller the change the less confidence we can have that it isn't just a statistical blip. 

Labour Market Statistics (Again)

In her recent reiteration of surprise at employment figures the BBC's  Stephanie Flanders expressed the general confusion on the small downward movement in unemployment reported in the ONS August Labour Market Stats. 

"The unemployment rate was 8.0 per cent of the economically active population, down 0.2 on the quarter. There were 2.56 million unemployed people, down 46,000 on the quarter." (ONS)

According to convention economic theory when GDP is shrinking employment should also shrink. Why is it not? Flanders proposes several possible reasons without much conviction that they explain the situation.

• GDP estimates might be wrong
• Rise in part-time work
• Rise in self-employment
• Pay not keeping pace with inflation (i.e. lower real wages)
• Retention of staff
• Olympics

"Finally we're left with the explanation favoured by Britain's finest economic detectives at the moment (not to mention senior policy makers): it's a mixture of all of these possible factors, plus, maybe "something else"."

Some members of the public, commenting on this story provide further insights:

"We're replacing full-time jobs with 2 or 3 part-time jobs or zero-hour contracts."

"The answer is simple. Having worked in the welfare to work sector as a manager for many years I eventually walked out in disgust at the manipulation of the unemployment figures by pushing the unemployed through training courses, work experience programs and by backdating and projecting the period of training. The real unemployment figures are much higher than stated, it's all smoke and mirrors."

"They move you from one set of statistics to another: I was on JSA, got moved to an outside Agency (responsible for me for the next two years) and was persuaded to become self employed. Instead of income support I now get tax credits."

My own doubts go the margin of error in these figures which are obtained by sampling and statistical manipulation. Such figures should always be stated with a margin of error, but they never are, not even by the ONS. I have written to ask about the margin of error on these figures in particular and will publish the result. All of these government statistics are estimates with built in error from sampling and statistical method, the magnitude of which we don't know. The statistics have an historical accuracy that is not stated. They're quoted to one decimal place but we have no idea if this reflects the precision or is simply rounded up or down. Most of these estimates are subsequently revised over the longer term. The message is that we can't really treat these figures as absolute in the way that commentators do.More caution is required.

I don't go in for conspiracy theories, but it does seem credible that Job Centre staff under considerable pressure from Whitehall are scrambling to find creative ways to meet their targets. This is what bureaucrats do when faced with impossible demands. Remember when hospitals cancelled all operations and made patients reschedule to achieve their waiting list targets? (Not every one remembers that the man who came up with waiting list targets, Alain Enthoven, is also credited with the 'body count' as an efficiency measure for the Vietnam war).

Many employers do see staff merely as a cost on the bottom line; a cost to be minimised. They look for ways around labour protection laws, ways not to pay overtime, sick and annual leave, etc. The zero hour contract is how employers get around laws intended to prevent the casualization of the workforce.

What's also interesting is that the fall is not even. Unemployment in the North is rising, not falling. This lends credence to the Olympics as a significant factor mainly being felt in London. It will be interesting to see what happens to the figures after Christmas. In the mean time business will be hiring in preparation for the Christmas rush (let's not forget that wholesalers and suppliers have to crank up their operations a few months ahead of the retail Christmas).

UPDATE: Just seen another view from the Prime think tank: UK economy: there is no puzzle – just more work on lower pay.

VXN5SWSZ94VA

Household Debt Stats

I just discovered the excellent Credit Action website which has useful statistics on levels of personal indebtedness. For instance they say that:

Outstanding personal debt stood at £1.459 trillion at the end of April 2012. Which is about equal to 2011 GDP

The average amount owed per UK adult (including mortgages) was £29,706 in April. This was around 122% of average earnings.

 As with all of the information that surrounds the indebtedness of the UK this is sobering stuff. It gives us some sense of the scope of the Modern Debt Jubilee which is required.