I seem to spend a large amount of my time in this job explaining in detail why a particular report produced by a particular organisation is particularly problematic.

But it’s increasingly a Canute-like effort. The tide of stupid is rising too fast. Every day, someone somewhere produces a report not with the intent of adding to the total of human knowledge about the objective state of the world (pffft, whatever “objective” is) but in order to influence policy in someone’s interests rather than the public interest, or to publicise themselves. And an ever-more-harried and under-resourced mainstream media lacks the resources to properly analyse these reports: there are only so many Ross Gittinses and Peter Martins to go around, ready to call bullshit on the stuff being served up by special interests and self-promoters.

So I thought I’d put together a brief guide to how you, too, can take up the fight … erm, OK, the bucket, against the tide of stupid. Not that you need my advice, as mere possession of a basic scepticism is probably sufficient, but nonetheless, this is what I do when confronted with a new report freshly dropped on a newspaper journalist and dutifully written up as presented in the executive summary or accompanying press release.

1. Who wrote the report, for whom, and why? Consultants do not write reports for free, indeed they charge a great deal for them, particularly if they contain economic modelling. The bulk of reports are commissioned by interested parties — industry bodies and companies, usually — with the goal of influencing debate in their own interests. And consultants provide exactly what is expected of them: independent modelling is never independent. The other type of report is the self-promotional one, in which a consultancy aims to generate publicity for itself by releasing a report with a nice grab for the media, or to stimulate demand for its own products. A great example was the recent report on music piracy by consultancy firm Musicmetric, which cleverly generated headlines in several different countries each suggesting downloaders in each of those countries were the world’s worst, to maximise publicity.

2. What is their record? Have they predicted the end of the mining boom for a decade? (Did they predict Tony Abbott would turn the Liberals into a “reactionary rump”?) Has the industry or company commissioning the report issued several different reports over time saying entirely different things?

3. What is their methodology? Have they bothered to provide one? A report that doesn’t explain in detail its methodology, or which is simply a press release for a report that isn’t publicly available, can be binned immediately. Next, is it based on a survey or poll? If so, what were the questions? Did the questions make sense? Did they include, for example, defining being a victim of cybercrime as getting a PC virus or being subject to “online harassment“? What was the sample size? Have they used a survey of 68 business executives to judge an entire economy, as the World Economic Forum did in its competitiveness report? Was the survey response weighted to ensure it accurately reflected the relevant population? How did the authors take into account the way people’s political beliefs influence the way they view the world, such that Liberal voters currently think the Australian economy is a disaster area and Labor voters think it’s Eden?

4. If the methodology is based on modelling, what were the assumptions? What was the “business as usual” model? Did the alternate model accurately reflect what it is purported to represent? Or was it like the Access Economics carbon pricing report commissioned by the Baillieu government that conveniently forgot to include carbon price compensation? Does the report simply apply overseas modelling to Australia and assume we’re exactly the same? Or compare Australia to irrelevant and misleading overseas examples (AKA the Economist Intelligence Unit scam)?

5. Does the report use the three most regular tricks in the book? Time and again the same tricks pop up, faithful friends who never go away no matter how much they’re mocked and derided:

  • The fictitious jobs scam is to compare the industry employment outcomes from the modelled scenario with the business-as-usual scenario and declare that a particular policy will “cost thousands of jobs,” as though thousands of workers will be pinkslipped tomorrow, even when the particular industry will in fact grow.
  • The time travel scam is to compare the GDP outcomes from the modelled scenario with the GDP outcome of the business-as-usual scenario and claim that a policy will “shrink the economy by 5%” as though we’ll immediately enter a prolonged recession rather than have a minutely slower growth path over decades. This was a trick that first came to prominence as ABARE’s contribution to the Howard government’s war against climate science under Brian Fisher.
  • And the third is the reverse magic pudding: to assume that money lost from one industry entirely vanishes from the economy, rather than being spent in another industry, supporting jobs and economic growth there rather than in the industry funding the report.

6. Has Major Events Mathematics been employed? A special branch of applied mathematics, Major Event Maths is studied closely in consultancies working for governments, and sporting bodies, and deals with the remarkable properties of Major Event Numbers, which do not comply with ordinary laws of maths but instead have an additional dimension of multiplication and a capacity to erase negative symbols. Thus, for example, a grand prix that costs Victoria $50 million somehow is estimated to generate $30 million in benefits. Or FIFA announces that 29 billion people have watched a soccer tournament on a planet with only 7 billion people.

7. Do the results actually make sense? Or are they so overstated that they can’t possibly be correct? Like the losses from filesharing that, extended pro rata, would be greater than the entire content industry revenue, or how smokers on average spend an hour every day outside workplaces smoking? Or the way cybercrime cost Australians more than burglary and assaults combined?

With this handy cut-out-and-keep guide, hours of fun are yours to be had playing along at home in the fun game of Report Whacking.