As a minister who claims to be evidence-driven, Jenny Macklin’s use of data is extremely suss. Her latest effort in justifying policy is her gross over-promoting of the results of a very small and dicey survey of 76 income-managed residents in four communities in the Northern Territory. Her sin is making unsubstantiated claims for the “findings”, which run counter to those reported by the Australian Institute of Health and Welfare.

The results? The Australian ran the following article, which was then picked up by other media, including the ABC, fairly mindlessly. Macklin’s media release came out much later, as did access to the AIHW report. She obviously chose to brief The Australian as the medium most likely to share her paternalistic views. The article says it all her way:

The Rudd government has received new evidence that the quarantining of welfare payments in indigenous communities has significantly improved health, with children eating more and gaining weight as their parents’ alcohol consumption fell.

An Australian Institute of Health and Welfare report into welfare quarantining shows three-quarters of the 16,000 people whose income is being managed by authorities were spending more money on food, with half buying more fruit and vegetables.

Indigenous Affairs Minister Jenny Macklin said last night the results reinforced her decision for a national rollout of income management into disadvantaged communities from 2012.

She told The Australian the findings showed income management, introduced under the Howard government’s emergency intervention into indigenous communities, was helping to improve living conditions for children and families in indigenous communities. She said the report showed that income management could produce benefits across the nation.

The AIHW report, which was a commissioned study, worked hard with inadequate data to fulfil the brief of a contracted research project. The report is a very good illustration of the conflicts that ethical research organisations go through when they know what the government wants but cannot deliver it.

If I was still teaching research methods to social science students, I would set it as compulsory reading. The following quotes express their serious doubts about the quality of the data, which incidentally they did not collect.

In the summary their report states:

The research studies used in the income management evaluation (point-in-time descriptive surveys and qualitative research) would all sit towards the bottom of an evidence hierarchy. A major problem for the evaluation was the lack of a comparison group, or baseline data, to measure what would have happened in the absence of income management.

Later in the same section:

The approach taken by the AIHW in writing the evaluation report was to triangulate the  findings of a number of different studies by looking for common issues and themes, and to draw these together around the key evaluation questions.

While this approach resulted in evidence that had more strength and validity than the results of a single study, the overall evidence about the effectiveness of income management in isolation from other NTER measures was difficult to assess..

Later again more doubts are stated:

The evaluation findings would have greater strength if these views were supplemented by empirical indicators that showed evidence of the changes reported by the various stakeholders. In addition, there were some data quality issues with the research conducted for the evaluation. The 2009 Client interviews, for example, included only a relatively small number of clients (76) from 4 locations, who were not randomly selected for interview. The stakeholder focus group report did not attribute many of the findings to particular stakeholders. It was therefore often difficult to identify whose views were reported, or whether they applied to the majority of stakeholders in the focus groups.

Again:

A major challenge for the evaluation was to separate the impact of income management from these other measures introduced as part of the NTER. In some cases it was difficult to attribute the outcomes achieved to one particular measure, especially for the higher-level outcomes which were common to a number of NTER initiatives.

In order to measure changes in spending patterns, quantitative data on expenditure before and after the implementation of income management would be required. There were, however, no quantitative baseline data on expenditure patterns. The evaluation, therefore, had to rely primarily on the perceptions of stakeholders about whether they had changed. A key source for this information was the 2009 Client interviews, which included data on expenditure after income management, but these data were based on a small sample of clients who were not randomly selected.

These quotes indicate the serious doubts of a reputable research institute.

Despite the quite strong doubts in the report,  Macklin declares that there is “proof” that income management works. She bases this mainly on very wobbly data, mostly collected in a very long survey (an hour) from people who would have wanted to give the “right” answers, as is shown by her use of statistics in her media release. Her basis is this small survey that cannot be extrapolated to 73 communities let alone the whole NT, plus some other dubious data

There is no adequate evidence that this type of compulsory program works anywhere. The AIHW report includes a brief summary of a literature search on the topic, which found such mixed reports that it could not come to any conclusions. Outside factors make such evaluations very difficult as they have done here.  Certainly for some communities and groups there is evidence they like the process and it works for them but the question is why not make it voluntary? Misusing data to support compulsion is not acceptable.

Macklin told The Australian the findings showed income management, introduced under the Howard government’s emergency intervention into indigenous communities, was helping to improve living conditions for children and families in indigenous communities. She said the report showed that income management could produce benefits across the nation. So watch out for the next episode of how to distort evidence to support political preferences.