As yet another round of headlines emerge about lack of progress on indigenous health and social indicators, the “Closing the Gap” initiative can look more than ever like wrestling with smoke, with vast resources being pitched into battle with a complex set of irremediable and ill-defined problems.

The Productivity Commission’s Key Indicators report, which PC head Gary Banks launched with Jenny Macklin yesterday, isn’t merely a catalogue of where we’ve gone backwards or achieved progress. The Commission, alive to the specific problems of trying to measure social, economic and health indicators in indigenous communities, has tried to go beyond mere numbers by including a descriptive “Things That Work” section for every indicator. It may look like an attempt to counter the bad impression left by deteriorating indicators in areas like child abuse, but the PC’s rationale – to reach beyond national and state indicators to identify effective initiatives at the community level – is sound.

However, the PC is clearly concerned about the poor quality of data it is receiving from Commonwealth and State agencies, which prevents it from determining what progress is being made, if any, in critical areas. Banks identified lack of data as a particular concern.

This isn’t entirely a failing on the part of Commonwealth and state data agencies and their Governments. Getting accurate data on social, economic, health and educational indicators for Aboriginal and Torres Strait Islander people is more difficult than for non-Indigenous Australians. Sampling surveys are less effective because the much smaller number of Indigenous people compared to non-Indigenous people leads to much larger error margins. And census-based data and data derived from usage of social services relies on self-identification by Indigenous people, when they may, with good reason, be reluctant to identify.

The ABS spent considerable time wrestling with these issues and trying to develop a plausible means of addressing under-reporting in relation to Indigenous mortality, eventually determining a new approach to Indigenous life expectancy that showed a narrower gap with non-indigenous people than previously thought.

Yesterday the PC outlined a range of indicators where further work was needed on the quality of data available. The indicators ranged from comparatively minor, though still important, areas like dental health, to better-quality data on child protection, data on drug use, criminal justice data, and quality of local services. In particular, the PC pointed out, there is very limited data on early childhood education (a COAG target area) and family and community violence.

There is also a lack of trend data in areas like disability and chronic disease. In those areas, we have a snapshot – usually two or three years old – but no idea of whether things are getting better or getting worse.

The data problems add to the perception of fighting an enemy that can’t even be properly seen and assessed, let alone materially affected by our efforts.

Fortunately COAG yesterday took a large step toward addressing the problem, agreeing on $46.5m to improve Commonwealth and State data collection. One bureaucrat close to the issue called it “putting their money where their mouths are”. The money will assist data collection agencies to establish better-quality data and deal with problems particular to Indigenous communities. Particularly important will be consultation with and support from the communities themselves, especially on the issue of identifying as Indigenous.

None of this will happen quickly. The next PC Indicators report, due in two years, will still have some of the same problems, and there will still be a lack of long-term trend data in critical areas. We still won’t have much idea whether we’re closing the gap for years to come.