In the perpetual discussions about journalistic bias, the most pervasive prejudice of all, is often overlooked. Journalists are biased towards that interpretation of the facts that yields the best story.

They are also biased in that they will always tend to see any new revelation as significant. Particularly if their byline is over it.

At its best, this bias is what yields that admirable terrier instinct of the good investigative journalist — the one who chases rabbits down holes and ferrets out the unsavoury fact. For example, it is probably what kept The Guardian’s Nick Davies drilling away at the News of the World scandal, despite the attacks and denigration of his peers.

It is also, doubtless, what has kept The Australian’s Hedley Thomas disinterring the Julia Gillard AWU matters — and uncovering fresh information. How significant the new revelations are, and whether they deserved the treatment they were given, is open to debate. But fresh they are, and in my view worthy of reporting.

But at its worst, the bias towards the good story results in some or all of the following sins: a tendency for ordinary human flaws to be depicted as gross and criminal; a tendency for stuff ups to become conspiracies; a tendency for systems failures to be blamed on individuals, whose careers and lives are wrecked as a result.

There can be a tendency for the journalist to become captive to those sources who yield the most salacious version of events — and for the sources’ own motives and antecedents to be overlooked or glossed in the process.

Perhaps worst of all, there can be a tendency for a partial understanding of the facts (sometimes incomplete, it must be acknowledged, because of the reluctance of governments and businesses to be frank) to be misinterpreted, and then for the journalists concerned to stick to their misinterpretation like limpets, in the face of unfolding evidence that they might have misunderstood.

We have seen all these faults, including in some stories that have won awards, in recent months. Driven by their hunger for the good story, the media can be like a herd of wild cattle — easily stampeded, trampling good and bad in its path.

There is something almost reassuring about this bias towards the good story. It is among other things a guarantee that political bias will rarely be the main driver of the newsroom. A good story is a good story, no matter if it damages the team you favour.

But when people in the know see a journalist sticking to a misinterpretation, blaming an individual for a systems failure or overplaying  the facts, it undermines the news outlet’s credibility — perhaps not with the public at large but with the audience that is most vitally interested in and concerned with the subject. And this matters, now more than ever before.

I was reflecting on these matters last week during the New News conference at the Melbourne Writers Festival, organised by the Centre for Advanced Journalism (which I head) in partnership with the Melbourne Press Club and the festival. (See here for coverage provided by University of Melbourne and Monash University journalism students.)

Each year the conference tries to take the debate about journalism and its future to a general audience, and each year it seeks to provide an update on where the profession is going. Predictably, this year a hot topic was news organisations seeking to get people to pay for content online. What will people be prepared to pay for?

A few snapshots. Matthew Pinkney, the head of content at the AFL’s new website, who used to be at the Herald Sun, told the audience that there had been an internal debate about whether Andrew Bolt’s blog should be behind the paywall. The decision had been to keep the blog free (though the Bolt column requires a subscription to access), because otherwise its vast following, and therefore its value, would disappear.

Which raises the question: if News Limited can’t make money from selling subscriptions to a commentator as well read as Bolt, what can it make money from?

Pinkney’s view was that paywalls can’t work, or at least not for general news and views — a view that should be weighted with the fact he used to be the editor of the heraldsun.com.au, and in his new job poses a threat to the Herald Sun‘s paywall business model because the AFL is providing reporting on footy for free.

On the other hand, David Higgins, the News Limited innovation editor, suggested packages that might pay, including content sliced and diced from the archives to produce online niche publications. Indeed, the consensus view from the industry specialists at the conference is that paywalls will only work for really high quality, specialised content. Already there are any number of subscription newsletters online that serve highly useful, specialised content to audiences with particular interests.

I think this means the bias towards the interpretation of the facts that yields the best story will need to be watched.

This is because the niche audience — the very same public servants, government officials, football followers, lawyers, academics, tradespeople, film buffs, hobbyists, etc, who are most interested in the content, and therefore most likely to pay — will also have enough knowledge and a sufficiently nuanced view, as well as access to informal networks, private forums and gossip, to allow them to know when journalists have overplayed the facts or simply misunderstood. They will not pay for that information, because they won’t trust it.

For example: the spats at Slater and Gordon in the ’90s have been fairly common knowledge in the legal community in Melbourne for years. It is a fair bet that people in that gossip network saw the revelations of the past few weeks in a different light to the general reader.

Among this select group, the revelations and the leaks would have been weighed in a balance that would have included a knowledge of the individuals concerned, views about their reputation, credibility and motives, and so forth.

The niche audience — the one that is likely to pay because of its intense interest — is therefore a very different one from the mass audience that relies on the journalist for the whole of its understanding of the facts. Sensationalism and beat-ups won’t wash.

Content behind the paywall, I think, whether it is analysis or news, will have to be born of journalists who take the time to understand context and nuance. Balancing that with the hunger for the good story and the courage to tell the bald and unvarnished truth — the truth that the niche audience may not want to hear — will be an important and complicated factor.

*Declaration: Margaret Simons is the director of the Centre for Advanced Journalism at the University of Melbourne, which organised the New News conference in partnership with the Melbourne Press Club.