Bicycle parking at one of Amsterdam’s main train stations

Popular cycling website Copenhagenize released the third instalment of its biannual ranking of the world’s 20 most bike-friendly cities last week. Amsterdam topped the 2011 and 2013 indexes, with Copenhagen second, but the big news in 2015 is the positions have reversed, with Copenhagen coming out on top.

The Copenhagenize Index uses more than the level of cycling in each city to derive its ranking. It scores each candidate on 13 equally weighted criteria: variables like mode split and the quality of cycling infrastructure are included, but so are other less instrumental ones like the strength of cycling advocacy, gender ratio, the political climate regarding cycling and the social acceptance of cycling. Cities with “particularly impressive efforts or results” may also score bonus points.

Rankings of cities on any and every imaginable dimension are fashionable because they attract readers, but I find they’re almost always problematic. Even those that are reasonably transparent, like The Economist’s world’s most liveable city ranking, have serious shortcomings (e.g. see Does the gong for “world most liveable city” mean anything?)

The Copenhagenize 2015 Index of bicycle-friendly cities has its share of problems, too. A key one is it lacks transparency. For example, we’re not even told each city’s total score, much less how they rate on each of the 13 criteria. Worse, there’s no indication how most of the indicators are measured, and there’s no information on sources.

What little we’re told about the methodology indicates it’s questionable, too. For example, each criterion appears to be given equal weight, so that whether or not a city has a (successful) bike-share program counts as much as the standard of cycling infrastructure or the share of trips made by bicycle. Moreover, some of the variables count the same thing.

This is a project that seems misconceived from the get-go. David Hembrow from the Dutch blog A view from the cycle path contends that “such lists are always false” because “there is no reliable way to make a ranking”. It isn’t possible to reliably make the sort of comparison Copenhagenize purports to make because the necessary data isn’t available on the same basis for all cities.

Estimates of cycling levels in cities or countries can and do differ for a range of reasons, such as the geographical area selected (e.g. central city v metropolitan area); types of trip selected (e.g. commuting v all purposes); the time of year selected (e.g. summer v winter); the method of counting selected (e.g. number of trips v kilometres of travel); the modes selected for mode split calculations (e.g. is walking excluded?); and so on and so on (David Hembrow translates an excellent Dutch article on this topic).

I’ve yet to see a really convincing comparison of cities that lines all these factors up on a like-for-like basis, but I’ve seen a few that are self-serving. There are a number of ways to present a city in the best possible light. For example, if the selected geography is small and covers only the denser central city rather than the metropolitan area; or if only commuting and education trips are counted and other trip purposes that are less attractive for cycling are omitted; or if only the summer period is counted and the winter data is overlooked; if walking is omitted from mode split estimates, etc.

Nor is it clear what the index is actually measuring. For example, I have no problem with the idea that the strength of cycling advocacy is an indication of something, but whatever it is, it’s not the same thing as what an outcome indicator like mode split measures, or even what intermediate indicators like subjective safety and quality of infrastructure measure.

It’s a bit like saying Australia performs wonderfully on refugee policy because we have strong advocacy groups and a politically engaged debate. But of course, our record is poor on this issue; it seems likely the line of causation runs in the other direction.

I suppose it could be argued that “bike-friendly” is such a broad term it covers a panoply of variables related to cycling; they’re all relevant. But I’m not convinced such disparate variables can be aggregated and reduced to a single summary statistic and still offer something sensible. If “bike-friendly” is so all-encompassing then I wonder if it’s measuring anything worthwhile.

I don’t have any problem with cities like Copenhagen and Amsterdam being at the top because there’s ample evidence from other sources that they do extraordinarily well on the most convincing measure of all: outcomes. Cycling captures a high proportion of trips in these cities because it’s used as an everyday means of transport. But I’m not going to put much stock by an index that doesn’t even tell me the current level of cycling in each city.