When cholera swept through London in 1854, people believed in the miasma theory, which held that bad air carried the disease. The physician John Snow turned his attention to the water supply. At the time, two companies supplied London's tap water: Vauxhall drew from the polluted lower reaches of the Thames, while Lambeth had moved its intake upstream. When Snow linked cholera case records with data on which company served each household, he found that infection rates had fallen in both areas, but the decline was far steeper in the Lambeth district. It was evidence that contaminated water carried cholera. Had he only the cholera records, he would have known merely that cases were falling; had he only the water company data, he would have known merely where the water came from. Only when the two data sets met did the cause of cholera emerge.
The same holds for government data today. Each agency's data is sound in itself, but because it is accumulated for a particular purpose, it shows only one side of people's lives. Suppose one person appears in employment data as an employed worker, in medical data as a patient with a chronic illness, and in credit data as a borrower in arrears. Scattered data alone will not reveal a household in crisis, because the person is working and has income. Only when the data is connected do the warning signs appear. Connecting data in this way makes it possible to find people who fall outside the reach of policy, and to examine how policy has changed people's lives.
How, then, should scattered data be connected? Think of an orchestra. Violins, cellos and wind instruments each play a complete melody of their own, but when they sit around a conductor and play from a single score containing every instrument's line, they produce a harmony no single instrument could create alone. The National Data Office's hub-and-spoke model works the same way. The office places basic statistical registers on population, households, housing and businesses at the hub, like the conductor's score, and connects administrative records held by individual agencies as spokes to create integrated data. Two leading examples, comprehensive pension statistics and the de facto resident population, have supported retirement income policy and measures for regions facing population decline. This year the office is building integrated data on older adults and deaths, so that policies can be designed to address social issues such as population aging and suicide.
Until now, however, the office has gathered and built the data itself. Other policy ministries and researchers seeking to use the data had to apply to each agency separately, and even once they obtained it, connecting the pieces was not easy. Data sat behind the walls of individual agencies, making it difficult to use multiple sources together.
The office aims to move from a government that holds data to a government through which data flows. A national data system open to all will be the answer. The system adds the integrated data built so far to the basic statistical registers, firmly establishing a central axis, and links data provided by the government-wide data centers through a dedicated network. Under this system, users will be able to request data in one place and use a range of sources together, without visiting each data center.
Just as the power of linked data became visible more than 170 years ago in the search for the cause of cholera, the office intends to focus on laying the groundwork for government-wide data to be connected and put to use. The value of connected data will extend beyond the public sphere to everyone. I hope the collective intelligence of the public, meeting this data, will produce new answers and help unravel the difficult problems our society faces.