Few people seem to be paying attention, but for me, the scariest tech story of 2015 wasn't any of the many giant hacks afflicting big retailers, shadow IT, the impending doom of many of our favorite tech companies, cockeyed drone delivery schemes, or even the use of social media by terrorists.
Nope, I'm most worried about China's planned Social Credit System, which the New Republic described as "fiercely ambitious, authoritarian, technologically sophisticated, and likely to disrupt the lives of millions of people." What makes China's SCS so insidiously terrifying is that it goes well beyond Western-style credit scores to create a mandatory scheme to "rate the trustworthiness of citizens in all facets of life, from business deals to social behavior."
According to the BBC, "By 2020, everyone in China will be enrolled in a vast national database that compiles fiscal and government information, including minor traffic violations, and distills it into a single number ranking each citizen."
Eventually, these scores will be used to determine eligibility for everything from employment and credit to various social benefits. In practice, private company tests of the system are already using similar scores on dating sites, and acknowledging that even legal activities like playing too many video games all day or buying socially unacceptable products could incur penalties.
The SCS is also intended to encourage informing on your neighbors. The system's planning document states that the "new system will reward those who report acts of breach of trust."
Building a sincerity culture?
I don't necessarily have a beef with the SCS's stated goals of "establishing the idea of a sincerity culture, and carrying forward sincerity and traditional virtues." But, as the New Republic noted, the SCS means "China's citizens will be in fierce competition with one another, jostling for rankings better than their peers."
And that's only the beginning of the problems I see. Even if the SCS is used only in the ways it's officially intended, having all that kind of information available on everyone represents a huge invasion of privacy. But it's naïve to think that such powerful information will always and only be used in appropriate and agreed-upon ways.
Information wants to be used
One of the key tenets of big data is that, once you collect the information and are able to analyze it, you'll quickly discover all sorts of new ways to use it. In this case, it doesn't take much imagining to see how SCS scores could drive all kinds of nefarious agendas. And that's true not just in China but pretty much everywhere.
After all, it's not like these same issues don't exist in the West. The LA Times noted that, "in some ways, Chinese authorities' desire to incentivize moral or healthy behavior through data mining may be no different … than American insurance companies giving discounts to customers who upload digital proof from their Fitbits that they exercise regularly," or from using credit scores to make hiring decisions.
But many of those Western practices are voluntary, and the West has a tradition of whistleblowers who call out the excesses of their government. It's easy to see how a government could use SCS scores to identify and punish people engaged in activities it doesn't like, either through individual targeting or simply by labeling dissent as a negative practice leading to lower scores.
Ultimate Orwellian instrument?
The LA Times also recognized that "some fear that marrying FICO-style credit scores with school, employment, criminal and other records will create the ultimate Orwellian instrument of social control in a one-party state, which in recent years has shown less and less tolerance for critical voices."
I think that history has shown that such fears are prudent, whether we're talking about China, the United States, or any other government or large organization.