How should we properly integrate AI into our lives and societies? In China, the answer seems to have taken more than just a page from science fiction scripts. In fact, it bears an uncanny resemblance to a scenario portrayed in Westworld, a popular tv show. In the West, the answer may depend on whether the liberal narrative can withstand this technological revolution.
The rapid integration of AI into our lives has sparked conversations on where the future of humanity could be headed. In China, the answer seems to have taken more than just a page from science fiction scripts. A few months ago I watched the third season of Westworld, a popular science fiction tv show. The story revolves around an AI system called “Rehoboam” that is built to monitor and control all aspects of human society, from individual behavior to global events. Programmed by an evil, Machiavellian villain, Rehoboam makes decisions to optimize societal outcomes and bring about a perfect utopia. Gathering large amounts of data from an unaware populace, the AI intervenes in people’s lives by manipulating job opportunities, personal relationships and finances, aiming to align reality with its predictions for a more orderly, peaceful, happy world.
The similarities with China are striking. Thanks to advanced facial recognition systems powered by AI, the Chinese Communist Party (CCP) is rapidly ramping up social control in ways that are uncannily similar to something we might see in a dystopian story. Through something called the social credit system, citizens are rated on a number of different data points, like heroic acts (points earned) or traffic tickets (points deducted). Those with lower scores face concrete hurdles when applying for jobs, buying a home, accessing certain services or even purchasing certain types of goods. China’s social credit system, according to a CCP slogan present in official documents, was developed to engineer a problem-free society by “allowing the trustworthy to roam everywhere under heaven while making it hard for the discredited to take a single step.”
However, this might be just the beginning of China’s own path towards utopia. It’s worth noting that a system like Rehoboam – what experts in the field would call Artificial General Intelligence (AGI) – might actually exist in the not-so-distant future. Contrary to narrow AI, which is the current state of the art of the technology, AGI would possess general problem-solving ability, that is the capacity to make decisions and solve problems autonomously in any domain, mimicking human versatility. In a 2022 survey of 356 AI experts, 90% of respondents predicted that AGI will be developed within the next 100 years, and 50% believe it’s likely to happen before 2061.
Most people in the Western world have an unfavorable view of China. Besides its oppressive and authoritarian policies, the country has a powerful military, increasing geopolitical and economic influence, and clear ambitions of challenging American and Western hegemony over the next century. These are all fair concerns, but I think our fears are deeper than mere strategic headaches.
China holds up a mirror to our worldview, and forces us to question what we truly believe in.
Let’s go back to Westworld for a moment (mild spoilers ahead). The plot of the third season unfolds exactly as you would expect. A man named Caleb, deemed by the system as useless or outright dangerous, is relegated to the margins of society and condemned to a grim existence with no possibility of redemption. Every time he tries to climb up the societal ladder, the system pushes him back down. The story shows Caleb gradually realizing that he is being unfairly held back because of the machine’s perception of the Greater Good, and his rebellion against the system as he attempts to reclaim his destiny through his free will, which, it turns out, is stronger than any algorithm.
If you are having a hard time accepting Caleb’s story as anything more than a feel-good fairy tale, you are not alone. Westworld’s first two seasons were acclaimed by critics and public alike, while ratings for the third season went down and average viewership per episode was halved compared to season two’s numbers. Obviously, this downward trend could be attributed to many factors, but I would argue that the perceived naivety of the narrative played a major role in the growing disengagement of an audience accustomed to much more cynical themes in previous seasons. Watching the show, I was personally taken over by a sense of dread at the possibility of something like Rehoboam having full control over our lives in the near future, while at the same time refusing to believe Caleb’s success in his impossible uprising. I kept thinking: how could anyone withstand the unhinged power of a future AGI aimed at enforcing complete social control, when we can hardly resist social media algorithms from nudging us to stay on our phones?
At one point, I thought the story might be believable, but only at one condition. Maybe, I thought, the problem isn’t that Rehoboam is too tyrannical and powerful, but too little. Maybe, just maybe, if an even more powerful AI was built and given access to more data and more control over people’s lives, it might just work, and the perfect utopia might ensue. But once I caught myself entertaining these thoughts, I was horrified by their utterly totalitarian nature, and I reminded myself that utopian visions in history have always, inevitably led to catastrophic results: if what you are aiming at is a perfect world, then any means can be used to justify the glorious ends. Stuck between my disbelief in the narrative and the horror, I was left with confusion.
Of course, Westworld is merely a tv show. But narratives have the immense power of shaping our thoughts and our actions, our social and personal identities. In the Western world, most people share one essential narrative: the liberal story. At its most basic core, that story was derived from a merging of Judeo-Christian and Enlightenment values, and it claims that every human being is of equal worth and imbued with free will. The idea of a society based on merit, individual rights, personal responsibility and democratic institutions shaped by the will of the people is linked to that central claim. But to the degree to which we struggle to believe Caleb’s story, that central axiom is in jeopardy, and so is the liberal narrative.
Today, religious belief stands at historic lows, while a scientific worldview brings us to challenge the reality of free will, which, according to some, is a mere trick our brain is playing on us. Populist rhetoric on both sides of the political spectrum tells us that the meritocracy is not really based on merit (besides, can the concept of merit even exist if one rejects free will?), but on powerful institutions and élite groups that perpetuate themselves at the expense of everyone else. The number of people living under democracies is shrinking, and a rising mistrust in democratic institutions is evident even in developed countries. The end of history as predicted by Fukuyama keeps receding from our horizon as the most fundamental bases of our civilization come under attack from multiple fronts.
Meanwhile, Chinese citizens are learning to live with automated monitoring and the consequences of stepping out of line. Crime levels are down, the occasional sparks of social unrest are quickly and efficiently suppressed, and middle and upper classes preserve the status quo. They have access to luxury goods and expensive cars never imagined by their parents and grandparents, while promises are made to lift all Chinese people out of poverty. For now at least, it seems like the absence of privacy, free speech, religious and sexual freedoms, are reasonable trade-offs for earning a desirable social credit score. Except for minorities like the Uyghurs, who are systematically detained and forcibly sterilized in internment camps. But I guess those are the people that, like Caleb, have been deemed expendable by the system.
The narratives we choose to believe in have the power to shape our future. Caleb’s story is not particularly new or original: it’s about a common trope depicting the individual triumphing over powerful systems that constantly seek to oppress him. We used to believe those stories because they were deeply entrenched into our liberal worldview, but we are losing our conviction in them. And if we cannot believe our own narratives anymore, while simultaneously regarding opposing worldviews such as China’s with horror, the result is profound confusion.
Our horror at China does not come from a place of virtue or moral superiority. Instead, we are worried that the Chinese experiment might actually succeed, bringing the final blow to what is left of our liberal beliefs. Technological revolutions have challenged our worldviews throughout history. New, powerful instruments for control are already here, and something akin to Rehoboam might soon be a reality in China, bringing with it a totalitarian state the likes of which the world has never seen. If social control through AI might further decrease crime and increase compliance and efficiency, what is our argument against using it in the same way? If we don’t believe in free will anymore, what is our argument against curtailing it? When the time comes, will we fall prey to utopian temptations, or will we chart a different path?
Some might argue that these are all irrelevant philosophical musings, that nothing is truly changing in the world of realpolitik. But totalitarian states are not born in a vacuum. They are born within corrupt philosophies and narratives. And when societies start doubting their own foundational stories, they risk collapsing. China’s President Xi Jinping seems to understand this. In a text regarding military political work, he declared: “The crumbling of a regime always starts in the realm of ideas. […] Once the front lines of human thought have been broken through, other defensive lines also become hard to defend.”
We need to look at ourselves in the mirror.
I am a student of Politics and Policy Analysis. I am interested in politics, philosophy and art and I mostly write at the intersection between those subjects. Writing to me means thinking on the page, exploring questions I can’t figure out the answer to. Often, I stumble upon it along the way.