Digital Appeasement

A session at the World Economic Forum’s 2023 summit in Switzerland posed the question “Are you ready for brain transparency?” In a brief animated video, a woman’s brainwaves are monitored via work-issued earbuds, maximizing her health and productivity. When she gets to the office the next day, however, the cheery panopticon takes a darker turn: A co-worker is arrested for wire fraud, and the government has subpoenaed the whole team’s brainwave records to search for accomplices. Our main character is innocent, but she was working with the accused on a secret side project, which means the feds may see a guilty conscience where none exists.

The clip ends, and the session’s presenter, Duke University law professor Nita A. Farahany, comes on stage. “What do you think?” she asks. “Is it a future you’re ready for?” The Davos audience answers with a uniform, unfaltering “no.”

“You may be surprised to learn that it’s a future that has already arrived,” Farahany replies. “After all, what you think, what you feel — it’s all just data, data that in large patterns can be decoded using artificial intelligence.” The basic technology for analyzing brainwaves already exists, she explains, soon ready to be wielded by corporations, health care systems, and the state.

What Farahany is describing is a kind of “meganet,” the titular concept of a provocative new book by David B. Auerbach, a former software engineer at Google and Microsoft. Meganets, Auerbach argues, are “invisible human–machine behemoths” that are “radically restructuring our lives” and even our “inner realities.” Some meganets are with us already — Facebook, the metaverse, massively multiplayer online games, cryptocurrency, and state-run citizen tracking systems, each of which receives extensive treatment in Meganets — but Auerbach contends that few people realize the extent of the change meganets herald:

These meganets are fundamentally new combinations of huge numbers of people and enormous amounts of computational processing power. They evolve faster than we can track them. Their workings are opaque even to their administrators. And they irreversibly occupy our lives with an ongoing persistence that makes them inextricable from the fabric of society.

In Auerbach’s timeline, we’re in the early meganet era, “immersed in a world administered by enormous computer networks fundamentally out of our control” but still generally in contact with — or, perhaps, naïvely clinging to — a pre-meganet world. The meganets we know now are just the beginning, and more tech won’t save us. We may think the problem is a “lack of will or ethics” on the part of meganet corporations, or that meganet operators simply sacrifice control for profit. But the problem runs much deeper, says Auerbach. It’s that every meganet will “invariably” become too complex for human supervision.

The totality and grimness of Auerbach’s prophecy is nearly unrelenting. He anticipates meganets degrading our thought and speech, herding us away from complex and nuanced human language and toward emojis that “help algorithms neatly classify us.” “Self-determination will be a distant myth,” he predicts, as we will, from birth, be sorted by meganets and “subjected constantly to the decisions of algorithms and AIs that, when questioned, cannot be explained or reversed.” Maybe most ominously, Auerbach often speaks of the meganet, singular, a coming chimera of what are now distinct and, therefore, less powerful networks.

Perhaps no one wants a single entity — such as a company, but more likely the state — to manage our social networks, “shopping habits, business profile, credit history, medical records, government benefits, and taxes,” but “the irresistible force of the meganet will gradually consolidate these spheres.” This is, as Auerbach says many times of many meganets throughout the book, “inevitable.” We can’t stop it, though perhaps we can apply some nice velvet linings to our shackles and take comfort in the fact that the meganet’s inherent chaos means they’re not as tight as they could be.

It’s easy to imagine Auerbach’s idea gaining widespread currency, not least in popular discourse. At a more serious level, however, Meganets fits comfortably in a larger critique of technological development and its reshaping of our lives that pictures it as a system that degrades human agency. “The will of these systems is greater than that of even the most powerful individual,” Auerbach writes. They “redefine the very concept of agency, absorbing individuals into larger homogenous groups in which the autonomy of a single person is, at best, a questionable quality.”

This kind of systems analysis is hardly without insight. The sense of mechanical constraint and gamified manipulation it evokes will be familiar to anyone who is too online, anyone who has felt the Internet “tinkering with [their] brain, remapping the neural circuitry, reprogramming the memory,” as Nicholas Carr famously wrote in the 2008 Atlantic essay that would become The Shallows.

It will be familiar, too, if you’ve noticed the fundamental inadequacy of so many proposals to “fix” social media and its deleterious effects on our politics. “The challenges we are facing are not merely the bad actors, whether they be foreign agents, big tech companies, or political extremists,” L. M. Sacasas has written in these pages. Rather,

we are in the middle of a deep transformation of our political culture, as digital technology is reshaping the human experience at both an individual and a social level. The Internet is not simply a tool with which we do politics well or badly; it has created a new environment that yields a different set of assumptions, principles, and habits from those that ordered American politics in the pre-digital age.

So too has it changed the set of options we normally consider available to us as we socialize, shop, work, worship, and more.

But the systems critique can go too far in absolving humans, as both individuals and institutions, of responsibility for the systems we build and to which we subject ourselves and one another. For all of its talk of an “inevitable” and “irresistible” descent into dystopia, Auerbach’s book itself shows cracks in the systems.

Auerbach’s first chapter begins with a trio of anecdotes intended to show that meganets are unprecedented in scale — that “until recently, the amount of knowledge was manageable” but now we have a “world too big to know.” But none of the accounts actually depicts a lack of agency or an incapacitating excess of data.

In one story, Iranian architects draw plans for a skyscraper that will serve as both a waterpark and a cryptocurrency mine — but they never build it. In another, Singapore’s government uses its Covid-19 tracing app for law enforcement after promising the public it wouldn’t be used that way. In the third, Estonia’s president announces blockchain-based tech that was later revealed to run on simpler software. These aren’t stories of a world too big to know or too complicated for meaningful human choice. They are examples of human fantasy, deception, and confusion, respectively.

More broadly, Auerbach doesn’t conclusively demonstrate the difference between meganets and other extremely large networks of diversified knowledge. Maybe there truly is a difference, but that he never clearly shows it is at least a possible fault line in his systems critique. Call it the “I, Pencil” problem — even seemingly simple things can turn out to be deeply complicated, like how many actors from how many parts of the world must be called on to create a lowly pencil. Was the knowledge of the world before the digital revolution really that manageable? Or have we long participated in incredibly complex, globe-spanning systems that can never have a mastermind? Is Facebook’s lack of control and knowability truly incomparable to, say, decentralized worldwide markets?

Auerbach might parry by contending that the speed of meganets’ development is unmatched in the marketplace: They “present constantly evolving visions of the world for us faster than we can test and correct these visions,” he writes. Meganets’ abilities are always a step ahead of our understanding.

But for this too there is precedent. It’s part of what in foreign policy is called an offense–defense asymmetry, a difficult but common quandary that in the long term is not necessarily insurmountable. New means of attack often emerge before the means to defend against them: the sword before the chain mail, the bullet before the Kevlar vest, the missile before the missile interception system. A novel technique or technology may lend invincibility when first introduced, but this isn’t always a permanent attribute. Auerbach might be proven correct that meganets will forever outpace human power to defend our data and our minds. Or it may be that, with time, we’ll hit upon some as-yet unconceived means to battle the meganets back.

Yet even if the rapid evolution of meganets proves to be a legitimately unprecedented complication, it’s generally not one we are forced to endure; it is a complication we choose. But Auerbach’s systems critique doesn’t admit that choice.

Consider his extensive discussion of Facebook. Along with Twitter and lesser social apps like Farmville, Auerbach casts Facebook as a classic contemporary meganet in which we can easily perceive meganets’ “essential” qualities of volume, velocity, and virality. Facebook is a very large group of people sharing a very large amount of information (volume), with speed so great as to overwhelm normal regulations of human behavior like fact-checking, ethical qualms, or even legal strictures (velocity), in a self-perpetuating process (virality).

The results are ugly, but a fix is apparently nowhere to be found. Facebook’s failure, in the face of pandemic misinformation on its network, “to address ongoing, stinging criticism from all sides on an issue that doesn’t greatly profit them in any way does not speak to malice and not even to incompetence,” Auerbach insists. “It speaks to an actual inability to solve the problems…. [Facebook CEO Mark] Zuckerberg’s creation had become as autonomous as nature — as the weather, as the tides, as plate tectonics.”

Has it? Of course, you and I can’t control Facebook. But is the Facebook meganet really as uncontrollable as an earthquake? Or is it rather that Facebook has made certain qualities of its meganet non-negotiable, then relabeled those choices as immutable constraints? For instance, we know Facebook has at times declined to deprioritize content that users deemed “bad for the world” because doing so reduced engagement, and engagement is what makes Facebook a profitable meganet. Profit clearly motivated the decision. Or did we forget that choices made for the sake of profit were, in fact, choices?

Auerbach gestures in this direction himself: For example, recounting Facebook’s decision to limit forwarding in its Messenger app in the run-up to the 2020 election, he describes the company as “constraining functionality artificially.” But he doesn’t seem to understand what this implies.

Facebook Messenger isn’t a given part of reality. It isn’t more real with unlimited forwarding, nor is it more artificial with a cap. The same is true of the entire Facebook platform. Put simply, there is no reason that Facebook must be a meganet. There’s no law of nature saying our Facebook feeds must be algorithmic instead of chronological; that Facebook must run its 2023 version instead of its 2007 version; that every tittle of user data must be stored forever instead of wiped on a regular basis.

These are all choices Facebook has made about its business model. They might be difficult to unwind, but they aren’t irreversible. They aren’t like the weather or the tides. Facebook-as-meganet may indeed be uncontrollable. But Facebook doesn’t have to be a meganet.

Facebook also doesn’t have to be part of our lives if we don’t want it to be.

But the same cannot be said of state-run ID meganets, like China’s Social Credit System or India’s Aadhaar system, which is a kind of hybrid, combining a Social Security number, a credit score, and a national ID. Programs like these cannot be opted out of by the public as a matter of law, practice, or both, and even within the governments that run them, there is no clear locale of human agency, no Zuckerberg who has final say over them, though Beijing certainly fosters the impression of that control.

As Auerbach writes, these systems are “invasive, prone to mistakes, and susceptible to data leakage.” They are “a permanent, immutable tie from a real person to the meganet’s digital mirage of that person,” an enormous risk for privacy and data security, resistant to public accountability, and will implacably travel from East to West. To fight the development of a unified digital identity administered and (functionally, if not explicitly) mandated by the state “is to put our collective head in the sand,” he says. “Big Brother can’t be stopped,” and our time is better spent making him a bit incompetent than “futilely trying to kill him.”

That may be an achievable plan — much of the chapter on state-run meganets, in an apparent attempt at reassurance, describes the practical limitations of these programs, which in China have meant that the government also continues surveilling and abusing its people the old-fashioned way. But in the West it’s awfully early in the fight to retreat as far as Auerbach does.

He mostly elides the distinction between state and private surveillance and devotes relatively little space to differences of political culture and infrastructure between China or India and America. He compares Reddit moderators to law enforcement without noting that the latter have a key additional tool at their disposal: the state’s monopoly on the legitimate use of force. He quotes a security analyst’s observation that you “can always deactivate Facebook. You cannot deactivate your Aadhaar.” But he doesn’t consider how significant that distinction would be for approval of an Aadhaar-style system by the public in the West, and especially in the United States.

Auerbach also recognizes that in China, “where paternalistic and invasive government has been long treated as the norm, there is far greater acceptance of state monitoring of everyday behavior” and enforcement of cultural values. Yet surely this “acceptance” hardly deserves the name. Beijing’s authoritarianism is a reality-warping system, and the Chinese people have no real say in what they must “accept” from their government. With a larger range of options in view in the freer world, we don’t have to accept the same thing. Auerbach is obviously correct that governments’ interest in meganets will only grow, but so far we still retain the agency to reject that totalitarian temptation.

“We fail to ask ourselves, ‘What are we doing with all this power?’” Auerbach warns, “because we forget that until recently, we did not have it.” This is the right question to raise, though, ironically, Auerbach maintains that it won’t matter whether we ask it or not. Still, with prophecies as ominous as Meganets, perhaps we’ll choose to constrain this burgeoning power while we still can.