Losing Ourselves

Horror stories of artificial intelligence overcoming us disturb our confidence that there is no problem that some combination of science, good intent, and a can-do attitude can’t solve. What if our well-intended ingenuity doubles back and devours us? So we say to ourselves that we need to pump the brakes, consider unintended consequences, and maybe even propose regulations.

But what if at the root of this is a deeper fear? We moderns seem precariously unsure of what it means to be human. We have shaken off the old hierarchy of living creatures — with humans at the top of the natural world and the bottom of the supernatural — and have replaced it with the notion that what truly sets us apart from the rest of nature is our superior intelligence, or having a mind at all. What is left of our dignity consists in this: we are thinking beings. But the prospect of truly human-like artificial intelligence, even if it is for now only a pipe dream, rattles that foundation. If our intelligence is all that defines us, who are we when AI matches it?

This is the question that Kazuo Ishiguro’s new novel, Klara and the Sun, helps us to explore. It’s not exactly a horror story. Or, if it is one, then it is not the AI that evokes the horror. On the contrary, Klara, an Artificial Friend, is the pleasant, even lovable, narrator. She is curious and loyal; she has experiences that resemble awe at beholding beauty; she practices something like a natural religion, worshipping a benevolent and powerful Sun. If Klara is the terminus of AI development, then we have no uprising to fear. And yet we can’t shake the feeling of deep unease in the novel’s atmosphere. That is because the threat AI poses to human dignity is far creepier than that of a mere robot revolt.

Sold at what seem like toy stores, or perhaps at their own boutiques, Artificial Friends are companions for lonely children. Klara’s first home is one of these stores, where she lives with other AFs and a character named Manager, who prepares them for selection by a child and an eventual life in that child’s service. The AFs rotate from display to display, with the most coveted location being the front window, where they are most likely to be selected by a young passerby and where they receive day-long nourishment from the Sun (AFs are solar-powered).

For Klara, who is especially observant, the front window offers the added benefit of an unobstructed view of the street and of the flow of urban life. She is exposed to humans displaying the widest array of emotions, including those bizarre ones like melancholy that seem to be a mixture of opposites. If she does not quite understand these, she at least suspects — and she must be right — that a human emotional life is so complex and contradictory as to be irrational.

While stationed in the front window, Klara meets a young girl named Josie, who eventually brings Klara home to be her AF. Josie, we learn early on, is sick, and from Klara’s life with Josie and her mother, Chrissie, we begin to piece together a world in which children have Artificial Friends not as toys but as desperately needed companions. Ishiguro never sets the scene for us; we must gather all we can from Klara’s observations, inflected as they are by her programming and relative inexperience, and from her interactions with Josie’s family and friends.

Josie’s world is one in which automation has brought about widespread unemployment and dislocation. Josie’s father — her parents are divorced — was once an expert engineer before losing his job, and now lives in some kind of anarchist community with other expendable people. To keep a step ahead of their mechanized competitors, some parents choose to subject their children to a genetic enhancement procedure known as “lifting.” (Note that the same explicit motivation is behind Elon Musk’s venture for brain–machine interfaces, Neuralink.) Some parents, however, decide against getting their children lifted, either because of its exorbitant cost or because it can lead to prolonged, often fatal illnesses, as it did with Josie — and with her now-deceased sister, Sal. How widespread these side effects are is unclear, but parents with the means to pay for the procedure assume the risk, or rather foist it on their children, in the belief that it is a child’s only path to a decent life. Even the most generous colleges reserve only a pitiful two percent of their spots for the unlifted. Joining wealth inequality is a new, heritable, genetic inequality.

This harrowing reality is so unsettling because the characters by and large are good, or at worst are normal, flawed people trying their best. Josie’s mother is not callous, unloving, or maniacal for choosing one horn of a tragic dilemma: lift Josie and risk her dying, or ruin her chances of any semblance of a good life. Josie herself — ambitious, adventurous, and ebullient — acknowledges she would have wanted to assume the risk of the lifting. She “wouldn’t wish it any other way.” Both mother and daughter love each other, and aside from the predictable strains of single motherhood and adolescence, their relationship is one of mutual tenderness and care. So, too, with the other characters’ relationships: Josie’s with her unlifted neighbor Rick; Rick’s with his own mother, who chose to leave him unlifted; Josie’s with her father. Presumably, there are plenty of bad people out there, but none of the characters we observe up close are malicious. Rather, they are thrown helplessly into a tragic arrangement.

Josie and Rick live in the countryside and have no other friends. The only time we encounter other children is at a function at Josie’s house for lifted kids to build socialization skills, which they badly need. Otherwise, Josie, her family, and her friends suffer through her illness and their own isolation.

If there’s an affront to human dignity here, it does not arise from an AI menace. Klara, as we will see, raises questions for us about human dignity, but she, and AFs generally, are not what threaten it. The real threat is a ruthless technological logic, the logic of the factory foreman, instituted by humans, which governs all aspects of life save the intimate relations of the main characters. It’s the logic of human capital — that is, of humans as capital — where health, education, friendship, all that we expect to contribute to a full, flourishing life, become inputs for maximizing productivity during the years of prime employment. “Lifting” is like a software upgrade, and a technological society makes it the rational choice. To choose otherwise, as Rick’s mother does, is to defy the reigning social logic — which is to say, to consign one’s child to failure, to penury, to dwelling on the fringes of society.

We don’t know how far in the future the story takes place, but we can easily see that the social logic is not far removed from our own. In a lot of ways, Josie’s world is simply ours further extended, with a little help from technological “progress.” Why imagine that AI will run afoul of human dignity when, reducing the human person to a machine, we are already perfectly good at that ourselves?

But Klara and the Sun is a defense of human dignity, not merely a depiction of how it can be offended. So what exactly needs defending?

From the day she arrives at Josie’s home, Klara’s mandate is to observe: to study Josie’s habits, mannerisms, and decisions, presumably, we think at first, to become a better caretaker. We grow to learn that Klara’s role will be much greater and much darker. Josie is sick, in all likelihood terminally. Her mother, Chrissie, terrified that Josie might die, takes out Klara as an insurance policy. Should Josie die, Klara will assume an artificial copy of Josie’s body and continue Josie’s life. Indeed, Chrissie tried something similar with Josie’s sister Sal, but the replacement ended in failure. Chrissie couldn’t love the robotic Sal as the same girl. Mr. Capaldi, the man tasked with creating Klara to become the new Josie, chalks this blip up to the earlier primitive technology, which several years of research and development since should take care of. What’s more, he says to Chrissie:

A part of us refuses to let go. The part that wants to keep believing there’s something unreachable inside each of us. Something that’s unique and won’t transfer. But there’s nothing like that, we know that now. You know that…. There’s nothing there. Nothing inside Josie that’s beyond the Klaras of this world to continue. The second Josie won’t be a copy. She’ll be the exact same and you’ll have every right to love her just as you love Josie now. It’s not faith you need. Only rationality.

The science is settled, the technology advanced. Those who hold to the old views are backward, superstitious. The rest of the novel tells of the characters’ struggle to grapple with this. We are obliged to do the same.

Capaldi’s view falls somewhere between unnerving and horrifying, although major strains of both neuroscience and the philosophy of mind have been making the same underlying claim for decades: We are nothing more than the sum total of material processes, so there’s nothing in us that cannot, in principle, be replaced. Transhumanists dream of “uploading” themselves to computer hardware. Once again, the seeds of the story set in the future are already in our present.

Josie’s parents offer two ways of responding. Her father thinks it might be true, and this thought terrifies him:

Deep down I suspect [Capaldi] may be right. That what he claims is true. That science has now proved beyond doubt there’s nothing so unique about my daughter, nothing there our modern tools can’t excavate, copy, transfer. That people have been living with one another all this time, centuries, loving and hating each other, and all on a mistaken premise.

That Klara might seamlessly become Josie is an unsettling thought to bear, because it disarms her father of the means to grieve. Josie would be endlessly replaceable. So why his deep need to grieve her loss, if there is no real loss? Grief would seem irrational. But then his memories of Josie would be called into question, too. Are they not memories of a person irretrievably lost, and thus of a unique love? This line of thinking would seem intolerable. Why would we still care about others the way we do, and about those poignant aspects of our lives that are attached to them — love, joy, grief, even nostalgia? We could try to jettison these, but we would no longer be living a recognizably human life. If what Mr. Capaldi says is true, how can anyone live with this knowledge?

Josie’s mother shows another way. She can never believe that Josie is replaceable. Her solicitation of Mr. Capaldi — “it is me asking” — is an act either of incredible self-deception or of willful repression arising solely out of fear of losing Josie, the real Josie: “There’s going to be no other way for me to survive.” She knows she will never accept a facsimile of her daughter, despite Mr. Capaldi’s assertions. And perhaps because of this, she is able to continue loving her daughter — Capaldi’s claim does not eat away at their relationship. But if he is right, then she is lying to herself.

Are these our only options — soul-crushing truth or rushing headlong out of reality? Maybe not, but this isn’t because of any explicit falsification of Capaldi’s thesis. We may never get that, and as neuroscience grows more sophisticated and predictive, the thesis may grow more plausible and its defenders more evangelical. The question is how to defend human dignity from it.

This is not the first time Ishiguro has broached this question. His 1989 novel The Remains of the Day follows an English butler who struggles — and fails — to come to terms with the twilight of the aristocratic dignity in dutifully serving one’s appointed role. In its stead is the democratic, universal notion of dignity, as the inalienable endowment of each person both in spite of and because of individual differences. In Klara and the Sun, it may be dusk for this kind of dignity; something else may supplant it. Or if Capaldi is right, perhaps nothing will.

This is where Klara is crucial. As admirable and genuinely endearing as she may be, she is not human. Yet according to Capaldi, she and Josie are functionally the same. It’s up to us to parse their differences.

The first thing to note is that Klara — whether she alone or her entire species of AI — has a kind of dignity. When she’s mistreated — referred to as a “vacuum cleaner” or asked if she can be thrown across the room like a ragdoll — we recoil more strongly, it seems, than if she were a dog (though we’d recoil at this too). At the novel’s end, after Josie leaves for college, Klara is in some sort of junkyard, all alone, living out the end of her lifespan watching the sky. This is a heartrending end for a character we have grown close to, and it is because she doesn’t deserve it, regardless of whether her intelligence is artificial or natural. We can’t avoid the conclusion that Klara has some kind of dignity. But what does it consist of?

We might say that it’s derivative of human dignity, that Klara’s dignity is that of a creation bearing the imprint of its creator, the imago of her human designers. But that just kicks the can down the road. What is the distinction between Klara and her human creators? Is she just a diminished form of them, with a lower number or intensity of human-like qualities? No, Josie has something that demands stronger respect, something Klara does not have.

Klara herself has an explanation for why she and Josie are not interchangeable. “There was something very special, but it wasn’t inside Josie. It was inside those who loved her.” The human capacity freely to give love is what distinguishes us from the non-human and in which our dignity consists. Perhaps Klara loves Josie, but it is not freely given. She was programmed to commit to Josie, and even her actions to save Josie’s life, admirable though they appear, are all in the service of an unchosen relationship. The novel’s human relationships, on the other hand, are either chosen or in need of constant recommitment. Josie and her neighbor Rick have loved each other from a very young age. As children they vow to spend the rest of their lives together, and even as it becomes apparent that their promise is unkeepable — Rick is unlifted and their paths must diverge — they continually renew their love. “Josie and I grew up together and we’re part of each other,” Rick says.

Early on, Klara is befuddled by Rick’s mother, who is glad to let Rick leave the home to pursue a future, even though this will leave her alone. Klara wonders, “I didn’t think that humans could choose loneliness.” It’s easy to understand why she wonders: Klara’s purpose is to assuage human loneliness, which must be an immense evil to be avoided if it is worth creating a race of Artificial Friends to eliminate. But she comes to realize that humans are willing to endure loneliness for the sake of “forces more powerful.” After Josie’s recovery and before she leaves for college, she and Rick have drifted apart for the first time in their lives. Despite the injustice of it all, Rick knows it’s best for Josie and, out of love for her, is content: “Now we’re no longer kids, we have to wish each other the best and go our different ways.” When he earlier said their love was “genuine and forever,” he was undoubtedly right, though perhaps not in the way he meant it then.

So what distinguishes the humans from Klara, from even the most benign artificial intelligence? The philosopher Charles Taylor has written about the difference in terms of  “mattering.” Against those who would reduce human intelligence to the operations of an especially sophisticated computer, he remarks that things — objects, other people, moral ideals — matter to people. The world has meaning and importance for us in a way that it doesn’t for computers. Ishiguro wants to go further, imagining that the world does matter to Klara. Her worship of the Sun as a sort of deity makes no sense otherwise. There is no indication she was programmed to do so. Of what use is a religious robot?

The story suggests that in a world whose logic reduces humans to machines and in which machines inch closer and closer to human appearance, humans are able to maintain their dignity because they can and will never cease being able to love. It is an expression of both tremendous freedom and transcendence — freedom because, as Rick shows, love can be illogically and yet unconditionally offered, and transcendence because it surpasses the social logic of order and control. The novel’s human beings attain their most recognizable humanity when they contradict the logic of efficiency — when they love.

When Josie awakes crying in anguish, she pleads for her mother, the most primal and most loving relationship she has: “Don’t want to die.” Her mother’s soft reassurances — “It’s okay. Okay” — soothe her and send her back to sleep. And when Klara hopes that sabotaging a polluting machine may earn the Sun’s favor for Josie, Josie’s father joins in this sabotage, against all reason and common sense, with all the irrational desperation of a man on the cusp of losing a daughter. “I want what’s best for Josie. Exactly the same as you. So, I’m willing to grasp at whatever chance comes our way.”

To restate this in terms of “mattering,” it seems that loving transforms the world such that someone matters so much that we cease mattering much to ourselves. It is this source of willing self-abnegation, a self-emptying for the sake of another that separates humans from Klara. It is what Klara drives at when she says that Josie’s uniqueness — we might add, her dignity — exists in “those who loved her.” “That’s why I think now Mr. Capaldi was wrong and I wouldn’t have succeeded” in becoming Josie.

In Klara and the Sun, we see the dignity of humans best when they upend this programmed logic. It is this logic — not the possibility that we might one day create intelligent machines, but the idea that that is what we’ve already become — that we should really be anxious about.