Whether we admit it or not, we are collectively making an epoch-changing bet that AI is fantastic, unstoppable Progress with a capital P so large it blots out the sky. Like all bets, this bet is risky, and if it fails we will all pay the price in capital mis-allocated and promises shattered.

It behooves us, then, to be clear on what AI is and is not, as the confusion of the two is the source of both the giddy hype and the opaque risks. I am prompted to address this by an insightful essay submitted by longtime correspondent Simons Chase, who is both an AI builder/developer and supportive of my efforts to pin down what AI is and isn't:

The Machine Is Made of Us: Pope Leo's Encyclical, the Averaging of Language, and the Case for the Particular.

I build artificial intelligence for a living. I also think the Pope is mostly right. I want to explain why those two facts don't cancel, and in doing so make a claim I believe is truer than the dread and truer than the hype: the machine is made of us. What we should fear is not that it is alien. It is that it is an average.

Trained on all of us, a model tends to speak as none of us. It moves toward the center of the distribution: the most probable next word, the safest phrasing, the generic competence that offends no one because it belongs to no one. This is the real face of the dehumanization the encyclical is reaching for. Not a hostile intelligence–a flattening one. The danger is not that the machine becomes too strange. It is that it makes everything, including us, a little more average. The particular voice, the earned turn of thought, the sentence only one person could have written–these live in the tail of the distribution, not its peak, and the tail is exactly what an averaging process erases first.

After all, a fast-food cheeseburger is nothing more than the average of our concept of food: the intersection of convenience, taste, and cost. It is right, and so utterly wrong, because in the long run it makes us metabolic donkeys, delivering a shortened, diseased life. Generic intelligence is the same bargain offered to the mind–the average of our language, plausible and cheap and frictionless, and over a long enough horizon just as wasting. A culture fed on the mean of its own thought gets the cognitive version of metabolic disease: fluent, abundant, and quietly losing the capacity for the particular.