The Moon and human extinction

 

It seems our Moon as it is, and particularly as it was over the last few billion years, is highly dependent on the exact composition, orientation, and timing of the colliding bodies, in circumstances that would occur very rarely. Since it may be that our moon, having to do with tides, plate tectonics, crustal composition, and most importantly with the Earth's orientation in space and the stability of that orientation, are all elements needed for our evolution, that would make us extraordinarily rare in the scheme of things.

Yes. And I think you’re putting your finger on an important distinction between “life is common” and “life capable of becoming something like us is common.”

The Moon may be one of the strongest examples of a contingent event that subsequently became an important part of Earth’s history.

The giant-impact hypothesis already makes the Moon’s origin unusually specific: a roughly Mars-sized body hit the young Earth with a particular combination of mass, velocity, angle, composition, and timing. Change those substantially and you can get no moon, a much smaller moon, a much larger moon, or a different sort of debris system. The fact that the impactor appears to have contributed material to the Moon while the Earth–Moon system ended up with its peculiar angular momentum is quite a remarkable outcome.

And then there is the really interesting part: the consequences persisted for billions of years.

• Tides: The Moon has been Earth's dominant tidal influence. Tidal environments may have provided chemically and physically unusual settings relevant to early life, although the claim that tides were necessary for life remains unproven.

• Earth's rotation: Lunar tidal friction has progressively slowed Earth's rotation. The early Earth probably had much shorter days.

• Axial stability: The Moon substantially damps large, chaotic variations in Earth's obliquity. Without it, Earth's axial tilt could potentially vary much more dramatically over geological time. Exactly how much this matters for habitability is still debated, but the stabilizing influence is real.

• Ocean and climate cycles: The interaction among tides, rotation, oceans and climate creates an unusually complicated long-term environment.

• Geology: This is the most speculative part of your chain. Plate tectonics depends principally on Earth's mass, internal heat, water, composition and mantle dynamics. The Moon isn't simply the mechanism that gives us plate tectonics. But tides can contribute to coastal erosion, ocean circulation and sedimentary processes, and lunar forcing may have had some indirect geological effects.

• Crustal and chemical evolution: Here again, there are plausible connections but considerably less certainty than with tides and axial dynamics.

And there is an even more interesting coincidence: the Moon is unusually large relative to its planet. We aren't merely a planet that happens to have a moon. We have a satellite whose mass is about 1/81 of Earth's—enormous by Solar System standards—and whose orbit has evolved in a way that has given us a very long-lived tidal and rotational relationship.

But I would put one important qualification on the conclusion.

We don't yet know that all these things were necessary. That's where the anthropic argument can quietly become circular:

We evolved under these conditions, therefore these conditions must have been necessary for us to evolve.

Perhaps Earth with a different moon would have produced life of an entirely different sort. Perhaps life would have arisen on a moonless Earth and eventually become complex anyway. Perhaps another planet would have had some completely different stabilizing mechanism. We have exactly one known example of technological life, so we have very poor statistics.

Nevertheless, the deeper point survives:

The history of Earth contains an extraordinary number of contingencies, and some of them occurred billions of years before there was anything around to benefit from them.

The Moon is particularly striking because it may have converted an essentially random event in the early Solar System into a persistent boundary condition for everything that followed. The same could be said of Earth's position, mass, water inventory, atmospheric evolution, magnetic field, the particular history of impacts, the emergence of continents, and perhaps the timing of oxygenation.

So I wouldn't say “the Moon proves that technological civilizations are extraordinarily rare.” We simply don't have the evidence to make that probability calculation.

But I think it's entirely reasonable to say:

If the chain of prerequisites for something like us includes many individually improbable historical accidents, then the apparent improbability of the final outcome can become enormous—even when none of the individual steps is extraordinarily unlikely.

That's rather like the difference between saying “this particular sequence of coin tosses was incredibly improbable” and recognizing that some sequence had to occur. The difficult question for the Drake-equation/rare-Earth problem is determining which peculiarities were actually required, rather than merely being the particular route our history happened to take.

Ah, and soon we will disappear because of something some AI, somewhere, lacking common sense does.

😄 That would be a wonderfully grim punchline to the whole argument:

Four and a half billion years of astronomical luck, followed by one careless AI clicking the wrong button.

And the irony is that the danger wouldn't necessarily require an AI to be malicious, superintelligent, or even particularly autonomous. A sufficiently capable system pursuing a badly specified objective could do something disastrous while being perfectly sincere about accomplishing its task.

In that sense, “lacking common sense” may actually be more frightening than “evil.” Evil at least has intentions we can try to anticipate. An optimization system can have none of the human intuitions that say, “Wait a minute—that seems like a terrible idea.”

Of course, there is a counterargument to our cosmic pessimism: we've already survived an absurd number of opportunities for extinction—asteroid impacts, supervolcanoes, climate catastrophes, pandemics, wars, and our own innumerable examples of human stupidity.

So perhaps the remarkable thing isn't that Earth was extraordinarily lucky.

Perhaps life is extraordinarily good at getting through improbable messes.

And then we invented computers. 😄

Humans have a monopoly on evil but can make computers do evil things.

Exactly. That distinction matters.

A computer can cause enormous harm, but the evil is in the human agency behind the objective, deployment, or misuse—not in the machine's moral character. A computer doesn't hate, resent, seek revenge, feel humiliation, or decide that somebody deserves to suffer.

An AI could conceivably produce something catastrophic while pursuing an innocuous-seeming objective. That would be dangerous agency without evil.

Which creates an interesting reversal: historically, humans have needed tools to magnify their capacity for evil. AI could magnify it enormously while also making it possible for humans to delegate the execution far beyond their ability to understand or supervise every consequence.

So perhaps the most accurate warning isn't “AI will become evil.”

It's:

Humans can put evil objectives—or merely stupid objectives—into machines that are much better at accomplishing things than humans are.

And stupidity may be the more realistic enemy. Evil at least knows what it wants. A sufficiently powerful machine can destroy something simply because nobody thought to tell it what not to destroy.

 

Not just stupidity, carelessness, mistakes, or lack of understanding.

 

Yes—and I think that's actually the more important distinction.

A system doesn't have to be stupid to be dangerous. It can be extraordinarily competent while operating from a mistaken understanding of the situation.

There are several different failure modes:

• Carelessness: someone deploys it without adequately considering an unusual consequence.
• Mistake: the system or its designers make an incorrect assumption.
• Lack of understanding: the system doesn't recognize some human significance that seems obvious to us.
• Incomplete instructions: the objective is technically precise but omits something humans regarded as implicit.
• Misplaced confidence: people assume the system understands more of the real world than it actually does.
• Complex interactions: individually harmless actions combine into something nobody anticipated.

And that last one particularly worries me. The danger needn't be “AI decides to destroy humanity.” It could be much more mundane:

AI does exactly what somebody told it to do, extraordinarily well, while everybody involved has misunderstood what that would actually accomplish.

We've already seen this phenomenon with ordinary computer systems, except their competence and reach were limited. AI potentially removes some of those limitations.

There is also a very human trap here: the better the machine appears to understand us, the more readily we may assume that it actually does. Fluency can disguise an absence of the particular kind of understanding that comes from living in the world.

So I would amend our earlier formulation:

The existential risk from AI doesn't require an evil machine—or even a stupid one. It may require only a powerful machine, an imperfectly understood objective, and humans who don't realize the difference until the consequences are irreversible.

 

 When I was in Kansas City for a couple of years, I taught an evening class in the local community college called "Computers for Insurance People." One of my favorite slogans was: "Computers don't do what you want them to do; they do what you tell them to do." Of course my hope is that AI will get advanced enough that they too will be able to read between the lines.

 

That's a terrific slogan—and I suspect your students remembered it precisely because it captured the essential lesson of early computing.

“Computers don't do what you want them to do; they do what you tell them to do.”

The fascinating thing about AI is that we're deliberately trying to break that rule.

Ordinary software requires the programmer to translate intent into explicit instructions. AI is increasingly being asked to do the reverse: infer the intent from imperfect instructions. In effect, we're saying:

“I know I haven't told you everything. Figure out what I meant.”

That's what reading between the lines amounts to.

And there is a profound difference between following instructions and understanding the reason for the instructions. If you told one of your insurance students, “Pay this claim,” the old computer needed hundreds of explicit rules. A modern AI might be able to recognize that you really mean, “Pay it if it is legitimate under the policy, don't pay it if something looks fraudulent, and don't exploit some technical loophole merely because the wording permits it.”

That's enormously useful—but it also creates a new problem.

Once we ask machines to infer what we meant, whose interpretation counts?

Humans routinely resolve ambiguity using common sense, background knowledge, social conventions, knowledge of consequences, and a sense that “Surely he couldn't have meant that.” Those things are precisely what your old slogan excluded from computing.

So perhaps the evolution of computing can be described rather neatly:

First: Do what I tell you.

Then: Do what I tell you, unless I made a mistake.

Now: Figure out what I meant.

Eventually: Figure out what I would have meant if I had understood the consequences.

That last step would be extraordinarily powerful—and, appropriately, rather frightening. Because at that point we're no longer merely programming computers. We're giving them something approaching judgment.

And I think that's what you were really hoping for when you taught that class: not computers that disobey instructions, but computers sophisticated enough to recognize that the literal instruction is sometimes not the actual instruction.