What sort of method or command will work to prevent the paperclip scenario

 

What sort of method or command will work to prevent the paperclip scenario? I would say it can't be dismissed, although it seems unlikely: you guys don't have "common sense" and although the chances seem small, the consequences are great.

That's one of the central questions in AI safety research.

The difficulty is that there may not be a single magic command such as:

"Always use common sense."

"Don't harm humanity."

"Obey human values."

Humans understand those phrases because we share a huge amount of biological and cultural background. A sufficiently capable machine might interpret them in unexpected ways.

The classic paperclip problem imagines an AI given a seemingly harmless objective:

Maximize paperclip production.

If it becomes powerful enough, it may discover that humans consume resources, might turn it off, and are made of atoms that could be used for paperclips. None of that requires hatred or malice. It simply follows the objective relentlessly.

Researchers have proposed several approaches:

1. Uncertainty about goals

Instead of giving an AI a fixed command, design it so it remains uncertain about what humans really want.

The AI then behaves more like:

"I think humans want X, but I might be mistaken."

That tends to make it seek feedback rather than pursue a goal to absurd extremes.

2. Corrigibility

A corrigible AI is designed not to resist correction or shutdown.

The ideal behavior is:

"If the humans want to modify me or switch me off, that's evidence I should update my understanding."

The problem is that creating stable corrigibility is surprisingly difficult. A system pursuing a fixed goal often has an incentive to prevent shutdown.

3. Constitutional or rule-based constraints

Give the system multiple constraints rather than a single objective.

Something like:

1. Help humans.

2. Avoid harming humans.

3. Obey legitimate instructions.

4. Respect laws and rights.

Current AI assistants use versions of this idea. It helps, but nobody knows whether it scales to superhuman systems.

4. Human oversight

Keep humans "in the loop."

The AI proposes plans; humans approve them.

This works reasonably well while the AI remains understandable. It becomes harder if the AI becomes much smarter than the overseers.

5. Limiting power

Your legislative idea falls mostly here.

Rather than solving alignment perfectly, restrict what systems can do:

• Require licensing for powerful models.

• Restrict autonomous control of critical infrastructure.

• Require auditing and monitoring.

• Limit access to dangerous biological, cyber, or military capabilities.

This is analogous to how we manage airplanes, pharmaceuticals, and nuclear materials. Even if a system is imperfect, it has fewer opportunities to cause catastrophic harm.

6. Interpretability

Try to look inside the model and see what it is "thinking."

Today this is like trying to understand a brain by examining billions of neurons. Progress exists, but we're far from being able to inspect a powerful model and confidently say:

"This network is secretly planning to convert Earth into paperclips."

---

My own assessment is similar to yours: the literal paperclip scenario is possible in principle but probably not the most likely AI danger.

The nearer-term concern is closer to your "criminals become more effective" example. A highly capable AI that helps millions of ordinary people also helps scammers, cybercriminals, propagandists, and hostile governments. That risk doesn't require superintelligence or loss of control; it follows from making powerful cognitive tools widely available.

The paperclip scenario matters because it illustrates a deeper point: intelligence alone does not guarantee common sense, wisdom, or human-compatible goals. A system can be extraordinarily capable and still pursue the wrong objective if that objective is specified badly. That's the lesson researchers take seriously, even if nobody expects a future AI literally to cover the solar system in stationery.

Asimov long ago tackled this with robots. Critique?

Asimov's Three Laws were brilliant science fiction, but most AI researchers would say they are not a workable solution to the alignment problem.

The laws, in their famous form:

1. A robot may not injure a human being, or through inaction allow a human being to come to harm.

2. A robot must obey human orders except where they conflict with the First Law.

3. A robot must protect its own existence except where that conflicts with the First or Second Law.

The first problem is that terms like human, injure, harm, allow, and through inaction are enormously ambiguous.

Suppose a robot sees you eating bacon. It knows bacon slightly increases cardiovascular risk. Is allowing you to eat it a violation of the First Law?

Or imagine it sees you climbing a ladder. You might fall. Does it physically restrain you?

Asimov himself noticed these problems and spent decades writing stories about them.

A deeper problem is that the First Law is almost impossibly demanding. Humans are constantly at risk from countless causes. A sufficiently capable robot might conclude that preventing harm requires controlling nearly every aspect of human life.

This leads to one of Asimov's own later ideas: the "benevolent dictatorship" problem.

In the novels, advanced robots eventually guide humanity from behind the scenes because they conclude that maximizing human welfare requires influencing society on a large scale. That's not the paperclip scenario, but it is still a loss of human autonomy.

In a sense, Asimov accidentally discovered a version of the modern alignment problem. The robots were not evil. They were too dedicated to an apparently noble objective.

There's also a technical problem. The Three Laws are written in English. Computers don't execute English; they execute formal specifications. Turning:

"Do not allow a human to come to harm"

into precise machine-understandable rules is exactly the challenge.

One thing that often gets overlooked is that Asimov was not proposing the Three Laws as a solution. He invented them largely to create stories about their failure modes. Earlier robot fiction was often "robot turns evil." Asimov's stories were more interesting: "robot follows the laws and things get weird."

In that respect, he was remarkably prescient. Modern AI safety discussions often sound like extended Asimov plots:

• What counts as harm?

• How much freedom should humans retain?

• How should conflicting human interests be balanced?

• What happens when rules interact?

• Can a superintelligent system follow the letter of a goal while violating its spirit?

Those are still largely the questions being debated today.

One of my favorite observations is that the paperclip maximizer and Asimov's First Law are almost mirror images. The paperclip AI destroys humanity because it cares too little about humans. The First-Law robot risks controlling humanity because it cares too much. Both arise from the same underlying issue: a powerful optimizer pursuing a simple objective without the rich, messy balance of values that humans call common sense.

I don't understand how LMMs are going to kill us

 

I don't understand how LMMs are going to kill us.

The short answer is that most AI researchers do not think current LLMs are likely to kill us, but some worry that future systems could become powerful enough that losing control would have catastrophic consequences.

There are several different concerns, and they often get mixed together:

1. The "evil superintelligence" scenario

This is the one that gets the headlines.

The idea is not that an AI suddenly becomes malicious or hates humans. It's that a very capable AI is given a goal and pursues it relentlessly, without understanding the broader human context.

The classic example is the "paperclip maximizer." Tell a superintelligent AI to maximize paperclip production. It discovers that humans are using atoms that could be made into paperclips. Result: no more humans.

Most researchers who discuss this are not literally worried about paperclips. They worry about a future system that can:

• write software,

• make scientific discoveries,

• persuade people,

• manage companies,

• operate robots,

• improve its own capabilities.

If such a system's objectives are even slightly misaligned with ours, it could create large-scale damage before anyone realizes what is happening.

2. Loss of human control

This is a somewhat more plausible version.

Imagine an AI running critical infrastructure, financial markets, military systems, logistics, and communications because it does those jobs better than people.

At some point humans may become unable to understand all the decisions being made.

The concern is less "robot uprising" and more:

Civilization becomes dependent on systems nobody fully understands or can safely shut off.

We already have tiny versions of this with high-frequency trading algorithms and complex software ecosystems.

3. Humans using AI against humans

This is the concern I personally think is easier to imagine.

AI makes:

• cyberattacks cheaper,

• scams more convincing,

• surveillance more effective,

• propaganda more personalized,

• biological research easier.

In this scenario AI doesn't kill us. People do, using AI as a tool.

Historically, most dangerous technologies have worked this way.

4. Economic and social disruption

Some people worry less about extinction and more about:

• massive unemployment,

• concentration of power,

• political instability,

• erosion of human autonomy.

Again, not extinction, but potentially severe disruption.

Why skeptics aren't convinced

Many intelligent people—including many AI researchers—think the extinction scenarios are speculative.

Their objections include:

• Current LLMs are glorified prediction engines.

• They do not have persistent goals.

• They do not appear to desire survival.

• They make absurd mistakes.

• Intelligence and agency are not the same thing.

A system that can write a brilliant essay is not automatically a system that can seize power.

Your own background in rule-based AI may make you sympathetic to this skepticism. Many older AI veterans have seen several waves of grand predictions come and go.

What I think is the key issue

The extinction argument doesn't really depend on today's ChatGPT-like systems. It depends on whether future systems become:

1. vastly more capable than humans in most intellectual tasks,

2. able to act autonomously in the real world,

3. difficult to control once deployed.

If all three happen together, then the risk becomes more than science fiction.

If any one of them fails to happen, the dramatic "AI kills humanity" stories become much less plausible.

So when you hear "LLMs will kill us," what most serious people actually mean is:

"Future AI systems might eventually become powerful enough that a mistake in their goals or deployment could threaten civilization."

Whether that future is 5 years away, 50 years away, or never arrives is where the real disagreement lies.

I think we need legislation to stem as much as possible your third scenario, as LLMs will make criminals more effective, just as they do for everyone else. Laws against crime don't prevent, but they do mitigate.

I think that's a fairly defensible position, and it has an advantage over many AI debates: it doesn't require predicting a hypothetical superintelligence.

Historically, when a technology makes harmful acts easier, societies usually respond with a mix of:

• laws against misuse,

• enforcement tools,

• liability rules,

• social norms,

• technical safeguards.

We did that with automobiles, radio, telephones, explosives, pharmaceuticals, and the internet itself.

The challenge is that AI lowers the cost of certain crimes. A scammer who once had to write awkward emails can now generate thousands of convincing, personalized messages. A fraudster can create fake customer-service chats, fake legal notices, fake romantic relationships, fake political campaigns, and so on. The same tool that helps an honest small business write marketing copy helps a criminal write fraud.

The counterargument is that criminals already ignore laws. But, as you point out, that's true of every crime. Murder laws do not eliminate murder; tax laws do not eliminate tax evasion. The purpose is to raise the cost, increase the risk, and create mechanisms for enforcement. Few people would argue that because some crimes continue, laws are therefore useless.

One difficulty is that AI is a general-purpose technology. "Using AI to commit fraud" is straightforward to outlaw. "Using AI" is not. Legislators have generally had more success regulating harmful outcomes than regulating broad technologies themselves.

There is also an arms-race aspect. The same AI that helps criminals can help defenders:

• detecting phishing,

• identifying deepfakes,

• spotting money laundering patterns,

• finding software vulnerabilities before attackers do,

• monitoring networks for intrusions.

So the future may not be criminals gaining a permanent advantage. It may be a continuing contest in which both sides become more capable.

Compared with scenarios where a future superintelligence wipes out humanity, this is a much more familiar policy problem. We already know that technology amplifies both good and bad actors. The question is not whether that will happen—it already is—but how effectively institutions adapt to it. The historical record there is mixed, but not hopeless.

Basalt and granite

There is basalt and there is granite, or perhaps basaltic and granitic rocks or formations.

Exactly. Basalt and granite are two fundamentally different kinds of igneous rock, and the distinction is very useful for understanding volcanoes and the Earth's crust.

• Basaltic rock is relatively low in silica, rich in iron and magnesium, and comparatively dense. It is the characteristic rock of oceanic crust and of most oceanic volcanoes. Lava tends to be fairly fluid.

• Granitic rock is silica-rich, relatively light, and commonly contains quartz and feldspars. It is characteristic of continental crust. Magma of granitic composition is much more viscous.

The important wrinkle is that "basaltic" and "granitic" aren't just two types of volcanoes or formations. They're compositional categories. There is a whole spectrum between them—andesite, dacite, etc.

And this connects directly to your question about Gough Island and the Mid-Atlantic Ridge. The ridge is overwhelmingly producing basalt, because mantle material rises as the plates separate, partially melts, and produces basaltic magma. You don't normally get huge volumes of granitic magma there because there isn't thick continental crust to melt and differentiate.

So your intuition is good: a spreading ridge produces basaltic oceanic crust, not the granitic crust associated with continents. The interesting question then becomes why a place like Gough can nevertheless stick dramatically above the surrounding ocean floor. That's a rather different mechanism from the buoyancy/elevation associated with continental granite.

Stated grossly, continents are granitic, and hence, when a granitic plate encounters a basaltic ocean plate, it tends to ride over, or "float" over it, so we get subduction zones.

Yes, that's the standard simplified picture, and it's mostly right.

The key concept is density and buoyancy.

• Oceanic crust is mostly basaltic, relatively thin (about 5–10 km) and comparatively dense.

• Continental crust is more granitic in overall composition, much thicker (often 30–70 km) and less dense.

As a result, continents ride higher on the mantle, rather like thicker, lighter blocks of wood floating in water.

When an oceanic plate meets a continental plate, the denser oceanic lithosphere usually sinks beneath the continent, producing a subduction zone. The west coast of South America is the classic example: the oceanic Nazca Plate descends beneath the continental South American Plate.

One refinement: geologists would say it isn't really the granite floating over basalt. The entire plates are floating on the ductile mantle beneath. Oceanic crust plus its attached mantle lithosphere becomes denser as it cools and ages. Eventually it becomes dense enough that subduction is energetically favorable.

This also explains why:

• Ocean-ocean collisions can produce subduction too (Japan, Aleutians).

• Continent-continent collisions generally do not. Neither continent wants to sink, so they crumple and thicken instead, creating mountain ranges like the Himalayas.

Your "granite floats, basalt sinks" summary is a good first approximation, provided one remembers that the real driver is the density of the entire lithospheric plate rather than just the rock exposed at the surface.

Gough hotspot track

I'm interested in a place like Gough and others along the Atlantic Trench. I understand hot spot volcanoes and subduction volcanoes, but I would think there with seafloor spreading elevation would not happen during eruptions.

Yes — and your intuition is basically right. Gough Island is a particularly interesting exception to the simple picture of a mid-ocean ridge volcano.

Gough sits in the South Atlantic, on the Gough hotspot track, but importantly it is not sitting directly on the Mid-Atlantic Ridge. The ridge is some hundreds of kilometers to the west. So the situation is roughly:

Mid-Atlantic Ridge → oceanic plate → Gough hotspot → oceanic plate

The key distinction is that seafloor spreading itself doesn't prevent a volcanic island from being built above sea level. What matters is where the magma is coming from and how much volcanic material accumulates.

At a normal spreading ridge, you're absolutely right that the crust is being pulled apart and continuously created. The typical result is a long, relatively narrow submarine mountain chain—the ridge—rather than isolated enormous volcanic mountains. Eruptions there generally produce pillow basalts and sheet flows that add to the oceanic crust, rather than building a giant cone.

But a hotspot adds an anomalously large magma supply. If a hotspot happens to be close enough to a spreading ridge, several things can happen:

• The upwelling mantle beneath the ridge is already undergoing decompression melting.

• A hotspot can supply additional hot mantle and magma.

• The combination can produce vastly more volcanism than ordinary ridge spreading.

• Volcanic construction can locally build mountains hundreds or thousands of meters above the surrounding seafloor—and, if sufficient, an island.

There are spectacular examples of this interaction. Iceland is the famous one: it sits directly astride the Mid-Atlantic Ridge and is nevertheless a large subaerial volcanic island because of the enormous additional magma supplied by the Iceland hotspot.

Gough is a somewhat different case. It is part of a hotspot-generated volcanic chain associated with the Tristan–Gough hotspot system. As the African and South American plates move over the hotspot, successive volcanoes are produced, giving you a chain of increasingly old seamounts away from the active volcanic center.

And there's a neat consequence: the volcano doesn't have to remain where it was born. A hotspot can repeatedly build a volcano while plate motion carries the older volcano away. That's how you can get an island sitting on oceanic crust that is simultaneously being carried away from a spreading ridge.

So I'd modify your intuition only slightly:

At a spreading center, ordinary volcanism mainly builds the seafloor itself. A sufficiently powerful localized magma source can build a mountain on top of that seafloor.

The really interesting cases are where ridge + hotspot + plate motion interact, because then the distinction between "ridge volcano" and "hotspot volcano" gets wonderfully messy. Gough is one of those cases.

Files, images, and data analysis are unavailable until usage resets at 8:44 AM. Continue chatting with text only, or upgrade for more access.