Flashcrash

 Watched the beginning of a YouTube video where the guy describes the "flashcrash" on the securities exchanges a couple of decades ago brought about by automated trading programs, where the same programs self corrected within about half an hour.  The video blames it on AI, but my reaction is that real AI would be smarter and not do such a thing.  This was just automation, code written without thinking it through properly and putting in stops on extremes.

I think your reaction is largely correct.

The 2010 "Flash Crash" is often described in popular media as an AI disaster, but by modern standards it wasn't AI in the sense that people use the term today. It was a complex interaction of automated trading systems following pre-programmed rules. The systems were fast, adaptive in a limited sense, and interacting with one another, but they were not reasoning about the market the way a human trader might.

What happened, in simplified form, was something like this:

    A very large sell order hit the market.

    Algorithmic trading systems responded according to their programmed strategies.

    Some algorithms sold because prices were falling.

    Others withdrew liquidity because conditions looked abnormal.

    The lack of buyers caused prices to fall even faster.

    The algorithms reacted to the worsening conditions, creating a feedback loop.

It's a bit like a crowd all heading for the same exit because each person sees everyone else moving that way. No one intends a stampede, but collectively one emerges.

The interesting part is that the market also recovered quickly because many of those same automated systems eventually recognized that prices had become wildly disconnected from reality and started buying again. So the system both created and corrected the problem.

Your observation about safeguards is one that regulators and exchanges largely agreed with afterward. Since then, exchanges have added various "circuit breakers" and trading pauses designed to stop exactly that sort of runaway feedback loop.

As for whether "real AI" would do better, I'm not entirely convinced. A sufficiently intelligent AI might recognize that a stock worth $50 probably isn't suddenly worth a penny and refuse to join the panic. On the other hand, if you had thousands of powerful AIs trained on similar data and pursuing similar objectives, they might still converge on the same mistake. Intelligence does not automatically eliminate herd behavior.

Humans provide a useful comparison. The stock market had spectacular crashes long before computers existed:

    1907

    1929

    1987 (when computers played some role, but humans designed the strategies)

    numerous smaller panics

Humans are capable of creating feedback loops all by themselves.

So I'd phrase it this way: the Flash Crash was less an AI failure than a systems-engineering failure. Lots of individually reasonable automated rules interacted in an unreasonable way because nobody fully anticipated how they would behave when all operating simultaneously under stress.

That's a recurring theme in finance, aviation, power grids, and other complex systems. The disaster often isn't one component failing; it's many components doing exactly what they were designed to do.

That makes the story even better.

From the operator's viewpoint:

    Normal runtime: 5 minutes
    Current runtime: 3 hours

That's exactly the signature of a hung process.

From your viewpoint:

    Of course it's taking 3 hours. It's touching every file in the system.

A data-cleaning pass is one of those jobs that completely changes the workload profile. A nightly run might normally process only new transactions or changed records. A cleanup run suddenly becomes "read everything, examine everything, potentially rewrite everything."

What strikes me is that you did the right thing: you warned him.

The amusing part is that operations people are trained to trust what the machine is telling them more than what some email from yesterday says. That's usually the correct instinct. If the machine says something is wrong, you investigate.

The trouble comes when the machine is reporting exactly what was planned.

I've seen variations on this throughout computing history:

    "The disk is full!" — yes, we're copying the archive.

    "The network is saturated!" — yes, we're restoring a backup.

    "The CPU is at 100%!" — yes, that's why we bought it.

The operator sees symptoms; the designer sees intent.

Your response also suggests a certain confidence in the system. You didn't tell him to kill the job, inspect logs, or restart anything. You told him to read his email because you already knew what was happening.

And then, naturally, while he was reading it, the job finished.

That's almost a law of computing: a process that has been running for hours will often complete the moment someone escalates it to the person who wrote it.

The machine equivalent of taking your car to a mechanic because of a strange noise and having the noise disappear the instant the mechanic gets in the passenger seat.

The operator apparently was in the habit of reading his emails when he had a little extra time.  Because of all the disk loading he was having to do (old days) he hadn't had any that night.  New rule: read emails as first task of the evening.

That is a very believable "operations lesson learned" story. The technical problem wasn't really the long-running job; it was the information flow.

The operator had developed a reasonable routine for normal nights:

    Start the scheduled jobs.

    While waiting for the system to finish its usual work, catch up on email.

    Investigate anything unusual.

But that routine had an unstated assumption: there will always be spare time before anything needs attention.

Your cleanup job broke that assumption. It consumed the very window in which he normally absorbed the information that would have prevented the alarm.

So the new rule—read email first thing—was actually a systems improvement. Not because people suddenly became more disciplined, but because the environment had changed. The old workflow depended on idle time; the new workflow did not.

It's a good example of why operational procedures evolve. Many "rules" that seem bureaucratic are actually fossils of previous incidents:

    "Check the change log before restarting anything."

    "Read the maintenance notices before calling an outage."

    "Verify the backup completed before formatting a drive."

    "Check scheduled jobs before killing a process."

Somebody, somewhere, learned those lessons the hard way.

And the old computing environment makes the story even more understandable. Today people forget that disk I/O used to be a major bottleneck. A full-system pass was not just "run a script"; it meant hours of mechanical activity: seek, read, process, write, repeat. The operator could hear the machine working, but without the context he couldn't know whether it was healthy work or a runaway.

Your example is actually a nice illustration of a broader principle in reliable systems: the human process around the computer is part of the system. A perfectly written program can still fail operationally if the people running it don't have the information they need at the right time.

Back to my main agenda here: the video was a scare piece about the danger of AI but I see it as a reason for more AI.

I think your interpretation gets at an important distinction that many of those "AI danger" videos blur.

The Flash Crash is actually an argument for more intelligent oversight, not necessarily less automation. The failure was not that a machine was "too intelligent"; it was that a collection of narrow, fast, rule-following systems interacted in a way humans had not anticipated.

In your programming example, the remedy was not:

    "Never run automated jobs."

It was:

    "Build better monitoring, better alerts, better safeguards, and better understanding of failure modes."

AI could be very useful in exactly that role.

Imagine a market-monitoring system whose job is not to trade, but to watch the traders and say:

    "The current behavior resembles a positive feedback event. Liquidity is disappearing faster than normal. Several independent algorithms are reacting similarly. Consider pausing."

That is a very different thing from an algorithm that simply says:

    "Price down. Sell."

The same applies to cybersecurity, power grids, aviation, medicine, and industrial systems. A lot of the danger comes from systems becoming too complex for humans to mentally simulate. AI could help by acting as a pattern-recognition layer over those systems.

Of course, the counterargument is that AI itself becomes another complex system that can fail. That's true. But the answer historically has rarely been to abandon powerful tools. It has been to add layers:

    calculators didn't eliminate the need for mathematical understanding;

    autopilots didn't eliminate pilots;

    computer-controlled reactors didn't eliminate engineers;

    electronic trading didn't eliminate market rules.

The interesting question is not "AI or no AI." It is "where do we put intelligence, and where do we put constraints?"

Your programming story is actually a good analogy. The best system would not be one that says, "My programmer would never create an infinite loop." It would be one that says, "My programmer may be brilliant, but I have a watchdog anyway."

A truly useful AI system should probably be designed with that same humility: assume that even intelligent systems can encounter situations they did not anticipate, and build mechanisms to catch those situations early.