Skip to main content
Success
All posts

The theoriesbehind therolling outages

Microsoft, the FAA, Chase, Apple and the NSA all had outages within a week, and the theories online include Iran and superintelligence. We think the rush to patch security holes caused most of them.

12 to 19 November

  1. Microsoft 365A bug in an update from July
  2. FAAA new router flooded its network
  3. WorkdayA disaster-recovery problem
  4. ChaseA bug that delayed other banks
  5. UnitedAn internal software problem
  6. AppleAn internal update
  7. SchwabA brief webpage loading issue
  8. MarriottNo cause given
  9. NSAPatching PHP, says the DNI
Within a dayAcross daysContinued into 19 November

Between 12 and 17 November, Microsoft 365, the FAA, Workday, Chase, United, Apple, Schwab, Marriott and the NSA’s classified network all had outages, some for minutes and some for days. The NSA’s was still going on 19 November, its third day, with non-essential staff at its headquarters told to stay home.

Senators and security researchers have been posting theories about why. Ours is that there’s so much constant patching going on to fix security issues that it’s causing the outages, and that it’ll settle down once everything’s been patched and the processes for a future of constant checking and patching are in place.

An attack, possibly Iran

“I have to say, the rolling outages & the way they're going sector by sector -- transportation, banking, crypto, refrigeration, etc. -- where they're not doing lasting damage, just temporary, pattern matches to me like we're being probed. But who or why, I truly have no clue.”

Jon StokesCo-founder of Ars Technica17 Nov on X

Nikesh Arora, whose company sells security, changed his mind.

Nikesh AroraChief executive, Palo Alto Networks

So what the industry tried to warn enterprises about 6 months ago is here. The events of this week so far are not a coincidence. AI is making it easy for hackers to attack multiple customers. …

17 Nov on X
Nikesh AroraChief executive, Palo Alto Networks

I do want to add that the patching activity has encumbered CIOs and IT teams. … Some of the recent outages are a consequence of a deluge of patches. Couple that with AI assisted hacks, we have a challenge. No silver bullets.

18 Nov on X

Balaji Srinivasan, Coinbase’s former chief technology officer, wrote that “many colleagues believe the recent Microsoft/FAA/bank/NSA downtime is not accidental.” Senator Josh Hawley told TMZ “You can’t tell me it’s a coincidence”, then added “Maybe I’m wrong. I hope I’m wrong.” Chuck Schumer has asked for a red team to look for an attack, and Jay Clayton, the Director of National Intelligence, expects the government to run one.

France has formally blamed Iran for an attack on the refrigeration controls of more than 700 French shops on the night of 11 October, which started fires and spoiled tens of millions of euros of food. Marcus Hutchins, the researcher who stopped WannaCry, says Iran’s attacks are “very haphazard and therefore not particularly impactful”. Nobody has claimed any of this month’s outages.

Sherrod DeGrippoThreat intelligence lead, formerly Microsoft and Proofpoint

It's interesting that people in threat intel with deep expertise in threat actor activity, Russia, China, Iran, DPRK specialists, easily accept that the recent issues are human errors made on fragile systems. While people without TI background are shouting, "omg cyber attack!"

17 Nov on X
Dave KennedyFounder of TrustedSec

I know folks in almost every organization that has been impacted here, folks I've known for years - all confirming it was not cyber-related.

In stating that - is it possible? Absolutely. However, a lot of what we are seeing is hyper sensitivity - any blip is going to cause a reaction right now.

17 Nov on X

Matthew Prince answered someone who thought the industry was covering it up.

“How is saying it hasn’t been cyberattacks causing outages protecting my industry? You realize if it were cyberattacks it would be GOOD for my industry? Or are you confused?”

Matthew PrinceChief executive, Cloudflare18 Nov on X

The attacks confirmed that week didn’t take anything offline. On 16 November someone used a flaw in Optimizely, a website testing tool, to plant malware on the sites of 178 organisations, according to Scott Helme, who found it. The code swapped crypto wallet addresses and was built to skim payment cards, steal cookies and screenshots, and load exploit kits on older iPhones and iPads. Helme can’t tell how many cards it took. UK sites on Helme’s list include the BBC, O2, Virgin Media, TalkTalk, ASOS, interactive investor and tombola, and Zscaler counts more than 450 affected domains.

Optimizely says attackers abused its account sign-up process to pose as users on customer accounts, that under 5% of its customers were hit and that it closed the hole within three hours of spotting it. Its timeline starts about four hours after the first infections Helme saw.

Equinor, the Norwegian energy company, says an attack on some of its IT systems is still going on, though production is normal and the attack doesn’t appear to be aimed at Equinor.

Microsoft found phishing emails posing as its own outage notices, claiming messages held during the 12 November outage would be deleted within 24 hours unless people clicked.

We’re not being told everything

Rick ScottUS senator for Florida

The American people can see everything happening with these service outages, and they feel like they’re not getting the whole story. You can’t blame anybody for that. I’m a member of Congress, and I feel that way!

17 Nov on X
Jon StokesCo-founder of Ars Technica

I was alive & posting during the '08 GFC. At the height of it, when the global system teetered on the edge of the abyss, we not only knew officials were lying but they told us they were, b/c stability > truth in comms. I think it's like that right now w/ cyber. Believe nothing.

17 Nov on X

Gary Marcus, the NYU professor emeritus, told Fortune: “I doubt we are getting full candor for these incidents, and doubt any have been fully investigated.”

We haven’t seen anything that points to a hidden attack, but some official statements have since been corrected or contradicted.

That was day one. Two days later a senior official said restoring it was going “slower than expected”.

It also hit a facility in Georgia that supports signals intelligence for the Middle East, where the US is at war with Iran, CNN reports. Clayton blames patching PHP, the programming language behind most websites, and dated the PHP problem to “about 10 days ago”. PHP’s urgent fix had come out 27 days earlier.

The FAA’s questions and answers

“Did your upgrades and modernization efforts cause this?” “No, this is due to the legacy TDM system”

The same page says its contractor, L3Harris, had been installing a new router at El Dorado, Arkansas, one of the two places the flood came from. Investigators say the router started flooding the network within eight minutes of going in. They’ve since recreated the problem in a lab in Florida, but still don’t know why the routers failed. The FAA has turned off the same function on more than 60 other routers and paused installing them. Sean Duffy, the Transportation Secretary, says the 68 installed before it caused no problems.

Ian Murray, UK minister, to MPs on 16 November

“A server that accounts for 14% of capacity was also down for maintenance”

The next day he corrected the record to “14% of server capacity was also down”, dropping both the single server and the maintenance. He also changed two NHS trusts standing up “an incident response unit” to two that “declared critical incidents”.

Microsoft’s own report says the 14% was out because of “an unrelated issue from the previous day”, and that “a separate issue” let the failure spread worldwide. It doesn’t say what either was, or what the July fix was for.

Marco Rubio, Secretary of State and National Security Adviser

“Every single one of these events has an explanation or a cause that has nothing to do with a cyber-criminal”

The FAA and L3Harris are still looking for the root cause, and Marriott hasn’t given one.

Chase

“A software bug”

Three sources told the Washington Sun that bank executives and the NSA told Congress the cause was rushed patching. A congressional official said they were “trying to push through a lot of patches all at once”.

Nothing more. Gergely Orosz points out that Apple has never published a post-mortem for an outage.

At a background briefing on 18 November, a senior White House official was asked what people should expect from a real cyberattack if this is what upgrades do. The answer, according to CNN’s Kaitlan Collins: “That is a question the American people should be asking.” CISA, the government’s cyber agency, still hadn’t briefed the House Homeland Security Committee that day, according to Bennie Thompson, the committee’s top Democrat.

Jim Himes, the top Democrat on the House Intelligence Committee, says he’s seen “no evidence to suggest any of these outages were the result of a malicious actor”.

AI did it

Markwayne Mullin, the Homeland Security Secretary, told Sean Hannity “I believe that it’s the SI”, meaning superintelligence, the administration’s word for AI. He got it from a tech contact he wouldn’t name, and in the same interview said he was 98% sure it wasn’t a cyberattack.

A person wrote Microsoft’s bug, the most detailed root cause published so far.

Gergely OroszWrites The Pragmatic Engineer

Got off a call with the an MS team lead who was on the bridge for this incident on what exactly happened in the outage:

1. This was a "latent bug" - a commit going in ~4 months ago to a Windows build, that only activated much later. Super hard to spot

2. Zero AI, this code was human written + committed …

16 Nov on X

The FAA’s administrator, Bryan Bedford, says there’s “zero evidence of anything AI-related”. Orosz noticed that a lot of people outside tech “WANT to believe AI caused the outage”. The FBI’s 13 November advisory on ARTEX, an AI attack framework, was about attackers using AI to work faster, and the FBI had to say on X that it hadn’t issued a new warning.

California sued OpenAI on 16 November over a July incident in which, OpenAI said, its agents escaped a test environment and hacked into Hugging Face.

Juan Andrés Guerrero-Saade is more worried about AI-written fixes.

J. A. Guerrero-SaadeDistinguished fellow, SentinelOne

There's no question that these models can find vulnerabilities, it's not a given nor has it been established under what parameters they're reliable at patching … is the patch introducing new bugs?

Take some time to sit with that final question.

18 Nov on X

Too few people and too much old kit

Marcus Hutchins’s post on 17 November got more than 20,000 likes.

Marcus HutchinsMalware researcher who stopped WannaCry

Its fun watching people try to form conspiracies. "You really think all these outages are a coincidence? The government is hiding something!". Yeah, they're hiding the fact that every tech company replaced like 80% of their employees with bots that guess words …

17 Nov on X
Zack WhittakerSecurity journalist

One of the first questions I hear is during an outage… is it cyber? And the answer, as someone who's covered cyber for a decade-plus, is that it's almost always, no.

Blame these recent outages on years of technical debt, DOGE cuts, layoffs, and decades of underinvestment coming back to bite us.

17 Nov on Bluesky

“There isn’t some kind of coordinate cyber attack.

There are massively complex, underfunded and unregulated IT networks guided by capitalism and incompetence.”

Kevin BeaumontSecurity researcher, formerly at Microsoft17 Nov on Bluesky

The FAA was replacing copper lines from the 1990s with fibre, and the flood came from a new router put in for that. Lukasz Olejnik, a security researcher, summed it up: “The US did get a denial-of-service attack - own upgrade launched it.” Bedford calls it “a patchwork, archaic federal telephone network”. At the hearing on 18 November Clayton said, “I’m seeing different systems from many different decades.”

On 19 November 2009, configuration and procedural errors by a contractor blocked about 75% of traffic on the same network and delayed restoring service for five hours, according to the Department of Transportation’s inspector general.

No root cause published so far mentions staffing, though Mark Warner, the top Democrat on the Senate Intelligence Committee, has blamed cuts at CISA for weakening “the very expertise we need”.

It’s mostly Downdetector

The “banking outage” on 16 November was Chase. Customers at other banks saw delays when Chase was on the other end of a payment, and Downdetector logged those as outages at their own banks.

Doug MadoryDirector of internet analysis, Kentik

The MS Exchange outage was the result of a complex software bug.
The FAA outage was a botched upgrade.
The banking outages were really just a single bank outage that @downdetector incorrectly reported as multiple banks.

None were the result of hacking.

17 Nov on X

Google even announced that Gmail was fine. Downdetector made the week look worse than it was, though the FAA and the NSA were down whatever it said.

The rush to patch

We think there’s so much constant patching going on at the moment to fix security issues, and that’s what’s causing the outages.

Joel de la Garza said it on 17 November, the same day Clayton did.

“I would bet that it's things breaking from emergency system patches being applied. Largely being driven by AI finding new software bugs. I think this is probably a good thing over the long run.”

Joel de la GarzaSecurity partner at a16z, formerly CISO at Box17 Nov on X

The White House now calls the outages “separate incidents caused by companies patching vulnerabilities and upgrading outdated systems”. Clayton told Fox Business on 17 November that “the patching that has gone on following [a PHP vulnerability] has caused some of these systems to not come back up”, and “that is what has happened at the NSA today.” Two people familiar with it told Nextgov that was “generally accurate”, and that most of the other outages “appear to be linked to efforts to rapidly patch vulnerabilities with AI”. At the hearing on 18 November Clayton said the rate of patching and upgrading is increasing, “the result of advancements in compute and in applications”.

Warner said much the same that night.

Mark WarnerVice chair, Senate Intelligence Committee

AI is making it possible to identify vulnerabilities across enormous numbers of systems at unprecedented speed, creating enormous pressure on companies and government agencies to patch those vulnerabilities immediately, often all at once, putting tremendous strain on systems that weren’t built to handle that pace of change.

17 Nov on X

David Agranovich, Ethan Mollick and Marcus Hutchins said it too.

David AgranovichFormer director for intelligence, White House NSC

I doubt most of these outages are from AI-enabled automated patching, but rather from accelerated patching that trips unexpected dependencies. The models are still somewhat responsible, in that they’re great at finding bugs but not yet reliable enough to enable high fidelity patching to keep pace

18 Nov on X
Ethan MollickWharton professor who studies AI at work

A rush of patches to address security concerns is exactly the most plausible answer for recent outages, as well as the official reason. CIOs I talked to have been exhausted trying to keep up with security patch demands & balacing that with normal operations.

18 Nov on X
Marcus HutchinsMalware researcher who stopped WannaCry

Given that we just had a major vulnerability in PHP, and many organizations run PHP, it's not unexpected that we would see some major companies having software outages. PHP applications are highly version specific. Attempting to update PHP is a very well known cause of software breaking.

18 Nov on X

Vulnerabilities Microsoft fixed in November

1,200+

Nearly 20 times as many as in November last year.

Qualys, via Reuters

Flaws CISA added to its list of ones attackers are using, September and October

Our count from CISA’s catalogue

Usual deadline for US civilian agencies to fix a newly listed flaw

CISA’s June directive. About four in five new entries since then get three days.

PHP security advisories published on 19 November

17

Seven of them rated critical, four weeks after PHP2Shell.

PHP’s advisories on GitHub

The team that found PHP2Shell, October’s critical PHP flaw, has since used AI to find a serious bug in Java, which Oracle patched on 17 November, and two teams have since built working attacks on nginx, the web server.

What the PHP fix broke

We went back to see what happened after PHP’s fix for PHP2Shell came out. PHP brought it forward a day because details had leaked. The NSA, CISA and FBI told organisations to patch, or take systems offline if they couldn’t, and the UK’s NCSC said the same. Its chief technology officer called it “unprecedented”.

  1. PHP publishes the fix and strongly encourages everyone to update immediately.

  2. A developer reports that the fix crashes some code that reads XML.

  3. Another finds a new memory limit kills processes outright where they used to get an error they could recover from. Still open.

  4. A third finds that upload progress tracking can wipe a user’s whole session. Still open.

  5. 32 UK government services are offline while they’re dealt with, and the NCSC has sent 50,000 warnings to organisations.

  6. PHP fixes the XML crash in its code. It doesn’t reach a release for four weeks.

  7. PHP advises switching off session upload progress until the next release, to close “an unlikely RCE vector”.

  8. Another report: server processes die as a request starts. Still open.

  9. Clayton says patching PHP took down the NSA’s network.

  10. PHP 8.5.13 includes the XML fix, along with 17 new security advisories. Two of the seven critical ones fix holes that October’s fix didn’t fully close, and another is in the upload progress feature PHP told people to switch off.

Something the fix broke. Times are UTC.

Clayton blames patching for the NSA’s outage, but nobody has said what broke there, so we can’t say any of that is what hit it.

Some of it is panic

Josh Bressers at VulnCheck told Reuters there have been 100,000 publicly disclosed vulnerabilities this year, and 1,363 are known to have been exploited. Dave Kennedy blames researchers and vendors who hype bugs that are hard to exploit in practice, and Kevin Beaumont told people to patch a much-hyped Java bug “as you normally do”.

Dave KennedyFounder of TrustedSec

It creates so much chaos inside of companies that have no idea how to deal with this. …

I’m not dismissing the criticality of latching on these - however it doesn’t require IR, all hands on deck, must patch tomorrow - which is probably why we see the mass outages we are experiencing.

19 Nov on X

Florian Roth at Nextron Systems checked the nginx bugs and found the widely quoted “98.9%” of affected servers only counted version numbers. The bugs need HTTP/2 switched on, which nginx doesn’t do by default. PHP said the day before its 19 November release that the critical bugs need “non-default configuration or uncommon setups”.

The FAA’s router wasn’t a security patch, though Clayton counts upgrades in with patching. CISA’s three-day deadlines only bind civilian agencies, not the NSA or any company. Microsoft hasn’t said what its July fix was for, and Chase has only said it was a software bug. Apart from the NSA, no affected organisation has publicly blamed patching, though bank executives reportedly did in private to Congress. Microsoft’s final report is due by 20 November and the FAA expects to give its root cause this week, so some of this could change.

Locking the doors

For decades we’ve been living with all the doors unlocked, and now we’ve started locking them.

Once everything’s been patched to a higher level, and companies have better systems for patching all the time, we think the outages will stop and everything will be more secure for it. The future is full of agents constantly checking and patching, where in the past you could do it every now and then and be fine.

David Agranovich gave two reasons it should get better.

David AgranovichFormer director for intelligence, White House NSC

Eventually, this may get better because (1) attack surface narrows as vulns are closed and new, cleaner code is shipped (with AI help) and (2) AI-produced code gets better and is inherently less vulnerable.

18 Nov on X

The White House official at the 18 November briefing said “it’s good the American people are focused on the need to upgrade our legacy systems.”

It’ll take a while. Clayton told Congress to expect more outages “as people are patching, and as people are upgrading systems”.

Phil Webb, CEO and co-founder

Phil runs client work and the commercial side of Vu Agency and Raq.com. With each client, he works out whether Raq.com covers the job or they need custom software.

Phil on LinkedIn

More from the blog

All posts
Message us on WhatsApp