Skip to main content
Success
All posts

Why Outlookwas downfor a day

From 4:20pm on 12 November, most Microsoft 365 users couldn’t reliably get into their email for almost 24 hours. Microsoft says the cause was a bug in an update it started rolling out in July.

  • Gateway servers already out (14%)
  • Not responding at the worst point (more than 80%)

At 4:03pm on Thursday 12 November, Microsoft’s monitoring noticed people failing to connect to Outlook. By 4:20pm it was an official incident, and for most people on Microsoft 365 it stayed that way, on and off, until 4:10pm on Friday. How bad it was depended on where you were, how you connected and when.

Microsoft’s preliminary post-incident report isn’t public. Admins can read it in the Microsoft 365 admin centre under Service health, filed under EX1550543 and MO1550821, and we’ve read it in ours, along with the 80 or so updates Microsoft posted while it was going on.

23h 50m

from the first alert to Microsoft saying nobody was still affected

80%+

of the gateway servers not responding at the worst point

7m+

Downdetector reports, the most for any outage in the past year

July

when the change with the bug in it started rolling out

A bug from July

A lot of Microsoft 365 sits behind what Microsoft calls a networking gateway, a huge fleet of Windows servers that take your connection and pass it on to the right service. Exchange Online, which is what Outlook talks to for your email, goes through it.

In July, Microsoft started rolling out a fix written specifically for those gateway servers. It reached every server over several weeks, and nothing went wrong for more than three months.

The fix had a bug in it. Under certain loads, parts of an internal Windows process got stuck competing for the same lock, which used up the server’s CPU and memory until it stopped responding. It only affects these servers, not Windows in general.

Its load testing didn’t catch it because of the machines it tested on. In the report’s words, the problem “is worsened on machines with more cores and newer hardware; whereas our load testing tended to focus on weaker hardware with fewer cores.”

Three things at once

Microsoft runs two separate gateway systems, built differently on purpose, so that services can move from one to the other when something goes wrong. On Thursday the bug went off because of three things happening together.

A bug from July

A fix written just for the gateway servers made them lock up under certain loads. It had been running worldwide for more than three months without a problem.

Teams moved over

At 1:07am, about 15 hours earlier, Teams failed over from Microsoft’s other gateway onto the one Exchange uses. More load, and a different mix of traffic.

14% already out

Nearly 14% of the gateway’s servers were out of service after an unrelated problem the day before. On its own that didn’t matter.

4:20pm Thursday, peak time in the USExchange Online alert, EX1550543

Between 1:07am and 4pm the extra load built up slowly, odd servers failed in some regions, and automation stepped in each time. Then US traffic hit its daily peak, requests started piling up, and servers stopped responding.

How it spread

When a server stops responding, Microsoft’s traffic management moves its users onto healthy ones. That happens all the time and it’s normally why you never notice anything.

This time it made things worse. Every server that froze pushed its users onto the rest, which made them more likely to hit the same bug. The report describes it as “a cascading failure from server to server as the same amount of load was spread across fewer servers.”

Wednesday

One server in seven is out after an unrelated fault. The rest have room to spare.

1:07am Thursday

Teams traffic arrives. Every server is busier, and the mix of traffic changes.

4:20pm Thursday

US peak. Servers hit the bug and freeze, and their users get moved onto the rest.

Within two hours

The extra load tips the others over too. More than 80% stop responding.

In less than two hours the bug had worked its way across the fleet. At the lowest point, more than 80% of the gateway servers weren’t responding.

It shouldn’t have spread worldwide. The gateway is split into thousands of what Microsoft calls capacity units, grouped into regions, with bulkheads between them so that a failure in one place stays there. They didn’t hold, “due to a separate issue” that the preliminary report doesn’t explain yet.

If you weren’t an admin, you heard about it late. Microsoft’s first post on its public status account on X went up at 5:54pm, 94 minutes after the alert, and only mentioned “the Americas”. Thirteen minutes later it said “any region”.

Some services managed to get out of the way. Teams, OneDrive, SharePoint and the Office apps moved off the broken gateway, which is why Teams chats and meetings carried on working for most people. Exchange Online couldn’t. Microsoft says it can’t fail out of that gateway “for scale reasons”.

Why it took a day

Microsoft’s updates show engineers hunting for a recent change. By 7:30pm on Thursday they’d paused every deployment to go through recent changes, and at 1am on Friday they began “reverting a recent network configuration change that correlates with the start of impact.” Even the closing update on Saturday blamed “an earlier network capacity upgrade” and “a recent network change”. The July bug only came up in the report two days later.

Every time engineers got servers back up, the traffic that had been queueing for hours landed on them at once and they froze again. Just after midnight the status account said “the servers we returned to service can become unresponsive again once they receive traffic”. Microsoft’s Charles Lamanna later called it “cascading failures + thundering herds at our network perimeter”, a thundering herd being what engineers call everyone piling back in the moment something comes up.

Someone on Y Combinator’s forum had pieced most of that together from the status updates at 5:16am on Friday, two hours before Microsoft started rolling out its fix: “Saturated servers fail health checks and get pulled, so the survivors take more load and fail too… That is why restarted servers die as soon as traffic returns.”

Microsoft’s automated recovery wasn’t designed for this many servers being down at the same time, and at 1:53am on Friday engineers switched off some of the automatic redirection that usually hides a failing server. And because the bug was months old, the usual ways of undoing a bad change, rolling it back or switching it off with a feature flag, couldn’t do it quickly.

So Microsoft built an update that put the old code back, then worked through the fleet in batches, sometimes more than half a region at once. Each batch was taken out of service, patched, restarted and put back.

Thursday 12 November

  1. Automation moves Teams traffic onto the gateway servers that Exchange Online uses.

  2. Monitoring picks up a rise in people failing to connect.

  3. High-priority alert for Exchange Online. Microsoft opens EX1550543 in the admin centre. The outage officially starts.

  4. Microsoft’s first post on its public status account on X. It mentions only the Americas.

  5. The same account says users in any region may be affected.

  6. A second incident, MO1550821, adds Teams, Copilot, Defender and more.

  7. Engineers work out the extra load isn’t extra traffic. It’s the same traffic on fewer working servers.

  8. A fix is tried, but many servers are now unresponsive. Restarts begin.

  9. Servers brought back stay healthy until traffic reaches them, then freeze again.

Friday 13 November

  1. The process that sets up secure connections is found to be overwhelmed.

  2. Machines are stuck in bad states and won’t take changes. More virtual machines are added.

  3. An update reverting the July change goes onto part of the fleet.

  4. It holds. The rollout widens, starting with regions in their working day.

  5. Email retry window extended to 48 hours so delayed messages don’t bounce.

  6. Europe back to healthy levels.

  7. Outlook on the web working normally worldwide.

  8. No active impact. Mail backlogs still clearing.

Saturday 14 November

  1. Final updates posted after a night of monitoring.

UK times. Penn has a public copy of Microsoft’s updates.

What else broke

Outlook got the headlines, but plenty of Microsoft 365 leans on Exchange Online, and those parts broke too. This is how Microsoft described it.

Exchange Online

Down

Most people couldn’t open their mailbox in Outlook, Outlook on the web or phone mail apps. Mail was delayed.

Copilot

Partly

Copilot Chat might not load or answer, and couldn’t reach email, calendars or files.

Microsoft Graph

Partly

API requests for mail, calendars and contacts timed out, which also broke some Teams management.

Power Automate and Power Apps

Partly

Flows and connectors that rely on Graph failed on and off, and some people couldn’t sign in to the Power Apps mobile app.

Teams

Mostly working

Calls, meetings and chats were fine. Calendars, scheduling, live events and new private or shared channels weren’t.

SharePoint and OneDrive

Mostly working

Files opened. Sharing with people outside the company had problems.

Purview

Partly

Data loss prevention, sensitivity labels, compliance checks, audit searches, eDiscovery and opening protected content.

Defender

Partly

Portal pages might not load, and some email policy checks were delayed.

Intune

Partly

Device enrolment, check-ins and compliance checks failed on and off.

Power BI and Fabric

Partly

Exporting or editing reports with sensitivity labels failed.

Admin centres

Partly

Some Microsoft 365 admin centre pages wouldn’t load, nor would cloud update and policy tools for Office apps.

Windows 365

Partly

Only Cloud PCs set up with Purview Customer Key were hit. They couldn’t connect.

Anything that reads Microsoft 365 data through its API, Microsoft Graph, had a bad night too. That includes AI assistants. Anthropic says Claude’s Microsoft 365 connector and Claude for Outlook returned errors for many requests from 4:03pm on Thursday to 4:40pm on Friday, and OpenAI kept its own connector incident open until just after midnight on Friday night.

We saw it ourselves. One of our client systems reads a Microsoft 365 mailbox through Graph every 15 minutes to import reports. From 5:45pm on Thursday it got server errors and requests that hung for 30 seconds. Nothing was lost, but Friday’s reports arrived 68 minutes late.

Email wasn’t lost. Mail servers normally give up on a message they can’t deliver and bounce it back, so on Friday morning Microsoft stretched that window to 48 hours. Which meant Friday’s inbox looked a lot like one person on X predicted on Thursday night: “300 emails, 3 that matter, and 40 reply-alls asking if anyone else’s Outlook is down.”

The most serious result we’ve seen reported was at Phantasialand, a theme park near Cologne. On the Friday, the automatic dosing system for the pool at its Hotel Ling Bao developed a fault and pumped in too much chlorine. The safety system raised the alarm by email, and the email never arrived. WDR reports 15 guests were hurt and four went to hospital. Most of the comments on Y Combinator’s forum blamed the hotel for letting a safety alarm rely on one email.

Friday morning in the UK

It started at 4:20pm UK time, so for a lot of people here it really hit on Friday morning. NHS staff told the Guardian they first noticed problems in the early hours of Friday, and that most realised how serious it was from about 6am.

NHS.net Connect, which more than a million NHS staff use for email, meetings and files, runs on Microsoft. Some staff went back to paper records, and some said they couldn’t send urgent referrals for cancer and other serious conditions.

The Met, Greater Manchester Police and Police Scotland told people to phone rather than use online forms. At Minshull Street Crown Court in Manchester, Judge Maurice Greene said staff had been looking for a fax machine, and sent the jury home for the weekend. The Supreme Court said it wasn’t receiving emails.

MPs couldn’t get into their parliamentary accounts and Holyrood’s email was down. Ofsted couldn’t receive email, and the Bank of England and the FCA were both hit, though the Bank said “all critical operations” were running normally.

Not everyone minded. It got nicknamed Microsoft Bank Holiday Friday, someone asked Andy Burnham to make it an annual bank holiday, and one person couldn’t believe “microsoft outlook managed to get a four day week before i did”.

It wasn’t just the UK. Reuters reported that Australia’s spy agency and federal police were hit, and the governments of Singapore, Japan, Greece and Norway told AP some of their ministries had lost email too.

Downdetector had more than 7 million reports about Microsoft’s services by 4pm on Friday, from more than 90 countries, the most Ookla, which owns it, has seen for any outage in the past year. Ciaran Martin, who used to run the UK’s National Cyber Security Centre, told NBC it “looks like being one of the biggest outages of all time”.

“This was a technical issue not related to cyber or AI / agents.”

Microsoft’s final update on both incidents, early on Saturday 14 November. Lamanna had said much the same on X the night before: “This was not a cyberattack or an issue caused by AI tooling.”

What Microsoft has changed

The preliminary report lists five repairs, all marked as done. Most of the longer-term changes will be in the final report, which Microsoft says is due by Friday 20 November.

Mitigation / Repair Items

ActionStatus
Developed an update that returned the connection handling code path to the previously known good stateCompleted
Extended the email delivery window to ensure delayed messages weren’t returned as undeliverable and all messages that were queued for delivery during the event were processedCompleted
Added additional capacity as a proactive measure to help with recoveryCompleted
Additional monitoring for traffic-redirection patterns and connection-handling healthCompleted
Implemented guardrails for traffic distribution to avoid cascading impactCompleted
A comprehensive list of repair items will be delivered as part of the final PIR20 Nov 2026

Preliminary Post Incident Report, EX1550543/MO1550821, 15 November 2026

If you’re on Microsoft 365

These are worth checking before the next one.

  • Claim the service credit, but don’t count on a set amount. Microsoft’s service level agreement promises Exchange Online 99.9% a month and pays back 25% of the month’s fee below that, 50% below 99% and all of it below 95%. A whole day locked out would be about 96.7%, so 50%, but it’s worked out from your own users’ downtime and only for the Exchange part of a bundled plan. It isn’t automatic. If you buy through a Microsoft partner, they claim it for you within a month of the end of that billing month. Office Watch has a guide.
  • Find anything that reads email or calendars automatically, whether that’s Power Automate, a Graph integration or an AI assistant connected to your mailbox. Check it retries on its own and tells someone if it’s still failing hours later. And if an alarm only reaches people by email, give it a second way to get through.
  • Keep a way to reach staff and clients that doesn’t depend on Outlook, with phone numbers somewhere other than your Outlook contacts. Teams chat kept working this time, but only because Teams could move off the broken gateway.
  • Be wary of emails about the outage. Microsoft Threat Intelligence found phishing emails posing as service health notices and quoting the real incident number, with subjects like “Outage -” and a six-character code, claiming mail held during the outage would be deleted within 24 hours unless you clicked to restore it. Check anything like that in the admin centre instead of clicking.

Phil Webb, CEO and co-founder

Phil runs client work and the commercial side of Vu Agency and Raq.com. With each client, he works out whether Raq.com covers the job or they need custom software.

Phil on LinkedIn

More from the blog

All posts
Message us on WhatsApp