Rackspace hosted Exchange suffered a devastating blackout starting December 2, 2022 and is still ongoing since 12:37 AM December 4th. Initially referred to as connectivity and login problems, the guidance was ultimately upgraded to announce that they were dealing with a security incident.
Rackspace Hosted Exchange Issues
The Rackspace system went down in the early morning hours of December 2, 2022. Initially there was no word from Rackspace about what the issue was, much less an ETA of when it would be resolved.
Clients on Buy Twitter Verified reported that Rackspace was not responding to support emails.
This has actually been quite the day with #Rackspace. Every hosted exchange customer has been down for 14 hours or two. Assistance isn’t reading/responding to tickets. Updates are unhelpful.
I am concerned now that they succumbed to something bad like the ProxyNotShell PoC hack. https://t.co/jchKsAO3Z7
— Joe Sinkwitz (@CygnusSEO) December 2, 2022
A Rackspace customer privately messaged me over social media on Friday to relate their experience:
“All hosted Exchange customers down over the past 16 hours.
Not sure how many companies that is, however it’s substantial.
They’re serving a 554 long hold-up bounce so individuals emailing in aren’t aware of the bounce for a number of hours.”
The main Rackspace status page used a running update of the failure however the initial posts had no information besides there was an outage and it was being investigated.
The first official update was on December second at 2:49 AM:
“We are examining a problem that is impacting our Hosted Exchange environments. More information will be published as they appear.”
Thirteen minutes later Rackspace began calling it a “connection concern.”
“We are examining reports of connection problems to our Exchange environments.
Users might experience a mistake upon accessing the Outlook Web App (Webmail) and syncing their e-mail customer(s).”
By 6:36 AM the Rackspace updates described the continuous problem as “connectivity and login concerns” then later on that afternoon at 1:54 PM Rackspace revealed they were still in the “examination stage” of the interruption, still attempting to find out what failed.
And they were still calling it “connectivity and login issues” in their Cloud Workplace environments at 4:51 PM that afternoon.
Rackspace Recommends Moving to Microsoft 365
4 hours later on Rackspace referred to the scenario as a “significant failure”and started offering their customers totally free Microsoft Exchange Strategy 1 licenses on Microsoft 365 as a workaround until they comprehended the problem and could bring the system back online.
The official guidance specified:
“We experienced a considerable failure in our Hosted Exchange environment. We proactively closed down the environment to prevent any further problems while we continue work to restore service. As we continue to resolve the root cause of the issue, we have an alternate solution that will re-activate your capability to send out and get e-mails.
At no cost to you, we will be supplying you access to Microsoft Exchange Plan 1 licenses on Microsoft 365 up until more notification.”
Rackspace Hosted Exchange Security Incident
It was not until almost 24 hr later on at 1:57 AM on December 3rd that Rackspace officially revealed that their hosted Exchange service was struggling with a security occurrence.
The statement further revealed that the Rackspace specialists had actually powered down and detached the Exchange environment.
“After further analysis, we have actually figured out that this is a security occurrence.
The recognized impact is separated to a part of our Hosted Exchange platform. We are taking needed actions to assess and safeguard our environments.”
Twelve hours later that afternoon they updated the status page with more info that their security team and outdoors experts were still dealing with solving the interruption.
Was Rackspace Service Affected by a Vulnerability?
Rackspace has actually not released details of the security occasion.
A security occasion normally involves a vulnerability and there are 2 severe vulnerabilities currently in the wile that were patched in November 2022.
These are the 2 most existing vulnerabilities:
Microsoft Exchange Server Server-Side Demand Forgery (SSRF) Vulnerability
A Server Side Request Forgery (SSRF) attack permits a hacker to read and alter information on the server.
Microsoft Exchange Server Remote Code Execution Vulnerability
A Remote Code Execution Vulnerability is one in which an enemy is able to run destructive code on a server.
An advisory released in October 2022 described the impact of the vulnerabilities:
“A verified remote opponent can perform SSRF attacks to escalate opportunities and execute arbtirary PowerShell code on vulnerable Microsoft Exchange servers.
As the attack is targeted against Microsoft Exchange Mailbox server, the assaulter can possibly gain access to other resources via lateral movement into Exchange and Active Directory site environments.”
The Rackspace outage updates have not suggested what the specific issue was, only that it was a security occurrence.
The most current status upgrade as of December fourth mentioned that the service is still down and consumers are encouraged to move to the Microsoft 365 service.
Rackspace posted the following on December 4, 2022 at 12:37 AM:
“We continue to make progress in attending to the event. The schedule of your service and security of your data is of high importance.
We have committed comprehensive internal resources and engaged world-class external proficiency in our efforts to decrease unfavorable effects to clients.”
It’s possible that the above kept in mind vulnerabilities relate to the security incident impacting the Rackspace Hosted Exchange service.
There has actually been no announcement of whether client information has actually been jeopardized. This occasion is still continuous.
Included image by Best SMM Panel/Orn Rin