Microsoft: China stole secret key that unlocked US govt email from crash debug dump
Mistakes were made, lessons learned, stuff now fixed, says Windows maker
Remember that internal super-secret Microsoft security key that China stole and used to break into US government email accounts back in July?
The Windows giant has, in its own words, today described how the Chinese spy team it tracks as Storm-0558 obtained that golden cryptographic key, which was then used to break into Uncle Sam's Outlook web mail accounts. The cyber-snoops stole the consumer key from a software crash dump which, as Microsoft was good enough to admit, should have been redacted and not have included the digital key in the first place.
Microsoft published these findings in a write-up titled "results of major technical investigations for Storm-0558 key acquisition" on Wednesday, and the tl;dr version is: mistakes were made, and Redmond assures us it has made changes to prevent them from happening again.
The IT titan keeps secrets like its consumer keys – which in the wrong hands can be used to create forged authentication tokens and log into other people's Microsoft accounts – in an isolated production network away from its day-to-day corporate network. As the biz put it:
Microsoft maintains a highly isolated and restricted production environment. Controls for Microsoft employee access to production infrastructure include background checks, dedicated accounts, secure access workstations, and multi-factor authentication using hardware token devices. Controls in this environment also prevent the use of email, conferencing, web research and other collaboration tools which can lead to common account compromise vectors such as malware infections or phishing, as well as restricting access to systems and data using Just in Time and Just Enough Access policies.
Be as that may, in April 2021 when software within that isolated environment that handled the consumer key broke down, a snapshot of the program was made. That crash dump, it turned out, contained a copy of that secret key.
"A race condition allowed the key to be present in the crash dump (this issue has been corrected)," the Microsoft Security Response Center explained in its detailed write-up.
"The key material's presence in the crash dump was not detected by our systems (this issue has been corrected)," it added.
Ideally, you don't want sensitive things like full secret cryptographic keys in your crash dumps, and these snapshots were expected to be automatically redacted. That said, you might expect the key to stay within something like a dedicated hardware module and not find its way into running production software, but hey, what do us vultures know?
If the dump had stayed within the production network, it wouldn't have necessarily been the end of the world: if an intruder could access the dump in prod, they could perhaps access a lot of other things anyway. However, as per Microsoft's "standard debugging process," workers moved the crash dump from the isolated production network into a debugging environment on the internet-connected corporate network.
Even after the move, credential scanning systems did not detect the key (Redmond also says "this issue has been corrected") and while the key was sitting in the crash dump on the general IT network, Storm-0558 compromised a Microsoft engineer's corporate account and swiped the digital key from the snapshot.
"Due to log retention policies, we don't have logs with specific evidence of this exfiltration by this actor, but this was the most probable mechanism by which the actor acquired the key," according to Redmond.
Wait, a consumer key signed tokens for enterprise email?
Back to the consumer key being used to access enterprise email: Microsoft explained this dates back to September 2018, when it began offering a converged API endpoint that applications could use to authenticate users, whether those users were within an enterprise or individual consumers.
At the time, Redmond updated its documentation and software libraries so that application developers could use this endpoint to ultimately provide a single-sign-on interface. Crucially, Microsoft did not provide enough automatic checks in those libraries to ensure that, say, an enterprise user wouldn't be validated using a consumer key, another issue it said has now been corrected.
When Microsoft's own engineers started using the endpoint in 2022 for its email system products, they didn't realize these checks weren't in place, either, we're told.
"Thus, the mail system would accept a request for enterprise email using a security token signed with the consumer key (this issue has been corrected using the updated libraries)," the postmortem report stated.
- Stolen Microsoft key may have opened up a lot more than US govt email inboxes
- Microsoft admits unauthorized access to Exchange Online, blames Chinese gang
- You patched yet? Years-old Microsoft security holes still hot targets for cyber-crooks
- Microsoft calls time on ancient TLS in Windows, breaking own stuff in the process
This also appears to validates earlier research by Wiz, an infosec biz founded by former Microsoft cloud security engineers.
About a week after Beijing's snoops used the stolen key to log into Microsoft cloud email accounts used by US government officials, including US Commerce Secretary Gina Raimondo and other State and Commerce Department officials, Wiz research boss Shir Tamari said the skeleton key "was more powerful than it may have seemed" and could have been used to breach more than just Outlook and Exchange Online accounts.
"Our researchers concluded that the compromised MSA key could have allowed the threat actor to forge access tokens for multiple types of Azure Active Directory applications," Tamari wrote in research published July 21.
Following the break-ins, and with a little push in the right direction from the US government, Redmond also agreed to provide all customers with free access to cloud security logs, but not until September this year. ®