Big Russian hack used a technique experts had warned about for years. Why wasn’t the U.S. government ready?

Cybersecurity researchers had warned for years that such an attack was possible. Those from one firm, FireEye, even released hacking tools in 2019 showing exactly how to do it — in hopes that the revelation would spur the widespread deployment of better defenses.

It didn’t.

Now there is urgent debate within cybersecurity circles about how best to respond to the hack, which was so extensive that experts describe it as historic.

Some are calling for stronger walls to keep out would-be intruders or better burglar alarms to alert network administrators that a hack had begun. Others, arguing that there’s no practical way to keep the most sophisticated hackers from breaking into important networks, say the smarter investment would be in building better tools for hunting and ejecting intruders once they inevitably get past security perimeters.

Meanwhile, questions remain about why this surge of corrective action didn’t happen earlier for a type of hack that had been discussed for years within cybersecurity circles and whether, even now, the potential solutions are being deployed widely enough to prevent future catastrophes.

Two months after the hack was discovered in December, cybersecurity researchers say spies are probably still active in some of the hundreds of breached networks. Victims included the departments of State, Treasury, Homeland Security, Energy and Commerce, and the National Institutes of Health and the National Nuclear Security Administration. Also penetrated were private companies in the consulting, technology, telecom, and oil and gas companies in North America, Europe, Asia and the Middle East, as well as FireEye itself, which first reported the attack on Dec. 8.

The Russians used a variety of sophisticated tricks to penetrate the networks in last year’s attack. But once inside, they often manipulated a piece of Microsoft software, Active Directory Federation Services, that vouches for the identities of authorized users by issuing digital identity documents called “SAML tokens.” An Israeli researcher had first described this technique, dubbed a “Golden SAML Attack,” in 2017, but it had not been seen in a major network intrusion until now, experts say.

Such systems for authenticating users are essential to securing the cloud services used widely by government agencies, corporations, hospitals, universities and most other places where people collaborate across long distances. And the ability to forge SAML tokens lets hackers roam widely among these cloud-based services, while also minimizing the chances of getting quickly caught.

“All of this outward security doesn’t mean squat if you don’t have this one thing locked up,” said Matthew D. Green, a Johns Hopkins cybersecurity and cryptology expert. “This is crazy.”

Authenticated SAML (rhymes with “camel”) tokens let intruders move easily among the computer systems affiliated with an organization, even if the individual elements are run by different companies, such as Microsoft, Amazon Web Services or Dropbox. Hackers can present these tokens as they seek access to different troves of valuable data — email, document repositories, payroll systems — while sidestepping common defensive measures, such as strong passwords and two-factor authentication.

There are possible protections against a Golden SAML Attack, including securing the encryption keys that create the tokens in their own, well-defended piece of hardware, or sharply limiting who has high-level access to the computers authorized to issue tokens. Alerts warning of unusual behavior might help defenders act more quickly, and more extensive logging could help the detective work after signs of trouble are detected.

Former National Security Agency hacker Jake Williams said his security consultancy has been helping clients respond to the recent Russian attack. But even now, it’s not entirely clear to him which defenses are best suited to prevent a repeat, given the sophistication of the attackers, which U.S. officials have said were from the SVR, Russian’s foreign-intelligence service. He favors bolstering systems for detecting intruders once they’re inside.

“We are not going to keep a nation-state attacker who has targeted you out,” said Williams, president of Rendition Infosec. “They are going to outfox you.”

The question then becomes: How best to keep a network intrusion from becoming a catastrophe?

As this debate plays out within the cybersecurity community, Sen. Ron Wyden (D-Ore.), a member of the Senate Intelligence Committee, asked Microsoft and FireEye last month to explain how a security weakness publicized years ago was not addressed before the Russians took advantage of it. Microsoft released a tool to help detect such attacks less than two weeks after the Russian hack was publicly revealed.

“The American people deserve to know why hackers were able to steal encryption keys from the U.S. government without anyone noticing,” Wyden said in a statement to The Washington Post. “I want to know why Microsoft didn’t provide its customers with tools to better protect and detect the theft of encryption keys, and why government agencies failed to deploy their own defenses. I’m also interested in what steps FireEye took to warn Microsoft, its customers and the U.S. government about a vulnerability it knew about nearly two years ago.”

Both companies defended their handling of the Golden SAML Attack in replies to Wyden — FireEye by letter, Microsoft by video call — according to a Wyden aide who spoke on the condition of anonymity to discuss communications not yet made public.

In comments to The Post, the companies noted the multiple weaknesses the Russians exploited in their intrusions and also the difficulty creating effective defenses against hackers who already have penetrated networks so deeply that they can issue their own SAML tokens. Both companies said better overall security practices are key to defending against this and other attacks, ideally before the initial intrusions succeed.

John Lambert, the head of Microsoft’s Threat Intelligence Center, said in an interview that the company long has recommended security measures that might have thwarted the Russians, such as stand-alone hardware to guard encryption keys, and that handling the issuance of SAML tokens through a cloud service, such as Microsoft’s Azure, would offer increased protection and potentially make hacks easier to detect.

He also said that some of the measures now under discussion among independent cybersecurity experts — such as installing the hardware modules Microsoft recommends for protecting encryption keys — would make a Golden SAML Attack harder to execute in the future.

“Defending identity has always been foundational,” Lambert said. “I think if you go back to any set of attacks at any point of time in the past, compromise of identities, abuse of identities, has always been a common element. … Securing identities and the secrets that underpin them have always been important.”

FireEye’s role in publicizing the Golden SAML Attack was highlighted in a Microsoft post that specifically cited one of the hacking tools FireEye released in 2019, ADFSDump. The post said that Microsoft’s Defender software could, as of Dec. 20, send alerts when it detected ADFSDump and called it “the initial tool used” in the Russian hacks.

Microsoft later revised this characterization after The Post questioned FireEye about it, saying that the Russians used a hacking tool resembling ADFSDump but that it was unclear whether ADFSDump itself was the one. The company’s updated version of the post removed the reference to the FireEye tool, saying instead that Microsoft’s Defender software now had an alert to “detect techniques used to obtain the information needed in order to generate security tokens,” as happened in last year’s Russian attack.

FireEye acknowledged that its engineers had raised alarm about Golden SAML Attacks and released a pair of hacking tools to exploit it during a security conference in Germany in March 2019. But the company said it found no evidence that these tools were used by the Russians, though it couldn’t rule out the possibility. The goal of releasing such tools is to help “red teams” of cybersecurity researchers probe networks for flaws that can be corrected before malicious hackers exploit them.

“FireEye develops red team tools to help improve enterprise cybersecurity by demonstrating the impacts of successful attacks and by showing the defenders … how to counter them in an operational environment,” said Dan Wire, vice president of global communications for FireEye. “Like many security companies, we have an internal process for responsibly releasing tools, and we review each release on a case-by-case basis.”

The U.S. government response to the Russian hack, meanwhile, came under fire Tuesday when the heads of the Senate Intelligence Committee, Chairman Mark R. Warner (D-Va.) and Vice Chairman Marco Rubio (R-Fla.), sent a letter to the heads of the FBI, National Security Agency and other federal agencies demanding the appointment of “a clear leader” to coordinate the response.

“The federal government’s response so far has lacked the leadership and coordination warranted by a significant cyber event, and we have little confidence that we are on the shortest path to recovery,” they wrote.

Russian spies began their attack by hacking SolarWinds, a Texas-based maker of network-monitoring software, and slipping what security experts call a “Trojan horse” into the networks of the company’s many clients during routine software updates. Once inside, the hackers roamed unchecked for months and might have stayed even longer had FireEye not found them within their own systems in December. That discovery triggered detection of the much wider, more troubling federal hack days later.

Many experienced network defenders point to the introduction of the Solar Winds trojan — in what’s called a “supply-chain attack” — as the problem most urgently demanding attention because federal government systems rely on software produced by many private companies, each of which offers targets for malicious hackers. Once they get inside, experts say, there are numerous options, beyond just a Golden SAML Attack, to exploit a network’s systems for verifying user identities.

“There are literally dozens upon dozens of ways,” said Dmitri Alperovitch, who co-founded cybersecurity firm CrowdStrike and now is executive chairman of Silverado Policy Accelerator, a think tank. “No one can possibly defend against all of them. … The idea that we should be chasing every single attack vector is a wrongheaded approach.”

Shaked Reiner, an Israeli cybersecurity expert who described the Golden SAML Attack in a 2017 blog post, said the method offers important advantages to hackers — namely its potential to enable unusually wide-ranging, long-lasting and hard-to-detect intrusions that may merit more robust defenses.

The initial blog post, made on the site of his employer, CyberArk Labs, initially generated only modest attention. News of the Russian hack, three years later, changed that. The National Security Agency cited Reiner’s post in its advisory on how to detect such intrusions on Dec. 17.

“Right away, we understood. This is what we were talking about,” Reiner said.

He added that hackers deploying the Golden SAML Attack “can pretty much impersonate any user in a network. … Detecting this type of attack can be extremely difficult.”

Some experts, including Green at Johns Hopkins, argue that sensitive government networks should invest in computer equipment called “hardware secure modules” that would house the encryption keys used to issue SAML tokens, making them almost impossible to steal. This equipment is expensive, ranging from tens to hundreds of thousands of dollars, and can add significant complexity to the operation of cloud-computing networks — factors that have been barriers to their widespread adoption.

Another approach would be to specify a small number of computers — perhaps ones at the physical desks of system administrators — that can gain high-level access to the identity-management software itself. Even a skilled hacker, for example, would find it much harder to execute a Golden SAML Attack from Moscow if only a handful of computers were vulnerable to such manipulation from afar. Even then, key computers could be left disconnected from the Internet, adding more barriers to hackers operating remotely.

Some other experts, however, say that even without actually stealing the encryption keys for issuing SAML tokens, hackers can still find ways to manipulate network identities in ways that allow them to expand and prolong intrusions.

Williams, of Rendition Infosec, said, “I agree that Microsoft could have done a better job of detecting any number of active-directory weaknesses or the exploitation of those weaknesses.”

But he added that more aggressive action by Microsoft, FireEye or others would have been unlikely to thwart the Russians, given their skills and resources.

“I’m confident that wouldn’t have changed the outcome here,” Williams said.

The most viable solution for the future, some experts say, may be in better alarms to rapidly alert defenders to suspicious behavior, along with more extensive network logging of network activities — preferably activated by default — to assist the detective work after hacks inevitably occur.

CORRECTION: A previous version of this story said incorrectly that Sen. Ron Wyden sent letters to FireEye and Microsoft last month asking for answers related to the Russian attack. But in fact only FireEye received a letter. The communication with Microsoft was oral.