SolarWinds Outlines ‘Triple Build’ Software Development Model to Secure Supply Chain
When FireEye (now Mandiant) disclosed the SolarWinds breach in December 2020, the security world was forced to accept the reality that given the motivation, time and resources, an advanced attacker can breach any organization. And if the breached organization is part of an important supply chain, the potential damage could be devastating.
In the SolarWinds incident, up to 18,000 companies could have received the malware injected into the SolarWinds software. Not all could have been affected. Many of these ‘victims’ did not install the infected version, and many others did so on servers with no internet connectivity. Of those companies that did receive the Nobelium Sunburst malware, only a relatively small number received any follow up attention from the hackers. In the final analysis, fewer than 100 victims’ servers communicated with the hackers. These were important companies and government offices that would be of interest to a foreign adversary state.
SolarWinds has attempted to be open and transparent about the incident and its effects. Its most recent ‘investigative update’ on the incident was published on May 7, 2021. In this analysis, SolarWinds outlines a ‘secure by design’ future, and adds, “We hope sharing of our learnings about this attack serves our customers—as well as the broader IT industry…”
The company has now published a formal description of its progress in securing its systems and software products into the future – and goes further by suggesting that its new software development methodology should be adopted by all third-party suppliers of software and demanded by all companies from their third-party suppliers.
The document — Becoming Secure by Design with SolarWinds — describes a new triple build model designed to ensure that software builds can never again be compromised in the way that Nobelium injected the Sunburst malware into SolarWinds Orion software. It also includes a summary of the security controls implemented to provide resiliency to the SolarWinds environment.
The company has spent approximately $21 million on this by the end of Q2, 2021. SolarWinds CISO Tim Brown told SecurityWeek that a key resiliency purpose for him was to increase visibility to decrease any possible dwell time. In the Nobelium incident the precise dwell time is unknown, but the initial breach occurred prior to October 2019 and was undoubtedly used for reconnaissance.
“They came back in October 2019,” said Brown, “and did a test run with no malicious code. It wasn’t until March 2020 that they inserted the malicious code.” And it wasn’t until December 2020 that Nobelium was discovered.
It is this extended dwell time that has focused Brown’s attention on visibility. “At the time,” he said, “I just had my own internal SOC. Now I have three. I have the CrowdStrike security operation center with my workstations and servers all feeding into it. That information as well as firewall, O365, Azure, AWS, and application logs also goes to SecureWorks. And then my SOC is still present as the third one. The purpose is to increase visibility across the entire environment so that I quickly see any inkling of malicious behavior, anywhere in the systems.”
But it is the triple build software development method that is the primary focus of the SolarWinds document. This has been introduced in three phases. Phase I, from January 2021, introduced dual build verification into the process. “This enabled us to take compiled binaries back to the source code files with the associated hashes and compare those hashes with the files in source control, thus ensuring no alteration or insertion occurred within the build pipeline,” according to the document.
Phase II incorporates the build in an AWS environment and adds several security enhancements. Phase III encompasses the triple build, SLSA Level 4 compliant environment.
“Multiple builds require collusion from multiple people to affect the build environment,” said Brown. Is it enough? “More builds would be more secure, but three is realistic. An intern and a malicious insider would not be enough – it would require three highly trusted architects to subvert the software builds.”
SolarWinds hopes that the build method it has developed can become standard practice for software developer firms. The company goes further by urging industry to demand proof of a similar level of build security from their software suppliers.
“Our priority,” the Solarwinds CISO told SecurityWeek, “was first to help our customers – to make sure they had the right information and could get through the incident as well as possible.” SolarWinds did this through openness and transparency. But then it wanted to help the wider industry.
“We don’t want this incident to be wasted,” Brown told SecurityWeek. “We can use what we’ve learnt to help the industry move forward. We want to be able to say, ‘Hey, this incident was an inflection point that has helped the industry become more resilient to attacks and make it harder for nation states and organized crime to perform their missions.’ If we can do that, the wider industry will ultimately benefit from the SolarWinds incident.”