Codecov Bash Uploader Dev Tool Compromised in Supply Chain Hack
Security response professionals are scrambling to measure the fallout from a software supply chain compromise of Codecov Bash Uploader that went undetected since January and exposed sensitive secrets like tokens, keys and credentials from organizations around the world.
The hack occurred four months ago but was only discovered in the wild by a Codecov customer on the morning of April 1, 2021, the company said in a note acknowledging the severity of the breach.
“On Thursday, April 1, 2021, we learned that someone had gained unauthorized access to our Bash Uploader script and modified it without our permission. The actor gained access because of an error in Codecov’s Docker image creation process that allowed the actor to extract the credential required to modify our Bash Uploader script,” Codecov said.
The company said a forensic investigation determined that the attacks began in late January and went undetected until a customer noticed a discrepancy between the shasum on Github and the shasum calculated from the downloaded Bash Uploader.
“Based upon the forensic investigation results to date, it appears that there was periodic, unauthorized access to a Google Cloud Storage (GCS) key beginning January 31, 2021, which allowed a malicious third-party to alter a version of our bash uploader script to potentially export information subject to continuous integration (CI) to a third-party server. Codecov secured and remediated the script April 1, 2021.”
Codecov said the breach allowed the attackers to export information stored in its users’ continuous integration (CI) environments. This information was then sent to a third-party server outside of Codecov’s infrastructure,” the company warned.
Codecov’s Bash Uploader is also used in several uploaders — Codecov-actions uploader for Github, the Codecov CircleCl Orb, and the Codecov Bitrise Step — and the company says these uploaders were also impacted by the breach.
According to Codecov, the altered version of the Bash Uploader script could potentially affect:
● Any credentials, tokens, or keys that our customers were passing through their CI runner that would be accessible when the Bash Uploader script was executed.
● Any services, datastores, and application code that could be accessed with these credentials, tokens, or keys.
● The git remote information (URL of the origin repository) of repositories using the Bash Uploaders to upload coverage to Codecov in CI.
Codecov chief executive Jerrod Engelberg said the company has rotated all relevant internal credentials, including the key used to facilitate the modification of the Bash Uploader; and conducted audits to determine where and how the key was accessible.
Engelberg said the company is setting up monitoring and auditing tools to ensure that this kind of unintended change cannot occur to the Bash Uploader again; and is also working with the hosting provider of the third-party server to ensure the malicious web server was properly decommissioned.
“We continually monitor our network and systems for unusual activity, but Codecov, like any other company, is not immune to this type of event. We are also working to further enhance security so we can stay ahead of this type of activity, including reinforcing our security tools, policies, and procedures,” Engelberg added.
Codecov is strongly encouraging software development teams to “immediately re-roll all of your credentials, tokens, or keys located in the environment variables in your CI process.”
“This is a significant compromise. [We are now] working to figure out blast radius, given that this was exfiltrating credentials for several months,” a Silicon Valley security response professional told SecurityWeek.
Codecov is considered the vendor of choice for measuring code coverage in the tech industry. The company’s tools help developers understand and measure lines of codes executed by a test suite and is widely deployed in big tech development pipelines
The company claims that more than 29,000 enterprises use its code coverage insights to check code quality and maintain code coverage. Codecov did not say how many customers were impacted or had data stolen in the incident.