Inside Toyota’s secret leak from a supply chain vulnerability

As the software supply chain has become an increasingly better target for attackers looking for ways to compromise enterprise systems, it has become common for corporations to take necessary precautions to block external threats from various common attack vectors, but they forget insiders.

Insider threats can be more dangerous to organizations than outside attackers, because it usually takes longer to detect the threat, giving the threat ample time to steal data and damage the integrity of a system. 

As a matter of fact, recent Apiiro research found that enterprise repositories are eight times more likely to publish secrets to publicly exposed repositories, meaning many organizations do not monitor their environment for potential internal developer threats. These threats are not always intentional, so developers are unaware of the potential of a data breach from storing secrets in code.

Secrets Insights 2022 Across the Software Supply Chain: https://apiiro.com/secrets-insights-2022/

Toyota recently suffered from a supply chain vulnerability when a third-party vendor stored access keys to an internal database in developer code. The vendor unknowingly stored secrets in code that handled one of Toyota’s critical customer systems, T-Connect. Hundreds of thousands of customer emails and control numbers were available to anyone who discovered the access keys, whether intentionally looking for secrets in code or accidentally discovered them in the developer’s public repository.

What’s interesting with this attack was that the secrets were exposed to the public for almost five years on GitHub. It wasn’t Toyota that was breached. Instead, it was a third-party web developer contracted to manage the car manufacturer’s T-Connect application. T-Connect is a Toyota service that lets drivers remotely monitor their parked vehicles for any unauthorized access and check the vehicle’s status. It’s a popular service that thousands of their customers use, leaving them open to data theft.

The Supply Chain in Software Development Can Be Your Weakest Link

Cyber-criminals look for an organization’s weakest link to breach systems, and humans are always a viable target. Software developers are no exception, even if they have reasonably good knowledge of cyber-attacks. Developers could purposely publish secrets in code without knowing the consequences, or they could accidentally publish secrets not knowing they’ve been exposed. Both situations have the same potential results, so public repositories should always be monitored for secrets, keys, passwords, credentials, and other access tokens.

In the case of Toyota, the third-party developer published keys to Toyota’s private systems that gave an attacker access to Toyota’s database that stored customers signed up for T-Connect service. The exposed secrets gave anyone access to customer email addresses and control numbers. In total, Toyota estimates that almost 300,000 customer emails and control numbers were publicly exposed.

Although Toyota did not confirm anyone had accessed their T-Connect customer database, they had to inform customers that their data might be breached. The exposed secrets were publicly available for five years, so it’s possible for an attacker to quietly access data without alerting any administrators since the secrets were a legitimate access method.

In addition to the software supply chain, Toyota suffered from another supply chain issue when one of their vendors, Kojima Industries, which supplied plastic components for their cars, suffered from a system compromise of their own and caused delays in manufacturing. Just like the physical supply chain, the software supply chain can cause any organization to suffer from downtime, customer loss, fees from compliance violations, and damage to brand reputation.

Supply Chain Compromises are Costly

IBM recently reported that supply chain compromises cost $4.55 million, and they are the most difficult to detect. Stolen credentials require a unique approach to cybersecurity to detect unauthorized access mainly because access is being performed using legitimate credentials. On average, IBM reported that it took 327 days for a business to detect unauthorized access from compromised keys.

Cost of a data breach 2022: https://www.ibm.com/reports/data-breach

Most businesses know of general vulnerabilities that threaten data integrity and adopt a zero trust approach to network security, which is the current standard for best practices. Zero trust has its benefits, but often administrators and security teams forget the human element, especially their developers. Developer software might be scanned and tested for vulnerabilities, but most testing tools are built to detect common vulnerabilities and not secrets and keys embedded in code.

Zero trust cybersecurity models reduce risk and save millions on breaches, but attackers always change their tactics, and targeting a third party gives attackers much larger potential to find at least one vulnerable vendor. Your organization might practice zero trust, but at least one of your third-party software vendors likely does not. This supply chain vulnerability is one of the most challenging for organizations. 

Third-party vendors providing libraries and software to your organization and your internal developers create a weak link difficult for security teams to control. It’s difficult to stop an outsider from attacking internal systems, but detecting issues from an insider poses a bigger challenge. Developers adding secrets in code or integrating a compromised third-party library are two forms of supply chain security challenges that require a unique approach to detect them before they reach production.

Once secrets are uploaded to public repositories, your data is at risk. Repository services such as GitHub keep archives of changes, so removing secrets even immediately after being uploaded to public sites leaves your data vulnerable. In most cases, however, secrets are left publicly visible for months, potentially years as with the Toyota compromise.

To make matters worse, cyber-criminals have scripts to automatically detect secrets, keys and credentials stored in code. Some cyber-criminals continually monitor specific organizations for code errors, vulnerabilities, and stored secrets. Using exposed secrets, any data behind an API or an infrastructure database would be accessible to the attacker. Most organizations are tied to compliance regulations requiring them to alert customers if their data was compromised, and even if the secrets might not have been used, organizations must still alert customers.

What Organizations Can Do to Detect and Stop Supply Chain Compromises

Monitoring for exposed secrets in code is necessary, but organizations can also take necessary precautions to help remediate the issue and stop it before it becomes a remediation issue. Ideally, developer education should stop the issue from ever happening, but humans make mistakes, including developers. Developers might not know the consequences of publishing secrets in code, or they might store them mistakenly. Regardless of the reason, they should understand what can be done with stored credentials, API keys, secrets, and access tokens in code. What can help here is having someone on staff to champion a security program. You need a person who understands the ins and outs of cybersecurity, vulnerabilities, and common attack vectors. This person usually helps define security policies and educates users on why specific policies are necessary to protect corporate data. 

Monitoring tools such as Apiiro will detect any attempts to commit secrets in code, and it will find any current repositories that store sensitive information including secrets, API keys, credentials (e.g., usernames and passwords), access tokens, and any other data that could pose a threat of unauthorized access stored in code. Apiiro’s monitoring tool will also attempt to remediate issues and alert developers of current vulnerabilities in code. The tool continually monitors the software development lifecycle (SDLC) even in a fast-paced DevOps development environment.

The risk of leaving your software supply chain open to possible human errors, malware, and threats exposing secrets can have a long-lasting impact on your organization’s brand, customer loyalty, and revenue. To avoid being the next target, educate your developers about the consequences of leaving sensitive data in code. If your organization has the budget, hire a staff member to promote good cybersecurity practices. Finally, to stop any threats that could be mistakenly made in code, use Apiiro to monitor your codebase for stored secrets and alert stakeholders of any current vulnerabilities.

To get started with the next generation in cybersecurity automation and vulnerability detection, check out our Cloud Application Security Platform and book a demo or contact us.