A recent blog post by Wiz detailed an attack performed on a customer tenant of the SAP AI Core platform. Here’s a short summary of the attack:
Access the Istio sidecar through Kubernetes configuration allowed Wiz to get an access token to the central Istiod server
The Istio token was used to scan the network
Wiz found a Loki config that had AWS S3 secrets
Wiz was able to use S3 secrets to access customer AI logs
EFS instances were discovered with full access that had customer AI data
A Tiller server for Helm charts (which is deprecated) was running in the cluster, which had Docker registry secrets
Wiz had write access to Tiller to run any Helm chart they wanted
Using this access level, an attacker could directly access other customer’s Pods and steal sensitive data
Secrets that are beyond the scope of SAP AI Core were accessible, including AWS accounts, Docker Hub accounts, and SAP HANA accounts
Image credit: WIZ Research
Major credit should be given to the SAP team, as they responded to the disclosure by addressing the vulnerabilities and rotating secrets. However, this attack is eye-opening in how easy it was for a customer of a SaaS product to move laterally within a cluster and access secrets in plaintext from other customers. The fact that this was an AI tool pulling data from sensitive sources adds to the concern, but this risk will only grow as AI becomes more mainstream.
The relative ease of movement within the SaaS cluster is concerning, but is also more common than you might think. If you’re like us and want to prevent these kinds of data leaks from happening to your environment, here are 5 takeaways that you can apply to your own systems.
The main concept behind the Zero Trust security model is "never trust, always verify", which means that users and devices should not be trusted by default, even if they are connected to a permissioned network. This has become a bit of a nebulous concept as the number of pillars and controls that enable Zero Trust seems daunting and unreachable, causing some IT admins to not even strive for it as a goal. The enterprise software market doesn’t help, with some vendors claiming to provide full Zero Trust even though they only touch on one aspect.
That said, a Zero Trust approach has real-world benefits. In this specific attack, Wiz was able to access AWS Elastic File System (EFS) instances by simply being on the same network. The assumption with the default EFS setup is that if a user or machine is on the same network, then it must mean it’s supposed to be there and can have full access. By applying Zero Trust principles to your EFS instances, you can lock down access to only verified users instead of relying on network controls.
It has become a baseline standard for human access to require multiple factors of authentication. This came about because despite increased password complexity and faster rotation schedules, any service that only uses a password for human access was vulnerable to cyber attacks. This same risk applies to service accounts and automated communication, but it gets ignored in many IT setups. In this attack, there were numerous secrets that were discovered and used that you might have thought were safely hidden, including:
Just by having these single-factor tokens, the Wiz engineers were able to potentially do the following:
“Using these secrets’ read access, a potential attacker could read internal images and builds, extracting commercial secrets and possibly customer data.”
“Using the secrets’ write access, an attacker could poison images and builds, conducting a supply-chain attack on SAP AI Core services. “
“Using this access level, an attacker could directly access other customer’s Pods and steal sensitive data, such as models, datasets, and code. This access also allows attackers to interfere with customer’s Pods, taint AI data and manipulate models’ inference. “
“Using our newfound access level, we queried for those secrets, and managed to access all of them in plaintext”
We know that single-factor isn’t good enough for humans, so we must apply the same concepts to machines in order to protect our sensitive data and systems.
AI tools are incredibly popular right now thanks to their promise of automated insights and quick correlation of your internal data. It’s tempting to bypass normal security procedures for your data in order to achieve these promises, but the risks must be weighed. Not only do you need to provide historical data to an AI tool, but you need to give continuous access to your most sensitive information in order to get the benefits. This level of access must be taken as seriously if not more so as any other human or tool.
SaaS applications can provide big benefits to the companies that write the apps as well as their customers. Economies of scale with shared clusters or servers provide immediate savings over self-hosting or standalone environments. However, it’s easy to forget that shared architectures are actually shared, which means there’s less barriers between another customer and your sensitive data. A small misconfiguration from the SaaS provider can lead to leaked data without any misuse on your end. Also, potential attackers see this as a better target for attacks, as one attack can lead to stolen data for multiple customers.
Keeping up to date with software patches and supported versions can be time consuming for your IT team, but it’s more important than ever with how fast knowledge of vulnerabilities can spread to bad actors. In this attack, a Tiller server was found for running Helm charts, but this tool has been deprecated since November 2020. In the deprecation announcement, it was flatly stated that “due to the vast number of possible security policies, our stance was to provide a permissive default configuration”. Maintaining supported software and versions is crucial for bug fixes and patches, but also to understand the security posture of the default settings for your tools.
About Chris Parlette
Chris is a Senior Solutions Architect at Corsha. He has a diverse background encompassing activities such as working with both on-premises and Software as a Service (SaaS) solutions, utilizing cloud and monitoring tools, offering consulting and professional services, conducting demonstrations and Proof of Concepts (POCs), managing Requests for Proposals (RFPs), contributing to marketing endeavors like blogging, participating in trade shows and conferences, engaging on social media platforms, navigating sales processes, providing customer support, handling custom integrations, and much more.
About Corsha
Corsha is an Identity Provider for Machines that allows an enterprise to securely connect, move data, and automate with confidence from anywhere to anywhere. Corsha builds dynamic identities for trusted machines and brings innovation like automated, one-time-use MFA credentials to APIs. This ensures automated communication across clouds, data centers, or shop floors is pinned to only trusted machines and helps an organization move past outdated, costly secrets management and reimagine identity and access for machines.