How Lacework caught an accidental insider threat

Staying ahead of threats, especially from within, is critical yet extremely difficult for teams using traditional or even “nextgen” security tools. Lacework is unique in this capability, and through its machine-learning technology, recently demonstrated its ability to protect cloud environments against a bleeding edge sophisticated threat. This blog post explores a recent story involving the detection of a developer-installed backdoor in a Databricks environment. It highlights the role of insider threats and explores the concept and implications of backdoors in cloud computing. 

Understanding the threat: what are backdoors?

In cybersecurity, a backdoor refers to a method by which authorized and unauthorized users can bypass normal authentication and security mechanisms to gain high-level user access to a computer system, network, or software application. Backdoors can be created for legitimate purposes such as the provision of remote access to IT support for troubleshooting. However, they are also leveraged by attackers to stealthily gain continued access to the target environment. These backdoors can be particularly alarming in a cloud computing context where they might allow attackers to access vast amounts of data or gain control over scalable cloud-based resources.

Examples of backdoors in cloud providers

Backdoors in cloud environments manifest in various forms, such as:

1. Embedded code in cloud applications: Malicious code inserted into cloud-based applications can provide backdoor access at the application level.
2. Compromised virtual machines, containers, or hosts: Attackers might install backdoor software on a VM to maintain continuous remote access, often for data theft or other malicious activities. This scenario unfolded in the situation covered later in this blog post.

The Lacework anomaly detection win

This success story involves a developer, who was a hired contractor and not a malicious hacker, that created a backdoor in a Databricks environment deployed into this customer’s cloud. Why and how did this happen? His experience was primarily on *nix systems, but he was tasked with working with Windows systems. To navigate around his lack of experience, he leveraged a Linux box and created a backdoor to do his work.

While this doesn’t necessarily constitute a threat because of the lack of malicious intent, the scenario illustrates how easy it is to open up potential vectors in the attack surface. And thanks to the sophisticated anomaly detections and Composite Alerts built into Lacework, this activity was swiftly identified and mitigated. Lacework Composite Alerts are indispensable to incident responders, not only because they streamline their alert investigation process but also because they effectively weave together multiple weak signals to unveil and address complex security threats. 

Rather than starting from a list of known good or bad behaviors that security teams must constantly maintain against an ever changing environment, Lacework continuously ingests vast amounts of data to establish an organization’s baseline behavior to then determine what is new or different in your environment before enriching with security context. Applied to Composite Alerts, this allows the identification of anomalous behavior early, which enables customers to find and react to issues before a full scale incident response. The ability to catch an attack in its earliest stages, detecting each anomaly with only the slightest hint of a signal, is a highly differentiated approach that makes Lacework unique from other solutions that claim to also have threat detection.

Out of five alerts that contributed to the investigation, the consolidated Composite Alert was instrumental. This alert played a crucial role in combining multiple individual alerts into a broader context, enabling the security team to quickly identify the complex security risk and ultimately removing the backdoor. Let’s look at the timeline of events, which all took place automatically in Lacework in the span of an hour: 

Part 1: The anomalies emerge

At about 3PM ET, Lacework detected something unusual. An alert popped up and flagged the execution of a new application, zsh, on a critical host. Initially, that alert sparked curiosity, but not alarm because new shell executions often fall into a gray area. This did, however, indicate that someone or something might be exploring unconventional tools or methods on the host system. And while the context into the depth of the alert was subtle, this initial alert contained enough security relevance for Lacework to automatically start the investigation for what could result in an eventual Composite Alert.

Part 2: The growing evidence

Lacework began connecting the dots within minutes after detecting more anomalies. Another alert triggered — this time for gs-netcat, an application known for its ability to establish a secure TCP connection behind a NAT/Firewall. Though not inherently malicious, gs-netcat can be used for reverse shells or remote command execution. This second alert, on the same host, intensified the scrutiny. Lacework automatically added this alert to its growing collection of evidence, while also triggering an early warning to the security team.

The alerts continued to paint a clearer picture of a potential security incident once Lacework detected that application gs-netcat, running on the host with root privileges , made an outbound connection to an external IP address on TCP port 80. This was the first time an outbound connection was made to this external IP address from this environment, which triggered an alert that flagged an “Outbound connection to a new external IP address from application”. Interestingly, according to VirusTotal, this new IP address did not appear to be malicious. Normally, an outbound connection from an application, such as gs-netcat, running with root privileges to an unknown IP address would merit further manual investigation. With Lacework, these steps are automatically carried out.  

Lacework then further detected that application gs-netcat running on the host made an outbound connection on port 7350 to an IP address resolving at l.gs.thc.org, another suspicious connection to a new domain. Fun fact: The Hacker’s Choice website, thc.org, associated with subdomain l.gs.thc.org, is a benign, known resource for security researchers.

Adding to the growing pile of evidence, Lacework detected the gs-netcat binary with hash 362b700c68ff2dc5c4188d32096b9c3d0f61073b9758cf25ab068b095460b9f9 as suspicious file by correlating threat intelligence data.

With this hash, there was the critical mass of correlated, security-focused detections needed to trigger the next step.

Part 3: The final Composite Alert

Less than an hour after the initial alert fired as an early warning signal, Lacework fired a final, consolidated alert: a Potentially Compromised Host Composite Alert notifying the security team that a host machine might be compromised.

By automating the detection and correlations of these anomalies, Lacework alleviated the burden on security analysts who otherwise would have been bogged down in low severity signals. This automation enables analysts to focus on higher-value tasks, swiftly and efficiently addressing the most critical security events. Without Lacework, an analyst would have needed to manually connect the dots, risking the possibility of overlooking this backdoor installation. 

The importance of addressing insider threats

Insider threats represent one of the most elusive yet potentially damaging risks to cloud environments. Unlike external attacks, insiders already have some level of authorized access which can be exploited to facilitate malicious activities or unintentional harm. This use case also underscores the need for robust internal controls, continuous monitoring, and the deployment of advanced security systems, like those provided by Lacework, to promptly detect and respond to such threats.

Machine learning trained threat detection is essential

The ability for Lacework to detect and respond to intricate security challenges from cloud environments to application code and system events is a testament to its leading-edge technology and strategic approach to cloud security. Utilizing advanced machine learning and Composite Alerts, Lacework not only addresses the evolving landscape of cloud threats but also fortifies organizations against future vulnerabilities, including internal risks.

Organizations that leverage cloud technology must recognize the critical nature of insider threats and the potential for backdoors. With that in hand, they must equip their security teams with the tools and strategies necessary to comprehensively protect their digital assets. While this organization was lucky that this backdoor was installed without malicious intent, this scenario also raises a potential policy issue in which enforcement needs to be addressed.

This story showcases the effectiveness of the advanced threat detection patented to Lacework and serves as a crucial reminder of the ongoing vigilance required to secure cloud environments against increasingly sophisticated threats.