Accelerating The Migration To Modern, Cloud-Native Data Lakes

By: Deb Banerjee, Co-Founder and CTO, Anvilogic

‍

Cisco bet big on cybersecurity last week when it announced its acquisition of Splunk for $28 billion. This makes it Cisco’s largest deal to date and follows the trend of tech giant growth through acquisitions. 

Splunk remains the workhorse SIEM for many enterprise SOC’s. However, enterprises have challenges with Splunk, specifically - (1) steeply rising costs, and (2) Splunk’s inability to migrate to a modern cloud-native architecture taking advantage of their dramatic cost-performance benefits. The Cisco acquisition shall exacerbate both these challenges. Innovative enterprise SOC’s that have been early adopters of modern cloud native data lakes such as Snowflake have seen significant cost-performance benefits up to an order of magnitude. The Cisco acquisition of Splunk shall be an inflection point on this adoption of modern, cloud-native data lakes in the SOC.

Innovative Enterprise SOC teams have taken the path of incrementally adopting modern, cloud-native data lakes to manage migration risk. Enterprises have a decade worth of security events and alerts ingested into Splunk and detection and response programs operating on that platform. They use an incremental approach to migrating feeds, and detection and response logic away from Splunk into modern, cloud-native data lakes such as Snowflake. Clearly this is not an overnight migration; the good news is that this migration is happening without a “rip and replace” thereby ensuring continuous detection and response efficacy and minimal additional operational overhead.

‍

So, how do SOC teams move forward?

Based on these early adoption patterns in enterprise SOC’s, we recommend introducing a modern, cost-effective data lake such as Snowflake in addition to your existing Splunk SIEM. This is enabled by a security analytics platform that works across both Splunk and these modern data lakes taking advantage of their innovation in terms of cost, scale, and performance.

‍

Migrating Away From Splunk To Modern, Cloud-Native Data Lakes: A Customer Story

A large financial institution was using Splunk as its sole security data lake. For cost reasons, they were unable to ingest CrowdStrike FDR feeds into Splunk because the high volumes would blow through their Splunk license costs. Instead, they used Anvilogic to bring that data set into Snowflake fully parsed and normalized, and were able to deploy over a hundred detections against it for a fraction of the cost it would have taken them to do it on Splunk. They use the Anvilogic platform for parsing and normalization as well as deploying detection logic developed by Anvilogic’s own threat researchers into Splunk and Snowflake, respectively, Further, the Anvilogic platform correlates across alerts in those two distinct data lakes for detecting custom attack sequences that threat actors may use.

This hybrid strategy (which our customers, including a top 20 US Financial Institution, do) allows organizations to adopt a SIEM-less architecture at their own pace, while saving more than two-thirds of what they were paying for Splunk. 

The journey often starts with a single data feed like CrowdStrike FDR data or AWS CloudTrail  that is unaffordable to bring into Splunk. These logs are ingested into Snowflake, detections deployed against the feed, and correlated with existing Splunk alerts. SOCs then add new feeds or migrate existing feeds from Splunk over time. By adopting a security analytics platform that works across multiple data repositories, security teams can slowly shift from Splunk to a modern, cloud-native data lake like Snowflake.

‍

Empowering SOC’s To Adopt Cloud-Native Technologies: The Great Unbundling

One of the fundamental trends we see in technology is the separation of the data platform from the threat detection, and response layer on top of it. Traditional SIEM vendors that have been bundling both storage and the threat detection response layers have failed to keep up with cost, performance and detection needs of modern SOC’s thereby accelerating this separation. This has been pointed out here https://ventureinsecurity.net/p/security-is-about-data-how-different  as well.

These modern detection and response platforms will be focusing on threat detection and response and allow enterprise SOC’s to use modern, cloud-native data lakes at a far lower cost and higher performance than traditional SIEM vendors could do. 

‍

The Incremental Approach To Managing Migration Risk

SOC teams need to ensure that there are tools in place to migrate the protections with the data to preserve detection efficacy. And as they get confidence in running detections in the new data lake and correlating across their detections in Splunk, they can start moving feeds and detections from Splunk.

‍

How does Anvilogic help SOCs move forward?

Anvilogic bridges the gap between Splunk and modern data lakes like Snowflake so that enterprise security teams can migrate at their own pace. By decoupling data from the logging and analytics layers, data can be left where it is, without compromising on the security.

Four years ago, we were primarily building our detection and automation platform on Splunk. Since then we have seen our customers ask for this hybrid strategy, being able to bring security events into Snowflake, a modern data lake, side-by-side with their existing Splunk fields. We build out threat, detection, and response across your data lakes and correlate across those transparently. Anvilogic has been doing this for the past two years for a variety of enterprise accounts with proven and mature tools, technologies, patterns, and practices that you can adopt right now.  

Want to chat about how well this has worked for our customers and the cost benefits they've gotten out of it? Get in touch.