Cutting Costs with Microsoft Azure Data Explorer (ADX)

The steep cost of data processing in Microsoft Sentinel – a crucial component of eXtended Detection & Response – poses a challenge for many large organizations that struggle to maintain effective cyber security processes while staying within their budget. At CyberProof, our team searched for a solution that would help lower these data processing costs for our clients. We became aware of the cost optimization potential that Microsoft’s Azure Data Explorer (ADX) provides.

ADX – a fully managed Platform-as-a-Service (PaaS) solution for Big Data analytics – parses, tags, and processes data at incredible speeds. It provides real-time analysis on large volumes of data streaming from applications, websites, IoT devices, and more. 

We decided to expand our use of this platform as a means of helping clients save money. To do so, we moved the long-term retention of the logs, and the processing of custom collected logs – the “heavy lifting”– over to ADX. We discovered that we were able to manage massive quantities of data for our clients, at a radically lower cost. 

Today, CyberProof is officially recognized as a Microsoft ADX partner.

Let’s explore just one example of how CyberProof assisted a large enterprise by leveraging ADX to reduce costs.

At CyberProof, our team searched for a solution that would help lower these data processing costs for our clients. We became aware of the cost optimization potential that Microsoft’s Azure Data Explorer (ADX) provides.

Background – The High Cost of Data Processing

Today, all our clients who work with Microsoft Sentinel – a cloud-native Security Information Event Management (SIEM) solution – use CyberProof Log Collector (CLC). The CLC, a tool developed in-house by CyberProof, was specifically designed to enhance the functionality of Microsoft Sentinel by enabling logs from custom sources to be collected and ingested for security analytics.

CyberProof Log Collector (CLC)

For large enterprises handling massive quantities of data, we found that it was not simple to manage the data cleansing and filtering processes. For the amount of data involved, extensive and costly computing was required.

The biggest challenge was the fact that ingesting large quantities of data to Microsoft Sentinel – and retaining it for the necessary time periods – could cost enterprises millions of dollars per month.

To solve these problems, we decided to leverage ADX – which was introduced by Microsoft as a long-term retention solution. We discovered that we could use ADX together with our CLC tool for data ingesting and processing – to help enterprise clients not only save money but also manage their data better.

The Problem – A Large, International Enterprise Needed to Cut Costs

CyberProof provides Managed Security Services to a large, international enterprise. For this client, the platform for Security Information and Event Management (SIEM) was Microsoft Sentinel. 

The challenge in this case related to the fact that the customer ingests huge amounts of data – approximately 9 terabytes of logs, per day. The monthly cost of managing this data was exorbitant. 

CyberProof searched for ways to help the client radically reduce its data processing costs without impacting its threat detection capabilities.

The CyberProof Solution

CyberProof moved the data processing step for this client – including the data parsing and aggregation, and the data cleansing – to an ADX cluster. 

We leveraged the ADX consumption model, paying for the infrastructure rather than paying for the quantity of data ingested (unlike Microsoft Sentinel). By using ADX, we were able to filter the data, retain all the information for reporting, compliance and regulation, and ingest only relevant Managed Detection & Response (MDR) events into Sentinel. 

This reduced the client’s expenses significantly while simultaneously providing exceptionally high execution capabilities.

A Deeper Look: How CyberProof Leveraged ADX

Moving all the parsing, tagging and aggregation to an ADX cluster for this client involved implementing the following process:

1. Events are ingested into the CLC. 

2. All the data is shipped to the ADX.

3. The parsing, cleansing, and tagging generally is done by the CLC within Parser POD - though in this use case, the Parser parses the CEF messages and ADX handles the processing of the data.

4. All the events specific to security use cases are passed to Sentinel/Log Analytics. 

5. The CyberProof Defense Center (CDC) platform provides MDR capabilities. (Read more here) 

The Results

When CyberProof first started using ADX for this client, we conducted a Proof of Concept (PoC) in which we measured samples of the data every 12 hours for a 48-hour period. Over a billion events were received within this period. 

We demonstrated that by formatting, cleansing, and tagging the data and combining fields, we were able to decrease the size of the data by 46%. This is because formatting was enabled within ADX via a mapping rule on ingestion, so that only the relevant data was kept.

We demonstrated that by formatting, cleansing, and tagging the data and combining fields, we were able to  decrease the size of the data by 46%.

Next, we transferred the relevant data from ADX to Sentinel and mapped out the log sources – so that we only took events that were relevant to MDR. The total decrease in the amount of data was 60%.

Benefits to the Client

Using ADX to reduce both the size and the amount of data, CyberProof was able to facilitate for this client almost 60% cost savings. 

Benefits to the client included:

• Lower costs – The entire yearly expenditure for the client’s ADX cluster was only $168,000 – a minimal sum compared to the millions of dollars in costs without the ADX/CLC solution.

• Better threat management – ADX is a great environment for threat hunting, compliance, and reporting – supporting better threat management and lowering the risk of attack.

• An available skill set – Anyone who works with Log Analytics understands how to use ADX, as it is the same system; this made it easier for the client’s team to work with ADX.

• Less maintenance – ADX is provided as a Platform-as-a-Service (PaaS); therefore, it requires less maintenance. ADX reduces the amount of investment in infrastructure that is necessary to achieve the same results.

Using ADX to reduce both the size and the amount of data, CyberProof was able to facilitate for this client almost 60% cost savings.

Getting the Most Out of Microsoft Sentinel with CyberProof’s CLC and Microsoft ADX

CyberProof, a Microsoft ADX partner, helps clients utilize Microsoft Sentinel more efficiently and at lower cost. Our ability to use ADX together with CyberProof’s proprietary CLC tool to filter and process data provides significant cost savings for large, enterprise clients. The CLC with ADX supports faster managed detection and response – further mitigating the potential risk of cyber attack.

Our ability to use ADX together with CyberProof’s proprietary CLC tool to filter and process data provides significant cost savings for large, enterprise clients.

CyberProof supports ingestion of data into native tables, which means that any enterprises utilizing the CyberProof Log Collector and ADX with Sentinel can leverage all of Sentinel’s native capabilities and the Microsoft Community Content.

Would you like to learn how we can help your company optimize processes for better Managed Detection and Response? Contact us today!