Part 1: CI/CD Pipelines Efficiently Delivers the Most Accurate and Updated Security Content

Applying Continuous Integration and Continuous Delivery (CI/CD) processes into your content workflow enables you to quickly refresh your security content to minimize potential attacker risks. However, in order to effectively utilize the CI/CD process for your content, you must first know what is the codebase to use.

In this guide we will discuss:

• What is Codebase from a Splunk perspective?
• What is CI/CD and how to incorporate it into your security content update process?
• How does BlueVoyant perform this today?

What is Codebase from a Splunk perspective?

Splunk is one of the top tier data storage and query engines in the market, most typically used by companies in the Security and Observability spaces. Splunk allows analysts a quick view of metrics, events, logs, and trace related data to determine suspicious activity. This is done through an advanced indexing and query engine based on data from source devices.

BlueVoyant’s MDR for Splunk services provide security detections, also known as correlation searches, and MDR-specific dashboards to detect malicious or suspicious activity from event logs. While this sounds simple enough, it requires multiple components in order to function correctly. BlueVoyant’s code base consists of:

• Use Cases: We first start with a Use Case to clearly outline the activity we are attempting to detect. This enumerates all the requirements from required data sources, specific fields, activity tracking, response procedures, and more. This is captured and tracked in JIRA.
  
• Security Detection Logic: Splunk Processing Language (SPL) is developed based on the BlueVoyant Information Model to generate the required results to alert a SOC analyst of suspicious activity. This SPL is then checked into a code repository.
• Technology Add-ons (TAs): BlueVoyant has developed its own collection of Splunk TA’s that parse the data according to the BlueVoyant Information Model and apply required enrichments for the Security Detection Logic to be successful. Configuration files such as lookups, props.conf, transforms.conf, eventtypes.conf, tags.conf, and more are also checked into the code repository.

CI/CD Steps

There are three key steps in CI/CD: Design & Build; Test; Deploy. Continuous Integration encompasses the first two steps, while Continuous Delivery covers the third one.

Continuous Integrations

At BlueVoyant, a SIEM Use Case is living, breathing code with an evolutionary life cycle. The GitLab concept of Continuous Integration, (CI), has a place in security best practices, too. CI is the practice of integrating your developer’s code changes into the main branch of a shared source code repository early and often, using “pipelines” to automatically test each change when they commit or merge them, and automatically kicking off a build. With continuous integration, errors and security issues can be identified and fixed much earlier in the development lifecycle. Our code base for the aforementioned correlations and TAs are a collection of Splunk files that make up a single Use Case and our code repository is stored in GitLab. This also means there is a record and a repository where your use cases are backed up beyond just being in Splunk.

• Design: Planning out and designing a use case is critical to the success of the detections. BlueVoyant uses a collection of resources to select detections. These resources are based on internal and external factors. Internal sources look at our current customer base and determine which new attacks or techniques are being used. External factors are from government and non-goverment sources detailing the attack vectors. We prioritize our development efforts by criticality, technology being exploited, and overall risk to our customers. A checklist of requirements is created to validate we have all the necessary elements to properly build a detection. The check-list is made up of Runbooks, data sources, parsing, Lookup tables and other items based on needs. Once everything is collected we move to building the detection.
• Build: Now the fun part. If the design was successful the building is super easy. There are three high level components to building a detection: Query, Parsing, Operations. BlueVoyant has created its own proprietary Information Model to support Query and Parsing activities. These are fully explained in another upcoming blog. Operations takes the concept that no alert should hit the Security Operations Center (SOC) without a runbook or a list of activities to be taken.
• Test: No detection is released without testing and validation from multiple parties. The SOC is critical in testing as we see them as the customers of the detections. Runbook development and validation of fields makes for a more streamlined and effective use of our analysts time. We also use a long list of lint checks in our CI/CD pipeline to automate as much of the configurations as possible. Everything from Btool checking, configuration typo’s, and broken searches are ran though an automated pipeline to ensure no human error is deployed to our customers. These test will be enumerated in details in a future article.

Continuous Delivery

Continuing with GitLabs concepts, we have Continuous Delivery (CD), CD is the software development practice that works in conjunction with CI to automate the infrastructure provisioning and application release process.

New content delivery across multiple Splunk instances takes hours. With our CD process, hundreds of instances can programmatically apply new SIEM detections for zero day vulnerabilities faster than a human could ever apply them so we can spend time triaging and containing bad actors.

• Deploy: BlueVoyant has a proprietary system known as “ASAP”, which packages and delivers the application(s) based on the targeted environment. For simplicity purposes, BlueVoyant elects to use three environments: Test, QA, and Production, where Production is connected to a customer's environment directly. ASAP leverages the Admin Configuration Service (ACS) to deploy the application(s) to Splunk Cloud environments.