Why You Should Be Ingesting AWS GuardDuty Logs

TL;DR Cost-effective ingestion of noisy GuardDuty logs requires investment in filtering, enrichment, tuning, and a cloud-native SIEM that eliminates infrastructure management overhead and outmoded licensing fees.

Cybersecurity professionals know that visibility is foundational to security. Quite plainly, how can you spot malicious activity if you can’t see what’s going on? But gaining system visibility at cloud-scale is not without its challenges: cloud providers like AWS generate voluminous logs that can break budgets and clog up the threat detection environment.

In this blog, you’ll dig into AWS GuardDuty logs to understand why you should be ingesting them into your Security Information and Event Management (SIEM) platform, and how you can manage these “noisy” logs to control costs and improve detection. With this in mind, you’ll learn how to filter raw log data to only ingest relevant security information and boost your signal-to-noise ratio with log enrichment and detection-as-code. You’ll also understand the limitations of SIEM infrastructure and licensing that may prevent you from cost-effective ingestion. 

Why you should be ingesting AWS GuardDuty logs

Amazon GuardDuty is a security monitoring service that uses threat intelligence feeds and machine learning to identify unexpected, potentially unauthorized, and malicious activity within your AWS environment. Examples include identifying anomalous behavior like unusual API calls, deployments, or network traffic. When GuardDuty identifies suspicious activity, it records the activity as a security finding within the GuardDuty console or another AWS integration.

Notably, GuardDuty is a security solution for AWS resources only. That’s why practitioners who secure a multi or hybrid cloud environment use a SIEM to ingest GuardDuty logs alongside other log sources to monitor their entire attack surface on one platform. With a SIEM, practitioners have granular control over aggregating, normalizing, analyzing, and correlating logs—across all log sources—to comprehensively identify and address threats. 

Noisy logs are problematic

While log monitoring is ubiquitous in cybersecurity, the nature and structure of logs is not, which is why log management is necessary. This practice includes normalizing, analyzing, and storing logs to render them useful. With particularly voluminous AWS logs, log management becomes vital. Left unmanaged, noisy AWS logs create two major problems.

The first problem is about budget constraints. The high volume of AWS GuardDuty logs can lead to prohibitive ingestion costs, forcing an impossible choice: increase the budget, or don’t ingest at all. While more logs always require greater storage capacity, a big culprit driving up spending is traditional SIEM infrastructure and licensing structures that are unfriendly to high-volume cloud logs. 

To maintain a baseline of performance, traditional SIEMs require ongoing database management to adjust how logs are indexed based on the write frequency and hardware profile. Then, there’s the high price of hot storage causing most data to reside in warm and cold buckets that take much longer to query during incident response and threat hunting. Facing these obstacles on top of additional licensing fees for cloud logs, security teams often do not ingest GuardDuty logs into their SIEM, but resort instead to siloing their log data in an S3 bucket, if at all.

The second problem is about threat detection. Any number of irrelevant logs ingested into your SIEM congests the threat detection environment. Congestion contributes to well-known operational issues that increase risk, including alert fatigue, false positives, and slow mean time to resolve (MTTR). A prime example of the impact of alert fatigue on effective threat detection is the March 2023 attack on 3CX’s supply chain.

Migrate to a cloud-based SIEM to control costs

If you are still using a traditional SIEM, migrating to a cloud-based SIEM will have the most significant impact on your ability to cost-effectively ingest AWS logs and gain full system visibility. 

Here’s why: a cloud-based SIEM leverages serverless architecture to eliminate infrastructure management; further, log data is saved in a security data lake that uses modern cloud database technology designed to not only handle cloud-scale data, but guarantee query performance. This enables cloud-based SIEM platforms to deliver exactly what modern security teams need:

  • Petabyte-scale search. By separating how data is stored and computed, the security data lake provides hot storage going as far back as 365 days for rapid investigation and response, even when running concurrent workloads.
  • Unified data model (UDM). A UDM configures and standardizes a set of unified fields across all log types, making this work optional rather than required.
  • Instant scalability. With a security data lake, you can scale resources up or down as needed for incident response and threat hunting.
  • Fully managed data. The security data lake includes data encryption, compression, governed policies and rules, data-masking, automated JSON parsing, and more, reducing data management workloads. 

Filter raw log data to reduce noise and control costs

Short of adopting a cloud-native SIEM, controlling noisy log data requires investment in filtering, a key log management practice. There are a few ways to create filters, depending on how your log pipeline is set up.

Within AWS, you can create GuardDuty finding filters that only report findings that match with your specified criteria. Unmatched findings are ignored and thrown out. You can also implement GuardDuty suppression rules to automatically archive findings that match the specified criteria. AWS recommends using suppression rules to “filter low-value findings, false positive findings, or threats you do not intend to act on,” reducing overall noise. 

However, if your log pipeline uses AWS CloudWatch to aggregate and forward logs, you may prefer to use subscription filters to control which logs are sent to your SIEM. 

Within your SIEM, you can create filters by log source to ignore—or throw out—log data before it’s analyzed for threats and saved to your data lake. Two common methods are raw data filters and normalized data filters.

A raw data filter processes and filters log data before the SIEM normalizes the data according to its schemas. A raw data filter specifies a pattern to match with. If any log matches the pattern, the entire log is ignored and not processed by the SIEM. The available tools for filtering raw log data depends on the SIEM you are working with, but these are the two most common methods:

  • Regular expressions (regex). A sequence of characters that represent a pattern that searches for, matches with, and manipulates strings in text. For example, you could specify the regex /[[:digit:]]/g to filter out any raw log data that contains numerical digits. The text “there was 1 fox” would be thrown out, and “there was one fox” would be processed.
  • Substrings. Text, or a sequence of characters, used as a matcher when analyzing other text. For example, the substring “hello” could be used to filter out any raw log data that contains “hello” at least once. The text “hello world” would be thrown out, but “world wide web” would be processed.

In contrast, a normalized data filter processes and filters log data after the SIEM normalizes the data. The benefit here is a granular filter that can throw out individual fields—or “keys”, pieces of data within a log—instead of the entire log.

To create a normalized data filter, you’ll typically need the following information:

  • The log type. For example, AWS GuardDuty.
  • A field. The identifier or “key” for the data you wish to remove, like any finding detail.
  • A condition. For example, “contains”, “is less than”, or “has any”.
  • A value. The state of the data for the field that you want to remove. This could include having no data.

With both filtering methods, any logs that are dropped during the filtering process should not contribute to your overall ingestion quota; verify pricing with your SIEM vendor.

An Example: Filtering out sample GuardDuty findings

Filtering starts by determining what information is irrelevant to security. Let’s look at an example of how to use a substring filter to exclude sample GuardDuty findings.

AWS GuardDuty assigns a severity level to all its findings, and it’s common to create detections that target these categories:

  • High severity. A resource is compromised and is actively being used for unauthorized purposes, and needs to be addressed immediately. High severity findings are assigned a numerical value from 7.0 – 8.9.
  • Medium severity. There is suspicious activity that deviates from normal behavior, and may be indicative of a resource compromise. The activity should be investigated as soon as possible.  Medium severity findings are assigned a numerical value from 4.0 – 6.9.
  • Low severity. There is suspicious activity that was attempted, but failed, like a port scan to discover weak points in a network. Low severity findings are assigned a numerical value from 1.0 – 3.9.

For testing purposes, you can generate sample GuardDuty findings. Like real findings, sample findings will also have severity levels assigned. To distinguish sample data from actual data, GuardDuty includes the value “sample”: true within the log data. Within your SIEM, you can use that exact text—”sample”: true—as a substring filter to exclude all sample data from being ingested into your SIEM.

Using GuardDuty finding filters, you need to specify filter criteria for what you want to surface: you’ll create a filter with the attribute sample set to the value false. With this finding filter, you’ll only see actual data and filter out any sample data. 

With a GuardDuty suppression rule, do the opposite: create a filter with the attribute sample set to value true, in order to filter out all sample data.

Enrich logs to increase alert fidelity

Alongside filtering, enriching your log data with context and threat intelligence increases the fidelity of your alerts and speeds up investigation and incident response. A classic example is adding information about business assets to log data, like a user-to-hardware mapping. Another example is mapping numeric IDs or error codes to human readable information.

To enrich logs, create a lookup table, a custom data set that you upload to your SIEM and configure to enrich one or more specified log types. The next image shows how this process works, where a lookup table of known bad actors enriches an incoming log by the matching IP address

Your SIEM may also partner with third-party threat intelligence providers for out-of-the-box log enrichment. For example, IPInfo is a threat intelligence provider that compiles IP address data, like geolocation. With IPInfo you can preemptively identify and block traffic from high-risk locations or networks.

Boost signal-to-noise ratio with detections-as-code (DaC)

Filtering enables you to reduce noise and control cost, and log enrichment improves the fidelity of your alerts. But another essential task is to increase your signal-to-noise ratio by tuning detections. 

Tuning detections is the process of customizing detections so they are optimized for your specific environment, address emerging threats, and cover any security gaps. When detections are specific, informative, and cover relevant security threats, you’ll be able to more accurately identify threats and resolve them faster. In other words, you’ll clearly hear the signal amid the noise. 

Boosting signal-to-noise has important downstream effects on your operations and cost: it reduces alert fatigue, false positives, and false negatives, and it improves mean time to detect (MTTD); practitioners can get out from the endless cycle of responding to alerts, and spend more time tuning detections, creating a positive feedback loop. 

But the ability to customize detections varies across SIEMs. It’s well known that legacy SIEMs suffer from inflexible tools that limit the extent to which you can tune and optimize detections for your environment. Platforms that offer detection-as-code (DaC) are taking customization and flexibility to a whole new level, by writing, managing, and deploying detections through code. The drive behind DaC is to make threat detection consistent, reliable, reusable, and scalable, all while controlling cost.

First, get a sense of what a code-based detection looks like. Check out the next image that shows an excerpt from a Python detection for high severity GuardDuty events. This is one of Panther’s 500+ pre-built detections. The logic in this rule will trigger an alert when the log data contains a “severity” key with a value between 7.0 and 8.9. What’s not shown in this excerpt is the logic that defines what information goes into the alert, tests for the detection, and other ways to customize how and when the alert is triggered. 

Now let’s connect the dots and understand how DaC’s core features give you the customization and flexibility to optimize your threat detection environment, and your operations:

  • Python. Widely-used coding languages like Python are expressive and flexible. They give you access to a host of built-in tools and libraries that provide useful out-of-the-box functionality, like crunching numbers (like float, line 9 in the code excerpt), or grouping routine processes or information into reusable functions (like deep_get, line 5 in the code excerpt). This makes writing detections a flexible process where it’s easy to customize detections and cover security gaps.
  • Testing and quality assurance (QA). Teams can create code-based unit tests that sit alongside the code-based detection rule. Code-based unit tests verify the efficacy and reliability of detection logic, ensuring correct, relevant, and high-fidelity alerts.
  • Managed in version control. Managing detection rules in version control facilitates effective change tracking, rollbacks, and audits. Coded detections are self-documenting and anyone with access can easily understand the detection environment. These systems also provide tools like pull requests for standardized peer review and collaboration, so you can maintain consistent quality in threat detection development.
  • Automated operations. Integrating DaC into CI/CD pipelines automates testing and deployment, streamlining operations and ensuring consistency and scalability by eliminating manual work.
  • No-code workflows. Security teams can benefit from DaC—including expressiveness, testability, CI/CD integration, and reusability—without requiring every team member to have programming experience. Practitioners can implement a console-based workflow using forms or a user-friendly markup language to create, test, deploy, and automate detections. 

To summarize, detection-as-code is all about efficiency and reliability; it gives security practitioners the flexibility to optimize their detections and the agility to stay on top of threats in an ever-changing cybersecurity landscape. To get a closer look at DaC, including no-code workflows, read how to create a code-based detection.

Choose a SIEM that’s built for the cloud

Traditional SIEMs cannot keep pace with cloud workloads, sacrificing timely and cost-effective service, and increasing overall vulnerability. Whether you control cost and noise with filtering, or boost your signal-to-noise ratio with detection-as-code, choose a SIEM that is built to handle cloud-scale data, without compromise.

Take a deep dive on how to keep AWS logs from running wild with Panther, a cloud-native SIEM that’s built for AWS. Panther empowers modern security teams with real-time threat detection, log aggregation, incident response, and continuous compliance, at cloud-scale. 

Ready to try Panther? Get started by requesting a demo.

Recommended Resources

Escape Cloud Noise. Detect Security Signal.
Request a Demo