Introduction: The Detection Blacksmith

“It is a strange fate that we should suffer so much fear and doubt over securing AWS.”
— Boromir, maybe

Given the return to the screen of shows like Game of Thrones and Lord of the Rings, we’ve been thinking in high fantasy archetypes lately. As security practitioners, tasked with outfitting our organizations with a strong defense against attacks, we see a corollary in the work of the blacksmith.

A smith starts with raw iron and materials then leverages their knowledge of potential attacks (bows, swords, maces) to smelt and cast the armor needed to protect the warriors of their house or city-state. (Gondor calls for aid!) As their house gains glory and prominence, the standing army a blacksmith needs to protect gets larger, requiring a sustainable way to produce and test the armor used in battle.

But, what house banner do modern security practitioners often find themselves protecting? To state some facts about how the pieces are set:

• The adoption and scale of cloud infrastructure is expanding rapidly.

• Amazon Web Services is the market leader for cloud infrastructure and service.

• Security Teams are often tasked with securing the data within AWS.

• A critical component of securing a cloud environments is quickly and effectively detecting malicious activity.

From here, things get murkier. If you’re hired to build out a detection & response function for a growing SaaS startup, where do you start? What if you have scars from trying to build an internal security stack in the past? Or, perhaps, scars from implementing a clunky, expensive tool that wasn’t a fit for your growing AWS environment?

This content is meant to provide some helpful guidance when it comes to creating and maintaining a viable detection & response function for an AWS-centric environment. This guide is meant to be introductory, but we’ve linked to relevant articles and technical documentation throughout - oh, and some gifs.

Crafting detections for AWS and crafting armor come with similar challenges:

• Which materials are best suited to the job and how can they be refined for applied use?

• Which tools should I use to build my defenses?

• How do I sustainably scale and improve production to meet the needs of a growing organization?

The Detection Blacksmith - Created by Dall-E 2

In the next few sections, we’ll lay out how to build a detection & response function for an AWS environment in the way a magical smith of yore might fashion Mithril shirts for a key battle against the forces of evil.

Part I: Refining Relevant Materials

As a blacksmith, the first thing you’ll need is relevant material.

• What types of raw material are available? Iron? Steel? Leather?

• What state are the materials in? Is the iron rusty? Will they need refinement?

Security teams are faced with a similar set of questions when tasked with securing an AWS environment, but instead of raw physical materials they’re supplied with logs. Logs are the raw materials that must be fashioned into usable detections.

For a blacksmith, the most common materials will most likely be steel and leather. However, the smith may also need specialized metals for certain use cases.

If a blacksmith has iron and leather, a security engineer has logs

For the security engineer, the most common material will be CloudTrail logs. These logs are the “steel” that gives teams the strength and ability to analyze every single activity (user, role, service, and even API) that occurs within an AWS environment. Properly configured, CloudTrail enables the security teams to discover and troubleshoot both operational and security issues. Given their breadth of service coverage and volume, there are many ways to use CloudTrail logs in practice - so it’s important to collect and protect them.

There are also cases when a security engineer would want to supplement CloudTrail logs with specific service logs to enable more granular detection. For example, since it’s still common to hear about unknowingly misconfigured S3 buckets, it can be beneficial to collect S3 Server Access logs to provide context into public S3 access. Or, given the use of RDP and SSH for high sensitivity remote traffic, it can be good to bring in VPC flow logs to highlight these connections to the AWS environment. Security engineers may also want to ingest CloudWatch logs. CloudWatch and CloudTrail are both AWS-wide logs, but CloudWatch is typically used for application monitoring and metrics. CloudWatch can help security engineers react to resource changes and fire alarms for certain events.

The supplementary logs a team chooses to ingest will depend on the AWS services deployed - if an organization is leveraging container infrastructure, they might include EKS logging. Or, if they’re using AWS security services like GuardDuty or Web Application Firewall, logs from those services also provide relevant context.

Unstructured, unorganized AWS logs are like rusty iron

For materials to be workable, they need to be melted down into a uniform substance that can applied quickly into any armor mold — an arduous task. It’s interesting to remember that the Bronze Age and Iron Age are named in part due to the heating ability of the forges in each age. Bronze Age forges could only reach temperatures hot enough to melt bronze. In order to smelt iron into steel, forges need to reach a certain temperature that wasn’t possible before the blast furnace came to prominence at the beginning of the Iron Age.

By normalizing AWS logs, you remove rust and add carbon to that iron - producing highly usable steel

For security teams, in order for logs to be usable they need to be ingested, normalized and organized effectively. However, the same technological constraints remain in place today. The forge in today’s SIEM tools is the ingestion and storage mechanism used to bring in the logs. There are SIEM tools today that are stuck in the Bronze Age - leveraging out-of-date technology leading to expensive ingest and difficult log normalization. When thinking about organizing logs, it’s important to use tools built in the Cloud Age, with modern architecture, specifically for cloud environments. SIEM tools leveraging a serverless backend and/or a security data lake are likely to be much more efficient and scalable when it comes to ingesting and organizing log data.

Part II: Enemy Tactics & Tools of the Trade

It’s time for the smith to heat, hammer, and craft them into relevant pieces of armor for the elves, dwarves, and humans to fend off the forces of a ring-crazed, eye in the sky - i.e. attackers.

A good blacksmith understands the weapons of their enemy

Before swinging the hammer, it’s important to understand common tactics used by the enemy and typical attacks and weapons they may use. Before writing that first detection, reviewing common AWS incidents or understanding theoretical responses to many AWS attacks will be helpful in conceptualizing the detections needed. Both MITRE ATT&CK matrix for cloud infrastructure and CIS AWS Foundations Benchmark are great places to start when laying out baseline requirements.

Blacksmith reviews enemy weapon designs - Dall-E 2

There are helpful, standard armor casts, but the creative craftsperson might want to customize a piece

After understanding the tactics, the trick for the smith is crafting useful basic armor but also implementing creative custom pieces when required. In many cases, standard armor would suffice (a one-size-fits-all helm for example).

However, custom armor designs may be required as well. For example, should giants pledge their swords to the cause, customization and creativity will be required to hammer new armor molds to their size. A smith needs both standard molds and also a proper hammer to craft new ones.

The same combination is valuable for a security engineer crafting detections for their environment. When possible, it’s helpful to leverage out-of-the-box detection logic. For AWS specifically, detections for highly sensitive activities like root logins, root password resets, or logins without MFA should come ready to turn on. Having a handy list of relevant detections in your back pocket is a nice way to start building out defenses. In addition, SIEMs should also provide easy-to-follow mappings to standard attacker techniques. A great example would be the previously mentioned MITRE ATT&CK framework. These features make it easier for security teams to ensure they cover the most common use cases with their detections.

For security engineers, Python is a versatile hammer for crafting

It is important to note that security engineers also need a “hammer,” or a way to customize detections for their specific environment or use cases. Black box detection logic that can’t be edited or modified often leads to frustrating false positives because detections don’t “fit” a given AWS environment. Given the number of AWS services and their potential configurations, it is important for security to have the ability to look under the hood to fix and tune detections to their situation. Moreover, when it is time to customize a detection, writing detections in a highly expressible language like Python is incredibly valuable. To cap off the analogy, Python is a great hammer.

Part III: Scaling The Process

While it may seem like the blacksmith’s job ends when a piece of armor goes out the door, that’s really just the beginning. There are a few key questions that must be answered to effectively win the war and not just one battle:

• How does armor production scale with different needs and different battles?

• How effective is the armor? Can we test it? Are there ways to improve it?

For a smith overseeing armor production for numerous battles, the process must adapt quickly. Being able to seamlessly re-use old armor molds with new forges and quickly get a new production up and running is critical to making sure defenses are ready when the new battle begins. Moreover, it is critical to improve armor over time. The smith needs to understand which armor design is being used and how each design is performing by testing it.

The last thing a blacksmith would want is to send a shipment of faulty armor to the frontlines. Therefore, careful testing of armor before it sees battle is important. Once a piece of armor is tested and validated, getting it into the hands of our heroes as quickly as possible may turn the tides of an important battle.

Detection-as-Code helps sustainably craft and deploy detections into battle

For our heroic security teams, scale and testing can be accomplished via Detection-as-Code. One key benefit of the Detection-as-Code revolution is code reuse. When DevOps spins up a new AWS service or additional infrastructure (or orc battle), the security team has a wealth of detection code at the ready to call on in the new situation - saving them time from re-writing logic.

By implementing Detection-as-Code, security engineers can version control their detections and thoroughly test them. This functionality is critical to improving or tuning the logic - and for adding the code to a CI/CD pipeline. Finally, just like testing armor leads to a reduction in injuries, testing detections before implementing them improves their efficacy. With Detection-as-Code, it is much easier to do unit testing or test using replayed data before deploying detections to production. By building a strong culture of testing and tuning detections, security teams can ensure they’re constantly improving their posture as they face the day-to-day challenges of securing their AWS environment.

Conclusion: Defending Your Banner

As a modern, cloud-native SIEM, Panther is built to help security teams defend AWS. If you’d like to make the first step on your quest to defend your kingdom, try Panther’s free trial here. In the trial, you can experiment with ingesting logs, leverage out-of-the-box AWS detections, and craft your own custom detections in Python.

In the same way it is intimidating for a blacksmith to be tasked with outfitting an army to charge at the Black Gate, it is no small ask for security teams to secure ever-changing, ever-growing AWS environments. However, just as a blacksmith can start with only raw materials and end up producing weapon-tested, custom armor for a variety of battle circumstances - we can help security engineers make the same transition from unorganized AWS logs to a sustainable Detection-as-Code practice. As always, we’re here to support you.

Sagon Mijegon Aūgagon | Be Without Fear
-High Valyrian Proverb

  Blacksmith and Panther - Dall-E 2