We regret to inform you that a malicious third party may have had read access to the secret data you stored in our system.
Any time we get a communication like the above, as recently occurred with CircleCI or LastPass, several security processes need to execute as quickly as possible. One, identify any exposed credential data. Two, audit the use of any exposed credential data. Three, begin creating and installing replacement credential data as quickly as possible. This blog will demonstrate a detective solution for Step 2, which you can put in place before a security incident is declared.
In a Continuous Integration and Delivery system, the implications of exposed credentials can be severe. CI/CD tasks necessarily leverage resource creation and deletion privilege. In a reduced-risk world, build and deployment systems leverage short-lived credentials across pre-configured trust boundaries. In the world that we live in, APIs get integrated with authentication tokens that never expire.
This blog will provide an example of an advanced, proactive detection for identifying exfiltrated credentials as used in Continuous Integration/Continuous Delivery systems.
We must establish the criteria for unauthorized use to detect authentication token exfiltration. Then we can start creating theses (and tests!) to confirm or invalidate our idea. If our theories pan out, we codify them as detections.
Let’s begin by stating our hope for the ideal state. “I expect all activity from $SpecialIdentity in Infrastructure as a Service (IaaS) to be the result of $SomeTask executing in my Continuous Integration and Continuous Delivery (CICD) system.” We can assert this accurately for all conditions with enough shared metadata between our IaaS and CICD systems. The data that we have may be insufficient to create formal proofs.
In a typical environment, the CICD and IaaS systems align exclusively via timestamps. There is no additional metadata that can be used to cross-reference events in the systems. Using this constraint, we will write technical expectations.
Our hypothesis is built with these expectations:
Panther with Snowflake provide a query capability that matches a pattern in a series of rows. This capability is called MATCH_RECOGNIZE. If you have used Regular Expressions, you’ll likely find MATCH_RECOGNIZE familiar. In this example, we will use the regular expression repetition operators * (meaning zero or more) and + (meaning 1 one or more).
The PATTERN() statement below declares how to group events along our timeline. This PATTERN says zero or more rows identified as row_with_CICD_events followed by one or more rows identified as same_IaaS_session. We will surface when to alert and when to ignore based on the event grouping later in our SQL statement.
The DEFINE statement declares how rows should be classified for the PATTERN.
The HAVING statement declares that we only want groupings where we did not find the CICD events happening before the IaaS events.
SELECT
…
MATCH_RECOGNIZE (
ORDER BY p_event_time ASC
…
PATTERN(row_with_CICD_events* same_IaaS_session+)
DEFINE
current_row AS true,
-- Assume sessions in IaaS will last less than 10 minutes
same_IaaS_session AS IaaS_event_name IS NOT NULL
AND DATEDIFF(MINS, lag(p_event_time), p_event_time) <= 10,
row_with_CICD_events AS count_CICD_events > 0
)
HAVING num_CICD_events = 0Code language: SQL (Structured Query Language) (sql)We will assert that our detection works as expected by stubbing in some test data. In this data, there are three sets of test data. The test data and what they demonstrate are described below. For this detection, the alert condition is when the num_CICD_events column in our final output has a value of 0.
These are our test data with explanations about what is demonstrated by them:
WITH CICD_events AS (
SELECT
$1 AS p_event_time,
$2 AS count_CICD_events,
-- We add a column for IaaS_event_name column in CICD_events
-- here in the CICD_events so that we can later UNION CICD to IaaS
$3 AS IaaS_event_name
FROM (VALUES
-- First event in CICD system.
('2023-01-31 01:00:00'::timestamp_ntz,1,NULL),
-- Second event in CICD system.
('2023-01-31 02:00:00'::timestamp_ntz,1,NULL),
-- Third event in CICD system.
-- Note this record is interleaved with the IaaS events for this hour
('2023-01-31 03:05:55'::timestamp_ntz,1,NULL)
)
),
IaaS_events AS (
SELECT
$1 AS p_event_time,
$2 AS count_CICD_events,
$3 AS IaaS_event_name
FROM (VALUES
-- First group of events in IaaS
('2023-01-31 01:11:32'::timestamp_ntz,0,'StartSession'),
('2023-01-31 01:12:32'::timestamp_ntz,0,'ListResources'),
-- Second group of events in IaaS
-- Note that this time window will be 15 minutes long because there is no
-- gap of more than ten minutes gap between any adjacent events.
('2023-01-31 02:06:32'::timestamp_ntz,0,'StartSession'),
('2023-01-31 02:06:32'::timestamp_ntz,0,'ListResources'),
('2023-01-31 02:10:32'::timestamp_ntz,0,'UpdateResource'),
('2023-01-31 02:11:32'::timestamp_ntz,0,'DescribeResource'),
('2023-01-31 02:15:32'::timestamp_ntz,0,'WriteLog'),
-- Third group of events in IaaS
-- Note that the timing of the first IaaS event is one minute
-- ahead of the event from the build system.
-- Uncomment line below to demonstrate
--('2023-01-31 03:03:32'::timestamp_ntz,0,'SendMFAToken'),
('2023-01-31 03:04:32'::timestamp_ntz,0,'StartSession'),
('2023-01-31 03:06:32'::timestamp_ntz,0,'ListResources'),
('2023-01-31 03:10:32'::timestamp_ntz,0,'UpdateResource'),
('2023-01-31 03:11:32'::timestamp_ntz,0,'DescribeResource'),
('2023-01-31 03:15:32'::timestamp_ntz,0,'WriteLog')
)
),
all_events AS (
SELECT * FROM CICD_events
UNION ALL
SELECT * FROM IaaS_events
)
SELECT * FROM all_events
MATCH_RECOGNIZE (
ORDER BY p_event_time ASC
MEASURES
match_number() AS match_number,
classifier() AS classifier,
first(p_event_time) AS start_time,
last(p_event_time) AS end_time,
count(*) + 1 AS rows_in_sequence,
coalesce(count(row_with_CICD_events.*), 0) AS num_CICD_events,
count(same_IaaS_session.*) + 1 AS num_IaaS_events
ONE ROW PER MATCH
-- read as
-- ZERO OR MORE ANTECEDENT EVENTS
--- followed by
-- AT LEAST ONE CONSEQUENT EVENT (AS A SESSION)
-- consider CONSEQUENT sessions to be valid for 10 minutes
PATTERN(row_with_CICD_events* same_IaaS_session+)
DEFINE
-- Assume IaaS events are in the same session if there’s a gap
-- of less than or equal to ten minutes between events
same_IaaS_session AS IaaS_event_name IS NOT NULL
AND datediff(MINS, lag(p_event_time), p_event_time) <= 10,
row_with_CICD_events AS count_CICD_events > 0
)
-- num_CICD_events will be greater than zero
-- if the leading event is found
HAVING num_CICD_events >= 0
ORDER BY start_time ASCCode language: SQL (Structured Query Language) (sql)A tangible application for this detection is GitHub Actions jobs integrated with AWS. In this detection, we want to alert if the AWS IAM Role arn:aws:iam::123456789012:role/DeploymentUpdateGitHubRole has activity and there was not a preceding pull_request.merge in the repo panther-labs/example-repo. At the end of the statement, we set HAVING num_CICD_events = 0 to ensure our output only contains suspicious windows of IAM activity.
WITH CICD_events AS (
SELECT
p_event_time,
1 AS num_github_target_events,
null AS cloudtrail_event_name
FROM
panther_logs.public.github_audit
WHERE
p_occurs_since('3 DAY')
AND
-- you can target a specific github action
--action = 'workflows.created_workflow_run'
--AND
--name = 'CI'
-- maybe you want pull_request.merge instead
action = 'pull_request.merge'
AND
repo = 'panther-labs/example-repo'
),
IaaS_events AS (
SELECT
p_event_time,
0 AS num_github_target_events,
eventname AS cloudtrail_event_name
FROM panther_logs.public.aws_cloudtrail
WHERE
p_occurs_since('3 DAY')
AND
arrays_overlap(
p_any_aws_arns,
array_construct(
'arn:aws:iam::123456789012:role/DeploymentUpdateGitHubRole'
)
)
AND
errorcode IS NULL
),
all_events AS (
SELECT * FROM CICD_events
UNION ALL
SELECT * FROM IaaS_events
)
SELECT * FROM all_events
MATCH_RECOGNIZE (
ORDER BY p_event_time ASC
MEASURES
match_number() AS match_number,
classifier() AS classifier,
first(p_event_time) AS start_time,
last(p_event_time) AS end_time,
count(*) + 1 AS rows_in_sequence,
coalesce(count(row_with_CICD_events.*), 0) AS num_CICD_events,
count(same_IaaS_session.*) + 1 AS num_IaaS_events
ONE ROW PER MATCH
-- read as
-- ZERO OR MORE CICD EVENTS
--- followed by
-- AT LEAST ONE IaaS EVENT (AS A SESSION)
-- consider IaaS sessions to be valid for 10 minutes
PATTERN(row_with_CICD_events* same_IaaS_session+)
DEFINE
current_row AS true,
-- Assume auths into IaaS will can last less than 10 minutes
same_IaaS_session AS cloudtrail_event_name IS NOT NULL
AND datediff(MINS, lag(p_event_time), p_event_time) <= 10,
row_with_CICD_events AS num_github_target_events > 0
)
-- Limiting results to HAVING num_CICD_events=0
-- means each row is an alert scenario.
-- IaaS events are found
-- CICD events are not found
HAVING num_CICD_events = 0
ORDER BY start_time ASCCode language: SQL (Structured Query Language) (sql)Detecting unauthorized use of authentication tokens is critical for security in CI/CD systems, especially in the event of a security incident. By establishing criteria for unauthorized use and codifying them as detections, security teams can proactively monitor for potential threats and take action before any damage occurs. The example presented here demonstrates how such detections can be implemented and tested. We hope that you will integrate this detective approach as a part of your security practice.
