Discovering Exfiltrated Credentials

We regret to inform you that a malicious third party may have had read access to the secret data you stored in our system.

Any time we get a communication like the above, as recently occurred with CircleCI or LastPass, several security processes need to execute as quickly as possible. One, identify any exposed credential data. Two, audit the use of any exposed credential data. Three, begin creating and installing replacement credential data as quickly as possible. This blog will demonstrate a detective solution for Step 2, which you can put in place before a security incident is declared.

In a Continuous Integration and Delivery system, the implications of exposed credentials can be severe. CI/CD tasks necessarily leverage resource creation and deletion privilege. In a reduced-risk world, build and deployment systems leverage short-lived credentials across pre-configured trust boundaries. In the world that we live in, APIs get integrated with authentication tokens that never expire.

This blog will provide an example of an advanced, proactive detection for identifying exfiltrated credentials as used in Continuous Integration/Continuous Delivery systems.

Detecting the Risk

We must establish the criteria for unauthorized use to detect authentication token exfiltration. Then we can start creating theses (and tests!) to confirm or invalidate our idea. If our theories pan out, we codify them as detections. 

Let’s begin by stating our hope for the ideal state. “I expect all activity from $SpecialIdentity in Infrastructure as a Service (IaaS) to be the result of $SomeTask executing in my Continuous Integration and Continuous Delivery (CICD) system.”  We can assert this accurately for all conditions with enough shared metadata between our IaaS and CICD systems. The data that we have may be insufficient to create formal proofs. 

In a typical environment, the CICD and IaaS systems align exclusively via timestamps. There is no additional metadata that can be used to cross-reference events in the systems. Using this constraint, we will write technical expectations. 

Our hypothesis is built with these expectations:

  1. IaaS Activity
    1. We will only use IaaS event logs for $SpecialIdentity
    2. All IaaS sessions of interest will contain at least two events. We will ignore any single event with no neighbor +/- 10 minutes
    3. Events separated by more than ten minutes are separate sessions. 
  2. CICD Activity
    1. We will only use CICD event logs for $SomeTask
    2. If $SomeTask executes fully in CICD, CICD will call IaaS within ten minutes.
    3. It is acceptable for $SomeTask to execute without calling IaaS. The CICD test suite might sometimes fail.

Implementing our hypothesis with SQL

Panther with Snowflake provide a query capability that matches a pattern in a series of rows. This capability is called MATCH_RECOGNIZE. If you have used Regular Expressions, you’ll likely find  MATCH_RECOGNIZE familiar. In this example, we will use the regular expression repetition operators * (meaning zero or more) and + (meaning 1 one or more).

The MATCH_RECOGNIZE portion of our query

The PATTERN() statement below declares how to group events along our timeline. This PATTERN says zero or more rows identified as row_with_CICD_events followed by one or more rows identified as same_IaaS_session.  We will surface when to alert and when to ignore based on the event grouping later in our SQL statement.

The DEFINE statement declares how rows should be classified for the PATTERN. 

The HAVING statement declares that we only want groupings where we did not find the CICD events happening before the IaaS events. 

SELECT
…
MATCH_RECOGNIZE (
    ORDER BY p_event_time ASC
    …  
    PATTERN(row_with_CICD_events* same_IaaS_session+)
    DEFINE
      current_row AS true,
      -- Assume sessions in IaaS will last less than 10 minutes
      same_IaaS_session AS IaaS_event_name IS NOT NULL
        AND DATEDIFF(MINS, lag(p_event_time), p_event_time) <= 10,
      row_with_CICD_events AS count_CICD_events > 0
)
HAVING num_CICD_events = 0Code language: SQL (Structured Query Language) (sql)

Putting it all together with tests

We will assert that our detection works as expected by stubbing in some test data. In this data, there are three sets of test data. The test data and what they demonstrate are described below. For this detection, the alert condition is when the  num_CICD_events column in our final output has a value of 0.

These are our test data with explanations about what is demonstrated by them:

  • 01:00 Hour data
    • This data is an alert scenario, indicated by the 0 value in the num_CICD_events column
    • This data is a CICD event that is followed by an IaaS event. 
    • The 11-minute gap between the CICD event and the IaaS event is greater than our threshold.
  • 02:00 hour data
    • This data is not an alert scenario
    • This data is a CICD event that is followed by an IaaS event
    • There is a 6-minute gap between the two systems, which is inside our tolerance.
    • The IaaS session in this data lasts a total of 15 minutes. This demonstrates that IaaS sessions can hold arbitrary lengths, as long as there is no gap between events longer than 10 minutes. 
  • 03:00 hour data
    • This data is not an alert scenario
    • This data is a single IaaS event followed by a single CICD event followed by several more IaaS events
    • There is a 4-minute gap between the CICD event and the following IaaS events, which is within our tolerance.
    • The single IaaS event before the CICD event is ignored because we consider that every IaaS session of interest will have at least two events. 
    • This data will become an alert scenario if you uncomment the SendMFAToken IaaS event
WITH CICD_events AS (
  SELECT
    $1 AS p_event_time,
    $2 AS count_CICD_events,
    -- We add a column for IaaS_event_name column in CICD_events
    -- here in the CICD_events so that we can later UNION CICD to IaaS
    $3 AS IaaS_event_name
 FROM (VALUES 
   -- First event in CICD system.
   ('2023-01-31 01:00:00'::timestamp_ntz,1,NULL),
   -- Second event in CICD system. 
   ('2023-01-31 02:00:00'::timestamp_ntz,1,NULL),
   -- Third event in CICD system. 
   -- Note this record is interleaved with the IaaS events for this hour
   ('2023-01-31 03:05:55'::timestamp_ntz,1,NULL)
 )
),

IaaS_events AS (
  SELECT 
    $1 AS p_event_time,
    $2 AS count_CICD_events,
    $3 AS IaaS_event_name
 FROM (VALUES 
   -- First group of events in IaaS
   ('2023-01-31 01:11:32'::timestamp_ntz,0,'StartSession'),
   ('2023-01-31 01:12:32'::timestamp_ntz,0,'ListResources'),
   -- Second group of events in IaaS 
   -- Note that this time window will be 15 minutes long because there is no 
   -- gap of more than ten minutes gap between any adjacent events. 
   ('2023-01-31 02:06:32'::timestamp_ntz,0,'StartSession'),
   ('2023-01-31 02:06:32'::timestamp_ntz,0,'ListResources'),
   ('2023-01-31 02:10:32'::timestamp_ntz,0,'UpdateResource'),
   ('2023-01-31 02:11:32'::timestamp_ntz,0,'DescribeResource'),
   ('2023-01-31 02:15:32'::timestamp_ntz,0,'WriteLog'),
   -- Third group of events in IaaS
   -- Note that the timing of the first IaaS event is one minute 
   -- ahead of the event from the build system. 
   -- Uncomment line below to demonstrate 
   --('2023-01-31 03:03:32'::timestamp_ntz,0,'SendMFAToken'),
   ('2023-01-31 03:04:32'::timestamp_ntz,0,'StartSession'),
   ('2023-01-31 03:06:32'::timestamp_ntz,0,'ListResources'),
   ('2023-01-31 03:10:32'::timestamp_ntz,0,'UpdateResource'),
   ('2023-01-31 03:11:32'::timestamp_ntz,0,'DescribeResource'),
   ('2023-01-31 03:15:32'::timestamp_ntz,0,'WriteLog')
 )


),

all_events AS (
    SELECT * FROM CICD_events
    UNION ALL
    SELECT * FROM IaaS_events
)


SELECT * FROM all_events
MATCH_RECOGNIZE (
    ORDER BY p_event_time ASC
    MEASURES
    match_number() AS match_number,
    classifier() AS classifier,
    first(p_event_time) AS start_time,
    last(p_event_time) AS end_time,
    count(*) + 1 AS rows_in_sequence,
    coalesce(count(row_with_CICD_events.*), 0) AS num_CICD_events,
    count(same_IaaS_session.*) + 1 AS num_IaaS_events
    ONE ROW PER MATCH
    -- read as
    -- ZERO OR MORE ANTECEDENT EVENTS 
    --- followed by 
    -- AT LEAST ONE CONSEQUENT EVENT (AS A SESSION)
    --  consider CONSEQUENT sessions to be valid for 10 minutes
    PATTERN(row_with_CICD_events* same_IaaS_session+)
    DEFINE
    -- Assume IaaS events are in the same session if there’s a gap
    -- of less than or equal to ten minutes between events
    same_IaaS_session AS IaaS_event_name IS NOT NULL
    AND datediff(MINS, lag(p_event_time), p_event_time) <= 10,
    row_with_CICD_events AS count_CICD_events > 0
)
-- num_CICD_events will be greater than zero 
-- if the leading event is found
HAVING num_CICD_events >= 0
ORDER BY start_time ASCCode language: SQL (Structured Query Language) (sql)

Putting it all together for the Real World

A tangible application for this detection is GitHub Actions jobs integrated with AWS. In this detection, we want to alert if the AWS IAM Role arn:aws:iam::123456789012:role/DeploymentUpdateGitHubRole has activity and there was not a preceding pull_request.merge in the repo panther-labs/example-repo.  At the end of the statement, we set HAVING num_CICD_events = 0 to ensure our output only contains suspicious windows of IAM activity.

WITH CICD_events AS (
    SELECT
        p_event_time,
        1 AS num_github_target_events,
        null AS cloudtrail_event_name
    FROM
        panther_logs.public.github_audit
    WHERE
        p_occurs_since('3 DAY')
        AND
        -- you can target a specific github action
        --action = 'workflows.created_workflow_run'
        --AND
        --name = 'CI'
        -- maybe you want pull_request.merge instead 
        action = 'pull_request.merge'
        AND
        repo = 'panther-labs/example-repo'
),

IaaS_events AS (
    SELECT
        p_event_time,
        0 AS num_github_target_events,
        eventname AS cloudtrail_event_name
    FROM panther_logs.public.aws_cloudtrail
    WHERE
        p_occurs_since('3 DAY')
        AND
        arrays_overlap(
            p_any_aws_arns,
            array_construct(
                'arn:aws:iam::123456789012:role/DeploymentUpdateGitHubRole'
            )
        )
        AND
        errorcode IS NULL
),

all_events AS (
    SELECT * FROM CICD_events
    UNION ALL
    SELECT * FROM IaaS_events
)


SELECT * FROM all_events
MATCH_RECOGNIZE (
    ORDER BY p_event_time ASC
    MEASURES
    match_number() AS match_number,
    classifier() AS classifier,
    first(p_event_time) AS start_time,
    last(p_event_time) AS end_time,
    count(*) + 1 AS rows_in_sequence,
    coalesce(count(row_with_CICD_events.*), 0) AS num_CICD_events,
    count(same_IaaS_session.*) + 1 AS num_IaaS_events
    ONE ROW PER MATCH
    -- read as
    -- ZERO OR MORE CICD EVENTS 
    --- followed by 
    -- AT LEAST ONE IaaS EVENT (AS A SESSION)
    --  consider IaaS sessions to be valid for 10 minutes
    PATTERN(row_with_CICD_events* same_IaaS_session+)
    DEFINE
    current_row AS true,
    -- Assume auths into IaaS will can last less than 10 minutes
    same_IaaS_session AS cloudtrail_event_name IS NOT NULL
    AND datediff(MINS, lag(p_event_time), p_event_time) <= 10,
    row_with_CICD_events AS num_github_target_events > 0
)
-- Limiting results to HAVING num_CICD_events=0 
-- means each row is an alert scenario. 
-- IaaS events are found
-- CICD events are not found

HAVING num_CICD_events = 0
ORDER BY start_time ASCCode language: SQL (Structured Query Language) (sql)

Concluding Thoughts

Detecting unauthorized use of authentication tokens is critical for security in CI/CD systems, especially in the event of a security incident. By establishing criteria for unauthorized use and codifying them as detections, security teams can proactively monitor for potential threats and take action before any damage occurs. The example presented here demonstrates how such detections can be implemented and tested. We hope that you will integrate this detective approach as a part of your security practice.

Table of Contents

Recommended Resources

Escape Cloud Noise. Detect Security Signal.
Request a Demo