Five years ago I joined Airbnb’s Security team to create a system for analyzing high volumes of critical security data and identifying suspicious behaviors that could indicate a breach. This system is known as a SIEM, which stands for Security Information and Event Management, and is one of the primary tools used by detection and response teams to secure enterprise environments.
Over the past 15 years the SIEM market has gone through multiple phases of evolution, from traditional “boxes”, to log analytics platforms, to the cloud-centric solutions offered today. At Airbnb, we evaluated all of the industry-leading options and ultimately decided to build our own, called StreamAlert. In this blog, I’ll explain the motivation behind that decision and how it inspired me to start Panther in August 2018.
In the beginning, traditional SIEMs like ArcSight and QRadar were introduced to ingest data and output alerts from built-in detection rules. But for years teams struggled to produce high quality alerts and handle the increasingly larger volumes of data companies produced as they scaled.
When these limitations became clear, the industry gravitated towards general-purpose log analytics solutions like Splunk, Sumo Logic, and Elastic that were designed for exploration of all types of log data. With some added development work, security teams could use these tools to collect and analyze data for security investigations and threat detection. Eventually, purpose-built SIEM offerings were introduced by these vendors, but the fundamental issues remained: Their architectures weren’t built for the cloud, and as data volumes and attack surfaces continued to explode, these tools became cumbersome and expensive to operate.
As we evaluated our options at Airbnb, we felt like the problems with traditional SIEMs had manifested themselves yet again and we were back to square one.
Founded in 2008, Airbnb is a quintessential modern tech company. Its production environment was born in Amazon Web Services and therefore has a very different set of requirements for security. Instead of data centers and on-premise applications, the focus is virtual machines, containers, and SaaS applications that power the business.
When I joined the Computer Security Incident Response Team (CSIRT) team, our goal was to build a cloud-native SIEM that could analyze Terabytes of data per day and enable a small team to detect threats faster with more expressivity than was ever possible before. After learning from our past experiences at other large tech companies (e.g. Facebook, Yahoo, Dropbox), our team decided to take a new approach and not use a log analytics platform. Instead, we set out to use the power of cloud services to avoid the pains of the past and created an open source project called StreamAlert, which I announced at Enigma 2017.
StreamAlert is a serverless, data analysis framework that analyzes logs in real-time. It works by comparing data against Python-based rules and sending alerts when matches are found. StreamAlert addressed many of the challenges we had at the time: It was easy to deploy, could be customized to fit internal business use-cases, and was less expensive to operate than incumbent solutions. One of the primary ways we drove down costs was by using Serverless technology (or FaaS), which offers a modern application hosting paradigm where developers can simply provide packaged code and leverage cloud services for orchestration, horizontal/vertical scaling, and reliability. In essence, Serverless is an abstraction above running a container-based application in a platform like Kubernetes or ECS that enables a system to scale with configuration options rather than infrastructure changes.
For two years I led the development of StreamAlert and helped grow Airbnb’s security engineering team from one to four members, complemented by several analysts. We fostered an open source community, scaled our internal deployment capacity to power Airbnb’s growing incident response (IR) program, and added new features that served both our internal needs and the needs of those in our community. As the project grew, I continued to see more and more high-tech companies adopting StreamAlert to alleviate the pain they also felt with the status quo for SIEM. This fueled my desire to work on the problem full-time and take the concept we developed at Airbnb to the next level.
In 2018, I left Airbnb to start Panther with the goal of redesigning StreamAlert to avoid the pitfalls I had discovered while operating the system at scale. Because StreamAlert is 100% command-line driven, security analysts often struggle to create new detections. In addition, over the years the platform accrued technical debt that’s resulted in issues with its data infrastructure and limited its ability to scale.
My goal with Panther was to recruit an A-team of engineers to build an Enterprise-grade detection platform that offered an intuitive interface, was designed for 10x scale, and was flexible enough to support the wide range of use cases security teams commonly struggle with today.
In early 2020, our team open sourced Panther v1, which incorporates many of the features people love about StreamAlert while also introducing a number of notable improvements, such as:
One of my foundational beliefs is that security teams should focus on detecting the bad guys, not operations. With Panther, more security teams can make this a reality.
The world is shifting from analysts and dashboards to automation and code. By enabling security teams to operationalize massive volumes of security data with cloud-first architectures and developer-driven workflows, I believe Panther can serve as the foundation for modern organizations to quickly bootstrap detection and response programs and secure cloud environments.
Today, Panther is a dedicated team of 20+ people and we just raised $15M to keep growing. The platform already supports a wide variety of use cases and data integrations and is well-positioned to innovate quickly on new ideas from our community. A few recent features we’re excited to introduce include:
My goal when starting Panther was to help all security teams deploy a SIEM that works at a modern scale. My time at Airbnb taught me that it was possible to decommission older platforms like Splunk in favor of new tools that follow the cloud-native paradigm validated by StreamAlert, and now Panther. Panther is the next step in this journey, and I’ve incorporated all of the lessons learned from my time at Airbnb and combined it with the diverse backgrounds of my engineering team.
I’m truly proud of what we’ve built thus far and what we’ll continue to build in the future. We’ll relentlessly push the traditional boundaries of SIEM to help teams prevent breaches at a cloud-scale.
To see for yourself, Run Panther!
