Graylog is an advanced log management system, capable of ingesting all of your corporate logs into a central repository for easy searching and analysis of your data. From Graylog’s Open Source beginnings to the powerful new Enterprise features we continue to create, we have time and time again run into the same request:
“Help me see my data right from the beginning”
At the start of 2020 Graylog assembled an Enterprise Intelligence team, a group of real-world practitioners with deep field expertise, and we got to work. For our first effort, we created usable enterprise visualizations geared around different sets of data. Six months later, we released Graylog Illuminate for Authentication.
The first release creates a foundation to normalize all authentication data, regardless of source. This gives you consistency in reporting, alerting, and analysis plus the power to easily correlate authentication data across different types of data sources. Not surprisingly our initial area of focus is around Windows authentication with the inclusion of the Windows Auth Spotlight app.
We want your input. Let us know what other authentication data you’d like to see Graylog build and maintain here.
Foundation
Before we could even start building Graylog Illuminate, we had to get a few things in place. First, we needed a way to make sure all the data, no matter the source, will be stored in the same format/layout. To do this, we created the Graylog Schema.
Graylog Schema is a blueprint for how we map data fields to a standardized field name in the product, allowing for pieces of Graylog to be developed more generically and across many technologies. For example, most users want to know when a user fails to login. Windows, Linux, Applications and VPN devices, for instance, all use different field names for the “user”, so putting in the data into a common field “user_name” allows for a Dashboard to show all “user_name” that have failed.
We also needed to include the ability to have normalized fields in the schema. To enrich the original log message, with new data not present originally. Not all logs use the same term for “Failed Login”. Some can use “Access Denied”, “Unauthorized” or “Invalid Password”. By putting the field “event_outcome” in the schema we could have a common name “success” or “failure” for all these types of logs. Now a search can be made for any event_outcome:failure and it will show up all the failed logins which have been processed by Graylog.
Processing
We have started by creating a multi-step method of log processing where the first few stages are normalizing the logs and getting them into our schema. Depending on the log source, this can be done in a different number of steps, but once the first stages are done, the logs are ready for Enrichment. We enrich the logs by adding data like Geolocation information or threat intelligence to the log message. We can utilize lookup tables to add severity to the host, or location of the asset if mapped out.
We tag IP addresses with “Internal” if they fall in the common RFC1918 address ranges, and then can limit the types of rules and dashboards they will show up in as a result. Understanding the direction will be a part of Graylog Illuminate as we expand into other areas like Networking, so we are aware if the attack is coming inbound or outbound.
Once the logs are normalized and enriched, we can write them to disk for storage. With the new schema in place, Graylog also has recommended templates for the data, to make sure different analysis can be done on the data types. We need to make sure we can “sum” bytes to get a count of data transferred or find averages of a numerical type.
Visualization
Included in Graylog Illuminate are dashboards built for the specific area of data. For this first release, we are introducing Graylog Illuminate for Authentication, so the dashboards and drill downs are all around authentication data. The first source we chose was Microsoft Windows Active Directory.
In the coming weeks, we will be adding in Okta support as an authentication source, at which point the dashboards for Authentication will show logs from both Microsoft and Okta, even combining the data where appropriate. This is the payoff for spending a lot of time on schema and normalization. Not only that but if you add your own custom data source and map it to our schema, it will automatically work with all downstream functionality.
With Graylog 3.3+, there will be Correlation and Aggregation alerts included, to find things like weak Kerberos cyphers, or Password Spraying attacks. These will continue to be developed and released with new versions.
How does this help me?
Graylog Illuminate wants to make sure you can see your data and find the items you might be missing in the noise. Making the data easy to read, allowing you to pivot around your data, can take your investigation time down from hours to seconds.
For example, we had a client who said it would take 3-4 hours to run down why an account was getting locked out over and over again. With the dashboards, and the ability to pivot into a user drill-down, they found out what machine was causing the issues and processes causing the authentication attempts in 3 min.
Authentication is the first area we are targeting, and additional expansions of the sources will continue to come, but we also want to cover other areas. Networking (Firewalls, IPS, Wireless), Endpoint (EDR, HIPS, Process Tracking), Applications (Web Server, DB) and Change Management round out the 5 pillars of coverage. We will be working to cover all of them on a high level first, and then drill down into each with product specific material. Our first example of this will be Okta. It will feed into the Authentication content but will also have its own SSO dashboards to dive deeper into what it can provide.
Conclusion
Graylog Illuminate is taking the first steps to getting you up and running with relevant information from the moment you get installed. We will continue to enhance Graylog Illuminate by adding additional product Spotlights and updating existing ones as needed.
Please join us in creating content based on the Graylog Schema once released and share all your ideas with our wonderful community!
HAPPY LOGGING!
Nick Carstensen provides a deeper dive into Graylog Illuminate and the Windows Authentication Spotlight in this recorded webinar.
