Logs are a wealth of information containing meta-data from IP addresses, User Names, and error codes. While this is all extremely helpful, the task of understanding all this can seem overwhelming at times to an untrained eye.
Other times, corporations might have additional resources they would like to enrich their logs with, i.e., adding a department name to a log message that depends on the username in the log. This could allow for grouping and visualization of logs based on this new field which was not included in the log.
Graylog has a few built-in lookup tables and common adapters to read in the data from various locations. The most common is usually a GeoIP lookup, taking an IP address and bouncing it off a list of information around that IP address. You can also upload your own data in a CSV file for instance to do the enrichment, which is what we will cover in this article.
Let’s go over the steps needed to configure your lookup table.
CONFIGURATION
The first step to using our CSV file will be to upload it to the Graylog server(s) and ensure it has the right permissions on it, to be read by Graylog. In my example, I will be putting it into:
/etc/graylog/lookup_error.csv
In this example, the file contents look like this:
Now we need to configure the Lookup Table to read the CSV when an input is given to it. We will first create the data adapter under System -> Lookup Tables and Select “Data Adapters.” On this page, select “Create data adapter,” and fill in one like below:
Next, we need to create a Cache with the “Create cache” button like below:
In the last step in the Lookup Table, we need to create the table itself, using the created data adapter and cache from the last two steps:
Now that the lookup table has been created and ready for use, we need to create a pipeline rule to utilize it and add in the metadata to each message with an IP address.
Go to (System -> Pipelines) and under the “Manage rules,” we will need to create a new rule. Give it a description so you can remember it, and in the Rule Source put:
rule “Windows: Error Log Cleanup”
when
has_field(“SubStatus”) AND contains(to_string($message.EventID), “4625”)
then
//Change Logon Code
let update_source = lookup_value(“lookup_error”, $message.SubStatus);
set_field(“SubStatus”, update_source);
end
The when condition is only allowing the rule to process when the log has the field SubStatus and has the EventID of 4625. NOTE: Please change this to the field name containing your SubStatus and EventID, as they can be different names. Once it finds those logs, it runs the lookup against our Lookup Table “lookup_error” with the data in the SubStatus field, then changes SubStatus to something more understandable.
Once the rules are created, add them to a stage in the pipeline processing the logs with an IP address in them, and the enrichment will start working.
And once new logs will come through the pipeline, you can see the enriched log entry:
The original log will still contain the Hex code as shown:
ADDITIONAL THOUGHTS
You can use this lookup table functionality in many ways, from changing error codes into something more readable, to adding in metadata fields from a CMDB repository to help add more context to the events as they happen.
