Have you ever needed to grab a log from a local server that is not part of the Windows Event Channel? Applications like IIS or DNS can write their logs to a local file, and you need to get them into your centralized logging server for correlation and visualization.
Graylog sidecar can help by creating and managing a centralized configuration for a filebeat agent, to gather these types of logs across all your infrastructure hosts. Graylog Sidecar can run on both Linux and Windows devices, but in this article, we will discuss the Windows version.
WHAT IS FILEBEAT USED FOR?
Filebeat is used for the collection of local text files, not present in the Microsoft event channel logs. For this example, we will use the DNS Query logging collection, but the process can be applied to any flat text file collection.
Filebeat allows for the collection of the local files while maintaining their position on the collection, so you don’t end up re-gathering the same logs again and again. The position is also needed to be kept across service restarts or system reboots to ensure no logs are left behind so that everything is sent to Graylog for long term retention.
CONFIGURATION OF GRAYLOG SIDECAR FOR FILEBEAT
After you know the location of the logs you want to collect by the filebeat agent, we can configure Graylog to do the collection. In this example, sidecar has been installed on a Windows host and is checking in already, so we need to configure the input and the collection of the logs.
First, we need to create the input on the Graylog server, at System -> Inputs. Drop down the Select input and select Beats from the menu, and pick “Launch new input”
Fill out the details, by selecting the node to start the listener on, or select the Global option, and pick the port for the listener to start on. BEATS by default is TCP/5044.
Click Save and the input should start up, noted with a green “1 RUNNING” box next to the name. Now we need to configure the Sidecar. System -> Sidecars, we can select “Configuration” in the upper right and pick “Create Configuration”
We give the Configuration a name and pick “filebeat on Windows” as the Collector from the dropdown. We need to change the configuration in two locations.
Hosts: Change IP to the IP of the graylog node you set up the input, on port 5044
paths: Pick the location where your logs are located at. You can include wildcards to pull all files with an extension as well, as shown in the example:
Click “Create” after this is done, and go to the “Administration” button to apply the new configuration to the agent. Next to the filebeat agent by the system you want to collect from, select the box, and then pick Configure -> Policy Name. This will pop up a window and confirm to apply.
Validate logs are coming in by using the search window.
CONCLUSION
Any text file can be collected by the filebeat agent, and some additional parsing and processing of the logs may be needed to get the full value from the logs. Pipelines are a great spot to start using to expand these text files into more useful data.
